Index
flt
¶
FLTLoopedGPT
¶
Bases: LoopedGPT
Fully Looped Transformer: K blocks applied R times.
Each repeat restarts the residual stream from rms(x0). The rms-normed output of the previous repeat, h, is passed to every block as the query source. architecture/n_layer_parameters is K and architecture/n_layers is K * R.
CarvedFLTLoopedGPT
¶
Bases: FLTLoopedGPT
topology() -> tuple[set[int], set[int], tuple[tuple[int, ...], ...]]
¶
Surviving attention blocks, surviving MLP blocks, skip sources per block.