looped

Bases: LoopedGPT

Fully Looped Transformer: K blocks applied R times.

Each repeat restarts the residual stream from rms(x0). The rms-normed output of the previous repeat, h, is passed to every block as the query source. architecture/n_layer_parameters is K and architecture/n_layers is K * R.