pretrain
pretrain
¶
DiversityScratchbubbles
¶
Bases: Scratchbubbles
Scratchbubbles training model with fork-diversity instrumentation.
measure_friend_diversity(x: jax.Array, token_index: jax.Array) -> jax.Array
staticmethod
¶
Measure mean cosine distance between non-rightmost siblings.
decode(x: jax.Array, padding_mask: Optional[jax.Array] = None, deterministic: bool = False, **kwargs: Any) -> Any
¶
Decode while recording states immediately after each fork.