Skip to content

theseus

mmlu

Initializing search

theseus

Getting Started
Tutorials
Tutorials
- Overview
- Running Experiments
- Adding Things
  Adding Things
  - Overview
  - Model
  - Experiment
  - Dataset
  - Evaluation
Design
Design
Reference
Reference
- base
  base
  - axis
  - chip
  - hardware
  - job
  - topology
- cli
- config
- data
  data
  - datasets
    
    datasets
    
    alpaca
    
    bbq
    
    ccaligned
    
    cfq
    
    clutrr
    
    dataset
    
    fever
    
    fineweb
    
    harmfulqa
    
    longbench
    
    longhealth
    
    mmlu
    
    mnli
    
    mtob
    
    pes2o
    
    pg19
    
    pile
    
    pile_detoxify
    
    qqp
    
    redcodegen
    
    redcodegen
    
    hardening
    
    siqa
    
    squad
    
    sst2
    
    winogrande
  - tokenize
  - tokenizer
- dispatch
  dispatch
  - bootstrap
  - config
  - dispatch
  - mailbox
    
    mailbox
    
    mailbox
    
    sidecar
  - slurm
  - solve
  - ssh
  - sync
- evaluation
  evaluation
  - base
  - datasets
    
    datasets
    
    bbq
    
    blimp
    
    ccaligned
    
    cfq
    
    clutrr
    
    fever
    
    longbench
    
    longhealth
    
    mmlu mmlu
    Table of contents
    
    mmlu
    
    MMLUEval
    
    mnli
    
    mtob
    
    perplexity_evals
    
    pes2o
    
    pg19
    
    pile
    
    qqp
    
    siqa
    
    squad
    
    sst2
    
    tinystories
    
    winogrande
  - huggingface
- experiments
  experiments
  - continual
    
    continual
    
    abcd
  - models
    
    models
    
    forking
    
    gpt
    
    gpt_neox
    
    llama
    
    qwen
  - redcodegen
    
    redcodegen
    
    hardening
- inference
  inference
  - base
  - huggingface
- job
- mock
- model
  model
  - activations
    
    activations
    
    swiglu
  - attention
    
    attention
    
    base
    
    forking
    
    grouped
    
    rope
  - axes
  - block
    
    block
    
    block
    
    forking
    
    gpt_neox
    
    llama
    
    qwen
  - huggingface
  - layers
    
    layers
    
    layernorm
    
    mlp
    
    rmsnorm
    
    rope
  - masks
  - models
    
    models
    
    base
    
    contrib
    
    contrib
    
    gpt_neox
    
    llama
    
    qwen
    
    thoughtbubbles
  - module
- plot
- quick
- registry
- training
  training
  - backbone
  - base
  - contrastive
  - flywheel
    
    flywheel
    
    contrastive
    
    padded
    
    pmd
    
    strategy
  - huggingface
  - kl_divergence
  - optimizers
    
    optimizers
    
    adamw
    
    muon
  - schedules
    
    schedules
    
    wsd
    
    wsds
  - utils
- web
  web
  - app
  - auth
  - generate_password_hash
  - models
  - routes
    
    routes
    
    api
    
    auth
    
    views
  - services
    
    services
    
    cache
    
    checkpoints
    
    logs
    
    status

Table of contents

mmlu
- MMLUEval

mmlu

`mmlu` ¶

`MMLUEval()` ¶

Bases: RolloutEvaluation

MMLU evaluation (validation split, all subjects).