siqa
siqa
¶
SIQAEval(split: str = 'validation')
¶
Bases: RolloutEvaluation
SIQA evaluation using validation split.
max_new_tokens(inference: Any) -> int
¶
Only need 1 token for A/B/C answer.