INNER CODE UNIT · Python

evaluate

Future-House/BixBench · generate_zeroshot_evals.py:93

async def evaluate(
    dataset: pd.DataFrame,
    zeroshot_agent: ZeroshotBaseline,
    output_dir: str = "results",
    output_file: str | None = None,
):
    results = []
    for _, row in dataset.iterrows():
        query = await zeroshot_agent.generate_zeroshot_answers(
            Query(
                id=row["question_id"],
                question=row["question"],
                target=row["ideal"],
                choices=row["distractors"],
                evaluation_mode=row.get("eval_mode", None),
            )
        )

View source record →