Adapter idea: your checkpoint-based eval output maps cleanly onto a portable eval schema

[adhabnr-ux]

Hi maintainers — nice benchmark design, particularly the per-testpoint binary checkpoint scoring paired with analysis text in eval/src/eval_common.py. It's more structured than most T2I benchmarks, and it maps cleanly onto a portable eval schema, so wanted to flag the idea.

I maintain EvalPort, an open, framework-agnostic schema (TestCase / Grader / Result / ResultSet / GraderResult) for eval data, with Python (evalport-sdk on PyPI, import openeval) and TypeScript SDKs, plus adapters for other eval tools (e.g. adapters/deepeval-openeval-adapter). The goal is a common format so eval results survive a move between tools instead of being locked into one CSV shape.

Looking at your actual code, here's how the pieces line up:

• Each CSV row (index, prompt_en, sub_dims_en with its Testpoints + Testpoint Description) → an EvalPort TestCase: input = the prompt text, tags = the testpoint list, metadata carries the sub_dims categories/subjects/sentence-structure fields verbatim.
• Each of your 27 sub-dimensions (the EXPLANATION_DICT_EN/_ZH keys) → a reusable Grader of type llm_judge, one per checkpoint type, description = your explanation text, params.prompt derived from SYSTEM_PROMPT_EN_TEMPLATE/_ZH_TEMPLATE.
• Since one MLLM call already scores every testpoint for an image and returns per-checkpoint 0/1 + analysis (result_json["score"] / ["analysis"]), that's naturally one Result per (prompt, image) with one GraderResult per testpoint — score, passed = score == 1, reason = analysis[i].
• The 4 generated images per prompt each get their own Result, with actual_output = the image path and metadata.img_path for traceability.
• calculate_scores()'s primary/sub-dimension accuracy breakdown maps almost 1:1 onto ResultSet.summary — you're already computing exactly the shape a portable results file wants there.

Rough sketch, using the real dataclasses from openeval.types:

import ast, json
from openeval.types import TestCase, Grader, Result, GraderResult

def grader_id(testpoint: str) -> str:
    return f"gr_{testpoint.split(' - ')[0].lower().replace(' ', '_')}"

def testcase_from_row(row, lang):
    subdims = json.loads(row[f"sub_dims_{lang}"])
    testpoints = subdims["Testpoints"]
    return TestCase(
        id=f"unigenbench_{lang}_{row['index']}",
        input=row[f"prompt_{lang}"],
        graders=[grader_id(tp) for tp in testpoints],
        tags=testpoints,
        metadata={"sub_dims": subdims, "lang": lang},
    )

def result_from_eval_row(row, lang, image_idx):
    testpoints = ast.literal_eval(row["testpoint"])
    rj = row["result_json"]
    rj = json.loads(rj) if isinstance(rj, str) else rj
    grader_results = [
        GraderResult(
            grader_id=grader_id(tp), type="llm_judge",
            score=float(s), passed=(s == 1), reason=rj["analysis"][i],
        )
        for i, (tp, s) in enumerate(zip(testpoints, rj["score"]))
    ]
    return Result(
        test_case_id=f"unigenbench_{lang}_{row['index']}",
        passed=all(gr.passed for gr in grader_results),
        grader_results=grader_results,
        actual_output=row["img_path"],
        metadata={"image_idx": image_idx},
    )

Happy to open a PR adding a small, purely additive --export_openeval path off calculate_score.py (or a standalone to_openeval.py next to eval_common.py) that writes a ResultSet JSON alongside your existing CSV/JSON outputs — wouldn't touch the current pipeline or output format at all. Or if you'd rather it live outside this repo, I can build it as a community adapter in EvalPort's adapters/ directory that consumes your result CSVs directly. Either way, just wanted to check whether this is useful before putting time into it.

— Sahi, independent contributor (not affiliated with this project)