Repository navigation
Add ViDoRe v3 HR retrieval evaluation notebook - #2763
Conversation
Signed-off-by: Chris Jarrett <cjarrett@nvidia.com>
Signed-off-by: Chris Jarrett <cjarrett@nvidia.com>
This reverts commit ed72381. Signed-off-by: Chris Jarrett <cjarrett@nvidia.com>
Signed-off-by: Chris Jarrett <cjarrett@nvidia.com>
| "import sys\n", | ||
| "\n", | ||
| "LOCAL_MODELS = True\n", | ||
| "INSTALL_DEPENDENCIES = True\n", |
There was a problem hiding this comment.
The notebook tells users to choose Run All Cells, and this flag defaults to True, but this cell replaces Torch/Torchvision and then execution immediately continues into imports and GPU work in the same kernel. On every later Run All it also repeats the uninstall/reinstall. That makes the documented happy path unreliable and unnecessarily expensive. Please default installation to False and make the one-time setup an explicitly separate step, or stop execution after installation with a clear restart instruction.
| "else:\n", | ||
| " print(\"Reusing the existing index.\")\n", | ||
| "\n", | ||
| "assert index.open_table(TABLE_NAME).count_rows() > 0, \"Empty index: set OVERWRITE_INDEX=True.\"\n", |
There was a problem hiding this comment.
This only verifies that some rows exist. If extraction skips pages, a prior run wrote only part of the corpus, or the cached table belongs to an older corpus snapshot, the notebook will silently compute plausible-looking metrics against an incomplete index. Because this pipeline promises exactly one vector per exported page, please validate count_rows() == len(corpus) before reuse; otherwise fail with the rebuild instruction.
Signed-off-by: Chris Jarrett <cjarrett@nvidia.com>
Description
Add
examples/vidore_v3_hr.ipynb, a compact example for ingesting the ViDoRe v3 HR corpus and computing page-level Recall and nDCG at 1, 3, 5, and 10.Checklist