Skip to content

Support transformers 5.x and fix empty coreference clusters - #16

Open
mjspeck wants to merge 10 commits into
SapienzaNLP:masterfrom
mjspeck:pr/transformers-5-support
Open

mjspeck wants to merge 10 commits into
SapienzaNLP:masterfrom
mjspeck:pr/transformers-5-support

Conversation

@mjspeck

@mjspeck mjspeck commented Oct 14, 2026

Copy link
Copy Markdown

This PR makes xCoRe run correctly on current PyTorch and on transformers 4.56 through 5.x, using the published checkpoints as they are. It also fixes the bug where predict() always returned empty clusters, adds paper-replication configs and docs, and adds tests.

Transformers 5.x support

Two transformers 5.x changes break xCoRe:

  • Missing [CLS]/[SEP]. Transformers 5.0.0–5.3.0 stop inserting them for DeBERTa-v3 when add_special_tokens=True; 5.4.0 restores them. Without them the encoder output collapses and no mentions are detected. A new encode_words() in xcore/common/tokenization.py tokenizes without special tokens and wraps [CLS] … [SEP] itself. Inference (xcore_model.py) and training (datasets.py) both use it, so the input layout no longer depends on the installed version.
  • Float16 loading. Transformers 5.x loads deberta-v3-large in float16 by default, and checkpoint loading keeps that dtype, so inference fails with mat1 and mat2 must have the same dtype, but got Half and Float. model_cross.py now loads the encoder with dtype=torch.float32.

With both fixes, encoder hidden states are bit-identical between transformers 4.57.6 and 5.19.0. sapienzanlp/xcore-litbank predicts the same clusters on 4.56.0, 4.57.6, 5.0.0 and 5.19.0 as the original code does on 4.57.6, so the checkpoints do not need retraining.

Other changes in this group:

  • The requirement becomes transformers>=4.56,<6. Version 4.56 is the first that accepts dtype=.
  • xcoreDataset refuses a processed-data cache that was built without [CLS] and asks you to rebuild it.

Other fixes

  • Empty clusters. In model_cross.py, three independent if blocks let the final else overwrite the soft-dot and soft-mc predictions with []. They are now if/elif.
  • PyTorch ≥ 2.6. Checkpoint loading in xcore_model.py and evaluate.py forces weights_only=False for the trusted Lightning checkpoint.
  • evaluate.py had broken imports, which are fixed. Hydra _target_ paths now use full xcore.* module paths, so commands run from the repository root.
  • xdeberta-large_80000steps.yaml now really uses 80,000 steps with 8,000 warm-up steps; it was a copy of the 10k config.
  • default_evaluation.yaml no longer hardcodes an absolute checkpoint path.

Additions

  • REPLICATION.md, with commands and expected scores for Tables 3 and 4, and conf/data/xanimalfarm.yaml.
  • tests/test_tokenization.py, covering:
    • the token layout against a transformers 4.x reference;
    • training offsets;
    • the stale-cache guard;
    • a full dataset build, cache reload, and collate_fn run;
    • the float32 encoder load;
    • an opt-in checkpoint run (XCORE_SLOW_TESTS=1).

Testing

pip install -e . pytest
pytest -q tests
XCORE_SLOW_TESTS=1 pytest -q tests   # downloads sapienzanlp/xcore-litbank, runs on CPU

All 11 tests pass on transformers 4.56.0, 4.57.6 and 5.19.0. On 5.0.0, 10 pass and one comparison test skips, because that version's tokenizer omits the special tokens.

Relation to #13

This branch already merges #13, so there are no conflicts:

🤖 Generated with Claude Code

mjspeck and others added 10 commits March 24, 2026 11:02
Three bugs caused predict() to always return empty clusters:

1. model_cross.py: independent if-blocks overwrote soft-dot/soft-mc results
   The three if-blocks at lines 1732-1807 were independent, so the final
   else branch (preds["full_coreferences"] = []) always ran for soft-dot
   and soft-mc models because coref_hs is always empty for those
   representations. Fixed by converting the second and third if to elif.

2. xcore_model.py: CLS/SEP tokens missing with transformers >= 5
   transformers 5.x changed DeBERTa-v3 tokenizer behaviour so
   add_special_tokens=True no longer inserts [CLS]/[SEP]. The model was
   trained with these boundary tokens, so without them the encoder
   produces nearly identical representations for all tokens and the
   start_token_classifier outputs ~0 probability for every token (logit
   ~-14), yielding zero detected mentions and empty clusters.
   Fixed by detecting the missing tokens and manually injecting them
   while patching word_ids() and word_to_tokens() for the +1 offset.

3. xcore_model.py: checkpoint loading fails with PyTorch >= 2.6
   PyTorch 2.6 changed torch.load to use weights_only=True by default,
   which blocks PL checkpoints containing non-tensor globals (AttributeDict,
   DictConfig, etc.). Since PyTorch Lightning passes weights_only
   explicitly downstream, functools.partial cannot override it.
   Fixed by temporarily replacing torch.load with a wrapper that forces
   weights_only=False only for the duration of the checkpoint load.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
transformers 5.x changed DeBERTa-v3 tokenizer behaviour in ways that
require additional compatibility work (see issue #2). Pin to 4.x as a
short-term safeguard until that work is complete.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Fix broken imports in evaluate.py (missing xcore. prefix)
- Add torch.load weights_only=False patch to evaluate.py for PyTorch >=2.6
- Fix all Hydra _target_ paths across conf/data/ and conf/model/ to use
  fully-qualified xcore.* module paths (previously used bare module names
  that only worked when running from inside the xcore/ subdirectory)
- Fix xdeberta-large_80000steps.yaml: set num_training_steps=80000 and
  num_warmup_steps=8000 (was incorrectly identical to the 10k-step config)
- Fix conf/evaluation/default_evaluation.yaml: replace hardcoded absolute
  checkpoint path with Hydra ??? (required override) placeholder
- Add conf/data/xanimalfarm.yaml for zero-shot Animal Farm evaluation
- Add REPLICATION.md with step-by-step instructions for all paper
  experiments (Tables 3 and 4), expected results, and hyperparameter table

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This change fixes the issue with xcore returning empty clusters. But there are still performance issues.
Two transformers 5.x changes broke xCoRe:

- transformers 5.0-5.3 stop inserting [CLS]/[SEP] for DeBERTa-v3 with
  add_special_tokens=True. Add encode_words(), which tokenizes without
  special tokens and wraps [CLS] ... [SEP] itself, and use it for both
  inference (xcore_model.py) and training (datasets.py). Training had no
  fallback before this change.
- transformers 5.x loads deberta-v3-large in float16 by default, so the
  published checkpoints crashed with a Half/Float dtype mismatch. Load
  the encoder with dtype=torch.float32.

With both fixes, encoder hidden states are bit-identical between 4.57.6
and 5.19.0, and sapienzanlp/xcore-litbank predicts the same clusters on
4.56.0, 4.57.6, 5.0.0 and 5.19.0, so the checkpoints need no retraining.

Also reject processed-dataset caches built without [CLS], and replace
the transformers<5 pin with >=4.56,<6 (4.56 adds the dtype keyword).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Add Superset workspace setup and teardown scripts

Setup builds a uv-managed .venv (torch, requirements.txt, editable xcore),
symlinks data/ to the root checkout's datasets, and smoke-tests imports.
Teardown removes only the data symlink. numpy is capped below 2 so the
spaCy 3.7 / thinc 8.2 stack imports.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Link datasets per entry so processed caches stay per workspace

Address review: keep data/ a real directory with one symlink per root
dataset (skipping cache_*), repair dangling links on re-run, recreate a
broken .venv with --clear, report dataset status, and drop the now
unneeded .gitignore rule.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Move dataset linking into link-data.sh with tests

Address second review: teardown removes only links that point at the
root's data/<name> and never runs in the root checkout, the root path
falls back to git's common dir when SUPERSET_ROOT_PATH is unset, and the
missing-data hint says to download into the root checkout.
.superset/tests/test_link_data.sh covers linking, repair, migration,
and removal.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Harden link-data.sh against foreign links and path spellings

Address third review: never write through a data symlink the script did
not create, run from the git toplevel, canonicalize the root path, find
the root via git worktree list, prune only dangling links into the root's
data/, and recreate a broken .venv without uv's newer --clear flag.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Simplify link-data.sh and fix dangling-link collision

Address final review: a user's dangling data/<name> link no longer makes
ln -s abort setup, one is_root_link helper decides ownership, the
never-shipped whole-directory migration path is dropped (any data symlink
is now left alone), setup.sh runs from the git toplevel, Python is pinned
to 3.12, and constraints.txt caps transformers<5.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Install in one resolve and tighten link-data.sh edge cases

Address review: overrides.txt settles the sentencepiece conflict so torch,
requirements.txt and the package resolve once with no churn on re-run;
setup rebuilds a .venv on the wrong Python; link-data.sh rejects unknown
arguments, cleans links to a deleted root data/, and warns about dangling
links it leaves.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Keep user state safe in setup and link-data.sh

Address review: setup refuses to delete a .venv on the wrong Python,
link-data.sh keeps links while the root data/ is missing (e.g. an
unmounted drive), tolerates a regular file named data, and warns about
dangling user links in the same pass that prunes root links. Tests pin
git discovery to the temp dir. Drop the transformers cap that the single
resolve made redundant.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Read venv version once and test the git worktree root fallback

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Resolve conflicts with SapienzaNLP#13: keep transformers>=4.56,<6
instead of the ==4.34 pin, and keep the fork's equivalent torch.load
weights_only patch. Take upstream's 0.1.3 version bump and .float() call.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The .superset/ directory configures the fork maintainer's local
development workspaces and is not part of xCoRe.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant