Skip to content

Fix xCoRe always returning empty coreference clusters - #1

Merged
mjspeck merged 3 commits into
masterfrom
steel-guan
Mar 25, 2026
Merged

mjspeck merged 3 commits into
masterfrom
steel-guan

Conversation

@mjspeck

@mjspeck mjspeck commented Mar 24, 2026 •

Copy link
Copy Markdown
Owner

Summary

  • Bug 1 (model_cross.py): Three independent if blocks in the cluster-representation dispatch meant the final else: preds["full_coreferences"] = [] always ran for soft-dot and soft-mc models, silently overwriting the correctly computed result. Fixed by converting the second and third if to elif.

  • Bug 2 (xcore_model.py): transformers 5.x changed the DeBERTa-v3 tokenizer so add_special_tokens=True no longer inserts [CLS]/[SEP] tokens. Without them the encoder produces nearly identical representations for every token and the start_token_classifier outputs ~0 for all positions, yielding zero detected mentions and empty clusters. Fixed by detecting the missing tokens after tokenization and manually injecting them, patching word_ids() and word_to_tokens() to account for the +1 offset.

  • Bug 3 (xcore_model.py): PyTorch ≥ 2.6 changed torch.load to use weights_only=True by default, blocking PL checkpoints that contain non-tensor globals. Fixed by temporarily wrapping torch.load to force weights_only=False only during checkpoint load.

  • Bug 4 (setup.py): Pin transformers to >=4.34,<5 to prevent accidental upgrade to 5.x until full compatibility work is complete (see issue Upgrade to transformers >= 5.x and retrain model checkpoints #2). Closes issue Pin transformers to 4.x to restore full model accuracy #3.

Verification

from xcore import xCoRe

model = xCoRe(device="cpu")
model.model = model.model.float()
text = "Barack Obama is traveling to Rome. The city is sunny and the president plans to visit its most important attractions"
pred = model.predict(text)
# Before: {'clusters_token_offsets': [], 'clusters_token_text': []}
# After:  {'clusters_token_text': [['Rome', 'The city', 'its']]}
assert any("Rome" in cluster for cluster in pred["clusters_token_text"])

Note: the ['Barack Obama', 'the president'] cluster is not returned — this is a pre-existing model quality issue unrelated to these fixes (the model scores 'Barack' at 0.472, just below the 0.5 threshold, in both transformers 4.x and 5.x). Tracked in issue #2.

Test plan

  • Run the verification snippet above and confirm a non-empty cluster is returned
  • Test with other coreference-heavy sentences (pronoun chains, named entity references)
  • Verify cross and long modes still work

🤖 Generated with Claude Code

Three bugs caused predict() to always return empty clusters:

1. model_cross.py: independent if-blocks overwrote soft-dot/soft-mc results
   The three if-blocks at lines 1732-1807 were independent, so the final
   else branch (preds["full_coreferences"] = []) always ran for soft-dot
   and soft-mc models because coref_hs is always empty for those
   representations. Fixed by converting the second and third if to elif.

2. xcore_model.py: CLS/SEP tokens missing with transformers >= 5
   transformers 5.x changed DeBERTa-v3 tokenizer behaviour so
   add_special_tokens=True no longer inserts [CLS]/[SEP]. The model was
   trained with these boundary tokens, so without them the encoder
   produces nearly identical representations for all tokens and the
   start_token_classifier outputs ~0 probability for every token (logit
   ~-14), yielding zero detected mentions and empty clusters.
   Fixed by detecting the missing tokens and manually injecting them
   while patching word_ids() and word_to_tokens() for the +1 offset.

3. xcore_model.py: checkpoint loading fails with PyTorch >= 2.6
   PyTorch 2.6 changed torch.load to use weights_only=True by default,
   which blocks PL checkpoints containing non-tensor globals (AttributeDict,
   DictConfig, etc.). Since PyTorch Lightning passes weights_only
   explicitly downstream, functools.partial cannot override it.
   Fixed by temporarily replacing torch.load with a wrapper that forces
   weights_only=False only for the duration of the checkpoint load.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
mjspeck and others added 2 commits March 24, 2026 12:22
transformers 5.x changed DeBERTa-v3 tokenizer behaviour in ways that
require additional compatibility work (see issue #2). Pin to 4.x as a
short-term safeguard until that work is complete.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Fix broken imports in evaluate.py (missing xcore. prefix)
- Add torch.load weights_only=False patch to evaluate.py for PyTorch >=2.6
- Fix all Hydra _target_ paths across conf/data/ and conf/model/ to use
  fully-qualified xcore.* module paths (previously used bare module names
  that only worked when running from inside the xcore/ subdirectory)
- Fix xdeberta-large_80000steps.yaml: set num_training_steps=80000 and
  num_warmup_steps=8000 (was incorrectly identical to the 10k-step config)
- Fix conf/evaluation/default_evaluation.yaml: replace hardcoded absolute
  checkpoint path with Hydra ??? (required override) placeholder
- Add conf/data/xanimalfarm.yaml for zero-shot Animal Farm evaluation
- Add REPLICATION.md with step-by-step instructions for all paper
  experiments (Tables 3 and 4), expected results, and hyperparameter table

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@mjspeck mjspeck self-assigned this Mar 25, 2026
@mjspeck
mjspeck merged commit a61ee2e into master Mar 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant