Tokenizer.add_special_case raises KeyError #656

Activity

1. soldni commented on Nov 23, 2016

   

   soldni

   on Nov 23, 2016

   Last edited by soldni

   Author

   More actions

   A bit of follow up: I was going through the definition for spacy.vocab.Vocab.make_fused_token, and it seems that code expects ORTH to be equal to 'F', POS to be equal to 'pos', and LEMMA to be equal to 'L'; however, ORTH equals 65, POS equals 74, and LEMMA equals 73.

   I am not sure if the values expected by make_fused_tokens are intentionally different from those defined in spacy.symbols.

   EDIT: Even when replacing keys for token_attrs argument as described above, I still encounter an error:

   import spacy
   
   nlp = spacy.load('en')
   
   nlp.tokenizer.add_special_case('gimme',
       [
           {
               'F': 'gim',
               'L': 'give',
               'pos': 'VERB'},
           {
               'F': 'me' }])
   
   # Traceback (most recent call last):
   #  File "test.py", line 13, in <module>
   #    'F': 'me' }])
   #  File "spacy/tokenizer.pyx", line 377, in spacy.tokenizer.Tokenizer.add_special_case (spacy/tokenizer.cpp:8460)
   #  File "spacy/vocab.pyx", line 342, in spacy.vocab.Vocab.make_fused_token (spacy/vocab.cpp:7907)
   #  File "spacy/morphology.pyx", line 39, in spacy.morphology.Morphology.assign_tag (spacy/morphology.cpp:3919)
   # KeyError: 97
2. honnibal commented on Nov 23, 2016

   

   honnibal

   on Nov 23, 2016

   Member

   More actions

   Thanks for this.

   The docs have gotten ahead of the library here — the current/old behaviour is pretty inconsistent, so I wrote up the intended usage, but haven't had time to fix it yet. Will definitely have this resolved in the next release, which should be up this week.

   🎉React with 🎉1Reacted by Luca Soldaini
3. honnibal

   added

   bugBugs and behaviour differing from documentationBugs and behaviour differing from documentation

   on Nov 23, 2016
4. soldni

   closed this as completedon Nov 23, 2016
5. soldni

   reopened this on Nov 23, 2016
6. honnibal

   closed this as completedin 1e0f566on Nov 25, 2016
7. honnibal

   added a commit that references this issue on Nov 25, 2016

   Test #656, #624: special case rules for tokenizer with attributes.

   6652f2a
8. lock commented on May 9, 2018

   

   lockbot

   on May 9, 2018

   More actions

   This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.
9. lock

   locked as resolved and limited conversation to collaborators on May 9, 2018