Skip to content

BUG: token.ent_iob_ is str not unicode #672

Closed

Description

@ELind77

Spacy version: 1.3.0
System: Ubuntu 14.04

Issue:
The value returned on token.ent_iob_ is a string, not unicode.

Code:
The above issue is reproducible with the following:

import spacy
nlp = spacy.load('en')

txt = u'''Lorem Ipsum is simply dummy text of the printing and typesetting industry.'''

doc = nlp(txt)
for tok in doc[:5]:
    print type(tok.ent_iob_)

Results in:

<type 'str'>
<type 'str'>
<type 'str'>
<type 'str'>
<type 'str'>

Comments:
Pretty sure this is caused by this line in token.pyx.
Possible solutions are to change that line or import unicode_literals in that file. I'm not sure how the project handles strings internally but having all modules use unicode_literals might not be a terrible idea.

Just fixing the single line would be easy though. If I want to submit a PR as small as this do I need to run a bunch of tests or can I just put u in front of each of those letters? That said, adding some kind of automated test builder to ensure that all properties and return values respect the contracts in the documentation might not be a bad idea. Alternatively, from what little I know about cython, maybe the properties could get type declarations that would be enforced by the compiler?

Followup question, is there a page with instructions for contributing?

-- Eric

Activity

  1. added
    bugBugs and behaviour differing from documentation
    on Dec 10, 2016
  2. honnibal commented on Dec 10, 2016

    @honnibal
    Member

    Thanks for the report!

    All modules should definitely have unicode_literals. Good suggestions re the testing, which currently needs to be refactored and improved. I don't know how to add a type declaration to a property in Cython, though. You can only specify return types for cdef and cpdef functions, I believe.

    You can find the contribution guidelines here. Thanks again!

  3. added a commit that references this issue on Dec 18, 2016
  4. honnibal commented on Dec 18, 2016

    @honnibal
    Member

    Should be fixed now.

  5. lock commented on May 9, 2018

    @lock

    This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

  6. locked as resolved and limited conversation to collaborators on May 9, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugBugs and behaviour differing from documentation

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions