Skip to content

Python 3.9/3.11 support, dropping Python 3.6 support, and moving to GitHub Actions - #559

Merged
sheriferson merged 14 commits into
AllenInstitute:masterfrom
sheriferson:feature/github-actions
Dec 13, 2023
Merged

sheriferson merged 14 commits into
AllenInstitute:masterfrom
sheriferson:feature/github-actions

Conversation

@sheriferson

@sheriferson sheriferson commented Nov 17, 2023 •

Copy link
Copy Markdown
Contributor

Overview:

This PR does the following:

  • Drops support for Python 3.6
  • Updates dependencies and code to run on Python 3.9/3.11
  • Moves IPFX tests to GitHub Actions (and runs onprem tests using self-hosted runner)

Type of Fix:

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing
    functionality to not work as expected)
  • Documentation Change

Unit Tests:

See here for the light and onprem test results from my fork:
https://github.com/sheriferson/ipfx/actions/runs/6898536678

Checklist

  • My code follows
    Allen Institute Contribution Guidelines
  • My code is unit tested and does not decrease test coverage
  • I have performed a self review of my own code
  • My code is well-documented, and the docstrings conform to
    Numpy Standards
  • I have updated the documentation of the repository where
    appropriate
  • The header on my commit includes the issue number
  • My code passes all tests
  • I have updated the CHANGELOG.md with the description of changes understandable by end users

Notes

I believe this addresses #558, #557, and #550. With the move away from Python 3.6 and the unpinning of most dependencies, pip install -e . succeeds on my M* Mac.

Action items

  • Deprecate CircleCI as well (?)
  • Nightly workflow not detected by GitHub Actions yet. Maybe after merging into the main branch.
  • Disable/deprecate outgoing internal CI/CD

@sheriferson
sheriferson marked this pull request as ready for review November 17, 2023 22:28
Comment thread README.md Outdated
Comment thread docker/README.md
Comment thread docker/py39/Dockerfile Outdated
Comment thread docker/py39/Dockerfile Outdated
self._reload_nwbfile()


def set_container_sources(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm going to invoke Chesterton's Fence here and ask for an explanation of what this existing code does? Why is it safe to remove this now but not previously?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with the invocation.
But I've been looking through #361 and the commit making the changes (1aeb641) and I am yet to find a good source of documentation for why the method existed.
There are a couple of internal confluence pages that I'm trying to get access to in case they are enlightening.
If you have suggestions for where to look or who to check with, please let me know.

Comment thread requirements.txt
Comment thread tests/test_nwb_utils.py Outdated
TEST_NWB_FILE_PATH = 'tests/data/nwb/Ctgf-T2A-dgCre;Ai14-495723.05.02.01.nwb'

def test_is_file_mies():
assert is_file_mies(TEST_NWB_FILE_PATH) == True No newline at end of file

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add a test to test for the other type? I think it was HBG?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is no specific test/check for HBG files. The usage is checking if the file is MIES, and if not it assumes it's HBG.
There is one nwb file in the repo that is not MIES, I could create a test that shows that the NWB data loads successfully when HBGNWBData class is used?

Comment thread requirements.txt

@njmei njmei Nov 20, 2023 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should have this requirements.txt be as flexible as possible in terms of dependency pinning. But we should have have our tests use a pinned-requirements.txt so that future maintainers don't get constantly hit with dependency update related test failures.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't you want to know if the actual requirements.txt file customers are using will cause test failures either on install or on run? (likely because a dependency update caused an issue) I'm not following why it's a good idea to use different dependencies for tests vs. when pip installing.

@sheriferson sheriferson mentioned this pull request Dec 8, 2023
@njmei
njmei self-requested a review December 12, 2023 17:07
After updating PyNWB, the pre-existing serialize method in the Nwb2Sink
class was throwing errors along the form of:

ValueError: Cannot change container_source once set: 'root' pynwb.file.NWBFile

This commit resolves this error by removing the use of set_container_sources()

After removal of the set_container_sources() method, tests pass and
testing done by scientists also did not reveal any issues, but because
the origin/reason for this code is not clear, we're not sure that
removing it doesn't break something else with NWB files, for example
with publishing to DANDI.

Should you run into issues with containers or other metadata in NWB
files, the work of the removed code might be the explanation. However
it's not clear how to modify that code to get around the original error
mentioned above.
Testing: re-created the images on the host for the self-hosted runner
and re-ran the onprem tests.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants