Skip to content

Flash buffer file consistency check and force creation - #239

Merged
zain-sohail merged 26 commits into
mainfrom
buffer-metadata-211
Nov 18, 2023
Merged

Flash buffer file consistency check and force creation#239
zain-sohail merged 26 commits into
mainfrom
buffer-metadata-211

Conversation

@zain-sohail

@zain-sohail zain-sohail commented Nov 4, 2023

Copy link
Copy Markdown
Member

With reference to #211

@coveralls

coveralls commented Nov 4, 2023 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Pull Request Test Coverage Report for Build 6854608712

  • 127 of 127 (100.0%) changed or added relevant lines in 2 files are covered.
  • No unchanged relevant lines lost coverage.
  • Overall coverage increased (+0.7%) to 91.122%
Totals Coverage Status
Change from base Build 6812060575: 0.7%
Covered Lines: 5060
Relevant Lines: 5553
💛 - Coveralls

zain-sohail
zain-sohail commented Nov 5, 2023
View reviewed changes
Comment thread sed/loader/flash/loader.py
steinnymir
steinnymir approved these changes Nov 7, 2023
View reviewed changes

@steinnymir steinnymir left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems to work now, after fixing the schema.
reloading works which is the most important feature, so LGTM!

Comment thread sed/loader/flash/loader.py
@steinnymir steinnymir linked an issue Nov 7, 2023 that may be closed by this pull request
Flash loader parquet file consistency check #211
Closed
@steinnymir

steinnymir commented Nov 7, 2023

Copy link
Copy Markdown
Member

With reference to #211

if you write: "closes #211" it will automatically close the issue on merge! ;)

rettigl
rettigl requested changes Nov 7, 2023
View reviewed changes

@rettigl rettigl left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generally looks good, but there are a few small things worth considering. Also, I would suggest to write a test for the new part.

Comment thread sed/loader/flash/loader.py Outdated
Comment thread sed/loader/flash/loader.py Outdated
Comment thread sed/loader/flash/loader.py Outdated
Comment thread sed/loader/flash/loader.py
Comment thread sed/loader/flash/loader.py Outdated
zain-sohail
zain-sohail commented Nov 10, 2023
View reviewed changes
Comment thread sed/loader/flash/loader.py Outdated
rettigl
rettigl approved these changes Nov 15, 2023
View reviewed changes

@rettigl rettigl left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good now, I have only a few very small comments left. Please merge after addressing these.

Comment thread tests/loader/flash/test_flash_loader.py Outdated
Comment thread tests/loader/flash/test_flash_loader.py
Comment thread pyproject.toml
@rettigl

rettigl commented Nov 15, 2023

Copy link
Copy Markdown
Member

Ah, and add the __init__.py file to the flash folder.

@steinnymir steinnymir added this to the FLASH Beamtime preparation milestone Nov 16, 2023
zain-sohail and others added 19 commits November 17, 2023 17:10
@zain-sohail
add reading metadata of buffer files
484193b
@zain-sohail
add consistency check and force recreation of buffer files
5c25717
@zain-sohail
fix the lint error
00e4a4b
@zain-sohail
fix the lint error
fea81ba
@zain-sohail
load metadata and schema after creating all buffer files
8e2e720
@pullrequesttoken @zain-sohail
Update dependencies (#251)
b2cd33d 
Co-authored-by: zain-sohail <zain-sohail@users.noreply.github.com>
@steinnymir @zain-sohail
fix schema comparison
823f4f1
@steinnymir @zain-sohail
docstring for linting
a4a64b5
@zain-sohail
add the test file
daa1f9a
@zain-sohail
allow to test all formats of channels and if group_name is not in h5
ef64d0a
@zain-sohail
add tests for flashloader
c02d0af
@zain-sohail
add pyfakefs for fake paths
0b31e50
@zain-sohail
fix the lint error of runs being an int
915024f
@pullrequesttoken @zain-sohail
Update dependencies (#254)
b773e0f 
Co-authored-by: zain-sohail <zain-sohail@users.noreply.github.com>
@zain-sohail
- update get_channels_by_format to a general method get_channels
2d6e9bf 
- remove the schema_from_config property
- update tests for get_channels to include many cases
- add docstrings and comments to all tests
- add comprehensive test for bufffer_schema_mismatch
@zain-sohail
fix lint error
522c5ed
@zain-sohail
fix linting problems arising from fixture
e49f2aa
@zain-sohail
add get, add return types
713f9ad
@zain-sohail
add get
7cb1c07
@zain-sohail
zain-sohail merged commit ac160b3 into main Nov 18, 2023
@zain-sohail
zain-sohail deleted the buffer-metadata-211 branch November 18, 2023 16:36
rettigl added a commit that referenced this pull request Nov 18, 2023
@rettigl
implement changes from PR #239: buffer file consistency check and tes…
f0094ec 
…ts into SXP loader
@rettigl rettigl mentioned this pull request Nov 21, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@steinnymir steinnymir steinnymir approved these changes
@rettigl rettigl rettigl approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

prep FLASH BeamTime

Development

Successfully merging this pull request may close these issues.

Flash loader parquet file consistency check

4 participants

@zain-sohail @coveralls @steinnymir @rettigl