Skip to content

feat: modernize to uv + pyproject.toml + semantic-release - #461

Merged
feanil merged 10 commits into
openedx:masterfrom
irfanuddinahmad:irfanuddinahmad/modernize-python-tooling
Sep 30, 2026
Merged

feanil merged 10 commits into
openedx:masterfrom
irfanuddinahmad:irfanuddinahmad/modernize-python-tooling

Conversation

@irfanuddinahmad

@irfanuddinahmad irfanuddinahmad commented Jul 27, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Modernize opaque-keys to uv + pyproject.toml (PEP 621/735) + python-semantic-release, and adopt src/ layout.

Part of openedx/public-engineering#506 (tracked in openedx/public-engineering#516).

  • Move opaque_keys/ to src/opaque_keys/ (see "src/ layout" below).
  • Replace setup.py with pyproject.toml (PEP 621 static metadata); remove the
    hardcoded __version__ in favor of importlib.metadata + setuptools-scm.
  • Switch from pip-compile to uv with PEP 735 dependency groups (test-base,
    test, django42, quality, doc, ci, dev); commit uv.lock.
  • Update tox.ini to use tox-uv's uv-venv-lock-runner.
  • Update CI to install uv (pinned to a commit SHA) and run uv sync / uv run tox.
  • Add python-semantic-release + release.yml, publishing to PyPI via OIDC.
    commitlint.yml already existed and needed no changes.

src/ layout

Per the 2026-07-15 decision on openedx/public-engineering#506, motivated by
flat-layout editable installs not being reliably resolvable by mypy. mypy.ini
switched from files = opaque_keys (a literal path) to packages = opaque_keys
(import-based resolution via the editable install), which is the actual fix. Verified
locally: editable install resolves correctly, mypy/pylint/pycodestyle all pass,
full test suite passes, and a built wheel installs opaque_keys at the top level.

One side effect: the top-level settings test-settings package (previously bundled
into the wheel only as an artifact of setup.py's bare find_packages()) is no
longer picked up, since it lives outside src/ — its correct scope.

Removed

Deleted files: setup.py, requirements/*.in, requirements/*.txt,
.github/workflows/pypi-publish.yml (superseded by release.yml's
publish_to_pypi job — the old token-based workflow triggered on
release: published and would otherwise race the new OIDC-based publish once
semantic-release starts creating releases/tags).

Not included

  • Ruff: per the 2026-07-15 discussion on Modernize Python repos: pyproject.toml + uv + semantic-release public-engineering#506, ruff
    adoption is out of scope for this migration cycle (its own separate epic). Existing
    pylint/pycodestyle/mypy config is left untouched.
  • edx-platform-style constraint-sync script: not applicable — this repo never
    had the requirements/edx-platform-constraints.txt + check_pins.py mechanism
    that pattern replaces.

Known out-of-scope gaps (flagging, not silently working around)

  • PyPI trusted publisher (OIDC) — pre-merge blocker. release.yml's
    publish_to_pypi job publishes via OIDC (id-token: write, no stored
    credentials) instead of the old PYPI_UPLOAD_TOKEN. This requires a PyPI
    project maintainer to add a trusted publisher for edx-opaque-keys (PyPI project
    → Publishing → Add a new publisher → GitHub, repo openedx/opaque-keys,
    workflow release.yml, no environment) before the first release can actually
    publish. Not a blocker for reviewing/merging this PR, but publish_to_pypi will
    fail on the first push to master until it's done.
  • .github/workflows/upgrade-python-requirements.yml. This repo has one,
    calling openedx/.github's shared reusable workflow, which hardcodes
    ADD_PATHS="requirements" for its PR-creation step with no add_paths-style
    input exposed via workflow_call. Now that requirements/ is deleted, make upgrade's real output (uv.lock/pyproject.toml changes) won't match that
    hardcoded glob — the scheduled job will keep running but silently stop producing
    real dependency-upgrade PRs (empty diff, no failure, no alert). This can't be
    fixed from this repo; it needs a fix in openedx/.github itself (parameterize
    add_paths, or auto-detect uv.lock vs. requirements/).

Versioning

Dynamic, via setuptools-scm reading the latest git tag. tag_format = "v{version}"
now, so a manual v-prefixed release will bridge the existing unprefixed tag history
(4.0.0, 3.1.0, ...) before python-semantic-release starts tagging.

Testing Notes

Verified locally (macOS, Python 3.12): uv lock resolves cleanly and stays in sync
with pyproject.toml; uv run tox -e django42 (407 passed), without-django (394
passed), docs, and quality (mypy/pylint/pycodestyle) all pass; python -m build (with SETUPTOOLS_SCM_PRETEND_VERSION set) produces correctly-versioned
wheels/sdists.

CI-only issues found and fixed after opening this PR: fully moving coverage config
off a stale .coveragerc that was still shadowing pyproject.toml;
.readthedocs.yaml (still pointed at the deleted requirements/doc.txt, switched to
method: uv/groups: [doc]); __init__.py's importlib.metadata fallback
try/except (not present pre-migration) simplified to a direct call, no pragma
needed.

codecov/project shows a one-time ~1.6 point drop (94.03%→92.46%) once test files
were correctly excluded from coverage measurement — not a real regression, patch
coverage is 100%. No codecov.yml threshold added to absorb it; expected to show
red on this PR.

⚠️ Regression found and fixed after approval

A commit landed after this PR's initial approval that was meant to be a trivial
"remove the unnecessary __version__ try/except" cleanup, but instead deleted 385
lines from src/opaque_keys/__init__.py — the entire OpaqueKey/
OpaqueKeyMetaclass/InvalidKeyError class hierarchy, everything after the imports.
This broke every consumer that imports from the top-level opaque_keys module
(e.g. opaque_keys.edx.django.models does from opaque_keys import OpaqueKey),
which is why CI's django42 job started failing with an ImportError. Restored the
full class hierarchy from the last good commit, keeping the intended __version__
simplification. All tox envs (django42, without-django, quality) re-verified
passing after the restore. Flagging explicitly here since the approval predates this
regression — worth a fresh look rather than relying on it.

Code reviewer notes

  • django42 is the only Django version group for now (this repo's tox.ini
    envlist has only ever tested one Django version at a time); adding a
    django52-style group later just needs [tool.uv].conflicts if two Django
    pins need to coexist in one lockfile.

This PR was created with Claude Code.

@openedx-webhooks openedx-webhooks added open-source-contribution PR author is not from Axim or 2U core contributor PR author is a Core Contributor (who may or may not have write access to this repo). labels Jul 27, 2026
@openedx-webhooks

Copy link
Copy Markdown

Thanks for the pull request, @irfanuddinahmad!

This repository is currently maintained by @openedx/axim-engineering.

Once you've gone through the following steps feel free to tag them in a comment and let them know that your changes are ready for engineering review.

🔘 Get product approval

If you haven't already, check this list to see if your contribution needs to go through the product review process.

  • If it does, you'll need to submit a product proposal for your contribution, and have it reviewed by the Product Working Group.
    • This process (including the steps you'll need to take) is documented here.
  • If it doesn't, simply proceed with the next step.
🔘 Provide context

To help your reviewers and other members of the community understand the purpose and larger context of your changes, feel free to add as much of the following information to the PR description as you can:

  • Dependencies

    This PR must be merged before / after / at the same time as ...

  • Blockers

    This PR is waiting for OEP-1234 to be accepted.

  • Timeline information

    This PR must be merged by XX date because ...

  • Partner information

    This is for a course on edx.org.

  • Supporting documentation
  • Relevant Open edX discussion forum threads
🔘 Get a green build

If one or more checks are failing, continue working on your changes until this is no longer the case and your build turns green.

Details
Where can I find more information?

If you'd like to get more details on all aspects of the review process for open source pull requests (OSPRs), check out the following resources:

When can I expect my changes to be merged?

Our goal is to get community contributions seen and reviewed as efficiently as possible.

However, the amount of time that it takes to review and merge a PR can vary significantly based on factors such as:

  • The size and impact of the changes that it introduces
  • The need for product review
  • Maintenance status of the parent repository

💡 As a result it may take up to several weeks or months to complete a review and merge your PR.

@codecov

codecov Bot commented Jul 27, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 92.46%. Comparing base (49ab347) to head (04d7fd1).
⚠️ Report is 2 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master     #461      +/-   ##
==========================================
- Coverage   94.19%   92.46%   -1.74%     
==========================================
  Files          31       11      -20     
  Lines        3068     1327    -1741     
  Branches      191      145      -46     
==========================================
- Hits         2890     1227    -1663     
+ Misses        154       78      -76     
+ Partials       24       22       -2     
Flag Coverage Δ
unittests 92.46% <100.00%> (-1.74%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@mphilbrick211 mphilbrick211 moved this from Needs Triage to Ready for Review in Contributions Jul 27, 2026
@mphilbrick211 mphilbrick211 moved this from Ready for Review to Waiting on Author in Contributions Jul 27, 2026
@mphilbrick211

Copy link
Copy Markdown

Hi @irfanuddinahmad - flagging that some checks and branch conflicts have popped up.

@irfanuddinahmad

Copy link
Copy Markdown
Contributor Author

Rebased on master — conflicts resolved, CI green.

Comment thread .github/workflows/ci.yml Resolved
Comment thread .github/workflows/ci.yml Resolved
Comment thread codecov.yml Resolved
Comment thread pyproject.toml Resolved
Comment thread pyproject.toml Resolved
@irfanuddinahmad
irfanuddinahmad force-pushed the irfanuddinahmad/modernize-python-tooling branch from 6458243 to 64f37bf Compare September 3, 2026 11:39
Modernize opaque-keys to uv + pyproject.toml (PEP 621/735) +
python-semantic-release, and adopt src/ layout.

Part of openedx/public-engineering#506 (tracked in
openedx/public-engineering#516).

- Move opaque_keys/ to src/opaque_keys/.
- Replace setup.py with pyproject.toml (PEP 621 static metadata);
  remove the hardcoded __version__ in favor of importlib.metadata +
  setuptools-scm.
- Switch from pip-compile to uv with PEP 735 dependency groups
  (test-base, test, django42, quality, doc, ci, dev); commit uv.lock.
- Update tox.ini to use tox-uv's uv-venv-lock-runner.
- Update CI to install uv (pinned to a commit SHA) and run uv sync /
  uv run tox.
- Add python-semantic-release + release.yml, publishing to PyPI via
  OIDC.
- Delete setup.py, requirements/*.in, requirements/*.txt,
  .github/workflows/pypi-publish.yml (superseded by release.yml).
- tag_format = "v{version}" -- a manual v-prefixed release will
  bridge the existing unprefixed tag history (4.0.0, 3.1.0, ...) so
  future releases stay consistent.

Addresses salman2013's PR review feedback:
- Remove unexplained fail-fast: false in ci.yml.
- Author metadata matches sample-plugin's convention (Open edX
  Project / oscm@openedx.org).
- Keep the push trigger on ci.yml alongside workflow_call, per code
  owners' call (double CI run on master pushes accepted as fine).
- Keep the CI job name as Tests, matching the pre-migration name
  (deferring the per-toxenv rename to avoid breaking
  required-status-check matching).
- No codecov.yml -- it didn't change any pass/fail outcome and isn't
  required by this migration itself.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@irfanuddinahmad
irfanuddinahmad force-pushed the irfanuddinahmad/modernize-python-tooling branch from 64f37bf to 154466c Compare September 3, 2026 11:42
@salman2013
salman2013 self-requested a review September 3, 2026 11:43
irfanuddinahmad and others added 6 commits September 9, 2026 14:24
…odernize-python-tooling

# Conflicts:
#	requirements/base.txt
#	requirements/ci.txt
#	requirements/dev.txt
#	requirements/django-test.txt
#	requirements/doc.txt
#	requirements/test.txt
…ecf0aa

eecf0aa ("fix: remove unnecessary try/except around __version__") was
meant to be a trivial 5-line cleanup but instead deleted everything
from the try/except through the end of the file -- 385 lines,
including the entire OpaqueKey class hierarchy. This broke every
consumer that imports from the top-level opaque_keys module (e.g.
opaque_keys.edx.django.models does `from opaque_keys import
OpaqueKey`), causing an ImportError in CI's django42 job.

Restored the full class definitions from the last commit before the
bad edit (154466c), keeping the intended __version__ simplification
(no try/except, matching the pattern used correctly elsewhere in this
migration effort).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…ernize-python-tooling

requirements/*.txt conflicts (modified on master by an automated
dependency-bump merge, deleted here in favor of uv.lock) resolved by
keeping the deletion -- superseded by uv.lock/pyproject.toml.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…ernize-python-tooling

# Conflicts:
#	requirements/ci.txt
#	requirements/dev.txt
#	requirements/django-test.txt
#	requirements/doc.txt
#	requirements/pip-tools.txt
#	requirements/test.txt
…e job

Matches openedx/sample-plugin's current release.yml; the custom
OPENEDX_SEMANTIC_RELEASE_GITHUB_TOKEN secret isn't needed.
Comment thread pyproject.toml Outdated
dynamic = ["version"]
description = "Create and introspect course and xblock identities through a clear and consistent API"
readme = "README.rst"
license = "AGPL-3.0"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AGPL-3.0 is deprecated in the SPDX list, and under [project].license this is now published as a validated expression rather than as free text. The built wheel's METADATA carries:

License-Expression: AGPL-3.0

LICENSE:637-638 grants "either version 3 of the License, or (at your option) any later version", so this should be AGPL-3.0-or-later.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correction on this one, sorry. LICENSE:637-638 is inside the "How to Apply These Terms to Your New Programs" appendix that ships with every verbatim copy of the AGPL-3.0 text, so it isn't a grant this repo makes. What we published is setup.py:137's License :: OSI Approved :: GNU Affero General Public License v3, with no "or later". Nothing in the source or docs grants "or later" either. AGPL-3.0-only is the one that matches what we published; the deprecation half of the comment still stands.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like this is the last thing to resolve and then this is good to Switch it to AGPL-3.0-only and then I think this is good to go.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done — switched to AGPL-3.0-only in 04d7fd1 (setup.py:137's classifier has no "or later").

Comment thread .github/workflows/release.yml Outdated
id: release
uses: python-semantic-release/python-semantic-release@9a026e9303981c866c3425723009becb2437c757 # v10.6.2
with:
github_token: ${{ secrets.OPENEDX_SEMANTIC_RELEASE_GITHUB_TOKEN }}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use secrets.GITHUB_TOKEN here. The org secret isn't needed going forward, and sample-plugin, DoneXBlock, xblocks-core and xblocks-extra are all on the default token already.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Already secrets.GITHUB_TOKEN here, verified — no change needed.

irfanuddinahmad and others added 2 commits September 28, 2026 15:43
Makefile now assumes an already-synced/activated local env; uv run
stays in CI workflow steps only. See openedx/ccx-keys#190 (review
comment r4097501942).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…odernize-python-tooling

# Conflicts:
#	requirements/base.txt
#	requirements/ci.txt
#	requirements/dev.txt
#	requirements/django-test.txt
#	requirements/doc.txt
#	requirements/test.txt

@feanil feanil left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pyproject.toml:10 is still AGPL-3.0. Switch it to AGPL-3.0-only, per the correction on that thread. The rest looks right.

Comment thread pyproject.toml Outdated
dynamic = ["version"]
description = "Create and introspect course and xblock identities through a clear and consistent API"
readme = "README.rst"
license = "AGPL-3.0"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like this is the last thing to resolve and then this is good to Switch it to AGPL-3.0-only and then I think this is good to go.

Pre-migration setup.py:137 classified this as "License :: OSI
Approved :: GNU Affero General Public License v3" with no "or
later"/"v3+" wording, so AGPL-3.0-only is the correct SPDX
identifier, not AGPL-3.0-or-later.

Per feanil's correction (review comment r4135968924).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@feanil
feanil merged commit 6eb7247 into openedx:master Sep 30, 2026
8 of 9 checks passed
@bradenmacdonald

Copy link
Copy Markdown
Contributor

With this PR merged, my editable install of opaque-keys is finally visible to mypy and other python tooling when I'm working on openedx-platform code. Thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core contributor PR author is a Core Contributor (who may or may not have write access to this repo). open-source-contribution PR author is not from Axim or 2U

Projects

Archived in project

Development

Successfully merging this pull request may close these issues.

6 participants