Repository navigation
Description
Environment
self-hosted (Docker Compose)
Version
26.9.0, upgraded from 26.8.0.
- Backend release:
26.9.0+91e940e6fe4df840c91d865b9d29df9a1b044faf - Python 3.13
sentry-sdk2.68.0taskbroker-client0.20.25
Steps to Reproduce
- Upgrade an existing self-hosted installation from 26.8.0 to 26.9.0 with the beacon enabled.
- Allow the hourly
sentry.tasks.send_beacontask to run. - In this installation it failed at both 11:00 and 12:00 UTC on 2026-09-16, and the internal
send-beaconcron monitor reported a failure.
I have not isolated which production module lookup changes the registry, so the upgrade steps alone may not reproduce this on every installation. The following deterministic reproduction exercises the installed helper in an isolated Python process. It simulates a module loading another module during version discovery, without modifying the real sys.modules or any files.
Run the following Python inside a 26.9.0 Sentry container:
from pathlib import Path
from types import SimpleNamespace
import sys
source = Path("/usr/src/sentry/src/sentry/debug/utils/packages.py").read_text()
ns = {}
exec(compile(source, "packages.py", "exec"), ns)
# Isolate the reproduction from the interpreter's real module registry.
modules = {}
class LazyModule:
def __getattr__(self, key):
if key == "get_version":
# Simulate a lazy import during version lookup.
modules["loaded_during_version_lookup"] = SimpleNamespace(__version__="2.0")
return lambda: "1.0"
raise AttributeError(key)
modules["diagnostic_lazy_module"] = LazyModule()
ns["sys"] = SimpleNamespace(modules=modules, version_info=sys.version_info)
ns["get_all_package_versions"]()Expected Result
Package-version collection should tolerate modules being imported during discovery. The beacon task should not crash while constructing its payload.
Actual Result
Production taskworker traceback:
Traceback (most recent call last):
File "/.venv/lib/python3.13/site-packages/taskbroker_client/worker/workerchild.py", line 509, in run_worker
_execute_activation(task_func, inflight.activation, app.context_hooks)
File "/.venv/lib/python3.13/site-packages/taskbroker_client/worker/workerchild.py", line 725, in _execute_activation
task_func(*args, **kwargs)
File "/.venv/lib/python3.13/site-packages/taskbroker_client/task.py", line 142, in __call__
return self._func(*args, **kwargs)
File "/usr/src/sentry/src/sentry/tasks/beacon.py", line 165, in send_beacon
"packages": get_all_package_versions(),
File "/usr/src/sentry/src/sentry/debug/utils/packages.py", line 44, in get_all_package_versions
for module_name, app in sys.modules.items():
RuntimeError: dictionary changed size during iteration
The standalone reproduction fails at the same loop with the same exception.
Analysis / Possible Fix
get_all_package_versions() iterates a live view of sys.modules and calls get_package_version() inside that loop. Version discovery includes attribute access and potentially calls a module's version function, so it can trigger imports. Imports from another thread can also invalidate the iterator.
A possible fix is to iterate a snapshot:
- for module_name, app in sys.modules.items():
+ for module_name, app in sys.modules.copy().items():I tested this change only in memory in an isolated process: the injected lazy-module reproduction fails with the installed implementation and succeeds with the snapshot, preserving the discovered version. No production patch has been applied, and an end-to-end beacon run with the change has not been tested.
Source: 26.9.0 packages.py.
Although the failure became visible after upgrading, the same unsafe loop is present in 26.8.0 and was still present on master when checked. I am not claiming that 26.9.0 introduced it.
Product Area
Crons
Metadata
Metadata
Assignees
Labels
Projects
- StatusShow more project fieldsNo status
- StatusShow more project fieldsNo status
Activity
getsantry commentedon Sep 16, 2026
Assigning to @getsentry/support for routing ⏲️
linear-code commentedon Sep 16, 2026
ENG-8633
aminvakil commentedon Sep 17, 2026
This is related to
getsentry/sentry, I opened getsentry/sentry#124713 to fix it.fix(beacon): Snapshot modules during package version discovery (#124713)