Skip to content

fix: retry LookupAccountName with exponential backoff to prevent ERROR_NONE_MAPPED on fresh EC2 instances - #262

Merged
leongdl merged 5 commits into
OpenJobDescription:mainlinefrom
leon-li-inspire:fix/named-pipe-system-sid-lookup
Jun 12, 2026
Merged

fix: retry LookupAccountName with exponential backoff to prevent ERROR_NONE_MAPPED on fresh EC2 instances#262
leongdl merged 5 commits into
OpenJobDescription:mainlinefrom
leon-li-inspire:fix/named-pipe-system-sid-lookup

Conversation

@leon-li-inspire

@leon-li-inspire leon-li-inspire commented Jun 9, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Summary

  • create_security_attributes() calls win32security.LookupAccountName("", username) to resolve the current user's SID for named pipe access control
  • This fails intermittently with pywintypes.error: (1332, 'LookupAccountName', 'No mapping between account names and security IDs was done.') on freshly started EC2 instances where the LSA (Local Security Authority) service has not finished initializing
  • The failure is intermittent — re-runs pass because LSA is warm by then
  • Add retry with exponential backoff (up to 3 attempts, sleeping 1s then 2s) on error 1332, then re-raise if all attempts are exhausted

Affected repos

Fixes intermittent Windows integration test failures in all DCC repos that use LOGNAME=SYSTEM in their CodeBuild Windows environment: Cinema 4D, Blender, Houdini, Maya, 3ds Max, VRED, and Unreal Engine.

Related ticket: https://t.corp.amazon.com/P447504169

Test plan

  • test_create_security_attributes_uses_lookup: verifies LookupAccountName is called with the current username
  • test_create_security_attributes_retries_on_lsa_not_ready: verifies retry succeeds after a transient 1332 error, with correct backoff sleep
  • test_create_security_attributes_raises_after_max_retries: verifies the original error is re-raised after exhausting all retries

@leon-li-inspire
leon-li-inspire requested a review from a team as a code owner June 9, 2026 05:18
@leon-li-inspire
fix: avoid LookupAccountName for SYSTEM user in named pipe security a…
1dbca63 
…ttributes

When CodeBuild Windows integration tests run with LOGNAME=SYSTEM, getpass.getuser()
returns "SYSTEM" and the subsequent LookupAccountName("", "SYSTEM") call fails with
ERROR_NONE_MAPPED (1332) on freshly started reserved-fleet EC2 instances because the
LSA service hasn't finished initializing.

SYSTEM (S-1-5-18) is a well-known SID constant that never varies across Windows
machines. Use ConvertStringSidToSid("S-1-5-18") to bypass the LSA lookup entirely,
consistent with how the network SID (S-1-5-2) is already handled in the same method.

Fixes intermittent Windows integration test failures in deadline-cloud-for-cinema-4d
and 6 other DCC repos that share the LOGNAME=SYSTEM CodeBuild workaround.

Signed-off-by: leon-li-inspire <2182521+leon-li-inspire@users.noreply.github.com>
@leon-li-inspire
leon-li-inspire force-pushed the fix/named-pipe-system-sid-lookup branch from bc7b020 to 1dbca63 Compare June 9, 2026 05:24
karthikbekalp
karthikbekalp reviewed Jun 9, 2026
View reviewed changes
Comment thread CHANGELOG.md Outdated
crowecawcaw
crowecawcaw previously approved these changes Jun 10, 2026
View reviewed changes
@crowecawcaw
crowecawcaw enabled auto-merge (squash) June 10, 2026 19:29
@crowecawcaw
crowecawcaw disabled auto-merge June 10, 2026 19:29
@crowecawcaw
crowecawcaw enabled auto-merge (squash) June 10, 2026 19:30
@leon-li-inspire
style: fix black formatting in test_named_pipe_helper.py
1735af2 
Signed-off-by: Leon Li <2182521+leon-li-inspire@users.noreply.github.com>
auto-merge was automatically disabled June 10, 2026 20:32

Head branch was pushed to by a user without write access

@leon-li-inspire

leon-li-inspire commented Jun 10, 2026

Copy link
Copy Markdown
Contributor Author

Pushed a formatting fix (commit 1735af2) — black was flagging 3 long lines in the test file. The new workflow runs need maintainer approval to execute. Could someone please approve the pending CI runs?

@leon-li-inspire
fix: use real PySID in tests to prevent TypeError on Windows
617a7c9 
LookupAccountName mock must return a real PySID object (not a string)
because downstream code passes it to dacl.AddAccessAllowedAce() which
validates the type on Windows.

Signed-off-by: Leon Li <2182521+leon-li-inspire@users.noreply.github.com>
@leon-li-inspire

leon-li-inspire commented Jun 10, 2026

Copy link
Copy Markdown
Contributor Author

Pushed another fix (commit 617a7c9) — the Windows tests were failing because the mock returned a string "fake_sid" instead of a real PySID object. The code downstream calls dacl.AddAccessAllowedAce(..., user_sid) which validates the type. Fixed by using win32security.ConvertStringSidToSid("S-1-1-0") to produce a real PySID.

The CI runs need maintainer approval again — could someone please approve the pending workflow runs?

crowecawcaw
crowecawcaw previously approved these changes Jun 10, 2026
View reviewed changes
@crowecawcaw
crowecawcaw enabled auto-merge (squash) June 10, 2026 21:17
leongdl
leongdl reviewed Jun 10, 2026
View reviewed changes
Comment thread CHANGELOG.md Outdated
@leon-li-inspire
chore: remove CHANGELOG entry, will be auto-generated at release
73f95a8 
Signed-off-by: Leon Li <2182521+leon-li-inspire@users.noreply.github.com>
auto-merge was automatically disabled June 10, 2026 21:55

Head branch was pushed to by a user without write access

@leon-li-inspire
leon-li-inspire requested a review from leongdl June 10, 2026 22:14
leongdl
leongdl reviewed Jun 12, 2026
View reviewed changes
Comment thread test/openjd/adaptor_runtime/unit/named_pipe/test_named_pipe_helper.py
):
named_pipe_helper.NamedPipeHelper.generate_pipe_name("AdaptorTest")

@patch("getpass.getuser", return_value="regularuser")

@leongdl leongdl Jun 12, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we will need to port this to openjd-rs too!

@leongdl
leongdl merged commit 0c85c36 into OpenJobDescription:mainline Jun 12, 2026
43 checks passed
This was referenced Jun 12, 2026
Merged
Merged
@client-software-ci client-software-ci mentioned this pull request Jun 18, 2026
Merged
leon-li-inspire added a commit to leon-li-inspire/openjd-rs that referenced this pull request Jun 21, 2026
@leon-li-inspire
fix: retry LookupAccountNameW with exponential backoff on ERROR_NONE_…
ee2061a 
…MAPPED

LookupAccountNameW fails intermittently with ERROR_NONE_MAPPED (1332) on
freshly started EC2 instances where the LSA service hasn't finished
initializing. Add retry with exponential backoff (up to 3 attempts,
sleeping 1s then 2s) to the lookup_sid() function, then re-raise if all
attempts are exhausted.

This is the Rust equivalent of the Python fix in
OpenJobDescription/openjd-adaptor-runtime-for-python#262 which addresses
the same intermittent failure in Windows integration tests across all DCC
repos that use LOGNAME=SYSTEM in their CodeBuild Windows environment.

Signed-off-by: Leon Li <2182521+leon-li-inspire@users.noreply.github.com>
mwiebe pushed a commit to leon-li-inspire/openjd-rs that referenced this pull request Jun 30, 2026
@leon-li-inspire @mwiebe
fix: retry LookupAccountNameW with exponential backoff on ERROR_NONE_…
374d89d 
…MAPPED

LookupAccountNameW fails intermittently with ERROR_NONE_MAPPED (1332) on
freshly started EC2 instances where the LSA service hasn't finished
initializing. Add retry with exponential backoff (up to 3 attempts,
sleeping 1s then 2s) to the lookup_sid() function, then re-raise if all
attempts are exhausted.

This is the Rust equivalent of the Python fix in
OpenJobDescription/openjd-adaptor-runtime-for-python#262 which addresses
the same intermittent failure in Windows integration tests across all DCC
repos that use LOGNAME=SYSTEM in their CodeBuild Windows environment.

Signed-off-by: Leon Li <2182521+leon-li-inspire@users.noreply.github.com>
mwiebe pushed a commit to leon-li-inspire/openjd-rs that referenced this pull request Jul 2, 2026
@leon-li-inspire @mwiebe
fix: retry LookupAccountNameW with exponential backoff on ERROR_NONE_…
837fa69 
…MAPPED

LookupAccountNameW fails intermittently with ERROR_NONE_MAPPED (1332) on
freshly started EC2 instances where the LSA service hasn't finished
initializing. Add retry with exponential backoff (up to 3 attempts,
sleeping 1s then 2s) to the lookup_sid() function, then re-raise if all
attempts are exhausted.

This is the Rust equivalent of the Python fix in
OpenJobDescription/openjd-adaptor-runtime-for-python#262 which addresses
the same intermittent failure in Windows integration tests across all DCC
repos that use LOGNAME=SYSTEM in their CodeBuild Windows environment.

Signed-off-by: Leon Li <2182521+leon-li-inspire@users.noreply.github.com>
jericht pushed a commit to leon-li-inspire/openjd-rs that referenced this pull request Jul 2, 2026
@leon-li-inspire @jericht
fix: retry LookupAccountNameW with exponential backoff on ERROR_NONE_…
5377982 
…MAPPED

LookupAccountNameW fails intermittently with ERROR_NONE_MAPPED (1332) on
freshly started EC2 instances where the LSA service hasn't finished
initializing. Add retry with exponential backoff (up to 3 attempts,
sleeping 1s then 2s) to the lookup_sid() function, then re-raise if all
attempts are exhausted.

This is the Rust equivalent of the Python fix in
OpenJobDescription/openjd-adaptor-runtime-for-python#262 which addresses
the same intermittent failure in Windows integration tests across all DCC
repos that use LOGNAME=SYSTEM in their CodeBuild Windows environment.

Signed-off-by: Leon Li <2182521+leon-li-inspire@users.noreply.github.com>
jericht pushed a commit to OpenJobDescription/openjd-rs that referenced this pull request Jul 2, 2026
@leon-li-inspire
fix: retry LookupAccountNameW with exponential backoff on ERROR_NONE_…
0da9968 
…MAPPED (#221)

* fix: retry LookupAccountNameW with exponential backoff on ERROR_NONE_MAPPED

LookupAccountNameW fails intermittently with ERROR_NONE_MAPPED (1332) on
freshly started EC2 instances where the LSA service hasn't finished
initializing. Add retry with exponential backoff (up to 3 attempts,
sleeping 1s then 2s) to the lookup_sid() function, then re-raise if all
attempts are exhausted.

This is the Rust equivalent of the Python fix in
OpenJobDescription/openjd-adaptor-runtime-for-python#262 which addresses
the same intermittent failure in Windows integration tests across all DCC
repos that use LOGNAME=SYSTEM in their CodeBuild Windows environment.

Signed-off-by: Leon Li <2182521+leon-li-inspire@users.noreply.github.com>

* fix: gate LookupAccountNameW retry on LSA-not-ready to avoid penalizing unmapped names

ERROR_NONE_MAPPED (1332) is overloaded: Windows returns it both when the LSA
service is still initializing (transient, only at instance startup) and when a
name genuinely does not map to an account (permanent). The previous retry fired
on every 1332, so steady-state lookups of a non-existent principal — notably
the WindowsSessionUser::is_process_user fallback path — blocked for ~3s before
returning, a regression for a path that previously returned immediately.

Gate the backoff on a process-global LSA-availability check: once any lookup
succeeds (or a probe of the current process user resolves), the LSA is known to
be up and a subsequent 1332 must mean the name is unmapped, so it fails fast.
The LSA never restarts within a process lifetime, so the result is cached.

Update test_lookup_sid_fails_for_nonexistent_user accordingly: it now asserts
the unmappable-name lookup fails fast (<1s) once LSA is confirmed up, instead of
asserting the old >=3s backoff.

Signed-off-by: Leon Li <2182521+leon-li-inspire@users.noreply.github.com>

---------

Signed-off-by: Leon Li <2182521+leon-li-inspire@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@leongdl leongdl leongdl approved these changes
@crowecawcaw crowecawcaw crowecawcaw approved these changes
@epmog epmog epmog approved these changes
+1 more reviewer
@karthikbekalp karthikbekalp karthikbekalp approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@leon-li-inspire @crowecawcaw @epmog @leongdl @karthikbekalp