Skip to content

Ad-hoc builds sometimes have the completely wrong code #81466

Closed

Description

@chuckdries

testBuild.yml seems to rely on Rock's fingerprint to uniquely identify the native build artifacts for ad-hoc builds, which causes a cache collision across different PRs that happen to have the same native code. This leads to ad-hoc builds on one pr actually containing code from a different PR.

For example, #81437 (comment) and #79473 (comment) are ad-hoc builds on completely different PRs, but link to the exact same S3 URLs

Issue OwnerCurrent Issue Owner: @LukasMod

Activity

  1. AndrewGable commented on Feb 5, 2026

    @AndrewGable
    Contributor

    For additional context: The web build worked correctly in the examples above, iOS shows identical S3 URLs after code has changed.

  2. LukasMod commented on Feb 5, 2026

    @LukasMod
    Contributor

    Hi! I suspect the issue happened because we started building ad-hoc builds more frequently (on push to main), and I accidentally allowed them to run in parallel, which doesn’t work well with Rock internals. The gh action checks whether a specific build already exists, and when we re-sign (which we do in these workflows), we add the commit hash to the build name.

    So instead of
    https://ad-hoc-expensify-cash.s3.us-east-1.amazonaws.com/rock-artifacts/ad-hoc/rock-android-Adhoc-FINGERPRINT/index.html

    it ends up being more like
    https://ad-hoc-expensify-cash.s3.us-east-1.amazonaws.com/rock-artifacts/ad-hoc/rock-android-Adhoc-SHA-FINGERPRINT/index.html

    In your case, I assume the build was replaced by another parallel build with the same native changes. This shouldn’t happen again, as the fix was merged a few hours ago.

    One thing I’m not sure about right now is why the subsequent builds didn’t work for you, since they’re already using the SHA in the name. I’ll try to figure this one #79473 (comment) out

  3. LukasMod commented on Feb 5, 2026

    @LukasMod
    Contributor

    I prepared another fix to put both adhoc workflows (manual + push) into the same concurrency group. Previous fix was focused on new workflow instead of both #81520

  4. LukasMod commented on Feb 5, 2026

    @LukasMod
    Contributor

    I may found potential bug with manual triggered workflows.
    It looks like in https://github.com/Expensify/App/actions/runs/21695712840 workflow was triggered for PR 79473, which set REF and properly checkout to create build, but after all it is saved based on GITHUB_SHA pointing to commit from main which triggered workflow:

    • Main was at commit 4d4f242, therefore GITHUB_SHA = 4d4f242 for the ENTIRE workflow run

    • Checkout action checks out ee673a9 (from needs.prep.outputs.APP_REF based on PR number)

    • The working directory HEAD is now ee673a9, building

    • GITHUB_SHA context variable still equals 4d4f242 so https://github.com/callstackincubator/ios/blob/8dcef6cc275e0cf3299f5a97cde5ebd635c887d7/action.yml#L416-L429
      sets ARTIFACT_NAME to rock-ios-device-AdHoc-4d4f242-54e89147d28f8e8638773d8a678888d5d25e206b

    • there is a risk that another workflow may be also triggered for the same GITHUB_SHA and same fingerprint

  5. LukasMod commented on Feb 5, 2026

    @LukasMod
    Contributor

    But even if that’s true, the build was triggered again, so it should have been overwritten or created from scratch after the manual deletion. I’m a bit worried there might be something else going on too.

    There was similar build triggered 15min later from #81465,
    which produced ios artifact link: https://ad-hoc-expensify-cash.s3.us-east-1.amazonaws.com/rock-artifacts/ad-hoc/rock-ios-device-AdHoc-4d4f242-54e89147d28f8e8638773d8a678888d5d25e206b/index.html - same as build from this comment #79473 (comment). But based on timings in responses it should not be issue, it was already marked as wrong one (10min before second build finished) 🤔

  6. self-assigned this
    on Feb 5, 2026
  7. rafecolton commented on Feb 5, 2026

    @rafecolton
    Member

    Temporary workaround here: #81578

  8. added
    ReviewingHas a PR in review
    and removed on Feb 5, 2026
  9. mountiny commented on Feb 5, 2026

    @mountiny
    Contributor

    Nice digging Lukasz! I commented on this PR #81520, I am not 100% sure we want to do that, but curious for your take

  10. LukasMod commented on Feb 6, 2026

    @LukasMod
    Contributor

    I'm glad that #81584 helps!
    I shared some details here #81520 (comment) and asked for rock team opinion to double check if my conclusions are correct. I believe it would make process safer and results more readable

  11. LukasMod commented on Feb 11, 2026

    @LukasMod
    Contributor

    I prepared PR for gh action and waiting for review there. It should resolve edge case with new fingerprint builds and keep consistent paths/names for adhocs callstackincubator/ios#25. Also it should save some space on remote -> 1 native build (zip) upload per 1 fingerprint.

    Update: Waiting for review on this one. I will share the results or prepare action bump if it will be accepted 👍

  12. LukasMod commented on Feb 13, 2026

    @LukasMod
    Contributor

    iOS action already merged. Testing how it works with #82391

    For fresh build it worked as planned ✅
    zip artifact (reused, no identifier in name): https://ad-hoc-expensify-cash.s3.us-east-1.amazonaws.com/rock-artifacts/rock-ios-device-AdHoc-fbaf59143f35e8cf884efc354d9e31d29d32fc5d.zip
    adhoc build (per commit, identifier in name): https://ad-hoc-expensify-cash.s3.us-east-1.amazonaws.com/rock-artifacts/ad-hoc/rock-ios-device-AdHoc-56f7872bc7ca30ea7127fbb454f2a77d67275c11-fbaf59143f35e8cf884efc354d9e31d29d32fc5d/index.html

    Waiting for next test build to verify also how it behaves for scenario with cached build.
    Meanwhile similar PR for android gh actions callstackincubator/android#22

  13. LukasMod commented on Feb 13, 2026

    @LukasMod
    Contributor

    Both scenarios tested on iOS ✅

    • zip uploaded with only {fingerprint} name
    • adhoc uploaded with {identifier}-{fingerprint} name
    • for cached build scenario it uploaded only fresh adhoc files without another zip

    Android action PR was already merged.
    PR with actions bump ready for review: #82391

  14. LukasMod commented on Feb 16, 2026

    @LukasMod
    Contributor

    Updated branch to include both App and Mobile-Expensify short sha in identifier section. Already merged, waiting for some runs to double check how its working.

  15. mountiny commented on Feb 16, 2026

    @mountiny
    Contributor

    Perfect, thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

Type

No type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions