Repository navigation
chore(ci): run AI integration tests on Xcode 27 - #16899
andrewheard wants to merge 2 commits into
Conversation
Run the standard, Live and GeminiLanguageModel integration tests exclusively on the xcode-27 runner with Xcode 27.0. Previously, the standard and Live tests ran on macos-26 with Xcode 26.2 and the GeminiLanguageModel tests were non-blocking. Rename the `build` matrix entry to `xcode-26-build` on PRs and `xcode-26-test` in nightly runs. It still uses macos-15 with Xcode 26.2, the oldest supported version, but now runs the standard integration tests nightly instead of only building for testing. Live and GeminiLanguageModel tests are intentionally omitted on Xcode 26. Also: - Make the GeminiLanguageModel job blocking. - Pass matrix values to the test step via `env` to avoid template injection. - Remove the unused `GEMINI_LANGUAGE_MODEL` environment variable.
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. |
Move the Live and GeminiLanguageModel integration test jobs into the `testapp-integration` matrix as `live` and `gemini-language-model`. Since both jobs now run on xcode-27 as blocking checks, they were identical to `testapp-integration` apart from their test filter. Each matrix entry still runs on its own runner, so Live tests stay isolated from the other suites (#16493). Also: - Remove the `GEMINI_LANGUAGE_MODEL` environment variable, which is no longer read since #16631. - Name failure log artifacts after the matrix entry and drop the `log_prefix` matrix key. - Skip test diagnostics collection in AI integration test runs, as in #16781, since `simctl diagnose` can stall for up to 10 minutes on Xcode 27 after a failed test attempt.
Summary of ChangesThis pull request optimizes the Firebase AI Logic integration test suite by migrating primary test execution to Xcode 27. By utilizing Xcode 27's enhanced test repetition logic, the CI pipeline can now target specific failing test cases rather than re-running entire test plans, significantly reducing execution time and backend resource usage. The changes also consolidate various test jobs into a unified matrix and improve security by hardening CI script execution. Highlights
New Features🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Ignored Files
Activity
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
Runs the Firebase AI Logic test app integration tests on Xcode 27 for PRs, while nightly runs continue to cover the oldest supported Xcode (26.2). Together, the PR and nightly runs exercise the integration tests across a wide range of environments, and the jobs that gate PRs take advantage of Xcode 27's improved retry handling for parameterized Swift Testing tests.
Motivation
The integration tests call live backends and run with
-retry-tests-on-failure -test-iterations 3(seescripts/build.sh) to absorb transient network and backend flakiness. Most of them (44 of 53@Tests) are parameterized Swift Testing tests, e.g.,@Test(arguments: InstanceConfig.defaultConfigs), which run one test case per backend and API configuration.Before Xcode 27, retrying a failed Swift Testing test reran the entire test plan, so a single flaky test case caused every integration test to run again. That made retries slow, used extra backend quota, and gave other tests another chance to flake. The Xcode 27 release notes (Testing > Resolved Issues) include fixes for this:
With Xcode 27, only the failing test case (a single argument of a parameterized test) is retried. Running the PR-gating jobs on Xcode 27 makes retries faster and more targeted without reducing coverage.
Integration tests for the oldest supported Xcode version still run nightly, so regressions specific to older toolchains are still caught.
Changes
All AI integration tests now run as entries of a single
testapp-integrationmatrix job:schedule)testapp-integration (xcode-26-build)/(xcode-26-test)macos-15/ Xcode 26.2,build-for-testingmacos-15/ Xcode 26.2, standard integration teststestapp-integration (standard)xcode-27/ Xcode 27.0xcode-27/ Xcode 27.0testapp-integration (live)xcode-27/ Xcode 27.0xcode-27/ Xcode 27.0testapp-integration (gemini-language-model)xcode-27/ Xcode 27.0xcode-27/ Xcode 27.0buildmatrix entry so its name reflects what it does:xcode-26-buildon PRs (compile-only check against the oldest supported Xcode) andxcode-26-testin nightly runs (runs the standard integration tests). It previously only built for testing.macos-26/ Xcode 26.2 toxcode-27/ Xcode 27.0. This also removes the stale "Bump to 26.4" TODOs.continue-on-error: true). Also removed theGEMINI_LANGUAGE_MODELenvironment variable: feat(ai): document and expose GeminiLanguageModel #16631 removed thePackage.swiftcheck that read it, andGeminiLanguageModelis now included unconditionally on Swift 6.4+ toolchains.continue-on-errorandGEMINI_LANGUAGE_MODELgone, thetestapp-integration-liveandtestapp-integration-gemini-language-modeljobs were identical totestapp-integrationapart from their test filter, so they're nowliveandgemini-language-modelmatrix entries (about 45 fewer lines of YAML). Each matrix entry still runs on its own runner, so Live tests stay isolated from the other suites (fix(ci): isolate AI Live integration test runs #16493).scripts/build.shnow passes-collect-test-diagnostics neverto the AI integration test run, as fix(ci): skip test diagnostics collection in SPM test runs #16781 did for SPM test runs. On Xcode 27,xcodebuildrunssimctl diagnoseafter a test attempt fails, which can stall for up to 10 minutes. With-retry-tests-on-failure, a single transient failure could otherwise trigger that stall in the now-blocking Xcode 27 jobs.envinstead of interpolating them directly into therunscript, which avoids zizmortemplate-injectionfindings now that some matrix values are expressions.xcodebuild-standard-…,xcodebuild-xcode-26-build-…). The Live and GeminiLanguageModel artifact names are unchanged.Intentionally omitted on Xcode 26
FoundationModels, so they're compiled out on Xcode 26 and are explicitly excluded there.Environment coverage
Between PRs and nightly runs, the integration tests now cover:
InstanceConfig.defaultConfigsentry in each environment above.This is in addition to the SPM unit tests in
_spm.yml, which continue to run onmacos-15/ Xcode 26.2,macos-26/ Xcode 26.4, andxcode-27/ Xcode 27.0 across all platforms.Notes for reviewers
testapp-integration (build)→testapp-integration (xcode-26-build),testapp-integration-live→testapp-integration (live), andtestapp-integration-gemini-language-model→testapp-integration (gemini-language-model).testapp-integration (standard)keeps its name.macos-15image (previously it only built for testing there)._spm.ymlremain non-blocking (continue-on-error), while the xcode-27 integration test jobs are now blocking.#no-changelog