[andrewheard]

GeminiLanguageModel's integration tests run against live Gemini backends, so they're slow, cost money, need credentials, and fail when a backend misbehaves. This PR adds record and replay support using google/test-server. Every GLM integration suite can now replay from committed recordings in under a second, without credentials or network access to Gemini. That means they can run on every pull request, including PRs from forks.

	Live	Replay
Credentials	Required	None
Runs on fork PRs	No	Yes
Time (20 tests, 40 cases)	Tens of seconds	~0.3 s
Flaky backends	Can fail tests	Deterministic

How it works

INTEGRATION_TEST_MODE selects how IntegrationTestingBackend sends requests:

Mode	Requests go to	Credentials
live (default)	The live backends	Required
record	test-server, which forwards them and saves each interaction	Required
replay	test-server, which answers from recordings	Not used

• makeModel() (and the new makeClient() for GeminiAPIClient) sends a Test-Name header derived from the calling test's file, function, and backend, so each parameterized case gets its own recording under GeminiLanguageModel/Tests/TestServer/recordings/.
• test-server matches a request by its method, URL, JSON body (key order doesn't matter), the previous request in the same recording, and every header that isn't redacted in test-server.yml. This makes replay a strict check on request encoding: an SDK change that alters a request fails replay until the affected tests are re-recorded.

cd GeminiLanguageModel
scripts/test-server.sh replay IntegrationTests              # No credentials needed
scripts/test-server.sh record ToolCallingIntegrationTests   # Requires credentials
scripts/test-server.sh diff IntegrationTests                # Explains replay mismatches
scripts/test-server.sh audit                                # Checks recorded headers

See Tests/TestServer/README.md for details.

Changes

Test harness (GeminiTestUtilities)

• Add IntegrationTestMode and TestServerConfiguration. An unrecognized INTEGRATION_TEST_MODE value stops the test run instead of silently falling back to live requests.
• In record and replay modes, route IntegrationTestingBackend requests to the local test-server proxy. In replay mode, every backend is available without credentials, the project ID is REDACTED, and the App Check exchange is skipped.
• Add an authenticated option to makeHeaderProvider so unauthenticated-request tests can be recorded.
• Comment out .developerAPI in IntegrationTestingBackend.allCases, so integration tests target Firebase AI Logic only. It can be uncommented for manual testing.

Tests

• Parameterize GeminiAPIClientIntegrationTests over IntegrationTestingBackend. The separate Firebase-specific test is now covered by the parameterized simple prompt test.
• Add 40 recordings covering all GLM integration suites (tool calling, basic content generation, guided generation, and API client) against both Firebase AI Logic backends.
• Add unit tests for the mode handling and the Python tooling.

Tooling (GeminiLanguageModel/scripts)

• test-server.sh:
  • Downloads a pinned, checksum-verified test-server (v0.2.9).
  • Passes credential values to test-server so they're redacted anywhere they appear while recording.
  • Runs tests serially, because test-server tracks request order per endpoint.
  • Removes credentials from the environment before replaying.
  • Re-records failed tests with a freshly started server. Retrying against the same server would leave the failed attempt in the recording.
• test_server_diagnostics.py (Python standard library only):
  • Explaining mismatches: when replay fails, the script re-runs the failed tests behind a capture proxy and diffs each unmatched request against its recording, because test-server itself only returns a hash.
  • Header audit: fails if a recording contains any header not listed in recording-policy.json, whatever its value. A new credential header such as Authorization: Bearer … can't be committed by accident, even though gitleaks and trufflehog both missed it in testing. Noisy response headers such as Date are stripped. Only header names are printed, never values.

CI (sdk.geminilanguagemodel.yml)

• audit-recordings (Linux): runs the tooling unit tests and audits the committed recordings.
• replay-integration-tests (macOS): replays all integration suites without secrets, including on fork PRs. It fails if suites are skipped because FoundationModels is unavailable, rather than passing without running anything.

Testing

• Recorded every suite against the live Firebase AI Logic backends. All tests passed on the first attempt.
• Replayed every suite with credentials removed from the environment, more than once: 19 tests (38 cases) pass, and 1 test is skipped as described below.
• Confirmed that no credential values appear in any recording, and that project IDs in URLs and error messages are REDACTED.
• Bug detection: temporarily reintroduced the bug fixed in fix(ai): attach signature-only reasoning to a part with data #16886 (a data-less signature part). Replay caught it on every backend in 0.2 s, and the mismatch explanation pinpointed the moved thoughtSignature. Recording the same bug live never reproduced the backend's nondeterministic 400 in 9 runs.
• Retries: simulated a flaky backend partway through a recording. The script re-recorded only the failed test, and the final recording contained no trace of the failed attempt.
• swift test --skip IntegrationTests, swift-format lint --strict, shellcheck, pyink, and the Python unit tests all pass.

Known limitations and follow-ups

• streamGenerateContentWithoutAuthThrows is skipped in replay mode. test-server v0.2.9 replays a JSON error body on a streaming (alt=sse) URL as a server-sent event, so GeminiAPIClient reports .httpError instead of .apiError. A ~15-line upstream fix that records each response's body format has been verified locally. Once it's released, the skip can be removed and the test-server version bumped.
• Re-running a test against the same replay server fails, so -test-iterations and -retry-tests-on-failure don't apply in replay mode. Replay is deterministic, so they aren't needed.
• Intentional changes to request encoding require re-recording, which needs credentials.
• Follow-ups:
  • Stop sending synthesized function call IDs to the backend (separate PR).
  • An exact-value secret check and gitleaks rules for recording bodies.
  • Nightly drift detection that compares fresh live recordings against the committed ones.

API changes

None. All changes are in tests, test utilities, scripts, and CI.

#no-changelog

[gemini-code-assist]

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.