Skip to content

fix(testing): add CPU execution flags and standardize E2E checkpoint conversion tests - #4886

Merged
copybara-service[bot] merged 1 commit into
mainfrom
jackyf/fix-e2e-hf-flags
Aug 14, 2026
Merged

fix(testing): add CPU execution flags and standardize E2E checkpoint conversion tests#4886
copybara-service[bot] merged 1 commit into
mainfrom
jackyf/fix-e2e-hf-flags

Conversation

@RexBearIU

@RexBearIU RexBearIU commented Aug 14, 2026

Copy link
Copy Markdown
Collaborator

Description

This PR fixes the E2E Hugging Face checkpoint conversion test suite when executed in CPU environments:

Key Changes

  1. CPU Execution Flags:

    • Added hardware=cpu and skip_jax_distributed_system=True across the conversion test scripts (gemma4, gpt-oss, llama3.1, qwen3, qwen3-vl).
    • Prevents multi-host TPU initialization hangs when executing on single-host CPU pods.
  2. Threshold Alignment:

    • gemma4-26b: Set --max_kl_div=0.10 to account for MoE router boundary sensitivity and 256k vocabulary BF16 tail accumulation on long sequences (peak single-token KL is 0.082, top tokens identical).
    • qwen3-vl-2b: Tightened --max_kl_div=0.03.

Notice 1: Once all tests pass, the "pull ready" label will automatically be assigned.
This label is used for administrative purposes. Please do not add it manually.

Notice 2: For external contributions, our settings currently require an approval from a MaxText maintainer to trigger CI tests.

Tests

  • Verified on GKE CPU cluster across the E2E conversion test suite.

Checklist

Before submitting this PR, please make sure (put X in square brackets):

  • I have performed a self-review of my code. For an optional AI review, add the gemini-review label.
  • I have necessary comments in my code, particularly in hard-to-understand areas.
  • I have run end-to-end tests tests and provided workload links above if applicable.
  • I have made or will make corresponding changes to the doc if needed, including adding new documentation pages to the relevant Table of Contents (toctree directive) as explained in our documentation.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates several end-to-end TPU test scripts to download pre-computed golden logits from Google Cloud Storage instead of generating them on the fly or running Hugging Face models. It also standardizes the test execution by adding hardware=cpu and skip_jax_distributed_system=True parameters. The reviewer suggests using [ ! -s ... ] instead of [ ! -f ... ] when checking for the downloaded logit files to robustly handle potentially empty files from interrupted downloads.

Comment thread tests/end_to_end/tpu/gemma3/4b/test_gemma3_to_mt.sh Outdated
Comment thread tests/end_to_end/tpu/gemma4/26b/test_gemma4_to_mt.sh Outdated
Comment thread tests/end_to_end/tpu/llama3.1/70b/test_llama3.1_70b_to_mt.sh Outdated
Comment thread tests/end_to_end/tpu/qwen3/30b/test_qwen3_to_mt.sh Outdated
Comment thread tests/end_to_end/tpu/qwen3/vl_2b/test_qwen3_to_mt.sh Outdated
@codecov

codecov Bot commented Aug 14, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@RexBearIU RexBearIU changed the title fix(testing): standardize E2E checkpoint conversion tests to use GCS golden logits and CPU flags fix(testing): add hardware=cpu flags to E2E checkpoint conversion tests Aug 14, 2026
@RexBearIU
RexBearIU requested a review from parambole as a code owner August 14, 2026 09:37
@RexBearIU
RexBearIU force-pushed the jackyf/fix-e2e-hf-flags branch from 32c4b08 to ad289d1 Compare August 14, 2026 10:15
@RexBearIU RexBearIU changed the title fix(testing): add hardware=cpu flags to E2E checkpoint conversion tests fix(layers,testing): fix YaRN RoPE dimension calculation and standardize E2E conversion tests Aug 14, 2026
@RexBearIU
RexBearIU force-pushed the jackyf/fix-e2e-hf-flags branch from ad289d1 to 80b202d Compare August 14, 2026 11:27
@RexBearIU RexBearIU changed the title fix(layers,testing): fix YaRN RoPE dimension calculation and standardize E2E conversion tests fix(testing): add CPU execution flags and standardize E2E checkpoint conversion tests Aug 14, 2026
@RexBearIU
RexBearIU force-pushed the jackyf/fix-e2e-hf-flags branch 2 times, most recently from 2965137 to 3d7a332 Compare August 14, 2026 13:23
…conversion tests

- Add missing hardware=cpu and skip_jax_distributed_system=True flags to E2E shell scripts for consistent CPU execution.
- Adjust max_kl_div threshold to 0.10 for Gemma 4 26B and tighten Qwen3-VL 2B threshold to 0.03.
@RexBearIU
RexBearIU force-pushed the jackyf/fix-e2e-hf-flags branch from 3d7a332 to 4b5761c Compare August 14, 2026 16:38
@copybara-service
copybara-service Bot merged commit 1544e08 into main Aug 14, 2026
68 of 70 checks passed
@copybara-service
copybara-service Bot deleted the jackyf/fix-e2e-hf-flags branch August 14, 2026 18:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants