[improvement](build) Run Parquet microbenchmarks in performance CI - #65936
Draft
hello-stephen wants to merge 19 commits into
Draft
[improvement](build) Run Parquet microbenchmarks in performance CI#65936hello-stephen wants to merge 19 commits into
hello-stephen wants to merge 19 commits into
Conversation
### What problem does this PR solve? Issue Number: N/A Related PR: #65921 Problem Summary: The native Parquet decoder and reader benchmark matrix added by #65921 is not built or executed by the community performance admission pipeline. Build the Release benchmark target after the normal performance artifacts, execute all 152 decoder and 137 reader cases as a 1 ms smoke gate, validate the JSON outputs, and publish the raw results as TeamCity artifacts. The smoke samples are diagnostic only and are not treated as performance regression measurements. ### Release note None ### Check List (For Author) - Test: Manual test - `bash -n` for the three performance pipeline scripts - Stubbed benchmark runner validation for the 152/137 case counts and JSON checks - `git diff --check` - Behavior changed: Yes. The community performance pipeline now builds and smoke-runs the native Parquet microbenchmark matrix. - Does this need documentation: No
Contributor
Author
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run performance |
Contributor
|
To prevent injection attacks, single test triggers ( Please review the pipeline script changes carefully. If they are safe, a committer can use |
### What problem does this PR solve? Issue Number: N/A Related PR: #65921 Problem Summary: The initial integration executed the native Parquet microbenchmark matrix inside the performance compile step. That mixed benchmark runtime failures with compilation failures and hid the benchmark duration and artifacts inside the compile stage. Keep only the Release benchmark target build in compile so it can reuse the existing toolchain and dependency setup, and move execution into a dedicated TeamCity step between compile and deploy. ### Release note None ### Check List (For Author) - Test: Manual test - `bash -n` for the performance compile and microbenchmark scripts - Stubbed container-side validation for all 152 decoder and 137 reader results - `git diff --check` - Behavior changed: Yes. Parquet microbenchmark execution now has an independent performance pipeline step. - Does this need documentation: No
### What problem does this PR solve? Issue Number: N/A Related PR: #65921 Problem Summary: The dedicated Parquet microbenchmark step used the triggering commit as a Docker name fallback. Make that fallback safe when the runner is invoked manually outside TeamCity parameters. ### Release note None ### Check List (For Author) - Test: Manual test - `bash -n regression-test/pipeline/performance/run-parquet-microbenchmark.sh` - `git diff --check` - Behavior changed: No - Does this need documentation: No
### What problem does this PR solve? Issue Number: close #65936 Related PR: #65921 Problem Summary: The performance tmp validation built the shared benchmark_test target with GCC and failed before the Parquet step because a pre-existing binary_cast memcpy warning was promoted to an error. Apply the GCC-specific warning downgrade only inside the Parquet benchmark build helper so the base and PR benchmark binaries use identical flags while all other Doris targets retain strict warnings. ### Release note None ### Check List (For Author) - Test: Manual test - bash syntax, ShellCheck, and git diff checks - Behavior changed: No - Does this need documentation: No
### What problem does this PR solve? Issue Number: close #65936 Related PR: #65921 Problem Summary: A clean target-branch worktree successfully built benchmark_test but build.sh --benchmark --output then failed while copying full BE bin files that benchmark-only builds do not produce. Export only the benchmark binary for benchmark builds so both the PR and base revisions can be packaged from clean worktrees. ### Release note None ### Check List (For Author) - Test: Manual test - bash syntax, build.sh help, ShellCheck, and git diff checks - Behavior changed: No - Does this need documentation: No
### What problem does this PR solve? Issue Number: close #65936 Related PR: #65921 Problem Summary: Repeated calibration builds could start minutes apart and merge the PR onto different target-branch revisions, making cross-run noise estimates incomparable. Allow calibration jobs to provide a validated target-branch ancestor SHA while preserving the existing latest-target behavior when the variable is absent. ### Release note None ### Check List (For Author) - Test: Manual test - bash syntax, pinned SHA ancestry, and git diff checks - Behavior changed: No - Does this need documentation: No
### What problem does this PR solve? Issue Number: close #xxx Related PR: #65921 Problem Summary: The performance runtime image does not contain jq, so the Parquet microbenchmark step exited before smoke execution even though the benchmark binaries built successfully. Validate the smoke JSON with Python 3, which is already required by the comparison tool, and suppress shell tracing while publishing the TeamCity artifact service message so it is parsed only once. ### Release note None ### Check List (For Author) - Test: Manual test - bash syntax, Python validator positive and negative fixtures, and git diff checks - Behavior changed: No - Does this need documentation: No
### What problem does this PR solve? Issue Number: close #xxx Related PR: #65921 Problem Summary: The current performance build image provides Python 2.7 but neither jq nor a python3 command. The Parquet microbenchmark therefore exited before smoke execution after both benchmark binaries had built successfully. Make the JSON validator and ABBA comparator run on both Python 2.7 and Python 3, preferring python3 when available and falling back to python without changing the gate policy. ### Release note None ### Check List (For Author) - Test: Manual test - Shell syntax and Python 3 bytecode compilation - Python 2 grammar parsing with 2to3 - Synthetic PASS and stable 20% regression cases - Comparator JSON parity within 1e-12 and byte-identical Markdown output - Behavior changed: Yes, the performance gate now runs in the current Python 2.7 build image - Does this need documentation: No
### What problem does this PR solve? Issue Number: close #xxx Related PR: #65921 Problem Summary: The Parquet benchmark binaries link against libjvm.so, but the runtime build container does not include the JDK server library directory in its dynamic linker path. The benchmark therefore exited before listing or running any smoke cases. Locate libjvm.so from the container JDK and prepend its directory to LD_LIBRARY_PATH before executing either benchmark binary. ### Release note None ### Check List (For Author) - Test: Manual test - Shell syntax, linker-path construction with empty and existing LD_LIBRARY_PATH, and git diff checks - Behavior changed: Yes, the benchmark binary can resolve libjvm.so in the performance runtime container - Does this need documentation: No
### What problem does this PR solve? Issue Number: close #xxx Related PR: #65921 Problem Summary: The current native Parquet benchmark registers 228 decoder and 167 reader cases, while the CI runner still required the earlier exact 152 and 137 counts. Reader benchmark names also gained an explicit value-type dimension. Keep the original counts as minimum coverage floors, validate every currently registered case in the smoke run, report the actual counts, and update the 11 reader performance canaries to their current int32 names. ### Release note None ### Check List (For Author) - Test: Manual test - Verified all 22 gate canaries against build 1006730 cases.txt - Verified the unique 228/167 matrix satisfies the 152/137 coverage floors - Shell syntax and git diff checks - Behavior changed: Yes, legitimate benchmark matrix additions are included in smoke instead of failing an exact stale count - Does this need documentation: No
Contributor
Author
TPC-H: Total hot run time: 29220 ms |
Contributor
Author
TPC-H: Total hot run time: 29470 ms |
Contributor
Author
TPC-DS: Total hot run time: 177419 ms |
Contributor
Author
ClickBench: Total hot run time: 25.2 s |
Contributor
Author
TPC-DS: Total hot run time: 176852 ms |
Contributor
Author
ClickBench: Total hot run time: 25.04 s |
### What problem does this PR solve? Issue Number: close #65936 Related PR: #65921 Problem Summary: The Parquet performance integration mixed Docker orchestration, matrix smoke validation, ABBA execution, retry policy, and threshold configuration in one shell script, and rebuilt the PR benchmark after the normal performance compile. Keep the shell step as a thin container and artifact wrapper, move the test workflow into a Python 2/3 runner, centralize cases and policy in JSON, build doris_be and benchmark_test together for the PR revision, and record raw and gzip-level-1 binary sizes for cache sizing. The target-branch benchmark remains an exact-SHA build; no shared binary cache is enabled before reuse data is available. ### Release note None ### Check List (For Author) - Test: Manual test - Shell syntax for build and performance pipeline scripts - Python 3 bytecode compilation and Python 2 grammar parsing - Synthetic PASS, stable 20% regression, and persistent INCONCLUSIVE retry paths - JSON validation and git diff checks - Behavior changed: Yes. The performance compile reuses its PR benchmark binary and the gate reports explicit infrastructure versus regression exit codes and binary size metrics. - Does this need documentation: No
### What problem does this PR solve? Issue Number: close #65936 Related PR: #65921 Problem Summary: PerformanceTmp injects the liblance_c.a download by matching the existing build.sh --fe --be --clean command. Keep that compatibility anchor while retaining --benchmark as the trailing option, so the injection succeeds and the same build invocation still produces doris_be and benchmark_test. ### Release note None ### Check List (For Author) - Test: Manual test - Shell syntax and git diff checks - Simulated the TeamCity sed injection and verified the resulting command retains --benchmark - Behavior changed: No - Does this need documentation: No
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Related PR: #65921
The native Parquet decoder and reader microbenchmark matrix is available in the
repository, but the community
performanceadmission pipeline neither runs itnor rejects a PR that causes a measurable regression.
What is changed?
parquet-microbenchmarkTeamCity step aftercompileand beforedeploy.that benchmark as the head binary, then build only the benchmark target once
for the exact base SHA in a detached worktree.
build.sh --benchmark --output ...path. This is whybuild.shand the BE CMake files change: the base comparison does not need asecond
doris_beexecutable.dependencies. Both binaries run sequentially on the same TeamCity agent and
pinned CPU.
and 167 reader cases), with minimum-count guards for matrix truncation.
(
base -> PR -> PR -> base), with warmup and five repetitions per phase.ns/raw_row; require identicalraw_rowsandselected_rowscounters before results are comparable.
raw phases, machine-readable comparison JSON, and a Markdown summary.
provides Python 2.
Admission policy
A case blocks the PR only when all of these conditions hold:
A stable 5%-15% regression is reported as a warning. If an apparent regression
of at least 5% exceeds the 3% CV limit, the complete ABBA measurement is retried
once; if it remains inconclusive, the performance step fails as a
measurement/infrastructure failure instead of treating the PR as healthy.
Other unstable cases are reported as noisy.
The full-matrix smoke run is an executability gate only and is not used for a
performance verdict.
After collecting 20-30 clean same-revision A/A runs, the global threshold can be
replaced by per-case thresholds derived from observed noise, for example
max(5%, p99 A/A noise + 2 percentage points).Cache cost decision
The trusted validation measured one
benchmark_testat 5.97 GB raw and 2.305 GBwith gzip level 1. Compressing the head and base binaries took 249 seconds in
total, while rebuilding the benchmark-only base on the existing remote ccache
took about 6 minutes. A separate binary cache would therefore add a 2.305 GB
artifact per base plus transfer, decompression, keying, and retention logic to
save at most about 6 minutes on a hit. This PR keeps the existing ccache and
does not add a second binary cache. The one-off compression measurement was
removed from the final gate, reducing the microbenchmark step from 13m08s to
8m59s.
Validation
Trusted TeamCity
PerformanceTmpbuild:finished / SUCCESS39a54af30babc56f08fdd411176f6515de91597c042e613b1347540e4baf9dc7a395f7009f2a451eremained below the 5% suspicious-regression threshold
The formal
Doris_DorisPerformance_Performanceconfiguration remains unchanged.Only
Doris_DorisPerformance_PerformanceTmphas the ninthparquet-microbenchmarkstep for this validation. Formal promotion shouldfollow A/A calibration and explicit pipeline-owner review.
Check List (For Author)
git diff --checkparquet-microbenchmarkafter compile and before deploy in PerformanceTmp