-
Notifications
You must be signed in to change notification settings - Fork 407
test: [M3-7852] Use test weights for Jenkins/Cypress parallelization #10310
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
jdamore-linode
merged 18 commits into
linode:develop
from
jdamore-linode:poc-cypress-test-weights
Aug 26, 2024
Merged
Changes from all commits
Commits
Show all changes
18 commits
Select commit
Hold shift + click to select a range
6fcf9a8
WIP
f803ce8
Merge branch 'develop' into poc-cypress-test-weights
94de069
Merge branch 'develop' into poc-cypress-test-weights
bc76b24
Add test weight data for core suite
e2948fd
Generate weights file after run when plugin is enabled
e6e3d70
Distribute parallelized tests using weight data
7f3dc89
Update weights
718b791
Merge branch 'develop' into poc-cypress-test-weights
80faaf3
Allow weights file read path to be determined using environment variable
5a898d4
Delete weights file
d260261
Document `CY_TEST_SPLIT_RUN_WEIGHTS` and `CY_TEST_GENWEIGHTS`
e973ec8
Improve type
150fb4d
Merge branch 'develop' into poc-cypress-test-weights
491e7ee
Feedback, first pass
20add0e
Improve type
956b1f0
Clean up run splitting / test weight logic
1a6c3de
Move helpers after plugin function
58ececf
Merge branch 'develop' into poc-cypress-test-weights
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
155 changes: 155 additions & 0 deletions
155
packages/manager/cypress/support/plugins/generate-weights.ts
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,155 @@ | ||
| import type { CypressPlugin } from './plugin'; | ||
| import { DateTime } from 'luxon'; | ||
| import { writeFileSync } from 'fs'; | ||
| import { resolve } from 'path'; | ||
| import { object, string, array, number, SchemaOf } from 'yup'; | ||
|
|
||
| // The name of the environment variable to read to check if generation is enabled. | ||
| // The value should be a path to the weights file. | ||
| const envVarName = 'CY_TEST_GENWEIGHTS'; | ||
|
|
||
| /** | ||
| * Describes spec file weights for a test suite. | ||
| */ | ||
| export interface SpecWeights { | ||
| /** | ||
| * Spec weight metadata. | ||
| */ | ||
| meta: { | ||
| /** | ||
| * Date and time that test spec weights were generated. | ||
| */ | ||
| datetime: string; | ||
|
|
||
| /** | ||
| * Total test weight. | ||
| */ | ||
| totalWeight: number; | ||
|
|
||
| /** | ||
| * Total test run duration in milliseconds. | ||
| */ | ||
| totalDuration: number; | ||
| }; | ||
| /** | ||
| * Array of spec weights. | ||
| */ | ||
| weights: SpecWeight[]; | ||
| } | ||
|
|
||
| /** | ||
| * Describes the weight of an individual spec file. | ||
| */ | ||
| export interface SpecWeight extends SpecResult { | ||
| /** | ||
| * Spec weight. | ||
| */ | ||
| weight: number; | ||
| } | ||
|
|
||
| /** | ||
| * Spec weights schema for JSON parsing, etc. | ||
| */ | ||
| export const specWeightsSchema: SchemaOf<SpecWeights> = object({ | ||
| meta: object({ | ||
| datetime: string().required(), | ||
| totalWeight: number().required(), | ||
| totalDuration: number().required(), | ||
| }).required(), | ||
| weights: array( | ||
| object({ | ||
| filepath: string().required(), | ||
| duration: number().required(), | ||
| weight: number().required(), | ||
| }) | ||
| ).required(), | ||
| }); | ||
|
|
||
| /** | ||
| * Describes the duration of an individual spec file. | ||
| * | ||
| * Used in the process of calculating weights for each spec. | ||
| */ | ||
| interface SpecResult { | ||
| /** | ||
| * Relative path to spec file. | ||
| */ | ||
| filepath: string; | ||
|
|
||
| /** | ||
| * Spec run duration in milliseconds. | ||
| */ | ||
| duration: number; | ||
| } | ||
|
|
||
| /** | ||
| * Enables test weight generation when `CY_TEST_GENWEIGHTS` is defined. | ||
| * | ||
| * @returns Cypress configuration object. | ||
| */ | ||
| export const generateTestWeights: CypressPlugin = (on, config) => { | ||
| const specResults: SpecResult[] = []; | ||
|
|
||
| if (!!config.env[envVarName]) { | ||
| const writeFilepath = config.env[envVarName]; | ||
|
|
||
| // Capture duration after each spec runs. | ||
| on('after:spec', (spec, results) => { | ||
| const duration = results.stats.duration; | ||
| if (duration) { | ||
| specResults.push({ | ||
| filepath: spec.relative, | ||
| duration, | ||
| }); | ||
| } else { | ||
| console.warn( | ||
| `Failed to record test information for '${spec.relative}'` | ||
| ); | ||
| } | ||
| }); | ||
|
|
||
| // Aggregate spec durations and save as a spec weights JSON file. | ||
| on( | ||
| 'after:run', | ||
| ( | ||
| results: | ||
| | CypressCommandLine.CypressRunResult | ||
| | CypressCommandLine.CypressFailedRunResult | ||
| ) => { | ||
| // Determine whether this is a failed run. "Failed" in this context means | ||
| // that Cypress itself failed to run, not that the test results contained failures. | ||
| const isFailedResult = ( | ||
| results: | ||
| | CypressCommandLine.CypressRunResult | ||
| | CypressCommandLine.CypressFailedRunResult | ||
| ): results is CypressCommandLine.CypressFailedRunResult => { | ||
| return 'failures' in results; | ||
| }; | ||
|
|
||
| if (!isFailedResult(results)) { | ||
| const totalWeight = 100; | ||
| const totalDuration = results.totalDuration; | ||
| const weights: SpecWeights = { | ||
| meta: { | ||
| datetime: DateTime.now().toISO(), | ||
| totalWeight, | ||
| totalDuration, | ||
| }, | ||
| weights: specResults.map( | ||
| (specResult: SpecResult): SpecWeight => { | ||
| return { | ||
| filepath: specResult.filepath, | ||
| duration: specResult.duration, | ||
| weight: (specResult.duration / totalDuration) * totalWeight, | ||
| }; | ||
| } | ||
| ), | ||
| }; | ||
|
|
||
| const resolvePath = resolve(writeFilepath); | ||
| writeFileSync(resolvePath, JSON.stringify(weights), 'utf-8'); | ||
| } | ||
| } | ||
| ); | ||
| } | ||
| }; |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should these be added to our
.env.example? What are the defaults you've found to be a good balance for these?Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hm, I don't think these belong there since they're more intended for CI and not super practical to use locally (you could, but the cost of having multiple Cypress and Chrome instances running at once plus the hassle of setting up separate test accounts for each runner probably makes it more trouble than it's worth).
However, I've never really considered documenting any of these in
.env.example, so I'll take a quick look later and add any that might make sense! Thanks Jaalah!Edit: It also just occurred to me that I've always just assumed our
.env.examplefile is specifically for documenting variables for local usage/development, but if we consider it to be a complete list of all the env vars that might be relevant to Cloud then I'm definitely in favor of adding all this there