(possibly) Flaky test-doctool-versions

[mmarchini]

I've seen this in a CI (from another collaborator) for a PR to master, and now locally while testing a commit to upstream/v12.x-staging. The CI failure only happened in one build for that PR, so I think it's safe to say this is a flaky.

=== release test-doctool-versions ===                   
Path: doctool/test-doctool-versions
/home/mmarchini/workspace/nodejs/node-v12-backport-postmortem-metadata/test/common/index.js:685
const crashOnUnhandledRejection = (err) => { throw err; };
                                             ^

AssertionError [ERR_ASSERTION]: Did not find entry for '12.x' in [
  { num: '13.x' },
  { num: '10.x' },
  { num: '8.x' },
  { num: '7.x' },
  { num: '6.x' },
  { num: '5.x' },
  { num: '4.x' },
  { num: '0.12.x' },
  { num: '0.10.x' }
]
    at test (/home/mmarchini/workspace/nodejs/node-v12-backport-postmortem-metadata/test/doctool/test-doctool-versions.js:55:12)
    at processTicksAndRejections (internal/process/task_queues.js:97:5) {
  generatedMessage: false,
  code: 'ERR_ASSERTION',
  actual: undefined,
  expected: true,
  operator: '=='
}
Command: out/Release/node /home/mmarchini/workspace/nodejs/node-v12-backport-postmortem-metadata/test/doctool/test-doctool-versions.js

[richardlau]

It's not flaky. #32080 changed the formatting of the changelog so anyone with the pre-updated test will see a failure (since the test fetches the current version of the changelog in master).

It's broken on v12.x-staging until #32146 (which updates the test) lands. v10.x-staging was also broken but #32149 has landed on v10.x-staging so that should be okay now.

[mmarchini]

Thanks, makes sense. I wonder why we saw this on CI then (bad timing between the CI starting and #32080 landing?).

[richardlau]

If the CI was resumed then it will build the original rebased commit from the job being rebuilt instead of being rebased onto the current master.

[mmarchini]

The CI was resumed. Ok, I think we can close this then. Thanks!