Timeouts in crypto tests on Raspberry PIs

[mhdawson]

Test

parallel/test-crypto-binary-default paralllel/test-crypto-dh-odd-key (3) parallel/test-crypto-binary-default (2) parallel/test-crypto-dh (2) parallel/test-crypto-dh-constructor (2) parallel/test-crypto-dh-leak (2)

Platform

ARM

Console output

22:20:01   ---
22:20:01   duration_ms: 240.146
22:20:01   severity: fail
22:20:01   exitcode: -15
22:20:01   stack: |-
22:20:01     timeout
22:20:01   ...

Build links

• https://ci.nodejs.org/job/node-test-pull-request/42994/
• • many others, check out reliability repoorts - CI Reliability 2022-03-17 reliability#223

Additional information

These have been flaky for quite a long time

I can see failures in parallel/test-crypto-dh all the way back to October 14 2021 in the reliability reportl - nodejs/reliability#86

[mhdawson]

The update to OpenSSL 3 was around that time

commit 66da32c045035cf2710a48773dc6f55f00e20c40
Author: Daniel Bevenius <daniel.bevenius@gmail.com>
Date:   Wed Apr 14 11:19:54 2021 +0200

    deps,test,src,doc,tools: update to OpenSSL 3.0

    This pull request updates the OpenSSL version that is statically
    linked with Node.js from OpenSSl 1.1.1 to quictls OpenSSL 3.0.0+quic.

    This pull request will replace the OpenSSL version that is currently
    in the deps directory and when performing a normal build
    OpenSSL 3.0+quic will be statically linked to the Node.js executable.
    We will still be able to dynamically link to OpenSSL 1.1.1 and we have
    a CI job which dynamically links to OpenSSL 1.1.1 which is run for
    every pull request to make sure that we maintain backward compatibility.

    PR-URL: https://github.com/nodejs/node/pull/38512
    Reviewed-By: Michael Dawson <midawson@redhat.com>
    Reviewed-By: Tobias Nießen <tniessen@tnie.de>

and it actually landed on the 11th.

My guess is that the update made things a bit slower which is why we are seeing the intermittent timeouts on the PIs

[Trott]

My guess is that the update made things a bit slower

My guess is that a bug was introduced or at least an OS bug or limitation is now being exposed.

One thing that could help would be to modify the tests to either break them up into many smaller test files and/or to add some console.log() statements so that when the test times out, we have some idea how far along it is and how long it was stuck in that state. Right now, all we know is that something locks up somewhere in these (often very long) test files but we have no idea where.

[richardlau]

Since we no longer have Raspberry Pi's in our CI cluster, should we close this?

[mhdawson]

agreed, closing.