First crypto ECDH sign() failes after convertKey() call.

[wdenbakker]

• Version: v11.10.0
• Platform: 64-bit Windows 10
• Subsystem: crypto

After calling ECDH.convertKey() (on an invalid public key) the next call to sign() fails, but subsequent calls succeed.
Example:

const crypto = require("crypto");

const publicKey = Buffer.from("02ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff", "hex");
try {
	crypto.ECDH.convertKey(publicKey, "secp256k1", undefined, undefined, "compressed");
} catch (error) {
	//Lies outside curve, so it should throw.
}

const secp256k1 = crypto.createECDH("secp256k1");
secp256k1.generateKeys();

//Pem format private key
let privateKey = secp256k1.getPrivateKey();
if (privateKey.length < 32) {
	privateKey = Buffer.concat([Buffer.alloc(32 - privateKey.length, 0), privateKey]);
}
const privateStart = Buffer.from("302e0201010420", "hex");
const privateEnd = Buffer.from("a00706052b8104000a", "hex");
const privateKeyPem = "-----BEGIN EC PRIVATE KEY-----\n" +
	Buffer.concat([privateStart, privateKey, privateEnd]).toString("base64") +
	"\n-----END EC PRIVATE KEY-----";

const toSign = "whatever";
try {
	crypto.createSign("SHA256").update(toSign).sign(privateKeyPem);
} catch (error) {
	console.log(error);
	console.log("That threw an error, lets try the same thing again.");
	crypto.createSign("SHA256").update(toSign).sign(privateKeyPem);
	console.log("This time it threw no error.");
}

Resulting error:

Error: error:10067066:elliptic curve routines:ec_GFp_simple_oct2point:invalid encoding
    at Sign.sign (internal/crypto/sig.js:84:29)

[bnoordhuis]

Thanks for the good bug report and test case. A fix is in #26153.