How best to handle ETIMEDOUT? #425

This is a follow up to #379, which never got resolved, I have created a new issue as I have now done a lot more research now.

Environment

Node version: v14.15.1 for local dev and I think the same on the Azure DevOps Hosted Agents
Npm version: 6.14.8
OS and version: Azure DevOps Hosted Agents
azure-devops-node-api version: 8.1.1

Issue Description

I have a Node based Azure DevOps Pipeline Extension that generates release notes based on a handlebars template.

• Source
• Marketplace

This task uses the Node SDK to make many calls, commonly well over 100, to the Azure DevOps API to get details of WI, CS, PR and tests associated with a build or release. These results are then injected into a Handlebars template.

In most cases this task works without error, but in some cases my users see a error in the form

Error: connect ETIMEDOUT 13.107.42.18:443

What can I do to handle these timeouts?

Is my only option to abandon the Azure DevOps Node SDK and make all my REST calls natively?

Expected behaviour

The task should reliably complete, irrespective of the number Azure DevOps REST API calls made. The SDK should handle timeouts, retry and throttling of the API.

Actual behaviour

There is intermittent failure, the error being in the form.

Error: connect ETIMEDOUT 13.107.42.18:443

The task could run find multiple times, then fail for a few runs. A retry of a job commonly fixes the immediate problem. It is assumed that the Azure DevOps REST API is being saturated and the SDK retry logic is in adequate.

Steps to reproduce

This is hard to reliable reproduce. It appears to occur more often

• On the hosted agents
• In the late afternoon and evening UK time - when USA based clients are busier?
• When a set of release note contains a lot of associated WI/CS and PRs, hence more API calls to get the details

What I have tried

I have tried all of the following, none have helped

API Retry Settings
I have altered the creation of my WebApi instance to increate the timeouts

 const credentialHandler = getCredentialHandler(pat);
 const options = {
       allowRetries: true,
       maxRetries: 20,
  } as vstsInterfaces.IRequestOptions;
  const organisation = new webApi.WebApi(tpcUri, credentialHandler, options);

Retry on failure
I added a try catch block around all of my SDK API calls. If there was a failure I retired the call (using my own code, the the SDK retry system). Within this retry logic I was able to set the number of retries and the time period to pause before a retry.

This had no effect, it seemed as if once there was an SDK reported timeout no re-connection was possible, even if I paused before the reconnect retry for 60 seconds.

Recreate the WebAPI instance on timeout
I refactored my retry logic to recreate the WebApi object on each retry. This again had no effect, the error still occurred

Logs

The only message seen is

Error: connect ETIMEDOUT 13.107.42.18:443

Examples can be seen in this issue 648 on my Release Notes Task Repo