JASMIN file availability very patchy

[valeriupredoi]

We notice fairly frequent ie 5-10% test failures of this kind - these are not created by issues with either client or Reductionist, it's JASMIN NGINX server not being up - I need to speak to CEDA about this.

[valeriupredoi]

RE: comms with JASMIN: April 24 2-8am was a bad time for the servers; Dave Poultier mentioned: 500 / 501- tell us every time
502 / 503 / 504: tell us if you get more than 1 in 10,000. Unexpected 401 or 403 - tell us every time (i.e. - it's a public file complaining about auth, or it's token you know is valid not working). Anything else - ignore

HTTPSConnectionPool(host='esgf.ceda.ac.uk', port=443): Max retries exceeded with url: /thredds/fileServer/esg_cmip6/CMIP6/AerChemMIP/MOHC/UKESM1-0-LL/ssp370SST-lowNTCF/r1i1p1f2/Amon/cl/gn/latest/cl_Amon_UKESM1-0-LL_ssp370SST-lowNTCF_r1i1p1f2_gn_205001-209912.nc (Caused by ConnectTimeoutError(<HTTPSConnection(host='esgf.ceda.ac.uk', port=443) at 0x7f116e2f6210>, 'Connection to esgf.ceda.ac.uk timed out. (connect timeout=None)'))

-> so 408 is most of the things we get

[valeriupredoi]

RE even more comms with JASMIN (Dave Poultier): OK I've got some numbers for you.

You tests generate 4614 requests to that netCDF file (ignoring extension - explanation below), during your tests this afternoon (including the .das and .dds endpoints for opendap protocol). Of these:

4278 yielded a 206 (successful RANGE request)
285 yielded a 200 (complete transfer of .das or .dds endpoint)
51 resulted in a 404

I see no evidence of a single 408 response

However, what I do see in my logs, is that everyone of those 404 requests was for a file ending in .ncx (which I am not familiar with).

Here is an example ingress-haproxy output:
2026-04-24T13:19:11+00:00 127.0.0.1 1 2026-04-24T13:19:11.080837+00:00 - ingress 2352 - - 20.171.125.211:44039 [24/Apr/2026:13:19:11.080] _front_https~ esgf_esgf-file-server_http/srv001 0/0/0/0/0 404 123 - - ---- 111/107/20/20/0 0/0 "HEAD /thredds/fileServer/esg_cmip6/CMIP6/AerChemMIP/MOHC/UKESM1-0-LL/ssp370SST-lowNTCF/r1i1p1f2/Amon/cl/gn/latest/cl_Amon_UKESM1-0-LL_ssp370SST-lowNTCF_r1i1p1f2_gn_205001-209912.ncx HTTP/1.1"

Is this you guys testing for a 404?

Either way - every single request that our ingress controller received for that file during that timeframe resulted in a 200, 206 or a 404 - all of which I think you are expecting.

Which is another way of saying, I can't find any timeouts from your tests, I wonder if the timeout is actually at the github end?

[valeriupredoi]

Myself and @varsiha-sothilingam we will set up a set of (very verbose) tests that should investigate this behaviour - if it's a 404 then we should be sure where that comes from (JASMIN, GHA machine, somewhere inbetween?)

[valeriupredoi]

Summary for call with CEDA on June 4, 2026

• simple GETs test to NGINX server, throttling the amount of stress it receives [DNM] Concurrency test that tests NGINX storage servers at CEDA #333
• replicate traffic to NGINX server via GETs throttled by max number of threads sent to Reductionist [DNM][Test CEDA NGINX] Reduce maxthreads #335 with interim solution (practical) to decrease max threads from 100 to 30 reduce max threads to 30 #336
• temporary unavailability of files https://github.com/NCAS-CMS/PyActiveStorage/actions/runs/26575027754/job/79316678805?pr=336
• comparison with Apache server - tests done by @varsiha-sothilingam
• comparison with DKRZ NGINX server

Call notes

[valeriupredoi]

we can safely close this one, file availability has increased manyfold now, thanks to Dan and the CEDA folks