fix: split H5D_CONTIGUOUS_REF datasets into virtual chunks again

[d-gski]

Problem

Since 1.0 the chunk shape comes from h5json.dset_util.getChunkDims, which returns the full dataset shape for any layout that isn't H5D_CHUNKED*. For an H5D_CONTIGUOUS_REF dataset that makes the whole dataset one chunk: reading a single element makes the DN range-get, decode and cache the entire dataset from the linked file. 0.9.x split contiguous references into virtual chunks sized from min_chunk_size/max_chunk_size. #468/#469 rightly keep the reference layout in the GET_Dataset response, but nothing puts that split back.

We hit this serving NREL's NSRDB TMY domains (nrel-pds-hsds), where each meta dataset is an H5D_CONTIGUOUS_REF (MSG msg_tmy-2022.h5: 2,693,287 × 134 B = 361 MB; GOES nsrdb_tmy-2025.h5: 262 MB). On 1.0.1 + #469:

• a one-row meta read fetched all 361 MB, which took 6–34 s from S3 in our deployment;
• the two meta datasets no longer fit a 512m chunk cache, so they evicted each other and every read re-fetched;
• requests that arrived during a fetch waited store_read_timeout (1 s) and then each started another full copy, taking a DN from 411 MB to 1.56 GB within a minute.

How this regressed

As far as I can tell, this is an unintended side effect of the h5json migration rather than a design change. Please say if single-chunk reads were meant, and I'll rethink the approach.

• 0.9.x chunked contiguous references on purpose. POST_Dataset gave every contiguous reference a virtual chunk shape (getContiguousLayout), and testContiguousRefDataset required a chunked layout with a chunk size between CHUNK_MIN and CHUNK_MAX. Selection can fail for H5D_CONTIGOUS_REF datasets #393/fix for contiguousref datasets #396 fixed an edge case of exactly this, the short last chunk, on an NREL meta dataset.
• b6016e0 removed it. That commit ("use hdf5-json util classes", in H5json #450) moved layouts under creationProperties and deleted HSDS's getContiguousLayout along with the computation. It also switched chunk-shape lookup to h5json.dset_util.getChunkDims. h5json's rule is sound for data HSDS stores itself, because generateLayout only picks H5D_CONTIGUOUS below chunk_min. A contiguous reference's size comes from the linked file instead.
• The read path still assumes virtual chunks. None of the following can trigger with a single chunk:
  • getChunkLocations computes a byte offset per chunk index, and warns when one runs past the end of the dataset;
  • the DN pads a short final chunk (fix for contiguousref datasets #396);
  • updateDatasetInfo describes contiguous references as "divided into equal size chunks";
  • dataset creation still rejects client-supplied dims for them.
• It isn't listed as a change. Neither the 1.0.0 release notes nor the README's "Breaking Changes in v1.0.0" mention it.
• Nothing surfaced it. The contiguous-reference read tests use 80- and 60-byte datasets from tall.h5. On domains linked by earlier versions, these datasets returned fill values instead, because of the reference-layout bug that fix: preserve reference layouts in GET_Dataset response #468/fix: preserve and test reference layouts in GET_Dataset response #469 fixed. The single-chunk reads only showed up once that fix landed.

This PR leaves the intended parts of b6016e0 as they are: layout lives only under creationProperties, GET_Dataset has no top-level layout, and a reference layout reports no dims.

Fix

Add getChunkDims to hsds/util/dsetUtil.py and use it in place of h5json's in the seven modules that imported it. It defers to h5json for every layout except H5D_CONTIGUOUS_REF. For that one it derives virtual chunks with h5json's getContiguousLayout, using the dataset shape, item size and min_chunk_size/max_chunk_size. That's the same split 0.9.x made: 21042 rows for the MSG meta above, identical to the dims stored with that domain.

The shape is derived, not read from a stored layout. No chunk objects exist for a contiguous reference, since each virtual chunk is a range get into the file, so the shape only has to agree between two places:

• the SN, which turns a chunk index into a byte range in getChunkLocations;
• the DN, which decodes and caches that range in get_chunk_bytes.

Both already share the inputs, so deriving it gives agreement without changing the GET_Dataset response. That response still reports the reference layout without dims, as #468/#469 and their tests intend, and clients see no change.

Testing

• New testGetChunkDims in tests/unit/dset_util_test.py. All unit tests in testall.py pass, and flake8 is clean.
• Against NREL's public NSRDB data, I compared native builds of 0.9.4, of 1.0.1 + fix: preserve and test reference layouts in GET_Dataset response #469 (20ec2e6 + 0131551), and of this branch. Each read one site: its meta record plus 12 series.

	0.9.4	1.0.1 + #469	this branch
meta storage read per row	2,819,628 B	360,900,458 B (whole dataset)	2,819,628 B (MSG), 4,099,680 B (GOES)
cold read of one site	17.6–19.2 s	62.6–78.3 s	18.2–21.9 s
alternating GOES/MSG sites, 512m cache, after warm-up	10.2–12.8 s	62.9–83.1 s (meta re-fetched every read)	12.1–16.5 s
4 concurrent cold MSG reads, DN peak memory	606 MB	2,553 MB	648 MB

The timings are from a laptop reading us-west-2 S3, so the absolute numbers are only indicative. The gap between builds is what matters.

• I also checked byte-for-byte equality with 0.9.4 for the meta record and all 12 series at 10 sites across both datasets: 130 of 130 arrays identical. The sites include:
  • the first row;
  • both sides of the first virtual-chunk boundary;
  • the last row, which sits in the short final chunk the DN pads out.

Upgrade note

This changes what a chunk id means for H5D_CONTIGUOUS_REF datasets: _0 goes from the whole dataset to the first virtual chunk. Chunk caches are in memory, so a restart clears them. But an SN and a DN on different versions would disagree on the chunk shape, so restart all nodes together rather than rolling them one at a time.

cc @joaopaulosr95 (#468) and @mattjala (#469).

[mattjala]

Thank you for the detailed writeup. I think it'd be best to fix this in h5json's dset_util.getChunkDims itself rather than covering it from HSDS's side with a parallel definition. Other h5json code that would still use its own getChunkDims (e.g. Hdf5db.ChunkIterator) would keep the same whole-dataset behavior, so fixing it over there would deal with multiple cases at once. I plan to put out an h5json patch release soon that includes this and bump the h5json requirement to pick it up.