fix: preserve reference layouts in GET_Dataset response

[joaopaulosr95]

Summary

Since b6016e0 ("use hdf5-json util classes"), GET_Dataset in hsds/dset_dn.py
overwrites the dataset's creationProperties.layout with the top-level layout key:

if "creationProperties" in dset_json:
    cpl = dset_json["creationProperties"]
else:
    cpl = {}
if "layout" in dset_json:
    cpl["layout"] = dset_json["layout"]
resp_json["creationProperties"] = cpl

Impact

Any dataset whose stored json has both keys — i.e. anything written by an older HSDS or by hsload --link — silently returns zeros/empty strings instead of its data. Metadata, dtype, shape and attributes are all still correct, so it does not look like a failure.

This covers NREL's public HSDS data on s3://nrel-pds-hsds. Reading NSRDB through 1.0.x returns an all-zero array with no error of any kind; the same read on 0.9.4 returns the real series.

Why CI didn't catch it

POST_Dataset in 1.0.x never stores a top-level layout key (it logs "unexpected key for POST Dataset: 'layout'"). So for datasets created by this version "layout" in dset_json is False, the overwrite never happens, and tests/integ/dataset_test.py::testChunkedRefIndirectDataset — which does assert that class, file_uri and chunk_table survive a GET — passes.

The bug only reaches datasets in the older on-disk shape, which the test suite never constructs.

Reproducer

Against any 1.0.x server, using the public NREL bucket (no credentials needed):

import h5pyd
f = h5pyd.File("/nrel/nsrdb/GOES/tmy/v4.0.0/nsrdb_tmy-2025.h5", "r",
               endpoint=ENDPOINT, bucket="nrel-pds-hsds")
print(f["ghi"].id.dcpl_json["layout"]["class"])
print(f["air_temperature"][:, 1092614].min(), f["air_temperature"][:, 1092614].max())

	0.9.4	1.0.0 / 1.0.1
layout class	H5D_CHUNKED_REF_INDIRECT (+ file_uri, chunk_table)	H5D_CHUNKED
air_temperature min/max	-84 / 377	0 / 0
time_index[:3]	b'2025-01-01 00:30:00', …	b'', b'', b''

The stored object confirms which one is right — s3://nrel-pds-hsds/db/02161936-77b022e7/d/bb36-de0d1a-f7d448/.dataset.json has creationProperties.layout.class == "H5D_CHUNKED_REF_INDIRECT" with file_uri and chunk_table, and a separate top-level layout of {"class": "H5D_CHUNKED", "dims": [2000, 500]}. 1.0.x reports the latter, contradicting its own stored metadata.

The change

GET_Dataset now delegates to a new getDatasetCreationProps() in hsds/util/dsetUtil.py, which keeps 1.0.x's behaviour of reporting the resolved chunk shape except when the creationProperties layout is one of CHUNK_REF_LAYOUTS, where it is preserved intact.

Two smaller things fixed along the way:

• The old code mutated dset_json["creationProperties"] in place. Since dset_json is the meta_cache entry returned by get_metadata_obj, that corrupted the cached metadata for the lifetime of the process, not just the response. The helper returns a copy.
• CHUNK_REF_LAYOUTS is currently duplicated in dset_lib.py and chunk_crawl.py. I put the shared copy in util/dsetUtil.py (a leaf module the DN can import — dset_lib pulls in servicenode_lib, so it isn't importable from dset_dn). Happy to follow up with a patch pointing the other two at it if you'd like that consolidated.

Tests

tests/unit/dset_util_test.py::testGetDatasetCreationProps covers:

• layout in creationProperties only → reported as-is
• both keys, plain chunked → top-level (resolved) shape wins, as in 1.0.x
• both keys, H5D_CHUNKED_REF_INDIRECT → reference preserved (this is the regression)
• same for H5D_CHUNKED_REF and H5D_CONTIGUOUS_REF
• the input dict is not mutated
• no creationProperties, and neither key, both handled

It fails on master and passes with the change. Full unit suite: 87 passed, 1 skipped.
flake8 clean.

Verification

Built a 1.0.1 image with the patch and read NSRDB through it against the real public bucket. The layout is reported correctly again, and a TMY CSV extracted through the patched 1.0.1 is byte-identical (sha256 7486b799…) to the same extraction through 0.9.4.

[joaopaulosr95]

@mattjala this fixes a critical regression. We're currently running a custom build of this in production.