Silent data corruption when two dictionaries share the same Dictionary_ID (no error raised) #4845

Describe the bug

When a zstd frame that was compressed with dictionary A is decompressed by a decoder that was given a different dictionary B — where B declares the same Dictionary_ID as A — all decoders we tested (libzstd, klauspost/compress, educational decoder) silently decode the frame to byte-identical wrong output. No error, warning, or diagnostic is produced. The corrupted output has the same length as the correct one, so downstream consumers cannot tell anything is wrong.

To Reproduce

Self-contained script (requires only pip install zstandard):

import hashlib, random
import zstandard as z

WORDS = ["temp","humidity","pressure","status","device","sensor","alert","battery","uptime","region"]
REGIONS = ["eu","us","ap","sa"]
def make(seed, n=300):
    rng = random.Random(seed); lines = []
    for i in range(n):
        w = rng.choice(WORDS); st = "OK" if rng.random() < 0.9 else "WARN"
        lines.append(f"{w}={rng.randint(0,1023)},region={rng.choice(REGIONS)},seq={i},status={st}")
    return ("\n".join(lines) + "\n").encode()

corpusA = [make(100+i) for i in range(40)]
corpusB = [make(500+i) for i in range(40)]   # different data, same forced dict ID
sample  = make(777)

def train(did, samples):
    return z.train_dictionary(11264, samples, dict_id=did, k=1024, d=8).as_bytes()

A = train(100, corpusA)   # team A's dictionary, ID 100
B = train(100, corpusB)   # team B's dictionary, SAME ID 100, different content

frame = z.ZstdCompressor(level=5, dict_data=z.ZstdCompressionDict(A)).compress(sample)
D = lambda b: z.ZstdDecompressor(dict_data=z.ZstdCompressionDict(b))
truth  = D(A).decompress(frame)
silent = D(B).decompress(frame)   # <- no error raised here

print("correct  output:", hashlib.sha256(truth).hexdigest()[:16], len(truth))
print("collision output:", hashlib.sha256(silent).hexdigest()[:16], len(silent))
print("identical:", truth == silent)
print(truth[:100].decode())
print(silent[:100].decode())

Actual output (zstandard 0.25.0 / libzstd 1.5.7, 2026-10-15):

dict IDs     : 100 100
correct  output sha256: 2ef0986eb28d2286 (11667 bytes)
collision output sha256: c784885d2655e6eb (11667 bytes)
identical: False
--- correct output, first 100 bytes ---
status=754,region=ap,seq=0,status=OK
sensor=81,region=sa,seq=1,status=OK
region=591,region=eu,seq=2,
--- collision output, first 100 bytes ---
39,regi7237,status=OK
dev38,status=OK
device826,region=ap,seqs=OK
status=82,regi5us,seq=223,status==

The collision output keeps the line structure and status=OK fields of the original, so it looks superficially plausible while being semantically garbage.

Scope: this is not specific to libzstd. We reproduced the identical silent corruption (byte-identical wrong output across implementations) with libzstd 1.5.7, klauspost/compress v1.20.3 (Go), and the educational decoder. Decoding the same frame with no dictionary at all does error out everywhere; the collision case is the one that slips through.

Context / why this matters

• The only mechanism the format provides against dictionary substitution is the Dictionary_ID itself, and a decoder can only check that the ID matches — it cannot verify the dictionary content is the one the frame was produced with.
• RFC 8878 §3.1.1.1.3 permits any Dictionary_ID ≥ 32768 "by private arrangement between participants", so two independent parties choosing the same ID is a realistic failure mode. (RFC 8878 §7.4 also states that no IANA registry was ever created.) An erratum against RFC 8878 §3.1.1.1.3 covering the contradictory dictionary-ID rules has been filed with the RFC Editor (pending screening).
• Partial mitigation that already works: if the frame carries a Content_Checksum, libzstd and klauspost detect the collision (Restored data doesn't match checksum / CRC check failed). However, RFC 8878 §2 explicitly permits a decoder to ignore the checksum ("It may also ignore informative fields, such as the checksum"), and we observed a decoder doing exactly that.

Proposal

Add a recommendation to doc/zstd_compression_format.md (and the CLI/man page for -D) that frames produced with a dictionary SHOULD include a Content_Checksum, since the checksum is currently the only mechanism that can detect same-ID/different-dictionary substitution.