OperatorSequence (fused): changing the runlist under the same name silently reuses the old ELF #237

Summary

FusedDispatch names its artifacts after the sequence name only:

• {name}_fused.mlir (iron/common/sequence.py:168);
• {name}.elf (iron/common/sequence.py:136).

SequenceMLIRArtifact depends only on the per-operator MLIR files (iron/common/compilation/sequence.py:48), and availability is decided by mtime alone (CompilationArtifact.is_available_in_filesystem, iron/common/compilation/base.py:312). The runlist, buffer layout and slice info are not part of either the name or the dependencies.

So if a sequence is rebuilt in the same build dir with a different runlist that uses already-built operators, nothing looks out of date. The old ELF runs: no error, no warning, exit code 0, wrong output.

All line numbers refer to devel @ 79c04e6.

Reproduction

This runs on an NPU2 (Strix) machine. It uses dispatch="fused", which is also what "auto" picks on NPU2.

git clone --branch devel https://github.com/amd/IRON.git
python3.12 -m venv venv && source venv/bin/activate
pip install mlir_aie -f https://github.com/Xilinx/mlir-aie/releases/expanded_assets/latest-wheels-4
pip install llvm-aie -f https://github.com/Xilinx/llvm-aie/releases/expanded_assets/nightly
pip install --index-url https://download.pytorch.org/whl/cpu "torch==2.13.0+cpu"
pip install --no-deps -e IRON ml_dtypes
source /opt/xilinx/xrt/setup.sh

python stale_fused.py --runlist add_relu --build-dir build        # out = relu(a + b), correct
python stale_fused.py --runlist add_add  --build-dir build        # expects (a + b) + b
python stale_fused.py --runlist add_add  --build-dir build_fresh  # control

Versions verified: IRON devel @ 79c04e6, mlir_aie==1.4.4.dev73+g18ca6c1, llvm-aie==22.0.0.2026092101+0006955e. Platform: Strix HX 370 (npu2), amdxdna 2.25.0.

# stale_fused.py
# Build and run a fused OperatorSequence named "repro" in a fixed build dir.
# Running --runlist add_relu and then --runlist add_add in separate processes
# reuses the first run's fused ELF for the second runlist.
import argparse
from pathlib import Path

import torch
import aie.utils as aie_utils

from iron.common.context import AIEContext
from iron.common.sequence import OperatorSequence
from iron.operators.elementwise_add.op import ElementwiseAdd
from iron.operators.relu.op import ReLU

SIZE, TILE, COLS = 4096, 1024, 4


def main():
    p = argparse.ArgumentParser()
    p.add_argument("--runlist", choices=["add_relu", "add_add"], required=True)
    p.add_argument("--build-dir", type=Path, required=True)
    opts = p.parse_args()

    ctx = AIEContext(build_dir=opts.build_dir)
    add = ElementwiseAdd(size=SIZE, tile_size=TILE, num_aie_columns=COLS, context=ctx)
    relu = ReLU(
        size=SIZE, num_aie_columns=COLS, num_channels=1, tile_size=TILE, context=ctx
    )
    if opts.runlist == "add_relu":
        runlist = [(add, "a", "b", "temp"), (relu, "temp", "out")]
    else:
        runlist = [(add, "a", "b", "temp"), (add, "temp", "b", "out")]

    seq = OperatorSequence(
        name="repro",
        runlist=runlist,
        input_args=["a", "b"],
        output_args=["out"],
        dispatch="fused",
        context=ctx,
    )
    seq.compile()

    torch.manual_seed(0)
    a = torch.rand(SIZE, dtype=torch.bfloat16) * 4 - 2
    b = torch.rand(SIZE, dtype=torch.bfloat16) * 4 - 2
    run = seq.get_callable()
    for name, data in (("a", a), ("b", b)):
        buf = run.get_buffer(name)
        buf.torch_view()[:SIZE] = data
        buf.to("npu")
    run()
    out = run.get_buffer("out").torch_view()[:SIZE].float()

    if opts.runlist == "add_relu":
        expected = torch.relu(a + b).float()
    else:
        expected = ((a + b) + b).float()
    relu_ab = torch.relu(a + b).float()
    print(f"--runlist {opts.runlist}")
    print(f"  max |out - expected|     = {(out - expected).abs().max().item():.4f}")
    print(f"  max |out - relu(a + b)|  = {(out - relu_ab).abs().max().item():.4f}")
    aie_utils.DefaultNPURuntime.cleanup()


if __name__ == "__main__":
    main()

Observed

--runlist add_relu                      # build/, first run
  max |out - expected|     = 0.0000
  max |out - relu(a + b)|  = 0.0000
--runlist add_add                       # build/, second run, exit code 0
  max |out - expected|     = 5.9062
  max |out - relu(a + b)|  = 0.0000     <- the first runlist's ELF ran
--runlist add_add                       # build_fresh/
  max |out - expected|     = 0.0000
  max |out - relu(a + b)|  = 5.9062

After the second run, build/repro_fused.mlir and build/repro.elf still carry the first run's timestamps. Both operator MLIR files predate them, so plan() finds nothing to do.

Where else this shows up

The same mtime-only, name-keyed check also misses:

• Edits to modules a design imports. PythonGeneratedMLIRArtifact depends only on SourceArtifact(generator.source_path) (base.py:503). binary_elementwise_design.py and channeled_unary_design.py both import iron/operators/_trace.py. Running touch iron/operators/_trace.py and then re-running --runlist add_relu --build-dir build regenerates nothing: both operator .mlir files, repro_fused.mlir and repro.elf keep their old mtimes.
• An mlir-aie or llvm-aie upgrade. The new compiler never sees the old MLIR or ELF.
• DesignGenerator.args / kwargs. These are covered only to the extent that the operator's name happens to encode them. name is built from dataclass fields, so a parameter passed to the generator but not stored in a field is not covered.

Suggested fix

Key the fused artifacts on content rather than on name:

• Hash the runlist (design names and buffer names), subbuffer_layout, buffer_sizes, slice_info, trace_size, extra_flags, and the operator MLIR file contents into the filename. SeparateDispatch already hashes seq.name (sequence.py:209); this would extend that idea to the inputs.
• Alternatively, keep the filename and write the hash to a sidecar file. is_available_in_filesystem would compare against it, the way RemoteFileArtifact already checks a sha256 (base.py:535).
• Fold the mlir_aie / llvm-aie versions into the same hash.

A cheap interim guard is to have SequenceMLIRArtifact.is_available_in_filesystem compare the stored runlist and layout against the current ones and return False on mismatch.

Notes

This issue was brought to my attention vis the discord.