[Performance] A2A3 fresh AICore stream per run regresses repeated HCA/SWA Effective latency by 3.8–4.4% #1791

Background

In pypto-lib, repeated same-image DeepSeek-V4 Flash HCA/SWA execution on
a2a3 with the tensormap_and_ringbuffer runtime has a performance
regression at Simpler PR #1540: median effective_us increases by 3.81–4.36%.

Repro cases:

• models/deepseek/v4-flash/decode_attention_hca.py
• models/deepseek/v4-flash/decode_attention_swa.py

Both entrypoints and their helpers are public at pypto-lib
1f48761cb5be21a8cf0611fad10d513ba40b0863 on main. The test also applies
the public pypto-lib commit c0751b7582618f7ccef29385236b1964cb480d42,
which only migrates 17 valid_shapes keyword arguments to valid_shape.

Reproduction environment:

Controlled A/B note: PyPTO 2e59c01c records Simpler dccb8379. To isolate
PR #1540, the runtime checkout was intentionally overridden with the PR parent
and PR commit above. PTOAS and PTO ISA remained fixed. A separate pin-consistent
PyPTO runtime-bundle A/B independently showed the same regression direction and
magnitude.

Diagnosis: simpler — the direct Simpler parent/commit A/B reproduces the
regression while pypto-lib, PyPTO, PTOAS, PTO ISA, model inputs, and benchmark
settings remain fixed.

Platform

a2a3 (Ascend 910B/C hardware)

Runtime Variant

tensormap_and_ringbuffer

Summary

Simpler PR #1540 changes the A2/A3 runner from retaining an AICore stream for
repeated runs of the same image to creating and retiring a fresh AICore stream
for every run. A direct parent/commit A/B on DeepSeek-V4 Flash HCA and SWA shows
median effective_us regressions of 4.36% and 3.81%, respectively.

Each endpoint used 5 warmups followed by 100 measured iterations. All four
runs passed kv_cache and x_out validation.

Git Commit ID

e51c3aeecfaf982137b6adc67ca13749b0c95717

CANN Version

9.0.0

Driver Version

26.0.rc1 (Ascend HAL 7.35.23)

Host Platform

Linux (aarch64) — Huawei Cloud EulerOS 2.0, kernel
5.10.0-182.0.0.95.r1941_123.hce2.aarch64

Reproduction

Use isolated checkouts and a separate virtual environment for each Simpler
endpoint.

git -C <pypto-lib-checkout> checkout \
  1f48761cb5be21a8cf0611fad10d513ba40b0863

git -C <pypto-lib-checkout> show --format= \
  c0751b7582618f7ccef29385236b1964cb480d42 \
  -- models/deepseek/v4-flash/hc_pre.py | \
  git -C <pypto-lib-checkout> apply -

git -C <pypto-checkout> checkout \
  2e59c01ccc34e586e19b49ecaa1a94b2e21db459
git -C <pypto-checkout> submodule update --init --recursive

git -C <pypto-checkout>/runtime checkout <runtime-revision>
git -C <pto-isa-checkout> checkout \
  83d01313d9bfc247c4b7c8bcf969d1019f0d106f

source <CANN-root>/set_env.sh
export PTO_ISA_ROOT=<pto-isa-checkout>

python -m pip install --no-build-isolation -e <pypto-checkout>
python -m pip install --no-build-isolation -e <pypto-checkout>/runtime

task-submit --ptoas 0.54 --device 4 --timeout 0 --max-time 1800 --run '
cd <pypto-lib-checkout>
export PYPTO_BENCH=1
export PYPTO_BENCH_WARMUP=5
export PYPTO_BENCH_ROUNDS=100
export PYPTO_BENCH_RAW=1
export PTO2_RING_TASK_WINDOW=262144
export PTO2_RING_DEP_POOL=262144
export PTO2_RING_HEAP=2147483648
export PYPTO_RUNTIME_LOG=error

python models/deepseek/v4-flash/decode_attention_hca.py \
  -p a2a3 -d "$TASK_DEVICE" \
  --start-pos 8192 --enable-l2-swimlane 0

python models/deepseek/v4-flash/decode_attention_swa.py \
  -p a2a3 -d "$TASK_DEVICE" \
  --start-pos 8192 --enable-l2-swimlane 0
'

Run once with baseline runtime
8e00319e28cd2ea9f7f7f795a2b1e182956b91bd, then repeat with
e51c3aeecfaf982137b6adc67ca13749b0c95717.

Expected Performance

With parent commit 8e00319e28cd2ea9f7f7f795a2b1e182956b91bd:

Repeated same-image execution should retain this steady-state latency after
warmup while preserving correctness when different images alternate.

Actual Performance

With PR #1540 commit e51c3aeecfaf982137b6adc67ca13749b0c95717:

Test window: 2026-08-11 19:05:39–19:10:05 UTC+8. One 100-round batch was
run per endpoint/case. A separate PyPTO runtime-bundle boundary A/B produced
HCA +12.5 us and SWA +8.7 us, independently corroborating the result.

Profiling Data (Optional)

Before PR #1540, ensure_run_stream_set(slot, aicore_image_hash) returns
immediately when the retained AICore stream carries the same image hash. The
5 warmups and 100 measured rounds therefore share one AICore stream.

After PR #1540:

• RunStreamSlots::acquire() always creates a fresh AICore stream.
• The stream is retired after every completed run.
• The 5-warmup/100-round benchmark therefore uses 105 fresh AICore streams.

The reported effective_us metric is the on-device merged orch/sched window.
Host-side rtStreamCreate occurs before that device window, and
rtStreamDestroy occurs after device completion and timing readback. The
9–11 us regression is therefore not direct host stream API latency.

Generated AICore binaries are unchanged across the direct A/B:

The strongest current explanation is a device-side cold-start effect: using a
fresh stream for every same-image run prevents retention of prior
instruction-fetch/I-cache/stream-affinity state. The commit boundary and stream
lifecycle change are proven; PMU icache_req/icache_miss data or a
stream-only toggle is still needed to confirm the exact hardware mechanism.

Additional Context

The fresh-stream policy was introduced for correctness when different images
are published to the same GM code address. A plain restoration of the old
per-slot image-hash cache may reintroduce stale-instruction risk for an
A -> B -> A sequence across depth-two slots.

A safer remediation direction is runner-wide, generation-aware same-image
reuse: reuse only for consecutive identical images when no overlapping code
range has been republished, and recreate the stream when image identity or code
address generation changes. Stable non-overlapping code banks or a supported
AICore instruction-cache invalidation operation are alternatives.

Acceptance should cover repeated same-image performance, alternating-image
correctness, depth-two slot alternation, and stream-creation counts.

Source inspection of Simpler main at
c5a587400721ea98f0a00321569088a5321539b9 shows that
RunStreamSlots::acquire() still unconditionally creates a fresh AICore stream.
The numerical A/B above was performed at the PR parent/commit, not at that
latest-main commit.

Related: #1717 discusses hiding per-run device launch latency and persistent
execution for host_build_graph; this report concerns repeated same-image
device Effective regression in tensormap_and_ringbuffer.