Skip to content

Add NEON for av2_trellis_loop_diagonal_st8 - #5489

Merged
urvangjoshi merged 3 commits into
AOMediaCodec:av2-encfrom
jjustiss-apple:jjustiss/neon-trellis-loop-diagonal
Oct 8, 2026
Merged

urvangjoshi merged 3 commits into
AOMediaCodec:av2-encfrom
jjustiss-apple:jjustiss/neon-trellis-loop-diagonal

Conversation

@jjustiss-apple

Copy link
Copy Markdown
Contributor

Full ARMv7-A NEON port of the TCQ trellis diagonal loop. Fuses all
sub-kernels (pre_quant, rate_dist, decide_states, update_states,
update_nbr_diagonal) into a single inlined loop, vectorizing quantization,
RD-cost computation, context extraction, and state update with NEON
intrinsics. The decide_states path remains scalar (no vcgtq_s64 on ARMv7)
but uses ccmp chains for efficient 3-way branching.

Also consolidates duplicated helpers (kGolombExp0Bits, get_dqv,
get_diag_ctx) from NEON/AVX2/C into trellis_quant.h.

CTC results (RA, cpu-used=1, 33 frames, A4+A5):

Metric Delta
Encode time -15.1%
BD-rate Y 0.000%
BD-rate Cb 0.000%
BD-rate Cr 0.000%

Micro-benchmarks (Apple Silicon M2 Ultra, total_us/coeffs, 100M iterations):

TX Size Coeffs C (us) NEON (us) Speedup
4x4 16 62255 38075 1.64x
8x8 64 176644 73406 2.41x
16x16 256 157300 54788 2.87x
32x32 1024 124584 41517 3.00x
4x8 32 94864 60213 1.58x
8x16 128 79436 29899 2.66x
16x32 512 61110 20567 2.97x
32x16 512 62941 20796 3.03x
8x32 256 155631 55228 2.82x
32x8 256 162384 55384 2.93x

Unit tests included (RandomValues + Speed).

@jjustiss-apple

Copy link
Copy Markdown
Contributor Author

@jianj-g I addressed all your comments, except the aarch64/armv8a instruction question which I'd like to get clarification on from @yunqingwang1 before incorporating that change. It seems that using annotated disassembly can catch some of these (like the a0_0 a0_1 issue) where the compiler found the reduction but I missed the simplification in source code.

Incorporating a new RTCD class for aarch64/armv8a would likely be overbearing as 99% of the code would be shared with armv7a, but would we be open to using pragmas to split the armv7a and aarch64 implementations?

ARM builds seem to already lose the dynamic run-time RTCD features, so perhaps using pragmas isn't unreasonable. What do you think?

@jjustiss-apple

Copy link
Copy Markdown
Contributor Author

@jianj-g from an email, it looks like @yunqingwang1 would prefer following the AV1-style of RTCD with multiple ARM/neon specializations. Since this one is armv7a compliant, how would you feel about tabling the armv8a optimizations until I can explore the multiple NEON specializations a bit more?

@jjustiss-apple
jjustiss-apple requested a review from jianj-g October 7, 2026 21:58
@jianj-g

jianj-g commented Oct 8, 2026

Copy link
Copy Markdown
Contributor

@jianj-g from an email, it looks like @yunqingwang1 would prefer following the AV1-style of RTCD with multiple ARM/neon specializations. Since this one is armv7a compliant, how would you feel about tabling the armv8a optimizations until I can explore the multiple NEON specializations a bit more?

Yeah that's fine. Currently it looks like armv7a vs aarch64 is a compile time flag:

#if AVM_ARCH_AARCH64

So it doesn't need additional RTCD. But aarch64 extentions would need that like (dotprod, i8mm, sve).

Thanks for looking into this!

@urvangjoshi
urvangjoshi force-pushed the jjustiss/neon-trellis-loop-diagonal branch from 6a1eab5 to d30bcc9 Compare October 8, 2026 16:05
@urvangjoshi
urvangjoshi merged commit 97b64d0 into AOMediaCodec:av2-enc Oct 8, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants