fix(scc/ccd): encode EIA-608 special/extended characters instead of raw internal bytes by x15sr71 · Pull Request #2301 · CCExtractor/ccextractor

x15sr71 · 2026-07-31T05:10:30Z

In raising this pull request, I confirm the following (please check boxes):

Reason for this PR:

This PR adds new functionality.
This PR fixes a bug that I have personally experienced or that a real user has reported and for which a sample exists.
This PR is porting code from C to Rust.

Sanity check:

I have read and understood the contributors guide.
I have checked that another pull request for this purpose does not exist.
If the PR adds new functionality, I've added it to the changelog. If it's just a bug fix, I have NOT added it to the changelog.
I am NOT adding new C code unless it's to fix an existing, reproducible bug.

Repro instructions:

Sample

From issue #2098 (posted by @yukichigai, a 26-second segment):
DresdenTest.ts — https://1drv.ms/v/c/8b5ee05414946e03/IQDLLzwLvNY3RJi8rYvDN7PyAVTV7dC2YotrGHTGfnRkFUM?e=4YsJPQ

Before this PR — the bug

The apostrophe is a single raw byte 0x99 (invalid UTF-8). On macOS/BSD grep (and GNU grep in a UTF-8 locale) a plain grep will silently drop the line because it contains that invalid byte — so view it with cat -v, LC_ALL=C grep, or hexdump:

# cat -v renders the invalid byte as "M-^Y" (= 0x99)
./ccextractor --out=ccd --stdout ./DresdenTest.ts 2>/dev/null | cat -v | grep -iE "mother|that"
# 00:00:08:27  {RCL}{1400}{TO3}- WHAT IS IT?{1500}{TO3}- THIS WAS YOUR MOTHERM-^YS.{EOC}{ENM}^M
# 00:00:23:27  {RCL}{1504}{TO2}{I}THATM-^YS A SHIELD BRACELET._{EOC}{ENM}^M

# hexdump the apostrophe cell -> byte 99
./ccextractor --out=ccd --stdout ./DresdenTest.ts 2>/dev/null | LC_ALL=C grep -a -i mother | hexdump -C
# ...4d 4f 54 48 45 52 99 53 2e...   ->   M O T H E R (99) S .

# SCC: the same 0x99 byte emitted verbatim
./ccextractor --out=scc --stdout ./DresdenTest.ts 2>/dev/null | grep -oiE "4552 [0-9a-f]{4} [0-9a-f]{4}"
# 4552 99d3 ae80          ->   ...(E R) 99 (S) (. pad)

Fixes #2098

Summary

Special/extended EIA-608 characters (apostrophes, quotes, music notes, accented letters, etc.) are corrupted in --out=scc and --out=ccd. The reported case is the possessive apostrophe in MOTHER'S / THAT'S:

--out=ccd writes the apostrophe as a single raw byte 0x99 (invalid UTF-8) — renders as � in a UTF-8 terminal, ™ under Windows-1252.
--out=scc emits that same 0x99 byte verbatim, which re-decodes as a stray control code and destroys the apostrophe and the following letter.

--out=ttxt, --out=srt, --out=g608 are all correct, which localizes the bug to the SCC/CCD writer.

Root cause

The SCC and CCD writers (src/lib_ccx/ccx_encoders_scc.c, both via write_cc_buffer_as_scenarist) re-encode the already-decoded eia608_screen grid. In that grid, special/extended characters are stored as CCExtractor's
internal codes (>= 0x80), not as valid single-byte characters. For the apostrophe, the 608 decoder collapses the on-air a7 80 + 92 29 into a single grid cell holding internal byte 0x99.

write_character() emitted that internal byte verbatim — one raw byte for SCC (odd_parity(0x99)), and the raw byte for CCD. The internal→Unicode table (get_char_in_utf_8) is already correct (0x99 → U+0027); the writer simply
bypassed it. The issue is in the writer, not the internal→Unicode mapping table.

SCC fix

Reverse-map the internal code back to its EIA-608 two-byte code and emit a padded fallback base character followed by the extended pair:

internal code	hi (odd channel)	lo
`0x80–0x8f`	`0x11`	`c − 0x50`
`0x90–0xaf`	`0x12`	`c − 0x70`
`0xb0–0xcf`	`0x13`	`c − 0x90`

(field 2 / CC3/CC4: hi += 8, matching the existing is_odd_channel convention.)

The extended code destructively backspace-replaces the preceding cell, so a padded base character must occupy that cell first, and the extended pair must stay 2-byte aligned (handled by check_padding). The base character is the single-byte ASCII form when the character has one (apostrophe → 0x27), otherwise a space.

Result for the apostrophe: a7 80 92 29.

CCD fix

Emit the real UTF-8 glyph via get_char_in_utf_8 instead of the raw byte. This fixes every special/extended character (the table is already correct for all of 0x80–0xcf), not just the apostrophe.

How FFmpeg does it

FFmpeg's raw 608 bytes for the same file (subcc + stream copy) encode the apostrophe exactly as this fix now does — base char + pad frame, then the extended frame:

$ ffmpeg -f lavfi -i "movie=DresdenTest.ts[out0+subcc]" -map 0:1 -c:s copy reference.scc
$ grep "cd4f 54c8 4552" reference.scc
00:00:08:11  ... cd4f 54c8 4552 a780 9229
             #      M  O   T  H   E  R   [a7=' , 80=pad]  [92 29 = extended ']

Our output matches the expected character sequence. FFmpeg additionally duplicates some control codes for transmission reliability, while CCExtractor emits each once; this is pre-existing behavior and unchanged by this PR.

Testing

After this PR — fixed

# CCD: plain apostrophe (byte 0x27)
./ccextractor --out=ccd --stdout ./DresdenTest.ts 2>/dev/null | grep -iE "mother|that"
# 00:00:08:27  {RCL}{1400}{TO3}- WHAT IS IT?_{1500}{TO3}- THIS WAS YOUR MOTHER'S._{EOC}{ENM}
# 00:00:23:27  {RCL}{1504}{TO2}{I}THAT'S A SHIELD BRACELET._{EOC}{ENM}

# SCC: correct extended sequence -> a7 80 (base+pad) then 92 29 (extended apostrophe)
./ccextractor --out=scc --stdout ./DresdenTest.ts 2>/dev/null | grep -oiE "4552 [0-9a-f]{4} [0-9a-f]{4}"
# 4552 a780 9229          ->   ...(E R) '(a7) pad(80) 92 29 ...

Round-trip (no character loss, no control-code misinterpretation)

./ccextractor --out=scc -o out.scc ./DresdenTest.ts
./ccextractor --input scc --out=ttxt --stdout out.scc | grep -iE "mother|shield"
# 00:00:08,901|00:00:12,066|POP|   - THIS WAS YOUR MOTHER'S.
  # 00:00:14,267|00:00:15,766|POP|       THAT'S A SHIELD BRACELET.

No regression to other formats

--out=srt, --out=ttxt, --out=g608, and --out=bin are byte-identical before and after this change (verified by rebuilding the parent commit and diffing each output). The change only touches the SCC/CCD writer path.

Scope & follow-up

Two files: Two files: src/lib_ccx/ccx_encoders_scc.c (the fix) and a one-line src/rust/src/demuxer/stream_functions.rs drive-by fix for a pre-existing clippy::byte_char_slices lint that blocks CI
Multiple special/extended characters across all three ranges (0x80–0xCF) verified to round-trip correctly (e.g. ♪ ' — " " Ã } ⌟ ¥ Ø).
Not in scope: calculate_caption_bytes still counts extended chars as 1 byte (now 3–4). This only affects --scc_accurate_timing (off by default) as a minor preroll undercount — left out to keep this fix minimal.

…al byte (CCExtractor#2098)

Pre-existing clippy::byte_char_slices lint on master fails 'cargo clippy -- -D warnings' for every PR. Identical to the fix in CCExtractor#2298 so the two branches merge without conflict regardless of order.

ccextractor-bot · 2026-07-31T05:59:44Z

CCExtractor CI platform finished running the test files on linux. Below is a summary of the test results, when compared to test for commit 2feb09a...:

Report Name	Tests Passed
Broken	9/13
CEA-708	2/14
DVB	1/7
DVD	3/3
DVR-MS	2/2
General	23/27
Hardsubx	1/1
Hauppage	3/3
MP4	3/3
NoCC	10/10
Options	72/86
Teletext	20/21
WTV	12/13
XDS	31/34

Your PR breaks these cases:

ccextractor --out=srt --latin1 611b4a9235...
ccextractor --autoprogram --out=ttxt --latin1 1020459a86...
ccextractor --autoprogram --out=srt --latin1 --quant 0 85271be4d2...
ccextractor --autoprogram --out=ttxt --latin1 99e5eaafdc...
ccextractor --autoprogram --out=ttxt --latin1 --ucla dab1c1bd65...
ccextractor --autoprogram --out=ttxt --latin1 01509e4d27...
ccextractor --out=srt --latin1 --autoprogram 29e5ffd34b...
ccextractor --out=spupng c83f765c66...
ccextractor --dru c83f765c66...
ccextractor --startat 4 --endat 7 c83f765c66...
ccextractor --codec dvbsub --out=spupng 85271be4d2...
ccextractor --startcreditstext "CCextractor Start crdit Testing" c4dd893cb9...
ccextractor --startcreditsnotbefore 1 --startcreditstext "CCextractor Start crdit Testing" c4dd893cb9...
ccextractor --startcreditsnotafter 2 --startcreditstext "CCextractor Start crdit Testing" c4dd893cb9...
ccextractor --startcreditsforatleast 1 --startcreditstext "CCextractor Start crdit Testing" c4dd893cb9...
ccextractor --startcreditsforatmost 2 --startcreditstext "CCextractor Start crdit Testing" c4dd893cb9...
ccextractor --out=srt --latin1 f23a544ba8...
ccextractor --autoprogram --out=srt --latin1 --ucla d037c7509e...
ccextractor --autoprogram --out=srt --latin1 --ucla 7d3f25c32c...
ccextractor --autoprogram --out=ttxt --latin1 --ucla --xds 7f41299cc7...

NOTE: The following tests have been failing on the master branch as well as the PR:

ccextractor --out=srt --latin1 --autoprogram 73d9313d64..., Last passed:
Test 8738
ccextractor --out=ttxt --latin1 001dd8cdf7..., Last passed:
Test 8738
ccextractor --out=srt --latin1 4d4e938ef6..., Last passed:
Test 8738
ccextractor --service 1 --out=txt --no-bom --no-rollup ea83ff7bcb..., Last passed:
Test 8738
ccextractor --service 1 --out=txt f17524b53f..., Last passed:
Test 8738
ccextractor --service 1 --out=txt 80848c45f8..., Last passed:
Test 8738
ccextractor --service 1 --out=txt --no-bom --no-rollup b5d6aad89f..., Last passed:
Test 8738
ccextractor --service 1[EUC-KR] --out=txt --no-rollup b5d6aad89f..., Last passed:
Test 8738
ccextractor --service 1 --out=srt da904de35d..., Last passed:
Test 8738
ccextractor --service 1 --out=sami da904de35d..., Last passed:
Test 8738
ccextractor --service 1[EUC-KR] b5d6aad89f..., Last passed:
Test 8738
ccextractor --service 1[EUC-KR] --no-rollup b5d6aad89f..., Last passed:
Test 8738
ccextractor --service all da904de35d..., Last passed:
Test 8738
ccextractor --service all[EUC-KR] b5d6aad89f..., Last passed:
Test 8738
ccextractor --service 1,2[UTF-8],3[EUC-KR],54 --out=txt da904de35d..., Last passed:
Test 8738
ccextractor --autoprogram --out=srt --latin1 f1422b8bfe..., Last passed:
Test 9208
ccextractor --autoprogram --out=srt --latin1 d41b53b504..., Last passed:
Test 8738
ccextractor --stdout --quiet --no-fontcolor 79a51f3500..., Last passed:
Test 8738
ccextractor --stdout --quiet --no-fontcolor 767b546f96..., Last passed:
Test 8738
ccextractor --service 1 c83f765c66..., Last passed:
Test 8738
ccextractor --myth c83f765c66..., Last passed:
Test 8738
ccextractor --in=raw fb79021542..., Last passed:
Test 8738
ccextractor --mp4vidtrack 5df914ce77..., Last passed:
Test 8738
ccextractor --xmltv=3 --out=null 96efd279cf..., Last passed:
Test 8738
ccextractor --datapid 2310 --autoprogram --out=srt --latin1 e639e54550..., Last passed:
Test 8738

Congratulations: Merging this PR would fix the following tests:

ccextractor --service 1 --out=ttxt da904de35d..., Last passed: Never
ccextractor --autoprogram --out=ttxt --latin1 132d7df7e9..., Last passed: Never
ccextractor --autoprogram --out=srt --latin1 b22260d065..., Last passed: Never

It seems that not all tests were passed completely. This is an indication that the output of some files is not as expected (but might be according to you).

Check the result page for more info.

ccextractor-bot · 2026-07-31T06:19:29Z

CCExtractor CI platform finished running the test files on windows. Below is a summary of the test results, when compared to test for commit 9f78685...:

Report Name	Tests Passed
Broken	10/13
CEA-708	2/14
DVB	2/7
DVD	3/3
DVR-MS	2/2
General	25/27
Hardsubx	1/1
Hauppage	3/3
MP4	3/3
NoCC	10/10
Options	79/86
Teletext	20/21
WTV	13/13
XDS	34/34

Your PR breaks these cases:

ccextractor --autoprogram --out=srt --latin1 --quant 0 85271be4d2...
ccextractor --autoprogram --out=ttxt --latin1 --ucla dab1c1bd65...
ccextractor --out=srt --latin1 --autoprogram 29e5ffd34b...
ccextractor --out=spupng c83f765c66...
ccextractor --codec dvbsub --out=spupng 85271be4d2...

NOTE: The following tests have been failing on the master branch as well as the PR:

ccextractor --out=srt --latin1 --autoprogram 73d9313d64..., Last passed:
Test 8611
ccextractor --out=ttxt --latin1 001dd8cdf7..., Last passed:
Test 8611
ccextractor --out=srt --latin1 4d4e938ef6..., Last passed:
Test 8611
ccextractor --service 1 --out=txt --no-bom --no-rollup ea83ff7bcb..., Last passed:
Test 8611
ccextractor --service 1 --out=txt f17524b53f..., Last passed:
Test 8611
ccextractor --service 1 --out=txt 80848c45f8..., Last passed:
Test 8611
ccextractor --service 1 --out=txt --no-bom --no-rollup b5d6aad89f..., Last passed:
Test 8611
ccextractor --service 1[EUC-KR] --out=txt --no-rollup b5d6aad89f..., Last passed:
Test 8611
ccextractor --service 1 --out=srt da904de35d..., Last passed:
Test 8611
ccextractor --service 1 --out=sami da904de35d..., Last passed:
Test 8611
ccextractor --service 1[EUC-KR] b5d6aad89f..., Last passed:
Test 8611
ccextractor --service 1[EUC-KR] --no-rollup b5d6aad89f..., Last passed:
Test 8611
ccextractor --service all da904de35d..., Last passed:
Test 8611
ccextractor --service all[EUC-KR] b5d6aad89f..., Last passed:
Test 8611
ccextractor --service 1,2[UTF-8],3[EUC-KR],54 --out=txt da904de35d..., Last passed:
Test 8611
ccextractor --autoprogram --out=srt --latin1 f1422b8bfe..., Last passed:
Test 9209
ccextractor --autoprogram --out=srt --latin1 d41b53b504..., Last passed:
Test 8611
ccextractor --stdout --quiet --no-fontcolor 79a51f3500..., Last passed:
Test 8611
ccextractor --stdout --quiet --no-fontcolor 767b546f96..., Last passed:
Test 8611
ccextractor --service 1 c83f765c66..., Last passed:
Test 8611
ccextractor --myth c83f765c66..., Last passed:
Test 8611
ccextractor --in=raw fb79021542..., Last passed:
Test 8611
ccextractor --mp4vidtrack 5df914ce77..., Last passed:
Test 8611
ccextractor --xmltv=3 --out=null 96efd279cf..., Last passed:
Test 8611
ccextractor --datapid 2310 --autoprogram --out=srt --latin1 e639e54550..., Last passed:
Test 8611

Congratulations: Merging this PR would fix the following tests:

ccextractor --out=srt --latin1 611b4a9235..., Last passed: Never
ccextractor --autoprogram --out=ttxt --latin1 1020459a86..., Last passed: Never
ccextractor --autoprogram --out=ttxt --latin1 132d7df7e9..., Last passed: Never
ccextractor --autoprogram --out=ttxt --latin1 99e5eaafdc..., Last passed: Never
ccextractor --autoprogram --out=ttxt --latin1 01509e4d27..., Last passed: Never
ccextractor --dru c83f765c66..., Last passed: Never
ccextractor --startat 4 --endat 7 c83f765c66..., Last passed: Never
ccextractor --startcreditstext "CCextractor Start crdit Testing" c4dd893cb9..., Last passed: Never
ccextractor --startcreditsnotbefore 1 --startcreditstext "CCextractor Start crdit Testing" c4dd893cb9..., Last passed: Never
ccextractor --startcreditsnotafter 2 --startcreditstext "CCextractor Start crdit Testing" c4dd893cb9..., Last passed: Never
ccextractor --startcreditsforatleast 1 --startcreditstext "CCextractor Start crdit Testing" c4dd893cb9..., Last passed: Never
ccextractor --startcreditsforatmost 2 --startcreditstext "CCextractor Start crdit Testing" c4dd893cb9..., Last passed: Never
ccextractor --out=srt --latin1 f23a544ba8..., Last passed: Never
ccextractor --autoprogram --out=srt --latin1 --ucla d037c7509e..., Last passed: Never
ccextractor --autoprogram --out=srt --latin1 --ucla 7d3f25c32c..., Last passed: Never
ccextractor --autoprogram --out=ttxt --latin1 --ucla --xds 7f41299cc7..., Last passed: Never

It seems that not all tests were passed completely. This is an indication that the output of some files is not as expected (but might be according to you).

Check the result page for more info.

x15sr71 added 2 commits July 31, 2026 10:32

fix(scc/ccd): encode 608 special/extended chars instead of raw intern…

c33cedf

…al byte (CCExtractor#2098)

style(rust): use byte-string literals to satisfy clippy byte_char_slices

96acc42

Pre-existing clippy::byte_char_slices lint on master fails 'cargo clippy -- -D warnings' for every PR. Identical to the fix in CCExtractor#2298 so the two branches merge without conflict regardless of order.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix(scc/ccd): encode EIA-608 special/extended characters instead of raw internal bytes - #2301

fix(scc/ccd): encode EIA-608 special/extended characters instead of raw internal bytes#2301
x15sr71 wants to merge 2 commits into
CCExtractor:masterfrom
x15sr71:fix/2098-apostrophe-scc-ccd

x15sr71 commented Jul 31, 2026 •

edited

Loading

Uh oh!

ccextractor-bot commented Jul 31, 2026

Uh oh!

ccextractor-bot commented Jul 31, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

x15sr71 commented Jul 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Sample

Before this PR — the bug

Summary

Root cause

SCC fix

CCD fix

How FFmpeg does it

Testing

After this PR — fixed

Round-trip (no character loss, no control-code misinterpretation)

No regression to other formats

Scope & follow-up

Uh oh!

ccextractor-bot commented Jul 31, 2026

Uh oh!

ccextractor-bot commented Jul 31, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

x15sr71 commented Jul 31, 2026 •

edited

Loading