[HesandaLiyanage]

Scope

Third incremental PR for JAMES-4231, following merged #3244 and #3245. Adds the typed BlobIdUpdater SPI and a Cassandra implementation for relocating message header, body, and attachment references.

Targets feature/s3-object-compaction. The complete implementation remains available in draft #3193.

Design

• Scan authoritative messageV3 and attachmentV2 references once, notify the liveness observer, and persist predicate-matching references in a paged Cassandra index scoped by run and blob ID.
• Apply conditional reference updates without overwriting changed references or recreating deleted rows. Retain the prepared index for retries after partial failure.
• Repair historical header references in both secondary tables when a header read reports a missing blob, using messageV3 as the source of truth.
• Expose explicit asynchronous index cleanup. No index TTL is used, because expiry during relocation could hide live references.
• Keep mailbox code independent of chunk layout and compression.
• Collocate Cassandra compaction classes, schema definitions, and dedicated tests in org.apache.james.mailbox.cassandra.object.compaction. Use RunId throughout the updater/index API.
• Recover missing blobs once for other fetch types, including FULL, preserving header/body content and propagating a persistent miss.

Validation

Production compilation passed with JDK 21:

mvn -o -q -pl mailbox/cassandra -am -Dmaven.test.skip=true compile

Added 41 tests after review, including coverage for typed run IDs and full-content recovery. All passed alongside 51 existing secondary-DAO regression tests after the package move: 92 tests, zero failures, errors, or skips. The configured Checkstyle check also passed.

• 15 real Cassandra updater tests: paging, predicate/observer behavior, conditional updates, partial-failure retries, run isolation, cleanup, null references, and permanent/expiring/near-expiry attachments.
• 7 factory lifecycle tests: preparation failures, cancellation, subscription isolation, cleanup retries, and snapshot ownership.
• 10 reconciliation tests with real Cassandra secondary tables: repair persistence, deletion/change safety, mailbox/UID reuse, missing canonical content, partial-failure retry, full-content recovery, and absent historical headers.
• 6 real mapper tests: missing-header recovery through both lookup paths, correct content/flags, persistent repair, no repeated canonical lookup, full-content recovery, and a bounded retry for persistent missing blobs.
• 3 RunId tests: UUID storage/log representation, equality, distinct generated identifiers, and null rejection.
• Existing CassandraMessageIdDAOTest (30 tests) and CassandraMessageIdToImapUidDAOTest (21 tests).

Successful test command, using JDK 21 and Docker/Testcontainers Cassandra:

mvn -o -q -pl mailbox/cassandra -am \
  '-Dtest=RunIdTest,CassandraBlobIdUpdaterTest,CassandraBlobIdUpdaterFactoryTest,CassandraMessageMetadataReconcilerTest,CassandraMessageMapperTest#headersShouldRepairStaleReferencesOnlyOnce+fullFetchShould*,CassandraMessageIdMapperTest#headersShouldRepairStaleReferencesOnlyOnce+fullFetchShould*,CassandraMessageIdDAOTest,CassandraMessageIdToImapUidDAOTest' \
  -Dsurefire.failIfNoSpecifiedTests=false test

Limits and follow-up

• The prepared updater assumes a completed generation with no new authoritative references into it; passes affecting that generation must be serialized.
• Cleanup currently scans index partition keys across retained runs and deletes only its own partitions. Highly shared blob IDs can still produce large Cassandra partitions.
• Very short remaining attachment TTL causes relocation to fail so the original physical object remains available for retry.
• Production schema registration, factory bindings, task lifecycle integration, and abandoned-run recovery are deferred to wiring work.
• PostgreSQL is next, followed by compaction core/property tests, wiring, WebAdmin/E2E, and final documentation.

[Arsnael]

No tests at all here for this?

[HesandaLiyanage]

No tests at all here for this?

You're right, I missed the tests here. My bad. I'll add them and push an update :)

[Arsnael]

You're right, I missed the tests here. My bad. I'll add them and push an update :)

No problem, it's alright. Thanks :)

[chibenwa]

Hello,

Thanks for this work.

I would like Cassandra class to be collocated in a dedicated package. How about org.apache.james.mailbox.cassandra.object.compaction ?

[HesandaLiyanage]

Hello,

Thanks for this work.

I would like Cassandra class to be collocated in a dedicated package. How about org.apache.james.mailbox.cassandra.object.compaction ?

Sounds good. I'll move the Cassandra compaction classes into org.apache.james.mailbox.cassandra.object.compaction and update the tests and imports accordingly.