Repository navigation
#4137 fix(hash-index): keep HASH index keys deterministic with data encryption - #4138
Conversation
…ion is on DefaultDataEncryption.encrypt() uses a fresh random IV per call, so the same plaintext yields different ciphertext on every invocation. HashIndexBucket serialized index keys through BinarySerializer.serializeValue, which auto- encrypts when dataEncryption != null. Insert and lookup of the same key produced different bytes and therefore different murmurHash64 outputs, so the HASH index never found rows when encryption was enabled. WHERE <> was the only predicate that worked, since it falls back to a full scan. Add an applyEncryption flag to BinarySerializer.serializeValue/deserializeValue (default true preserves existing behavior) and pass false for HASH index key serialization so the on-disk and lookup byte representations are byte-identical across calls. Index keys are reconstructible from the encrypted document, so not encrypting them does not weaken data-at-rest protection. Closes #4137
Up to standards ✅🟢 Issues
|
| Metric | Results |
|---|---|
| Coverage variation | ✅ -7.73% coverage variation |
| Diff coverage | ✅ 88.24% diff coverage |
Coverage variation details
Coverable lines Covered lines Coverage Common ancestor commit (2df76f8) 125093 92092 73.62% Head commit (bbe6451) 156498 (+31405) 103110 (+11018) 65.89% (-7.73%) Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch:
<coverage of head commit> - <coverage of common ancestor commit>
Diff coverage details
Coverable lines Covered lines Diff coverage Pull request (#4138) 17 15 88.24% Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified:
<covered lines added or modified>/<coverable lines added or modified> * 100%
NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.
There was a problem hiding this comment.
Code Review
This pull request introduces an applyEncryption flag to the BinarySerializer to allow disabling encryption for specific use cases, such as maintaining determinism in hash index keys. While this correctly addresses the issue for simple types, the review feedback highlights that the flag is not currently propagated to recursive calls for complex types like Lists or Maps during serialization and deserialization. This would lead to index corruption or lookup failures for complex keys. Additionally, it is recommended to expand the test suite to cover these complex data types.
| public void serializeValue(final Database database, final Binary serialized, final byte type, Object value, | ||
| final boolean applyEncryption) { |
There was a problem hiding this comment.
The applyEncryption flag is not propagated to recursive calls within serializeValue. For example, in the TYPE_LIST, TYPE_MAP, and TYPE_EMBEDDED cases, the nested calls to serializeValue (or serializeProperties) will use the default value of true. This results in nested components being encrypted non-deterministically even when the top-level call requested no encryption (e.g., from HashIndexBucket), which breaks the hash index for complex key types.
| public Object deserializeValue(final Database database, final Binary deserialized, final byte type, | ||
| final EmbeddedModifier embeddedModifier, final boolean applyEncryption) { |
There was a problem hiding this comment.
Similarly to the serialization path, deserializeValue does not propagate the applyEncryption flag to its recursive calls. When reading back a complex key that was stored without encryption (such as a List or Map in a HASH index), the nested calls will attempt to decrypt the data, leading to an EncryptionException or data corruption.
|
|
||
| import static org.assertj.core.api.Assertions.assertThat; | ||
|
|
||
| class BrokenHashIndexTest extends TestHelper { |
There was a problem hiding this comment.
Code ReviewOverviewThis PR fixes a real bug: AES-GCM generates a fresh random IV on every What works well
Issues and suggestions1. LSM tree has the same determinism bug AND an additional symmetry hazard (acknowledged but worth scoping)
However there are two narrower breakages in the LSM path:
Both of these are arguably worse than the HASH bug. Given that 2. Migration gap for existing encrypted HASH index dataDatabases written before this fix have index keys stored as 3. Test class name
4. Missing composite-key and string-key test coverageThe test exercises a single
5. Minor:
|
… test Per Gemini review: BinarySerializer.serializeValue/deserializeValue now thread the applyEncryption flag through recursive calls for TYPE_LIST, TYPE_MAP and the primitive-array path (and serializeListEntries). Without this, an outer applyEncryption=false would still encrypt nested elements, which would defect deterministic key serialization for any future composite/array index keys. Per claude-review: add a string-key HASH index lookup test to exercise a non-numeric key type alongside the existing Long-key cases.
|
Pushed Addressed
Declined / scoped out
CI: 264 tests pass across |
|
Thank you for the updates and the detailed breakdown of the changes. Threading the |
Code Review - PR #4138OverviewThis is a well-targeted fix for a tricky encryption bug. The root cause is correctly identified: AES-GCM uses a fresh random IV on every CorrectnessThe fix is complete for the HASH path. One minor note: the new Code Quality
Test CoverageThe new
Scope / Follow-upThe scope note about LSM-tree indexes is appreciated and the decision to keep this PR tight to the HASH path is correct. The linked issue #4137 should be updated (or a new issue created) to explicitly track the LSM fix so it does not get lost. SecurityNot encrypting HASH index keys is an acceptable trade-off: the indexed keys (e.g., IDs) are generally derivable from the encrypted documents themselves, so an attacker with raw file access gains little beyond what they could reconstruct from the data pages. The fix description explains this reasoning clearly. Summary
Good catch on this subtle AES-GCM behavior. |
…ionTest DataEncryptionTest already exercises encryption end-to-end and creates a unique LSM index without ever querying it - the gap that allowed #4137 to remain undetected. Moving the regression tests there makes that file the canonical home for "indexes must work with encryption" coverage and keeps related fixtures in one place. Consolidated the four cases into two methods (long key with eq/neq/count and string key with eq), referencing #4137 in a comment block above the new methods.
Code ReviewOverviewThis PR correctly identifies and fixes a real encryption-vs-determinism bug. AES-GCM generates a fresh random IV on every What works wellThe core fix is correct and minimal. No read-side change was needed in HashIndexBucket. The hash index compares raw murmur-hashed byte arrays, not deserialized values, so the hash bucket lookup never calls LIST and MAP types correctly propagate Clean backward-compatible API. The overloaded method with The new tests are solid. They follow the project's AssertJ idiom, are clearly structured, and confirm both the bug (pre-fix) and the fix (post-fix) for Issues and suggestions1. Test class naming
2. Migration gap for existing encrypted HASH index dataDatabases written before this fix stored index keys as 3. LSM-tree follow-up is more severe than acknowledgedThe PR description mentions the LSM follow-up is captured in #4137. However the LSM breakage has two sub-problems that are arguably worse than the HASH bug:
Both of these are data-integrity failures that go beyond a lookup returning 0 rows. It may be worth reflecting this in the #4137 issue priority. 4. Minor:
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #4138 +/- ##
=======================================
Coverage 64.59% 64.60%
=======================================
Files 1637 1637
Lines 125093 125097 +4
Branches 26724 26725 +1
=======================================
+ Hits 80807 80814 +7
+ Misses 33031 33028 -3
Partials 11255 11255 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
…ith data encryption (ArcadeData#4138)
… 2.67.0 to 2.68.0 [skip ci] Bumps [com.google.api.grpc:proto-google-common-protos](https://github.com/googleapis/sdk-platform-java) from 2.67.0 to 2.68.0. Release notes *Sourced from [com.google.api.grpc:proto-google-common-protos's releases](https://github.com/googleapis/sdk-platform-java/releases).* > v2.68.0 > ------- > > [2.68.0](googleapis/sdk-platform-java@v2.67.0...v2.68.0) (2026-03-17) > ------------------------------------------------------------------------------------------------ > > ### Features > > * Add client request duration metric. ([ArcadeData#4132](https://redirect.github.com/googleapis/sdk-platform-java/issues/4132)) ([6a76397](googleapis/sdk-platform-java@6a76397)) > * Add more attributes to golden signals metrics. ([ArcadeData#4135](https://redirect.github.com/googleapis/sdk-platform-java/issues/4135)) ([59d0624](googleapis/sdk-platform-java@59d0624)) > * **gax-httpjson:** add HttpJsonErrorParser utility ([ArcadeData#4137](https://redirect.github.com/googleapis/sdk-platform-java/issues/4137)) ([a1b7565](googleapis/sdk-platform-java@a1b7565)) > * **generator:** add extra allowed modules that will not be removed from the monorepo if they are present ([ArcadeData#4124](https://redirect.github.com/googleapis/sdk-platform-java/issues/4124)) ([774fe6e](googleapis/sdk-platform-java@774fe6e)) > * **o11y:** introduce `gcp.client.repo` and `gcp.client.artifact` attributes ([ArcadeData#4120](https://redirect.github.com/googleapis/sdk-platform-java/issues/4120)) ([105f644](googleapis/sdk-platform-java@105f644)) > * **o11y:** Introduce `rpc.system.name` and `rpc.method` in gRPC ([ArcadeData#4121](https://redirect.github.com/googleapis/sdk-platform-java/issues/4121)) ([7ab6d2e](googleapis/sdk-platform-java@7ab6d2e)) > * **o11y:** introduce server.port attribute ([ArcadeData#4128](https://redirect.github.com/googleapis/sdk-platform-java/issues/4128)) ([56aa343](googleapis/sdk-platform-java@56aa343)) > > ### Bug Fixes > > * add null checks for ApiTracerFactory in ClientContext ([ArcadeData#4122](https://redirect.github.com/googleapis/sdk-platform-java/issues/4122)) ([4b3dbe2](googleapis/sdk-platform-java@4b3dbe2)) > * Decrease log level for directpath warnings outside GCE ([ArcadeData#4139](https://redirect.github.com/googleapis/sdk-platform-java/issues/4139)) ([c9651e7](googleapis/sdk-platform-java@c9651e7)) > * **gax-grpc:** add pick\_first fallback to direct path service config ([ArcadeData#4143](https://redirect.github.com/googleapis/sdk-platform-java/issues/4143)) ([b150fe9](googleapis/sdk-platform-java@b150fe9)) > * Populate method level attributes in metrics recording ([ArcadeData#4149](https://redirect.github.com/googleapis/sdk-platform-java/issues/4149)) ([7b7e6c9](googleapis/sdk-platform-java@7b7e6c9)) > * suppress warnings in generated projects for non-idiomatic durations ([ArcadeData#4119](https://redirect.github.com/googleapis/sdk-platform-java/issues/4119)) ([4206e6e](googleapis/sdk-platform-java@4206e6e)) > * Use ServiceName + MethodName as the regex for Otel ([ArcadeData#2543](https://redirect.github.com/googleapis/sdk-platform-java/issues/2543)) ([b9ae73f](googleapis/sdk-platform-java@b9ae73f)) > > ### Documentation > > * **hermetic\_build:** fix config field name in readme ([ArcadeData#4130](https://redirect.github.com/googleapis/sdk-platform-java/issues/4130)) ([a0c8f67](googleapis/sdk-platform-java@a0c8f67)) Changelog *Sourced from [com.google.api.grpc:proto-google-common-protos's changelog](https://github.com/googleapis/sdk-platform-java/blob/main/CHANGELOG.md).* > [2.68.0](googleapis/sdk-platform-java@v2.67.0...v2.68.0) (2026-03-17) > ------------------------------------------------------------------------------------------------ > > ### Features > > * Add client request duration metric. ([ArcadeData#4132](https://redirect.github.com/googleapis/sdk-platform-java/issues/4132)) ([6a76397](googleapis/sdk-platform-java@6a76397)) > * Add more attributes to golden signals metrics. ([ArcadeData#4135](https://redirect.github.com/googleapis/sdk-platform-java/issues/4135)) ([59d0624](googleapis/sdk-platform-java@59d0624)) > * **gax-httpjson:** add HttpJsonErrorParser utility ([ArcadeData#4137](https://redirect.github.com/googleapis/sdk-platform-java/issues/4137)) ([a1b7565](googleapis/sdk-platform-java@a1b7565)) > * **generator:** add extra allowed modules that will not be removed from the monorepo if they are present ([ArcadeData#4124](https://redirect.github.com/googleapis/sdk-platform-java/issues/4124)) ([774fe6e](googleapis/sdk-platform-java@774fe6e)) > * **o11y:** introduce `gcp.client.repo` and `gcp.client.artifact` attributes ([ArcadeData#4120](https://redirect.github.com/googleapis/sdk-platform-java/issues/4120)) ([105f644](googleapis/sdk-platform-java@105f644)) > * **o11y:** Introduce `rpc.system.name` and `rpc.method` in gRPC ([ArcadeData#4121](https://redirect.github.com/googleapis/sdk-platform-java/issues/4121)) ([7ab6d2e](googleapis/sdk-platform-java@7ab6d2e)) > * **o11y:** introduce server.port attribute ([ArcadeData#4128](https://redirect.github.com/googleapis/sdk-platform-java/issues/4128)) ([56aa343](googleapis/sdk-platform-java@56aa343)) > > ### Bug Fixes > > * add null checks for ApiTracerFactory in ClientContext ([ArcadeData#4122](https://redirect.github.com/googleapis/sdk-platform-java/issues/4122)) ([4b3dbe2](googleapis/sdk-platform-java@4b3dbe2)) > * Decrease log level for directpath warnings outside GCE ([ArcadeData#4139](https://redirect.github.com/googleapis/sdk-platform-java/issues/4139)) ([c9651e7](googleapis/sdk-platform-java@c9651e7)) > * **gax-grpc:** add pick\_first fallback to direct path service config ([ArcadeData#4143](https://redirect.github.com/googleapis/sdk-platform-java/issues/4143)) ([b150fe9](googleapis/sdk-platform-java@b150fe9)) > * Populate method level attributes in metrics recording ([ArcadeData#4149](https://redirect.github.com/googleapis/sdk-platform-java/issues/4149)) ([7b7e6c9](googleapis/sdk-platform-java@7b7e6c9)) > * suppress warnings in generated projects for non-idiomatic durations ([ArcadeData#4119](https://redirect.github.com/googleapis/sdk-platform-java/issues/4119)) ([4206e6e](googleapis/sdk-platform-java@4206e6e)) > * Use ServiceName + MethodName as the regex for Otel ([ArcadeData#2543](https://redirect.github.com/googleapis/sdk-platform-java/issues/2543)) ([b9ae73f](googleapis/sdk-platform-java@b9ae73f)) > > ### Documentation > > * **hermetic\_build:** fix config field name in readme ([ArcadeData#4130](https://redirect.github.com/googleapis/sdk-platform-java/issues/4130)) ([a0c8f67](googleapis/sdk-platform-java@a0c8f67)) Commits * [`7eca195`](googleapis/sdk-platform-java@7eca195) chore(main): release 2.68.0 ([ArcadeData#4123](https://redirect.github.com/googleapis/sdk-platform-java/issues/4123)) * [`7b7e6c9`](googleapis/sdk-platform-java@7b7e6c9) fix: Populate method level attributes in metrics recording ([ArcadeData#4149](https://redirect.github.com/googleapis/sdk-platform-java/issues/4149)) * [`c9651e7`](googleapis/sdk-platform-java@c9651e7) fix: Decrease log level for directpath warnings outside GCE ([ArcadeData#4139](https://redirect.github.com/googleapis/sdk-platform-java/issues/4139)) * [`55df486`](googleapis/sdk-platform-java@55df486) build(deps): bump black from 24.8.0 to 26.3.1 in /hermetic\_build/common ([ArcadeData#4138](https://redirect.github.com/googleapis/sdk-platform-java/issues/4138)) * [`15c8abf`](googleapis/sdk-platform-java@15c8abf) chore(deps): update upper bound dependencies file ([ArcadeData#4146](https://redirect.github.com/googleapis/sdk-platform-java/issues/4146)) * [`8be3f9d`](googleapis/sdk-platform-java@8be3f9d) chore: update googleapis commit at Thu Feb 19 03:02:27 UTC 2026 ([ArcadeData#4115](https://redirect.github.com/googleapis/sdk-platform-java/issues/4115)) * [`b150fe9`](googleapis/sdk-platform-java@b150fe9) fix(gax-grpc): add pick\_first fallback to direct path service config ([ArcadeData#4143](https://redirect.github.com/googleapis/sdk-platform-java/issues/4143)) * [`4a24d92`](googleapis/sdk-platform-java@4a24d92) build: disable SonarCloud for fork pull requests ([ArcadeData#4142](https://redirect.github.com/googleapis/sdk-platform-java/issues/4142)) * [`59d0624`](googleapis/sdk-platform-java@59d0624) feat: Add more attributes to golden signals metrics. ([ArcadeData#4135](https://redirect.github.com/googleapis/sdk-platform-java/issues/4135)) * [`0275698`](googleapis/sdk-platform-java@0275698) chore: extract channel builder for internal use ([ArcadeData#4136](https://redirect.github.com/googleapis/sdk-platform-java/issues/4136)) * Additional commits viewable in [compare view](googleapis/sdk-platform-java@v2.67.0...v2.68.0) [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- Dependabot commands and options You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
…ith data encryption (ArcadeData#4138)
Summary
DefaultDataEncryption.encrypt()uses a fresh random IV per call (AES-GCM), so encrypting the same plaintext twice yields different ciphertext.HashIndexBucket.serializeKeysrouted index keys throughBinarySerializer.serializeValue, which auto-encrypts whendataEncryption != null. Insert and lookup of the same key produced different bytes and differentmurmurHash64outputs, so entries were unreachable.WHERE vid <> 0only worked because it falls back to a full scan.applyEncryptionflag (defaulttrue) toBinarySerializer.serializeValue/deserializeValue.HashIndexBucket.serializeKeysnow passesfalseso the on-disk and lookup byte representations are byte-identical across calls. Index keys are reconstructible from the encrypted document, so not encrypting them does not weaken data-at-rest protection.Scope note
LSM-tree indexes use the same serializer path for keys (
LSMTreeIndexAbstract.writeKeysand the read-sidegetKeyInPagePosition/compareKey), so they almost certainly have the same defect with encryption enabled. The existingDataEncryptionTestcreates an LSM unique index but never queries by it, so the suite doesn't currently exercise that combination. This PR keeps scope tight to the failing HASH path; the LSM follow-up is captured in #4137.Test plan
BrokenHashIndexTest(3 tests) reproduces the bug pre-fix and passes post-fixHashIndexTest,HashIndexInheritanceTest,HashIndexUniqueOnExistingDuplicatesTest,Issue3714HashIndexCorruptionTest,LSMTreeIndexCompactionTest,LSMTreeIndexCompositeTest,DataEncryptionTest,DefaultDataEncryptionTest: all greencom.arcadedb.index.*,com.arcadedb.serializer.*,com.arcadedb.database.*all pass