feat(platform)!: skipIfAbsent may skip on a derived index property (PV14)

book/src/contract-keywords/derived-index-properties.md

@@ -70,7 +70,7 @@ The name reads through a reference when its first segment names a top-level iden
 
 ## What Drive does
 
-- **Create.** The reference is validated as usual, which reads the referenced document, and Drive writes the entries under the values it holds, without reading it again. A referenced document that has no value for the field, or a reference left out, puts the document under null, as a missing property would.
+- **Create.** The reference is validated as usual, which reads the referenced document, and Drive writes the entries under the values it holds, without reading it again. A referenced document that has no value for the field, or a reference left out, puts the document under null, as a missing property would, or leaves it out of an index that [skips on the property](#skipping-a-document-without-the-value).
 - **Replace, transfer, purchase, price update and a moderator's change of fields.** Drive reads the referenced document once, for both versions of the document, since neither can point elsewhere. It reads it even when no index holding a derived value moves: every entry's reference is rewritten in place on an update, under the values it is filed by.
 - **Delete, a moderator's removal, and expiry by `ttl`.** Drive reads the referenced document to find the entries to remove.
 - **A removed referenced document.** Once a moderator removes a `moderatedDocument` target, Drive reads its owner, and any field its type keeps under [`deleteKeepsFields`](deletion.md#moderatorabilitiesdeletekeepsfields), from the removal record, which keeps them as the document held them. So the entries are found under the values they were written under. A restored document is read again.
@@ -88,9 +88,27 @@ A derived value must stay what it was when an entry was written, or Drive could
 - A schema property must exist on the referenced type, be stored (not `transient` or inside a transient object), be fixed once written there under the same rule as the reference property, and be one an index can key: not an object or an array, a string of at most 63 characters, a byte array of at most 255 bytes, tighter on a [ranked index](ranked.md).
 - Not `$id` of the referenced document, which is the reference property itself: index that instead. Nor any other system property.
 - The index is not `unique` or contested: a uniqueness check reads the values the create carries, and a derived value is not among them.
-- The derived property is not the source of a `timeRange` or an `integerRange`, nor a skip property of `skipIfAbsent`.
+- The derived property is not the source of a `timeRange` or an `integerRange`.
 - The type is not `indexOnly`: its entries hold every value of its documents, and its delete carries them.
 
+## Skipping a document without the value
+
+A derived property can be a skip property of [`skipIfAbsent`](indexes.md#skipifabsent). The index then leaves out a document whose reference is absent, or whose referenced document has no value for the field, instead of filing it under null. A reply that may quote a post, filed by the owner of the post it quotes:
+
+```json
+{
+  "name": "byQuotedOwner",
+  "properties": [{ "quoteId.$ownerId": "asc" }, { "$createdAt": "asc" }],
+  "skipIfAbsent": ["quoteId.$ownerId"]
+}
+```
+
+A reply quoting no post writes nothing into this index, and its delete looks for nothing there. Since neither the reference nor the field can change once written, a replace never moves a document into or out of such an index through its derived values.
+
+Only the array form names a derived property. `skipIfAbsent: true` skips on the type's own optional properties: whether a derived value can be absent depends on the referenced type.
+
+At registration, a derived skip property must be able to be absent: its reference property is not in `required`, or the field is a schema property the referenced type does not require, or one inside an object it does not require. `$ownerId` and `$creatorId` are absent only with the reference. A byte array the property reads must set `minItems` to at least 1 on the referenced type, since an empty one is keyed like a missing value. Every other [`skipIfAbsent` rule](indexes.md#skipifabsent) applies as to any skip property.
+
 ## Queries
 
 A query uses a derived property as any other index property: `==`, `in`, a range, `orderBy`. Its values encode as the field's type.

book/src/contract-keywords/indexes.md

@@ -139,7 +139,7 @@ DPNS sets `"nullSearchable": false` on its `records.identity` index, so domains
 | **Since** | protocol version 14 |
 | **On update** | Fixed (10217) |
 
-A document that leaves out a property of the index's **skip set** writes nothing into this index, and its delete looks for nothing there. `true` makes the skip set every optional property of the index; an array names it. The skip property can sit anywhere in the index, under a [`timeRange`](time-range.md) or [`integerRange`](integer-range.md) window too. The index then holds only the documents carrying every skip property, and its counts and rankings are "among the documents that have them". A present but empty value is not absent and is indexed.
+A document that leaves out a property of the index's **skip set** writes nothing into this index, and its delete looks for nothing there. `true` makes the skip set every optional property of the index; an array names it. A [value of a referenced document](derived-index-properties.md#skipping-a-document-without-the-value) is never in the `true` set, since whether it can be absent depends on the referenced type; the array may name one. The skip property can sit anywhere in the index, under a [`timeRange`](time-range.md) or [`integerRange`](integer-range.md) window too. The index then holds only the documents carrying every skip property, and its counts and rankings are "among the documents that have them". A present but empty value is not absent and is indexed.
 
 ```json
 "post": {
@@ -177,10 +177,10 @@ A query only uses a skip index when it constrains every skip property, so it can
 Rules at registration:
 
 - The skip set is not empty: an index whose properties are all required could never skip.
-- Each skip property is a top-level property of the type, not a system property, and not listed in `required`.
+- Each skip property is a top-level property of the type, not a system property, and not listed in `required`, or a [value of a referenced document](derived-index-properties.md#skipping-a-document-without-the-value) that can be absent.
 - No `rankedCountable` `at` level sits above the index's deepest skip property: it would count only documents carrying that property, and no query could read it without binding the property.
 - The index is not contested, and does not also set `nullSearchable: false` (the skip already leaves out a document with every indexed value missing).
-- On a stored type, a skip property that is a byte array sets `minItems` to at least 1: an empty byte array is keyed like a missing value.
+- On a stored type, a skip property that is a byte array sets `minItems` to at least 1 (on the referenced type, for a value read from a referenced document): an empty byte array is keyed like a missing value.
 - On a stored type, a ranking at a skip property's level does not share that level with an index that keeps null for the property: that index would create the null value, and the ranking would show it as a group with no documents.
 - On an [index-only type](index-only.md#skipifabsent) the skip set holds every optional property of the index, and each optional property needs a skip index of its own, without a `timeRange` (a window keeps the value only until the window drains).
 

packages/rs-dpp/schema/meta_schemas/document/v3/document-meta.json

@@ -2100,7 +2100,7 @@
                 "uniqueItems": true
               }
             ],
-            "description": "Conditional participation: a document that omits a property of this index's skip set writes no entry into this index (and a delete, or on a stored type a replace, recomputes the same skip), so the index holds only documents carrying every skip property. `true` makes the skip set every optional property of the index; an array names the skip set (each must be one of the index's properties; an index has at most 10, hence maxItems). A skip property may sit at any position of the index, and must be a top-level schema property NOT listed in `required` (system properties are never optional). On an indexOnly type the skip set must hold every optional property of the index (an index path has no representation for an absent value), every optional property needs a skip index whose skip set is that property alone, and at least one $createdAt-free index must remain non-skipIfAbsent (the executed-transition proof index); on a stored type the array may name a subset, the index's other optional properties keeping the null layout, and a skip index cannot be contested or set nullSearchable: false. No rankedCountable `at` level may sit above the deepest skip property. An absent property is distinct from an empty value: absence skips the index, while any present value — empty included — indexes normally. Available from protocol version 14."
+            "description": "Conditional participation: a document that omits a property of this index's skip set writes no entry into this index (and a delete, or on a stored type a replace, recomputes the same skip), so the index holds only documents carrying every skip property. `true` makes the skip set every optional property of the index; an array names the skip set (each must be one of the index's properties; an index has at most 10, hence maxItems). A skip property may sit at any position of the index, and must be a top-level schema property NOT listed in `required` (system properties are never optional), or, on a stored type and in the array form only, a derived index property (`<reference property>.<field>`) that can be absent: its reference property is optional, or the field is a schema property the referenced type does not require (nor every object around it); a document holding no value for it, null included, is skipped. On an indexOnly type the skip set must hold every optional property of the index (an index path has no representation for an absent value), every optional property needs a skip index whose skip set is that property alone, and at least one $createdAt-free index must remain non-skipIfAbsent (the executed-transition proof index); on a stored type the array may name a subset, the index's other optional properties keeping the null layout, and a skip index cannot be contested or set nullSearchable: false. No rankedCountable `at` level may sit above the deepest skip property. An absent property is distinct from an empty value: absence skips the index, while any present value — empty included — indexes normally. Available from protocol version 14."
           },
           "summableOffCountIndex": {
             "type": "string",

packages/rs-dpp/src/data_contract/document_type/class_methods/try_from_schema/common/mod.rs

@@ -23,12 +23,8 @@ use crate::data_contract::config::v0::DataContractConfigGettersV0;
 use crate::data_contract::config::v2::DataContractConfigGettersV2;
 use crate::data_contract::config::DataContractConfig;
 use crate::data_contract::document_type::class_methods::consensus_or_protocol_value_error;
-#[cfg(feature = "validation")]
-use crate::data_contract::document_type::index::{
-    parse_derived_index_property_name, DerivedIndexPropertyName,
-};
 use crate::data_contract::document_type::index::{
-    Index, IndexGrammarAdmissions, IntegerRangeKeyType,
+    reads_through_reference, Index, IndexGrammarAdmissions, IntegerRangeKeyType,
 };
 use crate::data_contract::document_type::index_level::IndexLevel;
 use crate::data_contract::document_type::property::DocumentProperty;
@@ -985,13 +981,20 @@ fn parse_indices(
                     // parse (validating or not), before the index structure is
                     // built from it. System properties are never in a skip set;
                     // `apply_index_only` refuses the index when none remains.
+                    // Nor is a value read through a reference: whether it can
+                    // be absent depends on the referenced type, which this
+                    // parse does not see, so only the array form skips on one.
                     if index.skip_if_absent && index.skip_if_absent_properties.is_empty() {
                         index.skip_if_absent_properties = index
                             .properties
                             .iter()
                             .filter(|property| {
                                 !property.name.starts_with('$')
                                     && !required_fields.contains(&property.name)
+                                    && !reads_through_reference(
+                                        &property.name,
+                                        flattened_document_properties,
+                                    )
                             })
                             .map(|property| property.name.clone())
                             .collect();
@@ -1447,14 +1450,7 @@ fn validate_index_properties(
         // the index and the referenced field are judged once the whole type, and then the
         // whole contract, is parsed
         if ctx.generation.admit_derived_index_properties
-            && !matches!(
-                parse_derived_index_property_name(
-                    &index_property.name,
-                    flattened_document_properties,
-                    ctx.data_contract_id,
-                ),
-                DerivedIndexPropertyName::NotDerived
-            )
+            && reads_through_reference(&index_property.name, flattened_document_properties)
         {
             return Ok(());
         }
@@ -3677,9 +3673,13 @@ pub(super) fn apply_retracted_when(
 /// - The skip set is non-empty: an index none of whose properties is optional
 ///   could never skip.
 /// - Every skip property is a top-level (non-dotted) schema property that is
-///   not a system property and not listed in `required`. Its presence is then
-///   a single lookup, the same for every walker, and it can actually be
-///   absent.
+///   not a system property and not listed in `required`, or a derived index
+///   property, which reads through a reference of the type. Its presence is
+///   then a single lookup, the same for every walker: Drive puts a derived
+///   value into the document's properties under the derived name before it
+///   keys the document. Whether a schema property can actually be absent is
+///   judged here; a derived one is judged where the referenced type is in
+///   hand (`resolve_derived_index_properties`).
 /// - No ranking level sits above the index's deepest skip property. Such a
 ///   level would count only the documents carrying the deeper property, but a
 ///   query may only read a skip index when it binds every skip property, so
@@ -3689,13 +3689,15 @@ fn skip_if_absent_index_error(
     index: &Index,
     document_type_name: &str,
     required_fields: &BTreeSet<String>,
+    flattened_properties: &IndexMap<String, DocumentProperty>,
 ) -> Option<String> {
     if index.skip_if_absent_properties.is_empty() {
         return Some(format!(
             "index \"{}\" on document type \"{}\" declares `skipIfAbsent`, but none of its \
              properties is optional: a required or system property can never be absent, so \
              the index could never skip — remove the flag, or remove a property from \
-             `required`",
+             `required` (`true` leaves out a value read through a reference: name one in \
+             the array form to skip on it)",
             index_name, document_type_name,
         ));
     }
@@ -3708,7 +3710,9 @@ fn skip_if_absent_index_error(
                 index_name, document_type_name, skip_property,
             ));
         }
-        if skip_property.contains('.') {
+        if skip_property.contains('.')
+            && !reads_through_reference(skip_property, flattened_properties)
+        {
             return Some(format!(
                 "index \"{}\" on document type \"{}\" skips on nested property \"{}\": a skip \
                  property must be a top-level property, so that presence is a single lookup \
@@ -3880,9 +3884,13 @@ pub(super) fn apply_index_only(
             if !index.skip_if_absent {
                 continue;
             }
-            if let Some(message) =
-                skip_if_absent_index_error(index_name, index, name, &document_type.required_fields)
-            {
+            if let Some(message) = skip_if_absent_index_error(
+                index_name,
+                index,
+                name,
+                &document_type.required_fields,
+                &document_type.flattened_properties,
+            ) {
                 return Err(structure_error(message));
             }
             // A contest is decided over the document's full value tuple; a
@@ -4198,9 +4206,13 @@ pub(super) fn apply_index_only(
         // index is in its skip set, and each has a skip index of its own) is
         // checked with coverage below.
         if index.skip_if_absent {
-            if let Some(message) =
-                skip_if_absent_index_error(index_name, index, name, &document_type.required_fields)
-            {
+            if let Some(message) = skip_if_absent_index_error(
+                index_name,
+                index,
+                name,
+                &document_type.required_fields,
+                &document_type.flattened_properties,
+            ) {
                 return Err(structure_error(message));
             }
         }