Add JSON schema caching - #1637
Conversation
There was a problem hiding this comment.
Pull request overview
This PR adds an in-memory, process-wide JSON Schema cache keyed by resource identity so repeated schema validations don’t re-fetch schemas (e.g., during test/set/get flows), addressing #1513.
Changes:
- Introduces a new
schema_cachemodule with a staticRESOURCE_SCHEMASstore and a lookup helper. - Adds cache lookup/logging and cache population to schema retrieval paths for command resources / resource schema invocation.
- Adds Pester coverage that exercises cache-hit behavior and updates localization strings; bumps the bundled jsonschema patch version metadata.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 8 comments.
Show a summary per file
| File | Description |
|---|---|
| lib/dsc-lib/src/dscresources/dscresource.rs | Adds cache lookup in schema() and (currently) also attempts to populate the cache. |
| lib/dsc-lib/src/dscresources/command_resource.rs | Adds cache lookup and cache population inside get_schema(). |
| lib/dsc-lib/src/configure/schema_cache.rs | New module defining the global in-memory schema cache and lookup helper. |
| lib/dsc-lib/src/configure/mod.rs | Exposes the new schema_cache module. |
| lib/dsc-lib/locales/en-us.toml | Adds log strings for “retrieved schema from cache” messages. |
| lib/dsc-lib-jsonschema/.versions.json | Updates latest patch metadata and list to include V3_2_3. |
| dsc/tests/dsc_config_get.tests.ps1 | Adds tests that validate cache-hit logging for repeated resource usage. |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 8 out of 8 changed files in this pull request and generated 1 comment.
Comments suppressed due to low confidence (1)
dsc/tests/dsc_config_get.tests.ps1:205
- This test name states the embedded schema is retrieved "only once", but the assertion only checks that a cache-hit message appears at least once. That can still pass even if the schema is retrieved multiple times (as long as there is at least one cache hit). Consider also asserting that the cache-miss log ("Invoking schema for ...") appears exactly once for this resource type during the run.
$LASTEXITCODE | Should -Be 0 -Because $errorLog
$result.hadErrors | Should -BeFalse
$result.results.Count | Should -Be 2
$errorLog | Should -BeLike "*Retrieved schema for resource 'Microsoft.DSC.Debug/Echo' with version '1.0.0' from cache*"
}
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 8 out of 8 changed files in this pull request and generated 1 comment.
Comments suppressed due to low confidence (1)
dsc/tests/dsc_config_get.tests.ps1:204
- This test claims the embedded schema should be retrieved "only" once, but the current assertion only checks that a cache-hit message appears at least once. That would still pass even if the cache is hit multiple times (or if the message is logged more than once), so it doesn't actually verify the "only once" requirement.
$errorLog | Should -BeLike "*Retrieved schema for resource 'Microsoft.DSC.Debug/Echo' with version '1.0.0' from cache*"
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 8 out of 8 changed files in this pull request and generated 2 comments.
Comments suppressed due to low confidence (1)
dsc/tests/dsc_config_get.tests.ps1:205
- This test name says the embedded schema should be retrieved "only" once, but the current assertion only checks that the cache-hit message appears somewhere in the log. If the message appears multiple times (or the schema is retrieved more than once), this would still pass. Consider asserting the exact number of cache-hit log entries to match the test intent.
$errorLog = Get-Content $TestDrive/error.log -Raw
$LASTEXITCODE | Should -Be 0 -Because $errorLog
$result.hadErrors | Should -BeFalse
$result.results.Count | Should -Be 2
$errorLog | Should -BeLike "*Retrieved schema for resource 'Microsoft.DSC.Debug/Echo' with version '1.0.0' from cache*"
}
c2ad345 to
994cffb
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 8 out of 8 changed files in this pull request and generated 1 comment.
Comments suppressed due to low confidence (2)
dsc/tests/dsc_config_get.tests.ps1:186
- This
Itdescription claims the embedded schema is retrieved "only once", but the assertions only verify that a cache-hit message appears (at least once). Rename the test to match what it actually asserts (that the schema cache is used).
It 'embedded schema should only be retrieved once for multiple instances of the same resource' {
dsc/tests/dsc_config_get.tests.ps1:207
- This
Itdescription doesn’t match the assertions: the test checks for cache-hit log lines, not that the schema is retrieved "only once". Rename the test to describe the cache usage being asserted.
It 'retrieves command resource schema from cache for multiple instances of the same resource' {
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 8 out of 8 changed files in this pull request and generated 1 comment.
Comments suppressed due to low confidence (2)
dsc/tests/dsc_config_get.tests.ps1:187
- This test claims the schema is retrieved "only" once, but it only asserts a cache-hit message appears. It also calls this an "embedded" schema even though
Microsoft.DSC.Debug/Echoretrieves schema via a schema command. Tighten the assertions (e.g., exactly one "Invoking schema" for this resource) and/or adjust the name to match what is being validated.
It 'embedded schema should only be retrieved once for multiple instances of the same resource' {
$config_yaml = @"
dsc/tests/dsc_config_get.tests.ps1:233
- These assertions only check that the cache-hit messages appear somewhere in the debug log. To validate the intended behavior (cache is keyed by version and avoids repeated retrieval), assert the expected number of schema-cache misses as well (one miss per version => two total "Invoking schema" lines here).
$result.results.Count | Should -Be 3
$errorLog | Should -BeLike "*Retrieved schema for resource 'Test/Version' with version '1.1.0' from cache*"
$errorLog | Should -BeLike "*Retrieved schema for resource 'Test/Version' with version '2.0.0' from cache*"
Mikey Lombardi (He/Him) (michaeltlombardi)
left a comment
There was a problem hiding this comment.
Implementation seems sensible - all improvements and thoughts I have around schema caching can be deferred to a future version, no need to block this release. Local testing showed it works as expected.
* feat: implement export filter functionality for resource exports (#1621) * feat: implement export filter functionality for resource exports * Remove comment * feat: implement export filter functionality for resource exports * Remove comment * Fix Copilot remarks * Wrong commit * Remove the wildcard support for resources * Fix Copilot remarks * Attempt to increase code coverage and fix test * Add additional test for coverage * Add test and fix dism_dsc * fix: correct code coverage calculation for uninstrumented files Files without LCOV data (e.g., platform-specific code behind #[cfg(windows)] when only Linux coverage is collected) were incorrectly counting ALL added lines as uncovered, including non-executable lines like comments and blanks. This caused coverage to drop significantly when Windows-only files were modified in a PR. Fix: skip files without LCOV data since coverage cannot be determined for uninstrumented code. Also handle the '\ No newline at end of file' diff marker which could cause off-by-one line number errors after deleted lines. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Refactor work on services * Remove unused key * Revert change * Restore native resource filtering and add engine filtering fallback * Update resource definitions * Fix Copilot remark * remove directive * Change comment wording and update tests * Add crate * Revert change and fix test * Update resource manifest --------- Co-authored-by: Steve Lee <slee@microsoft.com> Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * (GH-1563) Enhance the PSScript resources (#1657) * (GH-1563) Canonicalize `_inDesiredState` for PSScript schema Prior to this change, the PSScript resource manifests embedded a schema that minimally defined the `_inDesiredState` canonical property instead of referencing the canonical schema for that property. This change updates the embedded schemas to reference the canonical schema and adds the embedded schema by ID to the `$defs` keyword. * (GH-1563) Canonicalize script properties for PSScript schemas Prior to this change, the embedded schemas in the manifests for the transitional PowerShell script resources defined the script properties with a minimal schema allowing the value to be any string or null. They didn't include any other keywords. This change: 1. Defines a shared subschema for the script properties, which they now reference instead of redefining the constraints for each property. 1. Removed `null` from the allowed types, since defining a script property as `null` isn't valid but _not_ specifying the property at all _is_ valid. 1. Added the `writeOnly` keyword to the script properties, since they're only used for input and not output. 1. Added the `contentMediaType` keyword to the script properties to clearly indicate---but not validate---that the value is expected to be a PowerShell script. 1. Added the `minLength` keyword to the script properties to ensure that an empty string isn't used as a script value. 1. Adds the `title` and `description` keywords to the script properties to provide more context for each property in the schema. * (GH-1563) Canonicalize `input` for PSScript resource schemas Prior to this change, the `input` property for the PSScript resource schemas defined the valid types with two problems: 1. It allowed `null` as a valid type, which caused failures when the resource invoked the scripts. 1. It didn't allow non-integer numbers, erroneously preventing users from defining input values like `3.14`, even though `[3.14]` was accepted. This change: 1. Removes `null` from the valid types for `input`. 1. Adds `number` to the valid types for `input`, allowing non-integer numbers to be used as input values. 1. Adds the `title` and `description` keywords to the `input` property, providing better documentation for users. * (GH-1563) Add constraints to PSScript resource schema Prior to this change, the embedded schemas for the PSScript resources didn't require any properties to be defined. For input, a user should always define at least one script property. For output, the resource should always return one of three value shapes: 1. For `test` operations, the resource should always define the `inDesiredState` property. 1. For implemented `get` and `set` operations, the resource should define the `output` property when the script emits any output to the success stream. 1. When `get` and `set` operations aren't implemented or don't emit any output, the resource should return an empty object. This change adds constraints to the embedded schemas with the `oneOf` keyword and nested `oneOf`/`anyOf` keywords to enforce these rules. * (GH-1563) Add docs keywords for PSScript resources Prior to this change, the embedded schema for the PSScript resources didn't define the `title` or `description` keywords. This change: 1. Adds `title` and `description` keywords to the embedded schema for the PSScript resources to provide some documentation for them. 1. Defines the `$schema` keyword explicitly. * (GH-1563) Canonicalize output for PSScript resources Prior to this change, the PSScript resources: 1. Always returned an array of output objects, even if there was only one object. 1. Didn't ensure that enums in script output were serialized as strings, causing enums to emit as integers, which is likely not what the user intended and makes review more difficult. This change: 1. Ensures that if a script returns a single object, it's serialized as that object instead of nested in an array. 1. Adds the `-EnumAsString` parameter to the `ConvertTo-Json` call for emitting script output when invoked through PowerShell to ensure that enums are serialized as strings. This enhancement doesn't affect Windows PowerShell, which doesn't support the `-EnumAsString` parameter. 1. Updates the tests to reflect the new behavior of returning a single object instead of an array. 1. Fixes the tests to use correct casing for the script properties, now that the schema validation checks for those properties. * (GH-1563) Fix tests using invalid casing for PSScript resource Prior to this change, some tests were using invalid casing for the PSScript resource type, which didn't previously cause errors during the JSON Schema validation because the schema didn't forbid additional properties. The tests passed because PowerShell isn't case-sensitive, so was able to retrieve the properties even with incorrect casing. With the changes to the JSON schema _requiring_ one or more script properties, these tests began to fail because the invalidly-cased property names didn't satisfy the JSON Schema. This change corrects the casing in the test definitions to ensure both that the tests are correct and pass the schema validation. * Apply suggestions from review Co-authored-by: Steve Lee <slee@microsoft.com> --------- Co-authored-by: Steve Lee <slee@microsoft.com> * Add JSON schema caching (#1637) * Add JSON schema caching * address copilot feedback * fix copilot feedback to not deserializing the json * rename function to be singular * make helpers scoped to crate * make cache crate only * fix: add support for preserving quoted group names in sshd_config (#1639) * fix: add support for preserving quoted group names in sshd_config * Update with remarks Tess --------- Co-authored-by: Gijs Reijn <26114636+Gijsreyn@users.noreply.github.com> Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Co-authored-by: Mikey Lombardi (He/Him) <michael.t.lombardi@gmail.com>
PR Summary
Add an in-memory static cache of retrieved JSON schema for resources so if the resource is used more than once it doesn't need to be retrieved from the resource itself.
PR Context
Fix #1513