Skip to content

Add JSON schema caching - #1637

Merged
Mikey Lombardi (He/Him) (michaeltlombardi) merged 6 commits into
PowerShell:mainfrom
SteveL-MSFT:cache-schema
Jul 29, 2026
Merged

Add JSON schema caching#1637
Mikey Lombardi (He/Him) (michaeltlombardi) merged 6 commits into
PowerShell:mainfrom
SteveL-MSFT:cache-schema

Conversation

@SteveL-MSFT

Copy link
Copy Markdown
Member

PR Summary

Add an in-memory static cache of retrieved JSON schema for resources so if the resource is used more than once it doesn't need to be retrieved from the resource itself.

PR Context

Fix #1513

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds an in-memory, process-wide JSON Schema cache keyed by resource identity so repeated schema validations don’t re-fetch schemas (e.g., during test/set/get flows), addressing #1513.

Changes:

  • Introduces a new schema_cache module with a static RESOURCE_SCHEMAS store and a lookup helper.
  • Adds cache lookup/logging and cache population to schema retrieval paths for command resources / resource schema invocation.
  • Adds Pester coverage that exercises cache-hit behavior and updates localization strings; bumps the bundled jsonschema patch version metadata.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 8 comments.

Show a summary per file
File Description
lib/dsc-lib/src/dscresources/dscresource.rs Adds cache lookup in schema() and (currently) also attempts to populate the cache.
lib/dsc-lib/src/dscresources/command_resource.rs Adds cache lookup and cache population inside get_schema().
lib/dsc-lib/src/configure/schema_cache.rs New module defining the global in-memory schema cache and lookup helper.
lib/dsc-lib/src/configure/mod.rs Exposes the new schema_cache module.
lib/dsc-lib/locales/en-us.toml Adds log strings for “retrieved schema from cache” messages.
lib/dsc-lib-jsonschema/.versions.json Updates latest patch metadata and list to include V3_2_3.
dsc/tests/dsc_config_get.tests.ps1 Adds tests that validate cache-hit logging for repeated resource usage.

Comment thread lib/dsc-lib/src/dscresources/dscresource.rs Outdated
Comment thread lib/dsc-lib/src/dscresources/dscresource.rs
Comment thread lib/dsc-lib/src/dscresources/dscresource.rs Outdated
Comment thread lib/dsc-lib/src/dscresources/command_resource.rs
Comment thread lib/dsc-lib/src/dscresources/command_resource.rs Outdated
Comment thread lib/dsc-lib/src/dscresources/command_resource.rs Outdated
Comment thread dsc/tests/dsc_config_get.tests.ps1
Comment thread dsc/tests/dsc_config_get.tests.ps1 Outdated
@SteveL-MSFT
Steve Lee (SteveL-MSFT) marked this pull request as draft July 20, 2026 16:24
Copilot AI review requested due to automatic review settings July 21, 2026 20:08

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (1)

dsc/tests/dsc_config_get.tests.ps1:205

  • This test name states the embedded schema is retrieved "only once", but the assertion only checks that a cache-hit message appears at least once. That can still pass even if the schema is retrieved multiple times (as long as there is at least one cache hit). Consider also asserting that the cache-miss log ("Invoking schema for ...") appears exactly once for this resource type during the run.
        $LASTEXITCODE | Should -Be 0 -Because $errorLog
        $result.hadErrors | Should -BeFalse
        $result.results.Count | Should -Be 2
        $errorLog | Should -BeLike "*Retrieved schema for resource 'Microsoft.DSC.Debug/Echo' with version '1.0.0' from cache*"
    }

Comment thread lib/dsc-lib/src/dscresources/command_resource.rs Outdated
Copilot AI review requested due to automatic review settings July 21, 2026 20:31
@SteveL-MSFT
Steve Lee (SteveL-MSFT) marked this pull request as ready for review July 21, 2026 20:32

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (1)

dsc/tests/dsc_config_get.tests.ps1:204

  • This test claims the embedded schema should be retrieved "only" once, but the current assertion only checks that a cache-hit message appears at least once. That would still pass even if the cache is hit multiple times (or if the message is logged more than once), so it doesn't actually verify the "only once" requirement.
        $errorLog | Should -BeLike "*Retrieved schema for resource 'Microsoft.DSC.Debug/Echo' with version '1.0.0' from cache*"

Comment thread lib/dsc-lib/src/configure/schema_cache.rs Outdated
Copilot AI review requested due to automatic review settings July 22, 2026 01:44

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (1)

dsc/tests/dsc_config_get.tests.ps1:205

  • This test name says the embedded schema should be retrieved "only" once, but the current assertion only checks that the cache-hit message appears somewhere in the log. If the message appears multiple times (or the schema is retrieved more than once), this would still pass. Consider asserting the exact number of cache-hit log entries to match the test intent.
        $errorLog = Get-Content $TestDrive/error.log -Raw
        $LASTEXITCODE | Should -Be 0 -Because $errorLog
        $result.hadErrors | Should -BeFalse
        $result.results.Count | Should -Be 2
        $errorLog | Should -BeLike "*Retrieved schema for resource 'Microsoft.DSC.Debug/Echo' with version '1.0.0' from cache*"
    }

Comment thread lib/dsc-lib/src/configure/schema_cache.rs Outdated
Comment thread lib/dsc-lib/src/dscresources/command_resource.rs
Copilot AI review requested due to automatic review settings July 22, 2026 02:36

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (2)

dsc/tests/dsc_config_get.tests.ps1:186

  • This It description claims the embedded schema is retrieved "only once", but the assertions only verify that a cache-hit message appears (at least once). Rename the test to match what it actually asserts (that the schema cache is used).
    It 'embedded schema should only be retrieved once for multiple instances of the same resource' {

dsc/tests/dsc_config_get.tests.ps1:207

  • This It description doesn’t match the assertions: the test checks for cache-hit log lines, not that the schema is retrieved "only once". Rename the test to describe the cache usage being asserted.
    It 'retrieves command resource schema from cache for multiple instances of the same resource' {

Comment thread lib/dsc-lib/src/configure/mod.rs Outdated
Copilot AI review requested due to automatic review settings July 22, 2026 04:16

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (2)

dsc/tests/dsc_config_get.tests.ps1:187

  • This test claims the schema is retrieved "only" once, but it only asserts a cache-hit message appears. It also calls this an "embedded" schema even though Microsoft.DSC.Debug/Echo retrieves schema via a schema command. Tighten the assertions (e.g., exactly one "Invoking schema" for this resource) and/or adjust the name to match what is being validated.
    It 'embedded schema should only be retrieved once for multiple instances of the same resource' {
        $config_yaml = @"

dsc/tests/dsc_config_get.tests.ps1:233

  • These assertions only check that the cache-hit messages appear somewhere in the debug log. To validate the intended behavior (cache is keyed by version and avoids repeated retrieval), assert the expected number of schema-cache misses as well (one miss per version => two total "Invoking schema" lines here).
        $result.results.Count | Should -Be 3
        $errorLog | Should -BeLike "*Retrieved schema for resource 'Test/Version' with version '1.1.0' from cache*"
        $errorLog | Should -BeLike "*Retrieved schema for resource 'Test/Version' with version '2.0.0' from cache*"

Comment thread lib/dsc-lib/src/dscresources/command_resource.rs

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Implementation seems sensible - all improvements and thoughts I have around schema caching can be deferred to a future version, no need to block this release. Local testing showed it works as expected.

Merged via the queue into PowerShell:main with commit 23330fd Jul 29, 2026
20 checks passed
Steve Lee (SteveL-MSFT) added a commit that referenced this pull request Aug 10, 2026
* feat: implement export filter functionality for resource exports (#1621)

* feat: implement export filter functionality for resource exports

* Remove comment

* feat: implement export filter functionality for resource exports

* Remove comment

* Fix Copilot remarks

* Wrong commit

* Remove the wildcard support for resources

* Fix Copilot remarks

* Attempt to increase code coverage and fix test

* Add additional test for coverage

* Add test and fix dism_dsc

* fix: correct code coverage calculation for uninstrumented files

Files without LCOV data (e.g., platform-specific code behind #[cfg(windows)]
when only Linux coverage is collected) were incorrectly counting ALL added
lines as uncovered, including non-executable lines like comments and blanks.
This caused coverage to drop significantly when Windows-only files were
modified in a PR.

Fix: skip files without LCOV data since coverage cannot be determined for
uninstrumented code. Also handle the '\ No newline at end of file' diff
marker which could cause off-by-one line number errors after deleted lines.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* Refactor work on services

* Remove unused key

* Revert change

* Restore native resource filtering and add engine filtering fallback

* Update resource definitions

* Fix Copilot remark

* remove directive

* Change comment wording and update tests

* Add crate

* Revert change and fix test

* Update resource manifest

---------

Co-authored-by: Steve Lee <slee@microsoft.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* (GH-1563) Enhance the PSScript resources (#1657)

* (GH-1563) Canonicalize `_inDesiredState` for PSScript schema

Prior to this change, the PSScript resource manifests embedded a schema
that minimally defined the `_inDesiredState` canonical property instead
of referencing the canonical schema for that property.

This change updates the embedded schemas to reference the canonical
schema and adds the embedded schema by ID to the `$defs` keyword.

* (GH-1563) Canonicalize script properties for PSScript schemas

Prior to this change, the embedded schemas in the manifests for the
transitional PowerShell script resources defined the script properties
with a minimal schema allowing the value to be any string or null. They
didn't include any other keywords.

This change:

1. Defines a shared subschema for the script properties, which they now
   reference instead of redefining the constraints for each property.
1. Removed `null` from the allowed types, since defining a script
    property as `null` isn't valid but _not_ specifying the property at
    all _is_ valid.
1. Added the `writeOnly` keyword to the script properties, since
   they're only used for input and not output.
1. Added the `contentMediaType` keyword to the script properties to
   clearly indicate---but not validate---that the value is expected to
   be a PowerShell script.
1. Added the `minLength` keyword to the script properties to ensure that
   an empty string isn't used as a script value.
1. Adds the `title` and `description` keywords to the script properties
   to provide more context for each property in the schema.

* (GH-1563) Canonicalize `input` for PSScript resource schemas

Prior to this change, the `input` property for the PSScript resource
schemas defined the valid types with two problems:

1. It allowed `null` as a valid type, which caused failures when the
   resource invoked the scripts.
1. It didn't allow non-integer numbers, erroneously preventing users
   from defining input values like `3.14`, even though `[3.14]` was
   accepted.

This change:

1. Removes `null` from the valid types for `input`.
1. Adds `number` to the valid types for `input`, allowing non-integer
   numbers to be used as input values.
1. Adds the `title` and `description` keywords to the `input` property,
   providing better documentation for users.

* (GH-1563) Add constraints to PSScript resource schema

Prior to this change, the embedded schemas for the PSScript resources
didn't require any properties to be defined. For input, a user should
always define at least one script property. For output, the resource
should always return one of three value shapes:

1. For `test` operations, the resource should always define the
  `inDesiredState` property.
1. For implemented `get` and `set` operations, the resource should
   define the `output` property when the script emits any output to the
   success stream.
1. When `get` and `set` operations aren't implemented or don't emit any
   output, the resource should return an empty object.

This change adds constraints to the embedded schemas with the `oneOf`
keyword and nested `oneOf`/`anyOf` keywords to enforce these rules.

* (GH-1563) Add docs keywords for PSScript resources

Prior to this change, the embedded schema for the PSScript resources
didn't define the `title` or `description` keywords.

This change:

1. Adds `title` and `description` keywords to the embedded schema for
   the PSScript resources to provide some documentation for them.
1. Defines the `$schema` keyword explicitly.

* (GH-1563) Canonicalize output for PSScript resources

Prior to this change, the PSScript resources:

1. Always returned an array of output objects, even if there was only
   one object.
1. Didn't ensure that enums in script output were serialized as
   strings, causing enums to emit as integers, which is likely not what
   the user intended and makes review more difficult.

This change:

1. Ensures that if a script returns a single object, it's serialized as
   that object instead of nested in an array.
1. Adds the `-EnumAsString` parameter to the `ConvertTo-Json` call for
   emitting script output when invoked through PowerShell to ensure
   that enums are serialized as strings. This enhancement doesn't
   affect Windows PowerShell, which doesn't support the `-EnumAsString`
   parameter.
1. Updates the tests to reflect the new behavior of returning a single
   object instead of an array.
1. Fixes the tests to use correct casing for the script properties,
   now that the schema validation checks for those properties.

* (GH-1563) Fix tests using invalid casing for PSScript resource

Prior to this change, some tests were using invalid casing for the
PSScript resource type, which didn't previously cause errors during
the JSON Schema validation because the schema didn't forbid additional
properties. The tests passed because PowerShell isn't case-sensitive,
so was able to retrieve the properties even with incorrect casing.

With the changes to the JSON schema _requiring_ one or more script
properties, these tests began to fail because the invalidly-cased
property names didn't satisfy the JSON Schema.

This change corrects the casing in the test definitions to ensure both
that the tests are correct and pass the schema validation.

* Apply suggestions from review

Co-authored-by: Steve Lee <slee@microsoft.com>

---------

Co-authored-by: Steve Lee <slee@microsoft.com>

* Add JSON schema caching (#1637)

* Add JSON schema caching

* address copilot feedback

* fix copilot feedback to not deserializing the json

* rename function to be singular

* make helpers scoped to crate

* make cache crate only

* fix: add support for preserving quoted group names in sshd_config (#1639)

* fix: add support for preserving quoted group names in sshd_config

* Update with remarks Tess

---------

Co-authored-by: Gijs Reijn <26114636+Gijsreyn@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Mikey Lombardi (He/Him) <michael.t.lombardi@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Engine needs to cache JSON schema

3 participants