Skip to content

[BUG] Feature flags and key changes not refreshed on triggered app config update in azure-spring-cloud-starter-appconfiguration-config 2.1.0 #24049

Closed
Closed
[BUG] Feature flags and key changes not refreshed on triggered app config update in azure-spring-cloud-starter-appconfiguration-config 2.1.0#24049
Assignees
mrm9084
Labels
ClientThis issue points to a problem in the data-plane of the library.azure-springAll azure-spring related issuesazure-spring-app-configurationSpring app configuration related issues.customer-reportedIssues that are reported by GitHub users external to the Azure organization.questionThe issue doesn't require a change to the product in order to be resolved. Most issues start as that
Milestone
[2021] October

Description

@PbALpi7xEX
PbALpi7xEX
opened on Sep 10, 2021

Describe the bug
App configuration not refreshing with trigger key and refresh interval.

To Reproduce
Steps to reproduce the behavior:

Code Snippet
Implemented the following config in my bootstrap.properties:

spring.cloud.azure.appconfiguration.stores[0].monitoring.enabled=true
spring.cloud.azure.appconfiguration.stores[0].monitoring.refresh-interval=5s
spring.cloud.azure.appconfiguration.stores[0].monitoring.triggers[0].key=app.config.refreshTrigger
spring.cloud.azure.appconfiguration.stores[0].feature-flags.enabled=true
spring.cloud.azure.appconfiguration.stores[0].monitoring.feature-flags.watch-interval=5s
spring.cloud.azure.appconfiguration.stores[0].selects[0].label-filter=,default,${spring.profiles.active}
spring.cloud.azure.appconfiguration.stores[0].feature-flags.label-filter=,default,${spring.profiles.active}
spring.cloud.azure.appconfiguration.stores[0].fail-fast=true

Feature Flag check in code:

if (Boolean.TRUE.equals(featureManager.isEnabledAsync(FeatureFlags.READ_ONLY_MODE.name()).block())) {
			throw new ApplicationInReadOnlyModeException("Application in read-only mode!");
}

Expected behavior
Toggle a feature, but it is not picked up in code. App restart required for changes to apply.

Setup (please complete the following information):

  • OS: Windows 10
  • IDE: Eclipse
  • Library/Libraries: org.springframework.boot:spring-boot-starter-parent:2.5.2, com.azure.spring:azure-spring-cloud-starter-appconfiguration-config:2.1.0, com.azure.spring:azure-spring-boot-bom:3.7.0
  • Java version: 11
  • App Server/Environment: Embedded spring boot
  • Frameworks: Spring Boot

Activity

  1. ghost added
    needs-triageWorkflow: This is a new issue that needs to be triaged to the appropriate team.
    on Sep 10, 2021
  2. ghost added
    customer-reportedIssues that are reported by GitHub users external to the Azure organization.
    on Sep 10, 2021
  3. ghost added
    questionThe issue doesn't require a change to the product in order to be resolved. Most issues start as that
    on Sep 10, 2021
  4. Josh Free (joshfree)
    added
    azure-springAll azure-spring related issues
    azure-spring-app-configurationSpring app configuration related issues.
    ClientThis issue points to a problem in the data-plane of the library.
    on Sep 10, 2021
  5. ghost removed
    needs-triageWorkflow: This is a new issue that needs to be triaged to the appropriate team.
    on Sep 10, 2021

  6. joshfree commented on Sep 10, 2021

    @joshfree
    Josh Free (joshfree)
    on Sep 10, 2021
    Member

    Xiaolu Dai (@saragluna) could you follow up with PbALpi7xEX

  7. PbALpi7xEX commented on Sep 10, 2021

    @PbALpi7xEX
    PbALpi7xEX
    on Sep 10, 2021
    Author

    Documentation typo as well to note...

    In https://github.com/Azure/azure-sdk-for-java/tree/main/sdk/appconfiguration/azure-spring-cloud-starter-appconfiguration-config#supported-properties

    spring.cloud.azure.appconfiguration.stores[0].feature-flags.enable

    Should read

    spring.cloud.azure.appconfiguration.stores[0].feature-flags.enabled

    Missing a "d" on enable... my feature flags weren't loading at first until I noticed this and decided to try ;-)

  8. Xiaolu Dai (saragluna)
    assigned
    Matthew Metcalf (mrm9084)
    on Sep 13, 2021
  9. Xiaolu Dai (saragluna)
    added this to the [2021] October milestone on Sep 13, 2021

  10. mrm9084 commented on Sep 13, 2021

    @mrm9084
    Matthew Metcalf (mrm9084)
    on Sep 13, 2021
    Member

    PbALpi7xEX I think this might be more than one issue.

    I have been able to replicate a similar issue but only with feature flags. Are you having a case where the trigger value changing doesn't cause a refresh too?

    Part of the issue might also be that the feature flag label only supports a single label, not multiple. I can work on adding it to this existing work item for feature flags as it is similar. Azure/AppConfiguration#497

    Thanks for pointing out the typo. Will update that.

    I will continue to try and replicate the configurations not refreshing while I fix the feature flag one.

  11. PbALpi7xEX commented on Sep 13, 2021

    @PbALpi7xEX
    PbALpi7xEX
    on Sep 13, 2021
    Author

    Are you having a case where the trigger value changing doesn't cause a refresh too?

    Yes. Updating the trigger doesn't seem to get the application to refresh it's configuration.

  12. mrm9084 commented on Sep 13, 2021

    @mrm9084
    Matthew Metcalf (mrm9084)
    on Sep 13, 2021
    Member

    After you make the change do you see a logs that looks similar to this?

    2021-09-13 11:54:43.640  INFO 25168 --- [         task-3] c.a.d.a.ConfigurationAsyncClient         : Listing ConfigurationSettings - SettingSelector(keyFilter=app.config.refreshTrigger, labelFilter= , acceptDateTime=null, fields=ALL_FIELDS)
2021-09-13 11:54:43.726  INFO 25168 --- [ctor-http-nio-2] c.a.d.a.ConfigurationAsyncClient         : Listed ConfigurationSettings - SettingSelector(keyFilter=app.config.refreshTrigger, labelFilter= , acceptDateTime=null, fields=ALL_FIELDS)
2021-09-13 11:54:43.726  INFO 25168 --- [         task-3] c.a.s.c.config.AppConfigurationRefresh   : Configuration Refresh Event triggered by app.config.refreshTrigger

  13. PbALpi7xEX commented on Sep 13, 2021

    @PbALpi7xEX
    PbALpi7xEX
    on Sep 13, 2021
    Author

    No

  14. 2 remaining items

  15. mrm9084 commented on Sep 13, 2021

    @mrm9084
    Matthew Metcalf (mrm9084)
    on Sep 13, 2021
    Member

    PbALpi7xEX Did you not update the value of app.config.refreshTrigger? The watch key needs it's value changed. It is how we know to refresh. It is one of the major differences between the 1.3.0 and 2.0.0 versions.

  16. PbALpi7xEX commented on Sep 13, 2021

    @PbALpi7xEX
    PbALpi7xEX
    on Sep 13, 2021
    Author

    Yes, updated app.config.refreshTrigger

  17. PbALpi7xEX commented on Sep 16, 2021

    @PbALpi7xEX
    PbALpi7xEX
    on Sep 16, 2021
    Author

    Matthew Metcalf (@mrm9084) FYI.. I enabled debug and seeing the following:

    2021-09-16 16:24:47,752 DEBUG [threadPoolExecutor-2] com.azure.spring.cloud.config.AppConfigurationRefresh: Skipping configuration refresh check for https://****-secondary.azconfig.io
2021-09-16 16:24:47,753 DEBUG [threadPoolExecutor-2] com.azure.spring.cloud.config.AppConfigurationRefresh: Skipping feature flag refresh check for https://****-secondary.azconfig.io

    Something in my config is causing it to skip?

    spring.cloud.azure.appconfiguration.stores[0].endpoint=${FAILOVER_APPCONFIG_ENDPOINT}
spring.cloud.azure.appconfiguration.stores[0].enabled=true
spring.cloud.azure.appconfiguration.stores[0].monitoring.enabled=true
spring.cloud.azure.appconfiguration.stores[0].monitoring.refresh-interval=5m
spring.cloud.azure.appconfiguration.stores[0].monitoring.triggers[0].key=app.config.refreshTrigger
spring.cloud.azure.appconfiguration.stores[0].feature-flags.enabled=true
spring.cloud.azure.appconfiguration.stores[0].monitoring.feature-flags.watch-interval=5m
spring.cloud.azure.appconfiguration.stores[0].selects[0].label-filter=,default,${spring.profiles.active}
spring.cloud.azure.appconfiguration.stores[0].feature-flags.label-filter=${spring.profiles.active}
spring.cloud.azure.appconfiguration.stores[0].fail-fast=true

  18. mrm9084 commented on Sep 16, 2021

    @mrm9084
    Matthew Metcalf (mrm9084)
    on Sep 16, 2021
    Member

    PbALpi7xEX That means refresh is disabled for some reason, or the trigger key didn't change. By any chance does you app.config.refreshTrigger have a label on it? If so it isn't listed in the triggers section.

  19. mrm9084 commented on Sep 16, 2021

    @mrm9084
    Matthew Metcalf (mrm9084)
    on Sep 16, 2021
    Member

    I am working on a sample that might help. It can be found here Azure-Samples/azure-spring-boot-samples#48. Going to add some steps for sample store creation. Hope it helps.

  20. PbALpi7xEX commented on Sep 17, 2021

    @PbALpi7xEX
    PbALpi7xEX
    on Sep 17, 2021
    Author

    Thanks... still can't get it working... looking at debug a bit closer, and looked into code... seems the etag checks never mismatch and triggering the refresh:

     /**
     * Checks refresh trigger for etag changes. If they have changed a RefreshEventData is published.
     *
     * @param state The refresh state of the endpoint being checked.
     * @param endpoint The App Config Endpoint being checked for refresh.
     * @param refreshInterval Amount of time to wait until next check of this endpoint.
     * @return Refresh event was triggered. No other sources need to be checked.
     */
    private boolean refresh(State state, String endpoint, Duration refreshInterval) {
        Date date = new Date();
        if (date.after(state.getNextRefreshCheck())) {
            for (ConfigurationSetting watchKey : state.getWatchKeys()) {
                ConfigurationSetting watchedKey = clientStore.getWatchKey(watchKey.getKey(), watchKey.getLabel(),
                    endpoint);
                String etag = null;
                // If there is no result, etag will be considered empty.
                // A refresh will trigger once the selector returns a value.
                if (watchedKey != null) {
                    etag = watchedKey.getETag();
                }

                LOGGER.debug(etag + " - " + watchKey.getETag());
                if (etag != null && !etag.equals(watchKey.getETag())) {
                    LOGGER.trace(
                        "Some keys in store [{}] matching the key [{}] and label [{}] is updated, "
                            + "will send refresh event.",
                        endpoint, watchKey.getKey(), watchKey.getLabel());

                    this.eventDataInfo = watchKey.getKey();

                    // Only one refresh Event needs to be call to update all of the
                    // stores, not one for each.
                    LOGGER.info("Configuration Refresh Event triggered by " + eventDataInfo);

                    RefreshEventData eventData = new RefreshEventData(eventDataInfo);
                    publisher.publishEvent(new RefreshEvent(this, eventData, eventData.getMessage()));
                    return true;
                }
            }

            // Just need to reset refreshInterval, if a refresh was triggered it will updated after loading the new
            // configurations.
            StateHolder.setState(state, refreshInterval);
        }

        return false;
    }

    I see the logger.debug from above output the following in my console:

    2021-09-17 16:32:38,591 DEBUG [threadPoolExecutor-1] com.azure.spring.cloud.config.AppConfigurationRefresh: null - null
2021-09-17 16:32:38,592 DEBUG [threadPoolExecutor-1] com.azure.spring.cloud.config.AppConfigurationRefresh: Skipping configuration refresh check for https://****-secondary.azconfig.io
2021-09-17 16:32:38,691 DEBUG [threadPoolExecutor-1] com.azure.spring.cloud.config.AppConfigurationRefresh: FKtNVZp4e4MtZtPu5Dfj2q4aLYw - FKtNVZp4e4MtZtPu5Dfj2q4aLYw
2021-09-17 16:32:38,692 DEBUG [threadPoolExecutor-1] com.azure.spring.cloud.config.AppConfigurationRefresh: Skipping feature flag refresh check for https://****-secondary.azconfig.io

    Looking into the history of my watch key in app config, I don't see any of the etags matching what is output in the debug console... not sure where "FKtNVZp4e4MtZtPu5Dfj2q4aLYw " is coming from, but not found in my watch key etag history at that timestamp.

  21. mrm9084 commented on Sep 17, 2021

    @mrm9084
    Matthew Metcalf (mrm9084)
    on Sep 17, 2021
    Member

    PbALpi7xEX Can we try you creating a new basic sentinel key. Just to make sure do the following.

    az appconfig kv set --name <your-store-name> --key sentinel --value 0 -y

    Update you config client app to use the key sentinel, with no label. Once you get it running do:

    az appconfig kv set --name <your-store-name> --key sentinel --value 1 -y

  22. PbALpi7xEX commented on Sep 20, 2021

    @PbALpi7xEX
    PbALpi7xEX
    on Sep 20, 2021
    Author

    Matthew Metcalf (@mrm9084) Well interesting , after applying the sentinel key update as specified, and firing a request in the app, I saw the following:

    2021-09-20 09:38:14,394 DEBUG [threadPoolExecutor-4] com.azure.spring.cloud.config.AppConfigurationRefresh: ZKQYXJ4mdDIE0pspaCEZgXroWkr - pyfnH2a7xBljKmHsCiFV7axGNzD
2021-09-20 09:38:14,394 INFO  [threadPoolExecutor-4] com.azure.spring.cloud.config.AppConfigurationRefresh: Configuration Refresh Event triggered by sentinel
2021-09-20 09:38:23,768 DEBUG [threadPoolExecutor-4] com.azure.spring.cloud.config.AppConfigurationPropertySourceLocator: PropertySource context [/application/] is added.

    So the sentinel key is not using the /application/ prefix in app configuration...

    image

    All my other keys are, and loading correctly...

    My previous watch key was set with /application/ prefix:

    image

    So it seems we shouldn't be prefixing the watch key with /application/ prefix?

  23. mrm9084 commented on Sep 20, 2021

    @mrm9084
    Matthew Metcalf (mrm9084)
    on Sep 20, 2021
    Member

    That is correct. We don't auto add prefixes to watch keys. This was to enable the system to work for people who both want and don't want the prefix on any keys.

    Feel free to add the prefix to you watch key, you just need to add it to your watch configuration and everything should work fine. Here is a list of all of the configurations and their default values. https://github.com/Azure/azure-sdk-for-java/tree/main/sdk/appconfiguration/azure-spring-cloud-starter-appconfiguration-config#supported-properties

  24. PbALpi7xEX commented on Sep 20, 2021

    @PbALpi7xEX
    PbALpi7xEX
    on Sep 20, 2021
    Author

    Thanks... my bad, guess it wasn't clear to me in the documentation.

    Using the prefix in my boostrap properties works properly:

    2021-09-20 13:16:55,174 DEBUG [threadPoolExecutor-4] com.azure.spring.cloud.config.AppConfigurationRefresh: rl3psxDawV7VEJpkNYQy46N9wEG - RWFHLWJwkltFshI5FDLgWXXBUXY
2021-09-20 13:16:55,174 INFO  [threadPoolExecutor-4] com.azure.spring.cloud.config.AppConfigurationRefresh: Configuration Refresh Event triggered by /application/app.config.refreshTrigger

    So i guess this just leaves the issue with feature flag refreshes then.

  25. mrm9084 commented on Sep 21, 2021

    @mrm9084
    Matthew Metcalf (mrm9084)
    on Sep 21, 2021
    Member

    PbALpi7xEX triggering a refresh this was also updates feature flags. So that can be a work around. I have found the bug with the feature flags. It will be in the next release.

  26. mrm9084 commented on Oct 8, 2021

    @mrm9084
    Matthew Metcalf (mrm9084)
    on Oct 8, 2021
    Member

    PbALpi7xEX 2.1.1 is released and should fix this last issue. Let me know if you have any further problems.

  27. PbALpi7xEX commented on Oct 8, 2021

    @PbALpi7xEX
    PbALpi7xEX
    on Oct 8, 2021
    Author

    Thanks Matthew Metcalf (@mrm9084)

    Noticed the update to 2.1.1 yesterday so pulled it in and seems all is working as it should.

    Have a good weekend

  28. PbALpi7xEX
    closed this as completedon Oct 8, 2021
  29. github-actions
    locked and limited conversation to collaborators on Apr 11, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

ClientThis issue points to a problem in the data-plane of the library.azure-springAll azure-spring related issuesazure-spring-app-configurationSpring app configuration related issues.customer-reportedIssues that are reported by GitHub users external to the Azure organization.questionThe issue doesn't require a change to the product in order to be resolved. Most issues start as that

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions