Store modules settings in separate CRDs #269

Problem overview

ConfigMap has its limits.

• It is not convenient to write settings for many modules:
  • Users need to always remember about the |
  • Suffix Enabled is special
  • Differences between boolean declarations
  • No syntax highlighting

apiVersion: v1
kind: ConfigMap
metadata:
  name: deckhouse
  namespace: d8-system
spec:
  userAuthzEnabled: "true"
  userAuthz: |
    enabledMultitenancy: true

• It is hard to orient in a big configmap.
• There is no feedback for the user whether their changes are accepted or not.
• We need to validate, edit and migrate the whole configmap, which is a single point of failure.

Describe the solution you'd like to see

1. Create CRD manually (addon-operator will not be managing CRDs).
2. The structure of CRD fields must be like this:

apiVersion: deckhouse.io/v1alpha1
kind: ModuleSettings
metadata:
  name: prometheus # the name of the module
  namespace: d8-system
enabled: true / false
spec:
  {module settings here ... }
status:
  lastCheckTime: 2019-08-08T12:00:00
  condition: accepted

3. User tells add-operator which apiVersion and kind of the CRD to listen instead of ConfigMap.
4. On every change, addon-operator validates spec parameters with an OpenAPI config schema. If validation pass, it sets custom resource status to accepted.
5. If validation fails, it sets custom resource status to validation error with an error message:

status:
  lastCheckTime: 2019-08-08T12:00:00
  condition: error
  message: |
    ...

5. If the module is not supported by the addon-operator, it will write unknownModule to the status:

status:
  lastCheckTime: 2019-08-08T12:00:00
  condition: unknownModule

Additional context

We also need an implementation of MutatingWebhook to write migrations for modules settings to store them in Git.