Use a projected key to mount rabbitmq configuration provided by the Operator

[Zerpet]

Context

The file rabbitmq.conf should be used by our users to configure their instances of RabbitMQ. The configuration provided by the Operator should be separated from the end-user configuration.

Proposed solution

Leverage /etc/rabbitmq/conf.d location to place Operator configurations of RabbitMQ. The property .spec.rabbitmq.additionalConf should be modified to create a ConfigMap and mount the same as a projected key in /etc/rabbitmq/rabbitmq.conf

Something to test:

• What's the order of configuration load? First rabbitmq.conf, then conf.d/*.conf. Configuration in conf.d/*.conf have higher precedence over rabbitmq.conf.

Tasks

• Write our default or "mandatory" configuration first in the ConfigMap. Append contents of .spec.rabbitmq.additionalConfig. We already do this.
• Use the ConfigMap with rabbitmq.conf key and mount it as a projected key
• Remove copying the rabbitmq.conf via the init container
• Change the location of the plugins configuration file *

* /etc/rabbitmq is not an emptyDir anymore. Some Docker images may come with pre-set or vendor configuration already present in /etc/rabbitmq, therefore we should not mount this directory.

[ChunyiLyu]

As discussed earlier, I think we need to figure out the order of /etc/rabbitmq/conf.d and /etc/rabbitmq/rabbitmq.conf being loaded to understand whether users' configurations can still overwrite default configurations with this change. We need to make sure that users can overwrite default configurations either way.

[Zerpet]

Context

The configuration load order is rabbitmq.conf, then every file in conf.d/*.conf. This poses an additional challenge, as we want to keep flexibility of configuration, even allow the end user to override vendored configuration.

The solution to overcome this is to create a ConfigMap with RabbitMQ configuration, put our configuration at the begining of the rabbitmq.conf key and append any user provided configuration via the spec. In this way, any configuration provided by the user will override our configuration.

After #346, it is no longer possible to configure the default user and pass via .spec.rabbitmq.additionalConfig. I think this is ok, as we never intended to allow customisation of default user and pass via the spec's additionalConfig field. It is still possible to configure other users via the import definitions feature.

Edit: updated issue with tasks list with this context in consideration

[Zerpet]

Context

Use the ConfigMap with rabbitmq.conf key and mount it as a projected key

For this task, we have to add a mount path for each key we want to configure, similar to what we do in the TLS setup. This has a trick tho, if some keys are not present in the config map, this will mount the keys as empty directories, which would be confusing.

We have to modify the logic in the SteatefulSet resource so that it includes the mount points based on the presence of the keys in the config map. See https://github.com/rabbitmq/cluster-operator/tree/vendor-configurations for work in progress commit with this idea.

[Zerpet]

This is how our StatefulSet should look like:

Edit: The plugins file can't be mounted as a projected key because it must be writable by RabbitMQ process.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  labels:
    app.kubernetes.io/component: rabbitmq
    app.kubernetes.io/name: spike
    app.kubernetes.io/part-of: rabbitmq
  name: spike
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: spike
  serviceName: spike-rabbitmq
  template:
    metadata:
      labels:
        app.kubernetes.io/component: rabbitmq
        app.kubernetes.io/name: spike
        app.kubernetes.io/part-of: rabbitmq
    spec:
      containers:
      - env:
        - name: MY_POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: MY_POD_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        - name: K8S_SERVICE_NAME
          value: spike-rabbitmq
        - name: RABBITMQ_USE_LONGNAME
          value: "true"
        - name: RABBITMQ_NODENAME
          value: rabbit@$(MY_POD_NAME).$(K8S_SERVICE_NAME).$(MY_POD_NAMESPACE)
        - name: K8S_HOSTNAME_SUFFIX
          value: .$(K8S_SERVICE_NAME).$(MY_POD_NAMESPACE)
          # TODO choose a sensible location
        - name: RABBITMQ_ENABLED_PLUGINS_FILE
          value: /operator/enabled_plugins
          image: rabbitmq:3.8.9
        name: rabbitmq
        ports:
        - containerPort: 4369
          name: epmd
          protocol: TCP
        - containerPort: 5672
          name: amqp
          protocol: TCP
        - containerPort: 15672
          name: http
          protocol: TCP
        - containerPort: 15692
          name: prometheus
          protocol: TCP
        readinessProbe:
          exec:
            command:
            - /bin/sh
            - -c
            - rabbitmq-diagnostics check_port_connectivity
          failureThreshold: 3
          initialDelaySeconds: 10
          periodSeconds: 30
          successThreshold: 1
          timeoutSeconds: 5
        resources:
          limits: {}
          requests: {}
        volumeMounts:
        - mountPath: /var/lib/rabbitmq/mnesia/
          name: persistence
          # TODO subPath as filename
          # TODO mountPath for every key
        - mountPath: /etc/rabbitmq/rabbitmq.conf
          name: server-conf
          subPath: rabbitmq.conf
        - mountPath: /etc/rabbitmq/conf.d/default_user.conf
          name: rabbitmq-confd
          # TODO subPath for every key in confd
          subPath: default_user.conf
        - mountPath: /var/lib/rabbitmq/
          name: rabbitmq-erlang-cookie
        - mountPath: /etc/pod-info/
          name: pod-info
          # TODO new location for plugins file
        - mountPath: /operator
          name: rabbitmq-plugins
      dnsPolicy: ClusterFirst
      imagePullSecrets:
      - name: tanzu-registry-access
      initContainers:
      - command:
        - sh
        - -c
        - cp /tmp/erlang-cookie-secret/.erlang.cookie /var/lib/rabbitmq/.erlang.cookie
          && chown 999:999 /var/lib/rabbitmq/.erlang.cookie && chmod 600 /var/lib/rabbitmq/.erlang.cookie
          ; cp /tmp/rabbitmq-plugins/enabled_plugins /operator/enabled_plugins
          && chown 999:999 /operator/enabled_plugins
          # TODO plugins: adapt the path in the init container
        image: rabbitmq:3.8.9
        imagePullPolicy: IfNotPresent
        name: setup-container
        resources:
          limits:
            cpu: 100m
            memory: 500Mi
          requests:
            cpu: 100m
            memory: 500Mi
        securityContext:
          capabilities:
            add:
            - CHOWN
            - FOWNER
            drop:
            - ALL
          runAsUser: 0
        volumeMounts:
        - mountPath: /tmp/rabbitmq-plugins/
          name: plugins-conf
        - mountPath: /var/lib/rabbitmq/
          name: rabbitmq-erlang-cookie
        - mountPath: /tmp/erlang-cookie-secret/
          name: erlang-cookie-secret
          # TODO mount this path to receive the file from the init container
        - mountPath: /operator
          name: rabbitmq-plugins
      securityContext:
        fsGroup: 999
        runAsGroup: 999
        runAsUser: 999
      serviceAccount: spike-rabbitmq-server
      serviceAccountName: spike-rabbitmq-server
      terminationGracePeriodSeconds: 604800
      volumes:
      - configMap:
          defaultMode: 420
          name: spike-conf
          items:
            - key: rabbitmq.conf
              path: rabbitmq.conf
        name: server-conf
      - configMap:
          defaultMode: 420
          name: spike-plugins
        name: plugins-conf
        # TODO empty dir for plugins
      - emptyDir: {}
        name: rabbitmq-plugins
      - name: rabbitmq-confd
        projected:
          defaultMode: 420
          sources:
          - secret:
              items:
              - key: default_user.conf
                path: default_user.conf
              name: spike-admin
      - emptyDir: {}
        name: rabbitmq-erlang-cookie
      - name: erlang-cookie-secret
        secret:
          defaultMode: 420
          secretName: spike-erlang-cookie
      - downwardAPI:
          defaultMode: 420
          items:
          - fieldRef:
              apiVersion: v1
              fieldPath: metadata.labels['skipPreStopChecks']
            path: skipPreStopChecks
        name: pod-info
  volumeClaimTemplates:
  - apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      labels:
        app.kubernetes.io/component: rabbitmq
        app.kubernetes.io/name: spike
        app.kubernetes.io/part-of: rabbitmq
      name: persistence
      namespace: default
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 10Gi
      volumeMode: Filesystem