Allow user to specify custom setting in prometheus.yml configuration via variable

[mklassen]

Currently only a limited subset of possible configuration settings are supported in the template uses to generate the prometheus.yml configuration file. Since it is not practical to support all possible current and future settings, it would be useful to allow users to specify custom configuration settings by adding the following line to the prometheus.yml.j2 template:

{{ prometheus_custom_config | default('') | to_nice_yaml(indent=2, sort_keys=False) }}

Using the prometheus_custom_config variable would imply that the user is responsible for ensuring that it does not conflict with existing settings. Unfortunately, adding custom settings after templating via ansible.builtin.blockinfile breaks idempotency and produces flap since the role's template task see the file as changed and then the blockinfile sees the file as change, both trigger prometheus reloads on every run, even when the final version of the configuration is actually unchanged. My current work-around to avoid the flap involves the following complex play:

- name: Install prometheus
  hosts: prometheus_servers
  vars:
    prometheus_config_dir: /etc/prometheus
  pre_tasks:
    - name: Check if Prometheus config exists
      ansible.builtin.stat:
        path: "{{ prometheus_config_dir }}/prometheus.yml"
      register: prometheus_config_results
    - name: Update prometheus configuration for idempotency
      when:
        - prometheus_config_results.stat.exists
        - prometheus_config_results.stat.isreg
        - prometheus_custom_config is defined
      block:
        - name: Add custom prometheus configuration to check for changes
          ansible.builtin.blockinfile:
            path: "{{ prometheus_config_dir }}/prometheus.yml"
            block: "{{ prometheus_custom_config | default('') | to_nice_yaml(indent=2, sort_keys=False) }}"
            marker: "# {mark} custom managed block"
          register: prometheus_config_update

        - name: Remove custom prometheus configuration for role idempotency
          ansible.builtin.blockinfile:
            path: "{{ prometheus_config_dir }}/prometheus.yml"
            block: "{{ prometheus_custom_config | default('') | to_nice_yaml(indent=2, sort_keys=False) }}"
            marker: "# {mark} custom managed block"
            state: absent
          changed_when: false
  roles:
    - role: prometheus.prometheus.prometheus
  tasks:
    - name: Add custom prometheus configuration after role execution
      ansible.builtin.blockinfile:
        path: "{{ prometheus_config_dir }}/prometheus.yml"
        block: "{{ prometheus_custom_config | default('') | to_nice_yaml(indent=2, sort_keys=False) }}"
        marker: "# {mark} custom managed block"
      when: prometheus_custom_config is defined
      changed_when: prometheus_config_update is not defined or prometheus_config_update is changed
      notify: restart prometheus