Unable To Enable Prometheus Lifecycle API In Azure Container Instances

Question

I've been having difficulty adding the --web.enable-lifecycle flag to prometheus, to enable the lifecycle api.

I have this working locally with docker compose, like so:

  prometheus:
    image: prom/prometheus
    volumes:
      - ./tmp/prometheus:/prometheus
      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
      - ./prometheus/targets:/etc/prometheus/targets
    networks:
      - localprom
    ports:
      - 9090:9090
    command:
      - --web.enable-lifecycle
      - --config.file=/etc/prometheus/prometheus.yml

But Azure's YAML format for creating/updating container instances is a little different than docker. Here is my YAML file:

apiVersion: 2021-10-01
location: useast
name: my-prometheus-container-instances
tags: { environment: development }
type: Microsoft.ContainerInstance/containerGroups

properties:
  containers:
    - name: prometheus
      properties:
        image: prom/prometheus:v2.45.6
        command:
          - /bin/prometheus
          - --config.file=/etc/prometheus/prometheus.yml
          - --web.enable-lifecycle
        resources:
          requests:
            cpu: 1
            memoryInGb: 1.5
        ports:
          - port: 9090
        volumeMounts:
          - name: prometheus-fileshare-volume
            mountPath: /etc/prometheus/

  subnetIds:
    - id: MY_SUBNET_ID

  osType: Linux
  volumes:
    - name: prometheus-fileshare-volume
      azureFile:
        shareName: prometheus-fileshare
        readOnly: true
        storageAccountName: prometheus-storage-account
        storageAccountKey: MY_ACCESS_KEY

  RestartPolicy: OnFailure
  ipAddress:
    type: Private
    ports:
      - protocol: tcp
        port: 9090

Note that the Azure YAML file requires - /bin/prometheus in the command array, while docker does not. Without this, prometheus will not start after deploying. I've also confirmed that the --config.file command is working, wondering if there is something else that may be stopping me from enabling the lifecycle API after deploying to Azure.

Answer 1

To enable the Prometheus lifecycle API in an Azure Container Instance using a YAML configuration, you need to make sure that the command array is properly configured and that all necessary parameters are correctly passed.

Create a storage account

az storage account create --name prometheusstorageacct --resource-group arkorg --location eastus --sku Standard_LRS

Create a fileshare

az storage share create --name prometheusfileshare --account-name prometheusstorageacct

Get the storage key

Create your Prometheus Configuration Files. I am just using a basic example, you edit as per your own requirement.

prometheus.yml:

global:
  scrape_interval: 15s # Set the scrape interval to every 15 seconds.

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  # Example job to scrape metrics from a target
  - job_name: 'example'
    static_configs:
      - targets: ['example.com:80']

targets.yml:

- targets:
  - 'target1.example.com:9100'
  - 'target2.example.com:9100'

Upload Configuration Files to Azure File Share

az storage file upload --share-name prometheusfileshare --source ./prometheus.yml --account-name prometheusstorageacct --account-key $STORAGE_KEY
az storage file upload --share-name prometheusfileshare --source ./targets.yml --account-name prometheusstorageacct --account-key $STORAGE_KEY

Create the Azure Container Instance

apiVersion: 2021-10-01
location: eastus
name: my-prometheus-container-instances
type: Microsoft.ContainerInstance/containerGroups
properties:
  containers:
    - name: prometheus
      properties:
        image: prom/prometheus:v2.45.6
        command:
          - /bin/prometheus
          - --config.file=/etc/prometheus/prometheus.yml
          - --web.enable-lifecycle
        resources:
          requests:
            cpu: 1
            memoryInGb: 1.5
        ports:
          - port: 9090
        volumeMounts:
          - name: prometheus-fileshare-volume
            mountPath: /etc/prometheus/
  osType: Linux
  volumes:
    - name: prometheus-fileshare-volume
      azureFile:
        shareName: prometheusfileshare
        readOnly: true
        storageAccountName: prometheusstorageacct
        storageAccountKey: ...6tNg==
  restartPolicy: OnFailure
  ipAddress:
    type: Public
    ports:
      - protocol: tcp
        port: 9090

Deploy the Azure Container Instance

az container create --resource-group arkorg --file prometheus-aci.yaml

Verify and access Prometheus

az container show --resource-group arkorg --name my-prometheus-container-instances

access prometheus by getting the IP

az container show --resource-group arkorg --name my-prometheus-container-instances --query "ipAddress.ip" --output tsv

go to your browser and put the IP you see as output for the above command

http://<container-ip>:9090.

you can verify your lifecycle using something like

curl -X POST http://52.188.24.225:9090/-/reload

Reference-

prometheus | Prometheus