Director not deploying since icinga was updated to 2.12.3

[ecarreau]

Starting from jordan/icinga2:2.10.5 image I have migrated to jordan/icinga2:latest image and migrated the database on a separate container.

With the new context it has become impossible to deploy Director changes. I get a message API request failed: Stage creation failed. (RestApiResponse.php:113) in the Activity log interface, and the following lines (repeating) in my icinga container logs :

2021-04-21 11:00:52,030 INFO spawned: 'icinga2' with pid 42682
checking Icinga2 configuration.
Starting icinga2 monitoring daemon in foreground.
[2021-04-21 11:00:52 +0200] information/cli: Icinga application loader (version: r2.12.3-1)
[2021-04-21 11:00:52 +0200] critical/cli: Another instance of Icinga already running with PID 2015
icinga2 ended ... failed!
2021-04-21 11:00:53,842 INFO success: icinga2 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
ERROR: Icinga\Exception\NotFoundError in /usr/local/share/icingaweb2/modules/director/library/Director/Core/RestApiResponse.php:111 with message: The requested path 'v1/console/execute-script' could not be found or the request method is not valid for this path.
2021-04-21 11:00:57,209 INFO exited: icinga2 (exit status 1; not expected)

Here is my Icinga2 engine informations :

#icinga2 --version
icinga2 - The Icinga 2 network monitoring daemon (version: r2.12.3-1)

Copyright (c) 2012-2021 Icinga GmbH (https://icinga.com/)
License GPLv2+: GNU GPL version 2 or later <http://gnu.org/licenses/gpl2.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

System information:
  Platform: Debian GNU/Linux
  Platform version: 10 (buster)
  Kernel: Linux
  Kernel version: 3.10.0-514.10.2.el7.x86_64
  Architecture: x86_64

Build information:
  Compiler: GNU 8.3.0
  Build host: runner-hh8q3bz2-project-298-concurrent-0
  OpenSSL version: OpenSSL 1.1.1d  10 Sep 2019

Application information:

General paths:
  Config directory: /etc/icinga2
  Data directory: /var/lib/icinga2
  Log directory: /var/log/icinga2
  Cache directory: /var/cache/icinga2
  Spool directory: /var/spool/icinga2
  Run directory: /run/icinga2

Old paths (deprecated):
  Installation root: /usr
  Sysconf directory: /etc
  Run directory (base): /run
  Local state directory: /var

Internal paths:
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /run/icinga2/icinga2.pid

, my /etc/icinga2/conf.d/api-users.conf file :

/**
 * The APIUser objects are used for authentication against the API.
 */
object ApiUser "root" {
  password = "xxxxx"
  // client_cn = ""

  permissions = [ "*" ]
}

, my /etc/icinga2/conf.d/api-user.conf file (the password is dynamicaly regenerated when container restart) :

object ApiUser "icinga2-director" {
  password = "DAEx2aD3JKSIVT4"
  permissions = [ "*" ]
}

and my Docker stack :

version: "3" 
services:
  icinga:
    image: jordan/icinga2:latest
    container_name: icinga
    hostname: icinga
    ports:
    - "8007:80" 
    - "5665:5665" 
    environment:
    - ICINGA2_FEATURE_GRAPHITE=true
    - ICINGA2_FEATURE_GRAPHITE_HOST=graphite
    - ICINGA2_FEATURE_GRAPHITE_PORT=2003
    - DEFAULT_MYSQL_HOST=mariadb
    - MYSQL_ROOT_PASSWORD=xxxxxxxxxxxx
    - DEFAULT_MYSQL_USER=icinga2
    - DEFAULT_MYSQL_PASS=xxxxxxxxxxxxxxx
    volumes:
    - /docker/persistance/debian/localtime:/etc/localtime
    - /docker/persistance/debian/timezone:/etc/timezone
    - /docker/persistance/icinga/etc:/etc/icinga2
    - /docker/persistance/icingaweb/etc:/etc/icingaweb2
    - /docker/persistance/icinga/data:/var/lib/icinga2
    - /docker/persistance/icinga/php:/var/lib/php/sessions/
    depends_on:
    - mariadb
    deploy:
      replicas: 1
      placement:
        constraints:
          - node.role == manager

  mariadb:
    image: mariadb:10.4
    volumes:
    - /docker/persistance/icinga/mysql:/var/lib/mysql
    environment:
    - MYSQL_ROOT_PASSWORD=xxxxxxxxxxxxxx
    - MYSQL_DATABASE=icinga2
    - MYSQL_USER=icinga2
    - MYSQL_PASSWORD=xxxxxxxxxxxxxxxxx
    deploy:
      replicas: 1
      placement:
        constraints:
          - node.role == manager

  graphite:
    image: graphiteapp/graphite-statsd:latest
    container_name: graphite
    ports:
    - "2003:2003" 
    - "2004:2004" 
    - "2023:2023" 
    - "2024:2024" 
    - "8008:80" 
    - "8025:8025" 
    - "8026:8026" 
    volumes:
    - "/docker/persistance/graphite/conf:/opt/graphite/conf" 
    - "/docker/persistance/graphite/storage:/opt/graphite/storage" 
    - "/docker/persistance/graphite/statsd:/opt/statsd" 
    - "/docker/persistance/graphite/logrotate.d:/etc/logrotate.d" 
    - "/docker/persistance/graphite/log:/var/log" 
    deploy:
      replicas: 1
      placement:
        constraints:
          - node.role == manager

I've been searching for 2 days now, but found nothing.
Thank you for your help.

[ecarreau]

When I try to execute the failing command with curl from the icinga2 container, I get the following error :

root@icinga:/# curl -k -s -u icinga2-director:xxxxx -H 'Accept: application/json' -X POST 'https://localhost:5665/v1/console/execute-script?command=get_host(NodeName)'
{
  "error":404.0,
  "status":"The requested path 'v1/console/execute-script' could not be found or the request method is not valid for this path."
}

[jjethwa]

Hi @ecarreau

Thanks for the report and including all of the necessary details. I took a quick look at the director repo: https://github.com/Icinga/icingaweb2-module-director and the image is using the latest release: https://github.com/jjethwa/icinga2/blob/master/Dockerfile#L88 but looks like there has been a lot of commits since then, so another release is on the way. The only thing is, I don't see any fixes for this particular issue. While I do some more research, do you mind opening up an issue on the upstream project? https://github.com/Icinga/icingaweb2-module-director/issues

Thanks!