icinga2 pki ticket & icinga2 node setup on LXC unprivileged container error

[JimKlapwijk]

Describe the bug

So the issue is caused because of unprivileged containers.
Running the same commands in a privileged container works. So I guess it’s a bug related to Icinga, and caused by the error:
warning/Application: Failed to adjust resource limit for open file handles (RLIMIT_NOFILE) with error "Operation not permitted

When I run the commands to either generate a ticket (icinga2 pki new-cert), or use icinga2 node setup it generates an error.
This does not happen on a privileged container.

To Reproduce

Provide a link to a live example, or an unambiguous set of steps to reproduce this bug. Include configuration, logs, etc. to reproduce, if relevant.

1. ticket=$(icinga2 pki ticket --cn librenms.example.com --salt "a+uXX3LMrqMUuutfjo9IuSKyrpqZYEoT9IV1USyr")

2.
icinga2 node setup --ticket ${ticket} \
              --cn librenms.example.com \
              --endpoint icinga.example.com \
              --zone librenms.example.com \
              --parent_zone master \
              --parent_host icinga.example.com \
              --trustedcert /etc/icinga2/pki/trusted-parent.crt \
              --accept-commands --accept-config \
              --disable-conf

Expected behavior

A clear and concise description of what you expected to happen.

Screenshots

If applicable, add screenshots to help explain your problem.

Your Environment

Include as many relevant details about the environment you experienced the problem in

• Version used (icinga2 --version): 2.12.3-1.buster and v2.12.0-rc1.
• Operating System and version: Debian 10, Ubuntu 20.
• Enabled features (icinga2 feature list): api checker icingadb ido-mysql mainlog notification
• Icinga Web 2 version and modules (System - About):

Icinga Web 2 Version
2.8.2
Git commit
8a89839af94a247ee2149b2336c73b8251b477c0
PHP Version
7.3.19-1~deb10u1
Git commit date
2020-08-17
Copyright
© 2013-2021 The Icinga Project

• Config validation (icinga2 daemon -C):

[2021-01-22 21:50:41 +0100] information/cli: Icinga application loader (version: r2.12.3-1)
[2021-01-22 21:50:41 +0100] information/cli: Loading configuration file(s).
[2021-01-22 21:50:42 +0100] information/ConfigItem: Committing config item(s).
[2021-01-22 21:50:42 +0100] information/ApiListener: My API identity: icinga.klapwijk.it
[2021-01-22 21:50:42 +0100] information/ConfigItem: Instantiated 1 NotificationComponent.
[2021-01-22 21:50:42 +0100] information/ConfigItem: Instantiated 8 Hosts.
[2021-01-22 21:50:42 +0100] information/ConfigItem: Instantiated 4 NotificationCommands.
[2021-01-22 21:50:42 +0100] information/ConfigItem: Instantiated 1 FileLogger.
[2021-01-22 21:50:42 +0100] information/ConfigItem: Instantiated 1 IcingaApplication.
[2021-01-22 21:50:42 +0100] information/ConfigItem: Instantiated 1 IcingaDB.
[2021-01-22 21:50:42 +0100] information/ConfigItem: Instantiated 1 CheckerComponent.
[2021-01-22 21:50:42 +0100] information/ConfigItem: Instantiated 10 Zones.
[2021-01-22 21:50:42 +0100] information/ConfigItem: Instantiated 8 Endpoints.
[2021-01-22 21:50:42 +0100] information/ConfigItem: Instantiated 1 IdoMysqlConnection.
[2021-01-22 21:50:42 +0100] information/ConfigItem: Instantiated 1 ApiUser.
[2021-01-22 21:50:42 +0100] information/ConfigItem: Instantiated 1 ApiListener.
[2021-01-22 21:50:42 +0100] information/ConfigItem: Instantiated 238 CheckCommands.
[2021-01-22 21:50:42 +0100] information/ConfigItem: Instantiated 3 TimePeriods.
[2021-01-22 21:50:42 +0100] information/ConfigItem: Instantiated 21 Services.
[2021-01-22 21:50:42 +0100] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
[2021-01-22 21:50:42 +0100] information/cli: Finished validating the configuration file(s).

• If you run multiple Icinga 2 instances, the zones.conf file (or icinga2 object list --type Endpoint and icinga2 object list --type Zone) from all affected nodes.

Additional context

Hypervisor is running Ubuntu 20.04 with LXC and LXD via Snap:

lxd     4.10      19009  latest/stable/…  canonical✓  
root@server-01:~# lxc --version
4.10
root@server-01:~# lxd --version
4.10

[Al2Klimov]

Hello @JimKlapwijk and thank you for reporting!

Is it just the error message that confuses you or does anything actually not work because of it?

Best,
AK

[JimKlapwijk]

Hi AK,

Thanks for your reply. It actually doesn't work if you want to use automation software like Ansible or Saltstack.
Because when I use for example:
ICINGA2_CA_TICKET=$(icinga2 pki ticket --cn {{ grains.fqdn }} --salt "{{ salt['pillar.get']('ticketsalt') }}" to generate a ticket, and then want to use that to do an icinga2 pki request or icinga2 node setup it fails, because the variable includes the error message.

Thank you for your time.

[Al2Klimov]

Colleagues, any objections if we log to stderr unless on icinga2 daemon?

[julianbrost]

I'm surprised that we're not doing so already. So go for it from my side. Not sure if special handling for the daemon command is even needed. Do you have anything in mind where it would be a problem if that also logged to stderr?

[Al2Klimov]

Yes, I have – broken forwarding of logs from stdout.

[Al2Klimov]

PING @N-o-X