hswaw

rook-discover pods in rook-ceph-systems use `ceph-volume inventory` which gets stuck in subprocess.Popen due to a very high ulimit in the container:

[root@rook-discover-9b8r6 /]# ulimit -n
1073741816

This is like https://github.com/coreos/fedora-coreos-tracker/issues/329, but we already set the containerd limit and that seems to have helped in the past for some other containers (iirc some in ceph-waw3, osd-prepares I think?). This might be getting triggered here because this particular container is privileged (and likely has CAP_SYS_RESOURCE), and something bumps up ulimit -n up to fs.nr_open. But I'm not sure.

I've temporarily bypassed this by setting fs.nr_open to a lower limit on affected nodes and manually restarting the discovery pods, but this needs to be fixed better.