Excessive memory usage or stuck software due to increased file descriptor ulimit

Symptoms

Software uses excessive amounts of memory or is stuck in a busy loop.

Cause

The default number of file descriptors a process is allowed to open is much higher on Alma Linux 9 (1,073,741,816) than on CentOS 7 (1024).

Some software packages perform an exhaustive search of file descriptors or allocate data structures for each possible file descriptor.

The much higher number of file descriptors under Alma Linux 9 then leads to excessive memory usage that can quickly result in swapping or out-of-memory errors or just a busy loop.

Affected Software

• xrootd memory consumption scales unboundly with RLIMIT_NOFILE

• yum when running CentOS 7 docker container on newer host.

• ACE

Solution

If the software itself can’t be fixed, setting ulimit -n to a lower value than the default can fix the issue, e.g.:

$ ulimit -n 1024:2048

Docker has a command line option for this:

$ docker run --ulimit nofile=1024:2048 ...

These values can also be set using in a docker compose file:

services:
  example:
    image: ...
    ulimits:
      nofile:
        soft: 1024
        hard: 2048

The value of 1024:2048 is just an example, ACADA chose 2^20 = 1048576.