Ability to specify RAM VM watermark in absolute terms

[michaelklishin]

FYI, in RabbitMQ 3.5.7 there will be a way to specify VM memory watermark as an absolute value, which, we are told, should make using RabbitMQ easier in containerized environments.

[yosifkit]

Oh, neat. Thanks for the heads up. Do you think we need to add to the documentation on Docker Hub?

[michaelklishin]

@yosifkit I think so but lets wait till 3.5.7 is actually out. We'll keep you posted.

[tianon]

Now that 3.5.7 is out, I think it's probably worth documenting and/or implementing something along these lines. Looking at the docs (https://github.com/rabbitmq/rabbitmq-website/pull/96/files), I'm wondering if this might make sense for our entrypoint script to set automatically if it detects a memory limit (basically just letting RabbitMQ itself know that there is a memory limit so we get graceful behavior instead of unexpected OOM).

[mthenw]

I will just +1 @tianon's idea. Would be great!

[tianon]

Looking at that upstream PR again, it would appear that in the face of an enormously outrageous number, RabbitMQ would act intelligently and instead cap at available memory, so we can probably just use the container's memory.limit_in_bytes directly (converted to MB like RabbitMQ expects, of course):

$ docker run -it --rm alpine cat /sys/fs/cgroup/memory/memory.limit_in_bytes
9223372036854771712
$ docker run -it --rm -m 8m alpine cat /sys/fs/cp/memory/memory.limit_in_bytes
8388608

[tianon]

More explicitly:

$ docker run -it --rm -m 8m rabbitmq bash -c 'echo $(( $(< /sys/fs/cgroup/memory/memory.limit_in_bytes) / 1024 / 1024 ))'
8
$ docker run -it --rm -m 1g rabbitmq bash -c 'echo $(( $(< /sys/fs/cgroup/memory/memory.limit_in_bytes) / 1024 / 1024 ))'
1024
$ docker run -it --rm rabbitmq bash -c 'echo $(( $(< /sys/fs/cgroup/memory/memory.limit_in_bytes) / 1024 / 1024 ))'
8796093022207

[tianon]

Even more explicitly, here's the exact PR I'm contemplating:

diff --git a/docker-entrypoint.sh b/docker-entrypoint.sh
index e070eba..502b55b 100755
--- a/docker-entrypoint.sh
+++ b/docker-entrypoint.sh
@@ -212,6 +212,13 @@ if [ "$1" = 'rabbitmq-server' ] && [ "$haveConfig" ]; then
                "{ loopback_users, $(rabbit_array) }"
        )

+       # https://github.com/rabbitmq/rabbitmq-website/pull/96/files
+       # "If the absolute limit is larger than the installed RAM or available virtual address space, the threshold is set to whichever limit is smaller."
+       if [ -r /sys/fs/cgroup/memory/memory.limit_in_bytes ]; then
+               memLimit="$(< /sys/fs/cgroup/memory/memory.limit_in_bytes)"
+               rabbitConfig+=( "{ vm_memory_high_watermark, { absolute, $(( memLimit / 1024 / 1024 )) } }" )
+       fi
+
        if [ "$haveSslConfig" ]; then
                IFS=$'\n'
                rabbitSslOptions=( $(rabbit_env_config 'ssl' "${sslConfigKeys[@]}") )

[komuw]

I'm running and setting the memory limit to 80MB(80388608 bytes) but the rabbitMQ container doesn't seem to respect that, it goes ahead and allocates its own memory based on the total 2GB out of my machine.

root@test-broker-1:~# docker run -it --rm -m 80388608 rabbitmq:3.6.5-management

          RabbitMQ 3.6.5. Copyright (C) 2007-2016 Pivotal Software, Inc.

## Licensed under the MPL. See http://www.rabbitmq.com/

########## Logs: tty

## tty

##########
Starting broker...

=INFO REPORT==== 23-Sep-2016::11:01:18 ===
Starting RabbitMQ 3.6.5 on Erlang 19.0
Copyright (C) 2007-2016 Pivotal Software, Inc.
Licensed under the MPL. See http://www.rabbitmq.com/

=INFO REPORT==== 23-Sep-2016::11:01:18 ===
node : rabbit@b59f7874b4fb
home dir : /var/lib/rabbitmq
config file(s) : /etc/rabbitmq/rabbitmq.config
cookie hash : someHash==
log : tty
sasl log : tty
database dir : /var/lib/rabbitmq/mnesia/rabbit@b59f7874b4fb

=INFO REPORT==== 23-Sep-2016::11:01:23 ===
Memory limit set to 801MB of 2002MB total.

System stats:

1. root@test-broker-1:~# lsb_release -a
   No LSB modules are available.
   Distributor ID: Ubuntu
   Description: Ubuntu 16.04 LTS
   Release: 16.04
   Codename: xenial
2. root@test-broker-1:~# uname -r
   4.5.7-std-3
3. root@test-broker-1:~# docker -v
   Docker version 1.12.1, build 23cf638