Reducing a Server's Memory Footprint when OMK applications are installed

Owned by ROBIN TWIGGS (Unlicensed)
Last updated: Apr 20, 2022


Exercise caution while editing /usr/local/omk/conf/opCommon.nmis, /usr/local/omk/conf/opCommon.json or /etc/mongod.conf; if a syntax error is induced all OMK applications will cease to function.

Check out How to make configuration changes to opCommon.nmis and other files for some more details on best practices for modifying the configuration file.

Reducing a Server's Memory Footprint


NMIS 8

    • In omk/conf/opCommon.nmis:
      • omkd_workers is set at 10 by default. Try reducing this number to 2 and then restart the omkd service.
        If you THEN find YOU need more omkd_workers, increment this value by one and test again until you get to a suitable value for omkd_workers
      • /omkd/omkd_max_requests: 100 to 500 (start at 100 and increase from this value if needed)

NMIS 9

    • In omk/conf/opCommon.json:
      • omkd_workers is set at 10 by default. Try reducing this number to 2 and then restart the omkd service.
        If you THEN find YOU need more omkd_workers, increment this value by one and test again until you get to a suitable value for omkd_workers
      • /omkd/omkd_max_requests: 100 to 500 (start at 100 and increase from this value if needed)
    • in nmis9/conf/Config.nmis:
      • /system/nmisd_worker_max_cycles: 100

Consider using zram or zswap to extend available memory, with default settings, provided the server does not have high CPU usage.

Advantages of zram

  • An advantage of zram is that it does not need to be backed by an appropriately sized swap space - and this makes it easier to configure and maintain during the lifecycle of the server.
  • Another advantage of zram is that it is more likely to function in low memory environments whereas zswap should not be used when the server  has less than 1GB RAM.
  • Another advantage of zram is that I've not encountered a server kernel that is not supporting it (zram), whereas I have encountered this with zswap on occasion.

Disadvantages of zram

  • A disadvantage of zram is that the computer cannot hibernate when using zram - which is highly unlikely to be a requirement for a server.
  • Another disadvantage of zram is that it requires a few installs initially to get it running, although once configured it is easier to maintain.
  • Another disadvantage of zram is that pages will be "swapped in", i.e. decompressed when making a zram configuration change on machines with low memory.
    Instead of     restarting the service, rebooting might be a better option.
    I prefer to set up an equivalent sized swap space as a fallback for when the zram service is restarted to change its settings.

  • A final disadvantage that needs mentioning is that the command `free` reports 20% more RAM than is present when configured to use 20% of memory for zram swap, for example:

    # We have a server with 8GB RAM.
# Expecting a compression ratio of 3.00, this server has been configured to use 20% of RAM (1.6GB) as zram swap.
# This zram swap is displayed by the `free` command below as 4.7GB, being the uncompressed zram swap (1.6GB * compression ratio of 3 = 4.8GB).

# The 1.6GB of RAM used as zram swap has not been deducted from the total Mem 7.8GB as displayed by the `free` command:
# This server has 11GB of combined RAM + zram swap (7.8GB - 1.6GB + 4.7GB), not 12.5GB combined RAM + zram swap (7.8GB + 4.7GB);

free -h
              total        used        free      shared  buff/cache   available
Mem:          7.8Gi       1.1Gi       264Mi        32Mi       6.4Gi       6.4Gi
Swap:         4.7Gi        46Mi       1.9Gi

    • zram

      • https://www.kernel.org/doc/html/v5.5/admin-guide/blockdev/zram.html

        • init.d (recommended)

          # install bc using apt or yum
sudo apt install bc; # for Debian derivative OS
sudo yum install bc; # for Redhat derivative OS

# Git clone zram-services to the servers /tmp/zram-services directory:
cd /tmp/
git clone https://github.com/eylles/zram-service.git
# Change directory into the cloned ./zram-service/ directory
cd zram-service/
# Checkout our trusted commit:
git checkout fa033ceebb297e01b3215dd32c9c0aea56a6bb8b
# Make absolutely sure the command `sha512sum zram.sh` returns the following checksum for zram.sh:
sha512sum zram.sh
6bf8c67834d375419e536c5e5c3070f0a1756f83f5bbcd7cc2d0f77b977f0df72536aef9025b8fc787870cf89bfa2a8c66317242f6d6bccfdba86fca70e663f6  zram.sh
# Move the zram.sh script to /etc/init.d/zram (notice file at /etc/init.d/zram has no .sh extension):
sudo mv zram.sh /etc/init.d/zram
# set root as owner and group of /etc/init.d/zram:
sudo chown root:root /etc/init.d/zram

# Cleanup as we are finished with our cloned zram-service
cd /tmp/
rm -rf zram-service

# Create the config file:
sudo echo -e 'ALGORITHM=lzo\n# 20% * expected compression ratio of 3 = 60%\nRAM_PERCENTAGE=60\nPRIORITY=100'|sudo tee /etc/default/zram-config
# Check the config settings
cat /etc/default/zram-config 
ALGORITHM=lzo
RAM_PERCENTAGE=20
PRIORITY=100

# Enable the zram init.d service:
# EITHER, for Debian derivative OS:
sudo update-rc.d zram defaults
# OR, for Redhat derivative OS:
sudo chkconfig --add zram

# Start the zram service:
sudo service zram start

# Check the zram service status
# - Note that DISKSIZE is the UNCOMPRESSED SIZE
# - At a compression ratio of 3.00 the actual memory used by zram for this device will be 1.6GB:
sudo zramctl
NAME       ALGORITHM DISKSIZE   DATA  COMPR  TOTAL STREAMS MOUNTPOINT
/dev/zram0 lzo           4.7G 360.9M 109.3M 133.3M       2 [SWAP]

# Lastly, establish the best available compression-algorithm from best to worst: zstd | lz4 | lzo-rle | lzo
# zstd is only supported by kernel 5.1 and newer:
cat /sys/block/zram0/comp_algorithm
[lzo] lzo-rle lz4 lz4hc 842 zstd
# zstd is the best and available - edit /etc/systemd/zram-generator.conf and set compression-algorithm=zstd
sudo nano /etc/default/zram-config

# After changing compression-algorithm=zstd in /etc/default/zram-config, restart the systemd service for device zram0 to update to using zstd:
sudo service zram restart

# and check the change has taken effect:
cat /sys/block/zram0/comp_algorithm
lzo lzo-rle lz4 lz4hc 842 [zstd]
# Confirm the device is in use:
sudo zramctl
NAME       ALGORITHM DISKSIZE   DATA  COMPR  TOTAL STREAMS MOUNTPOINT
/dev/zram0 zstd          4.7G 360.9M 109.3M 133.3M       2 [SWAP] 


sudo swapon --show;echo -e '\nzramctl:';sudo zramctl;echo -e '\nmm_stat ( https://www.kernel.org/doc/Documentation/blockdev/zram.txt ):';cat /sys/block/zram0/mm_stat;echo -e '\ncompression ratio';cat /sys/block/zram0/mm_stat|awk '{print $1/$2}'
NAME           TYPE      SIZE   USED PRIO
/dev/dm-1      partition 708M     0B   -1
/dev/zram0     partition 4.7G 498.4M  100
/data/swapfile file        4G     0B   -2

zramctl:
NAME       ALGORITHM DISKSIZE   DATA  COMPR  TOTAL STREAMS MOUNTPOINT
/dev/zram0 zstd          4.7G 360.9M 109.3M 133.3M       2 [SWAP]

mm_stat ( https://www.kernel.org/doc/Documentation/blockdev/zram.txt ):
378421248 114624368 139722752 1675169792 660004864    21330   242167

compression ratio
3.3014


# And finally check memory:
free -h
              total        used        free      shared  buff/cache   available
Mem:          7.8Gi       1.1Gi       264Mi        32Mi       6.4Gi       6.4Gi
Swap:         4.7Gi     499.4Mi       1.9Gi

# Check the available swap devices:
sudo swapon --show
NAME       TYPE      SIZE   USED PRIO
/dev/zram0 partition 4.7G     0B  100

        • systemd

          # as a non-root user install Rust and Cargo:
curl https://sh.rustup.rs -sSf | sh
source $HOME/.cargo/env

# as a non-root user navigate to an apppropriate directory and git clone zram-generator
cd ~/git_repos/
git clone https://github.com/systemd/zram-generator.git
cd zram-generator/

make build
# disregard the one error at completion - we haven't installed ronn:
/bin/sh: 1: ronn: not found
make: *** [Makefile:41: man] Error 127

sudo make install NOBUILD=true
# again disregard the one error at completion - we haven't installed ronn
install: cannot stat 'man/zram-generator.8': No such file or directory
make: *** [Makefile:65: install] Error 1

# create the config file with zram-size set at 20% of RAM
sudo echo -e '[zram0]\nhost-memory-limit=none\n# 20% * expected compression ratio of 3 = 60%\nzram-size=ram/1.666666667\ncompression-algorithm=lzo'|sudo tee /etc/systemd/zram-generator.conf
# check the config settings
cat /etc/systemd/zram-generator.conf
[zram0]
host-memory-limit=none
zram-size=ram/5
compression-algorithm=lzo

# Once installed and configured, the generator will be invoked by systemd early at boot, there is no need to do anything else:
#
# Start the systemd service - it defaults to enabled when installed:
sudo systemctl daemon-reload
sudo systemctl start /dev/zram0
# check service status
sudo systemctl status /dev/zram0

# confirm the device is in use
# - Note that DISKSIZE is the UNCOMPRESSED SIZE
# - At a compression ratio of 3.00 the actual memory used by zram for this device will be 1.6GB:
sudo zramctl
NAME       ALGORITHM DISKSIZE   DATA  COMPR  TOTAL STREAMS MOUNTPOINT
/dev/zram0 lzo           4.7G 360.9M 109.3M 133.3M       2 [SWAP]

# Lastly, establish the best available compression-algorithm from best to worst: zstd | lz4 | lzo-rle | lzo
# zstd is only supported by kernel 5.1 and newer:
cat /sys/block/zram0/comp_algorithm
[lzo] lzo-rle lz4 lz4hc 842 zstd
# zstd is the best and available - edit /etc/systemd/zram-generator.conf and set compression-algorithm=zstd
sudo nano /etc/systemd/zram-generator.conf

# After changing compression-algorithm=zstd in /etc/systemd/zram-generator.conf, to restart the systemd service for device zram0 to update to using zstd:
sudo systemctl restart systemd-zram-setup@zram0.service

# and check the change has taken effect:
cat /sys/block/zram0/comp_algorithm
lzo lzo-rle lz4 lz4hc 842 [zstd]
# confirm the device is in use:
sudo zramctl
NAME       ALGORITHM DISKSIZE   DATA  COMPR  TOTAL STREAMS MOUNTPOINT
/dev/zram0 zstd          4.7G 360.9M 109.3M 133.3M       2 [SWAP]

  
# Calculate the compression ratio for zram0 - it may take a short while to return the realistic expected compression ratio: sudo swapon --show;echo -e '\nzramctl:';sudo zramctl;echo -e '\nmm_stat ( https://www.kernel.org/doc/Documentation/blockdev/zram.txt ):';cat /sys/block/zram0/mm_stat;echo -e '\ncompression ratio';cat /sys/block/zram0/mm_stat|awk '{print $1/$2}'
NAME           TYPE      SIZE   USED PRIO
/dev/dm-1      partition 708M     0B   -1
/dev/zram0     partition 4.7G 360.9M  100
/data/swapfile file        4G     0B   -2
 
zramctl:
NAME       ALGORITHM DISKSIZE   DATA  COMPR  TOTAL STREAMS MOUNTPOINT
/dev/zram0 zstd          4.7G 360.9M 109.3M 133.3M       2 [SWAP]
 
mm_stat ( https://www.kernel.org/doc/Documentation/blockdev/zram.txt ):
378421248 114624368 139722752 1675169792 660004864    21330   242167
 
compression ratio
3.3014


# And finally check memory:
free -h
              total        used        free      shared  buff/cache   available
Mem:          7.8Gi       1.1Gi       264Mi        32Mi       6.4Gi       6.4Gi
Swap:         4.7Gi   360.9.1Mi       1.9Gi

# Check the available swap devices:
sudo swapon --show
NAME       TYPE      SIZE   USED PRIO
/dev/zram0 partition 4.7G     0B  100

    • zswap

      • First check whether the linux kernel supports zswap:

        grep -i zswap /boot/config-$(uname -r)

# CONFIG_ZSWAP is not set

        The output 'CONFIG_ZSWAP is not set' signals that the kernel does not support zswap.
        The output 'CONFIG_ZSWAP=y' signals that the kernel does support zswap.

      • https://www.kernel.org/doc/html/latest/vm/zswap.html provides that:
        • Zswap is a lightweight compressed cache for swap pages. It takes pages that are in the process of being swapped out and attempts to compress them
          into a dynamically allocated RAM-based memory pool. zswap basically trades CPU cycles for potentially reduced swap I/O.
          This trade-off can also result in a significant performance improvement if           reads from the compressed cache are faster than reads from a swap device.
        • Zswap is a new feature as of v3.11 and interacts heavily with memory reclaim. This interaction has not been fully explored on the large set of potential configurations and workloads that exist.
          For this reason, zswap is a work in progress and should be considered experimental.
        • Overcommitted guests that share a common I/O resource can dramatically reduce their swap I/O pressure, avoiding heavy handed I/O throttling by the hypervisor.
          This allows more work to get done with less impact to the guest workload and guests sharing the I/O subsystem.
        • Users with SSDs as swap devices can extend the life of the device by drastically reducing life-shortening writes.
      • Performance Analysis of Compressed Caching Technique
        • See the CONCLUSION of this paper for insights as to why zswap should not be used on a server with less than 1GB RAM.

      • For zswap to function correctly it needs swap space equivalent to the maximum uncompresse