Table of Contents |
---|
...
Warning |
---|
Exercise caution while editing /usr/local/omk/conf/opCommon.nmis, /usr/local/omk/conf/opCommon.json or /etc/mongod.conf; if a syntax error is induced all OMK applications will cease to function. Check out How to make configuration changes to opCommon.nmis and other files for some more details on best practices for modifying the configuration file. |
Reducing a Server's Memory Footprint
...
- In omk/conf/opCommon.json:
- omkd_workers is set at 10 by default. Try reducing this number to 2 and then restart the omkd service.
If you THEN find YOU need more omkd_workers, increment this value by one and test again until you get to a suitable value for omkd_workers - /omkd/omkd_max_requests: 100 to 500 (start at 100 and increase from this value if needed)
- omkd_workers is set at 10 by default. Try reducing this number to 2 and then restart the omkd service.
- in nmis9/conf/Config.nmis:
- /system/nmisd_worker_max_cycles: 100
- /system/nmisd_worker_max_cycles: 100
- In omk/conf/opCommon.json:
Consider using zram or zswap to extend available memory, with default settings, provided the server does not have high CPU usage.
An advantage of zram is that it does not need to be backed by an appropriately sized swap space.
Another advantage of zram is that it is more likely to function in low memory environments whereas zswap should not be used when the server has less than 1GB RAM.
A disadvantage of zram is that the server cannot hibernate when using zram - which is highly unlikely to be a requirement.
Another disadvantage is that it requires a few installs to get it running.
zram
https://www.kernel.org/doc/html/v5.5/admin-guide/blockdev/zram.html
Code Block # as a non-root user install Rust and Cargo: curl https://sh.rustup.rs -sSf | sh source $HOME/.cargo/env # as a non-root user navigate to an apppropriate directory and git clone zram-generator cd ~/git_repos/ git clone https://github.com/systemd/zram-generator.git cd zram-generator/ make build # disregard the one error at completion - we haven't installed ronn: /bin/sh: 1: ronn: not found make: *** [Makefile:41: man] Error 127 sudo make install NOBUILD=true # again disregard the one error at completion - we haven't installed ronn install: cannot stat 'man/zram-generator.8': No such file or directory make: *** [Makefile:65: install] Error 1 # create the config file with zram-size set at 20% of RAM sudo echo -e '[zram0]\nhost-memory-limit=none\nzram-size=ram/5\ncompression-algorithm=zstd'|sudo tee /etc/systemd/zram-generator.conf # check the config settings cat /etc/systemd/zram-generator.conf [zram0] host-memory-limit=none zram-size=ram/5 compression-algorithm=lz4lzo-rle # Once installed and configured, the generator will be invoked by systemd early at boot, there is no need to do anything else: # # Start the systemd service - it defaults to enabled when installed: sudo systemctl daemon-reload sudo systemctl start /dev/zram0 # check status sudo systemctl status /dev/zram0 # confirm the device is in use: sudo zramctl NAME ALGORITHM DISKSIZE DATA COMPR TOTAL STREAMS MOUNTPOINT /dev/zram0 lz4lzo-rle 1.6G 46.1M 14.7M 15.7M 2 [SWAP] # In the above output we can calculate compression ratio from: DATA/COMPR = 46.1M/.14.7M = 3.136054422 # Lastly, establish the best available compression-algorithm from best to worst: zstd | lz4 | lzo-rle | lzo cat /sys/block/zram0/comp_algorithm lzo [lzo-rle] lz4 lz4hc 842 zstd # zstd is the best and available - edit /etc/systemd/zram-generator.conf and set compression-algorithm=zstd nano /etc/systemd/zram-generator.conf # restart the systemd service for device zram0 to update to using zstd: sudo systemctl restart systemd-zram-setup@zram0.service # and check the change has taken effect: cat /sys/block/zram0/comp_algorithm lzo lzo-rle lz4 lz4hc 842 [zstd] # confirm the device is in use: sudo zramctl NAME ALGORITHM DISKSIZE DATA COMPR TOTAL STREAMS MOUNTPOINT /dev/zram0 zstd 1.6G 46.1M 13.1M 15.7M 2 [SWAP]
zswap
First check whether the linux kernel supports zswap:
Code Block grep -i zswap /boot/config-$(uname -r) # CONFIG_ZSWAP is not set
The output 'CONFIG_ZSWAP is not set' signals that the kernel does not support zswap.
The output 'CONFIG_ZSWAP=y' signals that the kernel does support zswap.- https://www.kernel.org/doc/html/latest/vm/zswap.html provides that:
- Zswap is a lightweight compressed cache for swap pages. It takes pages that are in the process of being swapped out and attempts to compress them
into a dynamically allocated RAM-based memory pool. zswap basically trades CPU cycles for potentially reduced swap I/O.
This trade-off can also result in a significant performance improvement if reads from the compressed cache are faster than reads from a swap device. - Zswap is a new feature as of v3.11 and interacts heavily with memory reclaim. This interaction has not been fully explored on the large set of potential configurations and workloads that exist.
For this reason, zswap is a work in progress and should be considered experimental. - Overcommitted guests that share a common I/O resource can dramatically reduce their swap I/O pressure, avoiding heavy handed I/O throttling by the hypervisor.
This allows more work to get done with less impact to the guest workload and guests sharing the I/O subsystem. - Users with SSDs as swap devices can extend the life of the device by drastically reducing life-shortening writes.
- Zswap is a lightweight compressed cache for swap pages. It takes pages that are in the process of being swapped out and attempts to compress them
- Performance Analysis of Compressed Caching Technique
- See the CONCLUSION of this paper for insights as to why zswap should not be used on a server with less than 1GB RAM.
- See the CONCLUSION of this paper for insights as to why zswap should not be used on a server with less than 1GB RAM.
- For zswap to function correctly it needs swap space equivalent to the maximum uncompressed RAM zswap may contain in its pool
- To cater for a zswap compression ratio of 4.0, with the default 20% zswap pool (1/5th),
one would provision swap space in a server with 10GB RAM installed of 2 GB zswap pool * (4.0-1) = 6.0 GB RAM: Server with 10GB RAM + at least 6.0 GB swap space should then be provisioned - One can expect at least a zswap compression ratio of 3.1, so when enabling zswap on a server with 10GB RAM installed, one should provision a minimum of 4.2 GB swap space.
- To cater for a zswap compression ratio of 4.0, with the default 20% zswap pool (1/5th),
- Don't be tempted to increase maximum pool percent from the default setting of 20: this will most likely affect performance adversely.
- Command to view zswap info during operation when enabled:
- sudo grep -rRn . /sys/kernel/debug/zswap/
may require mounting debugfs :
sudo mount -t debugfs none /sys/kernel/debug
- sudo grep -rRn . /sys/kernel/debug/zswap/
https://www.ibm.com/support/pages/new-linux-zswap-compression-functionality provides that for a server with 10GB RAM and 20% zswap pool size:
- For the x86 runs, the pool limit was hit earlier - starting at the 15.5 GB data point
- On x86, the average zswap compression ratio was 3.6
- Average zswap compression ratio of 3.6 ties in with the 15.5 GB data point a which real disk swap started as follows
(as given in https://www.ibm.com/support/pages/new-linux-zswap-compression-functionality):- 10 GB RAM with 2 GB zswap pool =
- 8 GB available RAM + 2 GB zswap pool =
- 8 GB available RAM + (2 GB zswap pool * 3.6 average zswap compression ratio) =
- 8 GB available RAM + 7.2GB zswap compressed RAM =
- 15.2 GB RAM
- 15.2 GB RAM
- 8 GB available RAM + 7.2GB zswap compressed RAM =
- 8 GB available RAM + (2 GB zswap pool * 3.6 average zswap compression ratio) =
- 8 GB available RAM + 2 GB zswap pool =
- 10 GB RAM with 2 GB zswap pool =
Real world example 2021-12-22:
- The following code was placed in /etc/rc.local to use the best zswap options available,
as improvements have been made to zswap compression since this article was originally writtenCode Block title /etc/rc.local #!/bin/sh -e # # rc.local # # This script is executed at the end of each multiuser runlevel. # Make sure that the script will "exit 0" on success or any other # value on error. # # In order to enable or disable this script just change the execution # bits. # # By default this script does nothing. # enable zswap: echo 20 > /sys/module/zswap/parameters/max_pool_percent||:; echo 1 > /sys/module/zswap/parameters/enabled||:; # https://linuxreviews.org/Zswap sh -c 'CMP=/sys/module/zswap/parameters/compressor;echo zstd >"${CMP}"||echo lz4 >"${CMP}"||echo lzo-rle >"${CMP}"||:;' 2>/dev/null # use the best zpool compressed memory pool allocator available: sh -c 'ZP=/sys/module/zswap/parameters/zpool;echo z3fold >"${ZP}"||echo zsmalloc >"${ZP}"||:;' 2>/dev/null # enable zswap debug info mount -t debugfs none /sys/kernel/debug 2>/dev/null||:; exit 0;
On Debian 9.13 the following zswap parameters were set from the above /etc/rc.local script
Code Block sudo -i chmod +x /etc/rc.local source /etc/rc.local grep . /sys/module/zswap/parameters/* /sys/module/zswap/parameters/compressor:lz4 /sys/module/zswap/parameters/enabled:Y /sys/module/zswap/parameters/max_pool_percent:20 /sys/module/zswap/parameters/zpool:zsmalloc
When this server was swapping the following zswap stats were achieved:
Code Block free -h total used free shared buff/cache available Mem: 2.0G 1.9G 58M 728K 27M 17M Swap: 2.7G 1.4G 1.4G sudo sh -c 'echo;grep . /sys/module/zswap/parameters/*;echo;D=/sys/kernel/debug/zswap;grep . "${D}/"*;perl -E "say \"\nCompress Ratio: \".$(cat "${D}/stored_pages")*4096/$(cat "${D}/pool_total_size")" 2>/dev/null' /sys/module/zswap/parameters/compressor:lz4 /sys/module/zswap/parameters/enabled:Y /sys/module/zswap/parameters/max_pool_percent:20 /sys/module/zswap/parameters/zpool:zsmalloc /sys/kernel/debug/zswap/duplicate_entry:0 /sys/kernel/debug/zswap/pool_limit_hit:565978 /sys/kernel/debug/zswap/pool_total_size:287293440 /sys/kernel/debug/zswap/reject_alloc_fail:593 /sys/kernel/debug/zswap/reject_compress_poor:436 /sys/kernel/debug/zswap/reject_kmemcache_fail:0 /sys/kernel/debug/zswap/reject_reclaim_fail:1596 /sys/kernel/debug/zswap/stored_pages:360385 /sys/kernel/debug/zswap/written_back_pages:1145522 Compress Ratio: 5.1370556204920
- How to interpret the above stats:
- 'free -h' command provides that 1.4G of swap was used.
That stat refers to the uncompressed swap:
'/sys/kernel/debug/zswap/stored_pages * 4096' = '360385 * 4096' = 1476136960 bytes = 1.37476 GB - The 1.37476 GB of swap was compressed to '/sys/kernel/debug/zswap/pool_total_size' = 287293440 bytes = 0.26756 GB of RAM.
- Compression Ratio is therefore '1.37476 GB/0.26756 GB' = 5.13706
- Note: A compression ratio > 5.00 is unusually high, long term:
- when zpool is zbud, the typical compression ratio will be 1.70;
- when zpool is z3fold, the typical compression ratio will be 2.70;
when zpool is zsmalloc, the typical compression ratio will be 3.00,
as in the following example where swap did not hit the disk,
even though 9GB memory was allocated when the server had only 7.8GB memory available:Code Block free -h total used free shared buff/cache available Mem: 7.8G 7.5G 245M 20K 73M 148M Swap: 7.7G 1.5G 6.2G sudo sh -c 'echo;grep . /sys/module/zswap/parameters/*;echo;D=/sys/kernel/debug/zswap;grep . "${D}/"*;perl -E "say \"\nCompress Ratio: \".$(cat "${D}/stored_pages")*4096/$(cat "${D}/pool_total_size")" 2>/dev/null' /sys/module/zswap/parameters/compressor:lz4 /sys/module/zswap/parameters/enabled:Y /sys/module/zswap/parameters/max_pool_percent:20 /sys/module/zswap/parameters/zpool:zsmalloc /sys/kernel/debug/zswap/duplicate_entry:0 /sys/kernel/debug/zswap/pool_limit_hit:0 /sys/kernel/debug/zswap/pool_total_size:387317760 /sys/kernel/debug/zswap/reject_alloc_fail:1507 /sys/kernel/debug/zswap/reject_compress_poor:0 /sys/kernel/debug/zswap/reject_kmemcache_fail:0 /sys/kernel/debug/zswap/reject_reclaim_fail:0 /sys/kernel/debug/zswap/stored_pages:291753 /sys/kernel/debug/zswap/written_back_pages:0 Compress Ratio: 3.08537436548223
- Note: A compression ratio > 5.00 is unusually high, long term:
- 'free -h' command provides that 1.4G of swap was used.
- With regards to the VM, with Compress Ratio: 5.1370556204920 example further above,
swapsize was set knowing the compression ratios zswap achieved on this VM.
To allow zswap to achieve a compression ratio of 5.13706,
zswap needs swap space equivalent to the maximum uncompressed RAM zswap may contain in its pool when compressing at compression ratio of 5.13706.
Consequently, the minimum swap space this VM should then have is:- '2 GB RAM * /sys/module/zswap/parameters/max_pool_percent * Compression Ratio'
= '2 GB RAM * 20% * 5.13706'
= 2.055 GB swap
- '2 GB RAM * /sys/module/zswap/parameters/max_pool_percent * Compression Ratio'
- However, the reason why the swap on this VM was actually set at 2.75 GB is that the command
watch free -h
showed that this VMs' zswap compression ratio was fluctuating between 4.8 and 6.1
Consequently the decision was made to set
'2GB RAM * 20% * 6.875'
= 2.75 GB swap on this VM. Use a swapfile for swap as this provides better flexibility in sizing swap than a swap partition.
To resize an existing /swapfile to 2.75 GB (2883584 kilobytes):Code Block SWAPFILE=/swapfile; sudo swapoff "${SWAPFILE:-FAIL_HERE_!!!}"; # count in kilobytes sudo dd if=/dev/zero of="${SWAPFILE:-FAIL_HERE_!!!}" bs=1024 count=2883584; sudo chmod 0600 "${SWAPFILE:-FAIL_HERE_!!!}"; sudo mkswap "${SWAPFILE:-FAIL_HERE_!!!}"; sudo swapon "${SWAPFILE:-FAIL_HERE_!!!}"; sudo swapon --show; NAME TYPE SIZE USED PRIO /swapfile file 12G 512K -2
To make a new /swapfile persist after a reboot append this entry to /etc/fstab:
Code Block /swapfile swap swap defaults 0 0
- The following code was placed in /etc/rc.local to use the best zswap options available,
...