Re: Re: Question about ever growing swap space on raspi 4 4 GB By: Jan Panteltje to All on Fri Jan 24 2020 08:00:58
JP> What is the point of using all memory for caching?
because it is faster... much much much faster than polling the drive all the time... drives, spinning rust or similar, are slow...
)\/(ark
Question about ever growing swap space on raspi 4 4 GB
I run buster on a raspberry pi4 with 4 GB RAM. I notice swap space is getting exhausted slowly.
Uptime: 07:37:54 up 12 days, 17:46, 17 users, load average: 0.35, 0.62, 0.67 Top shows this now: MiB Swap: 100.0 total, 20.7 free, 79.3 used. 3375.6 avail Mem
Here I found a way to see swap space per running process:
Running their script for file in /proc/*/status ; do awk '/VmSwap|Name/{printf $2 " " $3}END{ print ""}' $file; done | sort -k 2 -n -r | less
shows: Xorg 9136 kB rxvt 5648 kB fvwm 4256 kB rxvt 2548 kB rxvt 2388 kB rxvt 2196 kB rxvt 2080 kB rxvt 1796 kB rxvt 1764 kB rxvt 1764 kB rxvt 1636 kB (sd-pam) 1580 kB zsh 1144 kB zsh 1124 kB bash 892 kB (sd-pam) 752 kB systemd-udevd 652 kB systemd-journal 624 kB systemd 584 kB systemd 548 kB su 508 kB xterm 504 kB xfm 504 kB wpa_supplicant 496 kB ssh 492 kB rxvt 488 kB rxvt 484 kB rxvt 464 kB rxvt 464 kB rxvt 464 kB etc etc sorted to ever less
So it seems the system is using more and more little bits of swap space for each program!
The same on my lapop with even less memory shows zero swap for all programs!
The consequence would be that if I leave this raspi running (security camera recordings) it will soon run out of swap space.
It this an OS (buster) problem or something else? All that runs is some terminals and ffmpeg instances. The same on my laptop works without swap usage Any ideas what causes this swapping and / or how to stop it?
On a sunny day (Fri, 24 Jan 2020 06:52:42 GMT) it happened Jan Panteltje wrote in :
Follow up think I found the problem, top shows:
MiB Mem : 3906.0 total, 71.6 free, 375.9 used, 3458.6 buff/cache MiB Swap: 100.0 total, 19.9 free, 80.1 used. 3364.1 avail Mem
kernel claims all memory space as cache, that leaves virtually nothing (here only 71.6 MB) for programs those programs then start swapping!
BAD BAD BAD Linux error!!!!
Big thread about it here with possible solutions:
I have an old PC running this software with old Slackware kernel with maybe 1G of memory for 6 month 24/7 no problem.. That is where the old setup came from. But it had a much bigger swap partition, maybe never noticed this.
What is the point of using all memory for caching? Makes no sense to me. Is Linux DEAD?
You have that backwards.
From your 3906 MB, 375 MB is used, the kernel is keeping 71 MB free for immediate needs, that leaves 3458 MB that would be unused, so it temporarily gets used for caching, if the 71 MB is used up, the cache will be lowered and that memory become available for use ... no point in letting memory be unused when they system can benefit from using it as cache.
Exactly. Cache memory is essntially mirroring the disk(s). If it gets full the disk is activated and any writable data written to it. Read data is simply discarded.
Why the Pi is using swap is another matter entirely.
If it bothers you stop it, or configure a small ramdisk as swap.
-- I would rather have questions that cannot be answered... ...than to have answers that cannot be questioned Richard Feynman
On 2020-01-24, Jan Panteltje wrote: [...]
It makes perfect sense :) Other guys alredy answered why is that no problem so I will not repeat...
Depends :)
If using of swap for "inactive" processes bothers you, you can always change swapiness setting.
Change it with "sysctl vm.swappiness=VALUE where VALUE is some value lower than default (60).
If you want to have it persistent across reboots, put it in sysctl.conf.
Search for swappiness keyword to find detailed explanation about what is it and what you can expect with changing of that value.
-- obruT
Putting swap space in a ramdisk doesn't help because that is (obviously!) mapped into RAM, and so will merely reduce the RAM available for other purposes. If you want to use fixed swap space, rather than the default swap file, create a dedicated swap partition on spinning rust or an SSD, BUT I would not put one on an SD card if its at all frequently used. A reasonable size for fixed swap space is twice RAM size.
Use gparted or one of the older command-line tools (cfdisk, fdisk etc.) to manage device partitioning. Running "apropos partition" will show you which partition managers are installed and, as always, 'man' describes what each can do and how to use it. Swap partitions don't need to be formatted after creation.
-- Martin | martin at Gregorie | gregorie dot org
On a sunny day (Fri, 24 Jan 2020 11:39:33 -0000 (UTC)) it happened Martin Gregorie wrote in :
OK, thank everybody for the replies. So it is a swap file... did not even know that. Anyways it does not explain why all programs now seem to use swap space. I did the following as test / solution perhaps: Added # clear cache every 15 minutes to prevent swap space usage and swap partition overflow due to too little free memory left
0,15,30,45 * * * * /bin/echo 3 > /proc/sys/vm/drop_caches in /root/crtab so every 15 minutes the cache is emptied and more memory stays available: MiB Mem : 3906.0 total, 2688.8 free, 377.1 used, 840.1 buff/cache MiB Swap: 100.0 total, 19.7 free, 80.3 used. 3379.0 avail Mem Seem to have absolutely no effect on recording cameras, but at least 2 GB free memory is always available. Now the 19.7 free swap space has been constant for at least an hour.Then I started first firefox, swap space stayed the same. Then also started chrome browser, swapspace now increased to MiB Mem : 3906.0 total, 2890.2 free, 378.1 used, 637.7 buff/cache MiB Swap: 100.0 total, 20.2 free, 79.8 used. 3380.0 avail Mem
closing both firefox and chrome does NOT release the new swap space, it stays at 20.2 free, so it seems other apps were forced to use swap, 2 GB not enough ??? LOL Indeed it is then probably safer to add a swap partition with huge size.
But IMO this is a basic problem in Linux. An ever growing swap space usage is a killer in the long run, be it in a file or on a disk. ?
On a sunny day (Fri, 24 Jan 2020 09:23:25 +0000 (UTC)) it happened obruT wrote in :
OK, found this link
Yes, cool, On my system it is set to 60, but WHY THEN is it filling almost the whole memory with cache?
Anyways, I just did this: sysctl vm.swappiness=5
And repeated my system load experiment by starting both firefox and chrome at the same time.. Now the swap size in not increasing. I am inclined to set the swappiness to zero, but if this works it is probably safer, Will add it to the startup script. Many many thanks!!
Because cache is NOT SWAP Linux allocates anything it has spare to cache.
Every time you read the disk it goes into the cache. If it doesn't get accessed for a while it gets overwritten by some other disk stuff. If you write to disk it goes into cache. Only if it runs out of cache will it actually write to the disk.
--
"The most difficult subjects can be explained to the most slow witted
man if he has not formed any idea of them already; but the simplest
thing cannot be made clear to the most intelligent man if he is firmly
persuaded that he knows already, without a shadow of doubt, what is laid
before him."
- Leo Tolstoy
I'm pretty sure you'll find that it usually gets written to disk before that, only when a dirty block has been written out can it be reused so flushing dirty blocks is an important activity.
-- Steve O'Hara-Smith | Directable Mirror Arrays C:\>WIN | A better way to focus the sun The computer obeys and wins. | licences available see You lose and Bill collects. | http://www.sohara.org/
I don't see ever growing swap. You have 3.38GB of swap you could be using. You have 100MB of swap actually enabled, of which ~80MB is used and 20MB is free. That's approximately nothing.
If your swap is a partition, it's a fixed size and can't grow. So if you exceed swap the kernel will just kill a process to free up space. The same thing will happen if you don't enable swap.
If you want a system with more consistent performance, you can disable swap. Then everything either fits in RAM or the system kills it. Then you don't end up in a disc thrash death spiral.
Theo
You can run without swap. The command swapoff disables swapping and swapon (re)enables it. It can also be permanently disabled by editing the boot parameters / scripts/ services (depending on which Linux you're using.
I think you may be misreading the 'top' display. There are two lines that show this type of information:
MiB Mem : 7647.2 total, 4483.4 free, 1206.3 used, 1957.5 buff/cache MiB Swap: 16500.0 total, 16500.0 free, 0.0 used. 6325.5 avail Mem
The above is from this Lenovo T440, which has 8GB RAM and a 16GB swap partition on its hard drive.
You can also see just these two lines by running 'free -h'.
The first of these lines describes how RAM is being used: 'buff/cache' is the amount of RAM occupied by file buffering and other data that isn't owned by any specific process.
The second line describes swap space: its used space is non-zero only when a process has caused RAM to be oversubscribed. The kernel will first try to free up RAM by discarding any file blocks that have only been read. If this doesn't release enough RAM then disk blocks that are full but not yer written out and written out and released. If more RAM is stil needed, then excess code and heap data in the requesting process is written to swapspace and is subsequently paged in/written out as needed. Swap space will be released when that process terminates.
Note the order: Fast operations are always done first, followed by slower ones of the previous operations didn't make enough RAM available to the requesting process.
Note also, that small files (both code images and data) will tend to end up entirely on buffer space and will stay there even after the requesting process has ended - until something needs to use the RAM they're occupying. You can see this happening on a system with enough memory: its why, if you're developing a reasonably large C (or Java) application, the first compile of the day is several times slower than the rest: for subsequent compilations your editor, 'make', the Makefile, the C compiler and the source files you're working on are already buffered in memory and don't need to be re-read from disk.
Nope it isn't because what you thought was swap space isn't, and I've never seen properly configured swapspace (i.e. RAM * 2) get filled.
The nearest I've seen to this is when a process exceeds its configured maximum virtual memory size. If this happens it will be killed by the kernel. The virtual memory size of a process is the total of RAM used plus swap space used. This is a limit enforced by the kernel - it can happen to any program that can grow to accommodate arbitrary amounts of data: I've only seen this happen to a Java 1.4 application of mine running on a 32 bit system. The cause was the garbage collector failing to clear released heap-space as fast as the program was grabbing new heap- space and loading more data into it. That has never been an issue since Java 1.6 and its improved garbage collector were released.
-- Martin | martin at Gregorie | gregorie dot org
Because otherwise the memory would be wasted. You paid for the silicon, you?re paying for the electricity that powers it, using it store idle pages is not a good use of those resources. You are attacking a non-problem.
-- https://www.greenend.org.uk/rjk/
Linux tries to make use of all free ram for buffers but ONLY if it is not needed by other applications. it should free the buffers before swapping only when it cannot recover enough ram,
for a PI running from a SD card it is better to switch swap off completely
buffer-cache-in-linux
-- The biggest problem with communication is the illusion that it has occurred.
On Fri, 24 Jan 2020 08:00:58 GMT, Jan Panteltje declaimed the following:
While the OS has claimed 3.4GB as cache, note the line below reports
3.3GB as available -- meaning the OS can release that much of the cache for use by applications. After all, the R-Pi doesn't have 6.7+GB of memory.
Note that the first entry on that link complains about how long it takes for the cache to be freed, thereby delaying the startup of fresh applications. It does not implicate a situation with page swapping starting up. The first "answer" refers to /file system buffers/ -- eg; output that has not yet been forced to the file system (SD card); and with a journaling file system, file system updates take multiple stages (things like: allocate fresh blocks and write data; write journal log indicating where the new data is; later update file system so log contents are now active -- read journal/write primary file system, remove relevant journal log information).
I really doubt that the cache usage part of the kernel has been changed recently in a way that it does not free pages when new applications are started.
In your original process list, you show FOURTEEN rxvt (X-Window terminal) processes, along with SSH and ONE xterm. My suspicion is that something you are doing is invoking terminals, but never closing them. Surely you aren't really running 14+ terminals at once! Each terminal likely having a few pages dedicated to the "screen contents" of the terminal (the main code is no doubt shared, or you'd have even more memory tagged "used").
-- Wulfraed Dennis Lee Bieber AF6VN wlfraed@ix.netcom.com http://wlfraed.microdiversity.freeddns.org/
On a sunny day (Fri, 24 Jan 2020 11:19:03 -0500) it happened Dennis Lee Bieber wrote in :
Yes, a number of scripts (starting with your last remark) are running, below is one that checks for a flag and then starts a new recording of a camera: flag is set from crontab ever few hours so I get nice bacth of recordings like this
-rw-r--r-- 1 root root 153587406 Jan 24 19:00 camera6-270.wav
-rw-r--r-- 1 root root 943014702 Jan 24 19:00 camera6-270.avi
-rw-r--r-- 1 root root 1081724392 Jan 24 19:00 camera4-275.avi
-rw-r--r-- 1 root root 5560318832 Jan 25 02:00 hcam_3_504.ts
-rw-r--r-- 1 root root 5608550044 Jan 25 02:00 hcam_4_335.ts
-rw-r--r-- 1 root root 5605979896 Jan 25 02:00 hcam_2_434.ts
-rw-r--r-- 1 root root 5607202836 Jan 25 02:00 hcam_1_487.ts
-rw-r--r-- 1 root root 1080950850 Jan 25 07:00 camera4-276.avi
-rw-r--r-- 1 root root 176570366 Jan 25 07:00 camera6-271.wav
-rw-r--r-- 1 root root 1081851124 Jan 25 07:00 camera6-271.avi
-rw-r--r-- 1 root root 1328002836 Jan 25 07:53 bp65.avi
-rw-r--r-- 1 root root 79959598 Jan 25 07:53 camera6-272.avi
-rw-r--r-- 1 root root 13107294 Jan 25 07:53 camera6-272.wav
-rw-r--r-- 1 root root 80483886 Jan 25 07:53 camera4-277.avi
-rw-r--r-- 1 root root 4062141192 Jan 25 07:53 hcam_3_505.ts
-rw-r--r-- 1 root root 4135147984 Jan 25 07:53 hcam_2_435.ts
-rw-r--r-- 1 root root 577800268 Jan 25 07:53 e2500_8.mkv
-rw-r--r-- 1 root root 4138018368 Jan 25 07:53 hcam_1_488.ts
-rw-r--r-- 1 root root 4138335524 Jan 25 07:53 hcam_4_336.ts
Example of such a script: raspi95: /usr/local/sbin # cat test-update-camera6 #!/bin/bash while true do
if test -f /video/camera6-update-flag then killall -KILL wget2 killall -KILL mcamip2
serial_number=/bin/cat /video/camera6_serial.txt #/bin/echo "serial_number=$serial_number"
wget2 --user=panteltje --password=1234567 http://192.168.178.160:84/videostream.asf -O - 2>/dev/zero \ | \ ffmpeg -f asf -i - -acodec copy -y /mnt/sda2/security/video/camera6-$serial_number.wav \ 1>/dev/zero 2>/dev/zero &
mcamip2 -a 192.168.178.160 -p 84 -u panteltje -w 1234567 -x -t -f 2 -q -m 1 -y 2>/dev/zero \ | \ ffmpeg -f yuv4mpegpipe -i - -vcodec mjpeg -an -y /mnt/sda2/security/video/camera6-$serial_number.avi \ 1>/dev/zero 2>/dev/zero &
let old_serial_number=serial_number-6
# This file may still be there if reboot. let older_serial_number=serial_number-7
#/bin/echo "updating serial number"
let serial_number=serial_number+1 /bin/echo $serial_number > /video/camera6_serial.txt
#/bin/echo "removing old file /video/camera6-$old_serial_number.*"
rm /mnt/sda2/security/video/camera6-$old_serial_number.avi rm /mnt/sda2/security/video/camera6-$old_serial_number.wav
#/bin/echo "removing older file /video/camera6-$older_serial_number.*"
rm /video/camera6-$older_serial_number.avi 2>/dev/zero rm /video/camera6-$older_serial_number.wav 2>/dev/zero
rm /video/camera6-update-flag fi
sleep 54 done
There are 9 terminals running, and in each terminal a zsh shell is running Those terminals are always on (9 virtual desktops, xfm, fvwm so not the usual interface, that same configuration runs (apart that there it is native zsh) on all my computahs.
As to 'does it help etc?' I am monitoring now the swap file it seems not to be getting bigger with the modifications I mentioned in the other posting.
Fri 24 Jan 2020 03:10:42 PM CET: MiB Mem : 3906.0 total, 2891.6 free, 378.2 used, 636.2 buff/cache MiB Swap: 100.0 total, 20.4 free, 79.6 used. 3378.8 avail Mem
Sat 25 Jan 2020 06:46:05 AM CET MiB Mem : 3906.0 total, 2653.7 free, 372.0 used, 880.4 buff/cache MiB Swap: 100.0 total, 24.7 free, 75.3 used. 3380.4 avail Mem
so that is good!
It will (hopefully) keep running for a long time (weeks etc is on an UPS) and will check every day what now happens to the swap file. Normally it runs via ssh, but I confess I have used it with HDMI monitor to do all sort of other things such as web browsing, watching youtube videos, recording satellite TV, etc, Raspi 4 is a very powerful thing, can indeed for many applications replace a desktop PC, have a 3.4 TB USB harddisk connected Filesystem 1K-blocks Used Available Use% Mounted on /dev/root 29771644 14374360 14106048 51% / devtmpfs 1867796 0 1867796 0% /dev tmpfs 1999892 0 1999892 0% /dev/shm tmpfs 1999892 33484 1966408 2% /run tmpfs 5120 8 5112 1% /run/lock tmpfs 1999892 0 1999892 0% /sys/fs/cgroup /dev/mmcblk0p1 258095 53032 205064 21% /boot tmpfs 399976 0 399976 0% /run/user/1000 /dev/sda2 3844510712 2643749788 1005400260 73% /mnt/sda2 tmpfs 399976 0 399976 0% /run/user/0
We will see where it goes :-)
On a sunny day (Fri, 24 Jan 2020 14:48:41 -0000 (UTC)) it happened Martin Gregorie wrote in :
OK, understood, I will keep monitoring the system free -h is nice too.
On a sunny day (Fri, 24 Jan 2020 14:54:44 +0000) it happened Richard Kettlewell wrote in :
I disagree, I did see the swap space increasing day after day and that set of an alarm with me (normally I do a quick view on 'xosview'). Also the same stuff running on a normal PC did not cause lots of unrelated things to get swapped. One of the links I gave shows there was a recent modification in the way the kernel handles caching. From:
Would not be the first kernel bug I encountered.... ;-)
Will see ...
It doesn?t matter.
What would matter is the system swqpping lots of things back in; that would impact performance. But you?ve not measured that, as far as I can tell. Nothing you?ve yet posted is inconsistent with the normal eviction of idle pages from RAM.
2.6 is not ?recent?, it was released in 2003.-- https://www.greenend.org.uk/rjk/
ElectronDepot website is not affiliated with any of the manufacturers or service providers discussed here. All logos and trade names are the property of their respective owners.