Question about ever growing swap space on raspi 4 4 GB

I was trying to keep it simple.
But if you examine the cache with top you will see its always full...which either means top is reporting 'reserved for cache' or it is actually full.
Anywsy the point I was trying to make was to someone who didn't know the difference between cache and swap.
--
I would rather have questions that cannot be answered... 
...than to have answers that cannot be questioned 
 Click to see the full signature
Reply to
The Natural Philosopher
Loading thread data ...
When a dirty page is flushed to disc it stays in the cache, just marked clean so it can be reused if needs be, if it's something like a page out of a database then it will likely get written to again before it gets reused for something else.
I just like accuracy :)
--
Steve O'Hara-Smith                          |   Directable Mirror Arrays 
C:\>WIN                                     | A better way to focus the sun 
 Click to see the full signature
Reply to
Ahem A Rivet's Shot
Is writing a page to swap the same as evicting it from RAM? Naively I would have thought it would just mark the page as ready for eviction, i.e. eviction from RAM would only occur after some further kind of ageing process and/or more memory pressure.
Apologies if my terminology isn't quite correct, I did used to know the basics of virtual memory for VMS but that was some decades ago. I don't think I've ever read a good description of how linux (or unix) handles it. Can anyone recommend a good technical overview of the subject, if such a thing exists.
FWIW I would worry about a slowly growing swap file as being symptomatic of a memory leak, but that may be due to my lack of understanding.
Reply to
Pancho
On Sat, 25 Jan 2020 18:05:05 +0000, Pancho declaimed the following:
Swap is typically only used when an application is requesting more memory than is physically available (that may not be the same as physically installed -- if you have a 4GB address range, but only 2GB of RAM, then only a 2GB subset of the space can be loaded at any time, the rest is swapped out; the MMU is responsible for tracking what addresses are loaded). If the system has cached pages it can free (especially pages that are only read-mode, not written to) it is supposed to free those cache pages and make them available to the application requesting memory. Dirty cache pages will need to be written out -- but those typically belong to some file, not swap.
Note that on decent systems, shared libraries (which are normally read-only code) shouldn't end up in swap -- the library file itself should be mapped, so unused pages can just be freed; when that section of code is required again, it can be loaded from the library file itself.
Other than reading kernel sources? Maybe
formatting link
as an introduction. And a decade old...
formatting link

--
	Wulfraed                 Dennis Lee Bieber         AF6VN 
	wlfraed@ix.netcom.com    http://wlfraed.microdiversity.freeddns.org/
Reply to
Dennis Lee Bieber
The FreeBSD memory management system is well described here:
formatting link

I gather it is similar in principle, but different in detail, to the Linux mechanism.
--
Steve O'Hara-Smith                          |   Directable Mirror Arrays 
C:\>WIN                                     | A better way to focus the sun 
 Click to see the full signature
Reply to
Ahem A Rivet's Shot
Also, these days it's common to compress pages in RAM in preference to swapping to storage - a RAM page is faster to decompress than fetching it from disc. Swapping to compressed RAM doesn't relieve memory pressure as much as disc (since you still need the compressed pages in RAM) but it can be a lot quicker than devices like eMMC or spinning disc.
Android, macOS, Chrome OS, and iOS all use it - in particular since swapping on a mobile device isn't great for power and performance reasons. You can enable it on your Pi too if you want:
formatting link

Theo
Reply to
Theo
On a sunny day (Sat, 25 Jan 2020 09:00:04 +0000) it happened Richard Kettlewell wrote in :
If the swap file is full then what will happen?
OK, There was more I found online however on that subject, changes in 4.?
Anyways, now it _decreases_ swap: Fri 24 Jan 2020 03:10:42 PM CET: MiB Mem : 3906.0 total, 2891.6 free, 378.2 used, 636.2 buff/cache MiB Swap: 100.0 total, 20.4 free, 79.6 used. 3378.8 avail Mem
Sat 25 Jan 2020 06:46:05 AM CET Sat Jan 25 06:44:19 CET 2020 MiB Mem : 3906.0 total, 2653.7 free, 372.0 used, 880.4 buff/cache MiB Swap: 100.0 total, 24.7 free, 75.3 used. 3380.4 avail Mem
Sun 26 Jan 2020 07:32:08 AM CET MiB Mem : 3906.0 total, 3265.9 free, 370.6 used, 269.5 buff/cache MiB Swap: 100.0 total, 26.9 free, 73.1 used. 3383.9 avail Mem
Seems a lot safer to me.
I was also wondering if a full swap space could explain mysterious crashes some people had with their SDcards?
Here is an other old model raspi with far less memory and older kernel running 24/7:
root@raspi73:~# uptime 07:50:10 up 14 days, 17:48, 11 users, load average: 2.88, 3.00, 3.10
loaded to the maximum, top: KiB Mem: 448776 total, 426296 used, 22480 free, 150392 buffers KiB Swap: 102396 total, 0 used, 102396 free, 61336 cached
root@raspi73:~# uname -a Linux raspi73 3.6.11+ #371 PREEMPT Thu Feb 7 16:31:35 GMT 2013 armv6l GNU/Linux
No swapping whatsoever.
?
Reply to
Jan Panteltje
A fair point. I don?t know how those pages (i.e. written to swap but still available in RAM) are accounted for in the figures we get to see.
It could be, but (at the levels shown here) I don?t think there?s any evidence of that.
--
https://www.greenend.org.uk/rjk/
Reply to
Richard Kettlewell
In practice the system will appear to hang for a while, possibly a long while, until applications which require more memory either exit cleanly or crash, freeing up some memory and/or swap.
---druck
Reply to
druck
Don't do this, it is pointless interfering with the cache system, which will reduce system performance and wont affect swap usage in any way. The kernel will always reduce the cache as far as possible and give the memory to applications, before it is forced to do any swapping.
Any pages of other applications which were forced in to swap will stay there until those pages are needed again, and which point they will be loaded back in to RAM and the swap space freed. They kernel wont flush out the swap unless you turn if off, as there is no benefit.
Over time unused parts of long running applications will end up in swap, and stay there. It's not a problem unless the system is under intense memory demand, and it has used both all the memory, and completely filled the swap file.
As you keep talking about how much cache is present, the system is obviously not under much memory demand, and you can safely ignore some minor usage of the swap file.
---druck
Reply to
druck
-=> The Natural Philosopher wrote to Andy Burns > Er no. calls to malloc() will fail and serious errors in applications
TNP> thats a serious error
That reminds me of the old joke in the Jargon File about the computer crashing because it swapped out the swapper.
... Eval Day 1005
Reply to
Kurt Weiske
Er no. calls to malloc() will fail and serious errors in applications will happen.
--
Ideas are more powerful than guns. We would not let our enemies have  
guns, why should we let them have ideas? 
 Click to see the full signature
Reply to
The Natural Philosopher
Or the OOMK will "off" a greedy process or two :-(
Reply to
Andy Burns
When calls to malloc fail, a well written application will handle the error and exit cleanly, a badly written one will ignore the failure and crash - as I just said.
There is a third option, which is actually worse - the application handles the error and continues to run. This will cause the system to run at a glacial speed as it swaps in and out each application, hoping that one of them does eventually exit or crash - hence the appearing to hang bit above.
---druck
Reply to
druck
I don't recall how big the swap file is when its initially created, mainly because I use swap partitions on all my systems except the RPi. You set the swap partition size when you create it and its size doesn't change unless you replace it with a different sized swap partition.
However, I think you'll find that the swap file is expanded if the kernel runs out of space in it but it is never automatically reduced in size, because the kernel can't know whether the workload that caused the increase was a one-off or just the first instance of the system's normal workload.
--
Martin    | martin at 
Gregorie  | gregorie dot org
Reply to
Martin Gregorie
On Mon, 27 Jan 2020 20:04:51 -0000 (UTC), Martin Gregorie declaimed the following:
Everything I've found in Google implies that one first allocates a file of known size (cf:
formatting link
) which sort of indicates that it does not grow dynamically.
That sounds more like Windows behavior, where swap files are given an initial size, and a maximum size (I think I normally make Windows swap a fixed size since the OS can allocate a contiguous chunk of disk, rather than risk a "split" swap).
And that brings up the biggest problem with a Linux swap file -- it is likely not going to be contiguous even on disk AND if on an SD card (or other flash media -- even an SSD can be killed by a system that is swapping heavily)... Flash memory behavior means that it is likely every page swap to SD card will trigger a new allocation unit erase/write cycle and wear leveling means nothing is contiguous.
--
	Wulfraed                 Dennis Lee Bieber         AF6VN 
	wlfraed@ix.netcom.com    http://wlfraed.microdiversity.freeddns.org/
Reply to
Dennis Lee Bieber
Very unlikely - Linux in common with pretty much everything else practices overcommit, malloc just makes an entry in the memory map nothing is actually allocated until it is touched. These days malloc usually only fails if it causes the process limits to be hit.
The usual consequence of filling swap on any unix like system is that the kernel will randomly kill a process in order to free memory to satisfy the demand.
--
Steve O'Hara-Smith                          |   Directable Mirror Arrays 
C:\>WIN                                     | A better way to focus the sun 
 Click to see the full signature
Reply to
Ahem A Rivet's Shot
I did say that I don't use swapfiles apart from the default one set up by noobs on my RPi !
man 8 swapon has just about all you need to know, including how to create a swapfile (using dd and /dev/zero - use dd to copy from /dev/zero, specifying the size with something like
dd if=/dev/zero of=/swapfile bs=512 count=3906250
should create a 2GB swapfile called 'swapfile' in the root directory. 'count' is the required file size in bytes divided by the block size specified with the 'bs' parameter 512 bytes is the default blocksize given on the 'dd manpage.
Could well be: as I said, I've never had to set up a swapfile because all my non-RPIs boxes use swap partitions.
man 8 swapon
is your friend.
And a reminder to those newer to Linux: running "apropos xxxxxx" gives you a list with single line descriptions of every manpage with "xxxxxx" in the manpage's initial NAME section. It has the format:
name(s) - one line summary description
Find one that looks interesting and then use the command "man progname" to see the manpage for that program or subject.
The command "man man" shows you the manpage describing 'man'. Read it to understand the significance of the '8' in my example of reading the swapon manpage.
--
Martin    | martin at 
Gregorie  | gregorie dot org
Reply to
Martin Gregorie
I think you'll find Raspbian (Debian?) uses systemd dphys-swapfile.service (/etc/init.d/dphys-swapfile & /etc/dhpys-swapfile + /lib/systemd/systemd-sysv-install enable/disable dphys-swapfile) to compute size and load a swapfile under /var/ No, it doesn't seem to change size in use. File size is (re)calculated at system start.
--

Chris Elvidge, England
Reply to
Chris Elvidge
Does that service recreate the swapfile every time it boots? If so, can the service definition be edited to change swapfile size if this should need doing? My RPi isn't running ATM (and isn't in this room) so I can't easily look at it.
--
Martin    | martin at 
Gregorie  | gregorie dot org
Reply to
Martin Gregorie

ElectronDepot website is not affiliated with any of the manufacturers or service providers discussed here. All logos and trade names are the property of their respective owners.