move /var/log to a RAMDISK

Thanks for posting that!

I'm not the OP, but I still found that info immensely useful for a totally different reason. And I'm probably not the only one who doesn't want to waste a day sifting through the systemd and journalctl docs to figure it out for myself.

On Tue, 1 Aug 2023 14:25:33 +0100, Dave snipped-for-privacy@cyw.uklinux.net> declaimed the following:

I'd expect the main feature to be: how many allocation units can be kept open at one time.

Class 10 cards are rated for streaming to a newly formatted card -- and with FAT file systems, they only need to keep two allocation units open. The FAT and the streamed file, and cheaper cards only allow for that. Opening a second file for I/O requires flushing the current allocation unit to the actual flash rather than RAM-type buffer, finding a free allocation unit (erasing it to all 1s), loading it (copying any data from the old allocation unit if the file is being /reopened/ before marking that old one free).

Class 2/4/6 were rated for still image operations on cards with deletions. They often support 4-6 parallel open allocation units.

On a journal ling file system, having lots of open allocation units is desirable.

It's stored in /run/log/journal (/run is tmpfs in RAM)

Despite the configuration specifying a maximum journal size of 32M and even 512MB Pi's having a 64MB /run, if you do a df /run and see it at

journal --vacuum-files=1

If you aiming for uptimes of 80+ days or more, then bung that command in a weekly or monthly cron job, as things can get a bit temperamental if /run is full.

---druck

TNP, FFS, SD cards cost buttons. Pack of two 32GB for less than a tenner. If you are worried about wear, clone a new one every year, or whatever.

Lucky. I had an older Pi which I had set to reboot every day, and I had lots of occasions when it didn't come back up by itself. Often just needed power cycling, occasionally filesystem corruption needed fixing. Turned out it was a dodgy WiFi dongle, and after replacing that I was able to abandon the daily reboots, and the reliability was rock solid after that.

My suggestion is to look at the logging levels of various processes, and ensure everything has a suitable log rotate configuration, so the size of your logging directory is kept pretty static over time without having to reboot. Particularly as booting does lots of logging, and once that is rotated out, the size may go down slightly.

I do nightly differential backups across my fleet of Pi's and the first thing I look for in the monitoring graphs is the size of data transferred, any significant increase is invariably due to some problem causing a log to grow massively. A quick intervention has often avoided considerable downtime.

---druck

I think you are right. I really should set it up better than it is. It's a classic "round tuit" problem as it works acceptably as is. I suppose the problem with Pi's and the like is they feel disposable along with their SD Cards and once you have something working the urge to do the job properly wanes. And you can always just stick the backup SD Card in and reboot!

Agreed. If you CAN keep it simple, DO keep it simple. Remember YOU might have to debug/enhance it a few years LATER :-)

Things like Pi's are special cases - not blazing fast, not a lot of RAM, run off SD cards/eMMC that have to be protected from re-write fatigue. If the need for space isn't excessive, and simplicity is worthwhile (almost always) then a RAMdisk is often your best solution.

Because of the filesystem overhead, RAMdisks just MIGHT not be fast enough ... then you have to go to "less simple" approaches alas. To each app, its own.

Cool stuff

"Pipes" are a kind of cheat - really just "invisible" R/W files. I used them kinda extensively on some forking servers I wrote a couple of years ago in Python and 'C' to make data gathered by new forks accessible to the mother process. They WORK just fine and are commonly used and I *think* a tad faster than a RAMdisk, but still ...

I'm fond of such 'servers' - even cobbled together a good "pre-forked" one - supposedly the highest-capacity/speed - for both TCP and UDP - but never had a good high-volume reason to use them. The best was almost like a 'chat' app, non-sync bi-directional where the queen server process could initiate, even push, tasks and messages to the clients. Fun !

Hmm ... now apparently some possible servants of Xi here are calling me a "troll" because I repeated a few things (said by HS/NSA/CIA) they didn't WANT to hear about CCP sabotage-ware infiltrating almost everything in our important infrastructure/military systems and 'devices' (which are almost entirely Linux/Unix-based). One day soon they ARE gonna get a big wake-up call ... but meanwhile denial means they won't DO anything about it :-)

ANYway, /var/log CAN be moved to a RAMdisk if you want. Not 100% sure WHY you'd want to, but it CAN. If a few very early logs get 'lost' as you re-direct /var/log then that MIGHT not be all so important. If you want it all on RAMdisk then you don't CARE if it all vanishes on reboot. I very rarely look in /var/log anyhow so ....

Some info I gleaned suggested that [some of] the better cards do have full wear levelling.

I could find no clear evidence of that.

-- “It is not the truth of Marxism that explains the willingness of intellectuals to believe it, but the power that it confers on intellectuals, in their attempts to control the world. And since...it is futile to reason someone out of a thing that he was not reasoned into, we can conclude that Marxism owes its remarkable power to survive every criticism to the fact that it is not a truth-directed but a power-directed system of thought.” Sir Roger Scruton

Yup. excellent advice.

Now my /var/log isn't being used at all

oot@heating-controller:/var/ramlog# du -sh /var/log

-rw-r--r-- 1 root root 21280021 Aug 2 16:42 access.log

-rw-r----- 1 root adm 119049 Aug 2 16:39 auth.log

-rw-r----- 1 root adm 88082 Aug 2 16:39 daemon.log

-rw-r----- 1 root adm 3119 Aug 1 11:05 debug

-rw-r--r-- 1 root root 31171 Aug 2 15:39 error.log

-rw-r----- 1 root adm 31714 Aug 1 11:05 kern.log

-rw-r----- 1 root adm 30892 Aug 1 11:05 messages

-rw-r--r-- 1 root root 0 Aug 1 11:05 other_vhosts_access.log

-rw-r----- 1 root adm 133829 Aug 2 16:39 syslog

But ramlog is. My server instructs the browser to to do an ajax poll every second

# ls -l /var/ramlog total 21240

-rw-r--r-- 1 root root 21296720 Aug 2 16:45 access.log

-rw-r----- 1 root adm 119049 Aug 2 16:39 auth.log

-rw-r----- 1 root adm 88082 Aug 2 16:39 daemon.log

-rw-r----- 1 root adm 3119 Aug 1 11:05 debug

-rw-r--r-- 1 root root 31171 Aug 2 15:39 error.log

-rw-r----- 1 root adm 31714 Aug 1 11:05 kern.log

-rw-r----- 1 root adm 30892 Aug 1 11:05 messages

-rw-r--r-- 1 root root 0 Aug 1 11:05 other_vhosts_access.log

-rw-r----- 1 root adm 133829 Aug 2 16:39 syslog

Next step is to turn off such verbose logging from apache...ah simply comment out the line dictating access logs in the sites-enabled location and no more access..log. its getting there.

Its looking OK. root@heating-controller:/run/log/journal# ls -la total 0 drwxr-sr-x+ 3 root systemd-journal 60 Aug 1 11:05 . drwxr-xr-x 3 root root 60 Aug 1 11:05 .. drwxr-s---+ 2 root systemd-journal 80 Aug 2 06:05

I will bear that in mind But power failures are relatively common here when the wind blows

This product may end up in places where the user isn't smart enough to do that.

This pi will be running my central heating. It will be tucked away in a boiler cupboard. The existing controller keeps crashing during power cuts so I decided that with a $5000 heating bill, making a rock sold controller to have different temperatures at different times might easily save me $1000 a year (as well as allowing me to hit the controller via the bedroom laptop and turn bathroom heaters on before I get out of bed).

And by not continuously writing to the SD card with log files the chance of a non booting unit is really rather small. In fact it is hardly writing to the SD card AT ALL. now.

Its going slower than I hoped but i haven't met any major issues on the controller - the Zero W.

The Pico Ws to measure temperature and oil tank levels may be more of a challenge. But the existing Wireless oil tank gauge sucks - hasn't got the range, but to my surprise, holding up the mobile next to the oil tank showed two fairly weak, but potentially useable WiFi points in the house.

WE shall see.Having a machine far more capable than a PDP/11 cost less than £25 with the SD card, and less than £50 all in with power supply relays and hopefully 3D printed box, is stunning for basic networked process control

Looking forward to hacking the Picos in due course. Ive been working on the simplest and cheapest electronics that will, if the Pi instructs it, shut all power down to the Pi and draw at most a few microamps until a capacitor charges up again, when it will restore power and reboot the PICO..my existing oil tank monitor wakes up once an hour. For a second or two. The battery lasts for years.

Oh, I learnt my code writing contract assembler for people who wanted it well commented enough that the stupidest permie could fix it years later. Or myself. Sometimes I look at stuff I wrote and marvel at ait.

I am not a computer scientist. Elegance isn't in my nature. I am a software and other sorts of engineer, and making it work, and keep on working at the lowets possible prices is burned into my soul

"An engineer is someone who can do for five bob what any damned fool can do for a quid"

Ha! compared with a Z80 with 48k of usable RAM or a 6809, they ARE blazingly fast. I cut my teeth writing C and assembler for those, stuff that was burned into ROM. There is so much on this zero ROM - 16GB! And I have ram coming out of my ears.

My access requrements are measured in seconds, not micro seconds.

Oh there surely are faster, but who is in a hurry? a central heating controller changes states every few HOURS. and if the users browser view of that is a second behind, frankly who gives a damn?

Like wise room temperatures and tank oil levels are not things that change much in an hour...

The problem is that no one knows what is really happening, and although the political narratives that are spun for consumption by hoi polloi, are obviously not the real truth, no one knows what is. Not even the spinners.

I myself know for example, that there couldn't have been any weapons of mass destruction *directly threatening British interests* in Iraq, and that our prime minister lied about that. I spent a couple of years designing missile parts and know in principle where the bleeding edge is, and it sure ain't in Iraq.

I know that Covid injections do not put goverment control chips in your brain, They don't need to. They have the mass mediafor that :-)

I know the moon landings were real, because there wasn't the technology then to fake it.

I know that 911 was a simple case of structural failure in a fire, because everything about it is consistent with my knowledge of structural engneering.

I dont know who apart from lee harvey oswald, might have shot JFK , or why. |Jury still out.

I know that 'renewable' energy is a greater disaster than any climate change, and I know that climate change is largely natural with very little man made components. Because I did the years of research. And the problem is within my pay garde as an engineer with a good STEM background

I don't know who runs the world banks, but I an fairly sure its not Jews, or Lizards. I am however sure that they haven't a clue what they are doing.

As I said, whoever you are, the less writing to an SD drive there is, the less chance there is of file system corruption when the power goes out.

Its simply a natural habit to ensure than whatever an embedded system does is the minimum necessary to get the job done with the maximum reliability.

The SD card looks like it might be the weakest link. So I am trying to reduce stress on it.

On Wed, 2 Aug 2023 17:25:36 +0100, The Natural Philosopher snipped-for-privacy@invalid.invalid declaimed the following:

I wonder if studying the boot configuration files of a Pi-Star system might not be of use. Pi-Star is a system for working with internet linked digital Amateur radio nodes, using a low-power radio board on top.

By default, Pi-Star runs with the main (SD) drive set up in READ_ONLY. All logging, etc., is in RAM.

pi-star@pi-star-3b(ro):~$ df Filesystem 1K-blocks Used Available Use% Mounted on /dev/root 15213412 1955360 12604368 14% / devtmpfs 464768 0 464768 0% /dev tmpfs 498048 0 498048 0% /dev/shm tmpfs 32768 7632 25136 24% /run tmpfs 5120 0 5120 0% /run/lock tmpfs 32768 0 32768 0% /sys/fs/cgroup tmpfs 65536 8 65528 1% /tmp tmpfs 16 4 12 25% /var/lib/logrotate tmpfs 32 16 16 50% /var/lib/dhcpcd5 tmpfs 65536 10740 54796 17% /var/log tmpfs 64 0 64 0% /var/lib/php/sessions tmpfs 16 0 16 0% /var/lib/sudo tmpfs 1024 72 952 8% /var/cache/samba tmpfs 4096 2176 1920 54% /var/lib/samba/private tmpfs 1024 0 1024 0% /var/lib/nginx/body /dev/mmcblk0p1 64456 51112 13344 80% /boot tmpfs 99608 0 99608 0% /run/user/1000 pi-star@pi-star-3b(ro):~$

Biggest problem is that it is a few OS versions (Debian) behind.

Heck -- some years ago there were instructions on how to set up two R-Pi as a VAXCluster (back when the hobby version of OpenVMS was a free package). Unfortunately, that is now a 404 page.

We are more likely to have power failures when the wind doesn't blow!

---druck

Yes it works, right up to when it doesn't :-)

That's why I set up the nightly differential backups, which write the changes to an image file of the SD card, so if the card fails I can just write the image to a new SD card and be back up and running in minutes.

That's fine for my Pi's at home, but now I have them running services at several other locations, which I may not be able to get to for weeks. So I've set them up far more carefully to avoid using up the SD write life, and have invested and trying out the Samsung and SanDisk high endurance variants.

---druck

Ah, well the qualifier 'local' versus 'national' comes into play here, and it's not exactly true either

high wind leads to far more fluctuation in generation as the wind can stop wuite quickly, and if all the other generators are shut down,m there ain't nuthin' left.

People are coming to realise that renewable energy is a far greater threat to civilisation than climate change, man made or not, could ever be...

Interesting philosophy. accept failure and plan round it rather than eliminating failure in the first place? I will take an SD card image once the design is finalised, but fir now the Piz are outside my nightly backup regime simply because they are not changing, and such changes as there are are simply a mater of remounting my server on a fresh pi and saying 'make;make install'