When does a program write to the SD card

If your program isn't writing to a filing system, it shouldn't be causing any disc writes (ignoring causing swapping due to large memory use).

What will be causing disc writes is whatever operating you are using, as this will be writing to the file system pretty constantly for one reason or another. There are many article on tuning various Pi OS's to reduce disc writes.

---druck

Hi Martin!

28 Jan 2016 13:10, from Martin Gregorie -> Jon:

MG> - pulling the SD card out of an RPi, satnav, PDA, etc. while its still MG> running MG> then you'll sooner or later you'll corrupt the card and it will be MG> your fault.

That is the trivial fault. The much bigger one is the number of writes.

Irrelevant for a satnav. But can be very relevant for a RPi, depending on how you use it.

That is the reason for eg ramlog

Believe me there IS a good reason to fear for the health of your SD card when used as the root FS for a linux system.

MG> I've been using SD cards for over 10 years in a variety of PDAs, MG> satnavs, RPis and flight loggers. Lucky you! Reads on low number of serial writes is usually no big problem.

Random writes can be a big problem. SD cards are not designed to for such workloads.

SD cards are also called probabilistic storage devices. They store something, hopefully the data you get back has something to do with the data that you put there.

Under the hood it is a real mess, and we are quit lucky that they work as good as they do, if you take into account the work that is done behind the scenes from the controller to present you something that LOOKS reliable.

CU, Ricsi

Well, commonly Python modules are compiled on first import and the result is saved in .pyc files. So if you do a lot of testing on the Pi that would cause some extra writes.

Whether this is a problem depends entirely on you and how you use the device.

If you always use reputable SD cards[*] and make a habit of doing any of the following:

- switching an RPi, satnav, PDA, etc. off without shutting it down correctly

- taking the SD card out of a PC's reader without using the PC's eject/unmount function and waiting for it to finish writing to the card

- pulling the SD card out of an RPi, satnav, PDA, etc. while its still running

then you'll sooner or later you'll corrupt the card and it will be your fault. The degree of corruption can vary from a single damaged file, through needing to reformat the card all the way up to a totally borked card. It all depends on what the was being done to the card at the time:

- if the card was only being read, i.e. there are no incomplete or pending writes to it, then damage is unlikely.

- if a file was being written to or has unwritten data in OS buffers then you may find that just that file is damaged

- if a file was being extended, deleted, or its directory entry was being changed then you may need to repair or reformat the card

- all SD cards use wear levelling. This involves re-mapping its memory blocks to swap the content of seldom used blocks with heavily used ones and you don't know when an SD card is being remapped. However, if the card was being remapped when the device was powered off or the card was pulled out, then you may find the card is totally corrupted and can't even be reformatted.

Bottom line: *ALWAYS* make sure the device was correctly shut down or the card has been ejected/unmounted before taking the SD card out of it.

I've been using SD cards for over 10 years in a variety of PDAs, satnavs, RPis and flight loggers. I have never had a corrupted SD card in that time but I'm careful to check that the card is unmounted or the device is properly shut down before I insert or remove a card.

[*] reputable cards are branded cards from suppliers who make the cards they sell, i.e. SanDisk, Samsung and Toshiba. Everybody else rebrands cards made by somebody else. Lexar (formerly Crucial) and Kingston may be OK but I won't touch noname cards or cards from other brands.

--
martin@   | Martin Gregorie 
gregorie. | Essex, UK 
org       |

Hi Martin!

28 Jan 2016 21:01, from Martin Gregorie -> Richard Menedetter:

2 over 3 years.

Please read this:

CU, Ricsi

Thanks, your answer was very helpful.

BR

Jon

Given the sort of data volumes I write to a PNA running LK8000 and used routinely for glider navigation:

- upgrade software twice a year ( 5 MB)

- update airspace: 200k x 4/year ( 1 MB)

- update landouts once a year (50 KB)

- record flight logs ( 2 MB)

IOW, about 8MB of writes in total. If we assume a block size of 512B, thats 16K writes a year to a 2MB card which has 4K blocks of that size, thats an average of 4 writes a year per block if we assume that the wear levelling mechanism works as advertised.

Most flash cards are claimed to be good for 100K writes before failure, so clearly the card is more likely to be lost, trodden on or have its contacts worn out long before the recording material starts to fail from excessive use.

OK, so how many cards have you worn out so far? Over what time period? Numbers or it hasn't happened.

Raspbian, as normally configured, seems to do much less writing than vanilla Debian or Fedora running on a PC. Given that the underlying SD block size is around 4KB (so you'll commonly find 8 ext2 blocks on an SD card block) very often you'll only get one SD card write when a file is updated, simply because user-generated files are relatively small under Linux.

Depends on the card. If you use one designed for video cameras (class 10?) which is designed for sequential access, that might be a problem, but performance will be rubbish, but hopefully you'd use a class

4 or 6 card, which seem to have better random access performance, in an RPi.

I'll agree that cards are seldom removed from an RPi, but people are very careless about doing orderly power-downs and this applies to RPi users too.

I remember discussing the need for care when shutting down a PDA and immediately we finished, one of the pilots pulled the card out of his Binatone satnav without turning it off. He'd been tweaking his LK8000 configuration and there was evidently still some unwritten data, because, when the card was put into a PC it was unreadable and had to be reformatted and, no, it was readable before this or the PDA wouldn't have run LK8000 off it. He swore blind that he'd done a proper shut-down before pulling the card, but as it happened, I was looking at him at the time.

--
martin@   | Martin Gregorie 
gregorie. | Essex, UK 
org       |

Hi The!

29 Jan 2016 09:45, from The Natural Philosopher -> Richard Menedetter:

Congratulations!

But what is the point? You can cross the traffic light when it is red. If there is nobody coming, than you will get away with it.

Claiming that it is secure to do so, because you have crossed red traffic lights for the last 3 months, and you still live is not really an argument.

CU, Ricsi

Ive had one on a ASUS EEEPC netbook running for a couple of years with no issues.

--
Future generations will wonder in bemused amazement that the early  
twenty-first century?s developed world went into hysterical panic over a  
globally average temperature increase of a few tenths of a degree, and,  
on the basis of gross exaggerations of highly uncertain computer  
projections combined into implausible chains of inference, proceeded to  
contemplate a rollback of the industrial age. 

Richard Lindzen

What brands were they?

Thats interesting, but is really just an expansion of my advice about only using SD cards from reputable manufacturers, preferably those who make them e.g. SanDisk, Samsung, Toshiba brands and to avoid noname or very cheap cards off bargain stalls. Read this to find out why using cheap cards is a very bad idea:

--
martin@   | Martin Gregorie 
gregorie. | Essex, UK 
org       |

Except that SSD wear is not random. Its highly predictable.

You are doing an apples and oranges faux logic game.

Learn a little more before opening mouth.

--
"It is an established fact to 97% confidence limits than left wing  
conspirators see right wing conspiracies everywhere"

SD wear is not highly predictable. The only predictable thing about it is that it will happen - but not when, or which bits will fail.

Dave

It is.

IN terms of error bit density per unit time all other things being equal/.

The only predictable thing about

Wrong. All bits wear on a per write basis. Chemistry and temperature are factors, so one chip will be better than another if its different chemically, and hotter chips IIRC wear faster BUT it is not random. ALL bits wear. Its only when SOME have worn ENOUGH that error correction is necessary, and when there is no error correction left, the DDS device is at EOL.

This is a reasonable article on the subject.

The upshot is for SSD, its no worse than a disk drive but SD cards are somewhat different. My experience of them is that they should do a couple of years if of decent and recent manufacture, even with no especial steps taken to avoid needless writes.

MTBF on my spinning rust is at wet finger estimates, 5 years or so, Ive gad one go in less than 2, some last beyond 10.

--
"What do you think about Gay Marriage?" 
"I don't." 
"Don't what?" 
"Think about Gay Marriage."

Chicken feed.

Used to have a Pi running as a web cam and generating a HD resolution time lapse movie from the still images grabbed every 30 seconds. There would be the orginal image capture, add a caption (date/time), save to sequentially numbered file, delete those files older than 24 hours for the still webcam side. The time lapse run at 0000, 0600,

1200 and 1800 would create a list of the sequential files to use, create the .mp4, the one at 0000 being kept, the others "overwritten".

I can't seem to mount the SD card that has gone into "safe" mode or the USB memory stick that has also decided it doesn't want to play anymore. But lets assume each still image was 1 MB (they where bigger maybe closer to 2 MB), anyway, that's 1 MB of writes every 30 seconds

24/7 (24*60*60)/30 = 2.8 GB plus plus 2MB every 30 seconds if it's not dark (ave 12 hours/day not dark) another 2.8 GB of writes. I acn't remember the size of the .MP4's but an 8GB SD or USB could hold about 30 days of time lapse and the still image "buffer". 250 MB each maybe.

So each day there would be 2.8 + 2.8 + 1GB = 6.6 GB of writes. The first 8GB SD card (Sandisk class 4, with OS and datastore) lasted about 6 weeks. I then shifted the datastore to an 8 GB USB stick (Lexar), that stick lasted about 8 weeks... I've taken the camera down. B-)

--
Cheers 
Dave.

So that's pretty predictable then :-)

--
Outside of a dog, a book is a man's best friend. Inside of a dog it's  
too dark to read. 

Groucho Marx

Hi Martin!

29 Jan 2016 11:41, from Martin Gregorie -> Richard Menedetter:

A-Data and Samsung (I think it was a rebranded Sandisk or maybe Samsung also makes SD Cards?)

Clever controllers can make a bit of difference, but the general problem is electromigration:

CU, Ricsi

Hi The!

29 Jan 2016 12:22, from The Natural Philosopher -> Richard Menedetter:

TP> Except that SSD wear is not random. Its highly predictable. TP> You are doing an apples and oranges faux logic game. TP> Learn a little more before opening mouth.

Feel free to enlighten me with facts. I see none in your post!

CU, Ricsi

see another post on electro migration. SSD 'wears' per write operation. all other things being equal it will die after a certain number of write ops. this is not random.

stepping in front of cars till one hits you is random.

Apples and oranges

--
No Apple devices were knowingly used in the preparation of this post.

What are we talking about, the predictability of the writes, or the physical wear of the flash cells?

Where the writes can be predicted. The filing system has an allocation strategy and hot points such as the storage indexes, the flash controller has an algorithmic wear levelling mitigation strategy.

The physical wear process is not predicable to the degree you can say when an individual cell will start giving bit errors. But failure rates can be statistically modelled.

So in summary, keep writing to the same part of the disc, and it will fail sooner rather than later.

---druck