Hello guys! I was wondering.. i'm building and API for my software that will use my rpi3 as a gateway to internet. For instance: My app will send information to my (aka BOX) then my box will send to 1 or N+1 servers the same data but will keep a backup internal for a couple months. Is there any restriction about the amount of traffic data the SD can handle? i'm afraid that my SD can be corrupted if a lot of information arrive at the same time. It's an option to put some HDD but i'm avoiding it, just in the worst case.
Yes, anything that does a lot of writing to an SD card will kill it sooner or later. If you are using a database, I recommend getting a small SSD (32/64/128GB) plus an SATA to USB adaptor, and put the OS and database on that. It will then last practically forever.
---druck
--
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus
Rate of data arriving I wouldn't expect to be a problem, thats what buffers and flow control are for. B-)
However if lots of data is being written erased rewritten etc you can "wear out" and SD card or memory stick. I had a Pi running as a time lapse camera, taking an 1920x1080 image every 30 seconds, adding a date time overlay, producing a timelapse every 6 hours (that took a couple of hours to render), rolling still image and time lapase store. This killed an 8GB micro SD card and an 8 GB memroy stick in around 6 weeks. An image every 30 seconds is 2,880 per day, double that for adding the overlay and add the 4 timelapse videos and rolling stores and aroound 8 GB/day of writes would be happening...
Er, isn't a SSD just a scaled up SD card or is the underlying storage technology radicaly different?
I'd have the OS and programs on an SD card but the database on something else. So when the media with the database falls over it doesn't take the OS and programs with it.
No, and SSD has a far more sophisticated controller and wear levelling strategy designed for use as a computer storage device. An SD card is designed to store large photos and videos from cameras, not 24/7 use.
Again no, if you use the SD for the OS, it will still wear out due to continually writing from logging. Only use the SD card to boot from and put everything else the more reliable medium which wont fall over. There are numerous easy to follow guides on how to do this on Raspberry Pi related websites.
---druck
--
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus
Unless there are a *huge* number of hidden writes, that doesn't jibe with the oft-quoted 100,000 write cycles expected before an SD card dies. I think that those quoting 100,000 write cycles are assuming that the SD card is built with SLC memory, so bear in mind that if 3 level MLC memory chips are used instead the chip could die after as few as 1000 write cycles. Two-level MLC memory falls somewhere between these two: 10,000 write cycles sticks in my mind, probably erroneously.
It was also mentioned several times that an SD card is thought to hold data for up to 5 years between refreshes though several others thought the usable *lifetime* of a card was 5 - 10 years. FWIW Sandisk guarantees its cards for 5 or 10 years depending on the card type, but says nothing about the of number write cycles the guarantee covers.
I've just had a scan for actual test figures, but found only one SD card test project - and that hid its final results behind a password and/or paywall. This project tested 2GB cards. Each test cycle copied 1MB chunks of data from one half of the card to the other and back again, giving effectively a whole-card read/write cycle. I assume that no SD cards have deduplication built in: if they have, that test design would have given bogus results. About the only test data they made available was that two no-name cheapie SD cards failed after less than 100 cycles. Other sites warned that using a journalling file system at least doubles the write load on the card, so using ext2 or FAT32 filing systems should give longer life than ext3 or ext4.
I should add that nobody seems to know how often wear levelling remaps a card, how many write cycles per block this consumes or how the frequency of wear levelling episodes varies with the fullness of the card. About the only conclusion I can come to is that, as I've never had an SD card fail in a camera, WinCE-based PNA or my Pi-1B (lightly used for program development) then you should get good long SD card life from doing office or development-type tasks with an RPi. However, If I was using an RPi as a temporary store for large amounts of data, e.g. hung on the back of a surveillance camera, then I'd either expect to replace the SD card reasonably regularly or use it as a fairly static boot drive with an ethernet-connected hard drive or a SATA-connected SSD added to hold the mass data store.
Does this help anybody?
--
martin@ | Martin Gregorie
gregorie. | Essex, UK
org |
IMHO once you make the decision to connect a harddrive to a pi (spinning-rust in my case) it makes sense to put everything on it and just have a minimal boot SD card.
Il 01/03/2017 18:01, snipped-for-privacy@gmail.com ha scritto:
use my rpi3 as a gateway to internet.
box will send to 1 or N+1 servers the same data but will keep a backup internal for a couple months.
handle? i'm afraid that my SD can be corrupted if a lot of information arrive at the same time.
Mysql work primary on ram memory, but also (in cadence) on the SD card. I have solved mounting 100Mb in RAM and moving DB (but every my script and /var/www) on this position at the boot. Every 30m i dump the DB on SD card.
I have 150 RPI (1/2/3) online from (on average) 3 years and only 2 SD Card are broken.
While any old HD can saturate a USB bus for large sequential transfers, an SSD is far faster for small random access operations. And its these sort of small operations which are used by database applications.
---druck
--
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus
you still have the latency of the USB bus, that is shared with the ethernet , so having an HD or a SSD will change nothing because the bus is used for other things other than the HD. Also it's very lickely that mysql and/or linux will buffer, queue and pack accesses to the fs, so again no change between SSD or HD connected via USB.
On the other hand, if you have some benchmarks that disprove what I just sa id, feel free to post them.
- those which do no wear levelling at all. A given logical disk block always maps to the same physical block, i.e. the same transistors. The blocks holding frequently re-written data wear out quickly.
- those which do dynamic wear levelling, so each time a given logical disk block is written, the hardware chooses, /from those currently not in use/, a different physical block to map it to. This helps a lot, but only if the device has a good number of unused blocks to choose from. Don't run this kind of device close to full.
- those which do static wear levelling. Blocks which hold in-use, but infrequently modified, data are occasionally rotated into more heavily used cells, so that the whole device -- free and in use -- wears out at the same rate.
Wiki has a longer explanation.
Internet Wisdom has it that thumb drives have better wear levelling algorithms than SD cards, and that SSD drives have better again. It's difficult to establish the truth of this, because (SSDs apart) the manufacturers do not document the algorithms their devices (and revisions of devices) use.
There's plenty of 'anecdotal evidence' that SD cards die quickly if used in a heavy write environment, but rather little actual comparative testing that I've found.
On Fri, 3 Mar 2017 10:17:10 -0000, John Aldridge declaimed the following:
Possibly of interest:
formatting link
Particularly for an RDBM that uses multiple files per table, one likely wants a card with enough "open allocation units" to dedicate a unit per file, rather than constantly swapping (and rewriting) units around...
This is one reason I don't always trust the hype about "class 10" cards. Class 10 cards are speed rated based upon streaming video on a freshly formatted card -- no fragmentation, and only needing one allocation unit for the data. Class 2/4/6 cards were rated on fragmented file systems (still image cameras wherein the user may have erased a few images between shooting new ones). A 2 AU class 10 could come out slower than a 6 AU class
6 when used with multiple files or a journaling file system.
--
Wulfraed Dennis Lee Bieber AF6VN
wlfraed@ix.netcom.com HTTP://wlfraed.home.netcom.com/
As someone who works with Enterprise grade flash storage in my day job, I do find it frustrating that there's so little information available about life of consumer grade flash storage.
Enterprise grade flash storage tends to be rated in total write data handled in device lifetime, or something called full device writes per day (DWPD) to achieve a specified device life (usually 3 or 5 years).
Low end storage tends to be rated 0.1 DWPD for 3 year life - typical of good quality SSDs built into laptops (which normally get nowhere near 0.1 DWPD in practice, so you'll get more than 3 years life).
The high-end disks we use are rated for 50 DWPD (can't remember if that's 3 or 5 years life).
However, the end of life for an Enterprise drive is not necessarily what you might imagine - it's the point where the drive can no longer guarantee to retain data whilst powered-off for 3 months (and stored in worse allowable temperature conditions for data retention). Some manufacturers will let you use a drive beyond end of life, and that works well for us because longest storage time we require on most of our SSD's is max length of a power outage, which is nowhere near 3 months. Other manufacturers will lock the drive at end-of-life, or allow only an extremely slow write rate (enough for you to mount a filesystem and copy the data off).
Consumer grade products don't tend to have this protection. You can often read back how worn out they are, but they tend to fail catastophically at end of life. Common failure is inability to write back the modified logical to physical block mapping table. That will leave you with a device you can't write to but can still read, until you switch the power off and it loses the only correct copy of the logical to physical block mapping table which is in the embedded controller RAM, after which the contents are scrambled. A drive which is dead after a power cycle is often due to the block mapping table having been lost.
Enterprise grade drives manage that sort of thing much better, and try to signal end-of-life before they lose data without warning. In some cases, they store enough energy to ensure they can write back the mapping table after a power fail, and they keep multiple copies and/or transaction history on the drive so they don't lose everything on failure to save the mapping table.
You may be able to increase the life of an SSD by partitioning it and creating a partition of 5-10% which you never format or write to. That will give the device more spare flash to play with when erasing blocks and reblocking to handle partial block writes. Behind the scenes, that's part of the way Enterprise flash devices are differently specified for longevity and performance - the amount of spare flash the controller has to play with whilst doing erases and handling wear leveling.
--
Andrew Gabriel
[email address is not usable -- followup in the newsgroup]
Though the orginal question "is the underlying storage technology radicaly different?" hasn't been answered. Only various higher level work arounds to eek out as much life as possible from the individual memory cells.
et, so having an HD or a SSD will change nothing because the bus is used fo r other things other than the HD.
k accesses to the fs, so again no change between SSD or HD connected via US B.
said, feel free to post them.
I don't have the exact numbers to hand, but I used the same SATA to USB adaptor and booted the RPi from both a conventional 2.5" laptop drive and a 64MB SSD. The boot time for the SSD was about 60% of the HD, and the time taken to do an dist-upgrade was about 1/3 of the HD. The HD was actually slightly slower than the original SD card.
---druck
--
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus
ElectronDepot website is not affiliated with any of the manufacturers or service providers discussed here.
All logos and trade names are the property of their respective owners.