I am using several raspberries and one of them is used for recording of the data stream coming from a webcam. The (Axis) webcam sends a stream of mjpeg data, which is transformed into avi files using ffmpeg.
This worked fine, but, yesterday evening this machine broke down after recording of 2 or 3 GB of data to a 16 GB micro SD card. There was no partition to be found any more on the sd card, and I could not get it back to life. This should not happen.
so, will this file be written until the end of transmission is reached and then closed, or, will it be re-opened continuously and due to "wear-levelling" then there will be thousands of physical locations written to on disk (card)? If so, then, in my understanding, this would be a plausible reason for the damage. But, then, this will occur again and again on every new installation.
Thanks for any info!
Best regards,
Markus
--
Please reply to group only.
For private email please use http://www.dipl-ing-kessler.de/email.htm
I use to break the data down to 1 hour pieces, around 20 - 90 MB each (depending on brightness, contrast etc.).
Ad 2.)
Not sure, but it was an "Intenso Micro SDHC 16GB Class 10" purchased from Amazon. Well, from the same shipment I have one more now in the raspberry, currently under install :-/. Should I test it first? How?
B.t.w., you do not see any risk caused by the ffmpeg command above?
Thanks, best regards,
Markus
--
Please reply to group only.
For private email please use http://www.dipl-ing-kessler.de/email.htm
How was your SD card formatted? Maximum filesize on FAT16 or FAT32 is 2 GB. Upon reaching that size, the recording should've been forced to end; if your card was scrambled, that would suggest another problem.
If you were using something like ext4, exFAT, NTFS, etc., that maximum-filesize limit doesn't apply. Sometimes SD cards (especially MicroSD) just crap out for no apparent reason. :-P
_/_ / v \ Scott Alfter (remove the obvious to send mail) (IIGS(
formatting link
Top-posting! \_^_/ >What's the most annoying thing on Usenet?
Useful test: Can you do the conversion of that particular file on a machine with a bigger hard disk? How big is the /mpeg file? How big is the .avi file? Are either close to the 4GB limit?
Could be anything. I won't buy any SD card that isn't Sandisk, Samsung or
- at a pinch, maybe from Crucial, but always from a reputable seller. The first two have their own chip factories and Crucial is a long-standing rebranding supplier.
BE VERY CAREFUL that you're writing to the suspect SD card and not anywhere else when doing what follows.
Since the card is now failed, there's nothing of value on it, so try using one of the formatters: cfdisk, cgdisk, fdisk or gparted to create an ext3 partition on it. The first three are command line tools. gparted has a graphical interface.
gparted will tell you how much space it finds and refuse to add any partition(s) that overfill the device. If gparted thinks its around 16GB, then try to put a max size ext3 partition on it and format the partition.
If that works, use dd to fill the new partition with zeros. This may take some time but will tell you how much it managed to write to the partition of it fails before filling it. If all was successful and yoy don't want an ext3 type partition, fire up gparted again to replace that with the type of partition you do want.
I don't see why its should be a risk: its a standard package for Fedora Linux and so is probably one for Raspbian as well (My Pi isn't running or I'd look...), but do install it from the Raspbian repository with apt-get or aptitude rather than downloading it from some random website.
The mentioned partition is / was ext4, with around 8 GB available. The avi output was below 100 MB, but I do not see exactly what data rate the stream has. It is an older Axis cam with 640 x 480 resolution, so this should not cause an issue.
Well, I had even worse experience regarding so-called "Kingston" sd cards. But, well, brands like Sandisk, Samsung EVO, Intenso, Imation may also be the target of fake attempts.
That was the reason why I use to purchase such memory cards only from Amazon themselves, and not from an "Amazon dealer".
Yes, I should have done such tests, but I broke it into pieces already yesterday...
Thanks, best regards,
Markus
--
Please reply to group only.
For private email please use http://www.dipl-ing-kessler.de/email.htm
On Tue, 19 May 2020 14:51:48 -0000 (UTC), Markus Robert Kessler declaimed the following:
Presuming the card was formatted for ext3 or ext4 (common for Linux), there should not be a 4GB size limit on individual files (any FAT32 boot partition, however, would have such a limit).
With a name like that, it could be anything...
Samsung, SanDisk, Lexar, and Transcend are names I tend to trust (though I've never seen a Samsung SD card in the wild -- all of the ones I buy are SanDisk -- My Lexar/Transcend experience is with CF cards as used by my digital cameras).
Pretty much any other name on a card is a product of "lowest bid batches" (two cards with the same label but from different batches could be completely different internally). Kingston, for example.
Class 10 cards are rated for streaming media (video) on a freshly FAT formatted file system (which does not perform journalling). Cheap cards will have terrible performance with any journalling file system and/or with smaller multiple file I/O. Fat file system only requires the card internals to handle two "allocation units" at a time -- one for the FAT bitmap, and one for writing/reading the stream. A journalling file system writes new data to one location, then writes the file metadata somewhere, and later updates the original file metadata with the journal contents followed by marking the journal "empty". The idea being that if the system goes down while writing the data or journal, a restart "finds" no changes to the original data, and no journal of changes to be committed; going down after the journal was written but before the changes are committed to the main file metadata means a restart will find the non-empty journal and can commit the changes.
Thing is, each of those writes likely involves getting a different allocation unit -- each change of allocation unit requires finding a free unit, /erasing it/, then copying the non-changing contents from the old unit before writing the new changes to it. That's a lot of erase cycles on a card with only a two allocation unit capability. Cards with four to six allocation unit capability can keep the journal open in one AU, write new data using another AU, have a bitmap of sectors in a third... etc.
Class 2/4/6 cards are rated for small files on /fragmented/ file system (still image cameras, in which the user has reviewed and deleted some images).
Well,
formatting link
would be a start.
-=-=-=- 16 GB Kingston Class 10 in an R-Pi 3B+ (1.2GHz) pi@rpi3bplus-1:~/benchmark$ sudo ./benchmark.sh
Raspberry Pi Dramble microSD benchmarks microSD clock: 50.000 MHz
Running hdparm test...
/dev/mmcblk0: HDIO_DRIVE_CMD(identify) failed: Invalid argument Timing buffered disk reads: 66 MB in 3.02 seconds = 21.85 MB/sec
Running dd test...
51200+0 records in
51200+0 records out
419430400 bytes (419 MB, 400 MiB) copied, 54.3339 s, 7.7 MB/s
Running iozone test... Iozone: Performance Test of File I/O Version $Revision: 3.434 $ Compiled for 32 bit mode. Build: linux-arm
Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins Al Slater, Scott Rhine, Mike Wisner, Ken Goss Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR, Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner, Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone, Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root, Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer, Vangel Bojaxhi, Ben England, Vikentsi Lapa, Alexey Skidanov.
Run began: Tue May 19 13:38:19 2020
Include fsync in write timing O_DIRECT feature enabled Auto Mode File size set to 102400 kB Record Size 4 kB Command line used: ./iozone -e -I -a -s 100M -r 4k -i 0 -i 1 -i 2 Output is in kBytes/sec Time Resolution = 0.000001 seconds. Processor cache size set to 1024 kBytes. Processor cache line size set to 32 bytes. File stride size set to 17 * record size. random random bkwd record stride kB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread 102400 4 264 242 5349 5803 5136 239
iozone test complete.
microSD card benchmark complete!
-=-=-=- 8 GB SanDisk Class 4 in a BeagleBone Black (1GHz) debian@beaglebone:~/benchmark$ sudo ./benchmark.sh [sudo] password for debian:
Raspberry Pi Dramble microSD benchmarks
Running hdparm test...
/dev/mmcblk0: HDIO_DRIVE_CMD(identify) failed: Invalid argument Timing buffered disk reads: 66 MB in 3.01 seconds = 21.89 MB/sec
Running dd test...
51200+0 records in
51200+0 records out
419430400 bytes (419 MB, 400 MiB) copied, 70.7736 s, 5.9 MB/s
Running iozone test... Iozone: Performance Test of File I/O Version $Revision: 3.434 $ Compiled for 32 bit mode. Build: linux-arm
Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins Al Slater, Scott Rhine, Mike Wisner, Ken Goss Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR, Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner, Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone, Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root, Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer, Vangel Bojaxhi, Ben England, Vikentsi Lapa, Alexey Skidanov.
Run began: Tue May 19 13:26:14 2020
Include fsync in write timing O_DIRECT feature enabled Auto Mode File size set to 102400 kB Record Size 4 kB Command line used: ./iozone -e -I -a -s 100M -r 4k -i 0 -i 1 -i 2 Output is in kBytes/sec Time Resolution = 0.000001 seconds. Processor cache size set to 1024 kBytes. Processor cache line size set to 32 bytes. File stride size set to 17 * record size. random random bkwd record stride kB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread 102400 4 1192 1249 6587 6597 6541 635
iozone test complete.
microSD card benchmark complete!
debian@beaglebone:~/benchmark$
-=-=-=-
The BBB with a Class 4 card took 7 minutes for iozone... I'm still waiting for the R-Pi Class 10 after over 10 minutes. Heck -- I killed that run and restarted, just in case my moving PuTTY windows around somehow froze the output.
... Finally finished, it took 22 minutes for iozone!
Things to note:
* HDPARM speeds are very close, the R-Pi at 21.85 MB/s, the "slower" BBB at 21.89 MB/s
The high-grade "slow" Class 4 SanDisk, in a slower computer, outperforms the low-bidder "faster" Class 10 Kingston in the faster computer. The Kingston appears to be killed by the swapping between data and journal metadata, and later commit of the journal.
--
Wulfraed Dennis Lee Bieber AF6VN
wlfraed@ix.netcom.com http://wlfraed.microdiversity.freeddns.org/
The 'dealers' are no different to eBay shops except that its a lot easier to work out the seller's reputation on eBay.
On eBay the (small print) rating under the seller's name is THEIR rating, while the item is as described, though misleading adverts will sooner or later get the seller kicked off.
On Amazon the rating is a summary of buyer reviews of the product, not the seller, though failure to deliver whats sold gets them kicked off.
Dunno which is best: in both cases yer pays yer money and (hopefully) gets what yer wanted.
Actually Amazon supplied stuff is potentially worse than eBay due to the way Amazon warehousing is organised. If more than one supplier is providing the same barcode items via Amazon then the actual physical stock is effectively mixed even when different prices are charged by each particular supplier. The sale is credited to the supplier ordered from but the actual goods supplied depends on which suppliers product happens to come to hand in the warehouse closest to the point of delivery. Thus if there is a rogue supplier injecting counterfeits into the system for a common item like SD cards then the counterfeits could apparently be supplied by any of the sellers providing that particular barcoded card and getting a good one or otherwise is chance. At least with eBay you know who you ordered from and supplied the item, of course it still may or may not be counterfeit.
Silly question: Isn't that fraud? I know German small scale politicians are hopping mad because large, American amazon sees no reason for listening to them and abiding by their whims. But aren't US fraud laws rather strict?
If one supplier offers Mercedes and Trabant and I order and pay for a Mercedes and get the much cheaper Trabant, even if it looks exactly the same, don't American lawyers, obnoxious as they are, have a field day?
Off topic, I know, but I'm surprised and curious.
--
/ \ Mail | -- No unannounced, large, binary attachments, please! --
It is very simple don't use Amazon - never ever! Same as don't use UBER ever, Facebook or whatever BS is coming from there.
As for the Mercedes - I can't tell for sure, but I would not buy a newly manufactured BMW for sure. May be only the most expensive is still produced in Germany, may be not. The Germans started years ago to produce in China and package in Germany selling it as Made in Germany. Isn't this called fraud?! Manipulating the ECUs another example.
On Tue, 19 May 2020 14:17:59 -0400, Dennis Lee Bieber declaimed the following:
Okay -- that may be erroneous; I keep forgetting that Raspbian is built as 32-bit, even when running on a R-Pi 3B, 3B+, 4B, and that OS may have a file size limit. The limit I don't think is inherent in the file system (unlike FAT32).
--
Wulfraed Dennis Lee Bieber AF6VN
wlfraed@ix.netcom.com http://wlfraed.microdiversity.freeddns.org/
On Tue, 19 May 2020 16:18:54 GMT, Scott Alfter declaimed the following:
The R-Pi runs from the SD card, it isn't used as an add-on data storage. The common NOOBS/Raspbian set up is a small FAT boot partition (visible to Windows -- so allowing one to put in stuff like WiFi connection information) with the rest of the card converted to ext4 during setup of the OS. NOOBS expects to be written to a FAT formatted card, then booted in the R-Pi upon which it reformats/partitions the card and expands the OS to the ext# partition.
Technically, FAT32 should support 4GB files, but so much software is written using signed 32-bit integers (I'm looking at you Java!) that half the range is lost.
--
Wulfraed Dennis Lee Bieber AF6VN
wlfraed@ix.netcom.com http://wlfraed.microdiversity.freeddns.org/
You can have 64-bit file offsets in 32-bit executables for 32-bit OSes. There is a separate set of APIs, which can be seen in places like MinGW and elsewhere.
I have no idea and had trouble believing that was how Amazon worked, which is why I supplied the Amazon documentation reference. I only found out after investigating how I ended up with a grey market import after apparently ordering (through Amazon for convenience) from (and paying a premium for) an in country supplier. Not a mistake I made twice.
ElectronDepot website is not affiliated with any of the manufacturers or service providers discussed here.
All logos and trade names are the property of their respective owners.