Are there any programs available in Raspbian that will "format" a flash device, in the sense that all blocks are reset to the new state? Basically I'd like to trigger a comple cycle of garbage collection, so that new writes can happen without waiting for on-the-fly garbage collection.
I've got a 128 GB Samsung Evo + microSD that worked extremely well until a couple months ago. Suddenly, it got _really_ slow. It's not full, but close, and one possiblility is that I'm now seeing just how fast (not!) the garbage collection is. If the garbage collection could be forcibly run maybe the device would then work as it did when new. It's so slow now there's little to lose, and the data on the card is expendable. If it shortens the overall life of the card that's an acceptable compromise.
Do you mean the trim command to indicate to the device that free blocks in the filing system can be used for wear levelling?
If so, it can be done but only for USB to SATA attached SSDs, not SD cards or USB sticks. See
formatting link
The rule with any SD card is; if its more than half full, get a new one. The fuller it gets, the sooner it will die.
The best thing to do is get a USB to SATA cable and attach a small Samsung or SanDisk SSD, its far quicker and will not degrade in the same way as an SD card. Second best is Samsung USB3.1 memory stick, which is almost as fast and has a better lifetime, but not as good as an SSD.
but I don't think that works for SD card readers. You can try writing zeroes to it:
dd if=/dev/zero of=/dev/sdX bs=1M
If you hve a Pi booted from USB you might be able to put the SD card in the Pi slot and do:
blkdiscard /dev/mmcblk0
(I suppose you could put an OS on the card, boot single-user, mount the SD card readonly, and then blkdiscard it underneath you - but no guarantees if it tries to read the card in the middle and crashes)
Could you elaborate? I tried bob@raspberrypi:~ $ sudo apt-get install smartctl [sudo] password for bob: Reading package lists... Done Building dependency tree Reading state information... Done Package smartctl is not available, but is referred to by another package. This may mean that the package is missing, has been obsoleted, or is only available from another source
E: Package 'smartctl' has no installation candidate
As it happens the microSD card was used with trim turned on.
Just to be clear, I'm not looking for an "on the fly" solution. I'd like to force a one-time reset of the card to the unwritten state.
No, I mean a stand-alone utility that will force all blocks to the state they were in at the factory. Perhaps "secure erase" has the right connotation.
Interesting thread, but I don't think it applies. This is a micrSD card, not an SSD. It could be placed in a USB adapter for experimenting, but I gather USB doesn't support anything like TRIM.
Shortening the working life of the card is acceptable in this case. I just want to see if the "slowdown" is related to ageing (rather, garbage collection) of the card.
It's installed, but wants to run as user pi. Can it be run as another user in the wheel group? Most other things, apt-get for instance, work fine with sudo and my password. I'd rather not unlock the pi account.
On closer reading of the link druck pointed out, the blkdiscard command seems to do what I want, if it works on a microsd card.
Not for SD cards, only SSDs support the trim command which tells it that blocks are no longer used by the filing system. See below.
Yes SSD only as I said, and trim is supported using USB to SATA converters.
In a word yes. When a flash device is new is will do every new write to an used block, but as the devices is used, it writes to an ever smaller pool of unused blocks. Without the trim command to tell it blocks are no longer in use, eventually it has to assume all the blocks are used. Then instead of just writing, it has to swap more and less frequently used blocks, which is much slower.
Saying all that, I suspect that some cards may detect the pattern of writes from a reformat of the whole card to the default of FAT32, and reset the used blocks.
There are several mechanisms in the SD spec, depending on which parameters you give to CMD38:
ERASE DISCARD Full User Area Logical Erase (FULE)
However if you have a USB card reader it's that which is issuing the commands, and it may not be able to pass these special commands through. A directly-connected SD card (eg in the slot in a Pi) should be able to issue them.
I doubt a reformat to FAT32 (which would be just the write of a partition table, root directory and FAT, leaving the rest unwritten) would be detectable, but a write to every block with all zeroes might be.
On Wed, 29 Apr 2020 22:42:35 -0000 (UTC), Nikolaj Lazic declaimed the following:
A freshly formatted SD card is set (erased) to all-1s, and can only change bits to 0s when written. Any reuse of an allocation block requires the card to first erase to 1s, before the new data can write to it.
Unfortunately, the sites with decent descriptions of how SD cards operate have expired and the links are now dead. The following link is more of a company presentation for their testing software.
formatting link
""" NAND Structure
A raw NAND flash device can be broken up into three distinct parts.
Cell: The smallest entity. A cell stores data at the bit level and is not directly addressable by the device controlling the NAND storage. Page: The smallest array of cells that can be addressed for read and program operations. A program operation consists of ?flipping? bits from the value 1 to the value 0. Page sizes are in the kilobyte range; e.g., 4 KB. Block: The smallest array of pages that can be addressed for erase operations. A block is also known as an eraseblock in some contexts, such as the Linux MTD stack. An erase operation consists of returning bits with a value of 0 to a value of 1. Block sizes are in the megabyte range; e.g.,
4 MB. Erase operations are much slower than program or read operations executed on pages. Erase operations wears out flash storage over time. When a block is no longer usable for storing data, it is labeled a bad block. """
A benchmark suite that might be of interest:
formatting link
Take into account that SD card speed ratings are based upon FAT filesystem, and even worse: Class 10 is based upon streaming transfers of a single large (video) file on a freshly formatted card. Class 2/4/6 is based upon transferring smaller/scattered files (still image photos) on a fragmented card (which would result from deleting some images at random). As a result of these rating differences, cheaper Class 10 cards can only buffer one allocation/erase "block" for the FAT, and a second one for the data. Better cards in the 2/4/6 range may have internal buffers for four or more allocations blocks at a time, which reduces the cycles of "commit block buffer to SD card, find free block, erase block, copy unchanged pages to new block buffer, write new pages to buffer..."
Now compound that with use of EXT3/EXT4 journalling file systems -- where first the new data gets written to free pages, then the journal gets written describing where the new data is located, then later when the system is idle the original file metadata (inodes?) gets updated from the journal, at which time the original file data pages are marked free, and eventually the journal gets marked free. If one only has two allocation block buffers, one ends up with a lot of block reuse/erases. Having four or more buffers can postpone some of the reuse as the journal may not be committed to card itself. So, in the end, a cheap Class 10 can be SLOWER than a higher end Class 4 card.
From a post on the Beagleboard forum:
The BBB has SanDisk Edge 8GB Class 4 HC I8227DTJLT009 Not sure of the eMMC version
The R-Pi3B+ has Kingston 16GB Class 10 HC I U1 SDC10G2/16GB N0581-002.A00LF
The Class 4 SanDisk, in the 1GHz single-core BBB readily beat the Class
10 Kingston in a 1.4GHz quad-core R-Pi3B+ in any meaningful test (the Kingston only won out on the "bdr" and "dd" test cases, and the BBB eMMC beat it on the "dd" test).
--
Wulfraed Dennis Lee Bieber AF6VN
wlfraed@ix.netcom.com http://wlfraed.microdiversity.freeddns.org/
Controllers are very good at spotting certain patterns, and doing something - its how most benchmarks are faked^H^H^H^H^H optimized.
I'm only going by anecdotal evidence that old knackered SD cards reformatted with the official SD Card reformatting tool seem to have close to original performance (before dying shortly afterwards). The tool only writes the MBR and FAT, its far too quick to zero every block.
It probably starts with something that says 'you are blank' and then writes MBR and FAT. Perhaps the "Full User Area Logical Erase (FULE)" command Theo mentioned.
--
Steve O'Hara-Smith | Directable Mirror Arrays
C:\>WIN | A better way to focus the sun
The computer obeys and wins. | licences available see
You lose and Bill collects. | http://www.sohara.org/
That's possible, but highly risky. A card deciding to wipe itself based on a pattern of writes to a few blocks is not one I'd want to have the liability for...
"What is the difference between Quick and Overwrite format methods? Quick format deletes all the file/directory entries by initializing file system parameters of the card, but it does not delete the data written in files. Overwrite format deletes file/directory entries by initializing file system parameters of the card (same as with Quick format), and erases all data by overwriting the user data area completely. The Overwrite format takes more time to complete than the Quick format method."
formatting link
It is possible that 'Quick' mode still zeroes some blocks, which are enough to give the card some breathing space in the free block pool, which would improve performance.
ElectronDepot website is not affiliated with any of the manufacturers or service providers discussed here.
All logos and trade names are the property of their respective owners.