What is your strategy to save non volatile data?

Do you have a question? Post it now! No Registration Necessary

Translate This Thread From English to

Threaded View
Many times I work on embedded projects where it is required to save some  
data in a non volatile way.  Usualy I use external (to the main  
microcontroller) serial/parallel EEPROM or internal Flash memory (with a  
software layer for EEPROM emulation).

However my goal isn't to discuss the hw alternatives, but the data  
format and strategy.

I know there are many serialization data format (XML, JSON, Protocol  
Buffer, SQLite), but I don't think they are valid solutions for  
medium/low end microcontrollers.  They are complex to implement and they  
aren't really necessary.
Data are written and read from the same platform, so the endianess and  
byte order are not a problem.

IMHO directly saving a C structure to the non-volatile memory is the  
best method.  When you will load bytes from the non volatile memory, you  
will have magically your C structure in RAM filled with the saved values.

Is it the same strategy that you use?

Re: What is your strategy to save non volatile data?
On 8/11/2016 5:48 AM, pozz wrote:
Quoted text here. Click to load it

There's no universally "best" approach -- much depends on the nature of the
data, it's criticality, the frequency of update, quantity, etc.  *And*, the
hardware facilities available (i.e., if you don't have a "power fail" signal
available, saving megabytes of data *at* loss-of-power can be impractical).

[There is also a dependency on the software structure -- if you can't
get your hands on an "intact" dataset at any given instant, then that
limits what you can *do* with that data when you "need" to preserve it.]

Just "storing struct's" (assuming each has a way of identifying itself,
later -- even if that is implicitly indicated by WHERE it is stored)
means you have to ensure they are stored when they are in a "self-consistent"
state (i.e., when all members "make sense" in the context of their peers).
You also don't have any explicit way of later verifying (at "restore" time)
that the data is intact (checksum/hash/hamming code, etc.).

[likewise, are these data consistent with those *other* data that you're
also saving AT THIS POINT IN TIME?]

It also assumes all the data in a struct *needs* to be saved.  E.g., if I have
a struct that is used to define a text window/pad, it's *dimensions* may
be worth preserving -- but, probably not its *content*, cursor, etc.

Or, that data have the same "value" (i.e., the window's size/shape/position
vs. contents).

Remember, you invariably have to use the saved data to bring the NON-saved
data to some consistent state on "restore".  So, *you* decide what you need
to be able to make that "reinitialization" proper.

The underlying hardware technology also has a significant impact on how and
when you store.  Updating FLASH (limited write cycles, *long* erase cycles)
is much different than BBSRAM; updating a "local" store is much different
than updating a *remote* store (esp via a service!).

I've seen naive approaches that (literally) stored three copies of data
in an attempt to ensure its integrity (despite the inefficiency and lack
of robustness that implies).  At the other extreme, I've seen "raw" data
stored with *no* attempt made to vouch for its integrity (i.e., a three-valued
byte-wide variable could conceivably be recovered with 253 bogus states;
yet the software ASSUMED it would be in ONLY one of the three legitimate
states).

I've used approaches where I constantly maintained a live set of diff(1)'s
against some known system state -- so a "restore" was "restore to known
state followed by this set of (TAGGED!) diff's".

Currently, my only persistent store is a RDBMS.  So, I rely on *it*
to ensure data integrity (I just have to make sure i get the data *to*
the RDBMS before losing power).  I deal with the other issues (frequency
of update, underlying hardware technology, etc.) by partitioning data into
different "tablespaces" (i.e., *where*, physically, each datum gets saved
"in memory") -- so, I can decide how important various data are /ex post
factum/ (cuz I have a limited amount of each type of "memory" and don't want
to have to rewrite my code when I have to move dataX into "precious"
memory and kick dataY *out* to make room for it!).

Re: What is your strategy to save non volatile data?
On 11/08/16 14:48, pozz wrote:
Quoted text here. Click to load it

As Don says, there is no single best method.  The answer to almost every
question in embedded development is "it depends".

But for the common case of storing a small set of parameters, I usually
work with a struct definition.  Along with the parameters in question, I
include a "magic number" for identification, a "parameter structure
version number" (very useful for upgrading the parameter set if a later
version of the firmware uses more parameters), an update serial number,
a CRC check field, and some space reserved for future additional parameters.

Typically I have a statically allocated struct in ram, and two copies in
eeprom or flash.  On startup, I check both non-volatile copies to see
which is the latest version (highest serial number) that has a valid
CRC, and copy that into RAM.  If the parameter structure version number
does not match the current firmware requirements, I upgrade the parameters.

When I need to save the parameters, I use the oldest of the two NV sets.
 First, I erase it (for eeprom, that's just clearing the magic number).
 Then I copy in most of the data and CRC, leaving the magic number for last.


I have had occasion to use a log structured parameter "file system" in
flash, but usually such complications are unnecessary.  And certainly
SQLite, or JSON, etc., are normally overkill.


Re: What is your strategy to save non volatile data?
On 8/11/2016 7:12 AM, David Brown wrote:

Quoted text here. Click to load it

To be clear, you are wrapping *all* of your parameters into a single
"parameters_to_be_saved" struct.

This can work if they are "configuration parameters" -- data that are
consulted when setting up the application, restarting it, etc.

This is different than trying to "save application/process state"
across power outages.  In that case, the "state" may be scattered
through many variables dispersed in memory.  And, more difficult to
ensure catching a "consistent" snapshot.

I.e., if *this* variable is changed, are there any others whose state
must be reconciled *as* it is being preserved?

[It's unclear what the OP is addressing]

Quoted text here. Click to load it

I've used a circular buffer into which "tagged" parameters are placed.
On restart, I "play back" the buffer to restore the latest versions of
the parameters encountered, there.  This lets me mix snapshots and
incremental updates in the same mechanism.

My current application is effectively a hybrid approach as I'm
effectively restoring "application state" and not "configuration data"
(as the system can be considered to be constantly in a state of
reconfiguration).  So, I let the RDBMS do the "log playback" from
its WAL.

In that way, I don't have to micromanage "what's important" *in*
the application(s) but, rather, leave that to the RDBMS.

("Is this datum something that should be preserved across reboots?")

Re: What is your strategy to save non volatile data?
On 8/11/2016 7:53 AM, Don Y wrote:
Quoted text here. Click to load it

A more concrete example:

When I'm in the car, I can access any of the "settings" menus to configure
a variety of options pertaining to how I like the car "configured" when
I'm the driver.  I can manipulate each setting which, eventually, will be
"saved" when(?) I exit the "settings menu".  I.e., if I change 10 parameters,
only one "save" needs to be performed (hand-waving, here).

OTOH, if I'm driving and I *tune* the radio to a station, then tell it
to store that frequency as "preset #4", I've effectively created another
*single* persistent parameter.  Without entering a "setup"/settings mode.

OToOH, if I change the volume level of the radio -- or switch to the
disk drive as "audio source" -- that fact is *also* persistent.  As is
the point in the song where power was interrupted!  When I return to the
vehicle, I expect it to resume playing "from disk", the particular song
that was playing at the time power was lost *and* at the particular point!

Do you put "audio source", "title", "playback point", "volume", etc. in
*THE* "parameters_to_be_saved" struct?  And, flush that entire struct to
persistent store each time *any* of the parameters changes?  Or, do you
break up the "state" into smaller chunks that can be identified and
updated in pieces?


Re: What is your strategy to save non volatile data?
On 08/11/2016 11:27 AM, Don Y wrote:

Quoted text here. Click to load it

That's a cool idea, but as far as I can recall I've yet to drive any  
vehicle which actually does this.

My car is a 2015 Chevy, and if the vehicle is turned off with a song  
playing from CD or Bluetooth, on restart it just returns to its initial  
state. It doesn't even seem to remember which Bluetooth device you had  
connected, so if you have two of them in the car and the one you were  
using is the second on the list of paired devices, you have to manually  
re-select it.

Re: What is your strategy to save non volatile data?
On 08/11/2016 12:11 PM, bitrex wrote:
Quoted text here. Click to load it

Oh, and on hot days when you start driving after the car has been  
sitting in the sun a while, the center console "information display"  
will sometimes crash and reboot.

Re: What is your strategy to save non volatile data?
On 8/11/2016 9:12 AM, bitrex wrote:
Quoted text here. Click to load it

We walked away from the Nissans (and some other vendor) for similar
reasons:  if I can crash the "system" while it's brand new and
under the watchful care of the dealership, what would it be like
"in the wild"?

Re: What is your strategy to save non volatile data?
On 08/11/2016 02:35 PM, Don Y wrote:
Quoted text here. Click to load it

You can select a submenu on the dash display, along with the trip  
counter and tire pressure monitor, etc. that will show you the licensing  
information for whatever open source software they used in the firmware.  
So that's nice, I guess.

It's probably just for something lame like the MP3 decoder, or something.

Re: What is your strategy to save non volatile data?
On 8/11/2016 9:11 AM, bitrex wrote:
Quoted text here. Click to load it

I was describing SWMBO's current vehicle.

If *I* drive it, it remembers which song on the thumb drive it was
playing when I last "accessed" it -- as well as where in that song
it was at the time.

The problem it has is that it restores the "audio state" to whatever
it was when the vehicle was last driven -- regardless of who the current
DRIVER happens to be!  So, if SWMBO drives the car after I have, *she*
hears the song off the thumb drive resuming -- instead of the music
from the *disk* drive (which is where HER music is stored)

Quoted text here. Click to load it

I was terribly disappointed with the quality of the electronics
(infotainment/navigation) in EVERY vehicle that we considered.  Its
as if the folks designing this stuff are clueless as to how cars are *used*
(why do I have to wait for the backup camera to "boot" before I
BACK OUT of my parking space?

Re: What is your strategy to save non volatile data?
Il 11/08/2016 17:27, Don Y ha scritto:
Quoted text here. Click to load it
 > [...]

I'm not sure, but I think many piece of code in the car doesn't save  
anything in a non volatile memory.  The electronics are always power  
supplied from the battery, so it is sufficient to have the variables in RAM.

Indeed when you unplug the battery for a while, all the parameters  
revert back to default.


Re: What is your strategy to save non volatile data?
On 8/11/2016 3:21 PM, pozz wrote:
Quoted text here. Click to load it

I haven't yet tried that.  It would be "unfortunate" as there are *lots* of
parameters involved:
    station presets (AM/FM/XM) -- for two drivers
    audio/GPS/telephone "favorites" -- for two drivers
    music/seat/mirror positions -- for two drivers
    when should the doors lock themselves?
    what should the remote key fob do?
    when the driver opens his/her door, how should other doors react?
    etc.
Plus, of course, the engine related settings...


Re: What is your strategy to save non volatile data?
On 11/08/16 16:53, Don Y wrote:
Quoted text here. Click to load it

Yes, exactly.  I find this to be quite a common situation - but we all  
agree that different systems have different requirements and therefore  
different "best" solutions.

Quoted text here. Click to load it

Usually in such cases, it is still easy enough to have everything in one  
structure.  At its simplest, all those different state variables  
scattered around the program can be fields of the one state struct.

Quoted text here. Click to load it

Indeed.


I've done something like that too (that's sort of what I meant be a log  
structure - though I didn't describe it in detail).  It worked well for  
a flash based system - once one sector is getting a bit full, I ran a  
garbage collection pass by running through the buffer copying the latest  
parameters into a new sector, and then erased the old buffer with the  
outdated values.

Quoted text here. Click to load it


Re: What is your strategy to save non volatile data?
Il 11/08/2016 23:07, David Brown ha scritto:
 > On 11/08/16 16:53, Don Y wrote:
 >> On 8/11/2016 7:12 AM, David Brown wrote:
 >>
 >>> But for the common case of storing a small set of parameters, I usually
 >>> work with a struct definition.  Along with the parameters in  
question, I
 >>> include a "magic number" for identification, a "parameter structure
 >>> version number" (very useful for upgrading the parameter set if a later
 >>> version of the firmware uses more parameters), an update serial number,
 >>> a CRC check field, and some space reserved for future additional
 >>> parameters.
 >>
 >> To be clear, you are wrapping *all* of your parameters into a single
 >> "parameters_to_be_saved" struct.
 >>
 >> This can work if they are "configuration parameters" -- data that are
 >> consulted when setting up the application, restarting it, etc.
 >
 > Yes, exactly.  I find this to be quite a common situation - but we all
 > agree that different systems have different requirements and therefore
 > different "best" solutions.
 >
 >>
 >> This is different than trying to "save application/process state"
 >> across power outages.  In that case, the "state" may be scattered
 >> through many variables dispersed in memory.  And, more difficult to
 >> ensure catching a "consistent" snapshot.
 >>
 >> I.e., if *this* variable is changed, are there any others whose state
 >> must be reconciled *as* it is being preserved?
 >
 > Usually in such cases, it is still easy enough to have everything in one
 > structure.  At its simplest, all those different state variables
 > scattered around the program can be fields of the one state struct.

"Everything in one structure" means you have a global (as the scope in  
C) struct definition that is accessed by every piece of code in the  
project, even if they are very different section of the project.

In the past I used this approach with a global struct container:
   /* In one .h file */
   struct settings_s {
     uint8_t backlight_level;
     enum DisplayTheme theme;
     uint16_t psu_output_voltage_calibration;
     char password[7];
     ...
   };
   extern struct settings_s settings;

   /* In one .c file */
   struct settings_s settings;

   /* In display module */
   backligh_set(settings.backlight_level);

Every module that manages one or more persistent parameters should know  
the presence of a global settings variable with an exposed type  
definition (struct settings_s).
I think this is a bad approach that increase the coupling between  
modules instead of decreasing it as it should be.


 >> [It's unclear what the OP is addressing]
 >
 > Indeed.

Mainly I'm interested in saving configuration parameters/settings (color  
theme, calibration data, backlight level, and so on).
They are parameters that are changed by the user not so frequent.


 >>> Typically I have a statically allocated struct in ram, and two  
copies in
 >>> eeprom or flash.  On startup, I check both non-volatile copies to see
 >>> which is the latest version (highest serial number) that has a valid
 >>> CRC, and copy that into RAM.  If the parameter structure version number
 >>> does not match the current firmware requirements, I upgrade the
 >>> parameters.
 >>
 >> I've used a circular buffer into which "tagged" parameters are placed.
 >> On restart, I "play back" the buffer to restore the latest versions of
 >> the parameters encountered, there.  This lets me mix snapshots and
 >> incremental updates in the same mechanism.
 >
 > I've done something like that too (that's sort of what I meant be a log
 > structure - though I didn't describe it in detail).  It worked well for
 > a flash based system - once one sector is getting a bit full, I ran a
 > garbage collection pass by running through the buffer copying the latest
 > parameters into a new sector, and then erased the old buffer with the
 > outdated values.
 >
 >>
 >> My current application is effectively a hybrid approach as I'm
 >> effectively restoring "application state" and not "configuration data"
 >> (as the system can be considered to be constantly in a state of
 >> reconfiguration).  So, I let the RDBMS do the "log playback" from
 >> its WAL.
 >>
 >> In that way, I don't have to micromanage "what's important" *in*
 >> the application(s) but, rather, leave that to the RDBMS.
 >>
 >> ("Is this datum something that should be preserved across reboots?")
 >


Re: What is your strategy to save non volatile data?
On 8/11/2016 3:17 PM, pozz wrote:

[much elided]

Quoted text here. Click to load it

No.  It only has to be *effectively* global.

E.g., you can have a global "update_parameter()" that takes a
"key" (name for a parameter -- this could be a literal string
*or* just a small integer) and a parameter value (or pointer to
a parameter value).  Update_parameter() then moves the "value"
into the referenced portion of the "parameters to be saved struct".

In this way, the definition of the struct can remain opaque.
The same sort of "accessor" can be used to retrieve those
parameters from the struct -- without exposing the struct
or the manner in which the individual parameters are "encoded" (!)
in the struct.

E.g., you can pack a bunch of (potentially unrelated) parameters
into a bitfield within that struct.  Data coming *into* it gets
massaged to manipulate the appropriate fields.  Likewise for data
coming *out*.

Finally, because this represents a monitor, of sorts, you can ensure
atomic operations on that "data to be preserved" -- up to and including
an action that automatically saves the struct to persistent store
when power is failing (the rest of the time, "updates" happen to
the "in-RAM" copy of the struct so you don't incur the costs of
updating FLASH more often than necessary)

Quoted text here. Click to load it

But, are they changed in a "configuration mode" or, rather, as part
of the normal use of the device?  E.g., if I change the time on my
clock, I don't really enter a special "configuration mode" to do so.
Nor if I change the time zone, alarm settings, etc.  The clock
continues to *be* a clock while these persistent changes are happening.

See my other example re: automobile audio system (and, think of the
similar situation for most other "digital appliances" -- your TV
remembers the channel, volume, picture settings, etc. WITHOUT you
ever having to enter "configuration mode")


Re: What is your strategy to save non volatile data?
Il 12/08/2016 00:30, Don Y ha scritto:
Quoted text here. Click to load it

You use update_parameter() function to only move the value from the  
variable in RAM (managed by the module that really uses that variable)  
in another place in RAM (the struct that will be saved in a persisten  
way), right?

In this case, exactly who call update_parameter() and when?


Quoted text here. Click to load it


Re: What is your strategy to save non volatile data?
On 8/12/2016 12:02 AM, pozz wrote:
Quoted text here. Click to load it

*You* define the semantics.

I.e., each "update_parameter()" *could* cause that parameter to be
flushed to FLASH, checksum updated, etc.

Or, it could simply cause a cached copy of the appropriate parameter
"member" in the struct to be updated.  Some *other* mechanism could
cause the struct to be flushed to FLASH (e.g., update_parameter(FLUSH))
A lot depends on how often you are twiddling those parameters AND
the hardware that forms your persistent store.

E.g., if you have BBSRAM available, there is little cost to keeping
that struct *in* the BBSRAM.  So, an "update" can tweek the struct
member and update the checksum in one shot.  No "slow writes" to
worry about or limited media durability.  And, now your power
fail handling is localized to *that* routine:
    take lock
    mark struct as inconsistent
    update member
    compute checksum
    mark struct as self-consistent
    release lock

If power fails at any time OTHER THAN while the lock is held, the
struct is intact -- as indicated by the "consistent" marking.
Otherwise, the contents are "suspect".  (some, none or all of them
may be dubious -- depends on how you've encoded them)

Quoted text here. Click to load it

Anyone who wants to update one of these "persistent parameters".

E.g., the radio in my car would update "current_station" each
time I change stations.  In this way, my most recent selection would
be preserved if I turned the ignition off unceremoniously.

Alternatively, a cached/shadow copy of the "current station"
parameter could be maintained in "normal" RAM.  And, when the
ignition is turned off, the shutdown procedure could quickly
issue the update_parameter() calls for all pertinent parameters.
It depends on how much "work" you have to do and how much
time you have to do it (as well as the hardware capabilities
alluded to previously).

Re: What is your strategy to save non volatile data?
On 12/08/16 00:30, Don Y wrote:
Quoted text here. Click to load it

First, /don't/ use strings for that sort of thing unless you have no
other choice.  And don't use arbitrary integers.  Use an enum, that is
defined in one place.  That way you get it right.  Your code refers to
the parameters using sensible names, rather than numbers, you don't get
overlap or gaps (unless you want them, of course), and you can easily
change things when you need to.  And when one part of your program
writes parameter "backgroundColour" and another part reads parameter
"backgroundColor", you get a compile-time error rather than weird
behaviour at run-time.

But passing the data back and forth through functions like
"update_parameter" and "get_parameter" is an alternative to accessing
the struct directly, and can be easier to manage.  It is certainly
convenient if you want to trigger automatic saving of parameters.

Quoted text here. Click to load it

Yes.


That sounds like a complication that is unlikely to be worth the effort,
but it's possible.  In many cases its enough to simply say "all
parameters are 32 bits - it's up to the app if this is 4 characters, a
float, an integer, etc.".

Quoted text here. Click to load it

Also true - it's always important to make sure your data is consistent
at all times.



Re: What is your strategy to save non volatile data?

Quoted text here. Click to load it

But having it all defined in one place is basically the situation you
started with, with the global struct.

I have tried putting the non-volatiles in a special section, which then
gets written out after a change. That avoids the global dependencies but
adding a new variable then messes up the ordering and corrupts any
stored struct copy. (E.g. a firmware update with new settings).

Besides as I think you said usually something needs access to everything
anyway, like a UI or diagnostics module.

Quoted text here. Click to load it

With the single struct idea I periodically run a checksum to detect a
change.

Quoted text here. Click to load it

--  

John Devereux

Re: What is your strategy to save non volatile data?
On 12/08/16 14:14, John Devereux wrote:
Quoted text here. Click to load it

It is similar, but there is less shared detail.  And if it all helps you
catch mistakes at compile-time rather than run-time, then it is a good
thing.

Quoted text here. Click to load it

Yes, that's a danger with such a solution.  You can improve consistency
by making each section have a different name (such as ".settings.123" or
".settings.backlight") and then using KEEP(*(SORT(.settings.*))) in your
linker file.  At least that way you get the same order each time.
However, that won't help if one section grows and therefore changes the
addresses of all the other sections.

Quoted text here. Click to load it

I am not sure if I actually /said/ that, but I did say that if your
backlight module needs to preserve information, and the settings module
handles this preservation, then some coupling between the two modules is
unavoidable.

Quoted text here. Click to load it

There are (as always!) many ways to handle this sort of thing - a
periodic check can be a good way to do it.



Site Timeline