In *regulated* industries (FDA, aviation, etc.), products are
> validate (hardware and software) in their "as sold" configurations.
> This adds constraints to what can be tested, and how. E.g.,
> invariants in code need to remain in the production configuration
> if relied upon during validation.
>
> But, *testing* (as distinct from validation) is usually more
> thorough and benefits from test-specific changes to the
> hardware and software. These to allow for fault injection
> and observation.
>
> In *unregulated* industries (common in the US but not so abroad),
> how much of a stickler is the validation process for this level > of "purity"?
>
> E.g., I have "test" hardware that I use to exercise the algorithms
> in my code to verify they operate as intended and detect the
> faults against which they are designed to protect. So, I can inject
> EDAC errors in my memory interface, SEUs, multiple row/column
> faults, read/write disturb errors, pin/pad driver faults, etc.
>
> These are useful (essential?) to proving the software can
> detect these faults -- without having to wait for a "naturally
> occurrence". But, because they are verified/validated on non
> production hardware, they wouldn't "fly" in regulated
> markets.
>
> Do you "assume" your production hardware/software mimics
> the "test" configuration, just by a thought exercise
> governing the differences between the two situations?
>
> Without specialty devices (e.g., bond-outs), how can you
> address these issues, realistically?
>
I think perhaps this is confusing systems testing with product testing. You need to make a clear distinction between the two.
Systems testing is about checking that a /design/ is correct. Much of that is usually about software testing, but it applies to hardware too. This will often be done using modified hardware so that you can, for example, inject /realistic/ faults and check that the hardware and software function as expected. Depending on the application, you might also run test boards at high temperatures or otherwise abuse them to confirm the design.
Production testing is about ensuring that the products made are correct according to the design. You don't check that the memory works, or the ECC handler works - you check that you have correctly mounted and soldered the memory chip and that the memory chip supplier has checked for production faults.
There are some products where the likelihood of developing partial faults in the field are high and the consequences of that are serious but it is useful to be able to keep a partially failed system in action. There are also products with user-serviceable parts. Then it is often helpful to have some kind of self-test to identify failing subsystems.
Unfortunately, in some regulated markets, or for some types of "safety certification", the rule-makers don't understand how this works. The result is that they insist on extra fault-checking hardware and/or software that actually decreases the total reliability of the system, and introduces new parts that in themselves cannot be checked (systematically, in production, and/or in the field).
How do you deal with it? You follow the rules, even though some of them were written by muppets.