Re: Diagnostics

However, only a minor percentage of all devices must comply with such safety regulations.

As I understand it, Don is working on tech for "smart home" implementations ... devices that may be expected to run nearly constantly (though perhaps not 365/24 with 6 9's reliability), but which, for the most part, are /not/ safety critical.

WRT Don's question, I don't know the answer, but I suspect runtime diagnostics are /not/ routinely implemented for devices that are not safety critical. Reason: diagnostics interfere with operation of <whatever> they happen to be testing. Even if the test is at low(est) priority and is interruptible by any other activity, it still might cause an unacceptable delay in a real time situation. To ensure 100% functionality at all times effectively requires use of redundant hardware - which generally is too expensive for a non safety critical device.

YMMV. George

Maybe, if you mean domain specific regulations. But there are general EC directives. One may spent money on lawyers and research to conclude that some protections are not required by law. Or one may implement things like for regulated domain. On small scale the second is likely to be cheaper.

Anyway, I do not know if there is anything specific about washing machines, but software for them is clearly written as if they were regulated. The same for ovens, heaters etc.

IMO, "smart home" which matter have safety implications. Even if they are not regulated now there is potential for liabilty. And new requlations appear quite frequently.

IIUC at low levels requirements are not that hard to satisfy, especially since in most cases non-working device is deemed "safe".

TI for several years has nice processors with two cores, which are almost in sync, but one is something like one cycle behind the other. And there is circuitry to compare that both cores produce the same result. This does not cover failures of the whole chip, but dramaticaly lowers chance of undetected erros due to some transient condition.

For critical functions a car could have 3 processors with voting circuitry. With separate chips this would be more expensive than single processor, but increase of cost probably would be negligible compared to cost of the whole car. And when integrated on a single chip cost difference would be tiny.

IIUC car controller may "reboot" during a ride. Intead of rebooting it could handle work to a backup controller.

Well, if device matters then there is implied liabilty and nobody want to admit doing bad job. If device does not matter, then answer to the original question also does not matter.

What to test is really domain-specific. Traditional thinking is that computer hardware is _much_ more reliable than software and software bugs are major source of misbehaviour. And among hardware failures transient upsets, like flipped bit are more likely than permanent failure. For example, at low safety level you may assume that hardware of a counter generating PWM-ed signal works correctly, but you are supposed to periodically verify that configuration registers keep expected values. IIUC cristal osciators are likely to fail so you are supposed to regularly check for presence of the clock and its frequency (this assumes hardware design with a backup clock).

I do not know how PLC manufactures implement checks. Small PLC-s are based on MCU-s with static parity protected RAM. This may be deemed adequate. PLC-s work in cycles and some percentage of the cycle is dedicated to self-test. So big PLC may divide memory into smallish regions and in each cycle check a single region, walking trough whole memory.

I am affraid that usually inconsequential failure gets promoted to complete failure. Before 2000 checking showed that several BIOS-es "validated" date and "incorrect" (that is after 1999) date prevented boot.

Historically OS-es had a map of bad blocks on the disc and avoided allocating them. In principle on system with paging hardware the same could be done for DRAM, but I do not think anybody is doing this (if domain is serious enough to worry about DRAM failures, then it probaly have redundant independent computers with ECC DRAM).

Everything here is domain specific. In cheap MCU-based device main source of failurs is overvoltage/ESD on MCU pins. This may kill the whole chip in which case no software protection can help. Or some pins fail, sometimes this may be detected by reading appropiate port. If you control electic motor then you probably do not want to sent test signals during normal motor operation. But you are likely to have some feedback and can verify if feedback agrees with expected values. If you get unexpected readings you probably will stop the motor.

The chips that I mentioned use static memory with ECC. Of course, ECC circuitry may fail. There may be error undetected by ECC. The two cores may have the same error or comparison circuitry may fail to detect the difference. Each may happen, but each is much less likely to happen than simple transient error.

Detecting such thing looks easy. Recovery is tricky, because if you have spare FET and activate it there is good chance that it will fail due to the same reason that the first FET failed. OTOH, if you have propely designed circuit around the FET, disturbance strong enough to kill the FET is likely to kill the controller too.

Use 3 (or more) and voting. Of course, this increases cost and one have to judge if increase of cost is worth increase in safety (in self-driving car using multiple sensors looks like no-brainer, but if this is just an assist to increase driver comfort then result may be different).

Sure. But for common failures or serious failures having non-negligible pobability redundancy may offer cheap way to increase reliability.

It does not matter if handover _always_ works. What matter is if system with handover has lower chance of failure than system without handover. Having statistics of actual failures (which I do not have but manufacturers should have) and after some testing one can estimate failure probablity of different designs and possibly decide to use handover.

You may have tests essentially as part of normal operation. Of course, if you have single-tasked design with a task which must be "always" ready to respond, then running test becomes more complicated. But in most designs you can spare enough time slots to run tests during normal operation. Tests may interfere with normal operation, but here we are in domain specific teritory: sometimes result of operation give enough assurance that device is operating correctly. And if testing for correct operation is impossible, then there is nothing to do, I certainly do not promise to deliver impossible.

That brings up one of the critical points about any kind of runtime diagnostics - what do you do if there is a failure? Until you can answer that question, any effort on diagnostics is not just pointless, but worse than useless because you are adding more stuff that could go wrong.

I think bad or useless diagnostics are a more common problem than missing diagnostics. People feel pressured into having them when they can't measure anything useful and you can't do anything sensible with the results.

I have seen first-hand how the insistence of having all sorts of diagnostics added to a product so that it could be "safety" certified actually result in a less reliable and less safe product. The only "safety" they provided was legal safety so that people could claim it wasn't their fault if it failed, because they had added all the self-tests required by the so-called safety experts.

You /could/ do that, but what is the point?

What are you checking for? What is the realistic likelihood of finding a problem, and what are the consequences of such a problem? How do you test your test routines - are you able to simulate the problem you are testing in a good enough manner? What are the circumstances that could lead to a fault that you detect with the tests but where you would not already see the problem in other ways? Is it realistic to assume that your diagnostic test and reporting systems are able to run properly when the this problem occurs? If some kind of problem actually occurs, will your tests realistically identify it?

/Those/ are the kinds of questions you should be asking before putting in some kind of tests. They are the important questions. Asking "why can't I do a test now?" is peanuts in comparison.

Each item above may contribute to a significant loss. And there could push to litigation (say by a consumer advocacy group) basically to establish a precedent. So, better have record of due diligence.

First, I mean relevant hardware, that is hardware inside a MCU. I think that there are strong arguments that such hardware is more reliable than software. I have seen claim based on analysis of discoverd failures that software written to rigorous development standars exhibits on average about 1 bug (that lead to failure) per

I another place I wrote the one of studies that I saw claimed that significant number of errors they detected (they monitored changes to a memory area that was supposed to be unmodifed) was due to buggy software. And DRAM is special.

You are supposed to regularly verify sufficiently strong checksum.

First, such checking is not my idea, but one point from checklist for low safety devices. Registers may change due to bugs, EMC events, cosmic rays and similar.

ECC is part of solution. It may reduce probability of error so that you consider them not serious enough. And if you really care you may try to increase error rate (say by putting RAM chips at increased temperature) and test that your detection and recovery strategy works OK.

Hi Don,

Same ol', same ol'. Nothing much new to report.

If you "know" a priori when some component will be needed, then you can do whatever you want when it is not. The problem is that many uses can't be easily anticipated.

Which circles back to testing priority: if the test is interruptible and/or resumeable, then it may be done whenever the component is available ... as long as it won't tie up the component if and when it becomes needed for something else.

If you're testing memory pages, most likely you are tying up bandwidth in the memory system and slowing progress of the real applications.

Also because you can't accurately judge the "minimum" needed. BSD and Linux both have this problem where a sudden burst of allocations exhausts the pool of zeroed pages, forcing demand zeroing of new pages prior to their re-assignment. Slows the system to a crawl when it happens.

Because then you are watering (however briefly) when it is not expected. What if there was a pesticide application that should not be wetted? What if a person is there and gets sprayed by your test?

Properly, valve testing should be done concurrently with a scheduled watering. Check water is flowing when the valve should be open, and not flowing when the valve should be closed.

Automotive is going the way of aircraft: standby running lockstep with the primary and monitoring its data flow - able to reset the system if they disagree, or take over if the primary fails.

The point here is that there is no "one fits all" philosophy you can follow ... what is proper to do depends on what the (sub)system does, its criticality, and on the components involved that may need to be tested.

Yes.

Actually due diligence /can't/ insulate a manufacturer if the issue goes to trial. Members of a jury may feel sorry for the litigant(s), or conclude that the manufacturer can afford whatever they award ... or maybe they just don't like the manufacturer's lawyer.

Unlike judges, juries do /not/ have to justify their decisions, Moreover, in some US juridictions, the decision of a civil case need not be unanimous but only that of a quorum.

So they created the "class action", where all the litigants individually may have very small claims, but when put together the total becomes significant.

On Fri, 18 Oct 2024, Don Y wrote: "The costs of litigation are reasonably high."

Hi Don,

Court cases' costs are unreasonably high.

On Fri, 18 Oct 2024, Don Y wrote: "> To ensure 100%

Apparently, there is noise about incorporating such hardware into

Hi Don,

We were lectured by a person whom an expensive-German-car manucfaturer paid to help make cars, but he told us that this rich German company did not pay him enough to buy 1 of these cars so he drives a cheap car from a different company. He might endanger a client's clients' lives, but less so his own (though of course a malfunctioning German car could crash into him).

On Sat, 19 Oct 2024, Don Y wrote: "[. . .] [. . .] "Well, MAYBE Publishers' Clearinghouse came to my house to give me that oversized/cartoon check but the doorbell MIGHT not have rung. So, I want '$3000/week for life' as compensation...""

Hi Don,

Grady Booch confessed at an International Conference on Software Engineering in 2000 that his anti-engineering UML policy produced a door bell which was not rung, so a friend spent money on a cellphone call to him to open a door! Fools pretend that Grady Booch is an engineering hero!

Mister Fabio Bertella (then at Optisoft Srl, Via A. Bertoloni, 15,

Regards.

Courts need not be involved ... just lawyers.

A few weeks ago there was an article in Forbes magazine saying that the /average/ billing by lawyers at moderately sized firms in the US is now $1500..$1800/hr, and billing by lawyers from top firms is now $2600..$3000/hr.

Litigating a product liability case routinely takes hundreds to thousands of hours depending. True, many of those hours will be for paralegals unless/until the case gets to arbitration or court - but even small firms now bill their paralegals at over $200/hr.

On Sun, 20 Oct 2024, George Neuner wrote: "A few weeks ago there was an article in Forbes magazine saying that the /average/ billing by lawyers at moderately sized firms in the US is now $1500..$1800/hr,"

Yikes!

The point is that you can not eliminate all bugs. Rather, you should have simple code with aim of preventing "cost" of bugs.

The tests where run on Linux boxes with normal memory protection. Memory protection does not prevent troubles due to bugs in priviledged code. Of course, you can think that you can do better than Linux programmers.

Using something like Linux means that you do not care about rare problems (or are prepared to resolve them without help of OS).

AFAICS the rule about checking image originally were inteded for devices executing code directly from flash, if your "primary truth" fails possibilities are limited. With DRAM failures one can do much better. The question is mainly probabilities and effort.

IIUC microwave, stove and furnace should be. In cell phone BMS should be safe and core radio is tightly regulated. Other parts seem to be at quality/reliability level of PC-s.

You clearly want to make your devices more reliable. Bugs and various events happen and extra checking is actually quite cheap. It is for you to decide if you need/want it.

Well, there is evidence that increased temperature inreases chance of errors. More precisely, expect errors when you operate DRAM close to max allowed temperature. The point is that you can cause errors and that way test your recovery strategy (untested recovery code is likely to fail when/if it is needed).

Good.

In a car you have signals from wheels and engine, you can use those to compute transmission ratio and check is it is expected one. Or simply have extra inputs which mountor FET output.

Matching images is now a stanard technology. And in this case "voting logic" is likely to be software and main trouble are possible bugs.

I doubt in general solutions. Various parts of your system may have enough common features to allow single strategy in your system. But it is unlikely to generalize to other systems. To put it differently, there are probabilites of various events and associated costs. Even if you refuse to quantify probabilites and costs your design decisions (assuming they are rational) will give some estimate of them.

Well, you write that you do not try to build high reliablity device. However device which correctly operates for years without interruption is considered "high availability" device which is a king of high reliablity. And techniques for high reliablity seem appropiate here.

Well, I always use no parity transmission mode. Standard way is to use checksums and acknowledgments. That way you know if transmission is working correctly. What extra info you expect from looking at detailed error info from UART?