Weird code crash

Sep 14 14:17:03 systemd[1]: Created slice system-systemd\x2dfsck.slice. Sep 14 14:17:03 systemd[1]: Listening on fsck to fsckd communication Socket. Sep 14 14:17:04 systemd-fsck[109]: e2fsck 1.46.2 (28-Feb-2021) Sep 14 14:17:04 systemd-fsck[109]: rootfs: clean, 51075/932256 files,

Pretty sure it's a standard, my arch install has it set.

If there are any corrupted files, diagnosing any problems they cause will be difficult. I strongly recommend re-installing.

Regards, Dave Hodgins

SIGABRT is a problem in your code. If you aren't seeing stuff in the kernel log then it almost certainly isn't a hardware fault. It is a very special skill to have a hardware fault without spewing lots of stuff there.

Post the code somewhere and someone can take a look. Otherwise you need to use the development tools available to you to debug the problem.

Theo

Maybe not applicable in this situation, but if something deleted the file between the time of the scan and the time of the fopen call, it might/would not exist.

Again, I agree with Theo. Reported behavior is not really consistent with a hardware fault.

Reported behavior is also not consistent with a corrupt filesystem.

Most likely a bug in your program then.

It’s extremely good at identifying memory corruption even in cases where that doesn’t immediately lead to a crash; that’s what it’s for. But if it doesn’t, you leave it running until the crash happens.

Up to you, of course, whether you use the tools available, or debug with one hand tied behind your back.

• The Natural Philosopher snipped-for-privacy@invalid.invalid | On 14/09/2023 16:29, Ralf Fassel wrote: | > * The Natural Philosopher snipped-for-privacy@invalid.invalid | > | One possibility is that it is opening and reading a file at the | > | precise time another process is writing it...in both cases the read | > | and write | > | operations are atomic and done with C code. | >>

| > | READ | > | ==== | > | fp=fopen(fullname, "r"); | > | len=fread(filbuf,1,255,fp); // read entire file | > Check for fp != NULL is missing here in this example code before | > fread(). If this also in the production version, it might be a problem | > if the file is not accessible for any reason. | > R' | Ralf, I already put that in this morning, re compiled the code and | after an hour, it crashed again.

| The filename is built by scanning a directory so the filename must exist.

That assumption does not hold. Since scanning and opening are separated by a time gap (albeit a 'small' one), there is a non-zero chance that the file vanished between scan and open.

Further possibilities:

- how is 'filbuf' used after the fread()? If you use it as C-string, make sure it is 0-terminated (fread() won't do that for you). Maybe use fgets(3) instead?

| I am leaning towards possibly a cracked solder joint or board.

Well, since the Raspi is cheap, that should be easily checked by simply using another one. I bet 1 beer that it is *not* a cracked board, since with that many more processes should run into trouble, not only this particular one.

R' (.sig not from me .-)

If it persists I may do that, but now it is been rock steady for 20 hours.

The actual code has been replaced because I recompiled it anyway, but the problem persisted after that.

Then I twisted the board a bit, and now it hasn't failed since, No guarantees of course.

Does anyone else remember Tracy Kidder's 'Soul of a New Machine'* where they had a wire wrapped backplane on the prototype and a strange intermittent bug? And the director came in, twisted the backplane and the bug instantly reappeared?

One of the more curious 'bugs' I encountered was early in my software career, when code that I wrote suddenly went crazy, in a way in which the actual software as written could not possibly have caused. And only on one machine, equipped with a custom video capture card. We removed the card, but it made no difference.

I then compared the code on the machine with the code as compiled. Two bytes were FFH

I burned a new floppy and transferred the code again, and the code ran correctly.

Then we reinstalled the video card. The code ran correctly. Then we copied over the code again with the video card installed. The code again was corrupted.

Then the hardware guys looked at the address decide in the video card. It was a mass of gates one after the other. The total delay was well out of spec. It dawned on us that what was happening was that the DMA controller on the floppy was using bus addresses that were being decoded by the card, and then the IO request came along to access the floppy and those addresses were still on the bus as far as the sluglike video card was concerned, so it grabbed the data bus and shoved FFH on it.

Hardware is not perfect. That is the lesson. And chasing software when its really hardware is a losing game.

Anyway, I have in reserve all the great techniques suggested, but for now I am playing a wait and see game to see if any pattern emerges. My experience suggests that the same code running a loop in the same memory wont crash and burn unless there is a malloc/free mismatch, and that happens fairly quickly and shows in 'top'.

This kind of weird utterly asynchronous behaviour is often hardware. And. since I trod on the bloody PCB, I may simple get another one and test that. It doesn't need to be installed till winter. There is time. And my PCB design for the relay and PSU module isn't back from China yet...

I looked it up, it merely specifies the order I think, so you are right in practice.

Exactly. That is a possibility, which I have now covered. It made no difference.

In practice the write code that *replaces* the file is very simple. It is fopen( "w") immediately followed by fwrite()

without knowing the exact code involved with the fopen("w"); I cant say if that actually deletes the file and creates a new one, or merely truncates it to zero length, or indeed just opens it and trips the length *after* the new data is written..

Very definite.

Are you sure about that?

If you aren't seeing stuff in the kernel

Even a corrupted bit in a ram disk?

I can post the code, but it may not help. You need the whole system including the perpiherals that write, to the daemon that writes the data files that the daemon that crashes reads.

At the moment it is behaving perfectly. Without a reproducible bug I can see no point in using a debugger.

Well that is an option for sure.

Tell me in what way a corrupted - say - libc file, or a faulty bit of memory would show up in the kernel logs?

The problem is that this thing is looping very frequently. loop() { while (1) { int i; readThermometers(); readZones(); readOverrides(); readTimerData(); setRelayState(); setRelays(); usleep (1120000); } }

And that means thousands of faultless iterations in a day.

So this bug ( if it is a bug) is a one in a million or worse.

I suppose I could make the thing loop ten times a second (or even faster) and see if it happens more often..

its not as though its chewing up CPU...

The problem I have is that these crashes only recently started happening: prior to that the code ran for days. And two things happened, a massive brownout, and then a full power cut, and I trod on it.

And I made systemd start it...

I see it crashed again last night, again with zero errors apart from SIGABRT...

I will start it manually and cut systemd out.

dir = opendir(VOLATILE_DIR);

if(!dir) return; while ((dp = readdir (dir)) != NULL) { filename=dp->d_name; // skip known bollocks if(!strcmp(filename, "." ) || !strcmp(filename, ".." ) || !strcmp(filename, "relays.dat" )) continue; // construct full path sprintf(fullname,"%s/%s",VOLATILE_DIR,filename); stat(fullname,&stats);// get tfile times if(time(NULL)-stats.st_ctime >1800) // skip files older than half an hour continue; len=strlen(filename); if(strncmp(filename+len-4, ".dat",4)) // .dat file but not relays.dat continue; fp=fopen(fullname, "r"); if(fp==0) //file has disappeared? continue; len=fread(filbuf,1,255,fp); if(len==0) // file has zero length goto baddata; filbuf[len]=0; if(len=strncmp(filbuf,"ZONE",4)) //supposed to reject a file whose contents do not start with ZONE goto baddata; // looking very much like a temperature file i=(int)filbuf[4] -'1'; // this is our zone from "ZONE2" etc. 1-4 is zone but index is 0-3 so subtract '1' p=strstr(filbuf,"\n"); if(p) { p++; if(q=strstr(p,"\n")) { *q++=0; thermometers[i].name=strdup(p); // make a copy of the name and attach it to our thermometer structure p=q; } else goto baddata; // now to fetch the temp data. if(q=strstr(p,"\n")) { *q++=0; thermometers[i].temp=atof(p); p=q; } else goto baddata; // what's left is the voltage. To hell with any crap after it thermometers[i].battery=atof(p); } baddata:fclose(fp); } // end of directory scan loop

Very dependent on the nature of the corruption. But you’ve already told us there’s nothing in the kernel logs.

Anyway, not responsible for advice not taken.

Well, it could be a cosmic ray. The Pi doesn't have ECC memory to it's possible to bit-flip in RAM or storage without it noticing. I don't know which part of the galaxy you inhabit, but cosmic rays are rare enough down here that random bit flips like this don't happen often - ballpark once a year for a server (which has a much greater surface area to absorb them than a Pi).

It is also possible to be marginal on signal integrity for PCB interconnect, but that would mostly be a design fault: either they all work or many of them don't. Since we don't have a lot of people complaining of the same problem, we can assume the design is not marginal in that respect.

If computers were that unreliable they would be failing all the time - and we'd fit ECC to everything. That they aren't suggests bit-flip corruption isn't a problem. In general random bit-flip errors are not a statistically major source of crashes, unless you're running a hyper-redundant mainframe and have eliminated all the other sources.

What are a well-known class of bugs are concurrency/timing races and memory safety violations. Which is odds-on what's happening here, especially given we've already picked up on potentially risky code like failing to check for NULL from fopen().

That would be a useful thing to try.

Most of those things would cause it to fail hard (ie not power up), rather than have a very rare random fault.

It is possible that being run from systemd changes the timing or environment that provokes the fault in some way, but I doubt it would be the cause of the fault.

Theo

Possible write overrun here.

Possible read under-run here. (But if it crashes then you’d expect SIGSEGV rather than SIGABRT, so that’s probably not the issue.)

I don’t think the declaration of filbuf has been posted, so there’s a possible write overrun if it’s less than 255 bytes.

You trust the contents of 'outside'-files very much, do you? ;-) I don't know who can create files in the directory you're scanning, but not *assuring* the input you expect is another possible cause for problems...

• The Natural Philosopher snipped-for-privacy@invalid.invalid | > Further possibilities: | > - how is 'filbuf' used after the fread()? If you use it as C-string, make | > sure it is 0-terminated (fread() won't do that for you). Maybe use | > fgets(3) instead? | > | dir = opendir(VOLATILE_DIR);

| if(!dir) | return; | while ((dp = readdir (dir)) != NULL) [looks good, error checks for stat() et al couldn't hurt]

--<snip-snip>-- | if(len=strncmp(filbuf,"ZONE",4)) //supposed to reject | a file whose contents do not start with ZONE | goto baddata; | | // looking very much like a temperature file | i=(int)filbuf[4] -'1'; // this is our zone from | "ZONE2" etc. 1-4 is zone but index is 0-3 so subtract | '1'

The access of filbuf[4] is ok (since you checked that there are at least

| p=strstr(filbuf,"\n"); | if(p) | { | p++; | if(q=strstr(p,"\n")) | { | *q++=0; | thermometers[i].name=strdup(p); // | make a copy of the name and attach it | to our thermometer structure

Memory leak if thermometers[i].name already contains something.

Other than that, I really would have it running under a debugger or valgrind, since then *if* it crashes, you *know* *where* in your code it crashes.

Good luck hunting! R'

I’ve seen one inarguable random bit flip in several decades. In that case the behavior was deterministic - chiark’s /bin/ls had got a single-bit error, and caching meant it crashed _every_ time anyone ran it.

Maybe TNP has taken a trip to Sizewell?