Need help with PI PICO...

Mar 23, 2024 Last reply: 2 years ago 36 Replies

Ok, this is the one destined to be an oil level sensor and I have been working on getting a stable TCP/IP and Wi-Fi stack, which seems to have been achieved, as its talking reliably, albeit with delay, to my most remote Wi-Fi AP at a signal level generally around -87dbM. That is not, however the problem (although I thought it was). The problem seems to be that very very occasionally and as far as I can tell



*completely randomly*, it fails to return from the function listed below. Cargo culted from the module manufacturers application notes

This is for the ultrasonic transducer module. When it fails, all GPIO pins to and from the transducer module measure LOW.



static float get_distance() { int i; absolute_time_t start; absolute_time_t end; static int64_t us_delay; gpio_put(ULTRASONIC_OUT,0); sleep_us(2); //set output pin high gpio_put(ULTRASONIC_OUT,1); sleep_us(10); gpio_put(ULTRASONIC_OUT,0); //reset the input // wait for echo pulse start while(!gpio_get(ULTRASONIC_IN)) ; //read clock and store start=get_absolute_time (); //wait for echo pin to go low; while(gpio_get(ULTRASONIC_IN)) ; end=get_absolute_time (); //get clock difference us_delay=absolute_time_diff_us(start,end); //convert to float and return it as cm return (((float)(us_delay))*0.034/2); }



It would seem from the pin states that it gets permanently stuck in



while(!gpio_get(ULTRASONIC_IN)) ;



Which as understand it is waiting for the module (HCSR04) to *start* to send a pulse.



Now obviously infinite loops with no termination condition under fault conditions are poor code, but I am leaving it there until I understand why the code is in fact hanging. Someone online suggested that asynchronous interrupts may be the issue, but I cant see why or what interrupts to disable.



Can anyone cast any light on this one?



Or suggest a bug hunting methodology?


Assuming you have access to the source of gpio_get() instrument the inside of it with tracers (I'd use printf if there's anything listening to stdout - otherwise find somewhere to put breadcrumbs that you can see in real time (in ancient times I'd just watch the blinkenlights). Wait for it to lock up and see what it's doing.

Alternatively run it under strace or similar and wait for it to lock up or wait for it to lock up and attach gdb (you'll want to compile with -g for that).

There's three - hopefully one of them will shed some light.

Well I did. That's how I got this far. I know it enters the routine, but never leaves, and the lack of GPIO voltage suggest it is being stuck where it is.

My choice tree is between the GPIO out signal never being received by the ultrasonic module, or the GPIO in signal is being missed by the Pi PICO on account of possibly some interrupt masking its appearance until it is too late and its gone low again.

AIUI those are linux tools.

We are running bare metal-ish here. Back in the day I would have used a chip emulator with hardware break points. And a cost of hundreds of thousands.

It's odd, it may be something to do with short ultrasonic distances. I have the PCB just lolling around on the desk, and facing a wall a few inches away seemed to make it crash moire predictably

Instrument /Inside/ gpio_get().

Right so the next step is the inside of the routine.

Unix tools but yes.

Ah - no way to attach a debugger via the SDK ?

An ICE is always nice if someone else is paying :)

Hmm is there a minimum range spec ?

Can you scope it to see if the module is actually sending a pulse?

Is the pulse perhaps too short for the Pico to detect? eg if the loop or gpio_get() function took some time, it could be the signal goes 0-1-0 in the middle of a loop iteration and so the gpio_get() never sees it go 1.

Theo

gpio_get (X) for presumably memory mapped IO is going to be little more than return (iobase & (1<<X ) ? 1:0);

However I found something i had forgotten...

There may be but it would be more complex.

Precisely!

Not really in practical terms.

However on trawling the internet I discovered a conversation with someone else *on here* (c.s.r-pi) last year, who was finding that

*sleep_us(x)* was highly inconsistent for certain (small) values of x. Sometimes taking a day to end.

Further investigation reveals that in fact it really is a 'sleep' with the processor being put in low power mode and requiring an interrupt to 'wake it up'.

I haven't thought it through, but it could be that too long a delay in some sense might miss a complete set of send and return pulses, leaving the thing stuck. Especially on a very short range echo.

So last night instead of leaving it facing a wall, I left it facing the opposite wall, and it's still blinken away...

There is an alternative to sleep_us, that simply involves a tight processor loop watching the clock, that might work better for these very small delays.

I am going to do a statistical test on both methods at long and short ranges.

(Fortunately, I managed to remove the existing radio sensor from the oil tank and on shaking, could hear water sloshing around inside. After emptying out and drying n the oven, it started working again. I don't think +60°C made it very accurate as it then showed a full tank, but it seems to have settled down to a figure consistent with what a dipstick shows. So there is no urgency to get the replacement PI sensor working:-)

I am getting to really love these little PICOS, but there are a lot of quirks and bugs in the hardware and software.

Thank's for your input.

I could, but in fact it was easier to simply look at it in 'stuck' mode with a DVM.

As you can see from my last reply that is roughly where I am headed. Or similar. Lacking full ICE., its all a bit 'poke the black box with different sized sticks, and try and infer from what it does, what is happening inside it'

I think this must be where it sticks, because this is the only infinite loop with both input and output to the module in a low state, which is what I measured:

gpio_put(ULTRASONIC_OUT,1); sleep_us(10); gpio_put(ULTRASONIC_OUT,0); //reset the input // wait for echo pulse start while(!gpio_get(ULTRASONIC_IN)) ;

I.e that it (allegedly) sends a 10µs wide high pulse, and then waits for that to trigger a response from the unit, but that response never happens.

However that should not vary with the echo *delay*, and a longer target distance seems to improve things..

...unless, thinking a bit more, the pulse is so short it comes *and* goes inside that loop, as you suggested.. it certainly should *not* be, as even on a few cm of target distance, its hundreds of microseconds (i make it 58µs per cm roughly)

Or there never is a signal on the module output. You should write your code to cope with such problems - sample code/libraries often skips the error handling, but that does not mean *you* can if you want reliable operation.

Hook up an oscilloscope or logic analyzer to check what is the case. A cheap 8-channel USB logic analyzer is a great tool to have around when working bare metal:

formatting link
Using the Saleae software on these is *not* allowed by the license, but sigrok/pulseview work great:

formatting link

Nowadays, you would use a SWD debugger, since the debug/emulation logic is already integrated in the RP2040!

Something like the JTAG Hat on a Raspberry PI:

formatting link
or any SWD probe supported by OpenOCD (FTDI, CMSIS-DAP work fine), like the Raspberry Pi Debug Probe (which works for the RP2040, but lacks reset signals and support for voltages other than 3.3V):

formatting link
That sets you back about 20€, and gives you instant breakpoint/single step operation with full view of registers and memory contents.

cu Michael

Start a hardware timer when sending the pulse, and set up an interrupt on the input with the return pulse. In the interrupt handler, read the timer value and set a flag for the main routine that the measurement is complete.

If you add a timer interrupt to periodically send the start pulse, the whole measurement operation runs in parallel to whatever the main program is doing, without blocking anything.

cu michael

One of the challenges with embedded debugging is that printf can be a very slow thing, because it's spitting out characters on a slow UART. Adding printfs can thus screw up timing.

Other techniques include writing things to memory you pick up later (don't forget that string handling and especially sprintf formatting can be slow) and wiggling GPIOs (eg output a different number in each stage of the program).

sigrok is logic analyser software that works with cheap (<$10) capture hardware - seems it can now use a Pi Pico for that as well.

Theo

Why not use threads/timers, wait on a lock/semaphore rather than sleep.

But looking at PICO code samples, they commonly use sleep, so I'd be surprised if it was that bad.

Good point Pancho, but I was really looking for the simplest code possible. Interrupts can be tricky things on a platform whose architecture you do not understand completely. In any case it was a learnning curve I preferred not to negotiate if i didnt need to

I am veering away from that explanation, as with the test board located at some distance from any target, the problem has not reappeared.

I am beginning to think that it may be possible for the echo pulse to 'come *and* go' before the high level PICO code has got round to actually looking for it in the first place.

That is, some asynchrounous event in this sequence

gpio_put(ULTRASONIC_OUT,1); sleep_us(10); gpio_put(ULTRASONIC_OUT,0); //reset the input //if asynch event lasting more than 100uS occurs here... // wait for echo pulse start while(!gpio_get(ULTRASONIC_IN)) ; //then the low-high-low echo pulse will never be detected.

It is also not clear from the documentation whether it is the low to high, or the high to low sequence, that triggers the ultrasonic board.

If it is low to high, then there is an opportunity for an occasional very long delay in the

sleep_us(10);

to delay resetting the pulse until Elvis has left the building,. so to speak...

So I have tow things to do. Understand how the ES module works in terms of timings, and replace that sleep_us with a different delay mechanism. It's now been 24 hours with no lockup with a distant target...

OIL-SENSOR OIL-TANK

-85dBm

57.60cm 23.7°C 4.6V

I have noticed that with absoluteley no change in sensor location I get up to ± 0.5cm variation in delay.

Assuming the "oil" you're talking about is kerosene/heating fuel, the speed of sound is 1330 m/sec so 0.5cm takes 3.76 micro-sec. I don't know the specs for the PICO, but perhaps comparing that to the time it takes to execute your code will give you an answer. I'd guess (emphasis on guess) that +/- 0.5 cm is doing fairly well.

Oh the sensor is above the oil. Its just an electronic dipstick - measures distance to the oil surface.

It did over a full 30 hours on long delays. Ive modded the code

*slightly* and put it back on uber short delays. So far so good...

I would have thought it's measuring the distance to the surface of the oil from above the oil so it would be the speed of sound in air that matters 300m/s.

Correct, Mrs Shot. Anyway it's died within 30 minutes of going back on 'short echo'... So its definitely sensitive to that in some way.

I'll add more debug code tomorrow

Are you sure the sensor isn't malfunctioning as a result of being in oil vapour?

David

A timer isn't complicated, just a call back routine, and a semaphore. Interrupts are something an OS does, not me :o). I hate multithread code, but async handling of an external resource is one of the two main places I would use another thread.

I had a look at your code, it looks extraordinarily like a python example on Tom's hardware.

I'm not clear how many times it is succeeding vs failing, but I suspect you really need to bite the bullet and introduce timeouts/error handling, if it fails try again, average out multiple results. i.e. accept it as flawed and that results are statistical, like GPS.

In many ways the resilience code will be simple, because it is just normal code, rather than cargo culting a novel ultrasonic device.

You can investigate further, by recording fail stats, as well as distance stats.

Since it is operating on the desk in front of me, fairly sure :-)

I haven't let it anywhere near the oil tank yet. The plan is to have it installed by the fall. ready for next winter. So it is being hammered to check for problems *before* it gets to a cold wet inaccessible oil tank.

Oh. The manufacturers sample code is the source of ALL the 'examples' that other people publish as their own., I am just being honest :-)

Well the averaging out will happen anyway at the server side. I make the clients as simple as possible for resilience. In practice oil levels only change quickly if you have had the oil stolen overnight or if a supplier has filled the tank up, so gross deviations can have code exceptions, the rest would be the running average of maybe 20 samples. BUT it isn't inaccuracy that worries me per se, it's that it may be an indication of underlying timing issues.

In fact the code in either case is simple.

It is: send a 10µs pulse to the unit, wait for the echo pulse start , get the time, wait for the echo pulse end, get the time, subtract the difference.

What appears to be happening is that at short range the echo pulse never starts, or ends before the code is aware of it.

Failure is very very rare. I am sampling for test purposes once a second, and its usually an hour or more before it locks up.

I could simply turn the while loop into a for loop with a counter so that even if I got a null result it wouldn't lock up. Missing one sample is no big deal: just take another!

I am slightly curious as to how the PICO could miss what is a several hundred microsecond wide pulse.

So far I have managed to get stuff reliable without having to unpick the ARM interrupt pandora's box. I am keen to leave it closed. The LWIP stack was bad enough...:-)

Obviously interrupt on GPIO pin state would be the thing, but it would take some research to find out what the ISR was allowed to do in terms of library code that was re-entrant..

Join the Discussion

Have something to add? Share your thoughts — no account required.

Didn't find your answer?

Ask the community — no account required