edgetx icon indicating copy to clipboard operation
edgetx copied to clipboard

Potential race condition in ADC burst sampling

Open raphaelcoeffic opened this issue 4 years ago • 15 comments

We have observed lately a 2 reports of "twitches" on the Yaw or Roll axis which are then transfered as-is to receiver and later on to betaflight.

This is for the "Roll twitch", seen on a radio flown in mode 2 (Roll is the last of the 4 Axis to be sampled; other ADC channels are normally sampled afterwards as well): roll_twitch

And this is the "Yaw twitch", on a radio flown in an unknown mode (user has been asked): yaw_twitch

After some research, I discovered a possible cause for this: https://github.com/EdgeTX/edgetx/blob/cbced4a81c1261411be5306af2c4c06eb8f8825a/radio/src/targets/common/arm/stm32/adc_driver.cpp#L216-L221

This code looks to me like asking for a perfect storm: in case the DMA is not finished within the 10.000 checks, it will proceed to just switching DMA off, but will keep these results.

raphaelcoeffic avatar Jul 26 '21 12:07 raphaelcoeffic

There are not only the sticks been analog sampled, the are pots and sliders too, after the sticks. And then those are sampled 4 times per run. So it would be interesting to assess if such an issue does exist. We definitely need some data on this. Do we know on what type of radio this could possibly have happened?

3djc avatar Jul 26 '21 16:07 3djc

There are not only the sticks been analog sampled, the are pots and sliders too, after the sticks. And then those are sampled 4 times per run. So it would be interesting to assess if such an issue does exist. We definitely need some data on this. Do we know on what type of radio this could possibly have happened?

It’s not at all clear this is it. But we know for a fact these traces exist. And this is the only thing I could find in this direction.

for the radios, the first one is X9D+. The second could be TX16S. So both running on only one ADC burst (second burst is VBat, and not waited for).

raphaelcoeffic avatar Jul 26 '21 17:07 raphaelcoeffic

I have run a test of 150k samples, max value was 251, so it would require more than 30 times more to exit prematurely

3djc avatar Jul 26 '21 18:07 3djc

There are not only the sticks been analog sampled, the are pots and sliders too, after the sticks. And then those are sampled 4 times per run. So it would be interesting to assess if such an issue does exist. We definitely need some data on this. Do we know on what type of radio this could possibly have happened?

It’s not at all clear this is it. But we know for a fact these traces exist. And this is the only thing I could find in this direction.

for the radios, the first one is X9D+. The second could be TX16S. So both running on only one ADC burst (second burst is VBat, and not waited for).

First trace is from an x7.

JyeSmith avatar Jul 26 '21 20:07 JyeSmith

We do NOT believe the suggested scenario can happen, and even more, if the proposed issue was indeed hapening, it would not lead to the charts above as previous value would be used and it would not look like that. Nevertheless, we will be making a change to close the possible discussion

3djc avatar Jul 28 '21 07:07 3djc

We do NOT believe the suggested scenario can happen, and even more, if the proposed issue was indeed hapening, it would not lead to the charts above as previous value would be used and it would not look like that. Nevertheless, we will be making a change to close the possible discussion

Thx a lot for the analysis! I'm with you on that one. So the search continues....

raphaelcoeffic avatar Jul 28 '21 07:07 raphaelcoeffic

It would be interesting to learn if the ADC twitch issue is still there when running with FreeRTOS instead of CooCox CoOS: https://github.com/EdgeTX/edgetx/pull/499 ?

rotorman avatar Jul 28 '21 09:07 rotorman

Is there anything particular I would need to do to reproduce this? Or can I just fly / control a blackbox equipped FC with a D16 receiver and look for the twitches in the log?

pfeerick avatar Jul 28 '21 09:07 pfeerick

It would be interesting to learn if the ADC twitch issue is still there when running with FreeRTOS instead of CooCox CoOS: #499 ?

That is the case, @JyeSmith reported it while testing with FreeRTOS, then checked it was the same with the older code.

raphaelcoeffic avatar Jul 28 '21 11:07 raphaelcoeffic

Is there anything particular I would need to do to reproduce this? Or can I just fly / control a blackbox equipped FC with a D16 receiver and look for the twitches in the log?

@JyeSmith i believe this question is for you.

raphaelcoeffic avatar Jul 28 '21 11:07 raphaelcoeffic

Is there anything particular I would need to do to reproduce this? Or can I just fly / control a blackbox equipped FC with a D16 receiver and look for the twitches in the log?

I logged this on the bench by setting blackbox mode to always on.

JyeSmith avatar Jul 28 '21 11:07 JyeSmith

Is there anything particular I would need to do to reproduce this? Or can I just fly / control a blackbox equipped FC with a D16 receiver and look for the twitches in the log?

I logged this on the bench by setting blackbox mode to always on.

Do you think we could find a way to find/flag these errors automatically? That would help to systemise the testing.

raphaelcoeffic avatar Jul 28 '21 11:07 raphaelcoeffic

I can't write code except for simple python and makerbot type stuff I but I have tried about 10 different versions and the one here it's lock solid on 0 yeah lock solid on 0 unless I shake the radio and it might blip but the twitch is gone in this version hope this helps yeah hope this helps the T18 radio king

On Wed, Jul 28, 2021, 7:22 AM Raphael Coeffic @.***> wrote:

Is there anything particular I would need to do to reproduce this? Or can I just fly / control a blackbox equipped FC with a D16 receiver and look for the twitches in the log?

I logged this on the bench by setting blackbox mode to always on.

Do you think we could find a way to find/flag these errors automatically? That would help to systemise the testing.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/EdgeTX/edgetx/issues/503#issuecomment-888231154, or unsubscribe https://github.com/notifications/unsubscribe-auth/AASL2VBI757GJKCXAZDGI2LTZ7R5VANCNFSM5A77GQRQ .

TramALot avatar Jul 28 '21 14:07 TramALot

I remember another discussion about that and some detailed analysis by @rotorman . Does this still happen? I have no flight controllers, so for me checking would be more difficult

gagarinlg avatar Sep 17 '22 06:09 gagarinlg

I suspect not - since there really haven't been any follow reports to this. Also, shortly after this the ADC driver was cleaned up in #496, which should have removed what was thought of as being a possible culprit. I'll first see if I can reproduce the original issue on TX16S, since it appears to have shown up there, and then try with a later release.

pfeerick avatar Sep 22 '22 01:09 pfeerick

closed, because this has been solved a while ago

gagarinlg avatar Feb 25 '23 20:02 gagarinlg