openpilot icon indicating copy to clipboard operation
openpilot copied to clipboard

[$300 bounty] Ford Q3 Hybrids have occasional incorrect PCM message checksums

Open sshane opened this issue 2 years ago • 19 comments

This is causing controls mismatches for some users, including at least a Maverick and Escape. Haven't hit it on our Bronco yet however. I checked all three message checksums we check in panda (one from RCM, one from ABS, and one from PCM) and only the PCM message has these mismatches (it's behind a gateway, other two are on the same bus...).

Results: 2114
f8eaaccd2a90aef8|2023-05-05--12-27-53--18  FORD ESCAPE 4TH GEN    addr 0x202: 1 mismatches / 3000 msgs
f8eaaccd2a90aef8|2023-05-05--12-58-21--10  FORD ESCAPE 4TH GEN    addr 0x202: 1 mismatches / 2999 msgs
bd37e43731e5964b|2023-05-05--21-42-22--19  FORD MAVERICK 1ST GEN  addr 0x202: 3 mismatches / 3000 msgs
f8eaaccd2a90aef8|2023-05-05--12-27-53--22  FORD ESCAPE 4TH GEN    addr 0x202: 3 mismatches / 3000 msgs
933ad9d04ac82693|2023-02-08--09-19-55--22  FORD MAVERICK 1ST GEN  addr 0x202: 1 mismatches / 3000 msgs
f8eaaccd2a90aef8|2023-05-05--12-58-21--192 FORD ESCAPE 4TH GEN    addr 0x202: 2 mismatches / 2999 msgs
bd37e43731e5964b|2023-05-05--21-42-22--7   FORD MAVERICK 1ST GEN  addr 0x202: 1 mismatches / 3000 msgs
933ad9d04ac82693|2023-03-23--12-18-36--15  FORD MAVERICK 1ST GEN  addr 0x202: 1 mismatches / 2999 msgs
bd37e43731e5964b|2023-05-04--17-29-57--31  FORD MAVERICK 1ST GEN  addr 0x202: 1 mismatches / 3000 msgs
933ad9d04ac82693|2023-03-07--15-35-14--16  FORD MAVERICK 1ST GEN  addr 0x202: 1 mismatches / 3000 msgs
bd37e43731e5964b|2023-05-05--21-42-22--20  FORD MAVERICK 1ST GEN  addr 0x202: 1 mismatches / 3000 msgs
bd37e43731e5964b|2023-05-05--16-59-49--26  FORD MAVERICK 1ST GEN  addr 0x202: 1 mismatches / 3000 msgs
bd37e43731e5964b|2023-05-04--20-59-54--8   FORD MAVERICK 1ST GEN  addr 0x202: 1 mismatches / 3000 msgs
bd37e43731e5964b|2023-05-04--17-29-57--18  FORD MAVERICK 1ST GEN  addr 0x202: 2 mismatches / 3000 msgs
bd37e43731e5964b|2023-05-04--12-06-21--5   FORD MAVERICK 1ST GEN  addr 0x202: 2 mismatches / 3000 msgs
bd37e43731e5964b|2023-05-05--21-42-22--27  FORD MAVERICK 1ST GEN  addr 0x202: 1 mismatches / 2999 msgs

Matches by addr {514: 6199603, 1045: 6211839, 145: 12395638}
Mismatches by addr {514: 23793, 1045: 0, 145: 0}

Checked segments:
FORD BRONCO SPORT 1ST GEN 793
FORD FOCUS 4TH GEN 147
FORD ESCAPE 4TH GEN 243
FORD EXPLORER 6TH GEN 205
FORD MAVERICK 1ST GEN 726

Checked dongles:
FORD BRONCO SPORT 1ST GEN 4
FORD FOCUS 4TH GEN 1
FORD ESCAPE 4TH GEN 3
FORD EXPLORER 6TH GEN 4
FORD MAVERICK 1ST GEN 3

Checked platforms:
FORD BRONCO SPORT 1ST GEN {'54827bf84c38b14f': 717, '62241b0c7fea4589': 38, 'bbe75f19590890c4': 33, '5a08e8ef78cebd9f': 5}
FORD FOCUS 4TH GEN {'86d00e12925f4df7': 147}
FORD ESCAPE 4TH GEN {'f8eaaccd2a90aef8': 205, '26b2cace68e36212': 34, '62241b0c7fea4589': 4}
FORD EXPLORER 6TH GEN {'59a107f5793d9cc0': 22, '62241b0c7fea4589': 16, 'f0709d2bc6ca451f': 151, '8e1bab39e558e773': 16}
FORD MAVERICK 1ST GEN {'bd37e43731e5964b': 374, 'c845bd8c366e6f3d': 96, '933ad9d04ac82693': 256}

sshane avatar May 07 '23 05:05 sshane

Hybrids have been removed for now, since they're the only one affected.

adeebshihadeh avatar May 15 '23 20:05 adeebshihadeh

No reason to make a new duplicate issue for this

sshane avatar May 15 '23 20:05 sshane

Is this going to be in before 1.0? I can get whatever data is needed to help you solve it. -koopaloopz

ncoop23 avatar Jul 12 '23 01:07 ncoop23

Is there anything we can do about this?

seanaguinaga avatar Oct 17 '23 16:10 seanaguinaga

I'm happy to collect more can data if needed. Anything the community can work on? The acceptable solution space is unclear.

ncoop23 avatar Oct 26 '23 11:10 ncoop23

The bounty is for root causing the incorrect checksums and getting the Hybrids shipped.

adeebshihadeh avatar Dec 13 '23 07:12 adeebshihadeh

@adeebshihadeh do you have any suggestions for getting a repro, I have force-enabled my hybrid and have yet to see an error present in the UI. Would it be in the logs somewhere under some recognizable tag or message?

seanaguinaga avatar Dec 13 '23 15:12 seanaguinaga

Matt the gas for a few seconds is 100% reproduce or ACC acceleration from 60 to 80mph is like 75% reproduce. I can get you some more logs. If you look on the discord at #ford hybrid broke there is the story

ncoop23 avatar Dec 13 '23 15:12 ncoop23

I did a lot of work around this in the comma discord. Adeeb banned me, so forgive me if I remember incorrectly. It should be under Ford then a thread for "Ford hybrid broke".

I took the messages that Koopa got behind the gateway for the different BUSes and then ran through all the messages on Identifier 514 (hex 0x202) for checksum issues:

image

Anyways most if not all the data is in that thread.

sheaduncan avatar Dec 13 '23 17:12 sheaduncan

Matt the gas for a few seconds is 100% reproduce or ACC acceleration from 60 to 80mph is like 75% reproduce. I can get you some more logs. If you look on the discord at #ford hybrid broke there is the story

Mash the gas? Tap the gas? Sorry

seanaguinaga avatar Dec 15 '23 20:12 seanaguinaga

Matt the gas for a few seconds is 100% reproduce or ACC acceleration from 60 to 80mph is like 75% reproduce. I can get you some more logs. If you look on the discord at #ford hybrid broke there is the story

Mash the gas? Tap the gas? Sorry

100% full throttle for a few seconds will rack up enough mismatches to throw an error. Also, it is a safety concern that code is blocked while mismatches count up. You get a good half a second or so of comma holding course regardless of whatever is changing on the road.

ncoop23 avatar Dec 15 '23 22:12 ncoop23

@ncoop23 i full throttled for several seconds several times and never got any messages on the comma via UI

Does it only present in logs?

seanaguinaga avatar Dec 17 '23 04:12 seanaguinaga

@ncoop23 i full throttled for several seconds several times and never got any messages on the comma via UI

Does it only present in logs?

What is your vehicle? This only happens on Hybrids that have the mobile eye v3

ncoop23 avatar Dec 17 '23 12:12 ncoop23

on

2023 Ford Maverick Hybrid

seanaguinaga avatar Dec 17 '23 15:12 seanaguinaga

on

2023 Ford Maverick Hybrid

This is good news for you and could solve the issue for others. Can you get a drive to do public share of the route?

sheaduncan avatar Dec 17 '23 15:12 sheaduncan

on

2023 Ford Maverick Hybrid

Can you share a route of you going 100 throttle from 50 to 80 mph on the freeway?

ncoop23 avatar Jan 13 '24 19:01 ncoop23

Yes, let me get that tomorrow

seanaguinaga avatar Jan 14 '24 03:01 seanaguinaga

on

2023 Ford Maverick Hybrid

Can you share a route of you going 100 throttle from 50 to 80 mph on the freeway?

I can't use the gas pedal can I? I just thought this 😂

seanaguinaga avatar Jan 17 '24 16:01 seanaguinaga

on

2023 Ford Maverick Hybrid

Can you share a route of you going 100 throttle from 50 to 80 mph on the freeway?

I can't use the gas pedal can I? I just thought this 😂

If comma is engaged, you can still steer and use the gas pedal. If you hit the brake it will disable. Think passing a car, you can have comma engaged and turn signal and hit the gas to get around a guy.

What we need is you to record a log of going like 50 with comma engaged on the freeway and go 100% throttle till you are going as fast as you are able to go safely. Ideally 80 mph or more. If you could start lower than 50 mph, it would help.

I ran some when the freeway was empty early on a weekend morning by pulling off the road till there was a large opening, then getting aligned in a lane, setting comma and matting the gas till I was going as fast as was safe to trigger the error more than once. If it pops up with a controls mismatch error, press resume for your cruise control and keep accelerating if possible.

ncoop23 avatar Jan 17 '24 17:01 ncoop23

on

2023 Ford Maverick Hybrid

Can you share a route of you going 100 throttle from 50 to 80 mph on the freeway?

I can't use the gas pedal can I? I just thought this 😂

Any luck with the route?

ncoop23 avatar Jan 29 '24 22:01 ncoop23

I got a 2022 Ford Escape Plug in hybrid with the code change to get out of dashcam etc. Using Sunnypilot currently. I could try some time to do such a run on my commute route. Do car testers get any part of the bounty? Just curious. I work night shift so may not be until 11PM. I have a week or two old Comma 3X.

TSLARSX3 avatar Feb 15 '24 09:02 TSLARSX3

on

2023 Ford Maverick Hybrid

Can you share a route of you going 100 throttle from 50 to 80 mph on the freeway?

I can't use the gas pedal can I? I just thought this 😂

Are you still around to give this a try?

ncoop23 avatar Feb 15 '24 12:02 ncoop23

I got a 2022 Ford Escape Plug in hybrid with the code change to get out of dashcam etc. Using Sunnypilot currently. I could try some time to do such a run on my commute route. Do car testers get any part of the bounty? Just curious. I work night shift so may not be until 11PM. I have a week or two old Comma 3X.

When you floor it for a couple seconds with comma active, do you get a controls mismatch error?

ncoop23 avatar Feb 15 '24 12:02 ncoop23

I have an Escape SE (not titanium) 2021 without ACC. Only have Cruise Control and Line Assist. Can work in this car?

InfoIntegra avatar Feb 21 '24 20:02 InfoIntegra

I have an Escape SE (not titanium) 2021 without ACC. Only have Cruise Control and Line Assist. Can work in this car?

If you add acc. I'd look at the escape forum for that.

ncoop23 avatar Feb 23 '24 01:02 ncoop23

The on-ramp I take is a circle so I can turn cruise/pilot on but have to help it steer since it’s over steering limit.  Do you want me to do in all electric mode or both mode?  I assume both gas and electric combined.  I have some old logs of gassing it but I’ll plan on doing one in the morning and try to get that out to you.On Feb 21, 2024, at 3:33 PM, InfoIntegra @.***> wrote: I have an Escape SE (not titanium) 2021 without ACC. Only have Cruise Control and Line Assist. Can work in this car?

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: @.***>

TSLARSX3 avatar Feb 23 '24 09:02 TSLARSX3

The on-ramp I take is a circle so I can turn cruise/pilot on but have to help it steer since it’s over steering limit.  Do you want me to do in all electric mode or both mode?  I assume both gas and electric combined.  I have some old logs of gassing it but I’ll plan on doing one in the morning and try to get that out to you.On Feb 21, 2024, at 3:33 PM, InfoIntegra @.> wrote: I have an Escape SE (not titanium) 2021 without ACC. Only have Cruise Control and Line Assist. Can work in this car? —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: @.>

The plug in hybrid is a bit different. This error is for normal hybrids but your log would be helpful. Ideally, the best log would be to:

find a location on the side of an empty freeway early on a weekend morning, pull off the side of the road and stop. Put on hazard's. Wait for cars to clear, pull out into a lane, accelerate to 20 mph, turn on cruise/comma, floor the gas 100% until about 70-80 depending on speed limits in your area.

ncoop23 avatar Feb 23 '24 15:02 ncoop23

You mean Only have Cruise Control and Line Assist Can work in this car? Please tell me how! i tried everything!

This is the error:

Uploading Imagen de WhatsApp 2024-02-23 a las 18.14.37_20746c4c.jpg…

InfoIntegra avatar Feb 23 '24 21:02 InfoIntegra

Please get on discord app and go to the Ford channel and someone can point you in the right direction. You may need to add a module to your car. This is unrelated to the topic of this issue though.

ncoop23 avatar Feb 23 '24 21:02 ncoop23

Yes, i ask in Ford Channel but nobody knows why do not work.

InfoIntegra avatar Feb 23 '24 22:02 InfoIntegra