Not working with Bluetooth Proxies (ESP32 based)

[bcutter]

While I was lucky to initialy set up this integration without any issues (https://github.com/GrumpyMeow/sonicare-ble-hacs/issues/2#issuecomment-1715326758), I discovered by manually reloading the integration that:

• it will find the toothbrush just fine if it is pretty close to the Home Assistant server - a Raspberry Pi 4 which uses native Bluetooth
• it will NEVER EVER find the toothbrush if next to a Bluetooth Proxy (ESP powered, https://esphome.github.io/bluetooth-proxies/) - at the same time it can be discovered using the "LightBlue" app on an iPhone also next to the BT Proxy ESP

So where's the issue? Is it the BT proxy or this Sonicare BLE integration?

I bet this is one of the reasons for the most open "Can't connect to XYZ" issues in this integration, so really worth looking into this @GrumpyMeow! We should probably start asking for the specific bluetooth environment people use. I already found few users stating they are also using Bluetooth Proxy (e. g. here: https://github.com/GrumpyMeow/sonicare-ble-hacs/issues/2#issuecomment-1556674265).

[badewanne1234]

Thanks for sharing this. As stated, I can see the advertising messages of the Toothbrush in the BT Proxy Log, so I think it might be an issue of how the Sonicare integration reads/interprets these advertisement messages OR is the raspi BT passing BT messages differently than the ESP32 based BT proxies?

[bcutter]

it might be an issue of how the Sonicare integration reads/interprets these advertisement messages OR is the raspi BT passing BT messages differently than the ESP32 based BT proxies?

That's exactly the question I hope bluetooth experts like @GrumpyMeow can verify.

By the way: by only activating my toothbrush next to the BT proxy I could not find any activity in the logs (ESPHome). If i trigger other bluetooth devices (like SwitchBot buttons) next to the BT proxy, I see quite some activity.

[badewanne1234]

I think I played around with the advertising window / intervalls and could see the activity immediately when lifting the TB out of the glass...

[edwardtfn]

So, I've teste getting quite close to the server (like 2m) and the toothbrush was finally discovered. 😃

Now I will vote to add support for Bluetooth proxy. 😉

[bcutter]

@GrumpyMeow

[PV-Web]

Same problem. Bluetooth proxy via the M5 atom lite via ESP home. @GrumpyMeow Hopefully you got time to check were this could go wrong.

[helmerzNL]

My toothbrush does connect if i'm close to my HA instance, but not from a Bluetooth Proxy. So, it would be great if support for Bluetooth proxies can be added.

[bcutter]

@GrumpyMeow are you reading/still maintaining this project?

[rigwig]

Same issue here, am able to connect via HA intgrated bt, but no luck via an esp32 based proxy.

[bcutter]

@GrumpyMeow are you reading/still maintaining this project?

@GrumpyMeow you seem to be active, unfortunately not in this repo. Can you please give us an update on this? Fixing this "does not work with bluetooth proxies" issue is a mandatory killer feature immediately resolving a ton of open issues.

[urosch]

This is the log output from home assistant after trying to connect my sonicare to ha via esphome: 2024-01-07 12:00:28.419 INFO (SyncWorker_7) [homeassistant.util.package] Attempting install of sonicare-bletb==0.0.8 2024-01-07 12:01:28.647 WARNING (MainThread) [sonicare_bletb.sonicare_bletb] Philips Sonicare: Init 2024-01-07 12:01:28.647 WARNING (MainThread) [sonicare_bletb.sonicare_bletb] Philips Sonicare: initialise 2024-01-07 12:01:28.647 WARNING (MainThread) [sonicare_bletb.sonicare_bletb] Philips Sonicare: Connecting; RSSI: None 2024-01-07 12:02:11.702 WARNING (MainThread) [sonicare_bletb.sonicare_bletb] Philips Sonicare: Init 2024-01-07 12:02:11.702 WARNING (MainThread) [sonicare_bletb.sonicare_bletb] Philips Sonicare: initialise 2024-01-07 12:02:11.702 WARNING (MainThread) [sonicare_bletb.sonicare_bletb] Philips Sonicare: Connecting; RSSI: None 2024-01-07 12:03:29.214 WARNING (MainThread) [sonicare_bletb.sonicare_bletb] Philips Sonicare: Init 2024-01-07 12:03:29.214 WARNING (MainThread) [sonicare_bletb.sonicare_bletb] Philips Sonicare: initialise 2024-01-07 12:03:29.214 WARNING (MainThread) [sonicare_bletb.sonicare_bletb] Philips Sonicare: Connecting; RSSI: None 2024-01-07 12:03:49.238 WARNING (MainThread) [aioesphomeapi.connection] esphome-web- @ 100.0.0.1: Connection error occurred: [Errno 104] Connection reset by peer 2024-01-07 12:03:49.238 INFO (MainThread) [aioesphomeapi.reconnect_logic] Processing unexpected disconnect from ESPHome API for esphome-web- @ 100.0.0.1 2024-01-07 12:03:49.251 INFO (MainThread) [aioesphomeapi.reconnect_logic] Successfully connected to esphome-web- @ 100.0.0.1 in 0.005s 2024-01-07 12:03:49.329 INFO (MainThread) [aioesphomeapi.reconnect_logic] Successful handshake with esphome-web- @ 100.0.0.1 in 0.078s

[adheaps]

Seems @GrumpyMeow has abandoned this project. Would love to see some progress on this. Is this something that the Home Assistant dev team would pick up? It appears to have some good interest from the community.

[bcutter]

I don't think that is gonna happen. Usually a custom integration needs to prove it is "good enough" (like function in general, reliable, flexible in terms of reaction time for changes in core etc.) to become a core/default integration.

Very sad here's no progress, which makes this basically really great integration basically completely useless for most users. Sell Philips, get a Oral-B? Just kidding.

[jlipsit]

Summary of Errors and Troubleshooting Steps for ESP32 Bluetooth Proxy with Sonicare BLE Toothbrush

Updated: I found that although it will connect to a BT Dongle directly connected to the controller, it does not reconnect reliably after the toothbrush goes to sleep. I have to RELOAD the integration when the brush is awake to get it to reconnect. It only stays connected to that "session" and then will not reconnect. Therefore I suspect the integration has some issues that are just more severe when used with a proxy. IS ANYONE USING THIS RELIABLY WITH A DIRECT CONNECTION TO A BT DONGLE?

I picked up a new Philips Sonicare and verified that it will not provision and connect when using a BT proxy, It will connect and operate properly when using a BT dongle connect to the Home Assistant Green controller.

Although the toothbrush is discovered successfully, when attempting to add it to HA, when using an ESP32 Bluetooth proxy the Sonicare BLE toothbrush disconnects with error messages indicating "Connection reset by peer" (Errno 104). This issue occurs consistently, preventing the ability to add it to HA.

I did a lot of troubleshooting............... Here are some of my logs and troubleshooting

ESP32 Device Log Extract:

BLE Connection Attempts and Errors:

First Connection Attempt: Timestamp: [08:05:24] Device MAC: D3:35:34:35:53:8C Event: [08:05:24] Attempting BLE connection [08:05:27] ESP_GATTC_CONNECT_EVT [08:05:27] ESP_GATTC_OPEN_EVT [08:05:27] Connected [08:05:36] Disconnecting [08:05:36] ESP_GATTC_CLOSE_EVT

Second Connection Attempt: Timestamp: [08:06:42] Device MAC: 24:E5:AA:75:27 Event: [08:06:42] Attempting BLE connection [08:06:42] ESP_GATTC_CONNECT_EVT [08:06:42] ESP_GATTC_OPEN_EVT [08:06:56] ESP_GATTC_DISCONNECT_EVT, reason 8 [08:06:56] ESP_GATTC_CLOSE_EVT [08:06:57] Attempting BLE connection [08:06:58] ESP_GATTC_CONNECT_EVT [08:06:58] ESP_GATTC_OPEN_EVT

Third Connection Attempt: Timestamp: [08:07:26] Device MAC: 24:E5:AA:75:27 Event: [08:07:26] Attempting BLE connection [08:07:33] ESP_GATTC_CONNECT_EVT [08:07:33] ESP_GATTC_OPEN_EVT [08:07:46] Disconnecting [08:07:47] ESP_GATTC_CLOSE_EVT [08:07:47] Attempting BLE connection [08:07:48] ESP_GATTC_CONNECT_EVT [08:07:48] ESP_GATTC_OPEN_EVT

Errors: Timestamp: [08:06:42] Error: hcif disc complete: hdl 0x0, rsn 0x3e Timestamp: [08:06:56] Error: gattc_conn_cb: if=3 st=0 id=3 rsn=0x8 Timestamp: [08:06:57] Error: BTM_BleScan scan not active Error: bta_dm_ble_scan stop scan failed, status=0x6 Timestamp: [08:07:26] Error: hcif disc complete: hdl 0x0, rsn 0x3e Timestamp: [08:07:46] Error: GATTC_ConfigureMTU GATT_BUSY conn_id = 3 Error: hcif disc complete: hdl 0x0, rsn:0x13 Error: No pending command

Summary: The ESP32 attempts to connect to the BLE device multiple times, experiencing intermittent success and errors such as disconnection events and scan failures. Each connection attempt and its outcome are logged with timestamps for detailed troubleshooting.

Successful Initial Connections:

• The log shows successful connection attempts to the toothbrush's BLE device, identified by its MAC address (e.g., [24:E5:AA:75:27:D9]).
• BLE events such as ESP_GATTC_CONNECT_EVT and ESP_GATTC_OPEN_EVT indicate that the ESP32 Bluetooth proxy is able to establish initial connections.
• the signal strength appears adequate (Signal strength: -58 dB)

Frequent Disconnections:

• The log contains several instances of disconnection events (ESP_GATTC_DISCONNECT_EVT), often followed by immediate reconnection attempts.
• Disconnection reasons (rsn) vary but commonly include 0x16 and 0x3e.

Warnings and Errors:

• Warnings such as Connection error occurred: [Errno 104] Connection reset by peer suggest that the connection is being unexpectedly terminated.
• There are several instances of hcif disc complete: hdl 0x0, rsn 0x16 indicating that the disconnection is being completed by the Bluetooth host.

Troubleshooting Steps Taken:

Configuration Adjustments:

Adjusted esp32_ble_tracker scan parameters to optimize BLE scanning:

esp32_ble_tracker:
  scan_parameters:
    interval: 320ms
    window: 160ms

esp32_ble_tracker:
  scan_parameters:
    interval: 160ms
    window: 80ms

Power Supply Verification:

Ensured that the ESP32 is connected to a stable power source to avoid power-related disconnections. Interference Minimization:

Reduced potential Bluetooth and Wi-Fi interference by positioning the ESP32 and Sonicare toothbrush closer together and away from other electronic devices.

Firmware and Software Updates:

Confirmed that all firmware and software (ESPHome, Home Assistant, and ESP32 firmware) are updated to the latest versions available at the time of testing.

Core 2024.7.1 Supervisor 2024.06.2 Operating System 12.4 Frontend 20240705.0

Logger Configuration:

Enabled detailed logging to capture more granular details for debugging:

logger:
  level: debug
  logs:
    esp32_ble_client: debug

Alternative Connectivity Test:

Verified that the Sonicare toothbrush connects successfully using a Bluetooth dongle plugged directly into the Home Assistant Green controller, indicating the issue is specific to the ESP32 Bluetooth proxy.

[bcutter]

Great work! For sure took a loooooot of time. How should we use those perfect information (you mentioned "the ESP home guys need to look at this issue")?

Maybe someone can ping them?

Another note: I also use Shelly's as bluetooth proxies, with the same (not working) results as with a ESP home powered bluetooth proxy. Shelly are also based on ESP, but not ESP Home - just as a small note.

[jlipsit]

Unfortunately, I don't think they will look at it until someone takes over this integration and reviews the code for possible errors. It's sad because the Oral B is very expensive and the Sonicare has excellent hardware and is very affordable.

[jlipsit]

I found that although it will connect to a BT Dongle directly connected to the controller, it does not reconnect reliably after the toothbrush goes to sleep. I have to RELOAD the integration when the brush is awake to get it to reconnect. It only stays connected to that "session" and then will not reconnect. Therefore I suspect the integration has some issues that are just more severe when used with a proxy. IS ANYONE USING THIS RELIABLY WITH A DIRECT CONNECTION TO A BT DONGLE?_

[edwardtfn]

IS ANYONE USING THIS RELIABLY WITH A DIRECT CONNECTION TO A BT DONGLE?

Unfortunately, no. 😩

[jlipsit]

IS ANYONE USING THIS RELIABLY WITH A DIRECT CONNECTION TO A BT DONGLE?

Unfortunately, no. 😩

So just to clarify, you have it connected to a BT dongle directly, but it is not reliable?

[edwardtfn]

Exactly. In the end, it's not useful.

[Dodoooh]

is there any good news to get this up and running?

[bcutter]

No. The author of this integration doesn't seem to be active in any way. This makes us (potential) users grumpy and leaves us with a non-working integration. Completely useless.

[v6ak]

I remember having similar issues when implementing QuietDrift for SwitchBot Curtain 3. Originally, it was a separate service called via HTTP from HA. However, I got into connection issues similar to those. I implemented various reconnecting mechanisms, but it was hard to make it working in long term. Then, I rewrote it as a custom component and let HA to manage the BT connection, which has fixed the connection issues.

My original code was obtaining the device object from bleak_retry_connector.get_device. And this this custom component does something similar:

https://github.com/GrumpyMeow/sonicare-ble-hacs/blob/5ab0c8fa3c0a44015a8b75fa7196a89ada48b95e/custom_components/sonicare_bletb/init.py#L27-L29

Well… It isn't exactly the same. It tries to use bluetooth.async_ble_device_from_address first. However, when the device isn't reachable, it returns None, which forces a fallback to bleak_retry_connector.get_device. And it is quite common that the device is unreachable. In this case, bleak_retry_connector.get_device is likely to also fail, but it doesn't have to, as it is called somewhat later, and apparently can wait a bit. In my logs, I've seen an error from bleak_retry_connector, which suggests that this sometimes happens.

The code is in async_setup_entry, and it is the only place where async_ble_device_from_address/get_device is called, which has few consequences:

• It usually fails on startup, unless the toothbrush is reachable. (It is rarely the case.)
• It seems HA retries later when it discovers such device.
• Once the connection is lost, there is probably no mechanism to recover from that. HA will probably still consider the setup entry as valid, but the component doesn't seem to have a mechanism for reconnecting.

Not sure about async_ble_device_from_address, but I am pretty sure that bleak_retry_connector.get_device retries just in the first connection. Once the connection is lost, the returned object doesn't retry.

Initially, I am removing the get_device fallback and will see how it affects the behavior. I am not sure why it was added, as 34508e38cf81532f29b787d390f814259c9397f5 doesn't explain much.

[v6ak]

Few more findings:

First, it has connected to the toothbrush even with all BT adapters disabled, leaving only BT proxies.

Second, it has still issues with disconnecting. Looking more about the code of this repo, the library and the documentation, it seems that the component (and/or its libraries) is responsible for reconnections. There is however some mechanism, but seemingly incomplete:

1. The library has _reconnect method, which is called from _disconnected and from itself. However, calling from itself happens only in case of except BleakNotFoundError:, not in case of other errors. But the errors are more diverse.
2. The component listens for new device with the toothbrush's address and calls set_ble_device_and_advertisement_data, which, aside of logging, just sets properties, but doesn't call _reconnect. Which means it cannot save us when the reconnect loop is broken.

Looking from 10 feets:

1. Not sure if bleak_retry_connector.get_device is to blame. It shouldn't be needed, maybe it was added as an attempt to fix something (the megacommit 34508e38cf81532f29b787d390f814259c9397f5 doesn't tell us much). Now, it seems to just add complexity and I'll probably keep it removed.
2. The reconnects depend on polling (quite frequent, 0.25s), but it's a bit weird to poll for a toothbrush, which is offline most of the time.
3. The polling frequency will affect what kind of issues we face. Frequent polling increases likelihood of errors other than BleakNotFoundError (that stop the polling). Less frequent polling increases likelihood that HA will not connect within the short window when toothbrush is online.

What to do:

1. First and foremost, the _async_update_ble should trigger reconnect ASAP. (Not sure what to do when it is still connected.) So we woulnd't rely on polling.
2. We should also register a disconnect handler which stops the polling when toothbrush is disconnected. This would make the polling effectively disabled (without actually removing the polling code from the library).
3. Bonus: don't rely on toothbrush being available on startup (in async_setup_entry).

I'll probably do it later, it doesn't look that hard.

EDIT: I got a bit struggles with the callbacks, HA's documentation isn't perfect. But maybe I was reinventing the wheel – it looks like ActiveBluetoothDataUpdateCoordinator has already implemented what I need.

[bcutter]

@v6ak Wow, impressive (as always when users see devs talking and doing their magic 😀).

Sounds like maybe Christmas takes place a bit earlier this year: toothbrush working with BT proxies sounds so 👍

[simbalasim]

@v6ak Wow, that sounds really promising. I'm not a developer either, but that sounds like you have found at least a direction how to progress further. Much appreciated Really great that you're taking this on and investing your time. Many thanks for that 😃,

I'm currently working with very unreliable workournds,. when the toothbrush connects to a tablet, the Soniccare app simply opens. Not particularly fancy ^^

Unfortunately I don't know coding, but if I can help e.g. with testing, please let me know.

[v6ak]

Well, not sure how to proceed. I've tried multiple ways to discover the device:

• bluetooth.async_register_callback
• ActiveBluetoothDataUpdateCoordinator
• bluetooth.async_get_scanner

Expected results: the device gets discovered when I make the toothbrush active (move, brushing, briefly after brushing, briefly after charging), so I can connect it.

Actual results: the device gets discovered immediately and seems to never disappear. However, I don't get a callback when the device is actually activated.

I feel like I am missing something trivial, but not sure what…

[jlipsit]

With all my testing, I got the impression that there was some bug in the brush itself. I just threw mine in the trash this week and purchased 4 Oral B Series 4 toothbrushes off eBay. Great brush and about $50 bucks new on eBay if you are patient. The are rock solid in Home Assistant.

Not Running View

Running View

[u20p17]

@v6ak : thank you very much for trying to solve the issue! Looking forward to test your fixes in the future with my philips toothbrush 👌

Regarding you missing callback: maybe you could as in the discord channel of HomeAssistant in the dev area. Maybe you get some hints 👍

[adheaps]

@v6ak - I'd like to thank you as well. So happy to see someone trying to solve this issue. Truly appreciate your efforts on this.