The future of BLE, Bluetooth, and USB

[laurensvalk]

Introduction

This issue gathers some ideas for Bluetooth and USB to help us keep track of the big picture while we work on low-level details.

Note that most of this won't be done any time soon (if ever). This is mainly intended to avoid implementation choices now which might come back to hurt us later. If you want us to prioritize this, please consider becoming a sponsor :rocket: .

Nomenclature

• The code in boxes is pseudocode for commands in user scripts.
• Arrows: wireless or wired communication.
• Each color is a different service/characteristic which may or may not share a common connection.

1. Single-hub coding scenario

This is what we are working on today. We are working on a Pybricks BLE service to handle things like starting and stopping programs in a clean way. Standard I/O (print/input) may be a separate service or characteristic on the same connection.

Until now, both were handled on one and the same characteristic, which made it easy to mess up the connection by sending the wrong characters.

Cleaning this up and documenting it also paves the way for other editors or extensions to support Pybricks.

This is the only use case we will target for the upcoming v3.0 release. That is likely also the last release for Move Hub; it won't have enough space for the other features listed below.

2. Multi-hub coding scenario

This might work as above, but there are many elements still missing such as:

• Distinguishing hubs (e.g. by name or color)
• Multiple Pybricks Code instances (e.g. tabs or windows)

3a. Multi-hub communication scenario

This is still to be explored. At the basic level, there might be serial i/o streams between hubs that could be used with standard MicroPython tools. Higher level abstractions such as mailboxes could be added on top of these later.

3b. Other BLE UART devices

This is really the same as above, but worth showing in a separate diagram because this really opens up interesting possibilities.

4. BLE HID Peripherals

We may support connecting to generic HID devices over BLE (and possibly classic). Users could build on this to add specific mappings for certain devices like popular gaming consoles.

5. Bluetooth classic scenario

This is only available on Prime Hub and Inventor Hub. Therefore, we will not use Bluetooth classic for any system purposes. So, you won't be able to download and run programs with it.

Rather, it might be used within end-user scripts. For example, you might set up a generic RFCOMM connection with a server or client. This could be another Prime Hub, an EV3 Brick, a laptop, and so on.

6. LEGO Powered Up Remote

See #186.

7. USB scenario

This is only available on Prime Hub and Inventor Hub. Therefore, we will probably not use USB for system purposes. So, you probably won't be able to download and run programs with it. USB will be used primarily for charging the hub. We may re-enable REPL over USB if users want this. Since REPL is treated as a regular program, we'd have to define a way to start and stop it. Perhaps USB can also be used for fast(er) file transfer.

[TechnicBRICKs]

This would be really cool!! My priorities would be, first scenario # 3 (although this also implies scenario # 2) and then scenario # 4.

My hope is that basic functionality may be feasible with ubluetooth module (# 3), even before you are able to prioritize it, as we start to see examples making use of it. It would be a matter of finding someone willing to share his work and code examples, or going the hard way and figure it out yourself.

[laurensvalk]

Partly inspired by the existing messaging module, here is one idea for what communication could look like.

Peripherals:

from pybricks.messaging import PeripheralHub

# Can make only one of these
peripheral = PeripheralHub()

# Starts advertising. 
peripheral.advertise(True)

# Waits for incoming connection. On success, stops advertising and returns a stream object, else raises TimeOutException.
hub_connection = peripheral.listen(timeout=1000)

# Now you can send and receive data using hub_connection.read and hub_connection.write

The central:

from pybricks.messaging import CentralHub

# Can make only one of these
central = CentralHub()

# Starts scanning for hubs that are advertising.
central.scan(True)

# Results will be stored in central.scan_results, a dict of  name:address pairs
# So you could wait until some devices are found, or look for a specific one:
while 'my_other_hub' not in central.scan_results:
     pass

# Found it!
address = central.scan_results['my_other_hub']

# Stop scanning
central.scan(False)

# Connect. Returns a stream object on success, else raises TimeOutException.
hub_connection = central.connect(address, timeout=1000)

# Now you can send and receive data using hub_connection.read and hub_connection.write

Peripherals (simplified API idea)

from pybricks.messaging import ble_wait_for_connection

# Waits for incoming connection. On success, stops advertising and returns a stream object, else raises TimeOutException.
stream = ble_wait_for_connection(timeout=1000)

# Now you can send and receive data using stream.read and stream.write

The central (simplified API idea)

from pybricks.messaging import ble_connect

# Connect. Returns a stream object on success, else raises TimeOutException.
stream = ble_connect(address_or_name, timeout=1000)

# Now you can send and receive data using stream.read and stream.write

[mwinkler]

Really like to see this roadmap! As TechnicBRICKs already mention, I hope that ubluetooth where available, so that we can experiment and build exciting use cases.

[TechnicBRICKs]

Partly inspired by the existing messaging module, here is one idea for what communication could look like.

Peripherals:

from pybricks.messaging import PeripheralHub

# Can make only one of these
peripheral = PeripheralHub()

# Starts advertising. 
peripheral.advertise(True)

# Waits for incoming connection. On success, stops advertising and returns a stream object, else raises TimeOutException.
hub_connection = peripheral.listen(timeout=1000)

# Now you can send and receive data using hub_connection.read and hub_connection.write

So it looks halfway done already! :D

Meanwhile maybe playing around these examples could lead us somewhere. https://github.com/micropython/micropython/tree/master/examples/bluetooth

[laurensvalk]

The examples for ubluetooth already work just fine on the official firmware.

Our objectives are just a little different. Compare those examples with the suggested code snippets above. We're aiming to let you do it with two lines of code and about 10x fewer system resources used. ubluetooth is great for communicating with everything including the kitchen sink, but we don't need all of that just to communicate between hubs.

[TechnicBRICKs]

Fully understood your approach! Just thought that for the time being, this could lead to an intermediate solution.

To get it working guess one needs to add the ubluetooth library into your firmware zipfile, correct!?

[laurensvalk]

No, it doesn't work that way. ubluetooth is written in C, not Python.

[mwinkler]

The examples for ubluetooth already work just fine on the official firmware.

Your referring to the official spike prime firmware, correct? Because there is some stuff missing (pair function for example).

[laurensvalk]

Ah, that's interesting. I haven't tried all of the examples. The ubluetooth module seems to be actively developed still, so perhaps newer spike/inventor firmwares will get the latest versions in future updates too.

@mwinkler, as you've expressed interest in having ubluetooth in the Pybricks firmware too, could you share a few use cases that really need it, which aren't already covered above?

[mwinkler]

With the described use cases, is all covered what I have currently in mind.

My idea is a track based racing game, where every player controls his own vehicle via game pad (or pu remote). The cars are also connected to a central unit/hub. The central unit then can send commands and receive events from the cars, like increase speed (give a boost), count a marker/checkpoint (to track lap count/time), stop all cars, and so on...

So in the end, if all is implemented in pybricks, I don't need the ubluetooth module. The only reason I'am asking for the module is, because I have started experimenting with it (HID over GATT with a BLE pad) on spike prime and stuck now because of the missing functions in the lego firmware. But I see, that providing these modules have impact on memory usage and needs to be carefully selected.

Btw, thank you for the great work you do (you both) with this project.

[NStrijbosch]

Interesting dicussion!

I have been playing around with the ubluetooth from the official SPIKE Prime firmware to get Scenario 3.A/B working.

I second @laurensvalk statement on the complexity of the code necessary to get everything setup and the resources the ubluetooth module requires. So, once a more resource efficient implementation is available in Pybricks that would be my goto firmware for multi hub projects.

Just a few questions that came to my mind on the implementation of Scenario 3A:

• what would be the maximum number of pheripherals that can connect to a single central?
• can a hub be both a central and peripheral at the same time? (e.g., to get a network of three hubs that each have a direct connection to the other two)
• what happens when a connection is lost? Would that require a complete restart of the programs to get the connection back? or does it trigger a peripheral to start advertising and the central to start scanning automatically?

[laurensvalk]

Thanks @NStrijbosch -- some good points there. We're not sure yet about the connection limits.

I think we could enable the user to monitor the connection status so they can reconnect if they want to. But hopefully it should be stable enough within reasonable distances to the point where most users might not need to do this.

How is your experience with connectivity and stability, since you've worked with 9 hubs at once?

[NStrijbosch]

From my experience the connection limit of the Robot Inventor/Spike Prime hub:

• one hub is able to connect to 4 other hubs (3 peripherals and 1 central in my case) But I am not knowledgeable enough to know if this is: a hardware limitation; a software limitation in the firmware; or my own implementation. To get 9 hubs to communicate I implemented a BLE Tree network structure, with a few hubs acting as both central and peripheral.

The stability has some issues from time to time, once again not sure if this is hardware/firmware/my software. But some of my observations:

• For small scale projects (<4 hubs) connection is fine. Although it seems that it it is important to not flood the network at the central side with messages: this leads to an out of memory error. I have not managed to prevent this when implementing the Nordic UART protocol, so I opted for a messaging protocol that prevents this flooding.
• for bigger projects establishing a connection fails once in a while (at the hub that acts as both a central and peripheral). If this happens an already established connection fails at the moment of connecting a new hub to the same central. Not a big issue if it would have been possible to reestablish the lost connection. But for now the only option I have is restarting all hubs... (which can be a pain with 9 hubs). When everything is connected and the network is not flooded with messages it works fine.

[laurensvalk]

Thanks a lot for sharing this. This will set the minimal benchmark for us :wink:

We would certainly be interested in your findings once you test our stuff. We welcome contributions to the code as well.

And good to know that a hub can be both a central and a peripheral, too. We need that, since we use BLE for programming as well, where it is the peripheral to the PC.

[giadefa]

Going back to a previous issue which is now referring to this issue for the solution: https://github.com/pybricks/support/issues/164

How do I communicate few floats back and forward between a normal python program running in my PC and the hub micropython? I don't quite see this simple scheme in the description above.

Thanks

[dlech]

For EV3, we have https://pybricks.github.io/ev3-micropython/messaging.html

For Powered Up hubs, this is not implemented yet.

[laurensvalk]

How do I communicate few floats back and forward between a normal python program running in my PC and the hub micropython? I don't quite see this simple scheme in the description above.

That would be use case 3b. This pictures a phone as an example, but it could be a PC as well. The PC could use any language with BLE support, such as Python with the bleak library, or JS with WebBLE, etc.

And indeed, this is all still in the concept stage.

[dlech]

One thing I have been thinking about that is specific to BLE UART is that we have write with response or write without response in one direction and indicate (with response) or notify (without response) in the other direction. In the with response case, there should be no packets dropped, but throughput will be lower since there is the extra overhead of a response sent for each packet. In the without response case, the transmitter has no way of knowing if the receiver actually received the data and the receiver has no way of knowing if they missed something (unless it is handled at the protocol level).

So I'm thinking we probably want to default to the slower, safer option. But we might want to add an option for the faster lossy option if there are use cases that require high bandwitdh.

[laurensvalk]

At the moment, we have pybricks.bluetooth for lower-level bluetooth classic and pybricks.messaging with higher level messaging tools. Along these lines, perhaps we can have pybricks.ble for lower level BLE stuff.

I've been brainstorming what that absolute minimum might be, which will still support all of the communication use cases above. It might all boil down to just this:

# SPDX-License-Identifier: MIT
# Copyright (c) 2018-2020 The Pybricks Authors

"""
pybricks.ble: Bluetooth Low Energy communication tools.
"""

class BLEDevice:

    address = "AA:BB"
    """Address of the connected device"""

    name = "my_city_hub"
    """Name of the connected device"""

    mtu = 23
    """Maximum transmission unit."""

    # Add methods and/or attributes such as to configure write with/without response, etc

class NUSDevice(BLEDevice, PybricksIOBuffer):
    pass

def nus_connect(address, timeout=3000) -> NUSDevice:
    """Connects to a peripheral that is currently accepting a connection.

    This hub will act as the central. It can connect to any other peripheral
    that advertises the Nordic UART Service (NUS).

    Arguments:
        address: Bluetooth address or name of the device you wish to connect to.
        timeout: How long to scan for the device before giving up, or
            ``None`` to wait indefinitely.

    Raises:
        TimeoutException: If the connection could not be made within the
            given timeout.

    Returns:
        Stream object representing the connection to the peripheral.
    """
    pass


def nus_wait_for_connection(timeout=None) -> NUSDevice:
    """Accepts an incoming connection from a central.

    This hub will act as a peripheral and advertise the Nordic UART Service
    until a connection is made or the timeout is reached.

    Arguments:
        timeout: How long to wait for a connection before giving up, or
            ``None`` to wait indefinitely.

    Raises:
        TimeoutException: If the connection could not be made within the
            given timeout.

    Returns:
        stream object representing the connection to the central.
    """
    pass

# Peripheral 

from pybricks.ble import nus_wait_for_connection

prime_hub_stream = nus_wait_for_connection()

prime_hub_stream.write("Hello!")

# Central

from pybricks.ble import nus_connect

city_stream = nus_connect("my_city_hub")
technic_stream = nus_connect("AA:BB:CC")

class LWP3Device(BLEDevice):
    pass

def lwp_connect("....") -> LWP3Device
    pass

class pybricks.pupdevices.RemoteControl(LWP3Device):
    pass

my_duplo_hub = lwp_connect("Duplo Hub")

So I'm thinking we probably want to default to the slower, safer option. But we might want to add an option for the faster lossy option if there are use cases that require high bandwitdh.

Could we add this as a parameter above? Or is this a hub-wide setting that will affect all subsequently made connections?

[NStrijbosch]

Is this the lowest level possible? I am thinking beyond Nordic UART, e.g., LWP ;)

My main question: what are your thoughts on support for LWP, i.e., either by allowing specific characteristics in this low-level BLE to communicate with LWP devices, or by extending scenario 6 beyond the PoweredUp Remote?

I have been diving into the LWP docs lately (that would be extending scenario 6). For myself I see a few advantages (and of course also disadvantages) in using this protocol to communicate between a SPIKE Prime/Robot Inventor hub to any other PU hub supporting LWP. The advantages I enjoy at the moment:

• the program runs from a single hub, this makes coding/debuging way more efficient compared to updating the program on each of the hubs.
• No third party firmware necessary on the peripheral hubs.

Of course the disadvantages are numerous:

• You rely on the computation power of a single hub
• The update frequency of everything on the peripheral hubs is limited by BLE
• ...

[laurensvalk]

Thanks for your quick response.

Is this the lowest level possible?

No, this is mainly the low level for the NUS communication stuff, hence the nus_ prefixes. And despite being "low-level", this is really quite usesable in end-user scripts already, since you get a stream object that is fairly easy to use. Higher level pybricks.messaging is optional and maybe we don't even need it.

My main question: what are your thoughts on support for LWP, i.e., either by allowing specific characteristics in this low-level BLE to communicate with LWP devices, or by extending scenario 6 beyond the PoweredUp Remote?

In other words, this leaves room for other functions in pybricks.ble that start with lwp_ :wink:

[michieldetailleur]

I hope this is the best issue to post this under...

I applaud the good progress on making the official Lego controller work with PyBricks #186

Allthough one thing the Lego controller seems to lack is analog input. I love to control my Powered Up sets with my PS4 controller via BrickController2. And I would love it even more if I could connect the PS4 controller directly to the Powered Up hub (https://github.com/pybricks/support/discussions/156#discussioncomment-802999). Unfortunately that controller is not BLE capable, neither is the PS5 controller.

However: the latest Xbox controller seems to support BLE! https://en.wikipedia.org/wiki/Xbox_Wireless_Controller#Xbox_Series_X/S_launch_model

Seeing that:

• probably many people might already or will in the future own this controller
• this controller will be available for sale for probably a long time
• it's a high quality controller for an affordable price (around €70)
• you can also use this controller on your PC (so it's multipurpose, even if you don't have an Xbox)

Would anyone else be interested in PyBricks support for this controller? It seems to be technically possible at least.

I would be very willing to donate to the devs so they can buy one of these controllers and start playing with it/developing support for it. (and buy one for myself once there is some code to test) Anyone else wanting to chip in? What do the devs think? Is there room to put this on the roadmap in the near future?

[laurensvalk]

I applaud the good progress on making the official Lego controller work with PyBricks

:+1:

However: the latest Xbox controller seems to support BLE!

This is good to know. It would be even better if it uses a standard protocol similar to other input devices. If it does, this increases the chances we could ever do it, since space on the hubs is limited.

A good place to start would be to make (or search for) a regular Python script (so not Pybricks) that runs on your PC which reads the controller button state. Then we'd know what's theoretically required to be done on the hub firmware side.

What do the devs think? Is there room to put this on the roadmap in the near future?

So if everything above checks out, it might be part of category 4 of the first post in this thread, though that's perhaps not for the near future.

I would be very willing to donate to the devs

Thank you. We accept sponsors for the project. However, while we highly appreciate all contributions, we can't make promises as to which features get done first, if ever, since we do this in our free time.

[mwinkler]

The latest Xbox controller with BLE, supports HID over GATT, so its compatible to defined standards. Possibly for some features like the rumble motors are special GATT characteristics needed.

[NStrijbosch]

class LWP3Device(BLEDevice):
    pass

def lwp_connect("....") -> LWP3Device
    pass

class pybricks.pupdevices.RemoteControl(LWP3Device):
    pass

my_duplo_hub = lwp_connect("Duplo Hub")

Now that I am thinking about this: Will the my_duplo_hub be a stream object similar to nus, that collects all notifications received from a LWP3Device? (I think this will mean that notifications cannot be handled parallel to your program, or could this be possible with generator functions?)

Since we are talking about the absolute minimum could it be an option to define your own notification handler to obtain similar behaviour as a remote, i.e., all notifications handled in the background of your program. Something like this:

from pybricks.ble import lwp_connect
from pybricks.tools import wait

# custom notification handler
def notification_handler(message,state):
    if message[2] == 0x45: 
         if message[3] == 0x00:
               if message[4]==0x01:
                     state.left.plus = true
               elif message[4] == 0xFF:
                     state.left.min = true
         if message[3] == 0x01:
             .... 

#setup connection
Remote = lwp_connect("Remote",notification_handler)

# subscribe to left remote button
Remote.write([0x0A, 0x00, 0x41, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x01]) 

# main program
while true:
      print(Remote.state.left.plus)
      wait(100)

[laurensvalk]

Instead of a notification handler, we were thinking to keep incoming LWP3 messages in a finite queue that you can process at your own convenience.

[Tcm0]

This is good to know. It would be even better if it uses a standard protocol similar to other input devices. If it does, this increases the chances we could ever do it, since space on the hubs is limited.

https://github.com/ndeadly/MissionControl/issues?q=is%3Aissue+is%3Aopen+label%3A%22BLE+controller%22 lists some BLE controllers and some of them have documentation/logs linked.

[giadefa]

HI, sorry if the question has been answered already somewhere. I asked several months ago if it was possible to communicate between the hub and a python program on my PC in input and output. I practically need to send commands to the hub and read sensors back, i.e. communicate a bunch of floats back and forward. The answer was back then that some new feature would allow this. Anything new?

[laurensvalk]

While the simplified version discussed here is not yet implemented, it's certainly already possible to send data back and forth to a PC.

Could you open a separate issue to describe your use case? It would be easier to help that way. It would be good to know what kind of code is running on the PC side (Python or something else.)

[darvin]

Subscribing to get notified when it's done. Need it for finishing my Lego assembly robot replicator (it's like a 3d printer but it assembles Lego models from Lego instead of printing. Can assemble itself - its replicator!)