chorus icon indicating copy to clipboard operation
chorus copied to clipboard

Voice Channel support

Open bitfl0wer opened this issue 2 years ago • 13 comments

Add support for sending and receiving live streams of WebRTC Video and Audio.

bitfl0wer avatar May 12 '23 21:05 bitfl0wer

At least this is well documented

Some relevant info:

  • Encryption is done using xsalsa (specifically xsalsa20_poly1305, xsalsa20_poly1305_suffix or xsalsa20_poly1305_lite), I've found the easiest way to implement this is probably the RustCrypto secretbox crate (which already combines salsa20 with poly1305) (Also, Discord says they themselves use libsodium and secretbox is part of NaCl-compat, which is a rust implementation of libsodium's API)

  • Audio is encoded with the Opus open audio codec (stereo, @ 48kHz). For opus in rust, we can use opus-rs, which provides high level bindings for libopus, so we can just consult the official opus documentation

(for encryption, see this part of discord's docs)

The encryption and encoding seems to be the hardest part of implementing this, the actual communication is very similar to the gateway

kozabrada123 avatar Jun 20 '23 08:06 kozabrada123

Oh and we also have no info about video, so we should probably implement voice first

kozabrada123 avatar Jun 20 '23 08:06 kozabrada123

Doing some research, video uses VP8 and VP9, maybe env-libvpx-sys in rust?

{
	"codecs": [
		{
			"name": "opus",
			"type": "audio",
			"priority": 1000,
			"payload_type": 109,
			"rtx_payload_type": null
		},
		{
			"name": "VP8",
			"type": "video",
			"priority": 2000,
			"payload_type": 120,
			"rtx_payload_type": 124
		},
		{
			"name": "VP9",
			"type": "video",
			"priority": 3000,
			"payload_type": 121,
			"rtx_payload_type": 125
		}
	]
}

kozabrada123 avatar Jun 20 '23 08:06 kozabrada123

Webrtc-rs may also be worth looking into, since it seems to support all the needed codecs

kozabrada123 avatar Jun 20 '23 08:06 kozabrada123

Potentially maybe perhaps looking into this on feature/webrtc I probably won't manage to implement it though lol

kozabrada123 avatar Jun 20 '23 17:06 kozabrada123

Good luck :)

bitfl0wer avatar Jun 20 '23 17:06 bitfl0wer

Also looking at how Serenity implemented this, they seem to have developed their own rtp parser and even their own opus bindings

kozabrada123 avatar Jun 21 '23 07:06 kozabrada123

I wonder if we could... yoink this :D

bitfl0wer avatar Jun 21 '23 07:06 bitfl0wer

yup :D

kozabrada123 avatar Jun 21 '23 07:06 kozabrada123

Slight change of plans, raw UDP first then webrtc because webrtc is really complex and raw udp seems to be way easier to implement for now

kozabrada123 avatar Jun 21 '23 12:06 kozabrada123

wasm cannot support udp.

kozabrada123 avatar Dec 29 '23 14:12 kozabrada123

See https://github.com/webrtc-rs/webrtc and https://github.com/wasm-peers/wasm-peers

kozabrada123 avatar Dec 29 '23 14:12 kozabrada123

A few notes:

  • First, it seems that UDP is not strictly the older and Webrtc the newer version; Discord seems to use UDP in their native clients and Webrtc on web - we could very likely do the same

  • Second, the official udp voice docs have been expanded a bit, notably in the encryption modes. Once I pick this back up again we should take a closer look at that

  • Third, I feel that #457 was about 90% of the way there. I just need to find a code snippet for how to decode opus from rtp packets in rust (or generally, how to use any of the opus bindings / libraries to decode opus packets)

kozabrada123 avatar Sep 11 '24 05:09 kozabrada123