roc-streaming
Add Opus support
Opus is a lossy streaming audio codec.
Specs:
Adding Opus support introduces a few challenges:
- currently RTP payload decoder is stateless; Opus decoder is stateful;
- currently RTP payload decoder and FEC decoder are separate pipeline steps; Opus decoder combines them.
Our fec::Decoder is packet-format-independent and works before decoding audio packet payload. Thus, it theoretically may be used in conjunction with Opus decoder, if it makes sense.
Opus decoder is stateful
How may it impact on code? And what does it mean literally? Do we have to provide every packet consecutively? We do so already. Are we disallowed to loose packets? This would mean that opus is designed for unrealistic channels.
Opus decoder combines them
I thought this is fec::Decoder's destiny.
How may it impact on code? And what does it mean literally?
Currently Streamer extracts samples from every packet independently using packet->read_samples().
With Opus, we need a different approach: Streamer (or anybody else who needs extracting samples) should ask opus decoder to extract samples from next (possibly lost) packet.
Are we disallowed to loose packets? This would mean that opus is designed for unrealistic channels.
No, of course.
I thought this is fec::Decoder's destiny.
It seems that Opus API combines decoding samples and repairing lost packets. Thus, if we need repaired packets before decoding samples (i.e. before Streamer), we probably should decode samples before Streamer and store them somewhere.
I don't think we could (and need) reuse fec::Decoder for Opus support. They're working on different layers: fec::Decoder works on packet layer and is payload-independent, while opus decoder actually works on audio stream layer.
One more issue (maybe):
Sampling rate to decode to (Hz). This must be one of 8000, 12000, 16000, 24000, or 48000.
May we make a special handle for Opus that looks like Reader of audio packets?
If Opus do FEC, we don't have to invoke OpenFEC and others. If not, we can and have to use fec::Decoder and Opus. I think we have to provide arguments letting user determine usage of third party stand alone FEC and Opuses FEC.
If Opus do FEC, we don't have to invoke OpenFEC and others. If not, we can and have to use fec::Decoder and Opus. I think we have to provide arguments letting user determine usage of third party stand alone FEC and Opuses FEC.
Agree. BTW, it may be interesting to combine them together.
May we make a special handle for Opus that looks like Reader of audio packets?
Not sure. In this case:
- for every received packet, we need to allocate additional buffer for decoded samples, or alternatively replace received packet with new non-opus packet (e.g. regular RTP PCM packet);
- for every repaired packet, we need to create new non-opus audio packet (e.g. regular RTP PCM packet) and fill it with repaired samples.
In other words, we pass packets to Opus decoder and it produces (virtually) audio stream. Then we split this stream on temporary (fake) packets and pass them to Streamer. And finally Streamer concatenates those packets into audio stream again. It makes little sense.
But we can avoid this:
- We introduce a concept of (stateful)
AudioSampleDecoder(allowed to perform FEC) which should be used inStreamer. This concept may be useful in future if we'll add support for more stream-level codecs. - Alternatively, we can implement
OpusStreamerthat replaces our regularStreamerwhen we're using Opus.
We have no places to mention sample rate anywhere except alsa driver or this kind of things
Another important note: we create Streamer for every channel and deinterleave packets before Streamer, while Opus decoder is used for all channels and we should deinterleave samples after it.
Note on FEC in Opus: http://lists.xiph.org/pipermail/opus/2013-January/001904.html