Add Real-Time Encoded Frame Streaming and Network Transmission Support

[MagellaX]

PR Description

Overview

This pull request introduces a robust, extensible real-time streaming architecture to the Windows Capture library. The new implementation enables direct access to encoded video (and audio) frames in memory, allowing for immediate transmission over network protocols such as TCP and UDP, without the need to write intermediate files. This addresses the requirements outlined in Issue #134 and lays the groundwork for advanced live streaming, remote desktop, and distributed video processing use cases.

Key Features

• StreamingVideoEncoder:
  A new encoder type that exposes encoded frames as soon as they are available, via a callback interface, rather than persisting them to disk. This enables low-latency, in-memory processing and transmission.

• FrameCallback Trait:
  A thread-safe, extensible callback interface for handling encoded video and audio frames, as well as stream lifecycle events. This allows users to implement custom frame consumers, including network transmission, in-memory analytics, or integration with external streaming protocols.

• Network Transmission Utilities:
  Built-in support for TCP and UDP streaming is provided via the NetworkCallback and supporting types. These utilities handle connection management, frame serialization, and rate limiting, and are designed for extensibility to support future protocols (e.g., WebRTC, RTMP).

• Backward Compatibility:
  The existing file-based encoding API and usage patterns remain fully supported. The new streaming functionality is opt-in and does not introduce breaking changes.

Technical Details

• EncodedFrame and EncodedAudioFrame:
  New data structures encapsulate encoded frame data, timestamps, frame type (key, delta, bidirectional), and relevant metadata. This enables precise synchronization and downstream processing.

• Threading and Synchronization:
  The streaming encoder and callback system are designed for thread safety and high throughput. Internal synchronization primitives ensure that frame delivery and resource cleanup are robust under concurrent workloads.

• Error Handling:
  All new APIs propagate errors using idiomatic Rust error types, with comprehensive coverage for network failures, encoding errors, and callback exceptions. Resource cleanup is guaranteed even in the presence of errors.

• Performance Considerations:
  The implementation leverages hardware acceleration where available, minimizes memory copies, and supports configurable frame rates and buffer sizes. Network transmission is rate-limited and supports backpressure.

• Extensibility:
  The architecture is designed to support additional protocols and custom frame consumers with minimal changes. The callback interface can be implemented for advanced use cases such as WebRTC, RTMP, or custom analytics pipelines.

Usage

• StreamingVideoEncoder can be instantiated with a user-defined FrameCallback implementation.
• The provided NetworkCallback supports TCP and UDP streaming out of the box.
• A new example (examples/streaming.rs) demonstrates real-time streaming and network transmission.

Migration

• Existing users are not affected unless they opt into the new streaming API.
• To migrate, replace VideoEncoder with StreamingVideoEncoder and provide a FrameCallback implementation.

Testing and Validation

• The implementation has been validated for thread safety, error propagation, and resource management.
• Example applications demonstrate end-to-end streaming and network transmission.
• All changes are covered by new or updated documentation.

Future Work

• Planned support for additional protocols (WebRTC, RTMP, HLS/DASH).
• Further optimizations for multi-stream and hardware-accelerated scenarios.
• Extended test coverage for edge cases and high-concurrency environments.

Summary

This PR provides a production-grade, extensible foundation for real-time video streaming and network transmission in the Windows Capture library, enabling a wide range of advanced use cases while maintaining backward compatibility and high performance.

[NiiightmareXD]

Wow, this is amazing.

I'll review it as soon as possible, just one thing, can we implement a trait system so it would get a stream to read and write to instead of manually opening a tcp or udp stream?

Thanks for contribution 😄

[lei1996]

Hope to add support for SRT stream transmission and RTSP stream transmission.

[amangupta20]

Hi @MagellaX, thank you for your work on this.

I noticed it was failing to compile with the latest dependencies, so I created a new pull request that incorporates your original commits and adds the necessary fixes.

Just wanted to give you a heads-up and credit for the initial implementation! You can see the updated version here: #171