AravisProject
Running two cameras simultaneously is problematic
Describe the bug If I run two cameras simulatenously application crashes in a few minutes.
I see following messages in output:
[09:31:06.748] 🅸 stream-thread> [GvStream::find_frame_data] Discard late frame 2590 (last: 2592)
[09:31:06.749] 🅸 sp> packet_type = ok (0x0000)
content_type = data-block (0x0003)
frame_id = 2590
packet_id = 1075
data_size = 1464
[09:31:06.749] 🅸 stream-thread> [GvStream::find_frame_data] Discard late frame 2590 (last: 2592)
[09:31:06.749] 🅸 sp> packet_type = ok (0x0000)
content_type = data-block (0x0003)
frame_id = 2590
packet_id = 1084
data_size = 1464
[09:31:06.749] 🅸 stream-thread> [GvStream::find_frame_data] Discard late frame 2590 (last: 2592)
[09:31:06.749] 🅸 sp> packet_type = ok (0x0000)
content_type = data-block (0x0003)
frame_id = 2590
packet_id = 1102
data_size = 1464
...
[11:36:50.804] 🅸 stream-thread> [GvStream::process_packet] Error packet at dt = 164855, packet id = 1292 frame id = 2462
[11:36:50.804] 🅸 sp> packet_type = packet-unavailable (0x800c)
content_type = data-block (0x0003)
frame_id = 2462
packet_id = 1292
data_size = 0
[11:36:50.804] 🅸 stream-thread> [GvStream::process_packet] Error packet at dt = 164855, packet id = 1300 frame id = 2462
[11:36:50.804] 🅸 sp> packet_type = packet-unavailable (0x800c)
content_type = data-block (0x0003)
frame_id = 2462
packet_id = 1300
data_size = 0
[11:36:50.866] 🆆 stream-thread> [GvStream::check_frame_completion] Timeout for frame 2462 at dt = 272625
[11:36:50.906] 🆆 stream-thread> [GvStream::check_frame_completion] Timeout for frame 2466 at dt = 153021
[11:36:50.920] 🆆 stream-thread> [GvStream::check_frame_completion] Timeout for frame 2462 at dt = 280730
[11:36:50.927] 🆆 stream-thread> [GvStream::check_frame_completion] Timeout for frame 2467 at dt = 132460
ERROR: from element /GstPipeline:pipeline0/GstAravis:aravis1: Internal data stream error.
Additional debug info:
gstbasesrc.c(3055): gst_base_src_loop (): /GstPipeline:pipeline0/GstAravis:aravis1:
streaming stopped, reason error (-5)
Execution ended after 0:01:41.273452363
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
...
[11:36:52.931] 🅸 stream> [GvStream::finalize] n_completed_buffers = 1777
[11:36:52.931] 🅸 stream> [GvStream::finalize] n_failures = 689
[11:36:52.931] 🅸 stream> [GvStream::finalize] n_underruns = 0
[11:36:52.931] 🅸 stream> [GvStream::finalize] n_timeouts = 689
[11:36:52.931] 🅸 stream> [GvStream::finalize] n_aborteds = 0
[11:36:52.931] 🅸 stream> [GvStream::finalize] n_missing_frames = 1
[11:36:52.931] 🅸 stream> [GvStream::finalize] n_size_mismatch_errors = 0
[11:36:52.931] 🅸 stream> [GvStream::finalize] n_received_packets = 3477608
[11:36:52.931] 🅸 stream> [GvStream::finalize] n_missing_packets = 546253
[11:36:52.931] 🅸 stream> [GvStream::finalize] n_error_packets = 4202
[11:36:52.931] 🅸 stream> [GvStream::finalize] n_ignored_packets = 3
[11:36:52.931] 🅸 stream> [GvStream::finalize] n_resend_requests = 2452851
[11:36:52.931] 🅸 stream> [GvStream::finalize] n_resent_packets = 1294976
[11:36:52.931] 🅸 stream> [GvStream::finalize] n_resend_ratio_reached = 7
[11:36:52.932] 🅸 stream> [GvStream::finalize] n_resend_disabled = 4202
[11:36:52.932] 🅸 stream> [GvStream::finalize] n_duplicated_packets = 2350
[11:36:52.932] 🅸 stream> [GvStream::finalize] n_transferred_bytes = 5104569508
[11:36:52.932] 🅸 stream> [GvStream::finalize] n_ignored_bytes = 24
[11:36:52.932] 🅸 stream> [Stream::finalize] Flush 50 buffer[s] in input queue
[11:36:52.932] 🅸 stream> [Stream::finalize] Flush 0 buffer[s] in output queue
...
[11:36:52.964] 🅸 stream> [GvStream::finalize] n_completed_buffers = 1804
[11:36:52.964] 🅸 stream> [GvStream::finalize] n_failures = 714
[11:36:52.964] 🅸 stream> [GvStream::finalize] n_underruns = 0
[11:36:52.964] 🅸 stream> [GvStream::finalize] n_timeouts = 714
[11:36:52.964] 🅸 stream> [GvStream::finalize] n_aborteds = 0
[11:36:52.964] 🅸 stream> [GvStream::finalize] n_missing_frames = 1
[11:36:52.964] 🅸 stream> [GvStream::finalize] n_size_mismatch_errors = 0
[11:36:52.964] 🅸 stream> [GvStream::finalize] n_received_packets = 3547379
[11:36:52.964] 🅸 stream> [GvStream::finalize] n_missing_packets = 622990
[11:36:52.964] 🅸 stream> [GvStream::finalize] n_error_packets = 3597
[11:36:52.964] 🅸 stream> [GvStream::finalize] n_ignored_packets = 10
[11:36:52.964] 🅸 stream> [GvStream::finalize] n_resend_requests = 2418993
[11:36:52.964] 🅸 stream> [GvStream::finalize] n_resent_packets = 1419183
[11:36:52.964] 🅸 stream> [GvStream::finalize] n_resend_ratio_reached = 11
[11:36:52.964] 🅸 stream> [GvStream::finalize] n_resend_disabled = 3597
[11:36:52.964] 🅸 stream> [GvStream::finalize] n_duplicated_packets = 3318
[11:36:52.964] 🅸 stream> [GvStream::finalize] n_transferred_bytes = 5208011788
[11:36:52.964] 🅸 stream> [GvStream::finalize] n_ignored_bytes = 80
[11:36:52.965] 🅸 stream> [Stream::finalize] Flush 48 buffer[s] in input queue
[11:36:52.965] 🅸 stream> [Stream::finalize] Flush 2 buffer[s] in output queue
To Reproduce
Run pipeline ARV_DEBUG=all gst-launch-1.0 aravissrc camera-name="192.168.5.38" ! fakesink aravissrc camera-name="192.168.5.39" ! fakesink
Also it is possible to run two separate pipelines (ARV_DEBUG=all gst-launch-1.0 aravissrc camera-name="192.168.5.38" ! fakesink` and ARV_DEBUG=all gst-launch-1.0 aravissrc camera-name="192.168.5.39" ! fakesink` in my case) with delay of several seconds to see interference between them.
Expected behavior Application works until manually stopped.
Camera description:
- Daheng Imaging
- MER2-503-23GM
- Ethernet
Platform description:
- Aravis commit 8144377dde5d7a0c391722cc9ad7c0f5e8e09865
- Ubuntu 20.04.4 LTS (Focal Fossa) 5.4.0-124-generic x86_64
- x86_64
Additional context Amount and intensity of errors and warnings in logs varies over time. I suspect there may be some network issue. How can I debug that?
Is there enough bandwidth for the 2 streams ?
Did you adjust with the inter packet delay ?
Could you also enable gstreamer debugging output using GST_DEBUG ?
Is there enough bandwidth for the 2 streams ?
nload shows that single camera consumes ~410 MBit/s without any issues, while two cameras consume ~820 MBit/s with problem described above.
Did you adjust with the inter packet delay ?
Can you please rephrase, I'm not sure I understand question.
Could you also enable gstreamer debugging output using GST_DEBUG ?
gstreamer provides lots of debug information, what exactly do you want to see?
Is there enough bandwidth for the 2 streams ?
nloadshows that single camera consumes ~410 MBit/s without any issues, while two cameras consume ~820 MBit/s with problem described above.
If you have a 1 Gb/s link, 820 Mb/s is not that far to use all the available bandwidth.
Please have a look at the ethernet performance related chapter in aravis documentation:
https://aravisproject.github.io/aravis/ethernet.html
Did you adjust with the inter packet delay ?
Can you please rephrase, I'm not sure I understand question.
Have a look at the stream channel packet delay chapter:
https://www.get-cameras.com/Files/8/122000/122647/FileBrowser/camera-manuals/mercury2-gige-cameras-user-manual.pdf
Could you also enable gstreamer debugging output using GST_DEBUG ?
gstreamer provides lots of debug information, what exactly do you want to see?
Any information indicating why the pipeline stops.
Long time, no news. I'm closing this issue. Don't hesitate to reopen if you have the requested information.
Cheers.
Hi! I am sorry for not replying sooner.
It seems it was a network issue. As we reconfigured network problem dissapeared (or at least are not that frequent).
Thanks for help!