ingress icon indicating copy to clipboard operation
ingress copied to clipboard

Fix: handle percent-encoded slashes (%2F) in RTMP publishing name and add diagnostics

Open giacaglia opened this issue 6 months ago • 14 comments

This PR fixes an issue where RTMP ingress could reject valid publish attempts from some encoders (e.g., Riverside) that send a percent-encoded slash in the publishing path — like x%2F instead of x/. LiveKit previously treated this as a single malformed stream name and returned 404 before tracks published.

Changes

  • Unescape percent-encoded characters in the incoming PublishingName (url.PathUnescape) to correctly handle x%2F.

giacaglia avatar Oct 28 '25 23:10 giacaglia

CLA assistant check
All committers have signed the CLA.

CLAassistant avatar Oct 28 '25 23:10 CLAassistant

@biglittlebigben - That makes sense. I'm looking for a way to solve an issue involving Riverside and LiveKit’s RTMP ingest. This bug is blocking our migration to use LiveKit (it currently works fine with Cloudflare Stream).

We’ve confirmed that LiveKit RTMP ingest works with other third-party providers (OBS, ffmpeg), but not with Riverside. Any ideas on how we might fix this issue or further debug it? 🙏

Here’s a detailed description of the issue: https://github.com/livekit/ingress/issues/390

giacaglia avatar Oct 29 '25 18:10 giacaglia

As you can see from the code you modified, ingress really doesn't pay attention to the "app" field if the RTMP URL ('x' in this case). It's only there because some tools (ie ffmpeg) break or are harder to use if there is no "app" section in the RTMP URL.

So leaving out the "/x" section of the URL entirely should work. Stepping back, The way this is expected to work is by appending the "/x" to the end of "Stream URL" and setting the "Stream Key" to the stream key value. Is neither of these options work, there is another incompatibility that needs to be investigated. Providing logs would be needed to investigate.

biglittlebigben avatar Oct 29 '25 20:10 biglittlebigben

@biglittlebigben That makes sense. Neither options work with Riverside. Here is more information (and logs that could help you identify the issue):

🕐 Timeline & Logs (UTC)

2025-10-27 21:25:28.001-05 → 2025-10-28 02:25:28 UTC

  • LiveKit ingress created:
ingressId: [id]
url: rtmps://pinetree-[identity].rtmp.livekit.cloud/x
streamKey: [key]
roomName: 30-livestream
participantIdentity: ingress:30
enableTranscoding: true
reusable: true
  • Verified working in OBS and ffmpeg with the same credentials.

2025-10-28 02:28:03 UTC Successful OBS session:

event: ingress_started
status: ENDPOINT_PUBLISHING
roomId: [id]
resourceId: [resource-id]

Followed by:

event: ingress_started

Then:

event: ingress_ended
status: CLEAN END

2025-10-27 23:55 UTC → 2025-10-28 00:05 UTC Riverside attempts connection:

event: ingress_ended
state: {
  status: 'ENDPOINT_ERROR',
  error: 'HTTP request failed with code 404',
  resourceId: '[resource-id]'
}

Multiple repeats over a few seconds:

resourceIds: ....

⛔ Each attempt fails immediately after connect handshake — no track_published, no audio/video frames received.

Is there anything else I can share that might help debug this, e.g. network traces, full webhook dumps, or ingress creation payloads? Happy to provide whatever is useful.

giacaglia avatar Oct 29 '25 20:10 giacaglia

Looking at all the sessions above: ingress parses the stream key properly and fetches a valid ingress context. However, the RTMP client (Riverside) disconnects immediately after the handshake. The error reported In the ingress status is due to a race condition in the teardown path when the ingress is shutdown at the same time it is being started. It doesn't change the fact that ingress doesn't connect to the room because the RTMP encoder disconnected immediately.

Riverside may have some logs explaining why it's disonnecting

biglittlebigben avatar Oct 29 '25 23:10 biglittlebigben

@biglittlebigben contacted their team! Will keep you updated!

giacaglia avatar Oct 30 '25 03:10 giacaglia

We found that this is caused by a bug in the underlying RTMP server library we're using that was only exposed by Riverside because they use small RTMP chunk size.

biglittlebigben avatar Oct 31 '25 03:10 biglittlebigben

@biglittlebigben Oh nice!!! 🤩

giacaglia avatar Oct 31 '25 04:10 giacaglia

@biglittlebigben Let us know once the bug on the RTMP server is fixed! That’ll unblock us from using LiveKit

giacaglia avatar Nov 03 '25 15:11 giacaglia

Hey @biglittlebigben, thank you for all the work on this 🙏.

Do you have a rough ETA for the Riverside ingress fix? We’re moving from Cloudflare Stream to LiveKit and this is our last blocker to release this product. Even a ballpark would help with planning

giacaglia avatar Nov 06 '25 18:11 giacaglia

Please try again on livekit cloud

biglittlebigben avatar Nov 06 '25 21:11 biglittlebigben

Hey @biglittlebigben thank you so much for all the help so far! 🙏

Now I see the room starting well, but I’m still hitting issues with Riverside that I don’t see with other 3rd party software studios (OBS, Meld Studio, etc...).

🧭 Summary

OBS / Meld Studio → ✅ audio and video publish and play for subscribers.

Riverside → ❌ no sustained video, and the audio track (opus) is published but inaudible to subscribers.

🔧 Ingress config (reusable)

ingressId: IN_DP9bfvMwc44H
url: rtmps://pinetree-0niuqhg1.rtmp.livekit.cloud/x
streamKey: ayxhkRaAkrHc
roomName: 30-livestream
participantIdentity: ingress:30
enableTranscoding: true
reusable: true

Viewer SDK: LiveKit JS 2.15.4 (auto_subscribe=1)

🕐 Timeline & Logs (UTC)

Riverside attempt (problem: video missing/unstable, audio published but inaudible)

Room & ingress

event: room_started
  room: 30-livestream (sid: RM_mdegYdJrkFhc)

event: ingress_started
  state.status: ENDPOINT_PUBLISHING
  participantIdentity: ingress:30
  state.resourceId: RT_DNHdodUpsFG4

event: participant_joined
  identity: ingress:30
  kind: INGRESS

Audio publishes (but can’t hear it on subscribers)

event: track_published
  participant: ingress:30
  track.source: MICROPHONE
  track.mimeType: audio/opus
  track.stereo: true

IDs for this Riverside session

Range: 2025-11-07 13:46:15.522-06 → 2025-11-07T19:46:16.245Z
Room:         RM_mdegYdJrkFhc
Ingress:      IN_DP9bfvMwc44H  (participantIdentity: ingress:30)
ResourceId:   RT_DNHdodUpsFG4
Egress IDs:   EG_5DpqhsyUSNpK, EG_aQQvgS3pT5zC

OBS attempt (works: audio + video sustained)

Highlights

event: room_started
event: participant_joined  identity: ingress:30  kind: INGRESS
event: track_published     source: MICROPHONE     mimeType: audio/opus
event: track_published     type: VIDEO  source: CAMERA  mimeType: video/H264
event: egress_started      EG_eEZSwYZcngBS, EG_DtXgf4Jgdv5g
event: egress_updated      status: EGRESS_ACTIVE (both)
event: egress_ended        status: EGRESS_COMPLETE (both)
event: track_unpublished   VIDEO then AUDIO
event: ingress_ended

IDs for this OBS session

Room:         RM_ZgznxDyQyUxe
Ingress:      IN_DP9bfvMwc44H  (participantIdentity: ingress:30)
ResourceId:   RT_2pLLH22qMC8u
Egress IDs:   EG_eEZSwYZcngBS, EG_DtXgf4Jgdv5g

With OBS (and also Meld Studio), both audio and video are visible/audible to subscribers without issues.

Is there anything else I can share that might help debug this, e.g. network traces, full webhook dumps, or ingress creation payloads? Happy to provide whatever is useful.

giacaglia avatar Nov 07 '25 20:11 giacaglia

@biglittlebigben have you seen the above? I think your fix surfaced another issue with video and audio tracks in Riverside

Mind taking a look and letting us know what you think?

pranaymohan avatar Nov 11 '25 17:11 pranaymohan

Please retry with the current version of ingress.

biglittlebigben avatar Nov 21 '25 03:11 biglittlebigben