How to fix image problems when streaming h.264 via gstreamer udpsink

Using gstreamer I want to stream images from several Logitech C920 webcams to a Janus media server in RTP/h.264 format. The webcams produce h.264 encoded video streams, so I can send the streams to a UDP sink without re-encoding data, only payloading it.

I’m using the gst-interpipe plugin to switch between the different webcams, so that the video stream received by Janus stays the same, but with images coming from whatever webcam I choose.

It works but I’m experiencing some problems with broken frames where the colors are gray and details are blurred away, mainly the first 5 – 10 seconds after I switch between webcam source streams. After that the images correct themselves.

First frames

After 5 – 10 seconds or more

First I thought it was a gst-interpipe specific problem, but I can reproduce it by simply setting up two pipelines – one sending a video stream to a UDP sink and one reading from a UDP source:

gst-launch-1.0 -v -e v4l2src device=/dev/video0 ! queue ! video/x- 
h264,width=1280,height=720,framerate=30/1 ! rtph264pay 
config-interval=1 ! udpsink host=127.0.0.1 port=8004


gst-launch-1.0 -v udpsrc port=8004 caps = "application/x-rtp, 
media=video, clock-rate=90000, encoding-name=H264, payload=96" ! 
rtph264depay ! decodebin ! videoconvert ! xvimagesink

NB: I’m not experiencing this problem if I send the video stream directly to an xvimagesink, i.e. when not using UDP streaming.

Am I missing some important parameters in my pipelines? Is this a buffering issue? I really have no idea how to correct this. Any help is greatly appreciated.

Answer

Due to the nature of temporal dependencies of video streams you cannot just tune in into stream and expect it to be decode-able immediately. Correct decoding can only start at Random-Access-Point frames (e.g. I- or IDR-frames). before that you will get image data that rely on video frames you haven’t received – so they will look broken. Some decoders offers some control on what to do on these cases. libavdec_h264 for example has a output-corrupt option. (But actually I don’t how it behaves for “correct” frames which just are missing reference frames). Or they may have options to skip everything until a RAP-frame occurs. This depends on your specific decoder implementation. Note however that on any of these options the initial delay before you will see any image will increase.