b.stefanuk.ca

Appearance

Using Gstreamer for AV Streaming ​

Terminology ​

  • RTP: Real-time Transport Protocol. A transport-layer protocol for delivering audio/visual streams over IP networks.
  • RTSP: Real-Time Streaming Protocol. A presentation-layer protocol for managing and displaying data from in RTP streams.

Basics ​

To start, it's a good idea to practice with a webcam as it illuminates a number of important concepts (Also see Using Video 4 Linux).

Streaming Webcam ​

sh
gst-launch-1.0 autovideosrc ! xvimagesink # *should* detect webcam
gst-launch-1.0 v4l2src ! xvimagesink      # *should* detect webcam via v4l2
gst-launch-1.0 autovideosrc ! xvimagesink # *should* detect webcam
gst-launch-1.0 v4l2src ! xvimagesink      # *should* detect webcam via v4l2

Typically you'll want more control, you can inspect the available video devices on your machine using v4l2-ctl --list-devices.

Multi-route Audio Streaming ​

You'll need a local WAV file on your computer, some examples can be found here. Start a Gstreamer pipeline with the gst-launch-1.0 command

sh
gst-launch-1.0 filesrc location=$HOME/Downloads/CantinaBand60.wav ! wavparse ! audioconvert ! rtpL24pay ! udpsink clients=localhost:12345,10.0.1.23:12345,192.168.15.16:12345
gst-launch-1.0 filesrc location=$HOME/Downloads/CantinaBand60.wav ! wavparse ! audioconvert ! rtpL24pay ! udpsink clients=localhost:12345,10.0.1.23:12345,192.168.15.16:12345

This will start an RTP stream and forward UDP packets to all of the destinations specified in the clients field.

Next, to receive the stream, run

sh
gst-launch-1.0 -v udpsrc address=localhost port=23456 ! 'application/x-rtp,media=audio,payload=96,clock-rate=22050,encoding-name=L24' ! rtpL24depay ! audioconvert ! autoaudiosink sync=false
gst-launch-1.0 -v udpsrc address=localhost port=23456 ! 'application/x-rtp,media=audio,payload=96,clock-rate=22050,encoding-name=L24' ! rtpL24depay ! audioconvert ! autoaudiosink sync=false
  • If the listening device isn't on localhost, then omit address=localhost
  • Pay attention to the clock-rate argument in the application/x-rtp capability. This is the effective sampling rate of the audio.

You should be able to hear a (potentially corrupted) audio stream at each of the configured clients.

Streaming Video Streams ​

Test UDP streaming

sh

gst-launch-1.0 videotestsrc name=mysource do-timestamp=true ! \
	videoconvert ! videoscale ! video/x-raw,width=480,height=300 ! videorate ! video/x-raw,framerate=1/1 ! \
	nvvidconv ! 'video/x-raw(memory:NVMM), format=(string)I420' ! \
	nvv4l2h264enc \
	preset-level=3 \
	profile=2 \
	iframeinterval=20 \
	control-rate=1 \
	bitrate=100000 \
	peak-bitrate=500000 \
	quant-i-frames=10 \
	quant-p-frames=30 \
	quant-b-frames=1 \
	num-B-Frames=0 ! \
	h264parse ! \
	udpsink host=192.168.168.123 port=1552

gst-launch-1.0 videotestsrc name=mysource do-timestamp=true ! \
	videoconvert ! videoscale ! video/x-raw,width=480,height=300 ! videorate ! video/x-raw,framerate=1/1 ! \
	nvvidconv ! 'video/x-raw(memory:NVMM), format=(string)I420' ! \
	nvv4l2h264enc \
	preset-level=3 \
	profile=2 \
	iframeinterval=20 \
	control-rate=1 \
	bitrate=100000 \
	peak-bitrate=500000 \
	quant-i-frames=10 \
	quant-p-frames=30 \
	quant-b-frames=1 \
	num-B-Frames=0 ! \
	h264parse ! \
	udpsink host=192.168.168.123 port=1552

To stream video you need to have an RTSP server, e.g., rtsp-simple-sever. You can use something like VLC as your RTSP client, which accepts and displays audio/video streams.

Using Gstreamer C++ with UDP ​

  1. In your C++ program, use OpenCV's gstreamer bindings to pass in a value to gstreamer, something like rtph264pay ! udpsink host=x.x.x.x port=12345. This informs gstreamer to stream video to the specified host:port endpoint.
  2. At the specified endpoint, run something like rtsp-simple-sever. Ensure that the ports and configurations of the RTSP server are set correctly. Depending on how it's set up, running this server will forward the UDP stream to other host:port locations, allowing for multiplexed streams. The default RTSP port is 554, though it can be configured to use a different port. It's best to run this as a docker-compose file.
  3. The stream should be available at a location like rtsp://localhost:554/stream1. Where stream1 was setup in the configuration.
  4. Make sure the RTSP server is up before you start the program running your stream.

Useful commands

docker compose up
docker container ls --all
docker compose up
docker container ls --all

ADDITIONAL RESOURCES