Introduction
User bocorps pointed me in the direction of gst-rpicamsrc. This is “… a GStreamer wrapper around the raspivid/raspistill functionality of the RaspberryPi, providing a GStreamer source element capturing from the Rpi camera.”
What this means is that instead of piping the output of raspivid into gstreamer, gstreamer has a source element to read the camera directly. This is similar to using the video4linux (v4l) source element, but negates the need for a v4l driver.
My hope was that by integrating the camera functionality into a gstreamer source element the latency would be reduced. Unfortunately, I actually saw an 18% increase in latency.
Installation
Before I could use the gst-rpicamsrc element, I needed to download the source and build it. As I was working with a minimal install of Raspbian Jessie, I needed to install the git package before I could do anything else
sudo apt-get install git
With git installed I could download the latest sources for gst-rpicamsrc.
git clone https://github.com/thaytan/gst-rpicamsrc.git
With that done a look in the REQUIREMENTS file indicated what other packages were needed in order to accomplish the build.
sudo apt-get install autoconf automake libtool libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev libraspberrypi-dev
Finally, I was able to complete the build and install.
./autogen --prefix=/usr --libdir=/usr/lib/arm-linux-gnueabihf/ make sudo make install
The command `gst-inspect-1.0 rpicamsrc’ produces a list of the available parameters.
Factory Details:
Rank none (0)
Long-name Raspberry Pi Camera Source
Klass Source/Video
Description Raspberry Pi camera module source
Author Jan Schmidt <jan@centricular.com>
Plugin Details:
Name rpicamsrc
Description Raspberry Pi Camera Source
Filename /usr/lib/arm-linux-gnueabihf/gstreamer-1.0/libgstrpicamsrc.so
Version 1.0.0
License LGPL
Source module gstrpicamsrc
Binary package GStreamer
Origin URL http://gstreamer.net/
GObject
+----GInitiallyUnowned
+----GstObject
+----GstElement
+----GstBaseSrc
+----GstPushSrc
+----GstRpiCamSrc
Pad Templates:
SRC template: 'src'
Availability: Always
Capabilities:
video/x-h264
width: [ 1, 2147483647 ]
height: [ 1, 2147483647 ]
framerate: [ 0/1, 2147483647/1 ]
stream-format: byte-stream
alignment: au
profile: { baseline, main, high }
Element Flags:
no flags set
Element Implementation:
Has change_state() function: gst_base_src_change_state
Element has no clocking capabilities.
Element has no URI handling capabilities.
Pads:
SRC: 'src'
Implementation:
Has getrangefunc(): gst_base_src_getrange
Has custom eventfunc(): gst_base_src_event
Has custom queryfunc(): gst_base_src_query
Has custom iterintlinkfunc(): gst_pad_iterate_internal_links_default
Pad Template: 'src'
Element Properties:
name : The name of the object
flags: readable, writable
String. Default: "rpicamsrc0"
parent : The parent of the object
flags: readable, writable
Object of type "GstObject"
blocksize : Size in bytes to read per buffer (-1 = default)
flags: readable, writable
Unsigned Integer. Range: 0 - 4294967295 Default: 4096
num-buffers : Number of buffers to output before sending EOS (-1 = unlimited)
flags: readable, writable
Integer. Range: -1 - 2147483647 Default: -1
typefind : Run typefind before negotiating
flags: readable, writable
Boolean. Default: false
do-timestamp : Apply current stream time to buffers
flags: readable, writable
Boolean. Default: true
bitrate : Bitrate for encoding
flags: readable, writable
Integer. Range: 1 - 25000000 Default: 17000000
preview : Display preview window overlay
flags: readable, writable
Boolean. Default: true
preview-encoded : Display encoder output in the preview
flags: readable, writable
Boolean. Default: true
preview-opacity : Opacity to use for the preview window
flags: readable, writable
Integer. Range: 0 - 255 Default: 255
fullscreen : Display preview window full screen
flags: readable, writable
Boolean. Default: true
sharpness : Image capture sharpness
flags: readable, writable
Integer. Range: -100 - 100 Default: 0
contrast : Image capture contrast
flags: readable, writable
Integer. Range: -100 - 100 Default: 0
brightness : Image capture brightness
flags: readable, writable
Integer. Range: 0 - 100 Default: 50
saturation : Image capture saturation
flags: readable, writable
Integer. Range: -100 - 100 Default: 0
iso : ISO value to use (0 = Auto)
flags: readable, writable
Integer. Range: 0 - 3200 Default: 0
video-stabilisation : Enable or disable video stabilisation
flags: readable, writable
Boolean. Default: false
exposure-compensation: Exposure Value compensation
flags: readable, writable
Integer. Range: -10 - 10 Default: 0
exposure-mode : Camera exposure mode to use
flags: readable, writable
Enum "GstRpiCamSrcExposureMode" Default: 1, "auto"
(0): off - GST_RPI_CAM_SRC_EXPOSURE_MODE_OFF
(1): auto - GST_RPI_CAM_SRC_EXPOSURE_MODE_AUTO
(2): night - GST_RPI_CAM_SRC_EXPOSURE_MODE_NIGHT
(3): nightpreview - GST_RPI_CAM_SRC_EXPOSURE_MODE_NIGHTPREVIEW
(4): backlight - GST_RPI_CAM_SRC_EXPOSURE_MODE_BACKLIGHT
(5): spotlight - GST_RPI_CAM_SRC_EXPOSURE_MODE_SPOTLIGHT
(6): sports - GST_RPI_CAM_SRC_EXPOSURE_MODE_SPORTS
(7): snow - GST_RPI_CAM_SRC_EXPOSURE_MODE_SNOW
(8): beach - GST_RPI_CAM_SRC_EXPOSURE_MODE_BEACH
(9): verylong - GST_RPI_CAM_SRC_EXPOSURE_MODE_VERYLONG
(10): fixedfps - GST_RPI_CAM_SRC_EXPOSURE_MODE_FIXEDFPS
(11): antishake - GST_RPI_CAM_SRC_EXPOSURE_MODE_ANTISHAKE
(12): fireworks - GST_RPI_CAM_SRC_EXPOSURE_MODE_FIREWORKS
metering-mode : Camera exposure metering mode to use
flags: readable, writable
Enum "GstRpiCamSrcExposureMeteringMode" Default: 0, "average"
(0): average - GST_RPI_CAM_SRC_EXPOSURE_METERING_MODE_AVERAGE
(1): spot - GST_RPI_CAM_SRC_EXPOSURE_METERING_MODE_SPOT
(2): backlist - GST_RPI_CAM_SRC_EXPOSURE_METERING_MODE_BACKLIST
(3): matrix - GST_RPI_CAM_SRC_EXPOSURE_METERING_MODE_MATRIX
awb-mode : White Balance mode
flags: readable, writable
Enum "GstRpiCamSrcAWBMode" Default: 1, "auto"
(0): off - GST_RPI_CAM_SRC_AWB_MODE_OFF
(1): auto - GST_RPI_CAM_SRC_AWB_MODE_AUTO
(2): sunlight - GST_RPI_CAM_SRC_AWB_MODE_SUNLIGHT
(3): cloudy - GST_RPI_CAM_SRC_AWB_MODE_CLOUDY
(4): shade - GST_RPI_CAM_SRC_AWB_MODE_SHADE
(5): tungsten - GST_RPI_CAM_SRC_AWB_MODE_TUNGSTEN
(6): fluorescent - GST_RPI_CAM_SRC_AWB_MODE_FLUORESCENT
(7): incandescent - GST_RPI_CAM_SRC_AWB_MODE_INCANDESCENT
(8): flash - GST_RPI_CAM_SRC_AWB_MODE_FLASH
(9): horizon - GST_RPI_CAM_SRC_AWB_MODE_HORIZON
image-effect : Visual FX to apply to the image
flags: readable, writable
Enum "GstRpiCamSrcImageEffect" Default: 0, "none"
(0): none - GST_RPI_CAM_SRC_IMAGEFX_NONE
(1): negative - GST_RPI_CAM_SRC_IMAGEFX_NEGATIVE
(2): solarize - GST_RPI_CAM_SRC_IMAGEFX_SOLARIZE
(3): posterize - GST_RPI_CAM_SRC_IMAGEFX_POSTERIZE
(4): whiteboard - GST_RPI_CAM_SRC_IMAGEFX_WHITEBOARD
(5): blackboard - GST_RPI_CAM_SRC_IMAGEFX_BLACKBOARD
(6): sketch - GST_RPI_CAM_SRC_IMAGEFX_SKETCH
(7): denoise - GST_RPI_CAM_SRC_IMAGEFX_DENOISE
(8): emboss - GST_RPI_CAM_SRC_IMAGEFX_EMBOSS
(9): oilpaint - GST_RPI_CAM_SRC_IMAGEFX_OILPAINT
(10): hatch - GST_RPI_CAM_SRC_IMAGEFX_HATCH
(11): gpen - GST_RPI_CAM_SRC_IMAGEFX_GPEN
(12): pastel - GST_RPI_CAM_SRC_IMAGEFX_PASTEL
(13): watercolour - GST_RPI_CAM_SRC_IMAGEFX_WATERCOLOUR
(14): film - GST_RPI_CAM_SRC_IMAGEFX_FILM
(15): blur - GST_RPI_CAM_SRC_IMAGEFX_BLUR
(16): saturation - GST_RPI_CAM_SRC_IMAGEFX_SATURATION
(17): colourswap - GST_RPI_CAM_SRC_IMAGEFX_COLOURSWAP
(18): washedout - GST_RPI_CAM_SRC_IMAGEFX_WASHEDOUT
(19): posterise - GST_RPI_CAM_SRC_IMAGEFX_POSTERISE
(20): colourpoint - GST_RPI_CAM_SRC_IMAGEFX_COLOURPOINT
(21): colourbalance - GST_RPI_CAM_SRC_IMAGEFX_COLOURBALANCE
(22): cartoon - GST_RPI_CAM_SRC_IMAGEFX_CARTOON
rotation : Rotate captured image (0, 90, 180, 270 degrees)
flags: readable, writable
Integer. Range: 0 - 270 Default: 0
hflip : Flip capture horizontally
flags: readable, writable
Boolean. Default: false
vflip : Flip capture vertically
flags: readable, writable
Boolean. Default: false
roi-x : Normalised region-of-interest X coord
flags: readable, writable
Float. Range: 0 - 1 Default: 0
roi-y : Normalised region-of-interest Y coord
flags: readable, writable
Float. Range: 0 - 1 Default: 0
roi-w : Normalised region-of-interest W coord
flags: readable, writable
Float. Range: 0 - 1 Default: 1
roi-h : Normalised region-of-interest H coord
flags: readable, writable
Float. Range: 0 - 1 Default: 1
Usage
Because of the way gstreamer works, the parameters for the feed needed to be split and re-arranged in the streamer pipeline. Previously all the parameters are specified as part of raspivid.
/opt/vc/bin/raspivid -t $DURATION -w $WIDTH -h $HEIGHT -fps $FRAMERATE -b $BITRATE -n -pf high -o - | gst-launch-1.0 -v fdsrc !
GStreamer parameters like width, height and frame rate are configured through capabilities (caps) negotiation with the next element. Other parameters like the bit rate and preview screen are controlled as part of the source element.
gst-launch-1.0 rpicamsrc bitrate=$BITRATE preview=0 ! video/x-h264,width=$WIDTH,height=$HEIGHT,framerate=$FRAMERATE/1 !
The new stream script is
#!/bin/bash source remote.conf if [ "$1" != "" ] then export FRAMERATE=$1 fi NOW=`date +%Y%m%d%H%M%S` FILENAME=$NOW-Tx.h264 gst-launch-1.0 rpicamsrc bitrate=$BITRATE preview=0 ! video/x-h264,width=$WIDTH,height=$HEIGHT,framereate=$FRAMERATE/1,profile=high ! h264parse ! rtph264pay config-interval=1 pt=96 ! udpsink host=$RX_IP port=$UDPPORT
Tests
This test was a straight comparison between the old and new scripts using the same settings and the same testing methodology as previously. The resolution was 1280 x 720 pixels with a 6Mbps bitrate.
Update : Since the original article was published use Jan Schmitt spotted that I had misspelled “framerate” as “framereate” in the gst-rpicamsrc script. He also suggested I should try using the baseline profile and a queue element to decouple the video capture from the UDP transmission. With this in mind I have re-run the tests.
Results
Almost immediately I had the feeling that gst-rpicamsrc has slower. Analysis of the video showed I was correct. The latency using gst-rpicamsrc was 18% higher than using raspivid.
Update:
Running the original erroneous script with debugging on showed that the capture was running at 30fps instead of the intended 48 fps. Here are the new results averaged from 10 cycles.
| Script | gst-rpicamsrc @ 48 fps |
raspivid @ 48 fps |
||
| Profile | No queue | With queue | No queue | With queue |
| baseline | 184.2 | 175.4 | 153.9 | 156.9 |
| high | 186.2 | 185 | 154.1 | 159.7 |
gst-rpicamsrc @ 30 fps, high profile, no queue = 198.2 ms
Analysis
The first thing to note is that the raspivid latency (no queue, high profile) has risen from the 126ms found in the last tests to 154ms. The only difference was that I cloned the Sandisk microSDHC card onto a Transcend 8GB. I’ll set up some more tests to compare the cards. As these tests were run from the same card and from the same boot, they are still valid for comparison.
It is immediately obvious that the gst-rpicamsrc latency is about 20% higher than the raspivid script, so the conclusion from the first publish of this article still stands.
What can be added is that using the baseline profile, does reduce the latency a little: 1 to 3ms in most cases.
Adding a queue element does provide a benefit for the gst-rpicamsrc script, especially with the baseline profile where a 9ms reduction in latency was observed. For the raspivid script adding a queue element actually increased the latency by 3 to 4ms. I suspect this is because the video stream is already decoupled from gstreamer by being piped in from an external process.
Conclusion
Using gst-rpicamsrc provides no benefit for reducing latency over raspivid. That is not to say gst-rpicamsrc provides no other benefits. For any use other than FPV, I would definitely use gst-rpicamsrc instead of having to pipe the video in through stdin. It provides plenty of options for setting up the video stream as the command `gst-inspect-1.0 rpicamsrc’ above showed.
The problem here is that I am targeting this development for FPV use where low latency is the driving factor. At the moment my lowest latency for a adequate quality HD stream is 125ms and I really need to get this under 100ms to compete with current analog standard definition systems. Whether it is possible to shave of another 25ms remains to be seen.
Update: Following the additional tests I would add that it is better to use the baseline profile over the high profile.
..you found latency using gst-rpicamsrc was 18% higher, but it may have other useful features, please post output of “gst-inspect-1.0 rpicamsrc”.
(This comment hit the wrong thread at first, belongs here IMO.)
Thanks for your comment. I did think my original conclusion as somewhat terse and as you say rpicamsrc
has plenty ofmay have some useful features even if it can’t improve the latency. Therefore, I’ve extended the conclusion to make it more balanced. If you know of any way to improve the speed of the gst-rpicamsrc element please let me know...I never said it “has plenty of useful features”, I said “it may have” some, to learn more on these, I asked you to post the output of “gst-inspect-1.0 rpicamsrc”.
I’ve added the output from gst-inspect-1.0 rpicamsrc as requested.
..running rpicamsrc, did you try to disable the preview?
..disabling the preview with the good old pipe way, got my latency lower, around 150ms, AFAIR.
I did have the preview turned off. I even connected it to a monitor to check. I did find that you can use preview=0 or preview=false. My best average time for a useable 1280×720 stream using raspivid is 125ms.
You misspelled ‘framerate’ in your capsfilter – “framereate”. You may not be getting the framerate you expect, which could affect your latency measure. You might also benefit from having a ‘queue’ element in your pipeline before udpsink to decouple the capture and the udp transmission into 2 separate threads.
You’ll get better latency using baseline profile instead of high too. B-frames will hurt you.
Hi Jan.
Thanks for the spot. As the post wasn’t created on the Pi, I can’t remember if pasted in my actual script or retyped it. Either way I will check what the script says and rerun the tests if necessary. I’ll also do some back to back testing of base versus high profile.
Hi Jan.
Thanks for the spot. As the post wasn’t created on the Pi, I can’t remember if pasted in my actual script or retyped it. Either way I will check what the script says and rerun the tests if necessary. I’ll also do some back to back testing of Base versus high profile and with/without a queue element.
Hello,
I am using Raspberry Pi to stream video. I type the comment : gst-launch-1.0 rpicamsrc bitrate=90000 preview=0 ! video/x-h264,width=1280,height=720,profile=high ! h264parse ! rtph264pay config-interval=1 pt=96 ! udpsink host=192.168.178.24(IP address of Pi) port=5000. But i do not know how to watch the stream. Can you show me the command for receiver? Thanks a lot!
Ragards,
SonNam
Have a look at the script 4 (tsplay) on the current scripts page at https://sparkyflight.wordpress.com/current-raspberry-pi-fpv-scripts/
Use script 5 to record the stream as well.
You will also need the video and network configuration files.
I use the script 4 to receive the stream. Do i need to type the first threes line? Because when i type it ubuntu it said : source ./video.conf no such file or directory
You need to create the video.conf and network.conf files also given on the current scripts page but modified to match your setup. Put them in the same directory as tsplay.
HI,
I did all but the ./video.conf is still getting error. I type:
export TX_IP=192.168.178.24
export RX_IP=$(ip -o 192.168.32.132 | sed -n ‘s/.*inet \([0-9.]*\)\/.. .*/\1/p’)
export TCPPORT=5001
export UDPPORT=5000
export USER=pi
In the line beginning export RX_IP you DON’T replace the text “addr” with an IP address. Leave it as “export RX_IP=$(ip -o addr show wlan0 | sed -n ‘s/.*inet \([0-9.]*\)\/.. .*/\1/p’)”
I leave it as eth0 because i dont have wlan0. But it still get error. Do i have to do more than these steps ? 😀
hi, since i got some helpful tips on your blog, i took the liberty to share it on diydrones http://goo.gl/3DRRlg
Hi Gary
Very interesting tests you did here! Maybe you want to take a look into a small project I’ve started: https://befinitiv.wordpress.com/2015/01/25/true-unidirectional-wifi-broadcasting-of-video-data-for-fpv/
The basic idea is: Broadcast video data with 802.11 packets directly into the air, without association to an access point (thus no risk of loosing it). A weaker link quality just degrades the image quality (as with analog links) but does not terminate it.
It would be nice if you could share your thoughts on this, maybe even participate on that project 🙂
Best regards,
befinitiv.
Hi befinitiv,
I’ve had a quick read of your post and it looks interesting. A have a couple of initial thoughts.
1. How does it handle a 720p HD stream. Finding a low latency HD solution is my main aim. For an SD stream, which would include 640×480, current analogue systems still offer the lowest latency solution.
2. Have you though of using MJPEG for the stream. As each frame would be complete you wouldn’t need to send them three times.
I’ll try to get it compiled and tested in the next couple of weeks.
I compared the latency of rpicamsrc to piping it into gstreamer, and oddly enough found rpicamsrc slightly faster. With a pipe I measured 160-180 ms of delay, while rpicamsrc managed 120-140. This might be because I was using lower resolution video (400×240, 30FPS) or possibly rpicamsrc has improved in the last year.
Here is the pipeline I used:
gst-launch-1.0 rpicamsrc keyframe-interval=10 preview=0 ! video/x-h264,width=400,height=240,framerate=30/1,profile=baseline ! h264parse ! queue ! rtph264pay config-interval=1 pt=96 ! udpsink host=192.168.1.23 port=9000
I’m quite surprised by the latency you are getting at 400×240 resolution. I was able to get 90ms at 640×480. What device are you using on the receiving end and what bitrate are you using?
Both ends are a Raspberry Pi 2B running Jessie, connected via a pair of TEW-820AP.
I’m currently using the default bitrate setting, it didn’t seem to have much effect when I played around with it. The default setting is something like 17mbit/s but it’s not using anywhere near that.
Reception end uses this pipeline:
udpsrc port=9000 caps=application/x-rtp,media=(string)video,clock-rate=(int)90000,encoding-name=(string)H264 ! rtph264depay ! avdec_h264 ! queue ! eglglessink
Using the hardware decoder (omxh264dec) is about the same as avdec_h264, maybe 5ms faster. Adding and removing the queue didn’t seem to make a reliably measurable difference either.
I’m measuring latency by putting a stop watch in the camera’s field of view, positioning my monitor next to the stop watch, and taking a photo of the stop watch and the monitor, and then subtracting to get the lag. Is there a better way to do this?
./autogen –prefix=/usr –libdir=/usr/lib/arm-linux-gnueabihf/
Shouldn’t it be autogen.sh instead of just autogen?