Raspivid v Gst-rpicamsrc (Updated)

Introduction

User bocorps pointed me in the direction of gst-rpicamsrc.  This is “… a GStreamer wrapper around the raspivid/raspistill functionality of the RaspberryPi, providing a GStreamer source element capturing from the Rpi camera.”

What this means is that instead of piping the output of raspivid into gstreamer, gstreamer has a source element to read the camera directly.  This is similar to using the video4linux (v4l) source element, but negates the need for a v4l driver.

My hope was that by integrating the camera functionality into a gstreamer source element the latency would be reduced.  Unfortunately, I actually saw an 18% increase in latency.

Installation

Before I could use the gst-rpicamsrc element, I needed to download the source and build it.  As I was working with a minimal install of Raspbian Jessie, I needed to install the git package before I could do anything else

sudo apt-get install git

With git installed I could download the latest sources for gst-rpicamsrc.

git clone https://github.com/thaytan/gst-rpicamsrc.git

With that done a look in the REQUIREMENTS file indicated what other packages were needed in order to accomplish the build.

sudo apt-get install autoconf automake libtool libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev libraspberrypi-dev

Finally, I was able to complete the build and install.

./autogen --prefix=/usr --libdir=/usr/lib/arm-linux-gnueabihf/
make
sudo make install

The command `gst-inspect-1.0 rpicamsrc’ produces a list of the available parameters.

Factory Details:
  Rank                     none (0)
  Long-name                Raspberry Pi Camera Source
  Klass                    Source/Video
  Description              Raspberry Pi camera module source
  Author                   Jan Schmidt <jan@centricular.com>

Plugin Details:
  Name                     rpicamsrc
  Description              Raspberry Pi Camera Source
  Filename                 /usr/lib/arm-linux-gnueabihf/gstreamer-1.0/libgstrpicamsrc.so
  Version                  1.0.0
  License                  LGPL
  Source module            gstrpicamsrc
  Binary package           GStreamer
  Origin URL               http://gstreamer.net/

GObject
 +----GInitiallyUnowned
       +----GstObject
             +----GstElement
                   +----GstBaseSrc
                         +----GstPushSrc
                               +----GstRpiCamSrc

Pad Templates:
  SRC template: 'src'
    Availability: Always
    Capabilities:
      video/x-h264
                  width: [ 1, 2147483647 ]
                 height: [ 1, 2147483647 ]
              framerate: [ 0/1, 2147483647/1 ]
          stream-format: byte-stream
              alignment: au
                profile: { baseline, main, high }


Element Flags:
  no flags set

Element Implementation:
  Has change_state() function: gst_base_src_change_state

Element has no clocking capabilities.
Element has no URI handling capabilities.

Pads:
  SRC: 'src'
    Implementation:
      Has getrangefunc(): gst_base_src_getrange
      Has custom eventfunc(): gst_base_src_event
      Has custom queryfunc(): gst_base_src_query
      Has custom iterintlinkfunc(): gst_pad_iterate_internal_links_default
    Pad Template: 'src'

Element Properties:
  name                : The name of the object
                        flags: readable, writable
                        String. Default: "rpicamsrc0"
  parent              : The parent of the object
                        flags: readable, writable
                        Object of type "GstObject"
  blocksize           : Size in bytes to read per buffer (-1 = default)
                        flags: readable, writable
                        Unsigned Integer. Range: 0 - 4294967295 Default: 4096 
  num-buffers         : Number of buffers to output before sending EOS (-1 = unlimited)
                        flags: readable, writable
                        Integer. Range: -1 - 2147483647 Default: -1 
  typefind            : Run typefind before negotiating
                        flags: readable, writable
                        Boolean. Default: false
  do-timestamp        : Apply current stream time to buffers
                        flags: readable, writable
                        Boolean. Default: true
  bitrate             : Bitrate for encoding
                        flags: readable, writable
                        Integer. Range: 1 - 25000000 Default: 17000000 
  preview             : Display preview window overlay
                        flags: readable, writable
                        Boolean. Default: true
  preview-encoded     : Display encoder output in the preview
                        flags: readable, writable
                        Boolean. Default: true
  preview-opacity     : Opacity to use for the preview window
                        flags: readable, writable
                        Integer. Range: 0 - 255 Default: 255 
  fullscreen          : Display preview window full screen
                        flags: readable, writable
                        Boolean. Default: true
  sharpness           : Image capture sharpness
                        flags: readable, writable
                        Integer. Range: -100 - 100 Default: 0 
  contrast            : Image capture contrast
                        flags: readable, writable
                        Integer. Range: -100 - 100 Default: 0 
  brightness          : Image capture brightness
                        flags: readable, writable
                        Integer. Range: 0 - 100 Default: 50 
  saturation          : Image capture saturation
                        flags: readable, writable
                        Integer. Range: -100 - 100 Default: 0 
  iso                 : ISO value to use (0 = Auto)
                        flags: readable, writable
                        Integer. Range: 0 - 3200 Default: 0 
  video-stabilisation : Enable or disable video stabilisation
                        flags: readable, writable
                        Boolean. Default: false
  exposure-compensation: Exposure Value compensation
                        flags: readable, writable
                        Integer. Range: -10 - 10 Default: 0 
  exposure-mode       : Camera exposure mode to use
                        flags: readable, writable
                        Enum "GstRpiCamSrcExposureMode" Default: 1, "auto"
                           (0): off              - GST_RPI_CAM_SRC_EXPOSURE_MODE_OFF
                           (1): auto             - GST_RPI_CAM_SRC_EXPOSURE_MODE_AUTO
                           (2): night            - GST_RPI_CAM_SRC_EXPOSURE_MODE_NIGHT
                           (3): nightpreview     - GST_RPI_CAM_SRC_EXPOSURE_MODE_NIGHTPREVIEW
                           (4): backlight        - GST_RPI_CAM_SRC_EXPOSURE_MODE_BACKLIGHT
                           (5): spotlight        - GST_RPI_CAM_SRC_EXPOSURE_MODE_SPOTLIGHT
                           (6): sports           - GST_RPI_CAM_SRC_EXPOSURE_MODE_SPORTS
                           (7): snow             - GST_RPI_CAM_SRC_EXPOSURE_MODE_SNOW
                           (8): beach            - GST_RPI_CAM_SRC_EXPOSURE_MODE_BEACH
                           (9): verylong         - GST_RPI_CAM_SRC_EXPOSURE_MODE_VERYLONG
                           (10): fixedfps         - GST_RPI_CAM_SRC_EXPOSURE_MODE_FIXEDFPS
                           (11): antishake        - GST_RPI_CAM_SRC_EXPOSURE_MODE_ANTISHAKE
                           (12): fireworks        - GST_RPI_CAM_SRC_EXPOSURE_MODE_FIREWORKS
  metering-mode       : Camera exposure metering mode to use
                        flags: readable, writable
                        Enum "GstRpiCamSrcExposureMeteringMode" Default: 0, "average"
                           (0): average          - GST_RPI_CAM_SRC_EXPOSURE_METERING_MODE_AVERAGE
                           (1): spot             - GST_RPI_CAM_SRC_EXPOSURE_METERING_MODE_SPOT
                           (2): backlist         - GST_RPI_CAM_SRC_EXPOSURE_METERING_MODE_BACKLIST
                           (3): matrix           - GST_RPI_CAM_SRC_EXPOSURE_METERING_MODE_MATRIX
  awb-mode            : White Balance mode
                        flags: readable, writable
                        Enum "GstRpiCamSrcAWBMode" Default: 1, "auto"
                           (0): off              - GST_RPI_CAM_SRC_AWB_MODE_OFF
                           (1): auto             - GST_RPI_CAM_SRC_AWB_MODE_AUTO
                           (2): sunlight         - GST_RPI_CAM_SRC_AWB_MODE_SUNLIGHT
                           (3): cloudy           - GST_RPI_CAM_SRC_AWB_MODE_CLOUDY
                           (4): shade            - GST_RPI_CAM_SRC_AWB_MODE_SHADE
                           (5): tungsten         - GST_RPI_CAM_SRC_AWB_MODE_TUNGSTEN
                           (6): fluorescent      - GST_RPI_CAM_SRC_AWB_MODE_FLUORESCENT
                           (7): incandescent     - GST_RPI_CAM_SRC_AWB_MODE_INCANDESCENT
                           (8): flash            - GST_RPI_CAM_SRC_AWB_MODE_FLASH
                           (9): horizon          - GST_RPI_CAM_SRC_AWB_MODE_HORIZON
  image-effect        : Visual FX to apply to the image
                        flags: readable, writable
                        Enum "GstRpiCamSrcImageEffect" Default: 0, "none"
                           (0): none             - GST_RPI_CAM_SRC_IMAGEFX_NONE
                           (1): negative         - GST_RPI_CAM_SRC_IMAGEFX_NEGATIVE
                           (2): solarize         - GST_RPI_CAM_SRC_IMAGEFX_SOLARIZE
                           (3): posterize        - GST_RPI_CAM_SRC_IMAGEFX_POSTERIZE
                           (4): whiteboard       - GST_RPI_CAM_SRC_IMAGEFX_WHITEBOARD
                           (5): blackboard       - GST_RPI_CAM_SRC_IMAGEFX_BLACKBOARD
                           (6): sketch           - GST_RPI_CAM_SRC_IMAGEFX_SKETCH
                           (7): denoise          - GST_RPI_CAM_SRC_IMAGEFX_DENOISE
                           (8): emboss           - GST_RPI_CAM_SRC_IMAGEFX_EMBOSS
                           (9): oilpaint         - GST_RPI_CAM_SRC_IMAGEFX_OILPAINT
                           (10): hatch            - GST_RPI_CAM_SRC_IMAGEFX_HATCH
                           (11): gpen             - GST_RPI_CAM_SRC_IMAGEFX_GPEN
                           (12): pastel           - GST_RPI_CAM_SRC_IMAGEFX_PASTEL
                           (13): watercolour      - GST_RPI_CAM_SRC_IMAGEFX_WATERCOLOUR
                           (14): film             - GST_RPI_CAM_SRC_IMAGEFX_FILM
                           (15): blur             - GST_RPI_CAM_SRC_IMAGEFX_BLUR
                           (16): saturation       - GST_RPI_CAM_SRC_IMAGEFX_SATURATION
                           (17): colourswap       - GST_RPI_CAM_SRC_IMAGEFX_COLOURSWAP
                           (18): washedout        - GST_RPI_CAM_SRC_IMAGEFX_WASHEDOUT
                           (19): posterise        - GST_RPI_CAM_SRC_IMAGEFX_POSTERISE
                           (20): colourpoint      - GST_RPI_CAM_SRC_IMAGEFX_COLOURPOINT
                           (21): colourbalance    - GST_RPI_CAM_SRC_IMAGEFX_COLOURBALANCE
                           (22): cartoon          - GST_RPI_CAM_SRC_IMAGEFX_CARTOON
  rotation            : Rotate captured image (0, 90, 180, 270 degrees)
                        flags: readable, writable
                        Integer. Range: 0 - 270 Default: 0 
  hflip               : Flip capture horizontally
                        flags: readable, writable
                        Boolean. Default: false
  vflip               : Flip capture vertically
                        flags: readable, writable
                        Boolean. Default: false
  roi-x               : Normalised region-of-interest X coord
                        flags: readable, writable
                        Float. Range:               0 -               1 Default:               0 
  roi-y               : Normalised region-of-interest Y coord
                        flags: readable, writable
                        Float. Range:               0 -               1 Default:               0 
  roi-w               : Normalised region-of-interest W coord
                        flags: readable, writable
                        Float. Range:               0 -               1 Default:               1 
  roi-h               : Normalised region-of-interest H coord
                        flags: readable, writable
                        Float. Range:               0 -               1 Default:               1

Usage

Because of the way gstreamer works, the parameters for the feed needed to be split and re-arranged in the streamer pipeline. Previously all the parameters are specified as part of raspivid.

/opt/vc/bin/raspivid -t $DURATION -w $WIDTH -h $HEIGHT -fps $FRAMERATE -b $BITRATE -n -pf high -o - | gst-launch-1.0 -v fdsrc ! 

GStreamer parameters like width, height and frame rate are configured through capabilities (caps) negotiation with the next element. Other parameters like the bit rate and preview screen are controlled as part of the source element.

gst-launch-1.0 rpicamsrc bitrate=$BITRATE preview=0 ! video/x-h264,width=$WIDTH,height=$HEIGHT,framerate=$FRAMERATE/1 !

The new stream script is

#!/bin/bash

source remote.conf

if [ "$1" != "" ]
then
  export FRAMERATE=$1
fi

NOW=`date +%Y%m%d%H%M%S`
FILENAME=$NOW-Tx.h264

gst-launch-1.0 rpicamsrc bitrate=$BITRATE preview=0 ! video/x-h264,width=$WIDTH,height=$HEIGHT,framereate=$FRAMERATE/1,profile=high ! h264parse ! rtph264pay config-interval=1 pt=96 ! udpsink host=$RX_IP port=$UDPPORT

Tests

This test was a straight comparison between the old and new scripts using the same settings and the same testing methodology as previously.  The resolution was 1280 x 720 pixels with a 6Mbps bitrate.

Update : Since the original article was published use Jan Schmitt spotted that I had misspelled “framerate” as “framereate” in the gst-rpicamsrc script.  He also suggested I should try using the baseline profile and a queue element to decouple the video capture from the UDP transmission.  With this in mind I have re-run the tests.

Results

Almost immediately I had the feeling that gst-rpicamsrc has slower.  Analysis of the video showed I was correct.  The latency using gst-rpicamsrc was 18% higher than using raspivid.

Update:

Running the original erroneous script with debugging on showed that the capture was running at 30fps instead of the intended 48 fps. Here are the new results averaged from 10 cycles.

Script gst-rpicamsrc @ 48 fps
raspivid @ 48 fps
Profile No queue With queue No queue With queue
baseline 184.2 175.4 153.9 156.9
high 186.2 185 154.1 159.7

gst-rpicamsrc @ 30 fps, high profile, no queue = 198.2 ms

Analysis

The first thing to note is that the raspivid latency (no queue, high profile) has risen from the 126ms found in the last tests to 154ms.  The only difference was that I cloned the Sandisk microSDHC card onto a Transcend 8GB.  I’ll set up some more tests to compare the cards.  As these tests were run from the same card and from the same boot, they are still valid for comparison.

It is immediately obvious that the gst-rpicamsrc latency is about 20% higher than the raspivid script, so the conclusion from the first publish of this article still stands.

What can be added is that using the baseline profile, does reduce the latency a little: 1 to 3ms in most cases.

Adding a queue element does provide a benefit for the gst-rpicamsrc script, especially with the baseline profile where a 9ms reduction in latency was observed.  For the raspivid script adding a queue element actually increased the latency by 3 to 4ms.  I suspect this is because the video stream is already decoupled from gstreamer by being piped in from an external process.

Conclusion

Using gst-rpicamsrc provides no benefit for reducing latency over raspivid.  That is not to say gst-rpicamsrc provides no other benefits.  For any use other than FPV, I would definitely use gst-rpicamsrc instead of having to pipe the video in through stdin.  It provides plenty of options for setting up the video stream as the command `gst-inspect-1.0 rpicamsrc’ above showed.

The problem here is that I am targeting this development for FPV use where low latency is the driving factor. At the moment my lowest latency for a adequate quality HD stream is 125ms and I really need to get this under 100ms to compete with current analog standard definition systems.  Whether it is possible to shave of another 25ms remains to be seen.

Update: Following the additional tests I would add that it is better to use the baseline profile over the high profile.

Advertisements

23 responses to “Raspivid v Gst-rpicamsrc (Updated)

  1. ..you found latency using gst-rpicamsrc was 18% higher, but it may have other useful features, please post output of “gst-inspect-1.0 rpicamsrc”.

    (This comment hit the wrong thread at first, belongs here IMO.)

    • Thanks for your comment. I did think my original conclusion as somewhat terse and as you say rpicamsrc has plenty of may have some useful features even if it can’t improve the latency. Therefore, I’ve extended the conclusion to make it more balanced. If you know of any way to improve the speed of the gst-rpicamsrc element please let me know.

  2. ..running rpicamsrc, did you try to disable the preview?

    ..disabling the preview with the good old pipe way, got my latency lower, around 150ms, AFAIR.

    • I did have the preview turned off. I even connected it to a monitor to check. I did find that you can use preview=0 or preview=false. My best average time for a useable 1280×720 stream using raspivid is 125ms.

  3. You misspelled ‘framerate’ in your capsfilter – “framereate”. You may not be getting the framerate you expect, which could affect your latency measure. You might also benefit from having a ‘queue’ element in your pipeline before udpsink to decouple the capture and the udp transmission into 2 separate threads.

    You’ll get better latency using baseline profile instead of high too. B-frames will hurt you.

    • Hi Jan.
      Thanks for the spot. As the post wasn’t created on the Pi, I can’t remember if pasted in my actual script or retyped it. Either way I will check what the script says and rerun the tests if necessary. I’ll also do some back to back testing of base versus high profile.

    • Hi Jan.
      Thanks for the spot. As the post wasn’t created on the Pi, I can’t remember if pasted in my actual script or retyped it. Either way I will check what the script says and rerun the tests if necessary. I’ll also do some back to back testing of Base versus high profile and with/without a queue element.

  4. Hello,
    I am using Raspberry Pi to stream video. I type the comment : gst-launch-1.0 rpicamsrc bitrate=90000 preview=0 ! video/x-h264,width=1280,height=720,profile=high ! h264parse ! rtph264pay config-interval=1 pt=96 ! udpsink host=192.168.178.24(IP address of Pi) port=5000. But i do not know how to watch the stream. Can you show me the command for receiver? Thanks a lot!
    Ragards,
    SonNam

      • I use the script 4 to receive the stream. Do i need to type the first threes line? Because when i type it ubuntu it said : source ./video.conf no such file or directory

      • You need to create the video.conf and network.conf files also given on the current scripts page but modified to match your setup. Put them in the same directory as tsplay.

      • HI,
        I did all but the ./video.conf is still getting error. I type:
        export TX_IP=192.168.178.24
        export RX_IP=$(ip -o 192.168.32.132 | sed -n ‘s/.*inet \([0-9.]*\)\/.. .*/\1/p’)
        export TCPPORT=5001
        export UDPPORT=5000

        export USER=pi

      • In the line beginning export RX_IP you DON’T replace the text “addr” with an IP address. Leave it as “export RX_IP=$(ip -o addr show wlan0 | sed -n ‘s/.*inet \([0-9.]*\)\/.. .*/\1/p’)”

      • I leave it as eth0 because i dont have wlan0. But it still get error. Do i have to do more than these steps ? 😀

  5. Hi Gary

    Very interesting tests you did here! Maybe you want to take a look into a small project I’ve started: https://befinitiv.wordpress.com/2015/01/25/true-unidirectional-wifi-broadcasting-of-video-data-for-fpv/

    The basic idea is: Broadcast video data with 802.11 packets directly into the air, without association to an access point (thus no risk of loosing it). A weaker link quality just degrades the image quality (as with analog links) but does not terminate it.

    It would be nice if you could share your thoughts on this, maybe even participate on that project 🙂

    Best regards,
    befinitiv.

    • Hi befinitiv,

      I’ve had a quick read of your post and it looks interesting. A have a couple of initial thoughts.
      1. How does it handle a 720p HD stream. Finding a low latency HD solution is my main aim. For an SD stream, which would include 640×480, current analogue systems still offer the lowest latency solution.
      2. Have you though of using MJPEG for the stream. As each frame would be complete you wouldn’t need to send them three times.

      I’ll try to get it compiled and tested in the next couple of weeks.

  6. I compared the latency of rpicamsrc to piping it into gstreamer, and oddly enough found rpicamsrc slightly faster. With a pipe I measured 160-180 ms of delay, while rpicamsrc managed 120-140. This might be because I was using lower resolution video (400×240, 30FPS) or possibly rpicamsrc has improved in the last year.

    Here is the pipeline I used:

    gst-launch-1.0 rpicamsrc keyframe-interval=10 preview=0 ! video/x-h264,width=400,height=240,framerate=30/1,profile=baseline ! h264parse ! queue ! rtph264pay config-interval=1 pt=96 ! udpsink host=192.168.1.23 port=9000

    • I’m quite surprised by the latency you are getting at 400×240 resolution. I was able to get 90ms at 640×480. What device are you using on the receiving end and what bitrate are you using?

      • Both ends are a Raspberry Pi 2B running Jessie, connected via a pair of TEW-820AP.

        I’m currently using the default bitrate setting, it didn’t seem to have much effect when I played around with it. The default setting is something like 17mbit/s but it’s not using anywhere near that.

        Reception end uses this pipeline:

        udpsrc port=9000 caps=application/x-rtp,media=(string)video,clock-rate=(int)90000,encoding-name=(string)H264 ! rtph264depay ! avdec_h264 ! queue ! eglglessink

        Using the hardware decoder (omxh264dec) is about the same as avdec_h264, maybe 5ms faster. Adding and removing the queue didn’t seem to make a reliably measurable difference either.

        I’m measuring latency by putting a stop watch in the camera’s field of view, positioning my monitor next to the stop watch, and taking a photo of the stop watch and the monitor, and then subtracting to get the lag. Is there a better way to do this?

  7. ./autogen –prefix=/usr –libdir=/usr/lib/arm-linux-gnueabihf/
    Shouldn’t it be autogen.sh instead of just autogen?

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s