A Pi Straight Flush – Comparing the Pi 2

Extract

The Pi 2 Model B was release on the 2nd February 2015 promising a 1.5 to 6 times performance boost over the earlier generation one hardware.  If that boost occurred across the entire pipeline then even a minimal 1.5 improvement would reduce the latency from 120 to 80 milliseconds (a 33% reduction) and make a very useable FPV system.  Not enough for racing, but enough for most everything else.  Anyway, a 250mm racing quad would be too small to carry a Raspberry Pi.

In reality the processor is only one small part of the video streaming pipeline and while the Pi 2 does reduce the latency, the improvement is not nearly as much as you might think.  In fact most of the improvement over previous tests has come from a change in test procedure.  The final conclusion though is that the Pi 2 does at least allow us to break the 100ms barrier.  If I can just get a 5 GHz wifi system running on the Pi 2, then the next step is to actually get it airborne.

In addition to the Pi 2, I also have a Pi 1 A+ that had not been tested.  The A+ is smaller and lighter than the previous boards which is an advantage if you’re trying to fit it in a tight fuselage.  As I was using a new build of Raspbian, I thought I would run the test on all versions of the Pi: the A, B, B+, A+ and Pi 2B; a straight flush of Pi.

PiStraightFlush

Setup

Nothing new here.  I used the build of Raspbian from the previous test.  As I haven’t managed to get 5 GHz WiFi running on the Pi 2 yet, the test was done using the Edimax EW-7612UAn V2 2.4GHz USB WiFi stick as this was compatible with all versions of the Pi.  Following the previous tests where the Dual Band AC600 Wifi Adapter had similar performance to the on board ethernet port, I also tested the B and 2B over LAN.  The idea being to see what the Pi 2B could achieve with a good WiFi adapter.

Procedure

These tests were a straight comparison between the five Pi devices using the same settings and the same testing methodology as previously.  The resolution was 1280 x 720 pixels with a 6Mbps bit rate.

Results

My first impression of the Pi 2B was that it was faster than the series one models, however when I started analysing the results, I could see no improvement after seven iterations.  So I went back and calculated the latency at each step.  This revealed a general trend for the latency to increase with each step.

With the Pi sampling at 48 fps, that’s 21ms for each frame.  The screen is refreshing at 60 fps or once every 17ms.  Depending on when the image appears on the screen bright enough for the camera to detect there could be a significant increase in latency towards the end of the iteration loop.  In practice these things tend to average out and the general trend seems to be a 20ms increase between iterations 1 and 7.

I started using the multiple iteration technique right back at the start of this testing when we were looking at a latency in the 200 to 300ms range. Back then there was also a lot of variation at each step and so this increasing trend was not immediately apparent.  Now that the latency is around 100ms and more consistent it has become so.

Straight Flush Iterations

Because of this, I have changed my result reporting to only use the average of the first iterations and I always collect at least 10.  From the above graph you can see that over WiFi the Pi 2B has the lowest latency at 103.3 ms, but that’s only  2.3 ms faster than the Pi 1B.

Using the ethernet port the Pi 2B does better with an average latency of 97.2 ms whereas the Pi 1B could only manage 107.4 ms.

Straight Flush Iterations Eth

Here is a summary of the final results.

Straight Flush Latency Summary

From fastest to slowest the final results are Pi2B, Pi1B, Pi1A+, Pi1A, Pi1B+.

Discussion

So where is the 1.5 to 6 times performance improvement, the 33% reduction?  The best result achieved is a 10% reduction.  The answer is that the processor represents only a part of the complete video capture and transmission pipeline and each stage involves converting the stream into different forms through multiple processors.  Remember each device (camera, bus controller, Pi, USB controller, USB hub, WiFi Adapters, PC CPU, Video Card and display) has their own processor and most are not as powerful as the Pi.  Unfortunately, there is no easy way to measure the latency at each stage to determine where the main bottlenecks are.

Video Pipeline

It must also be remembered that gstreamer is a complete pipeline in itself, with multiple conversions between the raw video and the packetized stream sent to the WiFi adapter.

It is interesting to note that the A models were faster than the Pi1B+ even though they have half its memory.  The A models only have a single USB interface.  The B models don’t have multiple USB interfaces, they have the same single USB interface and that feeds a built-in two or  four port hub.  This adds an extra stage to the transmission pipeline making the Pi1B+ slower than the A models even with twice the memory.   I assume the two port hub in the Pi1B is faster than the four port hub in the Pi1B+ which combined with the extra memory makes it faster than the A models.

Conclusion

If the Pi 2B can be made to work with one of the dual band WiFi adapters and achieve this sub 100ms average latency then it has promise to finally deliver a low cost and useable high definition FPV system.

At the moment Raspbian is still optimised for the arm6 architecture of the first generation hardware.  With an operating system optimised for the arm7 in the Pi 2B further latency improvements should be possible.

There are no immediate plans for a Pi 2A.  If this was released with the 512MB of memory of the current B models, a single USB interface and a small form factor, it could be the idea platform for HD FPV.

Perhaps the way to improve the latency further is to switch from a raspivid/gstreamer RTSP (Real Time Streaming Protocol) solution to some other protocol.  Perhaps MpegStreamer holds the answer.  Only more testing will tell.

Advertisements

Comparison of 2.4 GHz WiFi, 5 GHz WiFi and 100 MBit LAN

Extract

Up to now I have been using an Edimax EW-7812UAn V2 USB WiFi Dongle.  This has been very reliable and reasonably simple to install and configure.  The only problem is that it operates on the 2.4 GHz frequency band, which means it will interfere with the majority of modern radio control systems.  Up to now I have been using a 35 MHz RC system which was fine as long as I was flying on my own.  However, I’d like to fly with others who will be using 2.4 GHz systems.

There are now a number of Dual Band Wifi Dongles that operate in the 5 GHz band as well as at 2.4 GHz. This provides a solution so long as they work with the Raspberry Pi.  A couple of candidates were obtained, configured and tested on my Model B Raspberry Pi.  For an extra comparison I also ran a test over the wired LAN interface.

The results were interesting in that there was no clear winner in the speed stakes, there was however a clear loser.  What is clear is that useable 5 Ghz WiFi is possible but for now it’s not plug and play.

Setup

For these tests I used the following network devices.

  1. Raspberry Pi Model B Ethernet (100 Mbit/s)
  2. Tenga W522U Dual Band (2.4 GHz & 5 GHz)
  3. Edimax EW-7612UAn V2 (2.4 GHz)
  4. Edimax AC600 Dual Band (2.4 GHz & 5 GHz)

I setup my Model B, with the latest release of Raspbian (2015-01-31) and the current scripts.

5 GHz WiFi Prerequisites

To get 5GHz WiFi working you need to install the Central Regulatory Domain Agent (crda).  Basically, the 5 GHz band is more tightly regulated worldwide and you need to specify your regulatory domain so that hostapd knows which channels are available.

To install the necessary packages use:

sudo apt-get install hostapd crda iw

Then edit /etc/default/crda and set the REGDOMAIN variable to match your region.  Examples are US, JP, EU and in my case UK.

crda

Devices

The amount of work needed to setup networking varied by device.  I’m not going to give a detailed description of configuring each device.  I’ll do that in separate posts.

1. LAN

This simply required plugging the Pi and the Laptop into a switch and running ipconfig on the receiving device to get its IP address.  This was then used in remote.conf to direct the UDP stream.

2. Tenga W522U

The W522U is nl80211 compatible.  This means it works with a standard install of hostapd.  No special configuration is needed.

3. Edimax EW-7612UAn V2 (2.4 GHz)

The EW-7612UAn V2 uses the RTL8191SU chipset.  The kernel supports this chipset for client networking, but not for AP mode.  However, Realtek provide a custom version of hostapd that needs to be downloaded, compiled and installed which does work.

4. Edimax AC600 Dual Band (2.4 GHz & 5 GHz)

The AC600 is not supported by the latest kernel.  All is not lost, it just takes a little more work.  You need to download the kernel source code and the code for a Realtek 8812U driver.  Then you need to install the latest version of GCC from the Jessie repository.  This lets you build the kernel module.  Finally you need the same hostapd build as for the Edimax EW-7612UAn V2 device.

Hostapd Configuration

There are four combinations of hostapd.conf depending on the driver and band.  I’ve set them out in the table below.

Hostapd Version 2.4 GHz 5 GHz
nl80211
interface=wlan0
driver=nl80211
ssid=Pi_AP

hw_mode=g
channel=13
macaddr_acl=0 
auth_algs=1 
ignore_broadcast_ssid=0 
wpa=2 
wpa_passphrase=Raspberry 
wpa_key_mgmt=WPA-PSK 
wpa_pairwise=TKIP 
rsn_pairwise=CCMP
interface=wlan0
driver=nl80211
ssid=Pi_AP

hw_mode=a
channel=44
ht_capab=[HT40+]
ieee80211n=1
macaddr_acl=0 
auth_algs=1 
ignore_broadcast_ssid=0 
wpa=2 
wpa_passphrase=Raspberry 
wpa_key_mgmt=WPA-PSK 
wpa_pairwise=TKIP 
rsn_pairwise=CCMP
Realtek
interface=wlan0 
driver=rtl871xdrv
ctrl_interface=/var/run/hostapd

ssid=Pi_AP
channel=13

beacon_int=100
hw_mode=g
ieee80211n=1
wme_enabled=1
wpa=2
wpa_passphrase=Raspberry
wpa_key_mgmt=WPA-PSK 
wpa_pairwise=CCMP 
max_num_sta=8 
wpa_group_rekey=86400
interface=wlan0
driver=rtl871xdrv 
ctrl_interface=/var/run/hostapd
ssid=Pi_AP
channel=44
beacon_int=100 
hw_mode=a 
ieee80211n=1 
wme_enabled=1 
ht_capab=[SHORT-GI-20][SHORT-GI-40][HT40+]
wpa=2
wpa_passphrase=Raspberry

wpa_key_mgmt=WPA-PSK 
wpa_pairwise=CCMP 
max_num_sta=8 
wpa_group_rekey=86400

Procedure

These tests were a straight comparison between the four devices using the same settings and the same testing methodology as previously.  The resolution was 1280 x 720 pixels with a 6Mbps bit rate.

Results

The table below shows the resulting latency by device.

Device Latency
Edimax EW-7612UAn V2 – 2.4 Ghz 132 ms
Tenga W522U – 2.4 GHz & 5 GHz > 10 s
Edimax AC600 – 2.4 GHz 122 ms
Edimax AC600 – 5 GHz 126 ms
LAN 124 ms

Here they are as a graph.

WiFiAdapters

Analysis

The big surprise is how bad the Tenga W522U adapter was.  The latency was not just two or three times higher, it was two order of magnitudes higher.  There are a number of threads on various forums where other people have found similar issues, although not to this extent, but then they are probably not swamping the feed with a continuous video stream.  Whether this is just a compatibility issue with the Pi remains to be seen.

The Edimax AC600 was 10 ms faster than the previously used Edimax 7612UAn V2 at 2.4 GHz.  At 5GHz it was still faster, but only by 6ms.  Interestingly, the LAN connection was not any faster than the AC600.  On the Pi the LAN connection is 100 MBit/s as is the USB bus, so although the AC600 is theoretically capable of 433 Mbit/s it is never going to manage more than 100.

Conclusion

What we can take away from this test is that there is a limit to how much the latency can be reduced by changing the WiFi adapter.  It is a shame that to get a 5 GHz WiFi link requires a lot of compiling and configuration.  Hopefully this will improve with time.  For now we have another small reduction in the video latency to add to the current optimisations.

Adendum

Just as these tests were being done, the new Raspberry Pi 2 was released promising a x1.5 to x6 speed increase.  As I had used the latest release of Raspbian, I swapped the microSD card from from the Model B to the Model 2B.  What I found was that the Edimax AC600 was no longer detected and I couldn’t find a way to compile the module so that it would.  So that’s my next task.  I may run a quick comparison of the B versus the 2B with the Edimax 7612UAn V2 to see how much improvement the new processor has made.

Raspivid v Gst-rpicamsrc (Updated)

Introduction

User bocorps pointed me in the direction of gst-rpicamsrc.  This is “… a GStreamer wrapper around the raspivid/raspistill functionality of the RaspberryPi, providing a GStreamer source element capturing from the Rpi camera.”

What this means is that instead of piping the output of raspivid into gstreamer, gstreamer has a source element to read the camera directly.  This is similar to using the video4linux (v4l) source element, but negates the need for a v4l driver.

My hope was that by integrating the camera functionality into a gstreamer source element the latency would be reduced.  Unfortunately, I actually saw an 18% increase in latency.

Installation

Before I could use the gst-rpicamsrc element, I needed to download the source and build it.  As I was working with a minimal install of Raspbian Jessie, I needed to install the git package before I could do anything else

sudo apt-get install git

With git installed I could download the latest sources for gst-rpicamsrc.

git clone https://github.com/thaytan/gst-rpicamsrc.git

With that done a look in the REQUIREMENTS file indicated what other packages were needed in order to accomplish the build.

sudo apt-get install autoconf automake libtool libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev libraspberrypi-dev

Finally, I was able to complete the build and install.

./autogen --prefix=/usr --libdir=/usr/lib/arm-linux-gnueabihf/
make
sudo make install

The command `gst-inspect-1.0 rpicamsrc’ produces a list of the available parameters.

Factory Details:
  Rank                     none (0)
  Long-name                Raspberry Pi Camera Source
  Klass                    Source/Video
  Description              Raspberry Pi camera module source
  Author                   Jan Schmidt <jan@centricular.com>

Plugin Details:
  Name                     rpicamsrc
  Description              Raspberry Pi Camera Source
  Filename                 /usr/lib/arm-linux-gnueabihf/gstreamer-1.0/libgstrpicamsrc.so
  Version                  1.0.0
  License                  LGPL
  Source module            gstrpicamsrc
  Binary package           GStreamer
  Origin URL               http://gstreamer.net/

GObject
 +----GInitiallyUnowned
       +----GstObject
             +----GstElement
                   +----GstBaseSrc
                         +----GstPushSrc
                               +----GstRpiCamSrc

Pad Templates:
  SRC template: 'src'
    Availability: Always
    Capabilities:
      video/x-h264
                  width: [ 1, 2147483647 ]
                 height: [ 1, 2147483647 ]
              framerate: [ 0/1, 2147483647/1 ]
          stream-format: byte-stream
              alignment: au
                profile: { baseline, main, high }


Element Flags:
  no flags set

Element Implementation:
  Has change_state() function: gst_base_src_change_state

Element has no clocking capabilities.
Element has no URI handling capabilities.

Pads:
  SRC: 'src'
    Implementation:
      Has getrangefunc(): gst_base_src_getrange
      Has custom eventfunc(): gst_base_src_event
      Has custom queryfunc(): gst_base_src_query
      Has custom iterintlinkfunc(): gst_pad_iterate_internal_links_default
    Pad Template: 'src'

Element Properties:
  name                : The name of the object
                        flags: readable, writable
                        String. Default: "rpicamsrc0"
  parent              : The parent of the object
                        flags: readable, writable
                        Object of type "GstObject"
  blocksize           : Size in bytes to read per buffer (-1 = default)
                        flags: readable, writable
                        Unsigned Integer. Range: 0 - 4294967295 Default: 4096 
  num-buffers         : Number of buffers to output before sending EOS (-1 = unlimited)
                        flags: readable, writable
                        Integer. Range: -1 - 2147483647 Default: -1 
  typefind            : Run typefind before negotiating
                        flags: readable, writable
                        Boolean. Default: false
  do-timestamp        : Apply current stream time to buffers
                        flags: readable, writable
                        Boolean. Default: true
  bitrate             : Bitrate for encoding
                        flags: readable, writable
                        Integer. Range: 1 - 25000000 Default: 17000000 
  preview             : Display preview window overlay
                        flags: readable, writable
                        Boolean. Default: true
  preview-encoded     : Display encoder output in the preview
                        flags: readable, writable
                        Boolean. Default: true
  preview-opacity     : Opacity to use for the preview window
                        flags: readable, writable
                        Integer. Range: 0 - 255 Default: 255 
  fullscreen          : Display preview window full screen
                        flags: readable, writable
                        Boolean. Default: true
  sharpness           : Image capture sharpness
                        flags: readable, writable
                        Integer. Range: -100 - 100 Default: 0 
  contrast            : Image capture contrast
                        flags: readable, writable
                        Integer. Range: -100 - 100 Default: 0 
  brightness          : Image capture brightness
                        flags: readable, writable
                        Integer. Range: 0 - 100 Default: 50 
  saturation          : Image capture saturation
                        flags: readable, writable
                        Integer. Range: -100 - 100 Default: 0 
  iso                 : ISO value to use (0 = Auto)
                        flags: readable, writable
                        Integer. Range: 0 - 3200 Default: 0 
  video-stabilisation : Enable or disable video stabilisation
                        flags: readable, writable
                        Boolean. Default: false
  exposure-compensation: Exposure Value compensation
                        flags: readable, writable
                        Integer. Range: -10 - 10 Default: 0 
  exposure-mode       : Camera exposure mode to use
                        flags: readable, writable
                        Enum "GstRpiCamSrcExposureMode" Default: 1, "auto"
                           (0): off              - GST_RPI_CAM_SRC_EXPOSURE_MODE_OFF
                           (1): auto             - GST_RPI_CAM_SRC_EXPOSURE_MODE_AUTO
                           (2): night            - GST_RPI_CAM_SRC_EXPOSURE_MODE_NIGHT
                           (3): nightpreview     - GST_RPI_CAM_SRC_EXPOSURE_MODE_NIGHTPREVIEW
                           (4): backlight        - GST_RPI_CAM_SRC_EXPOSURE_MODE_BACKLIGHT
                           (5): spotlight        - GST_RPI_CAM_SRC_EXPOSURE_MODE_SPOTLIGHT
                           (6): sports           - GST_RPI_CAM_SRC_EXPOSURE_MODE_SPORTS
                           (7): snow             - GST_RPI_CAM_SRC_EXPOSURE_MODE_SNOW
                           (8): beach            - GST_RPI_CAM_SRC_EXPOSURE_MODE_BEACH
                           (9): verylong         - GST_RPI_CAM_SRC_EXPOSURE_MODE_VERYLONG
                           (10): fixedfps         - GST_RPI_CAM_SRC_EXPOSURE_MODE_FIXEDFPS
                           (11): antishake        - GST_RPI_CAM_SRC_EXPOSURE_MODE_ANTISHAKE
                           (12): fireworks        - GST_RPI_CAM_SRC_EXPOSURE_MODE_FIREWORKS
  metering-mode       : Camera exposure metering mode to use
                        flags: readable, writable
                        Enum "GstRpiCamSrcExposureMeteringMode" Default: 0, "average"
                           (0): average          - GST_RPI_CAM_SRC_EXPOSURE_METERING_MODE_AVERAGE
                           (1): spot             - GST_RPI_CAM_SRC_EXPOSURE_METERING_MODE_SPOT
                           (2): backlist         - GST_RPI_CAM_SRC_EXPOSURE_METERING_MODE_BACKLIST
                           (3): matrix           - GST_RPI_CAM_SRC_EXPOSURE_METERING_MODE_MATRIX
  awb-mode            : White Balance mode
                        flags: readable, writable
                        Enum "GstRpiCamSrcAWBMode" Default: 1, "auto"
                           (0): off              - GST_RPI_CAM_SRC_AWB_MODE_OFF
                           (1): auto             - GST_RPI_CAM_SRC_AWB_MODE_AUTO
                           (2): sunlight         - GST_RPI_CAM_SRC_AWB_MODE_SUNLIGHT
                           (3): cloudy           - GST_RPI_CAM_SRC_AWB_MODE_CLOUDY
                           (4): shade            - GST_RPI_CAM_SRC_AWB_MODE_SHADE
                           (5): tungsten         - GST_RPI_CAM_SRC_AWB_MODE_TUNGSTEN
                           (6): fluorescent      - GST_RPI_CAM_SRC_AWB_MODE_FLUORESCENT
                           (7): incandescent     - GST_RPI_CAM_SRC_AWB_MODE_INCANDESCENT
                           (8): flash            - GST_RPI_CAM_SRC_AWB_MODE_FLASH
                           (9): horizon          - GST_RPI_CAM_SRC_AWB_MODE_HORIZON
  image-effect        : Visual FX to apply to the image
                        flags: readable, writable
                        Enum "GstRpiCamSrcImageEffect" Default: 0, "none"
                           (0): none             - GST_RPI_CAM_SRC_IMAGEFX_NONE
                           (1): negative         - GST_RPI_CAM_SRC_IMAGEFX_NEGATIVE
                           (2): solarize         - GST_RPI_CAM_SRC_IMAGEFX_SOLARIZE
                           (3): posterize        - GST_RPI_CAM_SRC_IMAGEFX_POSTERIZE
                           (4): whiteboard       - GST_RPI_CAM_SRC_IMAGEFX_WHITEBOARD
                           (5): blackboard       - GST_RPI_CAM_SRC_IMAGEFX_BLACKBOARD
                           (6): sketch           - GST_RPI_CAM_SRC_IMAGEFX_SKETCH
                           (7): denoise          - GST_RPI_CAM_SRC_IMAGEFX_DENOISE
                           (8): emboss           - GST_RPI_CAM_SRC_IMAGEFX_EMBOSS
                           (9): oilpaint         - GST_RPI_CAM_SRC_IMAGEFX_OILPAINT
                           (10): hatch            - GST_RPI_CAM_SRC_IMAGEFX_HATCH
                           (11): gpen             - GST_RPI_CAM_SRC_IMAGEFX_GPEN
                           (12): pastel           - GST_RPI_CAM_SRC_IMAGEFX_PASTEL
                           (13): watercolour      - GST_RPI_CAM_SRC_IMAGEFX_WATERCOLOUR
                           (14): film             - GST_RPI_CAM_SRC_IMAGEFX_FILM
                           (15): blur             - GST_RPI_CAM_SRC_IMAGEFX_BLUR
                           (16): saturation       - GST_RPI_CAM_SRC_IMAGEFX_SATURATION
                           (17): colourswap       - GST_RPI_CAM_SRC_IMAGEFX_COLOURSWAP
                           (18): washedout        - GST_RPI_CAM_SRC_IMAGEFX_WASHEDOUT
                           (19): posterise        - GST_RPI_CAM_SRC_IMAGEFX_POSTERISE
                           (20): colourpoint      - GST_RPI_CAM_SRC_IMAGEFX_COLOURPOINT
                           (21): colourbalance    - GST_RPI_CAM_SRC_IMAGEFX_COLOURBALANCE
                           (22): cartoon          - GST_RPI_CAM_SRC_IMAGEFX_CARTOON
  rotation            : Rotate captured image (0, 90, 180, 270 degrees)
                        flags: readable, writable
                        Integer. Range: 0 - 270 Default: 0 
  hflip               : Flip capture horizontally
                        flags: readable, writable
                        Boolean. Default: false
  vflip               : Flip capture vertically
                        flags: readable, writable
                        Boolean. Default: false
  roi-x               : Normalised region-of-interest X coord
                        flags: readable, writable
                        Float. Range:               0 -               1 Default:               0 
  roi-y               : Normalised region-of-interest Y coord
                        flags: readable, writable
                        Float. Range:               0 -               1 Default:               0 
  roi-w               : Normalised region-of-interest W coord
                        flags: readable, writable
                        Float. Range:               0 -               1 Default:               1 
  roi-h               : Normalised region-of-interest H coord
                        flags: readable, writable
                        Float. Range:               0 -               1 Default:               1

Usage

Because of the way gstreamer works, the parameters for the feed needed to be split and re-arranged in the streamer pipeline. Previously all the parameters are specified as part of raspivid.

/opt/vc/bin/raspivid -t $DURATION -w $WIDTH -h $HEIGHT -fps $FRAMERATE -b $BITRATE -n -pf high -o - | gst-launch-1.0 -v fdsrc ! 

GStreamer parameters like width, height and frame rate are configured through capabilities (caps) negotiation with the next element. Other parameters like the bit rate and preview screen are controlled as part of the source element.

gst-launch-1.0 rpicamsrc bitrate=$BITRATE preview=0 ! video/x-h264,width=$WIDTH,height=$HEIGHT,framerate=$FRAMERATE/1 !

The new stream script is

#!/bin/bash

source remote.conf

if [ "$1" != "" ]
then
  export FRAMERATE=$1
fi

NOW=`date +%Y%m%d%H%M%S`
FILENAME=$NOW-Tx.h264

gst-launch-1.0 rpicamsrc bitrate=$BITRATE preview=0 ! video/x-h264,width=$WIDTH,height=$HEIGHT,framereate=$FRAMERATE/1,profile=high ! h264parse ! rtph264pay config-interval=1 pt=96 ! udpsink host=$RX_IP port=$UDPPORT

Tests

This test was a straight comparison between the old and new scripts using the same settings and the same testing methodology as previously.  The resolution was 1280 x 720 pixels with a 6Mbps bitrate.

Update : Since the original article was published use Jan Schmitt spotted that I had misspelled “framerate” as “framereate” in the gst-rpicamsrc script.  He also suggested I should try using the baseline profile and a queue element to decouple the video capture from the UDP transmission.  With this in mind I have re-run the tests.

Results

Almost immediately I had the feeling that gst-rpicamsrc has slower.  Analysis of the video showed I was correct.  The latency using gst-rpicamsrc was 18% higher than using raspivid.

Update:

Running the original erroneous script with debugging on showed that the capture was running at 30fps instead of the intended 48 fps. Here are the new results averaged from 10 cycles.

Script gst-rpicamsrc @ 48 fps
raspivid @ 48 fps
Profile No queue With queue No queue With queue
baseline 184.2 175.4 153.9 156.9
high 186.2 185 154.1 159.7

gst-rpicamsrc @ 30 fps, high profile, no queue = 198.2 ms

Analysis

The first thing to note is that the raspivid latency (no queue, high profile) has risen from the 126ms found in the last tests to 154ms.  The only difference was that I cloned the Sandisk microSDHC card onto a Transcend 8GB.  I’ll set up some more tests to compare the cards.  As these tests were run from the same card and from the same boot, they are still valid for comparison.

It is immediately obvious that the gst-rpicamsrc latency is about 20% higher than the raspivid script, so the conclusion from the first publish of this article still stands.

What can be added is that using the baseline profile, does reduce the latency a little: 1 to 3ms in most cases.

Adding a queue element does provide a benefit for the gst-rpicamsrc script, especially with the baseline profile where a 9ms reduction in latency was observed.  For the raspivid script adding a queue element actually increased the latency by 3 to 4ms.  I suspect this is because the video stream is already decoupled from gstreamer by being piped in from an external process.

Conclusion

Using gst-rpicamsrc provides no benefit for reducing latency over raspivid.  That is not to say gst-rpicamsrc provides no other benefits.  For any use other than FPV, I would definitely use gst-rpicamsrc instead of having to pipe the video in through stdin.  It provides plenty of options for setting up the video stream as the command `gst-inspect-1.0 rpicamsrc’ above showed.

The problem here is that I am targeting this development for FPV use where low latency is the driving factor. At the moment my lowest latency for a adequate quality HD stream is 125ms and I really need to get this under 100ms to compete with current analog standard definition systems.  Whether it is possible to shave of another 25ms remains to be seen.

Update: Following the additional tests I would add that it is better to use the baseline profile over the high profile.

B or B+

Extract

The Raspberry Pi Model B+ has recently been released with improved USB hardware.  This test aimed to determine if this new hardware would reduce the latency compared to the Model B.  And the answer I’m afraid is No.

FPiV Model B v B+

Setup

The new USB hardware requires different firmware from previously.  As I was going to have to rebuild the operating system anyway, I opted to try a minimal installation using raspbian-ua-netinst.  There was a second reason for doing this, which was to install the “Jessie” build of Raspbian rather than “Wheezy” because “Jessie” comes with the latest build of gstreamer as standard.  This is currently at version 1.4.1.  In “Wheezy” you only have access to version 1.0.

Two installs were completed using identical procedures. One on my old model B and one on a new B+.

The test environment was set up the same as previously, with the camera pointing at the laptop screen.  The camera was positioned so that at least ten iterations of the image was visible at once. I used the LED on my phone with a flashlight app for measurement indicator.  The tests were run with no overclocking.

I used a UDP stream using the same scripts as for the previous Transport Stream and Overclocking Test.

A GoPro 3 Black was used to record the screen at 120 fps.

Three identical tests were run.  One for each Raspberry Pi using the operating system that had been built on it and then a third test using the B+ micro SD card in the model B with an adapter.  A number of led on/led off cycles were recorded and then two cycles from each run were picked at random for analysis.

Results

The GoPro video was analyzed using VideoReDo H264 to establish the number of frames between the LED switch on (or off) and the tenth iteration switch on/off.  This value was then divided by 1.2 to get the latency in milliseconds.

  1. Model B, Native built OS : 156ms, 150ms, 138ms, 147 ms = an average of 147ms +/- 10ms
  2. Model B+, Native built OS : 153ms, 154ms, 157ms, 154ms = an average of 154 ms +/- 3ms
  3. Model B with B+ built OS : – 148ms, 142ms, 158ms, 148ms = an average of 149 ms +/- 8 ms

Conclusion

For now the Model B+ appears to give a slightly higher latency than the Model B.  This is a shame as the B+ has a much cleaner layout.  Hopefully, the latency will improve with time as the USB drivers are optimised.  The “Jessie” build of Raspbian has not been officially released yet.

The results for the Model B were slightly better than the previous Transport Stream and Overclocking Test by about 10 millseconds so it appears the minimal raspbian-ua-netinst build was worthwhile.  See this post for details on how to configure raspbian-ua-netinst for “Jessie”.

Raspberry Pi Camera Latency Testing – Part 2

Netcat v Gstreamer

Extract

After a number of tests I have established that the Raspberry Pi is able to provide a low latency HD stream over a wifi connection.  There are limitation, but the results have been encouraging enough for me to push from desk to airborne testing.

Introduction

Back in May 2013 when the RPi camera appeared I did some latency testing to see if the platform might be suitable for FPV duties.  Using the default NetCat based solution the minimum latency I could achieve was 300ms and the stream wasn’t particularly smooth, so the conclusion was no.

Then a few couple of months later I was asked to try running the tests again using gstreamer instead of netcat.  It’s taken longer than I had hoped, but I scrapped my first set of results and re-ran them with a more structured methodology.  As the results were encouraging, I expanded the tests to see how far it would go.

In fact the tests were so encouraging I kept extending and extending and never got around to posting the results.  Now with the UK locked in perpetual rain and wind, I though I’s take the time to share the results.

Hardware Components.

For the tests I used:

  • Raspberry Pi model B
  • Raspberry Pi Camera.
  • Edimax EW-7612UAn V2 USB WiFi Adapter (with high gain antenna).
  • MSI Wind U100 Single Core 1.6 GHz Atom Netbook, Circa 2008
  • DELL XPS 17, Quad Core i7-2630QM 2.0GHz Laptop, Circa 2012

Raspberry Pi System Setup.

Initial setup was done with the Pi connected via LAN using the then current build of Raspbian.  The first tasks were to expand the storage to fill the SD card, enable the Camera, enable ssh and update the system using apt-get.

I followed Gbaman’s instructions on setting up gstreamer on the Pi, although I used Lubuntu as the client instead of mac os.  Lubuntu was run from a USB stick so I could move it between laptops to compare performance. This was the system I used for the first set of tests which was later discarded as I felt the system could be improved.

I wanted the Pi to act as a WiFi access point.  This removes the need for a WiFi base station without having to use an ad-hoc network. I followed Adafruit’s instructions to setup hostapd, but not isc-dhcp-server or NAT translation. For dhcp I setup dnsmasq based on the first half of Adafruit’s Ad blocking access point setup,  stop at “dig adafruit.com”.  With this setup I could connect the laptop to the Pi’s WiFi and login over ssh.  Apart from plugging in the power, no other connections to the Pi are required.

I created a script on the Pi to start the video stream and another on the laptop to receive it.  The Pi camera was pointed at the laptop screen so multiple iterations could be recorded and the whole test was recorded on a GoPro at 120 frames per second.  For some of the early tests a screen was connected to the Pi’s hdmi output to show the local preview.

Test 1

The first tests started where the first video finished at 640 x 480 pixel resolution and 0.7Mbit/s bitrate run from the RPi desktop with local preview.  The gstreamer feed was about three times faster than netcat.  The gstreamer feed was also more consistent.

GS2

Netcat : 505 ms ±50%, Gstreamer : 167 ms ± 17%

This sub 200ms latency for gstreamer is probably a usable value for FPV, so initial results were promising.

Test 2

For the second test gstreamer was started from the Pi’s command prompt instead of the desktop

NetCat : 514 ms 60%,    Gstreamer 174ms 30%

NetCat : 514 ms ±64%, Gstreamer 174ms ±30%

The results were actually slightly slower for both systems but gstreamer was still under the 200ms mark.

Netcat testing was dropped at this point as it was never going to compete with gstreamer.

Tests 3 & 4

 The next tests were run with the Pi preview removed as this would never be needed in a FPV system. The tests were run for 640 x 480 x 0.7 Mbit/s, 720 x 480 x 1 Mbit/s and 1280 x 720 x 2.5 Mbit/s streams. I also ran another test at the native screen size of the Netbook 1024 x 600 x 1.6 Mbit/s, but this is not shown on the video or graph.

Test 3 & 4


640 x 480 x 0.7 Mbit/s 186ms ± 18%
720 x 480 x 1.0 Mbit/s 169ms ± 20% (Ignoring initial long iteration)
1024 x 600 x 1.6 Mbit/s 163ms ± 5%
1280 x 720 x 2.5 Mbit/s 194ms ± 24%

The averaged results are strange because for the first three results the latency comes down as the resolution and bit rate go up.  The difference is still withing the measurement accuracy so is probable that the latency for the lower resolutions are actually all in the same ballpark.  Only once we hit the lower HD resolution of 1280 x 720 x 2.5 Mbit/s does the latency go back up, suggesting that a saturation point has been reached somewhere in the pipeline.

Tests 5 & 6

These tests were run to compare results between having a Single core 1.6 Ghz Atom Netbook and a Quad Core i7-2630QM 2.0GHz Laptop at the receiving end. The tests were run with 640 x 480 x 0.7 Mbit/s and 1280 x 720 x 2.5 Mbit/s streams

640 x 480 x 0.7 Mbit/s    Netbook     186ms ± 18% Laptop       154ms ± 30% 1280 x 720 x 2.5M bit/s  Netbook     194ms ± 24% Laptop       171ms ± 17%

640 x 480 x 0.7M bit/s Netbook 186ms ± 18%
640 x 480 x 0.7 Mbit/s Laptop 154ms ± 30%
1280 x 720 x 2.5 Mbit/s Netbook 194ms ± 24%
1280 x 720 x 2.5 Mbit/s Laptop 171ms ± 17%

On the quad core laptop the 1280 x 720 stream has dropped back down to the 170ms range and the 640 x 480 stream has dropped to almost 150ms.

Test 7

I re-ran the test again at the full HD resolution of 1920 x 1080 x 6 Mbit/s.  This worked OK on the laptop although the latency rose to 233ms ± 27%. This may be usable on a big powered glider or attitude stabilised multicopter, but is pushing it for rate or acro mode flying.

The netbook took almost 9 seconds (8982ms ± 2%) for one iteration.  If you want to run full HD you are going to need a good laptop at the receiving end.

Test 8

Now I had a usable stream, the next requirement is to record it.  On the Raspberry Pi we are sending the camera output to gstreamer through a stdout/stdin pipe.  This makes it easy to to capture the stream to a file using the tee command.  The *nix tee command copies the stdin stream to stdout and saves a copy to a specified file. This allows a copy of the video stream to be saved to the SD card.

raspivid <options> | tee file.h264 | gstreamer <options>

tee

The results showed that the tee command added nothing to the latency.  This means the stream can always be recorded at the sending end without worrying about missed frames due to WiFi dropout.

Recording the stream at the receiving end is a much harder problem.  Without writing a dedicated gstreamer application I can’t see an easy way to save the stream to a valid mp4 file.  I’ve got a workaround that allows me to capture the stream using gst-launch-1.0 and mux it into a usable file, but at the moment, it requires multiple steps across Linux and Windows.   I’ll document that in a later post.

Summary

  • Best SD latency (640 x 480): 154ms
  • Best HD latency (1280 x 720): 171ms
  • Best Full HD latency (1920 x 1080): 233ms
  • Saving a copy of the stream on the RPi using the tee command does not affect the latency.

Conclusions

Based on these results I believe the Raspberry Pi with camera could be used as an HD FPV platform in conjunction with a reasonably good laptop, although I would limit it to 720p HD and not 1080p.  Because the system uses 2.4Ghz Wifi consideration needs to be given to interference from other WiFi networks and 2.4Ghz RC transmitters.  This could be solved using 5Ghz WiFi dongle provided a USB dongle can be found that’s compatible with the Raspberry Pi and features a high gain antenna.

A second consideration is range.  In a house WiFi range can be as little as 10m due to various walls.  However, in free air the range can be a lot more.  Testing is required to establish the actual usable range.

During the tests 1 & 2 I recorded the time to display the preview on the RPi.  The latency for this was typically 86 ms.  As this preview was shown before the stream was compressed it can be assumed that at least half of the latency comes in reading and encoding the stream in the Raspberry Pi.  Any optimization that could be made here would have a significant benefit.

The Next Step

I’ve got an old slope soarer that uses a 35Mz RC system which won’t interfere with the 2.4GHz Wifi.  I’ve made a power pod for it and will start some some line of sight range testing with the Raspberry Pi banded on top of the wing.  I’ve soldered up some cables to connect a Hobbywing 3A BEC to a USB Micro B plug to feed the required 5V to the Raspberry Pi.

Raspberry Pi Camera Latency Testing – Part 1

When the Raspberry Pi Camera was first released I ran some latency testing with the netcat based pipeline to see if it was suitable for FPV use.
The initial results at full HD resolution were not promising as the following video shows.

This is a test of the Raspberry Pi camera streaming over 802.11g WiFi to a netbook running Windows 7.

The stream is 1920×1080 pixels with at the default data rate. The large screen is the HDMI feed directly from the Raspberry Pi. The small screen is the netbook showing the received stream.

The stream was created and received using the inbuilt commands given at www.designspark.com and recorded at 29.97 frames/s.

By analyzing the video in Movie Maker the latency between real life and the big screen was 0.2 to 0.3 seconds. The latency from real life to the netbook as around 4.5 seconds. This is obviously too high for remotely piloting a vehicle or even aiming a bigger camera. The next step is to reduce the camera resolution and bitrate to try to improve this.

After gradually reducing the picture quality a bit at a time I arrived at a 640×480 pixel resolution with a 1 MBits/s data rate.  There was no noticeable latency between real life and the big screen (Raspberry Pi preview).

The Movie Maker analysis shows the latency averaging around 0.3 seconds, which is still probably too high for remotely piloting a vehicle although could be used for aiming a second camera such as an SLR. However it may be possible to reduce this using dedicated code instead of the piped the output of multiple commands.

Orginially posted on Youtube on 21 May 2013.