Effect of Frame & Bit rates on Latency

Extract

Back in March, an update to the Raspberry Pi camera software introduced some new video modes with higher frame rates for resolutions below full 1080p HD.  For 720p HD, frame rates up to 49 fps are now available and for VGA (640×480) there are new modes for 60 and 90 fps.  There were reports that, at these higher frame rates, the latency was reduced.  This didn’t come as a surprise as it has been known for a while that the GoPro video out has lower latency at 48 fps than at 30 fps.  The reasons for this are not totally clear, but is thought to be due to the differences between frames being less at the higher frame rates allowing for more efficient compression.

In order to investigate this I ran some back-to-back tests at various frame rates.  I was expecting that with twice as many frames being squeezed into the same bit rates, the video quality would suffer, so the tests where repeated at three different bit rates.  The result of this testing was a 25% reduction in latency, with a new minimum of 121 milliseconds at 720p and 48 fps.  As expected the video quality at the old bit rate of 2.5Mbps suffered for fast moving backgrounds, but even when increased to 6.0Mbps, the latency was still only 126 Mbps; a 23% reduction.

With VGA resolution at 60fps and 2.5 Mbps the latency dropped to 87 ms and while this is useful, in terms of hardware size, convenience and even latency, it is still bettered by dedicated FPV hardware.  If you include the cost of the laptop, it looses out on price too.  Where the Pi has the advantage is at the 1280 x 720p HD resolution where there is currently no affordable competition.

Setup

The test environment was set up the same as previously, with the camera pointing at the laptop screen.  The camera was positioned so that at least ten iterations of the image were visible at once. I used a single LED torch as the measurement indicator.  The tests were run with no overclocking.

I used a UDP stream with some improved scripts to simplify running all the variations. The scripts can be found on the Current Scripts Page.

A GoPro 3 Black was used to record the screen at 120 fps.

Tests were run at frame rates of 25, 30, 36, 42, 45, 48 and 49 fps and at bit rates of 2.5, 4.5 and 6.0 Mbps.   Up to 5 LED on/led off cycles were recorded on the GoPro for later analysis.  With 10 iteration for each of the 5 on/offs, the calculated latency represents the average of 100 tests.

Results 1 – Latency

Across all the bit rates, the results show a definite reduction in latency with increasing frame rate right up to 48 frames per second. At 49 fps the latency increased slightly. The lowest latency achieved was 121ms at 48 fps for the 2.5 Mbps stream.  A reduction of 25% over the 25 fps result.  The 6.0 Mps stream still managed a reduced latency of 126 ms at 48 fps and the 4.5 Mbps stream achieved 125 ms.

The VGA test showed a much lower latency at 87ms, even though it was using the same bit rate as the smallest 720p HD stream.

The graph shows some odd behavior between the 4.5 and 6.0 Mbps curves.  Below about 40 fps the 4.5 Mbps stream has higher latency than the 6.0 Mbps stream.  Beyond 40 fps the curves switch over to what you would expect.

fps

In addition to the average latency, the graph below shows the minimum and maximums for each bit rate.  What is apparent is that there is a lot of overlap between the results and that the latency can vary +/- 20 fps, particularly at the lower bit rates.

fps2

Results 2 – Video Quality

The frame grabs below show a clear difference between the three bit rates at 48 fps. The 2.5 Mbps stream shows extreme pixelation and a general blurring of the image.

Quality25

Frame grab from 2.5 Mbps stream @ 48 fps

The  4.5 Mbps shows a reduction in the pixelation.  This is apparent in the sky and on the tiled roof.

Quality45

Frame grab from 4.5 Mbps Stream @ 48 fps

The 6.0 Mbps is the clear winner, with a minimum of pixelation and the sharpest image.

Quality60

Frame grab from 6.0 Mbps Stream @ 48 fps

Conclusion

The total latency is made up of capture, compression, transmission and display components.  The capture and  display components should be the same for the three bit rates as the resolution is the same.  The transmission component should increase in relation to the higher bit rates.  That just leaves the compression component.  The compressor has to work hard to squeeze the stream into a smaller bit rate whilst maintaining the best video quality possible.  For this reason decreasing the bit rate can increase the latency, although there is probably a point where the extreme pixelation starts reducing the latency again.

These opposing affects between compression and transmission are likely what has caused the odd behavior between the 4.5 Mbps and 6.0 Mbps streams.  At the lower frame rates, the compression delay for the 4.5 Mbps stream was more significant.  At the higher frame rates the transmission delay in the 6.0 Mbps stream became more significant.

While the VGA test used the same 2.5 Mbps bit rate as the HD test, with only a third of the pixels, the capture, compression and display components resulted in a much lower latency of 87 ms.

While the lowest latency was achieved for the 2.5 Mbps stream, the extreme pixelation of the image renders it pretty much unusable.  The 4.5 Mbps stream could be used but it only has a 1 ms advantage over the superior image quality of the 6.0 Mbps stream.  For this reason I plan to adopt the 48 fps, 6.0 Mbps, 126 ms stream as my new baseline when flying.

I will probably also switch to a 24 fps pipeline for youtube videos to get the best image quality from the 48 fps video, dropping every other frame.  I have seen this done where people have used the 48 fps recording on GoPro cameras.

Transport Stream and Overclocking Comparison

Extract

Various issues have kept me grounded for a couple of months, but I have managed to get some bench testing done.  Two things I wanted to test were a comparison of TCP versus UDP for the transport stream and the effect of overclocking the Pi.  In both cases the results were disappointing with only minimal reductions in overall latency.

TransportAndOverclockLatency

Setup

The test environment was set up the same as previously, with the camera pointing at the laptop screen.  The camera was positioned so that at least ten iterations of the image was visible at once. I used the LED on my phone with a flashlight app for measurement indicator.  The tests were run with the no overclocking and then the maximum (Turbo) setting available in raspi-config.

The scripts for transmitting and receiving the video stream are shown below.

A GoPro 3 Black was used to record the screen at 120 fps.

TCP – Pi Transmitter Script

#!/bin/sh
cat $0

# Zero duration is continuous
DURATION=0
WIDTH=1280
HEIGHT=720
FRAMERATE=30
BITRATE=2500000
IP=$(ip -o addr show wlan0 | sed -n 's/.*inet \([0-9.]*\)\/.. .*/\1/p')
PORT=5000

/opt/vc/bin/raspivid -t $DURATION -w $WIDTH -h $HEIGHT -fps $FRAMERATE -b $BITRATE -n -o - | gst-launch-1.0 -v fdsrc ! h264parse ! rtph264pay config-interval=1 pt=96 ! gdppay ! tcpserversink host=$IP port=$PORT

TCP – Laptop Receiver and Recording Script

#!/bin/bash
source ./settings.conf

NOW=`date +%Y%m%d%H%M%S`
FILENAME=$NOW-Rx.ts

gst-launch-1.0 -v tcpclientsrc host=$IP port=$TCPPORT ! \
  gdpdepay ! rtph264depay ! \
  h264parse config-interval=96 ! \
  tee name=t ! queue ! \
  video/x-h264, framerate=25/1 ! avdec_h264 ! videoconvert ! autovideosink sync=false t. ! \
  queue ! mpegtsmux ! filesink location=$FILENAME

UDP – Pi Transmitter Script

#!/bin/sh
cat $0

# Zero duration is continuous
DURATION=0
WIDTH=1280
HEIGHT=720
FRAMERATE=48
BITRATE=2500000
IP=$(ip -o addr show wlan0 | sed -n 's/.*inet \([0-9.]*\)\/.. .*/\1/p')

#Edit the address below to match the receiving computer
TARGETIP=192.168.42.26

UDPPORT=5001
NOW=`date +%Y%m%d%H%M%S`
FILENAME=$NOW-Tx.h264

/opt/vc/bin/raspivid -t $DURATION -w $WIDTH -h $HEIGHT -fps $FRAMERATE -b $BITRATE -n -o - | gst-launch-1.0 -v fdsrc ! h264parse ! rtph264pay config-interval=1 pt=96 ! udpsink host=$TARGETIP port=$UDPPORT

UDP – Laptop Receiver and Recording Script

#!/bin/bash
source ./settings.conf

NOW=`date +%Y%m%d%H%M%S`
FILENAME=$NOW-Rx.ts

gst-launch-1.0 -v udpsrc port=$UDPPORT \
  caps='application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264' ! \
  rtph264depay ! h264parse config-interval=96 ! \
  tee name=t ! queue ! \
  video/x-h264, framerate=48/1 ! avdec_h264 ! videoconvert ! autovideosink sync=false t. ! \
  queue ! mpegtsmux ! filesink location=$FILENAME

settings.conf

#!/bin/bash
export IP=192.168.42.1
export TCPPORT=5000
export UDPPORT=5001
export FRAMERATE=25

Results

The GoPro video was analyzed in Sony Movie Studio 11 to establish the number of frames between the LED switch on and the tenth iteration switch on.  This value was then divided by 120 to get the number of seconds for ten iterations, divided by 10 to get the number of seconds per iteration and finally multiplied by 1000 to get the latency in milliseconds.

  1. TCP – No Overclock – 160 ms
  2. TCP – Turbo Overclock – 160 ms
  3. UDP – No Overclock – 157 ms
  4. UDP – Turbo Overclock – 155 ms

As you can see from the graph, there is very little difference between the values. Switching to UDP gives a 3ms reduction in latency over TCP and the Turbo overclock removes another 2ms.  TCP doesn’t seem to benefit from overclocking at all.

Conclusion

Based on these results, I plan to switch future testing to UDP.  I’m not going to bother overclocking though, prefering the lower current draw and better long term stability of no overclocking.