Back in March, an update to the Raspberry Pi camera software introduced some new video modes with higher frame rates for resolutions below full 1080p HD. For 720p HD, frame rates up to 49 fps are now available and for VGA (640×480) there are new modes for 60 and 90 fps. There were reports that, at these higher frame rates, the latency was reduced. This didn’t come as a surprise as it has been known for a while that the GoPro video out has lower latency at 48 fps than at 30 fps. The reasons for this are not totally clear, but is thought to be due to the differences between frames being less at the higher frame rates allowing for more efficient compression.
In order to investigate this I ran some back-to-back tests at various frame rates. I was expecting that with twice as many frames being squeezed into the same bit rates, the video quality would suffer, so the tests where repeated at three different bit rates. The result of this testing was a 25% reduction in latency, with a new minimum of 121 milliseconds at 720p and 48 fps. As expected the video quality at the old bit rate of 2.5Mbps suffered for fast moving backgrounds, but even when increased to 6.0Mbps, the latency was still only 126 Mbps; a 23% reduction.
With VGA resolution at 60fps and 2.5 Mbps the latency dropped to 87 ms and while this is useful, in terms of hardware size, convenience and even latency, it is still bettered by dedicated FPV hardware. If you include the cost of the laptop, it looses out on price too. Where the Pi has the advantage is at the 1280 x 720p HD resolution where there is currently no affordable competition.
The test environment was set up the same as previously, with the camera pointing at the laptop screen. The camera was positioned so that at least ten iterations of the image were visible at once. I used a single LED torch as the measurement indicator. The tests were run with no overclocking.
I used a UDP stream with some improved scripts to simplify running all the variations. The scripts can be found on the Current Scripts Page.
A GoPro 3 Black was used to record the screen at 120 fps.
Tests were run at frame rates of 25, 30, 36, 42, 45, 48 and 49 fps and at bit rates of 2.5, 4.5 and 6.0 Mbps. Up to 5 LED on/led off cycles were recorded on the GoPro for later analysis. With 10 iteration for each of the 5 on/offs, the calculated latency represents the average of 100 tests.
Results 1 – Latency
Across all the bit rates, the results show a definite reduction in latency with increasing frame rate right up to 48 frames per second. At 49 fps the latency increased slightly. The lowest latency achieved was 121ms at 48 fps for the 2.5 Mbps stream. A reduction of 25% over the 25 fps result. The 6.0 Mps stream still managed a reduced latency of 126 ms at 48 fps and the 4.5 Mbps stream achieved 125 ms.
The VGA test showed a much lower latency at 87ms, even though it was using the same bit rate as the smallest 720p HD stream.
The graph shows some odd behavior between the 4.5 and 6.0 Mbps curves. Below about 40 fps the 4.5 Mbps stream has higher latency than the 6.0 Mbps stream. Beyond 40 fps the curves switch over to what you would expect.
In addition to the average latency, the graph below shows the minimum and maximums for each bit rate. What is apparent is that there is a lot of overlap between the results and that the latency can vary +/- 20 fps, particularly at the lower bit rates.
Results 2 – Video Quality
The frame grabs below show a clear difference between the three bit rates at 48 fps. The 2.5 Mbps stream shows extreme pixelation and a general blurring of the image.
The 4.5 Mbps shows a reduction in the pixelation. This is apparent in the sky and on the tiled roof.
The 6.0 Mbps is the clear winner, with a minimum of pixelation and the sharpest image.
The total latency is made up of capture, compression, transmission and display components. The capture and display components should be the same for the three bit rates as the resolution is the same. The transmission component should increase in relation to the higher bit rates. That just leaves the compression component. The compressor has to work hard to squeeze the stream into a smaller bit rate whilst maintaining the best video quality possible. For this reason decreasing the bit rate can increase the latency, although there is probably a point where the extreme pixelation starts reducing the latency again.
These opposing affects between compression and transmission are likely what has caused the odd behavior between the 4.5 Mbps and 6.0 Mbps streams. At the lower frame rates, the compression delay for the 4.5 Mbps stream was more significant. At the higher frame rates the transmission delay in the 6.0 Mbps stream became more significant.
While the VGA test used the same 2.5 Mbps bit rate as the HD test, with only a third of the pixels, the capture, compression and display components resulted in a much lower latency of 87 ms.
While the lowest latency was achieved for the 2.5 Mbps stream, the extreme pixelation of the image renders it pretty much unusable. The 4.5 Mbps stream could be used but it only has a 1 ms advantage over the superior image quality of the 6.0 Mbps stream. For this reason I plan to adopt the 48 fps, 6.0 Mbps, 126 ms stream as my new baseline when flying.
I will probably also switch to a 24 fps pipeline for youtube videos to get the best image quality from the 48 fps video, dropping every other frame. I have seen this done where people have used the 48 fps recording on GoPro cameras.