Netcat v Gstreamer
After a number of tests I have established that the Raspberry Pi is able to provide a low latency HD stream over a wifi connection. There are limitation, but the results have been encouraging enough for me to push from desk to airborne testing.
Back in May 2013 when the RPi camera appeared I did some latency testing to see if the platform might be suitable for FPV duties. Using the default NetCat based solution the minimum latency I could achieve was 300ms and the stream wasn’t particularly smooth, so the conclusion was no.
Then a few couple of months later I was asked to try running the tests again using gstreamer instead of netcat. It’s taken longer than I had hoped, but I scrapped my first set of results and re-ran them with a more structured methodology. As the results were encouraging, I expanded the tests to see how far it would go.
In fact the tests were so encouraging I kept extending and extending and never got around to posting the results. Now with the UK locked in perpetual rain and wind, I though I’s take the time to share the results.
For the tests I used:
- Raspberry Pi model B
- Raspberry Pi Camera.
- Edimax EW-7612UAn V2 USB WiFi Adapter (with high gain antenna).
- MSI Wind U100 Single Core 1.6 GHz Atom Netbook, Circa 2008
- DELL XPS 17, Quad Core i7-2630QM 2.0GHz Laptop, Circa 2012
Raspberry Pi System Setup.
Initial setup was done with the Pi connected via LAN using the then current build of Raspbian. The first tasks were to expand the storage to fill the SD card, enable the Camera, enable ssh and update the system using apt-get.
I followed Gbaman’s instructions on setting up gstreamer on the Pi, although I used Lubuntu as the client instead of mac os. Lubuntu was run from a USB stick so I could move it between laptops to compare performance. This was the system I used for the first set of tests which was later discarded as I felt the system could be improved.
I wanted the Pi to act as a WiFi access point. This removes the need for a WiFi base station without having to use an ad-hoc network. I followed Adafruit’s instructions to setup hostapd, but not isc-dhcp-server or NAT translation. For dhcp I setup dnsmasq based on the first half of Adafruit’s Ad blocking access point setup, stop at “dig adafruit.com”. With this setup I could connect the laptop to the Pi’s WiFi and login over ssh. Apart from plugging in the power, no other connections to the Pi are required.
I created a script on the Pi to start the video stream and another on the laptop to receive it. The Pi camera was pointed at the laptop screen so multiple iterations could be recorded and the whole test was recorded on a GoPro at 120 frames per second. For some of the early tests a screen was connected to the Pi’s hdmi output to show the local preview.
The first tests started where the first video finished at 640 x 480 pixel resolution and 0.7Mbit/s bitrate run from the RPi desktop with local preview. The gstreamer feed was about three times faster than netcat. The gstreamer feed was also more consistent.
This sub 200ms latency for gstreamer is probably a usable value for FPV, so initial results were promising.
For the second test gstreamer was started from the Pi’s command prompt instead of the desktop
The results were actually slightly slower for both systems but gstreamer was still under the 200ms mark.
Netcat testing was dropped at this point as it was never going to compete with gstreamer.
Tests 3 & 4
The next tests were run with the Pi preview removed as this would never be needed in a FPV system. The tests were run for 640 x 480 x 0.7 Mbit/s, 720 x 480 x 1 Mbit/s and 1280 x 720 x 2.5 Mbit/s streams. I also ran another test at the native screen size of the Netbook 1024 x 600 x 1.6 Mbit/s, but this is not shown on the video or graph.
The averaged results are strange because for the first three results the latency comes down as the resolution and bit rate go up. The difference is still withing the measurement accuracy so is probable that the latency for the lower resolutions are actually all in the same ballpark. Only once we hit the lower HD resolution of 1280 x 720 x 2.5 Mbit/s does the latency go back up, suggesting that a saturation point has been reached somewhere in the pipeline.
Tests 5 & 6
These tests were run to compare results between having a Single core 1.6 Ghz Atom Netbook and a Quad Core i7-2630QM 2.0GHz Laptop at the receiving end. The tests were run with 640 x 480 x 0.7 Mbit/s and 1280 x 720 x 2.5 Mbit/s streams
On the quad core laptop the 1280 x 720 stream has dropped back down to the 170ms range and the 640 x 480 stream has dropped to almost 150ms.
I re-ran the test again at the full HD resolution of 1920 x 1080 x 6 Mbit/s. This worked OK on the laptop although the latency rose to 233ms ± 27%. This may be usable on a big powered glider or attitude stabilised multicopter, but is pushing it for rate or acro mode flying.
The netbook took almost 9 seconds (8982ms ± 2%) for one iteration. If you want to run full HD you are going to need a good laptop at the receiving end.
Now I had a usable stream, the next requirement is to record it. On the Raspberry Pi we are sending the camera output to gstreamer through a stdout/stdin pipe. This makes it easy to to capture the stream to a file using the tee command. The *nix tee command copies the stdin stream to stdout and saves a copy to a specified file. This allows a copy of the video stream to be saved to the SD card.
raspivid <options> | tee file.h264 | gstreamer <options>
The results showed that the tee command added nothing to the latency. This means the stream can always be recorded at the sending end without worrying about missed frames due to WiFi dropout.
Recording the stream at the receiving end is a much harder problem. Without writing a dedicated gstreamer application I can’t see an easy way to save the stream to a valid mp4 file. I’ve got a workaround that allows me to capture the stream using gst-launch-1.0 and mux it into a usable file, but at the moment, it requires multiple steps across Linux and Windows. I’ll document that in a later post.
- Best SD latency (640 x 480): 154ms
- Best HD latency (1280 x 720): 171ms
- Best Full HD latency (1920 x 1080): 233ms
- Saving a copy of the stream on the RPi using the tee command does not affect the latency.
Based on these results I believe the Raspberry Pi with camera could be used as an HD FPV platform in conjunction with a reasonably good laptop, although I would limit it to 720p HD and not 1080p. Because the system uses 2.4Ghz Wifi consideration needs to be given to interference from other WiFi networks and 2.4Ghz RC transmitters. This could be solved using 5Ghz WiFi dongle provided a USB dongle can be found that’s compatible with the Raspberry Pi and features a high gain antenna.
A second consideration is range. In a house WiFi range can be as little as 10m due to various walls. However, in free air the range can be a lot more. Testing is required to establish the actual usable range.
During the tests 1 & 2 I recorded the time to display the preview on the RPi. The latency for this was typically 86 ms. As this preview was shown before the stream was compressed it can be assumed that at least half of the latency comes in reading and encoding the stream in the Raspberry Pi. Any optimization that could be made here would have a significant benefit.
The Next Step
I’ve got an old slope soarer that uses a 35Mz RC system which won’t interfere with the 2.4GHz Wifi. I’ve made a power pod for it and will start some some line of sight range testing with the Raspberry Pi banded on top of the wing. I’ve soldered up some cables to connect a Hobbywing 3A BEC to a USB Micro B plug to feed the required 5V to the Raspberry Pi.