Raspberry FPiV – Flight 3 – Recording on the Ground

Extract

After much trawling through the forums, I found a way to record the video stream at the receiving end using gst-launch-1.0 in such a way that it can be converted to a playable format later on without re-encoding and the resulting loss of quality.  Eventually, replacing gst-launch-1.0 with a dedicated gstreamer application would allow a valid video file to be recorded straight from the stream without the current post processing. This flight also included a test of the external amplified antenna for the Ultimate GPS Breakout board. The results were disappointing, not attaining sufficient accuracy for the desired application. The flight was cut short due to a combination of high, turbulent wind and dumb thumbs, but was long enough to get meaningful results. You need to watch the video at 720p for a meaningful comparison. Please bear in mind that the video has been re-encoded at least once, possibly three times between the Pi and your browser. https://www.youtube.com/watch?v=eCazn-JbR_0

Hardware Setup

The amplified GPS antenna comes with 5 metres of cable and a heavy magnet.  I shortened the cable and removed the magnet, but the parts were still unwieldy when strapped to the top of the plane.  Hardware wise this was the only addition to the previous test. 2014-04-25 13.15.03

 Software Setup

I had already generated and swapped ssh keys between the Pi and the laptop to remove the need to enter a password when connecting. While I’ve been testing different GPS setups , I’ve started gpsd using a small shell script

#!/bin/bash
echo Starting gpsd
sudo gpsd /dev/ttyAMA0 -n -F /var/run/gpsd.sock

I want to use the current time and date for the video and gpx file names.  As the Pi does not have a real time clock, I set the time from GPS using a  script based on one I found at  http://blog.petrilopia.net

#!/bin/bash
sudo date -s '01/01/2014 00:01'
sleep 1
GPSDATE=`gpspipe -w | head -10 | grep TPV | sed -r 's/.*"time":"([^"]*)".*/\1/' | head -1`
echo $GPSDATE
sudo date -s "$GPSDATE"

Next I run cgps to check the GPS accuracy.  Once it stops improving I exit cgps and run the third script to start gps logging and record the video stream.  The video is recorded as the raw h264 stream with a Tx (Transmitter) suffix.

#!/bin/sh
cat $0

# Duration in milliseconds. Use 0(Zero) for continuous
DURATION=1800000
WIDTH=1280
HEIGHT=720
FRAMERATE=25
BITRATE=2500000
NOW=`date +%Y%m%d%H%M%S`
FILENAME=$NOW-Tx.h264
gpxlogger -i 2 -f $NOW.gpx &
IP=$(ip -o addr show wlan0 | sed -n 's/.*inet \([0-9.]*\)\/.. .*/\1/p')
PORT=5000

/opt/vc/bin/raspivid -t $DURATION -w $WIDTH -h $HEIGHT -fps $FRAMERATE -b $BITRATE -n -o - | tee $FILENAME | gst-launch-1.0 -v fdsrc ! h264parse ! rtph264pay config-interval=1 pt=96 ! gdppay ! tcpserversink host=$IP port=$PORT

Finally, back on the laptop, in a new terminal window, I run a script to capture and display the video stream.  The h264 file is multiplexed (muxed) into an mpeg transport stream (ts) container with a Rx (Receiver) suffix.

#!/bin/bash
# Read the Settings file
source ./settings.conf

# Set file name from current time
NOW=`date +%Y%m%d%H%M%S`
FILENAME=$NOW-Rx.ts

# Read the stream.
gst-launch-1.0 -v tcpclientsrc host=$IP port=$PORT ! \
 gdpdepay ! rtph264depay ! \
 h264parse config-interval=96 ! \
 tee name=t ! queue ! \
 video/x-h264, framerate=25/1 ! avdec_h264 ! videoconvert ! autovideosink sync=false t. ! \
 queue ! mpegtsmux ! filesink location=$FILENAME

# Try to copy the file from SD card to Hard Disk
if [ ! -d $TARGETDIR ]; then
 sudo mount /dev/sdb4 /mnt
fi
cp $FILE $TARGETDIR/$FILENAME

Here is a sample settings.conf

#!/bin/bash
export IP=192.168.42.1
export PORT=5000
export FRAMERATE=25
export TARGETDIR=/mnt/home/user

The Flight

When I arrived at the field the wind felt to be blowing at a constant 10 mph.  In the short time it took to get everything ready it had started gusting much harder.  Having driven to the field and got everything setup I was determined to fly so opened the throttle and gave it a good chuck. My first realisation was that the wind was stronger than anything I had flown in before.  In order to get the aircraft to track up the field and not be blown out the back, I had to point it 20 to 30 degrees past the desired direction of flight.  Also, as the aircraft got further up the field and closer to the trees, the turbulence made it harder to control.  Eventually, it got bucked around to the point I lost orientation, dumb thumbed the rudder flipping it upside down and I wasn’t able to recover it before it hit the ground. When I checked later, the met office were quoting 26 mph gusts.

The Aftermath

The previous months rain had softened the ground enough that the aircraft nose was stuck in three or four inches with no apparent damage.  The rubber bands had separated, releasing the wings.  Any electronics that weren’t screwed down were scattered around the crash site and given a liberal coating of mud. Everything was dismantled and cleaned in fresh water, then left to dry.   When I did reconnect everything, it all worked.  The only problem was that camera now has a green tint.  I suspect that that the mud has abraded the lens coating, either on impact or during cleaning.  As they are cheap I bought a new one.

Results

I was worried that the crash would have lost or corrupted the on-board video recording.  As it turned out, only the last few seconds of the flight were lost, as you can see in the video.  After collecting the scattered components, I found the X terminal on the laptop reporting that the SD card had become read only.  It was still receiving and recording a video stream, although this was mostly a totally brown frame which turned to a greeny orange when I removed most of the mud.

GPS

I’ll cover the GPS first as it’s going to be short. The results using the larger amplified antenna were better, but still not good enough.  There seems to be either a lag or an offset in the computation.  The two images below show the problem.  For the next flight I’m going to use the CRIUS CN-06 V2 U-blox GPS module. Flight3-CrashSite Flight3-CricketSquare

Video 

The video recorded on the Pi (Tx) was copied to a Windows PC using WinSCP.  Then I used MP4BoxGUI to mux it into a MP4 file.  I have not had any problems using the generated file. With the video recorded on the laptop (Rx) as a transport stream, the first task was to convert it into an mp4. The following script takes the file name as an argument and creates two new videos.  This need to be run on a machine using Gstreamer version 1.2 (Check using gst-launch-1.0 –version).

#!/bin/bash
echo .
echo Remux
echo .
gst-launch-1.0 -v filesrc location=$1 ! tsdemux ! h264parse ! video/x-h264,framerate=25/1 ! mp4mux ! filesink location=$1.remux.mp4
echo .
echo Re-encode
echo .
gst-launch-1.0 -v filesrc location=$1 ! tsdemux ! h264parse ! video/x-h264,framerate=25/1 ! avdec_h264 ! videoconvert ! x264enc bitrate=2500 ! mp4mux ! filesink location=$1.reenc.mp4 

The first gstreamer pipeline (Remux), simply extracts the h264 video from the transport stream container and them multiplexes it into an mp4 container.  The h264 stream is unchanged, so there is no loss of video quality.  To make this file viewable on Windows required a couple of extra steps.  With the file copied on to the Windows PC I used MP4BoxGUI to demux and extract the h264 stream (again) and then to Mux it back into an MP4 file.  This new file was playable in Windows media Player. The second gstreamer pipeline (Re-encode), re-encodes the video as a new h264 file and multiplexes it into an mp4 container.  While re-encoding has to reduce the picture quality, it is not obvious without close inspection.  I use the re-encoded video in Sony Movie Studio 11.0, which had trouble reading the first remuxed file.  I also had to manually set the frame rate in Sony Movie Studio 11 to 25 fps as by default it was using 29.97 fps making it impossible to sync the Tx and Rx videos together.

Conclusion

The video posted on Youtube shows the Tx and Rx videos side by side.  They are fully synced and without a gap between I don’t think you can spot a join or see any difference in video quality.  Although a bit of a faff, for now I have a working pipeline to record usable video on the ground. As for the GPS, the search goes on.

Raspberry FPiV – Flight 1 (Line of Sight)

Extract

Having established from my tests with gstreamer that a low latency video stream over WiFi could be achieved from the Raspberry Pi and that a usable range could be achieved,  I decided to mount the system on an aircraft for some line of sight testing.  In order to extend the range test, I added an Adafruit Ultimate GPS Breakout Board.

The results were mixed: the video stream remained consistent throughout the flight although with higher latency than expected and the GPS accuracy was poor.   The reasons for the problems have been identified and fixed, so the overall results are still promising.

Computing Setup

To save booting from a thumb drive every time, I resized the Windows partition on my laptop and installed Lubuntu to the end of the hard disk.  This was fine for a while, but then an update to gstreamer made it impossible to install the bad and ugly plugin packages together.  In the end I gave up on Lubuntu and installed Arch Linux on both the Laptop and the RPi.  The driving force for this was that version 1.2 of gstreamer was available on both platforms.  Having never used Arch Linux before it took me most of the Christmas holidays to get everything running properly.

To save a bit of weight I replaced the Raspberry Pi model B with a model A.  The SD card can be moved between the two boards without changes.  An  Adafruit Ultimate GPS Breakout board was added to extend the range testing.  This was connected to the Pi’s UART pins.  I didn’t solder on the battery holder for the real time clock.

The board was mounted in a bright raspberry pink case, so it would be easier to find should it become detached from the plane.  I trimmed the plastic around the SD card slot so the camera cable could be run out above the card.  The camera was mounted in a PiBow holder and bolted to the case.  As the PiBow holder clips together, I glued the joints to secure everything.

Finally, I created some scripts to get everything configured and talking.

Aircraft Setup

I retrieved my old “Sorcerer” slope soaring trainer from the loft.  This design dates from 1969, although mine was built during the eighties.  It’s not pretty, but it is tough.  The radio system was a Futaba Attack 4 channel system on 35MHz and so wouldn’t interfere with the 2.4GHz WiFi signal.  As it hadn’t been used for 25-30 years, I replaced the transmitter battery, but apart from that everything worked fine.

As this was a glider, I built a power pod from birch and lite plywood to sit in place of the hatch.  Having the motor up front put the centre of gravity in the right place without adding additional weight.  The flight battery was replaced with a 1300 mAh LiPo which powered the receiver and servos through the ESC’s Battery Eliminator Circuit (BEC). I used a propeller that had previously been used on a multirotor that was well balanced.

2014-01-10 13.28.19

The Pi was powered by a dedicated 5V 3A BEC connected to the LiPo battery’s balance connector.  Everything was held on with rubber bands.

2014-01-10 13.28.28

The Flight

The first launch turned into a powered glide.  I hadn’t allowed enough throttle for the extra weight and drag.   The second launch had it climbing and zig-zagging up the field.  The Sorcerer has a huge rudder on a fin that is little more than a post.  With the added prop wash over the rudder, turbulence from the everything strapped on top of the wing and the gusting wind, it was difficult to keep it straight and smooth.  Once some height had been achieved and the throttle reduced everything settled down. I flew for about 5 minutes doing big circuits of our local playing field, before bringing it in to land.

Flight1-ScreenSetup

Results

As I was flying without a spotter, I didn’t look at the laptop screen until the flight was finished.

After retrieving the glider, I found the stream was still running which was a good start.  I hadn’t checked the latency with the new system before the flight, so it was disappointing to find it was running noticeably higher than I had achieved in my earlier tests.  I still had the original Raspian SD card and Lubuntu Thumb drive and a quick check showed that that setup still ran with low latency.  Research suggests the higher latency is due to incompatibilities between Arch Linux and the particular Intel chipset in the laptop.  It could also be that the hard drive speed is slower than the thumb drive, especially as the linux partition is at the end of the available drive space.

The next check was the GPS track.   I used WinSCP to get the video and gpx files off the RPi.  I found a really useful website that animated the track in real-time: www.gpsanimator.com/StarTraX.   I did a video capture from the screen using Applian Replay Video Capture which I then synced with the on-board and head camera footage in Sony Movie Studio 11.  As well as producing the video, this also allowed me to get frame grabs to calculate the gps error.

The animation showed the track jumping all over the place.   If it is to be believed, most of the flight was spent on the wrong side of the trees and I was flying behind where I was standing.  When synchronized with the video it is easy to see the inaccuracy.

Here is an example.  This image shows the glider crossing left to right in front of me.  In the bottom right hand corner is a tree.

Flight1-GPS1

 

Now look at the GPS track. I’ve marked on where I was stood, the position of the tree and the observed track.  The recorded track shows aircraft moving right to left, well behind me and over a small wood. The error is about 60 metres.

Flight1-GPS3

 

An even clearer example is the landing.  The error is about 130 metres.  Seeing as the current maximum tested range is only 240 metres, this is not good.  After being stationary for a little while, the GPS eventually caught up.

 

 

 

Flight1-Landing

I’ve got a Crius CN-06 V2 U-Blox based GPS which has a much larger antenna as the picture below demonstrates.  I also have the high gain remote antenna for the Ultimate GPS breakout, which is the same size as the Crius.  Finally, I’m going to add the real time clock battery holder, which should improve the warm start times.  At least one of these should improve things. 

 

 

GPS Size

 

Conclusion

While there were issues with the latency and the GPS, as a first flight the main purpose was achieved.  The streaming from air to ground worked and remained stable throughout the flight.  There is certainly enough promise to keep the project moving forward.

 

Raspberry Pi Camera Wifi Ground Range Testing

Inspired by the results from the gstreamer latency testing, I wanted to check the usable WiFi range before getting airborne.

With my portable power setup, I took a trip to our local playing field one lunchtime and set the Raspberry Pi on the roof of my car in the car park pointing it towards the far end.

2013-11-03 11.58.21

Everything was switched on, the WiFi link was established to the laptop and the stream started.  I used the Strava app on my phone to record the GPS track. Finally I started walking up the field monitoring the stream to check for drop-out.  At the far end of the field, I still had a solid stream.  I could have gone further, but the Raspberry Pi would have been out of sight where anyone passing could have just walked off with it.

Feeding the GPS output into Google Earth gave a distance of 212 metres (232 yards).

GroundRangeTest

Based on this encouraging result, the next step is to load the Pi+Camera onto an aircraft.

Raspberry Pi Camera Latency Testing – Part 2

Netcat v Gstreamer

Extract

After a number of tests I have established that the Raspberry Pi is able to provide a low latency HD stream over a wifi connection.  There are limitation, but the results have been encouraging enough for me to push from desk to airborne testing.

Introduction

Back in May 2013 when the RPi camera appeared I did some latency testing to see if the platform might be suitable for FPV duties.  Using the default NetCat based solution the minimum latency I could achieve was 300ms and the stream wasn’t particularly smooth, so the conclusion was no.

Then a few couple of months later I was asked to try running the tests again using gstreamer instead of netcat.  It’s taken longer than I had hoped, but I scrapped my first set of results and re-ran them with a more structured methodology.  As the results were encouraging, I expanded the tests to see how far it would go.

In fact the tests were so encouraging I kept extending and extending and never got around to posting the results.  Now with the UK locked in perpetual rain and wind, I though I’s take the time to share the results.

Hardware Components.

For the tests I used:

  • Raspberry Pi model B
  • Raspberry Pi Camera.
  • Edimax EW-7612UAn V2 USB WiFi Adapter (with high gain antenna).
  • MSI Wind U100 Single Core 1.6 GHz Atom Netbook, Circa 2008
  • DELL XPS 17, Quad Core i7-2630QM 2.0GHz Laptop, Circa 2012

Raspberry Pi System Setup.

Initial setup was done with the Pi connected via LAN using the then current build of Raspbian.  The first tasks were to expand the storage to fill the SD card, enable the Camera, enable ssh and update the system using apt-get.

I followed Gbaman’s instructions on setting up gstreamer on the Pi, although I used Lubuntu as the client instead of mac os.  Lubuntu was run from a USB stick so I could move it between laptops to compare performance. This was the system I used for the first set of tests which was later discarded as I felt the system could be improved.

I wanted the Pi to act as a WiFi access point.  This removes the need for a WiFi base station without having to use an ad-hoc network. I followed Adafruit’s instructions to setup hostapd, but not isc-dhcp-server or NAT translation. For dhcp I setup dnsmasq based on the first half of Adafruit’s Ad blocking access point setup,  stop at “dig adafruit.com”.  With this setup I could connect the laptop to the Pi’s WiFi and login over ssh.  Apart from plugging in the power, no other connections to the Pi are required.

I created a script on the Pi to start the video stream and another on the laptop to receive it.  The Pi camera was pointed at the laptop screen so multiple iterations could be recorded and the whole test was recorded on a GoPro at 120 frames per second.  For some of the early tests a screen was connected to the Pi’s hdmi output to show the local preview.

Test 1

The first tests started where the first video finished at 640 x 480 pixel resolution and 0.7Mbit/s bitrate run from the RPi desktop with local preview.  The gstreamer feed was about three times faster than netcat.  The gstreamer feed was also more consistent.

GS2

Netcat : 505 ms ±50%, Gstreamer : 167 ms ± 17%

This sub 200ms latency for gstreamer is probably a usable value for FPV, so initial results were promising.

Test 2

For the second test gstreamer was started from the Pi’s command prompt instead of the desktop

NetCat : 514 ms 60%,    Gstreamer 174ms 30%

NetCat : 514 ms ±64%, Gstreamer 174ms ±30%

The results were actually slightly slower for both systems but gstreamer was still under the 200ms mark.

Netcat testing was dropped at this point as it was never going to compete with gstreamer.

Tests 3 & 4

 The next tests were run with the Pi preview removed as this would never be needed in a FPV system. The tests were run for 640 x 480 x 0.7 Mbit/s, 720 x 480 x 1 Mbit/s and 1280 x 720 x 2.5 Mbit/s streams. I also ran another test at the native screen size of the Netbook 1024 x 600 x 1.6 Mbit/s, but this is not shown on the video or graph.

Test 3 & 4


640 x 480 x 0.7 Mbit/s 186ms ± 18%
720 x 480 x 1.0 Mbit/s 169ms ± 20% (Ignoring initial long iteration)
1024 x 600 x 1.6 Mbit/s 163ms ± 5%
1280 x 720 x 2.5 Mbit/s 194ms ± 24%

The averaged results are strange because for the first three results the latency comes down as the resolution and bit rate go up.  The difference is still withing the measurement accuracy so is probable that the latency for the lower resolutions are actually all in the same ballpark.  Only once we hit the lower HD resolution of 1280 x 720 x 2.5 Mbit/s does the latency go back up, suggesting that a saturation point has been reached somewhere in the pipeline.

Tests 5 & 6

These tests were run to compare results between having a Single core 1.6 Ghz Atom Netbook and a Quad Core i7-2630QM 2.0GHz Laptop at the receiving end. The tests were run with 640 x 480 x 0.7 Mbit/s and 1280 x 720 x 2.5 Mbit/s streams

640 x 480 x 0.7 Mbit/s    Netbook     186ms ± 18% Laptop       154ms ± 30% 1280 x 720 x 2.5M bit/s  Netbook     194ms ± 24% Laptop       171ms ± 17%

640 x 480 x 0.7M bit/s Netbook 186ms ± 18%
640 x 480 x 0.7 Mbit/s Laptop 154ms ± 30%
1280 x 720 x 2.5 Mbit/s Netbook 194ms ± 24%
1280 x 720 x 2.5 Mbit/s Laptop 171ms ± 17%

On the quad core laptop the 1280 x 720 stream has dropped back down to the 170ms range and the 640 x 480 stream has dropped to almost 150ms.

Test 7

I re-ran the test again at the full HD resolution of 1920 x 1080 x 6 Mbit/s.  This worked OK on the laptop although the latency rose to 233ms ± 27%. This may be usable on a big powered glider or attitude stabilised multicopter, but is pushing it for rate or acro mode flying.

The netbook took almost 9 seconds (8982ms ± 2%) for one iteration.  If you want to run full HD you are going to need a good laptop at the receiving end.

Test 8

Now I had a usable stream, the next requirement is to record it.  On the Raspberry Pi we are sending the camera output to gstreamer through a stdout/stdin pipe.  This makes it easy to to capture the stream to a file using the tee command.  The *nix tee command copies the stdin stream to stdout and saves a copy to a specified file. This allows a copy of the video stream to be saved to the SD card.

raspivid <options> | tee file.h264 | gstreamer <options>

tee

The results showed that the tee command added nothing to the latency.  This means the stream can always be recorded at the sending end without worrying about missed frames due to WiFi dropout.

Recording the stream at the receiving end is a much harder problem.  Without writing a dedicated gstreamer application I can’t see an easy way to save the stream to a valid mp4 file.  I’ve got a workaround that allows me to capture the stream using gst-launch-1.0 and mux it into a usable file, but at the moment, it requires multiple steps across Linux and Windows.   I’ll document that in a later post.

Summary

  • Best SD latency (640 x 480): 154ms
  • Best HD latency (1280 x 720): 171ms
  • Best Full HD latency (1920 x 1080): 233ms
  • Saving a copy of the stream on the RPi using the tee command does not affect the latency.

Conclusions

Based on these results I believe the Raspberry Pi with camera could be used as an HD FPV platform in conjunction with a reasonably good laptop, although I would limit it to 720p HD and not 1080p.  Because the system uses 2.4Ghz Wifi consideration needs to be given to interference from other WiFi networks and 2.4Ghz RC transmitters.  This could be solved using 5Ghz WiFi dongle provided a USB dongle can be found that’s compatible with the Raspberry Pi and features a high gain antenna.

A second consideration is range.  In a house WiFi range can be as little as 10m due to various walls.  However, in free air the range can be a lot more.  Testing is required to establish the actual usable range.

During the tests 1 & 2 I recorded the time to display the preview on the RPi.  The latency for this was typically 86 ms.  As this preview was shown before the stream was compressed it can be assumed that at least half of the latency comes in reading and encoding the stream in the Raspberry Pi.  Any optimization that could be made here would have a significant benefit.

The Next Step

I’ve got an old slope soarer that uses a 35Mz RC system which won’t interfere with the 2.4GHz Wifi.  I’ve made a power pod for it and will start some some line of sight range testing with the Raspberry Pi banded on top of the wing.  I’ve soldered up some cables to connect a Hobbywing 3A BEC to a USB Micro B plug to feed the required 5V to the Raspberry Pi.

Raspberry Pi Camera Latency Testing – Part 1

When the Raspberry Pi Camera was first released I ran some latency testing with the netcat based pipeline to see if it was suitable for FPV use.
The initial results at full HD resolution were not promising as the following video shows.

This is a test of the Raspberry Pi camera streaming over 802.11g WiFi to a netbook running Windows 7.

The stream is 1920×1080 pixels with at the default data rate. The large screen is the HDMI feed directly from the Raspberry Pi. The small screen is the netbook showing the received stream.

The stream was created and received using the inbuilt commands given at www.designspark.com and recorded at 29.97 frames/s.

By analyzing the video in Movie Maker the latency between real life and the big screen was 0.2 to 0.3 seconds. The latency from real life to the netbook as around 4.5 seconds. This is obviously too high for remotely piloting a vehicle or even aiming a bigger camera. The next step is to reduce the camera resolution and bitrate to try to improve this.

After gradually reducing the picture quality a bit at a time I arrived at a 640×480 pixel resolution with a 1 MBits/s data rate.  There was no noticeable latency between real life and the big screen (Raspberry Pi preview).

The Movie Maker analysis shows the latency averaging around 0.3 seconds, which is still probably too high for remotely piloting a vehicle although could be used for aiming a second camera such as an SLR. However it may be possible to reduce this using dedicated code instead of the piped the output of multiple commands.

Orginially posted on Youtube on 21 May 2013.