Tonight, I stumbled on an interesting post on Reddit. It linked to two Nest Cams livestreaming the landfall of hurricane Irma from a Miami condo. I popped the streams open on my phone (the hurricane had not hit yet, and the sky was mostly clear) and thought about the fact that I was probably going to fall asleep before the power went out and killed the stream.
Being an incredibly good-looking and overconfident dweeb, I then thought about the fact that there’s got to be a good way to rip and save a Nest Cam livestream so that I could watch it tomorrow, maybe post a time-lapse, and gain all of the Karma. After all, its already been done with tons of other streams from various sites. It seemed like a decent Friday night hackathon in the making, and would at least solve the problem of me falling asleep lest I fail.
I decided I wanted something that was (1) automatic, and (2) running in the background so that I didn’t need to stay up all night or keep my computer on. I ran through the list of options in my head:
- Just stream them in a browser window and use screen capture (lame and fails both #1 and #2),
- Use one of the millions of Chrome or Firefox plugins that allows for saving of streams (extra lame and still fails #2),
- Use some sort of stream-ripping software built for Linux so I could load it on my always-running Pi (not lame, but impossible for me to find something that worked after looking for an hour or so), or
- Hack it and do it myself.
If you haven’t guessed already, I went with #4. I’m going to show you how I figured it out and how to do it yourself. This assumes some basic knowledge of Linux command line and shell scripting.
First thing first, I loaded one of the Nest Cam streams using the links provided on Reddit. The video livestream itself sits inside a Nest-branded HTML page that does this really annoying thing where it auto-pauses and pops over a Nest advertisement every once in a while. If I wasn’t going to rip it out already, I would have been sufficiently annoyed by this to figure out how to get to the base stream.
I poked around inside the page source using the Safari dev tools to see if I could find any obvious stream container or link, but didn’t see anything. I did find a more minimal stream that is formatted for Twitter but it still does the popover thing. Boo. I also poked around in the javascript (warning: there’s a lot) to see if the stream was being lazy-fetched from any obvious source. Again, nothing. Boo.
I decided to use the Timelines tool to see what’s being loaded on the network. I recorded for a few seconds and saw what was clearly a periodic fetch taking place. There’s an XHR request going out approximately every 4 seconds. It’s loading a media_xxxxxxxx_123.ts file and a chunklist_xxxxxxxx.m3u8 file after each request. This is definitely an MPEG-2 stream, with the chunklist serving as a manifest for the media.ts file. Bingo!
.m3u8 files are commonly used to define video streams, and so I knew I was on the right track. Right-clicking on the m3u8 file and choosing “Copy Link Address” and pasting it into the Safari address bar yielded a base-level video stream with no extra junk (*cough*) on top of it. It looks like Nest streams their livestream content from stream-bravo.dropcam.com or from stream-delta.dropcam.com. (Both are currently using Wowza Streaming Engine 4 Subscription Edition 4.7.1 build20635)
The next step was saving the stream using this URL. Time to break out the Pi! I figured I could use ffmpeg to do this, and after a quick google search, my assumptions were confirmed. This StackOverflow question gave me what I needed, except I wanted to ensure that the ffmpeg command was always running (in the event the stream broke up and was restarted, a network issue occurred, etc).
For those of you who just want to save a Nest Cam stream to disk using Raspbian/Raspberry Pi/Debian/Other Linux, this is the command that will do it for you (you need ffmpeg installed in order to use this): ffmpeg -i http://your_stream_chunklist_link.m3u8 -c copy -bsf:a aac_adtstoasc /path/to/output/file.mp4. For example, this is the command I used to save the stream I was watching to my home directory: ffmpeg -i https://stream-delta.dropcam.com/nexus_aac/a8a645a10ef24a50b250c14a08b02ef9/chunklist_w719996219.m3u8 -c copy -bsf:a aac_adtstoasc Stream.mp4
In order to make sure that ffmpeg was always restarted in case of any issues, I whipped up the following shell script (named runStream.sh) to be run as a cronjob:
#!/bin/bash
#make-run.sh
#make sure a process is always running.
process=LivingRoom
now=$(date +%Y%m%d%H%M%S)
makerun="ffmpeg -i https://stream-delta.dropcam.com/nexus_aac/a8a645a10ef24a50b250c14a08b02ef9/chunklist_w719996219.m3u8 -c copy -bsf:a aac_adtstoasc /media/HDD/Stream_$now.mp4"
if ps ax | grep -v grep | grep $process > /dev/null
then
exit
else
$makerun &
fi
exit
The script checks to see if the ffmpeg command is running using ps ax and grep. If it is, there is no need to start it, so it exits. If it isn’t, the script is started using the makerun shell command. Note the $now variable at the end of the filename: it automatically appends a puncuation-less timestamp to each video file, so that the previous file is not lost when ffmpeg is automatically restarted.
The last thing to do was to make the script executable using chmod +x runStream.sh and add it to the crontab using crontab -e. I set it to run every minute (can’t miss any of the action!) using the following crontab:
# m h dom mon dow command
* * * * * /home/pi/runStream.sh
After saving the changes and waiting a minute, I saw the first video file pop up. After running for a few hours, the auto-restart was a great idea, because it’s kicked in several times (likely due to haphazard internet because there a HURRICANE).
Stay safe out there, Florida. It’s going to get crazy.
