Saturday, March 15, 2014

Apple TV and tidying up my media library

So the big news chez Gekko is the recent purchase of an Apple TV 3. Precipitated by the timely demise of my 7 year-old iMac, this latest homage to iFanboy-ism was conceived with the intent of streamlining my home entertainment infrastructure and just generally reducing the number of cables within reach of Gekko Jr.

So far, it's working out very well. For the most part, as per Apple's catch cry, everything "just works". AirPlay works really well from computer or iPad (except if you try to stream HD video while mirroring your computer screen --- but of course that's a silly thing to try to do anyway).

One caveat is that everything "just works" a little better if you are able to manage all your media content in iTunes, since you can access your iTunes library remotely (thus avoiding the need to use AirPlay explicitly).

And one problem with this is that iTunes only has truck with a very limited number of video (mp4 or m4v) and audio (mp3,aac,m4a) formats, so if you happen to have a rather sizeable collection of, um, "home" videos in other formats (such as avi, mkv), it isn't at all obvious how you might shoehorn these into the iTunes ecosystem.

To Gekko of course, this presented an irresistibly geeky challenge, one whose chief insights will now be shared.

Converting file formats

Firstly, a fortunate turn of events is that many of the more recently disseminated "home" video content is in mp4 format (where it used to be avi), and this works without problems. So going forward, all is well, and our problems lie primarily with the content previously amassed.

The mkv ("Matroska") format, meanwhile, doesn't work directly, but generally uses the same "codecs" as mp4 (x264 and whatnot) under the hood, so it is a relatively inexpensive task to convert from this to mp4 (a process referred to as "remuxing" in geek speak). There is a very nice (free) command-line tool for this called "ffmpeg". A straightforward remuxing is handled by the following:

> ffmpeg -i Forbrydelsen.S01E01.mkv -acodec copy -vcodec copy Forbrydelsen.S01E01.mp4

The -acodec and -vcodec options are set to "copy" which means they are just being copied without any re-encoding into the new mp4 container.

In some irritating cases, I've found simple remuxing leads to a file with perfect video but no audio. In this case, you want to copy the video, but re-encode the audio. I didn't want to get too bogged down in the quagmire of different options for audio encoding, but the following seems to work just fine:

> ffmpeg -i Transformers.Prime.S01E01.mkv -c:v copy -c:a aac -strict -2 Transformers.Prime.S01E01.mp4

There are some other cases where the audio seems to be in 5.1 surround sound which results in weird echos when played through my stereo output. Again, an audio re-encoding does the trick:

> ffmpeg -i filename.mp4 -c:v copy -c:a aac -strict experimental -b:a 128k -ac 2 -scodec copy new_filename.mp4

Okay, so what about the crappy avi files which unfortunately comprise roughly 99% of my "home" video collection? Well, since leaving well enough alone obviously wasn't an option, it seems that here there was no choice but to roll up my sleeves and re-encode the video. And for this task I found the (free) program Handbrake to be just the thing. The gui version works well but is a bit clunky at first (hint: the "File>Add all titles to queue..." menu item comes in very handy). I've found the command-line version to be much more useful for en masse re-encoding. Example:

> HandbrakeCLI -i 'Blackadder.S00E03.avi' -o 'Blackadder.S00E03.m4v' --preset="AppleTV 3"

Note the handy preset here which seems to give pretty good quality. The resulting mp4 file is iTunes-ready.

As an aside, Handbrake is an excellent tool for importing DVDs in m4v format, although I find myself needing to do this increasing rarely.

I, Subtitle

I have found subtitles to be a perennial bugbear with downloaded videos. Sometimes they're hardcoded, which makes life easy (unless they're in swedish); other times they come in the form of an .srt file or (uselessly) as a dvd-style .vobsub file. As you'd expect, .srt files are no use when dealing with iTunes, and the only way I've to get subtitles working is to use a little app called "iSubtitle". All you need to do is load up your mp4 file, add the appropriate .srt file as a subtitle track using the toolbar/tray on the right, and then save. No re-encoding necessary. Once that's done the subtitles should be available via the controller in iTunes. It's a hassle but the end result is nice because you can switch subtitles on and off.

Managing iTunes library
One issue that I've found problematic with iTunes is managing the physical storage of the music and video files. By default, iTunes wants to create its own copy of every file that you import into it. This means that (1) you have to constantly go back and delete the file you just imported (unless you want to keep duplicate copies); and (2) you have to trust iTunes to manage your files for you.

The latter became a real problem for me when my collection became too large for the local hard-drive and I had to shift to an external drive. For example, if the external drive happened to be disconnected for whatever reason when I imported something, it would go ahead and copy the files to an alternate library on the local hard drive. If left unchecked,this kind of thing could easily cause headaches down the track when changing machine or whatever.

So I've decided to disable the copying on import option (by unchecking "Preferences>Advanced>Copy files to iTunes Media folder when adding to library"), and just manage my files myself. One advantage of this is that I can maintain files on different drives, which allows me to keep music and video that I frequently use on the local drive (whilst relegating more esoteric "I might listen to that one day" content to an external drive).

One cool thing too is that, provided you keep the files on the same drive, you can go ahead and rename them or move to different folders, and iTunes will still know where they are.

Metadata

This is a slightly pedantic issue, but one cool thing about Apple TV (and other media solutions such as XBMC) is that you can retrieve "metadata" about your media files which is then displayed all nicely on the Apple TV.

For my music files I've always managed the metadata manually and, although I've kept things pretty tidy, I've never obsessed too much about it. I admit I do generally make the effort to add the album cover (particularly if it's an Iron Maiden album); but beyond that, really only what's needed to keep things sorted.

For video content, however, I wanted to tap into the rich meta-databases of thetvdb.com and themoviedb.com. There seemed to be several options out there for doing this. None of them, however, were free, so it was only after much research and some experimentation that I settled on "iFlicks 2", a nice little app from the Mac App Store that does a great job. At $25.99 it's a little pricey, but after wasting $3 on the awful "MetaTV" app I can attest you get what you pay for.

One nice thing about this approach is that the metadata is stored in the file, and not away in an application-specific database somewhere (as with XBMC, say). ITunes reads this metadata on the fly, and it will also make it easier if I decide to move away from iTunes in the future.

Command-line power
I mentioned above the command-line tools ffmpeg and HandrakeCLI, which I use on the Mac OSX Terminal app. The great thing about the command-line is that it makes things very scalable when you're dealing with more than a couple of files.

And along the way I've learned a couple of neat Bash tricks that come into play when working with multiple files. The first is "for" loops, which allow the same command to be applied to multiple files. The second is Bash's text replacement syntax, which is an easy way to change file extensions. This example uses both:

> for i in *.mkv; do ffmpeg -i "$i" -acodec copy -vcodec copy "${i/.mkv/.mp4}"; done

"Arse sinking"

One final hack worth a mention pertains to the function "rsync", which is a standard unix command-line utility for backing up filesystems. By default, it works by comparing the files present in source and target directories; whenever they are different and the version in source is newer it will replace the version in target.
I have used rsync for my backup needs for a long time, but in the course of this project I found myself wanting to rsync between two locations where, for whatever reason, the files in the target directory were newer than the ones in the source. (Look, it probably shouldn't happen in a perfect world, but I think I renamed some directories or something and suddenly it wanted to recopy everything.) So a useful trick to know is the --ignore-existing option, which basically does nothing when there is a file of the same name in target (regardless of its size or age). As I say, this probably isn't something I should need in the course of a well-ordered backup operation, but it happened to be very useful when I did!

> rsync -av --ignore-existing TV/ /Volumes/Gekko_2TB/ExternalData/Video/TV

Future proof

So to summarize, I'm pretty chuffed with Apple TV so far. But that's not to say I want to blindly hitch my cart to the Apple bandwagon going forward. For that reason, I think that managing music and video files manually, and storing metadata locally, is the way to go. It means that should iTunes stop being a good option down the track, it won't be too hard to simply take my business, and files, elsewhere. And in the meantime, it makes juggling files across multiple hard drives a lot more manageable.