Vodpod Geek Blog

My name’s Kyle, and one of my projects at Vodpod has been building an Android app for Vodpod.com. I’d like to talk briefly about what sort of challenges we had to solve in making the app, and hopefully give other developers a leg up on the process.

Architecture

The app comprises a constellation of activities, including a home-screen launcher, two services, classes representing each type of data provided by the API (Video, User, Collection, etc.), a singleton client for interacting with the API, an SQLite-backed content provider, and a few helper classes for formatting.

Most of the activities are straightforward: they load an XML view, pull data from the API in an AsyncTask, and when the network request is done, populate the view with the appropriate information. For our list views, we use the CWAC Thumbnail library to load thumbnails on demand.

The largest service handles several tasks, including synchronizing some preference data and uploading files to the service. We use the Apache Commons http client with mime4j and multipart for all requests.

The second, smaller service is purely for feed synchronization. It downloads new feed items and populates our content provider, issuing notifications as necessary.

The feed view is different than most of our network-backed views. It links directly to our feed content provider, which lets us store tens of thousands of feed items and scroll through them at incredible speed.

Finally, all the network requests share common paths, serialization, and response validation code. A singleton client class handles authentication and provides nicely typed methods for retrieving lists of videos, feed items, etc. Each NewsFeedItem, Video, Comment, and so forth are responsible for populating themselves recursively from a JSONObject. They also persist themselves to ContentValues objects for serialization. Instantiating extra objects increases parse time, but simplifies error handling, data lookup, and cuts down on memory footprint.

Storage

The Vodpod data model is a deeply nested directed graph of objects, which doesn’t lend itself well to serialization in row-column oriented datastores. We investigated using db4o as an object store.

Db4o is a fantastic database with excellent capabilities, but there are caveats. First, even the base db4o jar increases build and install times by several seconds, which lengthens development cycles. Second, Dalvik doesn’t always play nicely with the db4o serializer on third-party classes. For example, the Joda DateTime class can’t be deserialized correctly. Your only choices are extensive subclassing (which involves awkward casts on basically every operation) or implementing custom serializers.

Eventually, we abandoned Db4o in favor of an SQLite-backed content provider. The Android API is designed to support this storage model out of the box, and it’s extremely fast. The drawback, of course, is that our nested data needs to be denormalized into a table. We pre-render as much content as possible to save processing during display. For example, we colorize usernames and generate appropriate titles for each feed item before storage.

Latency is Key

We originally wanted to deliver much of vodpod.com as a mobile-optimized HTML5 web app. That approach would have let us share the vodpod experience across all platforms, instead of separate development tracks for iPhone and Android. We built a pared-down version of the video page for mobile browsers, and made some interesting discoveries: high latency kills the experience.

When you’re on your phone, waiting two to three seconds for a page to load over wifi is obnoxious. It breaks the applications “flow” and makes interacting with the app more like work. Over a 3g network the effect is stronger. Near-instant responses dramatically improve the subjective feeling of the application.

We tried to cut out as much fat as possible from the video page, but were still not able to offer a reasonable load time. In contrast, our mobile apps use a combination of caching and efficient API requests to offer pages in under 50 milliseconds, with videos loading in 0.5 to 4 seconds depending on payload. The performance difference (along with the advanced lifecycle management offered by the android platform) is a good reason to spend the time writing a native app.

Optimization

Declare everything you can final. What isn’t final you should be re-using. Try to avoid object instantiation overhead, since you pay a price in creation *and* in GC sweeps. Android provides plenty of opportunities to re-use objects: for example, in ListViews, views that scroll off the top of the screen can be re-used for the incoming items at the bottom.

Cache everything. Don’t create a new handler subclass for each button; use a single handler and store the data to be acted on in a tag. And recognize that *how* the user interacts with the app influences performance more than raw algorithmic complexity: it can be more efficient to parse and create a new intent when a button is pressed than to precompute and store that intent ahead of time.

Locking and thread management in Java are nontrivial. Inevitably you’ll find yourself accepting that a probabilistically successful implementation is significantly easier than reasoning about a correct one. With limited time, this tradeoff can be worthwhile.

Don’t sweat the small stuff: when one of your requests involves 100ms of network time, an extra 20ms of parsing doesn’t really matter. But do it in the background. Progressive enhancement is worth it: grab the basic metadata first, then the images, then the video. Above all, let the UI remain responsive.

Accept that efficent code is more complex. Managing a service, multiple threads, observer callbacks, DB transactions, and view refreshing means more time spent debugging and implementing simple features. It’s only worth doing for cases where performance (or in our case, offline storage) is critical.

Prioritizing

We’re only six people total, and provide content to something like 10 million unique visitors per month. I’m the sole Android and API developer, and about half of my time goes to building the app.

It *is* possible to build a functional app on limited dev time. But that means focusing carefully on getting the most out of every hour of development. We’ve left graphical glitches, stutters, crashes, and obtuse error messages unfixed simply because it takes so much time to track them down.

Instead, we’ve been focusing on implementing new features and critical fixes. I’m normally a perfectionist about this stuff, but it’s better, I think, to offer a feature which is broken 10% of the time than to not have it available at all.

Release strategy

I’ve been working with Lighthouse on this project, which helps a lot with planning improvements. We started out with monolithic releases including a huge set of of changes. Lately, though, I’ve been grouping our to-do items into clusters of five or ten issues around a specific feature, and trying to complete a release in a week to 10 days.

For instance, our next release includes numerous adjustments to the video page layout (margins, colors, and thumbnail sizes), a link to the original video page, and some new features such as user profiles and following. It also includes a couple of small bugfixes that I knew were easily solved. Trickier bugs, like “Uploader locks up in HTTPClient during intermittent 3g connectivity on large uploads” are deferred because they are so difficult to reproduce and analyze.

Android makes it easy to iterate rapidly–it takes only 10 minutes to go from finished code to release in the market. I’ve found that releasing small sets of changes more frequently makes it easier to set deadlines and stay motivated.

Resources for developers

The Android SDK docs are pretty darn good, but there are still undocumented corners. To fill in the gaps, I refer often to the included API demos in android/platforms/*/samples/ApiDemos. #android-dev on Freenode can be hit-or-miss, but the folks there tend to have that elusive best-practices knowledge. “Should I rebind my service in onResume or onStart?” They know.

Finally, StackOverflow has a wealth of Android Q&A. Unfortunately, finding one that pertains to your particular problem can be difficult.

Vodpod for Android is now available in the Android Market.

We’ve finally recovered from the thumbnail overload. You should be seeing fewer of those missing images on the site, now.

High S3 error rates and lots of incoming videos are overwhelming our thumbnails again. We’re bringing up additional workers and a more efficient dispatcher now.

We’ve recovered from the backlog. It may take a few minutes for thumbnails to pass through the CDN.

We’ve been receiving an unusually high number of new videos this morning, and our thumbnailer is having trouble keeping up. It may take a couple hours before thumbnails appear for new videos.

We’ve added support for OAuth to our twitter autopost system; you won’t have to update your twitter passwords on vodpod.com again. We’ve also introduced a new friend finder using oauth, which can suggest people you follow on twitter who are also on vodpod. That’ll be coming as a part of our new account pages, sometime in the next month.

If you’re building a web site that interacts with vodpod.com, you’ve probably wanted to let your users comment on those videos. We’ve added that feature today.

Make a post request to http://api.vodpod.com/my/collections//videos//collection_comments/new, and pass it your API/auth keys, ‘text’, ‘user[email]‘, and ‘user[name]‘. You can retrieve those comments from the collection_comments path later.

You can read more at the API documentation.

I’ve put together a basic transcoding service which lets us accept videos from the Nexus One and other weird codecs. I also removed the API restriction for those codecs, so you should be able to upload video from the N1 and friends again.  It’ll take a few seconds (typically less than 30) for your videos to be playable after upload. Go try out the Android app and let us know what you think.

In related news, the thumbnailer can now thumbnail your uploads directly: high-quality thumbnails should be present almost instantly.

I just released version 0.2 of our android app, which lets you upload videos to vodpod.com from your phone. This version retries on network errors, continues uploading after you lock the device, and provides an upload progress bar.

Next up, I’ll be improving the error handling–offering a message after server failures and checking form fields before upload. Further ahead, we’ll be introducing collection views, and the ability to watch some videos from vodpod directly on your phone. We’re also sketching out mobile versions of some pages on vodpod.com.

When we started this project, we presumed that all the Android and iPhone clients would hand us web-ready mpeg-4, h264 videos. Unfortunately, it looks like some phones–for instance, the Nexus One, use less common encodings, like the patent-encumbered AMR codec. Hence, videos from those phones can’t be viewed in flash (or, for that matter, on most desktops). We’re building a transcoding system to convert them to more standard codecs. When it’s done, any videos you’ve uploaded from the Nexus One (or other broken phones) should be viewable.