I’ve been working on an HTML5 offline app for just over a year now. Yes, the same app. It hasn’t been dedicated time, but the development has dragged on for awhile now. Primarily this has been due to the production of data (there’s a lot of data), though many delays have been caused by the developing spec and its idiosyncrasies across browsers. So, I decided to write a multi-part series in what it’s taken to get this app out the door (which it isn’t yet, but by the time I finish this series it should be).
When all is said and done, this app should run on Windows, OS X (Mac), Linux, iOS (iPhone/iPad/iPod Touch) and Android.
This first part is just an introduction to what’s involved and will mostly be fluff. The meat of it won’t start until part 2.
Application Background
The application is a language learning app (Thai). It allows users to search in both English and Thai against a database of more than 10,000 records. Some are individual words, some are sentences, some are short phrases or other word combinations. Each record will have an associated audio file in both English and Thai that users can click to hear the word/phrase/sentence pronounced by a native speaker. It supports multiple users, and each user can mark words/phrases/sentences as Favourites for quick access. It also includes adult phrases, and so includes a content filter option to remove these phrases from the results if desired.
Additionally, this particular application has different sections for Categories, Classifiers and Grammar. I won’t necessarily go into all of the application’s functionality in this series, but I will go into any functionality that wasn’t easily achieved using old-school JS (i.e. pre-HTML5).
Learning Experiences
Application Cache (AppCache) Offline Storage Limits
Offline storage is nowhere near consistent across browsers. This particular application requires (or will when all the audio is recorded) somewhere between 150 MB – 300 MB of storage. At the time of this writing (and my testing) the following holds true:
| Browser | Application Cache (AppCache) Storage Limit |
|---|---|
| Safari Desktop (Mac & Win) | Unlimited |
| Safari Mobile (iOS) | 10 MB |
| Chrome Desktop (Mac & Win) | 5 MB * |
| Chrome Mobile (Android) | Unlimited ** |
| Firefox 4 Beta | Unlimited (with user prompt) |
| IE | No idea. It sucks. *** |
* Chrome only allows a limited amount of storage UNLESS you jump through some hoops. More on that later.
** Android appears to allow unlimited storage, at least with the 2.3 API/emulator. However it continually errored out after 1092 files when I had more than that in a particular directory.
*** IE just plain sucks. I have no idea how much storage it allows because it loads my JS files out of order it doesn’t load some of my JS files because they have type=”application/x-javascript”, so pretty much everything breaks. I still hate IE.
Just think about this for a second. Safari’s desktop browser offers unlimited storage, but their mobile doesn’t. Chrome is the opposite – their desktop browser limits storage, but their mobile browser offers unlimited. What a PITA.
So, with > 150 MB to be deployed I ran into a bit of a conundrum. The app was originally designed for Safari only, but when Chrome and Firefox started showing promise, the sponsor wanted to extend the scope. Well, Firefox (beta right now, but I’m sure it’ll be release soon) and Safari are no problem. But Chrome bit me in the ass when I started testing with extreme amounts of data. Also, what about Safari Mobile? Didn’t Steve Jobs, for 2 versions of the iPhone, refuse to allow native apps, saying that the web was where it’s at? Well, if that’s the case, why tie my hands and limit me to 10 MB of local storage?? Oh ya, it’s because he released the App Store so now I have to pay and go through a Draconian authorization process if I want more than 10 MB of storage. Nice.
Templating (or basic MVC)
Since the original application was only targeting the desktop, I did what most people do with JavaScript and mixed presentation with logic. I tried to keep them mostly separated, but inevitably I ended up returning formatted HTML strings in functions where it might be better to simply return an array or an object. This is especially true now that I’m dealing with at least two different UIs for the same application – desktop and mobile. And that might grow again if separate, native-feeling iOS and Android versions are required.
If I were to start over, I’d probably use something like PURE. Hell, when I started this project I didn’t even realize that JS templating frameworks existed. Now, however, it’s a little too late to dig into this (next time). I will however detail the road that I’ve started down in order to separate presentation from logic.
In part 2 I’ll go into laying the foundation, including a few things that I didn’t do but should have (and now have to revisit because I didn’t). Stay tuned.
Is the AppCache storage limit for one html-file? Or for the whole domain? Or the whole browser?
If I have two entry files (two html-files), do they both get the limit of say 10 MB on Safari Mobile? If that was true, would it not be possible to split the app over two files?
I’d never even considered splitting it up, likely because my app is built around a single HTML file with all functionality/navigation handled by JavaScript. But I just did some testing on the iOS Simulator and it appears that it is a per-manifest limit. So if you have 2 separate HTML files with 2 separate manifests, sitting in the same directory, they are considered separate applications, and the 10MB limit applies to each separately.
It also appears that the two “applications” (HTML docs with unique manifests) have access to each other’s cache if located on the same server. I just cached 4 images in 1.html, took my server offline, then accessed them from 2.html, which didn’t include them in its manifest.
The downside to this approach is that it causes a disconnect/continuity issue in the app, especially on iOS where most apps are simply one HTML document with navigation handled via JavaScript. However this might be a small price to pay to work around this restriction.
Hi Garth,
in your test are you sure that 2.html took the images from the application cache and not the normal browser cache?
I tried to reproduce this on the iPad.
In my test I have two html-files with two different manifests.
When I went offline, the two html-files could only access the files that were cached with their respective manifest.
Thank you for your Tests!
We might take that route with one of our apps. We need about 30 MB to be cached.
It will need some experiments to find the best way to handle the navigation part.
If we really use this technique in our final version, I will post a link here.
No problem. Note however that I only did this test on iOS. Other browsers may behave differently, so you’ll have to test the gamut.
This project sounds really interesting. I’ve been looking all over for some kind of framework to do offline “full text” searching. How do you plan on approaching this?
I essentially wrote my own algorithm as I’m not really doing “full text” searching, but rather full text searching within individual records. So I’m effectively looping through every record and using JavaScript’s string .search() method with a regex. Though if I were to do it over again I’d probably use .indexOf() on the array in some form or another, as this would likely be faster (assuming the browser supports it, which I believe all modern browsers do).
Pingback: Get off(line) | Web Directions
I tried on OIS an app with 60 MB of scripts. First it asked if I want to increase the localstorage to 10 MB. After that, I had to reload to page to continue the caching process. It asked again to increase it to 25 MB (after which I reloaded) and then increase it to 50 MB. But it didn’t cache the entire app, apparently at the 50 MB it doesn’t ask to increase any more. But 50 MB is a hell of a lot better than a meagre 5 MB
just reload between the requests to increase the localstorage (will try to find something to counter this)
Pingback: Taking your web sites and apps offline with the HTML5 appcache « van der Straten
Regarding mobile safari size limit, i think tight limits on mobile web apps make sense due to the limited amount of storage space they have.
That said I think it should be an opt in, so if say the web app wan’t 150 meg to operate offline it should prompt the user to confirm this is okay. This way you get the best of both worlds, offline apps, and the apps dont chew space in the background with no checks.