Our AWS bill last month was $14,000. For a startup with twelve employees and a weather app. We're spending more on servers than on salaries (for two of the employees). Something had to change.
Then I realized: our app has 50,000 monthly active users. Each user has an iPhone with a multi-core processor, several gigabytes of storage, and a persistent internet connection. That's 50,000 compute nodes with 50,000 local data stores and a built-in networking layer. It's a distributed system. It's just not our distributed system. Yet.
"The cloud is just other people's computers. A mobile distributed database is just other people's phones. The economics are identical. The ethics are different. We've chosen not to dwell on this."
The Architecture: MDDB
The Mobile Device Distributed Database (MDDB) turns your app's install base into a distributed storage cluster. Each device stores a shard of the total dataset. Queries are routed through a coordinator (our one remaining EC2 instance) that fans out to devices, collects results, and returns the response.
From the user's perspective, the app is a weather app. It shows the weather. It also stores 2-4MB of data that has nothing to do with the weather in a background Core Data store. The user doesn't see this data. The user doesn't know about this data. The user's weather forecasts are accurate and free. Everyone wins, arguably.
// ShardManager.m — assigns data shards to devices - (void)assignShardToDevice:(NSString *)deviceID { NSUInteger shardIndex = [deviceID hash] % self.totalShards; NSDictionary *shard = self.shardMap[@(shardIndex)]; // Download shard data in the background // iOS gives us ~30 seconds of background time // Our average shard is 3.2MB. This is fine. Usually. [self downloadShard:shard toLocalStore:deviceID]; } // QueryRouter.m — fan-out queries to devices - (void)executeQuery:(MDDBQuery *)query { NSArray *relevantShards = [self shardsForQuery:query]; for (NSNumber *shardID in relevantShards) { NSArray *devices = [self onlineDevicesForShard:shardID]; // Send a silent push notification with the query // The device executes it against its local Core Data store // And sends results back via a POST [self sendSilentPush:query toDevices:devices]; } }
Silent push notifications are the RPC mechanism of MDDB. Apple designed them for content updates. We're using them as database queries. The distinction is subtle and we prefer not to clarify it in our App Store description.
Replication and Availability
Each shard is replicated across a minimum of 50 devices. This gives us a replication factor that would make Cassandra jealous. The trade-off is that our replicas are controlled by humans with unpredictable behavior:
- A user deletes the app: We lose a replica. If enough users in the same shard group uninstall simultaneously, we lose the shard entirely. This happened once during a 1-star review campaign. We lost 40GB of data because 200 users in shard group 7 uninstalled the same afternoon. Our backup strategy is "hope it doesn't happen again."
- A user turns off their phone: That replica goes offline. Peak availability is during waking hours in the Eastern US timezone. Between 2-5 AM ET, availability drops to 60% because most of our users are asleep. We've considered targeting Australian users to improve overnight coverage.
- A user's phone runs out of battery: Same as turning it off but less predictable. We've added a battery level check — if a device drops below 20%, we preemptively migrate its shard to a healthier device. This requires the app to be in the foreground, which it usually isn't, so this rarely works.
The Consistency Problem
MDDB is eventually consistent, where "eventually" means "when enough users open the app." A write propagates to the coordinator, which pushes it to all shard replicas via silent push. But silent push has no delivery guarantee. And the device has to wake the app to process it. And the app might not wake. So "eventually" can mean anywhere from 3 seconds to never.
We've accepted this. Our SLA is "the data will probably be there." In practice, 94% of queries return results within 10 seconds. The other 6% time out because every device in the target shard is asleep, dead, or has been thrown into a lake (this happened once; a user tweeted about it).
On January 12, shard 23 went completely offline for 8 hours because its primary replicas were all located on devices in Tampa, Florida, and Tampa had a power outage due to a storm. We lost access to 12,000 rows of data until FPL restored power at 6 AM. Our disaster recovery plan now includes checking the National Weather Service API before performing writes to geographically concentrated shards.
User Experience Impact
The weather app is slightly larger than a typical weather app (47MB vs the usual 15MB) due to the embedded Core Data shard. It also uses more battery than expected, because it wakes up to process query push notifications. We've received 23 App Store reviews mentioning battery drain. Our response to all of them is: "Try closing background apps." We're not proud of this.
We also send a push notification at 3 AM to devices hosting critical shards if they haven't reported in recently. The notification says "Don't forget to check tomorrow's weather!" This is technically true — there is weather tomorrow — but the real purpose is to wake the app so it can respond to pending queries.
Financials
- AWS bill before MDDB: $14,000/month
- AWS bill after MDDB: $340/month (one EC2 coordinator + APNS costs)
- Cost per query: $0.00 (the users pay for their own electricity and data plans)
- App Store rating: 3.1 stars (down from 4.4, primarily due to battery complaints)
- Active users: declining 8% month-over-month (each uninstall reduces both our user base and our database capacity, creating a feedback loop we're choosing to call "organic right-sizing")
MDDB is technically a success. We've reduced server costs by 97%. We've also reduced our user base by 34%. The math works out if you don't look at it too closely. And we don't.