🍫 CocoaPatterns — iOS & Mac Development Since 2009 RSS
🍫 CocoaPatterns
Because someone has to write this stuff down
"We ship apps. Sometimes they even work."

RegexDB: When Your Database Is Also a Parser

Why maintain a schema when a sufficiently creative regular expression can store and retrieve any data structure?

GA
Gordon Ashby
@gashby_data · CocoaPatterns

Relational databases are constrained by their schemas. Document databases are constrained by their document models. Key-value stores are constrained by their keys and values. What if I told you there's a storage system constrained by nothing except your imagination and a small set of metacharacters?

RegexDB stores all data as a single flat string. All queries are regular expressions. It is approximately as fast as you'd expect and approximately as maintainable as you're imagining.

"Some people, when confronted with a problem, think 'I know, I'll use regular expressions.' Now they have two problems, and one of them is a database."

The Data Model

RegexDB uses a proprietary serialization format we call RSSV (Regex-Separated Structured Values). All records are stored inline in a master string, delimited by a sequence we're confident won't appear in user data: |||~|||

MASTER_STRING =
"user:1:Brad:brad@example.com:2014-01-01|||~|||
user:2:Melissa:mel@example.com:2014-03-15|||~|||
order:1001:user:1:total:49.99:status:shipped|||~|||
user:3:O'Brien:obrien@exam|ple.com:2014-08-22|||~|||"
⚠️ Known Issue

User 3 has a pipe character in their email address, which interacts badly with our delimiter. User 3's records are currently "in an undefined state." We have advised User 3 to change their email address. They have declined. We are at an impasse.

Query Syntax

# Fetch all users
user:(d+):([^:]+):([^:]+):([^|]+)

# Fetch all shipped orders over $40 (in development, ETA unclear) order:(d+):user:(d+):total:(4[0-9].d{2}|[5-9]d.d{2}):status:shipped

Performance Characteristics

  • Write speed: O(1) — we append to the string
  • Read speed: O(n × m) where n is string length and m is regex complexity
  • After 90 days of production use, MASTER_STRING is 847MB
  • Our longest query regex: 2,847 characters. It has one capturing group we cannot explain.
  • Deletes: not yet implemented. We're calling it "immutable storage."

RegexDB is available on GitHub. It has 12 stars, 3 of which are from me on alternate accounts, and 1 issue titled "is this a joke" which I have labeled "enhancement" and closed.

3 Comments — Powered by Disqus
PL
p_lin_data · 2 weeks ago
User 3 situation is sending me. They exist in a sort of Schrödinger's row.
GA
gashby_dataAuthor · 2 weeks ago
@p_lin_data We can see User 3's data, we just can't reliably read it. They exist in a sort of Schrödinger's row.
WK
wkurtis_eng · 12 days ago
"is this a joke" labeled enhancement is peak open source maintainer energy.