VimDB: The Database You Exit When You're Done Querying

All operations are modal. Reads happen in NORMAL mode. Writes happen in INSERT mode. Deletes happen when you press dd on the wrong line.

After the qualified success of RegexDB, several readers asked if I had plans for a follow-up. "Gordon," they wrote, "you've shown us that a database can be powered by regex. What other tools can be inappropriately repurposed as storage engines?"

The answer, it turns out, was in front of me the whole time. I was writing the RegexDB post in Vim. And I thought: Vim already has buffers. Buffers hold data. Data is what databases store. I'm already inside a database. I just need to formalize it.

"Every text editor is a database if you're brave enough and your data model is 'lines of text.'"

Core Concepts

VimDB is a modal database. This is its defining innovation and its most controversial feature. In traditional databases, you can read and write in the same operation. In VimDB, you must explicitly switch modes:

NORMAL mode: Read-only queries. Navigate data with h, j, k, l. Search with /pattern.
INSERT mode: Press i to enter write mode. All mutations happen here. Press Esc to commit and return to NORMAL.
VISUAL mode: Select ranges of rows for bulk operations. This is our version of a SELECT ... WHERE clause.
COMMAND mode: Administrative operations. :w flushes to disk. :q terminates the connection. :q! terminates without saving, which we're calling a "forced rollback."

" Connect to VimDB
$ vim /var/data/vimdb/users.vdb

" Query: find all users named Brad
/Brad
" Press n to iterate through results (this is our cursor)
n
n
n

" Insert a new user
o
Brad Kowalski | brad@example.com | 2014-01-15
<Esc>

" Commit to disk
:w

" Close connection
:q

💡 Key Principle

In VimDB, the cursor is the query pointer. Your physical position in the file is your position in the result set. If you get lost, that's a you problem.

The Data Format

VimDB stores records as pipe-delimited lines in a plain text file. There is no schema. There is no header row. You are expected to remember what each column means. If you forget, that's between you and your .vimrc.

We considered adding column headers, but column headers are just data that thinks it's special, and VimDB doesn't support class hierarchies.

Transactions and ACID Compliance

VimDB provides a unique transaction model based on Vim's built-in undo tree:

Atomicity: Each keystroke is atomic. A multi-character insert is a sequence of atomic operations. If your finger slips, only the slipped character is wrong. The rest is fine.
Consistency: The file is always valid UTF-8. That's our consistency guarantee. That's the whole thing.
Isolation: If two people open the same .vdb file, Vim shows a swap file warning. This is our lock manager. First person to press (E)dit wins. Concurrency resolved.
Durability: :w writes to disk. If you forget to :w, your data is gone. We consider this a feature. It teaches discipline.

⚠️ Known Issue

One of our beta testers pressed dd (delete line) instead of jj (move down). They deleted a production user record. We told them to press u to undo. They had already pressed :w. There is no u after :w. We're calling this "eventual consistency with user error."

Advanced Features

Macros as Stored Procedures

Vim macros (q{register}) serve as VimDB's stored procedure system. Record a sequence of operations, save it to a register, and replay it with @{register}. Our most complex stored procedure — a user deduplication routine — is stored in register z and is 847 keystrokes long. Nobody remembers what it does. We just run @z and check if anything broke.

Replication

VimDB supports replication via scp. Every 30 seconds, a cron job copies the .vdb file to a backup server. If both the primary and the backup are being edited simultaneously, the last scp wins. We're calling this "last-write-wins replication" and several people have told us this is a real thing in distributed systems, which makes us feel better.

Benchmarks

Single-row lookup: 0.003ms (if you know the line number), indeterminate (if you don't)
Full table scan: depends on how fast you can press n
Write throughput: limited by typing speed (our fastest DBA writes at 140 WPM)
Maximum tested file size: 2.1GB (Vim got slow around 800MB; we consider this the sharding threshold)
Concurrent users: 1
Users who can exit VimDB without Googling how: 0

VimDB is available now. Our GitHub repo has 8 stars and one pull request titled "please delete this repository" which I have marked as "won't fix."

4 Comments — Powered by Disqus

bradkow_ios · 5 days ago

Gordon I ran a benchmark and VimDB is actually faster than SQLite for single-row reads. I don't know what to do with this information.

gashby_dataAuthor · 5 days ago

@bradkow_ios I know. I've been sitting with this for three weeks. It keeps me up at night.

priya_api_first · 4 days ago

The fact that your transaction model is 'undo with u' is either the worst or best thing in database theory.

ebaxter_dev · 3 days ago

I accidentally typed :wq in my Postgres terminal and honestly, it should have worked.