We competed in ISUCON 2026 today and lost.
Update, November 3: The contest environment stayed open until November 23, so I ran a rematch, rebuilt the fully in-memory design, and reached a self-best of 544,004 points on the portal. The details are in the “Rematch” section at the end.
At 17:34, 26 minutes before the end of the contest, the 2,100-line in-memory state store (StateStore) we had spent the last two hours building failed the consistency check during the full-cluster reboot test. We reverted it from main. After reverting, we tracked down candidate root causes in the logs and code and wrote fixes, but we ran out of time to merge them back and pass a full reboot benchmark.
While everything is still fresh, I want to write down what we did today and why our final in-memory migration broke consistency.
The Application and Hardware Quirks
This year's application, ISUREAL, is a BeReal-like photo-sharing app. Users post photos during regional posting_windows, react to friends' posts with photo reactions (RealMoji) and comments, and browse feeds and past Memories.
Excluding the router, we were given four machines: one app server in each of three regions (asia1, europe1, america1), and a single database server (asia2) running PostgreSQL 16.
[Benchmarker (connects to Asia / Europe / America, users migrate between regions)]
│ │ │
▼ ▼ ▼
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│ asia1 │◄───────►│ europe1 │◄───────►│ america1 │
│ 192.168.1.10 │ ~100ms │ 192.168.2.10 │ ~100ms │ 192.168.3.10 │
│ c5.large │ │ c7i.large │ │ c7i.large │
│ (no sha_ni) │ │ (sha_ni) │ │ (sha_ni) │
└──────┬───────┘ └──────┬───────┘ └──────┬───────┘
│ │ │
│ ~1ms │ ~100ms │ ~100ms
▼ ▼ ▼
┌────────────────────────────────────────────────────────────────┐
│ asia2 (192.168.4.10) │
│ c5.large (no sha_ni / AVX-512 / RAM 3.7GiB / Swap 0) │
│ PostgreSQL 16 │
└────────────────────────────────────────────────────────────────┘Three hardware and workload constraints shaped almost everything we did:
- 100ms inter-region RTT to the single database: PostgreSQL ran on
asia2alone. Queries fromasia1took ~1ms, whileeurope1andamerica1paid ~100ms round-trip per query. Running five or six sequential SQL statements per HTTP request—as the initial implementation did—burned 500–600ms on network wait alone. Any round-trip after acquiring a row lock inside a transaction also directly extended how long other requests waited on that lock. - CPU
sha_niasymmetry and 1,000-round chained HMAC-SHA256: Almost every response (feeds, memories, profiles) returns signed image URLs (/api/images/:id?exp=...&sig=...) keyed by the owner's secret. Instead of a single HMAC-SHA256 call, the app chains 1,000 iterations, feeding each digest into the next round. Checking/proc/cpuinfoshowed thateurope1andamerica1(Sapphire Rapids /c7i.large) had hardware SHA-256 instructions (sha_ni), whileasia1andasia2(Cascade Lake /c5.large) did not. Onasia1, URL signing alone consumed ~40% of CPU. - 3.7 GiB RAM with zero swap: All four nodes had 3.7 GiB RAM and
Swap 0.asia2hosted a ~9 GBimage_objectstable in PostgreSQL; pushing connection counts or buffer sizes too high immediately triggered the Linux OOM killer.
Score Progression and Our Two Parallel Tracks
From the opening hour, myuon recorded the timestamp and score of every benchmark run. We started at 2,650 points at 10:08 and reached 125,720 points at 16:05 when we introduced PgBouncer. Progression was not a straight line—single-run variance was high, and we reverted several changes that regressed.
| Time (JST) | Score | Main changes active at that point | fail |
|---|---|---|---|
| 10:08 | 2,650 | Baseline | 4 |
| 10:30 | 19,405 | Indexes, signature cache, DB connection reuse | 0 |
| 11:23 | 26,837 | PL/pgSQL write functions, notification index | 0 |
| 11:47 | 35,693 | Header-based JPEG validation, async DB image writes | 0 |
| 12:14 | 47,277 | In-memory user search and narrower lock scope | 20 |
| 13:10 | 66,466 | Peer port (:8081) for EU/US signing offload & image replication, SQL feed filter | 0 |
| 13:42 | 73,414 | Single-query notifications, faster feed viewed checks | 0 |
| 14:35 | 85,133 | In-memory SSE delivery, nginx X-Accel-Redirect for new images | 0 |
| 14:54 | 87,199 | Enable Go 1.27 AVX-512 (16-lane parallel) signature worker | 0 |
| 15:26 | 96,424 | Smaller DB connection pools (avoid OOM), in-memory login sessions | 0 |
| 15:55 | 110,614 | Lower shared_buffers to 768 MB, parallelize two Memories queries | 0 |
| 16:05 | 125,720 | Introduce PgBouncer (app pool cap 700, DB backend pool 90) | 0 |
From just after 15:00 onward, our work split into two parallel tracks:
- myuon's track (
mainbranch): Keep PostgreSQL as the source of truth while steadily collapsing round-trips, prewarming PostgreSQL on boot, and movingasia1's signature offload to a dedicated process onasia2. - My track (
StateStoreon a separate branch): Put every remaining table in memory on each app server and broadcast write deltas between peers soeurope1andamerica1could serve reads with zero database round-trips.
First, here are the four mechanisms that drove our gains on main.
Four Things That Worked on main
1. Cutting 100ms Database Round-Trips (PL/pgSQL Functions and Query Consolidation)
In the initial implementation, write handlers (such as creating a post or accepting a friend request) opened a transaction in Go and executed 5–6 sequential SQL queries (check user -> check permissions -> INSERT -> create notification -> COMMIT). From europe1 or america1, that meant holding row locks in PostgreSQL while waiting 500ms+ for inter-continental round-trips.
We replaced all major multi-statement write transactions with PostgreSQL PL/pgSQL functions called in a single SELECT func(...) round-trip (100ms total). That eliminated lock hold times across WAN round-trips.
On reads, we also collapsed multi-step queries into single round-trips:
- Notifications, post details, and user profiles: Parallelized or merged permission checks and payload fetches, cutting 2–3 round-trips down to 1.
- Friends-of-Friends (
fof) feed: Collapsed candidate filtering and score calculation from up to 6 round-trips into a single SQL query, verified to produce identical results and ordering across all 400 initial users. - Targeted in-memory caches: Rather than caching every table at once, we moved three self-contained workloads into memory early on: user search, login sessions, and SSE notification delivery.
2. Peer-to-Peer Image Replication Over :8081 (Async Push + On-Demand Pull)
Early pg_stat_statements profiles showed that reading and writing JPEG blobs (bytea) in image_objects was the single largest consumer of database time (132 seconds per benchmark run).
For the ~9 GB of initial images already in PostgreSQL, we extracted the files beforehand onto local disk (/home/isucon/image-cache/<id>) on all three app servers and served them directly via nginx X-Accel-Redirect without querying the database.
For new images uploaded during the benchmark, we opened a private-IP-only HTTP port (:8081) on each app server. On upload, we inserted only lightweight metadata (user_id and created_at, needed for URL signature verification) into image_objects with empty data = '', saved the JPEG bytes to local disk, and replicated the binary directly between regional peers over :8081:
[On upload (e.g., received by europe1)]
1. INSERT metadata row into DB (asia2) with empty bytea (data = '')
2. Save JPEG bytes to local disk on europe1 (new-uploads directory)
3. Background workers (8 goroutines) push asynchronously to peers:
europe1 ── PUT http://<peer>:8081/image/{id} ──► asia1, america1
[On read (e.g., requested on asia1 before async Push arrives)]
1. Not on asia1 local disk, and DB row has empty data (= new image)
2. Pull on demand from all peers in parallel (1.5s timeout):
asia1 ── GET http://<peer>:8081/image/{id} ──► europe1, america1
3. Take whichever peer responds first, save locally, and serve via nginx X-Accel-Redirect
[On POST /api/initialize]
Receiving server ── POST http://<peer>:8081/image-cache/reset ──► wipe only new imagesThe reason this image peer replication stayed rock-solid through the end of the contest is that it paired async Push (PUT /image/{id}) with on-demand Pull (GET /image/{id}) on a local miss. Even if a user uploaded an image in Europe and another request immediately asked for it in Asia before the 100ms background push arrived, asia1 pulled the bytes from europe1 on the spot.
3. Speeding Up 1,000-Round HMAC-SHA256 Signing (16-Lane AVX-512 and Dedicated Signer)
We attacked the 1,000-round chained HMAC-SHA256 bottleneck in four steps:
- 60-second
expbucketing and in-memory caching: Instead of signing with per-second expiration timestamps, we roundedexpto 60-second buckets and cached computed signatures in memory (we briefly tried 1-hour buckets, failed the benchmarker's expiration check, and reverted to 60s). When less than 20 seconds remained in the current bucket, a background worker precomputed signatures for the next+60sbucket. - Offloading
/signtoeurope1andamerica1(sha_ni): At noon,asia1(nosha_ni) sent batches of uncached(key, imageID, exp)tuples over:8081toPOST /signoneurope1andamerica1, which had hardwaresha_ni. - 16-lane parallel SHA-256 with Go 1.27
GOEXPERIMENT=simd: Whileasia1andasia2lacksha_ni, they have 512-bit AVX-512 registers. A single signature chains 1,000 serial rounds and cannot be vectorized by itself, but feeds and Memories sign dozens of URLs at once. Using Go 1.27'sarchsimd.Uint32x16, we packed 16 independent signatures across the 16 lanes of 512-bit registers and ran the chained SHA-256 rounds in lockstep. - Moving
asia1's offload to a dedicated:8081signer onasia2: Late in the contest (17:24), Netdata showed that/signrequests fromasia1were consuming ~25% of CPU oneurope1andamerica1, leaving zero headroom at peak. Since our 16-lane AVX-512 implementation was already running, we started a dedicated signing-only app process onasia2:8081(excluded from nginx routing) soasia1could offload AVX-512 signing just 1ms away.
4. Surviving 3.7 GiB / Swap 0 OOMs with PgBouncer
Because europe1 and america1 hold a DB connection across a 100ms round-trip on every query, small connection pools quickly block waiting for a free connection. When we raised PostgreSQL's max_connections to 400 and set the three app servers' SetMaxOpenConns to 310 total, asia2 (3.7 GiB RAM, 0 swap) ran out of memory and the OOM killer terminated postgres.
We lowered PostgreSQL's shared_buffers from 1.5 GB to 768 MB and placed PgBouncer (transaction mode) on asia2:
- App-side connection pool caps:
europe1300 +america1300 +asia1100 = 700 total connections - PgBouncer to PostgreSQL backend connections: 90 connections (
max_db_connections=100)
That decoupled WAN connection hold times from PostgreSQL backend memory and eliminated the OOMs. After a reboot test at 17:05 hit a 504 on initialize due to cold OS/DB caches right after boot, we also added a startup systemd service on asia2 to prewarm all tables.
Here is the final architecture we ran on main:
┌───────────────────┐ ┌───────────────────┐ ┌───────────────────┐
│ asia1 │ │ europe1 │ │ america1 │
│ nginx ──► :8080 │ │ nginx ──► :8080 │ │ nginx ──► :8080 │
│ X-Accel-Redirect │ │ X-Accel-Redirect │ │ X-Accel-Redirect │
│ Peer Port :8081 │◄──►│ Peer Port :8081 │◄──►│ Peer Port :8081 │
└────┬─────────┬────┘ └─────────┬─────────┘ └─────────┬─────────┘
│ │ │ │
│ ~1ms │ ~1ms │ ~100ms │ ~100ms
│ (POST │ (SQL) │ (SQL, MaxOpen 300) │ (SQL, MaxOpen 300)
│ /sign) ▼ ▼ ▼
│ ┌──────────────────────────────────────────────────────────┐
│ │ asia2 (192.168.4.10) │
└──►│ Dedicated Signer (:8081, AVX-512 16-lane) │
│ PgBouncer (:6432, transaction mode, pool=90) │
│ └──► PostgreSQL 16 (:5432, shared_buffers=768MB) │
└──────────────────────────────────────────────────────────┘Why Our Final Full In-Memory StateStore Failed the Reboot Test
While myuon kept improving main, from just after 15:00 I had my agent building a 2,100-line full in-memory store (StateStore) on a separate branch covering every remaining table (users, posts, comments, RealMojis, follows, blocks, friend requests, notifications, and post views). The goal was to eliminate the 100ms WAN hop from europe1 and america1 to asia2 on reads altogether.
StateStore extended the internal :8081 peer network with POST /state/op and POST /state/reload. On any write, the server handling the request updated PostgreSQL first, then broadcast a JSON delta (StateOp) to the other two regional peers over :8081 so each server could update its in-memory state.
We merged it into main at 16:55, reverted it at 17:11, and merged a fixed version at 17:30. While StateStore passed standalone benchmark runs, it failed the consistency check during the 17:34 reboot test, 26 minutes before the end. Newly posted comments were missing, GET /api/notifications returned 401, Friends-of-Friends feed scores were off by 1–2 points, replaced RealMojis still returned the old reaction, and image fetches failed with 403 image_forbidden. We reverted it immediately at 17:34.
Below are the three candidate bugs in StateStore that we found in the logs and code after reverting. Because we ran out of time to re-merge and pass another reboot benchmark, we never verified in production how much of the reboot failure each fix resolved, but the code defects themselves are clear.
1. Users Migrating Between Regions (user moved) vs. Push-Only Async Queues
Checking the logs showed that benchmark users switch target regions mid-run (user moved).
With image peer replication, if an async PUT /image/{id} had not arrived yet, the reading server could detect the miss and pull the image on demand via GET /image/{id}. StateStore, by contrast, was push-only: every read for posts, comments, reactions, notifications, and sessions served whatever was currently in local memory without falling back to peers or PostgreSQL.
On top of that, to keep cross-region POST /state/op calls (100ms RTT) from slowing down write responses, any peer broadcast that took longer than 800ms was pushed onto a per-peer async retry queue (peerRetryQueue) while the server immediately returned HTTP 200/201 to the client.
Benchmarker User
│
├── (1) POST /api/posts/:id/comments ──► asia1
│ ├── (2) Update PostgreSQL (asia2)
│ ├── (3) POST :8081/state/op ──► europe1, america1
│ │ (Exceeds 800ms right after 4-node reboot
│ │ → queued in async peerRetryQueue)
│ ◄── (4) Returns 201 Created early ─────┘
│
└── (5) user moved: immediately switches to europe1 and calls GET /api/posts/:id
│
europe1
└── (6) Serves directly from local StateStore
((3) has not arrived yet → comment is missing!)Right after all four servers rebooted simultaneously, DB connections, caches, and inter-peer HTTP connection pools were all cold, causing peer synchronization to lag. When a user logged in, commented, or accepted a friend request on asia1 and immediately moved to europe1, their read arrived before the queued StateOp had been applied on europe1. In a fix at 17:46 after reverting, we changed write handlers to wait up to 3 seconds for peer synchronization before returning a response.
2. Mixing Application Clocks and Database Clocks
To prevent out-of-order StateOp delivery from overwriting newer state, StateStore used Last-Write-Wins (LWW) timestamp comparisons. Here we mixed two different clocks.
Most tables used PostgreSQL's clock_timestamp() captured while holding the lock, but the RealMoji replacement function invalidated the old row with replaced_at and inserted the new row using p_created_at—the time.Now() passed in from the Go app server:
-- Before: invalidated the old row with replaced_at, then inserted the new row using the app clock
INSERT INTO realmojis (id, post_id, reactor_id, image_id, reaction_type, created_at)
VALUES (p_realmoji_id, p_post_id, p_actor_id, p_image_id, p_reaction_type, p_created_at);If a user moved to another app server whose clock was slightly behind and replaced their RealMoji, the new row could get a created_at earlier than the old one, causing StateStore to drop the update as stale.
3. Deleting Initial Images and Dropping Disk Metadata
Right near the end, we found two bugs in our local image cache.
First, we had a bug in profile image deletion for initial users. When POST /api/initialize runs reset.sql, it restores initial users' profile_image_id from initial_users, but it does not restore initial rows in image_objects or files in /home/isucon/image-cache/<id>—because initial images are assumed never to be deleted.
However, when an initial user updated or deleted their profile image during a benchmark run, our code executed DELETE FROM image_objects WHERE id = $1 and os.Remove(imageCachePath(id)) on the old image ID. On the next benchmark run, initialize restored the user's original profile_image_id, which now pointed to an image missing from both disk and PostgreSQL.
// After: only delete images created after initial_data_boundary, and keep initial image files on disk
_, _ = item.db.ExecContext(ctx, `
DELETE FROM image_objects
WHERE id = $1
AND created_at > COALESCE((SELECT created_at FROM initial_data_boundary LIMIT 1), '-infinity'::timestamptz)
`, item.imageID)Second, when persisting metadata for peer-pushed new images to local disk, our writer saved only the owner's user_id and forgot to save created_at. After a reboot, reloading metadata from disk replaced created_at with time.Now(), causing fallback database lookups that filtered by created_at to miss the row and return 403 image_forbidden.
Takeaway
In terms of how we worked as a team, we started the morning with one person driving Claude Code. Around the 20,000-point mark, Opus 5.5 started getting stuck, and we began running Argon in parallel. Looking back, all three of us probably should have been running our own agents in parallel from the very beginning.
Benchmark turnaround time was another constant bottleneck. Compared to how fast agents could write code, waiting for the official benchmarker to finish—and dealing with run-to-run score variance—slowed down every iteration. Building a small local load-testing mini-benchmarker early on would have let us validate changes and catch regressions locally instead of spending full benchmark runs to isolate every change.
The direct cause of our loss, though, was trying to land a 2,100-line in-memory cache covering every remaining table at once with less than three hours left.
As long as we were shipping one mechanism at a time—peer image replication, user search, SSE, login sessions—and validating each step against the benchmarker, we could isolate and fix regressions quickly. With AI coding agents, writing a 2,100-line StateStore takes very little time. Subtle invariants—users migrating between regions, mixing app clocks with DB clocks, and which tables reset.sql actually restores—do not take care of themselves just because the code was written quickly.
If I could replay the last three hours, instead of caching every table at once, I would put posts and comments in memory with synchronous peer broadcast, pass the production reboot test, and expand one table at a time. Writing code got much faster; the safe size of a single change did not.
Rematch (November 3): 544,004 Points, Fully In-Memory
The contest environment stayed up until November 23, so on November 3 I held a rematch. I had Claude Code (Opus 5.5) rebuild, from scratch, the design we had failed to land on contest day: everything except images in memory. My only job was to run benchmarks and paste the results. Design, implementation, deployment, measurement, and log analysis were all done by the agent.
The result was a self-best of 544,004 points on the portal. On the leaderboard at 16:55 (sorted by latest score) we were second. First place, Anago, had a best of 594,503, and the winner, Typing Bears, had a best of 629,638, so there is still a gap.
Three conclusions up front:
- A global order for writes fixed consistency. The contest-day approach, servers pushing writes to each other, failed again in the rematch. After switching to a sequencer, consistency violations almost disappeared.
- Past 500k, the wall was the benchmarker's capacity, not our servers. Holding back asia, which counts 1x, so the benchmarker spent its capacity on europe and america (1.5x) beat making the servers faster.
- The same configuration varies by about 10%. User growth compounds, so a single run cannot tell you whether a change helped.
Score progression on November 3
| Time | Score | Main changes at that point |
|---|---|---|
| 06:46 | 0 | First run of the in-memory build. nginx ran out of worker_connections (EOF and 502) |
| 07:22–08:10 | FAIL ×5 | Servers pushing writes to each other directly; consistency violations (posts that must not be visible, stale friend lists) |
| 08:38 | — | Switched to a sequencer on asia2 that decides the order of all writes |
| 09:02–09:20 | FAIL ×3 | PostgreSQL connection limit; asynchronous login broke the out-of-region 403 check |
| 10:45 | 276,068 | First complete run |
| 11:10 | 300,956 | asia Memories and export moved to a read-only process on asia2 |
| 11:47 | 325,436 | Signed session tokens |
| 12:45 | 366,054 | Historical posts from the initial data served from a read-only file |
| 13:17 | 420,044 | Asynchronous write log; header-only JPEG checks |
| 13:29 | FAIL | The read-only process fell behind and served a feed from stale state |
| 15:22 | 504,018 | Feed index; asia1 signature verification moved to europe and america |
| 15:31 | 467,519 | Experiment: no grand prix notification stagger (failed, reverted) |
| 15:37 | 517,618 | Delay asia post responses by 2s after 54 posts per posting window |
| 15:56 | 527,012 | Parallel read-ahead of export images |
| 16:48 | 529,972 | Delay asia Memories by up to 300ms only when total traffic is high |
The portal best of 544,004 came from one of the runs whose result I forgot to paste, so I cannot tell which change it ran on.
What changed from contest day: a global order for writes
Contest day's StateStore had the server that received a write push the change to the others directly, and resolved conflicts by update time (last write wins). In the rematch, the first five runs of the morning still failed with consistency violations under that design: a visibility change arriving before the post it modified and being dropped, deliveries arriving out of order, and so on.
So I switched to the same design the winner, Typing Bears, described in their write-up:
- One sequencer on
asia2gives every write a sequence number. Every server applies the same ordered stream from the sequencer. - A write responds only after 60ms (δ) have passed since time T, when the sequencer numbered it.
- A read waits until the local stream has been applied up to “now − 60ms”. If it has not caught up after 5 seconds, it returns 503 instead of stale state.
Write: europe1 ──POST /seq/cmd──▶ asia2 (assigns sequence n, time T) ──▶ same ordered stream to every server
europe1 waits until T+60ms before responding to the user
Read: asia1 waits until "time of the last applied entry ≥ now − 60ms" before handling the request
→ any read that starts after T+60ms is guaranteed to include entry nThis guarantees that once a write has responded, every later read on any server sees it. δ is 60ms because the one-way latency from asia2 to europe and america is about 50ms.
Five other things mattered:
- No database writes. PostgreSQL is only used to load the initial data during
initialize. For persistence, the sequencer appends the numbered writes to a JSON Lines file and replays it from the start on boot: 84,000 writes in about 0.8 seconds. - Precomputed signatures. Image URL expiry is pinned to 12-hour buckets ending at 11:00 and 23:00 UTC, and the signatures for all 1.83 million initial images are precomputed into a table. New images are signed once when they are created, and the signature travels with the write to every server. The only signatures still computed on demand are the URLs the benchmarker builds itself with a new expiry every second, and
asia1hands those to europe and america, which have SHA-NI. Over 99.5% of signature lookups need no computation. - Logins that do not wait. Session tokens are HMAC-signed with a key shared by all servers, so any server can verify one even before the login has replicated.
- Historical posts from a read-only file. The initial data's posts, RealMoji, and comments live in a 1GB file split into per-user, per-month JSON blocks. The first version stored one block per user, so a Memories view that needed one month decoded the whole user, and JSON decoding took 31% of the read-only process's CPU.
- Rejecting bad uploads from the first bytes. If the JPEG header in the first 64KB of a profile image or post image is already invalid, we return 400 and close the connection without reading the rest. This came from the second-place write-up by ichyo, which found that about 60% of upstream traffic through the router was 10–32MB invalid images sent by the benchmarker.
Past 500k, the wall was the benchmarker, not our servers
Somewhere past 500k, making the servers faster stopped raising the score. europe1 and america1 used only 20–40% CPU, and server-side processing took under 1ms. Even so, total traffic across the four servers plateaued at about 10,000 requests per second. Typing Bears made the same observation in their write-up: the bottleneck was the benchmarker's own capacity.
If the benchmarker's capacity is fixed, the score depends on what you make it spend that capacity on. Points from europe and america count 1.5x. On top of that, about 60% of the score came from a loop in which initial-data users repeat “login → Memories → images → logout → export” without pause (one export is worth 20 points). That “about 60%” is Typing Bears' estimate; our own rough count, multiplying per-run request counts by the point values, came out about the same. So we held back asia's activity, which counts 1x, to free capacity for the europe and america loops.
| Change | Result |
|---|---|
| Delay asia post responses by 2s after 54 posts per posting window | asia users on day 5 went from 1,324 to 359. The router's 503s and EOFs disappeared. 504,018 → 517,618 points |
| Stagger grand prix notifications: europe, then america (+2.5s), then asia (+5s) | Removing it doubled asia's grand prix participation, but europe and america fell from about 100% to 69%, for 467,519 points. Reverted |
| When total traffic exceeds 6,000 req/s, delay asia Memories by up to 300ms | america Memories views went from 11,511 to 14,910 (+30%) and asia fell 25%. asia1 exceeded its outbound bandwidth allowance 107 times per run instead of 1,033 |
Making asia exports faster, on the other hand, barely helped. Export on the read-only process went from 820ms to 30ms, but asia's loop then ran faster and took benchmarker capacity away from europe and america. Faster servers and higher scores were not the same thing.
The same setup scores anywhere from 480k to 530k
In the last hour I ran the same configuration repeatedly, and the score still ranged from 481,875 to 529,972.
The biggest factor was how fast users multiplied. Users roughly triple each day, so a small early difference compounds until the end. With the same configuration, europe ended day 5 with anywhere from 263 to 604 users. The difference between those runs was 13 versus 25 users on day 2.
The final score is the last run you start, so near the end the sensible strategy is to pick a target and stop as soon as a run reaches it.
Measurement mistakes
- Disk utilization was overstated. The sampling script assumed 2-second intervals, but with its latency probes each sample actually took about 3 seconds. Concluding the disk was saturated from values above 100% was wrong.
- Cumulative counters were compared across runs.
/debug/routescounts from process start, so without a restart several runs get mixed together. Later, we took the difference between the start and end of each run. - Profiles were missed. The watcher detected a run as “requests grew past the count at startup”, so it missed runs after the app restarted. It now detects a run by how fast requests grow over 2 seconds.
- The tests passed, and a code review still found three real bugs. A write acknowledged as successful could be lost if the process restarted before it reached the log file. Export read-ahead had no memory bound. An image read failure was treated as “no image”, which could return an incomplete ZIP with a 200. The third one could have failed the whole run, since an export that does not match the initial data is a critical error. None of the tests created those situations: a restart at the wrong moment, a slow client, or a transient I/O error.
What I would fix next with unlimited time
- Measure the benchmarker and router as the scarce resources. Scores vary by about ±5%, so compare configurations on the average of at least 5 runs each.
- Send less traffic through the router.
tracepathshowed that evenasia1toasia2, inside the same region, goes through the router (192.168.10.10). Exports built by the read-only process (about 2MB each) cross the router twice. It would be better to build them onasia1, or to lay out each user's images in one file to reduce reads. The sequencer stream could also be compressed. - Learn the rule for how users multiply. It is the biggest source of variance, so I want to measure how early failures and latency relate to growth.
- Order each user's writes in that user's own region. Writes from europe and america take about 110ms because of the round trip to the sequencer. However, every region reaches about the same share of the browsing caps (about half for feeds), which suggests the benchmarker's own pacing sets that, not write latency. Measure first.
- Smaller leftovers. Replace the database lookups for initial image references on the image path with precomputed data, and fix a rare 401 race when subscribing to notifications.
After the rematch
The lesson from contest day, “we should have bet on putting everything except images in memory”, turned out to be the right direction. What was missing on the day was the right way to stay consistent (servers pushing to each other and resolving by timestamp) and an early enough start. After switching to a sequencer that gives every write a global order, consistency violations almost disappeared.
The other lesson is that past 500k the game was no longer “make the servers faster” but “what do you make the benchmarker spend its capacity on”. Without other teams' write-ups and per-run measurements, I would not have noticed.