How We Lost ISUCON 2026 (ISUREAL)

We competed in ISUCON 2026 today and lost.

At 17:34, 26 minutes before the end of the contest, the 2,100-line in-memory state store (StateStore) we had spent the last two hours building failed the consistency check during the full-cluster reboot test. We reverted it from main. After reverting, we tracked down candidate root causes in the logs and code and wrote fixes, but we ran out of time to merge them back and pass a full reboot benchmark.

While everything is still fresh, I want to write down what we did today and why our final in-memory migration broke consistency.

The Application and Hardware Quirks#

This year's application, ISUREAL, is a BeReal-like photo-sharing app. Users post photos during regional posting_windows, react to friends' posts with photo reactions (RealMoji) and comments, and browse feeds and past Memories.

Excluding the router, we were given four machines: one app server in each of three regions (asia1, europe1, america1), and a single database server (asia2) running PostgreSQL 16.

[Benchmarker (connects to Asia / Europe / America, users migrate between regions)]
       │                        │                        │
       ▼                        ▼                        ▼
┌──────────────┐         ┌──────────────┐         ┌──────────────┐
│    asia1     │◄───────►│   europe1    │◄───────►│   america1   │
│ 192.168.1.10 │  ~100ms │ 192.168.2.10 │  ~100ms │ 192.168.3.10 │
│   c5.large   │         │  c7i.large   │         │  c7i.large   │
│  (no sha_ni) │         │   (sha_ni)   │         │   (sha_ni)   │
└──────┬───────┘         └──────┬───────┘         └──────┬───────┘
       │                        │                        │
       │ ~1ms                   │ ~100ms                 │ ~100ms
       ▼                        ▼                        ▼
┌────────────────────────────────────────────────────────────────┐
│                     asia2 (192.168.4.10)                       │
│      c5.large (no sha_ni / AVX-512 / RAM 3.7GiB / Swap 0)      │
│                         PostgreSQL 16                          │
└────────────────────────────────────────────────────────────────┘

Three hardware and workload constraints shaped almost everything we did:

1. 100ms inter-region RTT to the single database: PostgreSQL ran on asia2 alone. Queries from asia1 took ~1ms, while europe1 and america1 paid ~100ms round-trip per query. Running five or six sequential SQL statements per HTTP request—as the initial implementation did—burned 500–600ms on network wait alone. Any round-trip after acquiring a row lock inside a transaction also directly extended how long other requests waited on that lock.
2. CPU sha_ni asymmetry and 1,000-round chained HMAC-SHA256: Almost every response (feeds, memories, profiles) returns signed image URLs (/api/images/:id?exp=...&sig=...) keyed by the owner's secret. Instead of a single HMAC-SHA256 call, the app chains 1,000 iterations, feeding each digest into the next round. Checking /proc/cpuinfo showed that europe1 and america1 (Sapphire Rapids / c7i.large) had hardware SHA-256 instructions (sha_ni), while asia1 and asia2 (Cascade Lake / c5.large) did not. On asia1, URL signing alone consumed ~40% of CPU.
3. 3.7 GiB RAM with zero swap: All four nodes had 3.7 GiB RAM and Swap 0. asia2 hosted a ~9 GB image_objects table in PostgreSQL; pushing connection counts or buffer sizes too high immediately triggered the Linux OOM killer.

Score Progression and Our Two Parallel Tracks#

From the opening hour, myuon recorded the timestamp and score of every benchmark run. We started at 2,650 points at 10:08 and reached 125,720 points at 16:05 when we introduced PgBouncer. Progression was not a straight line—single-run variance was high, and we reverted several changes that regressed.

From just after 15:00 onward, our work split into two parallel tracks:

• myuon's track (main branch): Keep PostgreSQL as the source of truth while steadily collapsing round-trips, prewarming PostgreSQL on boot, and moving asia1's signature offload to a dedicated process on asia2.
• My track (StateStore on a separate branch): Put every remaining table in memory on each app server and broadcast write deltas between peers so europe1 and america1 could serve reads with zero database round-trips.

First, here are the four mechanisms that drove our gains on main.

Four Things That Worked on main#

1. Cutting 100ms Database Round-Trips (PL/pgSQL Functions and Query Consolidation)#

In the initial implementation, write handlers (such as creating a post or accepting a friend request) opened a transaction in Go and executed 5–6 sequential SQL queries (check user -> check permissions -> INSERT -> create notification -> COMMIT). From europe1 or america1, that meant holding row locks in PostgreSQL while waiting 500ms+ for inter-continental round-trips.

We replaced all major multi-statement write transactions with PostgreSQL PL/pgSQL functions called in a single SELECT func(...) round-trip (100ms total). That eliminated lock hold times across WAN round-trips.

On reads, we also collapsed multi-step queries into single round-trips:

• Notifications, post details, and user profiles: Parallelized or merged permission checks and payload fetches, cutting 2–3 round-trips down to 1.
• Friends-of-Friends (fof) feed: Collapsed candidate filtering and score calculation from up to 6 round-trips into a single SQL query, verified to produce identical results and ordering across all 400 initial users.
• Targeted in-memory caches: Rather than caching every table at once, we moved three self-contained workloads into memory early on: user search, login sessions, and SSE notification delivery.

2. Peer-to-Peer Image Replication Over :8081 (Async Push + On-Demand Pull)#

Early pg_stat_statements profiles showed that reading and writing JPEG blobs (bytea) in image_objects was the single largest consumer of database time (132 seconds per benchmark run).

For the ~9 GB of initial images already in PostgreSQL, we extracted the files beforehand onto local disk (/home/isucon/image-cache/<id>) on all three app servers and served them directly via nginx X-Accel-Redirect without querying the database.

For new images uploaded during the benchmark, we opened a private-IP-only HTTP port (:8081) on each app server. On upload, we inserted only lightweight metadata (user_id and created_at, needed for URL signature verification) into image_objects with empty data = '', saved the JPEG bytes to local disk, and replicated the binary directly between regional peers over :8081:

[On upload (e.g., received by europe1)]
  1. INSERT metadata row into DB (asia2) with empty bytea (data = '')
  2. Save JPEG bytes to local disk on europe1 (new-uploads directory)
  3. Background workers (8 goroutines) push asynchronously to peers:
     europe1 ── PUT http://<peer>:8081/image/{id} ──► asia1, america1
 
[On read (e.g., requested on asia1 before async Push arrives)]
  1. Not on asia1 local disk, and DB row has empty data (= new image)
  2. Pull on demand from all peers in parallel (1.5s timeout):
     asia1 ── GET http://<peer>:8081/image/{id} ──► europe1, america1
  3. Take whichever peer responds first, save locally, and serve via nginx X-Accel-Redirect
 
[On POST /api/initialize]
  Receiving server ── POST http://<peer>:8081/image-cache/reset ──► wipe only new images

The reason this image peer replication stayed rock-solid through the end of the contest is that it paired async Push (PUT /image/{id}) with on-demand Pull (GET /image/{id}) on a local miss. Even if a user uploaded an image in Europe and another request immediately asked for it in Asia before the 100ms background push arrived, asia1 pulled the bytes from europe1 on the spot.

3. Speeding Up 1,000-Round HMAC-SHA256 Signing (16-Lane AVX-512 and Dedicated Signer)#

We attacked the 1,000-round chained HMAC-SHA256 bottleneck in four steps:

1. 60-second exp bucketing and in-memory caching: Instead of signing with per-second expiration timestamps, we rounded exp to 60-second buckets and cached computed signatures in memory (we briefly tried 1-hour buckets, failed the benchmarker's expiration check, and reverted to 60s). When less than 20 seconds remained in the current bucket, a background worker precomputed signatures for the next +60s bucket.
2. Offloading /sign to europe1 and america1 (sha_ni): At noon, asia1 (no sha_ni) sent batches of uncached (key, imageID, exp) tuples over :8081 to POST /sign on europe1 and america1, which had hardware sha_ni.
3. 16-lane parallel SHA-256 with Go 1.27 GOEXPERIMENT=simd: While asia1 and asia2 lack sha_ni, they have 512-bit AVX-512 registers. A single signature chains 1,000 serial rounds and cannot be vectorized by itself, but feeds and Memories sign dozens of URLs at once. Using Go 1.27's archsimd.Uint32x16, we packed 16 independent signatures across the 16 lanes of 512-bit registers and ran the chained SHA-256 rounds in lockstep.
4. Moving asia1's offload to a dedicated :8081 signer on asia2: Late in the contest (17:24), Netdata showed that /sign requests from asia1 were consuming ~25% of CPU on europe1 and america1, leaving zero headroom at peak. Since our 16-lane AVX-512 implementation was already running, we started a dedicated signing-only app process on asia2:8081 (excluded from nginx routing) so asia1 could offload AVX-512 signing just 1ms away.

4. Surviving 3.7 GiB / Swap 0 OOMs with PgBouncer#

Because europe1 and america1 hold a DB connection across a 100ms round-trip on every query, small connection pools quickly block waiting for a free connection. When we raised PostgreSQL's max_connections to 400 and set the three app servers' SetMaxOpenConns to 310 total, asia2 (3.7 GiB RAM, 0 swap) ran out of memory and the OOM killer terminated postgres.

We lowered PostgreSQL's shared_buffers from 1.5 GB to 768 MB and placed PgBouncer (transaction mode) on asia2:

• App-side connection pool caps: europe1 300 + america1 300 + asia1 100 = 700 total connections
• PgBouncer to PostgreSQL backend connections: 90 connections (max_db_connections=100)

That decoupled WAN connection hold times from PostgreSQL backend memory and eliminated the OOMs. After a reboot test at 17:05 hit a 504 on initialize due to cold OS/DB caches right after boot, we also added a startup systemd service on asia2 to prewarm all tables.

Here is the final architecture we ran on main:

┌───────────────────┐    ┌───────────────────┐    ┌───────────────────┐
│       asia1       │    │      europe1      │    │     america1      │
│  nginx ──► :8080  │    │  nginx ──► :8080  │    │  nginx ──► :8080  │
│  X-Accel-Redirect │    │  X-Accel-Redirect │    │  X-Accel-Redirect │
│  Peer Port :8081  │◄──►│  Peer Port :8081  │◄──►│  Peer Port :8081  │
└────┬─────────┬────┘    └─────────┬─────────┘    └─────────┬─────────┘
     │         │                   │                        │
     │ ~1ms    │ ~1ms              │ ~100ms                 │ ~100ms
     │ (POST   │ (SQL)             │ (SQL, MaxOpen 300)     │ (SQL, MaxOpen 300)
     │  /sign) ▼                   ▼                        ▼
     │   ┌──────────────────────────────────────────────────────────┐
     │   │                    asia2 (192.168.4.10)                  │
     └──►│  Dedicated Signer (:8081, AVX-512 16-lane)               │
         │  PgBouncer (:6432, transaction mode, pool=90)            │
         │     └──► PostgreSQL 16 (:5432, shared_buffers=768MB)     │
         └──────────────────────────────────────────────────────────┘

Why Our Final Full In-Memory StateStore Failed the Reboot Test#

While myuon kept improving main, from just after 15:00 I had my agent building a 2,100-line full in-memory store (StateStore) on a separate branch covering every remaining table (users, posts, comments, RealMojis, follows, blocks, friend requests, notifications, and post views). The goal was to eliminate the 100ms WAN hop from europe1 and america1 to asia2 on reads altogether.

StateStore extended the internal :8081 peer network with POST /state/op and POST /state/reload. On any write, the server handling the request updated PostgreSQL first, then broadcast a JSON delta (StateOp) to the other two regional peers over :8081 so each server could update its in-memory state.

We merged it into main at 16:55, reverted it at 17:11, and merged a fixed version at 17:30. While StateStore passed standalone benchmark runs, it failed the consistency check during the 17:34 reboot test, 26 minutes before the end. Newly posted comments were missing, GET /api/notifications returned 401, Friends-of-Friends feed scores were off by 1–2 points, replaced RealMojis still returned the old reaction, and image fetches failed with 403 image_forbidden. We reverted it immediately at 17:34.

Below are the three candidate bugs in StateStore that we found in the logs and code after reverting. Because we ran out of time to re-merge and pass another reboot benchmark, we never verified in production how much of the reboot failure each fix resolved, but the code defects themselves are clear.

1. Users Migrating Between Regions (user moved) vs. Push-Only Async Queues#

Checking the logs showed that benchmark users switch target regions mid-run (user moved).

With image peer replication, if an async PUT /image/{id} had not arrived yet, the reading server could detect the miss and pull the image on demand via GET /image/{id}. StateStore, by contrast, was push-only: every read for posts, comments, reactions, notifications, and sessions served whatever was currently in local memory without falling back to peers or PostgreSQL.

On top of that, to keep cross-region POST /state/op calls (100ms RTT) from slowing down write responses, any peer broadcast that took longer than 800ms was pushed onto a per-peer async retry queue (peerRetryQueue) while the server immediately returned HTTP 200/201 to the client.

Benchmarker User
  │
  ├── (1) POST /api/posts/:id/comments ──► asia1
  │                                          ├── (2) Update PostgreSQL (asia2)
  │                                          ├── (3) POST :8081/state/op ──► europe1, america1
  │                                          │        (Exceeds 800ms right after 4-node reboot
  │                                          │         → queued in async peerRetryQueue)
  │   ◄── (4) Returns 201 Created early ─────┘
  │
  └── (5) user moved: immediately switches to europe1 and calls GET /api/posts/:id
                                             │
                                           europe1
                                             └── (6) Serves directly from local StateStore
                                                      ((3) has not arrived yet → comment is missing!)

Right after all four servers rebooted simultaneously, DB connections, caches, and inter-peer HTTP connection pools were all cold, causing peer synchronization to lag. When a user logged in, commented, or accepted a friend request on asia1 and immediately moved to europe1, their read arrived before the queued StateOp had been applied on europe1. In a fix at 17:46 after reverting, we changed write handlers to wait up to 3 seconds for peer synchronization before returning a response.

2. Mixing Application Clocks and Database Clocks#

To prevent out-of-order StateOp delivery from overwriting newer state, StateStore used Last-Write-Wins (LWW) timestamp comparisons. Here we mixed two different clocks.

Most tables used PostgreSQL's clock_timestamp() captured while holding the lock, but the RealMoji replacement function invalidated the old row with replaced_at and inserted the new row using p_created_at—the time.Now() passed in from the Go app server:

-- Before: invalidated the old row with replaced_at, then inserted the new row using the app clock
INSERT INTO realmojis (id, post_id, reactor_id, image_id, reaction_type, created_at)
VALUES (p_realmoji_id, p_post_id, p_actor_id, p_image_id, p_reaction_type, p_created_at);

If a user moved to another app server whose clock was slightly behind and replaced their RealMoji, the new row could get a created_at earlier than the old one, causing StateStore to drop the update as stale.

3. Deleting Initial Images and Dropping Disk Metadata#

Right near the end, we found two bugs in our local image cache.

First, we had a bug in profile image deletion for initial users. When POST /api/initialize runs reset.sql, it restores initial users' profile_image_id from initial_users, but it does not restore initial rows in image_objects or files in /home/isucon/image-cache/<id>—because initial images are assumed never to be deleted.

However, when an initial user updated or deleted their profile image during a benchmark run, our code executed DELETE FROM image_objects WHERE id = $1 and os.Remove(imageCachePath(id)) on the old image ID. On the next benchmark run, initialize restored the user's original profile_image_id, which now pointed to an image missing from both disk and PostgreSQL.

// After: only delete images created after initial_data_boundary, and keep initial image files on disk
_, _ = item.db.ExecContext(ctx, `
    DELETE FROM image_objects
    WHERE id = $1
      AND created_at > COALESCE((SELECT created_at FROM initial_data_boundary LIMIT 1), '-infinity'::timestamptz)
`, item.imageID)

Second, when persisting metadata for peer-pushed new images to local disk, our writer saved only the owner's user_id and forgot to save created_at. After a reboot, reloading metadata from disk replaced created_at with time.Now(), causing fallback database lookups that filtered by created_at to miss the row and return 403 image_forbidden.

Takeaway#

In terms of how we worked as a team, we started the morning with one person driving Claude Code. Around the 20,000-point mark, Opus 5.5 started getting stuck, and we began running Argon in parallel. Looking back, all three of us probably should have been running our own agents in parallel from the very beginning.

Benchmark turnaround time was another constant bottleneck. Compared to how fast agents could write code, waiting for the official benchmarker to finish—and dealing with run-to-run score variance—slowed down every iteration. Building a small local load-testing mini-benchmarker early on would have let us validate changes and catch regressions locally instead of spending full benchmark runs to isolate every change.

The direct cause of our loss, though, was trying to land a 2,100-line in-memory cache covering every remaining table at once with less than three hours left.

As long as we were shipping one mechanism at a time—peer image replication, user search, SSE, login sessions—and validating each step against the benchmarker, we could isolate and fix regressions quickly. With AI coding agents, writing a 2,100-line StateStore takes very little time. Subtle invariants—users migrating between regions, mixing app clocks with DB clocks, and which tables reset.sql actually restores—do not take care of themselves just because the code was written quickly.

If I could replay the last three hours, instead of caching every table at once, I would put posts and comments in memory with synchronous peer broadcast, pass the production reboot test, and expand one table at a time. Writing code got much faster; the safe size of a single change did not.