Back to work

Consistency in distributed transactions and event streams

I verified candidates flagged by an AI analysis tool and in reviews myself, against the code and library source, decided the scope of each fix, and fixed an ambiguous commit in 2PC and a Kafka rewind that did not work as intended. I found and blocked TTL deletes that nearly flowed into CDC myself, and after launch I checked retry safety across all paths.

Situation

The game server models users and guilds as event-sourced actors, and wraps requests that change several actors in a two-phase commit (2PC). Change events flow to Kafka through CDC and a transactional outbox. In this setup, a small defect turns into data mismatches that rarely show up in tests.

Decisions

Ambiguous 2PC commits: When the DB answers that it cannot tell whether a commit happened, the events may already be written, but the rollback path reverted only the in-memory state. From the fixes the AI analysis tool proposed, I chose and implemented the one that checks again whether the commit actually happened and brings the journal and memory back in line. An extension that widened this to the coordinator side could misjudge under contention, so I reverted it myself and narrowed the decision to the actor that directly holds the entity.

Kafka rewind: Code meant to rewind a consumer a few minutes on startup did not match how the library behaves, so it was actually always starting from the latest position. At first I judged recovery from the DB to be enough and did not restore the rewind, but I accepted a review that pointed out the gap between reading the DB and starting the subscription, and changed my call. I made the start position follow a point in time and made the overlapping range absorbed idempotently. For consumers where a rewind creates duplicates, such as notifications, I made it explicit in the code that they start from the latest position.

TTL deletes: I reproduced locally that TTL deletes on the outbox table emitted empty events through CDC, which could write empty default values downstream, and blocked it right after the outbox was introduced, before launch. I had also added handling that skipped failures, but reverted it following review feedback that skipping silently would also bury real problems.

Retry safety: On mobile networks, the same request can arrive more than once. While checking whether one guild feature was safe to retry, I widened the scope to the whole server. Together with AI, I traced call paths back from the points that involve currency and rewards, and systematically found and closed paths that were not safe to retry.

Result

I fixed the 2PC, Kafka and TTL defects before launch, and after launch I widened the retry-safety check to all paths. I also built a metric and an alarm that immediately surface messages whose processing keeps failing.

My role

A colleague's AI code-analysis tool first flagged the 2PC and Kafka defects, and I chose the fix directions from its suggestions. My part was to verify the candidates against the code and library source, decide the impact and the scope of each fix, and fix them with review feedback applied. The TTL delete fix and the retry-safety check were work I started myself.