I led the migration of a set of high-throughput, latency-sensitive processing jobs from an older async stack to a C++ stream-processing stack. On paper it was a port. In practice the translation was the least interesting part. At high volume, cost is decided by what happens per record, and most of that was in the execution model and the data format, not in the language.
When a job handles a very large number of small records, the fixed cost of handling one record matters more than the work done on it. In the old jobs it hid in places that look free: allocating a future or a closure, handing work to another scheduler, copying a record into a new buffer, touching memory that was no longer in cache. Each costs little. Multiplied by every record, they outweighed the actual logic.
So most of my work went into the processing architecture and execution model: how records move through a job, and which work gets done once instead of again for every record. For each job I started from what it did per record that it didn't need to, and wrote the C++ after those decisions were made.
The data path carried JSON. JSON is easy to debug and expensive to process. Every field name travels with every record. Every number is decimal text that has to be parsed back into binary, and parsing a floating-point number from text is surprisingly expensive. The parser also has to scan every byte to find where strings start and end, handle escapes, and usually build string objects for keys it will immediately compare and throw away.
Compact Thrift encodes the same record as tagged binary. Each field starts with a single header byte: four bits for the field ID, written as a delta from the previous field when the delta is small enough to fit, and four bits for the type. Integers are zigzag-encoded and written as variable-length bytes, so small values take one byte. Here is one illustrative record both ways:
{"user_id":123456789,"ts":1712345678,"score":0.73,"country":"US","flags":5}
JSON: 75 bytes
Compact Thrift: 27 bytes
user_id 1 header + 4 varint ts 1 header + 5 varint
score 1 header + 8 double country 1 header + 1 length + 2
flags 1 header + 1 varint end of struct 1The payload is about a third of the size, which helps the network. The bigger win is on the CPU: decoding is reading a byte, a varint, a fixed-width double. There is no text to scan and no decimal conversion.
These jobs carried production traffic, so being faster wasn't enough. Every job moved behind a feature flag and went out in stages, with the old path ready to take traffic back. Before each cutover I wrote down what failure would look like and what to do about it, and turned that into runbooks. None of it is clever. It's what makes it safe to move quickly, and in a migration like this, being able to undo a step instantly is worth more than any single optimization.
Both compute usage and CPU cycles fell by a large factor, and CPU cycles fell further. That gap is expected once you look at what each measures. CPU cycles count work done, while a compute-usage figure also reflects capacity a job holds whether or not it's busy, so removing per-record work shows up more sharply in cycles than in reserved capacity.
The habits from this project carried over: look for what is repeated per unit of work, price the data format before anything else, and never ship a change you can't take back in one step.