How fixing our data infrastructure before it broke us gave back 6,200 engineering hours

Moving before we were forced to

We ended up changing the setup before the bank went live, while we still had enough buffer to do it carefully. We came off the hosted Elasticsearch service and onto open source OpenSearch, and brought in NetApp Instaclustr to run both OpenSearch and Postgres, which underpinned our observability platform. That relationship extended to our ClickHouse analytics clusters not long after, which is where the bulk of our processing actually lives. Importantly, all of these are 100% open source technologies, which could ensure portability without lock-in. The principle behind the move, however, mattered just as much as the technology itself. I wanted my engineers out of the 2 am migration business and the emergency-patch business, and I wanted that time pointed squarely back at the product.

One condition did matter more than the rest, the one where many of these moves quietly die. We sell into regulated industries (like that large bank) and cannot just pour our data into somebody else’s cloud and give up control of where it physically sits. Zero wiggle room, so whatever we picked had to let us keep the data inside our own environment. No amount of “more stable” would have saved us if we flunked that test.

The payoff turned out to be easy to put a number on, and I’ll give you ours because it genuinely changed how I think about all of this. We got back somewhere north of 6,200 engineering hours a year. That was time that used to disappear into the black hole of back-end upkeep and the kind of fire drills that eat a sprint (and whatever weekend you were hoping to have). We have since pushed past 500,000,000 interactions and the trillions of insights our ClickHouse analytics layer chews through, and the performance scares we used to brace for mostly stopped happening. Failovers turned, well, boring, while the upgrades I once lay awake over became just a Tuesday afternoon.