From Spreadsheets
to Error Budgets

SLOs in Code, Starting with CloudWatch

I started by getting SLO definitions out of tool UIs and into version control. I wrote a Terraform module that defines SLOs as code for AWS CloudWatch, the primary observability platform for most Loyalty services, and moved every definition into a central SLO monorepo: versioned, reviewable, team-owned.

That gave teams their first automated reliability data without asking them to change how they thought about their services yet.

Writing them in code also let me make the SLIs sharper than a tool UI allows. I made latency thresholds conditional within a single SLI definition, so what counts as a "good" event depends on the combination of HTTP method and API route. A GET on a read endpoint is held to a tight threshold; a POST on the same API that legitimately does more work gets its own. That gives one SLI and one error budget, with thresholds that match how each endpoint actually behaves, instead of a single loose number that hides read regressions or a tight one that cries wolf on every write.

This applies an Observability 2.0 instinct to SLOs. I keep the high-cardinality dimensions, method and route, inside the definition of a good event instead of flattening them into one pre-aggregated latency number and setting a single threshold on top. The SLI gets to be as specific as the system it measures.

Sliisloo: One View Across All Tools

With CloudWatch SLOs flowing, fragmentation became the next problem. CloudWatch, Dynatrace, and Splunk each showed only their own services, so I built a normalisation layer.

Sliisloo is the Splunk application I wrote for it, and it runs inside the Splunk the organisation was already paying for. It pulls 10-minute samples of SLO state into a shared index from CloudWatch, Dynatrace, Sentry, and synthetic monitors. For Splunk-native log-search SLOs I added a report that pushes records into the same index. One schema. One dashboard. Regardless of source.

Most of the work was in the seams. CloudWatch will hold an SLO object but won't show it next to anything else; Splunk will visualise anything you can get into an index but has no native concept of an SLO. Neither limitation was going to change, so Sliisloo is the set of small components that make one tool's output legible to the other. No new platform, no migration, no second place to look.

I applied the same Observability 2.0 principle at the platform level: one event store as the source of truth, with the status page, trends, and historical reporting all derived from it, instead of four tools each maintaining a partial picture nobody can reconcile.

Beyond status and trends, I gave Sliisloo an annotations layer: engineers log what happened at any point on the SLO timeline, whether that is an incident, a deployment, or a campaign launch. The context lives alongside the data, so the chart tells the full story, not just the numbers.

The name came from the team, not from me. One of the Loyalty engineers coined Sliisloo (pronounced SlìSlò), a portmanteau that solved a genuine problem. I'm Czech, the team is Finnish, and saying "S-L-I, S-L-O" aloud in every meeting was a mouthful in either language. It stuck, and ended up an internal brand, stickers included.

I took that as a good sign. People don't nickname something they're merely tolerating, and by that point the team knew the concepts well enough to play with them.

Burn Rate Alerting + PagerDuty

With SLOs measured and visible, I made them actionable. I configured burn rate alerting for every service, so alerts fire when the error budget burns too fast, before a breach becomes inevitable. That replaced the alert-on-everything default with a signal that tells on-call engineers what actually matters. I wired PagerDuty in as the alerting destination to close the end-to-end on-call loop.

Workshops: Theory, Then Practice

Ahead of the live session I recorded a 17-minute video covering SLO fundamentals, error budget mechanics, and burn rate calculations, using examples from the Sliisloo dashboard. Engineers watched it at their own pace as preparation.

For the live workshop I wrote a fictional scenario: teams took a made-up service to production together, defining its SLIs and SLOs as a group exercise, and I facilitated the discussion afterwards. I built the format to stand on its own, so future engineers can work through the same video and workshop to onboard themselves, without me in the room.

Fault Injection Testing

With SLOs defined and alerting running, one question remained: does any of this actually work under failure? I set up fault injection testing on core Loyalty services, deliberately inducing failures to confirm the SLIs measured them accurately and the burn rate alerts fired as expected. Along the way we documented the failure symptoms the teams had never mapped.

Business Status Page

I built a service status page powered entirely by live SLO data. No manual updates. No engineer needed to interpret it. Leadership reads system health at a glance, and it feeds straight into wider business reporting.

SLOs as Default Practice

The shift that made the program stick wasn't a technical one. Putting an SLO on a new service still takes deliberate work. Someone writes the definition, picks the thresholds, opens the pull request. Nothing lands automatically. What changed is that nobody argues about whether to do it.

Before I started, measuring a new service meant building the measurement first: no module to call, no repository to add to, no agreed definition of a good event to borrow. That made an SLO easy to defer, and deferring it was the normal outcome. With the module and the monorepo in place, the work is small enough that skipping it is harder to justify than doing it.

So teams now treat an SLO as part of shipping a service rather than something to retrofit once it's already in production. That expectation is the durable outcome. The tooling only made it cheap enough to hold.