The practice of observability within organizations hasn’t changed much: anticipate possible failures, monitor the signals, and wire up alerts. The usual suspects: RUM, RED, USE, HTTPxx codes, latency percentiles, RPS, CPU %, memory usage, the load avg trifecta [1m/5m/15m].

Yet, despite this vast amount of telemetry – an average enterprise produces terabytes of telemetry per day1 – we often struggle to answer even the most basic questions about an incident.

The issue, as I see it, is that we never directly observe cause and effect in our telemetry. If we plot them on a dashboard, we often see correlations among the signals. It also doesn’t help that, in production, “too many” things happen at once. A marketing campaign might cause a surge in traffic exactly when a new deployment introduced a regression. What caused the latency spike?

Good alerts and playbooks can quickly tell us something is wrong and help stop the incident. But they don’t necessarily tell us why it happened. A playbook might mitigate the symptom without addressing the cause, making it likely for the incident to reappear.

Engineers carefully reason over the metrics and piece together a Root Cause Analysis (RCA). While I get the appeal of playing “detective”, it is generally error prone. Can we automate what our brains have been implicitly doing?

Causal Machine Learning

While looking for a better way to reason about incidents, I came across Causal Machine Learning which looked promising, and to my luck, some good folks at Microsoft and AWS have already done much of the heavy lifting in a library called DoWhy . The documentation does an excellent job of showcasing practical applications of causal modeling.

To put DoWhy into practice, imagine you’re running a simple three-tier web app – frontend, backend, database.

On a normal day at some hour, your operations look something like this:

Dashboard showing normal system operation
Fig. 1: Normal system operation

All data generated is synthetic to show if DoWhy works, check out Behind the Scenes below

Then, one day, you’re staring at this:

Dashboard showing the incident with a deployment regression
Fig. 2: Campaign and deployment with a regression

The dashboard gives us a lot of context about what changed, but it still doesn’t tell us which explanation to favor: Did the deployment introduce a regression? Or is this simply what the system would look like at this level of traffic that the campaign brought in?

Here’s another deployment without the regression while keeping everything else unchanged.

Dashboard showing the campaign and deployment without a regression
Fig. 3: Campaign and deployment without a regression

Once again, there’s the same spike in traffic. Latency also shifted though not nearly as much as before. The campaign and deployment took place just as they did in the previous incident. Yet this time, we controlled the deployment to not have a regression.

This dashboard likely already gives you some clue about the plausible relationships among various metrics, but remember: in production, we don’t get to simply flip the toggles and observe what happens.

So how does one tell them apart? In the first incident, you’d want to investigate the deployment. In the second incident, you could’ve safely ignored the deployment and looked elsewhere.

Hello DoWhy

Causal Graph

A causal graph is a DAG describing the cause-and-effect relationships between different variables. DoWhy has an experimental Graphical causal model based inference which is what we’re gonna use.

The graph gives the model a structure to work with. Domain experts in your organization can encode what they know about the system’s relationships over time as best as they can.

Here’s one possible causal graph for our example which should be fairly self-explanatory.

  graph LR
      Campaign --> Traffic
      Traffic --> CPU
      Traffic --> DBLoad
      Deployment --> CPU
      Deployment --> DBLoad
      CPU --> Latency
      DBLoad --> Latency

A campaign may influence traffic, which in turn affects CPU usage and database load. A deployment can also affect CPU and database load independently of traffic. Both CPU usage and database load contribute to the request latency.

Individual teams could build and maintain causal graphs for the systems they understand best. In a large organization, a platform team could compose them together into an organization-wide view, allowing outages to be reasoned about across team boundaries if desirable.

Demo

Let’s now look at what DoWhy has to say about the two incidents. We’ll use the Distribution Change recipe for this. The question it answers is:

Distribution Change explains why a target variable changed between two datasets by attributing that change back to the causal mechanisms in the graph that contributed to it.

It attributes the change in a variable (e.g., latency) back to the nodes in the causal graph that may have caused it, and quantifies how much of the change each one was responsible for.

The following summarizes the distribution change DoWhy predicts for Fig. 2. Notice that it hasn’t singled out a single root cause.

Attribution of the latency change in the incident with a regression
Fig. 4: Latency change attribution for Fig. 2

We know from Fig. 2 that our mean latency increased by roughly 100ms. The model attributes about 67ms to the deployment and 34ms to the campaign.

Well, at this point, it feels like I’m just throwing random numbers at you and saying, “Trust me, bro.”

Let’s see what happens when the deployment introduces no regression? Here’s the attribution for the second experiment, Fig. 3.

Attribution of the latency change in the incident without a regression
Fig. 5: Latency change attribution for Fig. 3

Since we designed this experiment so that the campaign was the only influential event, the shift in mean latency matches the campaign’s attributed impact exactly. This also suggests that the remaining 66ms observed in the previous experiment must be attributable to the deployment.

CPU usage increased in both cases, but it was never the root cause: adding capacity might have relieved the symptom, but fixing the regression is what would have resolved the underlying problem.

Apart from that, DoWhy also provides what causal ML calls intervention. Knowing the root causes is great, but engineers ultimately need to fix the situation. Intervention helps us answer “what-if” kinds of questions. For example, “what-if we vertically scaled the service and reduced CPU utilization to X%, how much would we expect latency to improve?”. Now that’s valuable! More on this in a follow-up post soon.

Footguns

Causal ML is still a tool and the literature is clear about the possibility of misleading results when the graph or data are insufficient. Imagine the deployment never had a regression, but there was an unknown factor affecting CPU usage that wasn’t modelled in our graph. If the data doesn’t contain enough variation to distinguish that factor from the deployment, DoWhy might attribute the latency effects back to the deployment.

That doesn’t make the approach unusable. Luckily production has enough chaos. We can also continuously revise the graphs as we better understand what impacts our systems, add more telemetry, and use canary deployments to produce useful variation & comparison groups (control/treatment).

On this topic, I’d recommend reading about Confounders, Colliders, Mediators .

Applications

Cloud FinOps

Suppose your AWS bill jumps by 30% this month. Was it the increase in traffic? A new deployment consuming more resources? The team switching to larger instances? More expensive queries? Discount expiring? I’ve seen teams spend days figuring out what’s moving their cost up.

Debugging slow queries

This one has been particularly painful & seems like a good fit for causal attribution. Instead of showing which metrics moved alongside a slow query, a causal model could estimate how much each of those changes (CPU and I/O pressure, buffer cache hits/misses, lock waits, scan size, sorting) actually contributed to the slowdown instead of trial-and-error which is often the norm.

Automatic remediation

In our example, the same latency alert can have two different action items: investigate a release or accommodate more traffic. Rolling back a healthy deployment won’t make the campaign go away. Adding capacity might help, but you’re paying for the regression.

Causal attribution could help automations choose the right action.

Manageable on-calls and better postmortems

Enterprises face an awkward trade-off: keep rotations fine-grained, with every team carrying its own on-call burden, or consolidate them into fewer rotations. The former is expensive (hourly wage is a thing in some countries), the latter puts unfamiliar systems in front of you increasing on-call anxiety.

If causal attribution can narrow down the likely source, you’ve got free money.

Closing Thoughts

This demo was deliberately small and used synthetic data. Applying it to production systems might require more work. Still, the possibility of turning telemetry and domain knowledge into an explanation is a useful capability.

I hope this has intrigued you enough to question whether our current approach to observability is really state-of-the-art. Surely, what we need isn’t yet another time-series database.

If you don’t want to set all of this up yourself, Datadog offers a similar RCA capability, though I haven’t tested it. That said, it’s still worth understanding the underlying ideas, since their applications go well beyond observability.

Here are some books you can follow up on:

HMU if you think I’ve got anything wrong here :)

Behind the Scenes

The synthetic data was generated with the following Python function, which returns N samples for each variable:

def samples(
    n: int,
    rng: np.random.Generator,
    *,
    campaign: bool = False,
    campaign_multiplier: float = 2.0,
    deployment: bool = False,
    regression: bool = False,
) -> pd.Dataframe:
    campaign_effect = np.full(n, campaign_multiplier if campaign else 1.0)
    traffic = rng.normal(100, 8, n) * campaign_effect

    deployed = np.full(n, 1.0 if deployment else 0.0)

    # traffic naturally raises cpu
    cpu = 15 + 0.3 * traffic + rng.normal(0, 3, n)

    # new deployment adds extra cpu overhead (synthetic, so we know!)
    if deployment and regression:
        cpu += 35 + rng.normal(0, 2, n)

    cpu = np.clip(cpu, 0, 100)

    # db load mostly follows traffic
    db_load = 20 + 0.28 * traffic + rng.normal(0, 3, n)

    latency = (
        50
        + 2.0 * cpu
        + 0.5 * db_load
        + rng.normal(0, 12, n)
    )

    return pd.DataFrame({
        "traffic": traffic,
        "campaign": np.full(n, 1.0 if campaign else 0.0),
        "deployment": deployed,
        "cpu": cpu,
        "db_load": db_load,
        "latency": latency,
    })

Fig. 1: sample(100, rng, campaign=False, campaign_multiplier=None, deployment=False, regression=False)
Fig. 2: sample(100, rng, campaign=True,  campaign_multiplier=1.4,  deployment=True,  regression=True)
Fig. 3: sample(100, rng, campaign=True,  campaign_multiplier=1.4,  deployment=True,  regression=False)

The model never sees this generator code. It learns from the generated observations and the causal graph.

  1. The Observability Cost Crisis  ↩︎