Paving observability: from Grafana to Datadog

Every incident has two problems: the root cause and the path to understanding it. For a long time, the second one was disproportionately expensive at Barte. Not because we lacked tooling, but because we had too many tools, each covering one part of the picture.

Four tools, one incident

Our first complete observability solution was a composition: self-hosted Grafana as the dashboard layer, pulling from several sources (CloudWatch metrics and logs, the database, custom metrics); New Relic for APM (application performance monitoring, request by request) and synthetic health checks; CloudWatch as the log destination for every service on ECS; and Opsgenie as the middleman for alerts, born in Grafana or New Relic, before they reached Slack.

Each of those pieces was a reasonable choice at the moment it came in. The problem was not any single tool, but the composition. Investigating an incident meant opening the dashboard in Grafana, jumping to APM in New Relic and searching logs in the CloudWatch console, correlating the three views manually. The alert that kicked all of this off arrived in Slack after two steps and with little context: it said something was wrong, but rarely pointed to where to start. Keeping it all running also had a price: self-hosted Grafana was one more system for us to operate, with infrastructure, upgrades and access control on our plate.

There was also a quieter cost: every new service repeated the same decisions. How to log? What to measure? At what threshold to alert? Which channel? Each team answered in its own way, and the answers drifted apart over time.

Observability needs a paved road too

If you’ve read our post on the paved road, you already recognize the pattern: energy spent on repeated decisions is energy that doesn’t become product. Monitoring was exactly one of those cases.

That’s how we framed the migration to Datadog: not as a tool swap, but as building a paved road for observability. Centralizing logs, metrics, traces (the trail a request leaves as it crosses services) and alerts in one place was the visible half of the work. The other half was turning observability best practices into the easiest path, so that the right way to monitor a service would also be the fastest way.

It helps to describe the ground this road would be built on. Our core stack is Kotlin with Spring Boot, running in containers on ECS Fargate, on AWS. Asynchronous communication between services goes through SQS and SNS, infrastructure is declared in Terraform and CI/CD runs on GitHub Actions. That’s the scene where telemetry is born, and instrumentation is where the paving began.

Swap the backend, not the instrumentation

Trace migration could have been the most expensive part. Re-instrumenting several services is the kind of project that spans quarters. In our case it wasn’t, thanks to a decision made well before: our services were already instrumented with the OpenTelemetry agent, exporting to a tracing backend that ran internally (New Relic still covered APM for part of the services, but the new path was already OTel).

Because OpenTelemetry is an open standard, migrating meant pointing the same agent at a different destination. The -javaagent stayed the same; only the delivery address changed:

environment = [
  # the same OTel agent as before, only the destination changed
  { name = "OTEL_EXPORTER_OTLP_ENDPOINT", value = "http://localhost:4318" },
  { name = "OTEL_EXPORTER_OTLP_PROTOCOL", value = "http/protobuf" },
  { name = "OTEL_SERVICE_NAME",           value = "my-service" },
]

That localhost:4318 is the central detail: traces are now received by a Datadog Agent sidecar running in the same ECS task, with the OTLP receiver (OpenTelemetry’s export protocol) enabled. The application doesn’t know the backend changed. If we ever want to switch again, the cost goes back to being an environment variable.

This instrumentation brings another benefit that supports everything else: context propagation. The agent injects and reads trace headers on calls between services, so the same trace_id follows a request end to end. A flow that starts in one service, goes through a queue and finishes in another shows up as a single distributed trace, and the logs of every service involved carry the same trace_id. The correlation that used to be done manually, tool by tool, became one click.

The lesson we took away: instrumentation built on an open standard is freedom bought in advance. The upfront cost is small, and the value shows up precisely on the day a migration like this becomes necessary.

The before and after in the task definition

In practice, the switch happened in the ECS task definition (the file that declares which containers run together for a service). The old one had one container: the application, with the awslogs log driver sending everything to CloudWatch.

flowchart LR
  A1[app] -- awslogs --> CW1[CloudWatch Logs]
  A1 -- OTLP --> T[Internal tracing backend]

The new one has three containers:

flowchart LR
  A2[app] -- OTLP localhost:4318 --> DA[datadog-agent]
  A2 -- stdout JSON --> FB[log_router / Fluent Bit]
  FB -- FireLens --> DD[Datadog Logs]
  FB -- copy --> CW2[CloudWatch Logs]
  DA -- traces + metrics --> DD2[Datadog]

1. The application, now shipping logs to Datadog via FireLens (the ECS mechanism for routing logs to destinations other than CloudWatch):

logConfiguration = {
  logDriver = "awsfirelens"
  options = {
    Name       = "datadog"
    Host       = "http-intake.logs.datadoghq.com" # adjust to your account's site
    dd_service = "my-service"
    dd_source  = "java"
    dd_tags    = "env:prod,team:my-team"
    TLS        = "on"
    provider   = "ecs"
  }
  secretOptions = [
    { name = "apikey", valueFrom = "<secrets-manager-secret-arn>" }
  ]
}

2. The datadog-agent sidecar, which receives OTLP traces and metrics from the application and handles APM:

{
  name      = "datadog-agent"
  image     = "public.ecr.aws/datadog/agent:7.76.1"
  essential = false
  environment = [
    { name = "ECS_FARGATE",    value = "true" },
    { name = "DD_SITE",        value = "datadoghq.com" }, # same: your account's site
    { name = "DD_APM_ENABLED", value = "true" },
    { name = "DD_OTLP_CONFIG_RECEIVER_PROTOCOLS_HTTP_ENDPOINT", value = "0.0.0.0:4318" },
    { name = "DD_ENV",     value = "prod" },
    { name = "DD_SERVICE", value = "my-service" },
    # drops the whole trace when the root span matches (health checks)
    { name = "DD_APM_IGNORE_RESOURCES", value = "GET /actuator.*" },
  ]
  secrets = [
    { name = "DD_API_KEY", valueFrom = "<secrets-manager-secret-arn>" }
  ]
}

3. The log_router, a Fluent Bit (a lightweight log collector) managed by FireLens: a container with firelensConfiguration = { type = "fluentbit" } and the aws-for-fluent-bit image, routing the application’s stdout. It exists because, on Fargate, the Datadog Agent cannot see other containers’ stdout; logs on Fargate leave via FireLens or directly through the API. Our logs were already structured JSON, which simplified landing in Datadog: each log field becomes a search filter (a facet), and since every line already carries the trace_id, Datadog links logs and traces automatically (the default remapping recognizes the OTel format).

All of this is Terraform living in the service’s own repository, with the Datadog API key coming from Secrets Manager, so there is no plaintext credential in the task definition. The deploy pipeline (GitHub Actions) only updates the image: Terraform owns the task’s structure, and the deploy owns the version that runs.

One destination to investigate, another to store

The diagram above has a detail that tends to raise questions: logs go to Datadog and keep going to CloudWatch. That is not indecision; it is a tiered retention architecture.

Logs have two consumers with opposite needs. Whoever investigates an incident wants fast search, facets and correlation, and almost always looks at the last few days. Whoever audits (and a fintech lives with audits) needs years of history, rarely queried. Indexing years of logs in an investigation tool is expensive; investigating an incident in cold storage is too slow.

The split ended up like this: Datadog indexes the hot 15-day window, where investigation happens; CloudWatch keeps the 5-year long tail at a far lower cost than indexing it. Fluent Bit handles this with one extra configuration file (an additional [OUTPUT] pointing at CloudWatch, which the aws-for-fluent-bit init image loads from S3). The same log line leaves the application once and reaches both destinations, each playing a different role.

Monitors as code

Centralizing telemetry solved only part of the problem. We still had to structure what to do with it, and this is where the most important piece of the paving came in: an internal Terraform observability module.

Instead of each team creating monitors by hand in the UI, a service declares what it has, and the module generates the corresponding monitor suite:

module "observability" {
  source = "git::https://github.com/<org>/terraform-modules//terraform-datadog-observability?ref=<version>"

  service_name               = "my-service"
  team                       = "my-team"
  notification_slack_channel = "@slack-team-alerts"

  sqs_queues = [
    { name = "domain-events", dlq_name = "domain-events-dlq" },
  ]

  log_monitors = [
    {
      name         = "flow-escalated-to-manual"
      headline     = "processing escalated to manual intervention"
      query_filter = "@flow:my_flow MANUAL_INTERVENTION_REQUIRED"
      reason       = "the flow exhausted its retries and needs a human"
      threshold    = 0
      window       = "30m"
    },
  ]

  rds_instances = [{ identifier = "my-instance" }]

  dependencies = [
    { name = "my-external-dependency", host = "api.dependency.example" },
  ]
}

Declared an SQS queue? The module creates monitors for the DLQ (dead-letter queue, where failed messages end up) and for aging messages. Declared an HTTP dependency? It creates 4xx/5xx rate and p95 latency monitors (the time within which 95% of requests complete) for that integration, built on metrics Datadog automatically derives from the traces. Declared a critical log pattern? The module generates a log alert, and the reason field goes into the alert message as a short action guide (a runbook).

The module carries the conventions that used to be decided case by case: naming ([env][service] resource condition), severity mapped to priority, standardized tags (service, team, sli) and notification channel. It also brought a valuable side effect: alerts now go through code review. Creating a monitor means opening a PR, with history, discussion and rollback, like any other production change.

It’s the same move we described in engineering principles: decide once, in the convention, what used to be decided every week, in every team.

The alert’s path got shorter

In the old design, an alert was born in Grafana, became a ticket in Opsgenie, and only then showed up in Slack. Two steps, little context.

Today the monitor notifies Slack directly, and the message arrives ready for action: what fired, in which service and environment, why it matters (the reason declared in the module) and a link to the query already scoped to the problem window. The question in the channel stopped being “does anyone know what this is?” and became “who’s taking it?”.

What was hidden became visible

Perhaps the subtlest change was one of posture. In the old design, incident response was essentially reactive: we saw whatever someone had built a dashboard or an alert to see, and everything else only surfaced when it became an incident.

Here is something worth calling out: much of the new visibility required no building at all. With services instrumented and the AWS integration on, Datadog generates on its own a set of views that used to depend on someone creating a dashboard: the service dependency map, latency and error rate per endpoint and per external integration (derived from the traces themselves), exceptions grouped by pattern with the volume of each one, ready-made dashboards for RDS, SQS and Lambda. All of that comes for free with instrumentation, and for every service at once, whether the team asked for it or not.

It was through those views that hidden problems came to the surface: recurring errors nobody was tracking, queues piling up at specific hours, integrations degrading slowly. What used to be scattered noise became a list we can prioritize.

That visibility also exposed a debt of our own: log classification. Messages marked as errors that require no action, important warnings hidden at info level, exceptions too generic to group well. We have ongoing work to reassess log levels and error classification, and it became much easier now that we can see, by pattern and by volume, what each line actually represents.

What was left behind, on purpose

No migration ever finishes at 100%, and the leftovers are worth recording:

• One custom metric is still emitted through both paths. The old one, through CloudWatch, and the new one, through OpenTelemetry, coexist until the last consumer migrates.
• One service is already instrumented but has not declared its monitors in the module yet. The paved road is ready, but not everyone has finished merging onto it.

Knowing exactly what was left behind, and why, is worth more than a migration that merely looks complete.

Telemetry became context for AI

One effect of centralization that wasn’t in the original plan: it prepared the ground for AI. With logs, metrics, traces and monitors in one place, under standardized tags and conventions, telemetry became consumable not only by people but also by agents.

Today we make heavy use of the Datadog MCP: AI agents investigate an alert by querying logs, traces and metrics directly, following the same trail an engineer would. And access stopped being an engineering exclusive: people from other areas can ask questions about system behavior in natural language, without learning each tool’s query language.

We had already written in the paved road post that standards and guardrails are what make it safe to automate with AI. Centralized observability is one more example of that: AI can only investigate well because the road is paved.

What changed in the day to day

In the end, the value of the migration shows up in the routine:

• Investigation in one place. From alert to log, from log to trace, from trace to metric, without switching tools and without correlating anything by hand.
• The same trace_id end to end. A flow that crosses several services is a single trace, and the logs of all services meet through that identifier.
• Alerts that arrive ready for action, with context and a link to the exact problem window, straight in Slack.
• New services are born monitored. Declaring queues, dependencies and log patterns in the module is part of the setup, and monitors go through review like any code.
• Telemetry accessible beyond engineering, by people and by AI agents, via MCP.
• Nobody keeps an observability tool running anymore. Self-hosted Grafana was a system of ours to operate; Datadog is managed, and the energy that went into the tool now goes into the monitors.

Detection, however, is only the beginning of an incident. Who takes ownership? How does communication flow? What becomes learning afterwards? For that chapter, a new piece entered our ecosystem: incident.io, integrated with everything described here. This was the first case study in the series we promised when we launched the blog; how incidents became a process, with an owner, a timeline and a post-mortem, is the subject of the next post.