The client is pushing, but you have no idea why the system is slow? It's not entirely your fault

On paper, Observability should be a tool accessible to every developer, product manager, and operations person. But the reality is completely different.

By Yuval Susmanovich

You are probably familiar with the following scenario: the system is running in production, it looks fine, but suddenly the client reports that it is slow, or that a feature is not working as expected. You start searching: you examine logs in one system, review metrics in a second, check traces in a third, and try to cross-reference information to understand if the root cause is in the code, infrastructure, or client data.

The systems we develop do not always behave as expected. Reasons range from missing requirements to unexpected bottlenecks or factors not accounted for in client usage. The responsibility for real-time understanding lies with the development teams.

This is not simple: modern applications consist of dozens of microservices, cloud infrastructure, and databases, each monitored differently. This is where Observability comes in—the ability to observe what is happening inside the system and why, based on metrics, logs, and traces. Instead of spending hours trying to reproduce a failure, we see immediately where the problem occurred, even in cases of "Unknown Unknowns."

Yes, but…

On paper, Observability should be accessible to everyone, but in practice, it often remains the domain of experts like DevOps or SRE teams. There are three main reasons:

Information overload: Dashboards are full of graphs, logs, and complex traces. Developers must dive into a sea of data, perform complex queries, and filter out noise.
Lack of standardization: In many organizations, there is no uniformity. One project uses Node.js, another Python, and a third runs in a different cloud environment. There is no common language or "golden path." Developers moving between projects must learn how to investigate failures from scratch every time.
More code, less time: AI tools allow developers to produce code faster, but the sheer volume of services makes the system so complex that investigators no longer know their own code in depth. Observability requires knowing what to look for, but with the dramatic growth of systems, developers often don't know where to start.

Observability has turned from a tool for answers into a huge database requiring expertise. The natural solution is AI agents, but if every project has a different log structure and toolset, we would need hundreds of different agents. Standardization is the prerequisite for one agent to help the entire organization.

Four steps to creating Observability

Deciding to change how we create Observability requires courage, as it involves time and money. However, breaking the task into clear steps makes the path simpler.

1. Changing the reporting method

First, report Telemetry (metrics, logs, traces) uniformly. OpenTelemetry has become the standard for creating a "golden path." This requires revisiting existing projects to ensure flexibility as development speed increases.

2. Moving to open-source projects

Next, replace edge technologies. We opted for open-source projects instead of paid providers (Grafana, Coralogix, Dynatrace, New Relic, Datadog), choosing Prometheus (metrics), OpenSearch (logs), and Jaeger (traces). This ensures data remains under our control and allows us to process it according to our needs.

3. Intelligent Alerting infrastructure

Once telemetry is aligned, build a proactive alert array. The goal is to identify anomalies and bottlenecks in real-time. This is a "breathing" process: as user bases and loads grow, constant tuning is required to minimize False Positives. Alert definitions are managed in a Git repository and deployed via Helm.

4. Making Observability accessible via IDP

We built an IDP (Internal Developer Portal) that makes telemetry accessible for individual projects and in consolidated views. This allows for seeing correlations between metrics, logs, and traces. Organizations can use Spotify’s Backstage or develop a custom portal for more flexibility.

The next evolution is connecting an intelligent agent to monitoring tools via MCP. Since all projects use the same format, one high-quality agent can serve everyone via a chatbot in the IDP.

Not just replacing one tool with another

Two years later, we have achieved fundamental changes:

No dependency on a select few: Information is accessible to everyone, and every developer can investigate problems independently.
Super-agent: We built an agent that works across all data, providing full analysis within minutes with 90% accuracy.
Predictive systems: We use ML algorithms to alert about potential problems based on past patterns, allowing us to prevent them in advance.

Ultimately, the "golden path" of Observability is a conceptual change: moving from a world where every team chooses its own tools to a world with a uniform language and infrastructure. In the AI era, this is not a luxury, but a condition for an organization to evolve, automate, and solve problems quickly.