LLM Observability: Tracing, Logging, and Cost Tracking

Fractional CTO · Founder of Injuria.ai

November 2, 2026•11 min read•

LLM Observability OpenTelemetry Software Architecture APM

Applications built on Large Language Models (LLMs) present a unique observability challenge. Unlike traditional software, where logic is deterministic, LLMs produce non-deterministic outputs whose quality, cost, and latency can vary dramatically with each call. Standard Application Performance Monitoring (APM) tools can tell you that an API call to an LLM provider succeeded, but they can't tell you what was generated, whether the answer was helpful or a hallucination, or how much that specific generation cost in fractions of a cent.

LLM observability extends traditional principles to address these new dimensions. It provides the necessary visibility into the content, quality, and cost of LLM interactions, making it possible to debug, optimize, and reliably operate AI-powered features. This involves capturing not just system metrics but the full context of each LLM call: the prompt, the response, the token counts, and performance metadata. By instrumenting applications to capture this data, engineering teams can move from treating LLMs as opaque black boxes to managing them as transparent, measurable components of their software stack.

The Core Mechanism: Extending Distributed Tracing

The foundation of modern LLM observability is distributed tracing. This concept, originally detailed in papers like Google's on its Dapper system^[1], involves tracking a single request as it flows through multiple services in a distributed architecture. The industry has largely standardized around OpenTelemetry^[2] (OTel), a project of the Cloud Native Computing Foundation^[3] (CNCF), which provides a unified set of APIs, SDKs, and data formats for telemetry.

Here is how the mechanism works, step by step, for an LLM application:

Instrumentation. Your application code is instrumented, either manually using an OTel SDK or automatically by an agent. This instrumentation wraps key operations, such as incoming web requests, database queries, and, most importantly, calls to LLM clients (like the official OpenAI or Anthropic libraries).
Trace and Span Creation. When a request initiates a workflow (e.g., a user submits a form), the instrumentation creates a root span and assigns the entire workflow a unique trace ID. A span represents a single unit of work, like a function call or an API request. As the request progresses, new child spans are created for each subsequent operation. Each child span records its own ID and the ID of its parent span, allowing the entire sequence to be reconstructed.
Context Propagation. The trace ID and current span ID (known as the trace context) must be passed between services. This is typically done via HTTP headers. The W3C Trace Context specification^[4] defines standard headers like traceparent that ensure interoperability between different systems and tracing tools. When your application calls an external service like a vector database or an LLM API, the OTel instrumentation automatically injects these headers into the outgoing request.
Capturing LLM-Specific Attributes. This is the critical extension for LLM observability. The specialized instrumentation for LLM client libraries captures details specific to the AI interaction and attaches them as attributes to the corresponding span. The OpenTelemetry semantic conventions for GenAI define standard names for these attributes, but the llm.* namespace is legacy. The current conventions use the gen_ai.* namespace. For example, llm.request.model is now gen_ai.request.model, and llm.usage.prompt_tokens is gen_ai.usage.input_tokens.
Export and Visualization. The collected spans (with all their attributes) are exported from the application via an OTel exporter. They are sent to an observability backend, which can be an open-source tool like Jaeger^[5] or a commercial APM or LLM-specific platform. The backend ingests this data, reconstructs the traces, and provides a user interface for searching, filtering, and visualizing the entire request lifecycle as a waterfall diagram. This allows an engineer to see the duration of each step and inspect the attributes of any given span, including the full LLM prompt and response.

A Concrete Example: Tracing a RAG Application

Retrieval-Augmented Generation (RAG) is a common pattern for building chatbots that can answer questions over a specific set of documents. A RAG application involves multiple steps, making it a perfect candidate for distributed tracing.

Consider a Python application using LangChain to interact with OpenAI and a Pinecone^[6] vector database. Here’s a simplified example of how you might set up OpenTelemetry instrumentation.

# app.py
# (Assumes OTel SDKs, exporters, and instrumentation libraries are installed)
# E.g., pip install opentelemetry-api opentelemetry-sdk opentelemetry-exporter-otlp opentelemetry-instrumentation-openai

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.openai import OpenAIInstrumentor

# --- OTel Boilerplate: Configure the tracer and exporter ---
trace.set_tracer_provider(TracerProvider())
tracer = trace.get_tracer(__name__)

# Configure an exporter to send data to an OTel-compatible backend
otlp_exporter = OTLPSpanExporter(endpoint="https://api.your-observability.dev/v1/traces")
span_processor = BatchSpanProcessor(otlp_exporter)
trace.get_tracer_provider().add_span_processor(span_processor)

# --- Auto-instrumentation for the OpenAI client ---
OpenAIInstrumentor().instrument()

# Assume 'rag_chain' is a pre-configured LangChain RAG chain
# that handles embedding, vector search, and the final LLM call.

def answer_question(query: str):
 # This creates the root span for our trace
 with tracer.start_as_current_span("answer_question") as span:
 span.set_attribute("user.query", query)
 
 # When this is invoked, the OpenAIInstrumentor will automatically create
 # child spans for any calls made by the chain to the OpenAI API.
 # If the chain also called a database, we would instrument that client too.
 response = rag_chain.invoke(query)
 
 span.set_attribute("app.response", response)
 return response

# Example usage
user_query = "What is the policy on remote work?"
answer = answer_question(user_query)
print(answer)

When answer_question is called, a trace is generated that might look like this in an observability platform:

answer_question (Root Span): The overall operation, lasting 5.5 seconds.
openai.embeddings (Child Span): A 150ms call to OpenAI's embedding API to vectorize the user's query. Attributes on this span would include the model name (text-embedding-ada-002) and token counts.
pinecone.query (Child Span): A 300ms call to the vector database to find relevant document chunks. Attributes would include the namespace and the number of results returned.
openai.chat.completions (Child Span): The main LLM call, lasting 5 seconds. This span is information-rich. An inspector would show:

Prompt: A long string containing the system message, the retrieved document chunks, and the original user query.
Completion: The final answer generated by the LLM.
Attributes: llm.request.model: gpt-4o, llm.usage.prompt_tokens: 3102, llm.usage.completion_tokens: 158.

This unified view instantly reveals bottlenecks (e.g., slow vector search), errors in any step, and the exact data flowing between components. An engineer can debug a bad answer by examining the context retrieved from Pinecone and the final prompt sent to GPT-4.

Beyond Tracing: Evaluation, Logging, and Cost

While tracing provides the narrative structure, a complete observability solution requires more.

Evaluation (Evals)

It's not enough to know that an LLM generated a response; you need to know if the response was good. Evaluation, or "Evals," is the process of scoring the quality of LLM outputs. This can be done in several ways:

Heuristic Evals: Simple checks, like measuring if the output is valid JSON, is below a certain length, or does not contain specific flagged words.
Model-Based Evals: Using another, often more powerful, LLM call to act as a judge. For example, you can use GPT-4 to score a GPT-3.5 response on a scale of 1 to 5 for correctness, conciseness, or adherence to a specific persona. The prompt to the judge model might be: "Does the following response accurately answer the user's question based on the provided context? Answer only 'yes' or 'no'."
Human Feedback: The ultimate ground truth. Integrating a mechanism for users to provide feedback (e.g., a thumbs-up/thumbs-down button) and attaching that feedback to the corresponding trace provides an invaluable dataset for fine-tuning models and improving prompt templates.

These evaluation results are typically stored as additional attributes on the main LLM span, allowing teams to search for all traces with negative feedback or low quality scores.

Logging

Traditional logging remains crucial. While traces excel at capturing the request flow, logs are better for recording discrete events, background job outputs, or rich, unstructured state information. The best practice is to correlate logs with traces by injecting the trace_id and span_id into every log message. When viewing a trace, you can then jump directly to all the logs associated with a specific span, providing deep context for debugging.

Cost Tracking

LLM usage is metered by tokens, and costs can accumulate quickly. Relying on the monthly invoice from your provider is insufficient for operational control. LLM observability platforms solve this by using the token counts captured in the trace data. The platform maintains a rate card of model prices (e.g., from the OpenAI pricing page^[7]) and calculates the cost of every single LLM call in real-time:

Cost = (prompt_tokens × price_per_input_token) + (completion_tokens × price_per_output_token)

This data is then attached to the span. It allows engineering leaders to aggregate costs and answer critical business questions: What's the average cost per user query? Which feature is driving the most LLM spend? Is a new prompt template more or less expensive than the old one? This granular cost analysis is impossible without deep instrumentation.

The Decision Lens: Tradeoffs and Adoption Strategy

Implementing LLM observability involves clear tradeoffs. Technical leaders must weigh the benefits against the costs.

When to adopt it:

When your application moves from a prototype to a production system with real users.
When your LLM logic involves multiple steps (e.g., RAG, agents) and debugging becomes complex.
When you need to monitor and control rising LLM costs.
When you need to track response quality and user feedback to improve the product.

When to delay:

During initial, small-scale experimentation. The overhead of setting up an observability pipeline can slow down rapid prototyping where print() statements and notebooks are more efficient.

Costs and Considerations

Performance Overhead: Instrumentation is not free. SDKs add a small amount of CPU and memory overhead to your application. Exporting telemetry data consumes network bandwidth. For very high-throughput applications, you may need to implement sampling, where only a percentage of traces are collected to manage performance and cost.
Financial Cost: Observability platforms are typically priced based on data volume (gigabytes of spans ingested) or number of traces. This SaaS subscription adds to your operational expenses. Self-hosting open-source alternatives like Jaeger and Prometheus^[8] avoids subscription fees but incurs engineering costs for setup, maintenance, and scaling.
Data Security and Privacy: Prompts and completions can contain highly sensitive user data or proprietary company information. Sending this data to a third-party observability vendor requires rigorous security vetting. Look for vendors with strong compliance certifications like SOC 2 and features for data masking, redaction, and PII detection to mitigate risk.
Implementation Complexity: While auto-instrumentation simplifies setup, a full-fledged observability practice is another system for your team to learn and manage. It requires an investment in training and process.

Common Misconceptions

Misconception: "My cloud provider's logs are sufficient."
Correction: Logs from AWS CloudWatch^[9] or Azure Monitor^[10] can show that an API call was made. However, they don't link that call to the initial user request or other internal service calls. They provide a list of isolated events, not a coherent, end-to-end trace. They also don't typically parse the LLM-specific payload to provide quality or cost analysis out of the box.
Misconception: "LLM observability is just another name for APM."
Correction: It builds on APM principles but is a distinct discipline. Traditional APM is largely blind to the content and meaning of data payloads. LLM observability's primary focus on the prompt/response text, quality evaluations, and token-based cost calculations is a significant departure from monitoring CPU, memory, and latency alone.
Misconception: "I can just install OpenTelemetry to solve this."
Correction: OpenTelemetry is a specification and a collection of tools (SDKs, collectors) for generating and collecting telemetry data. It is not a complete, standalone backend. You must pair OTel instrumentation with a compatible backend, either a commercial vendor or a self-hosted open-source system, to store, visualize, and analyze the data.

Where We Are Today

LLM observability is a rapidly maturing field. Adoption is becoming standard practice for any serious LLM-powered application. OpenTelemetry has won the standards war for instrumentation, providing a vendor-neutral foundation for the ecosystem. As of mid-2026, the GenAI semantic conventions are still not stable, but their status is "Development", not "experimental". This status means they can change without backward compatibility guarantees. Some core attributes are considered mature, but the overall specification is evolving and even moved to a separate repository in June 2026 to allow for faster iteration.

The tooling landscape is vibrant. Established APM leaders are retrofitting their platforms with LLM-specific features. Simultaneously, a new class of LLM-native observability platforms has emerged, offering deeper, more specialized workflows for prompt management, versioning, and integrated evaluation. While open-source tools like Jaeger are excellent for tracing, the more advanced features like automated cost analysis and sophisticated evaluation frameworks are currently a major differentiator for commercial products.

Sources (10)