LLM Observability Tracing: A Guide to OpenTelemetry Integration

Tracing an LLM pipeline with OpenTelemetry is the easy half. The harder decisions are what you are allowed to store, which traces you can afford to keep, and how to record a cost you do not know. This post shows the instrumentation, then argues for three choices: record metadata for every request and content for few, sample after the trace ends instead of before, and store a missing price as null.

What a trace needs to answer

A request to an AI workflow is a chain: retrieval, a prompt build, one or more model calls, tool calls. A 500 status will not tell you that step three fed the model stale context. A trace should answer two questions without opening application logs: which step was slow, and which inputs did the model see. The first needs span timings. The second needs content, and that is where the trouble starts.

Instrument the model call

Wrap each model call in a span and record the request model, token counts and finish status. The attribute names below follow OpenTelemetry's GenAI conventions, which now live in their own semantic-conventions-genai repository. Check that repository for current names before you standardize on them, since a dashboard built on invented attribute names cannot be shared across tools.

from opentelemetry import trace
from openai import OpenAI

tracer = trace.get_tracer(__name__)
client = OpenAI()

def generate(messages, model):
    with tracer.start_as_current_span("chat " + model) as span:
        span.set_attribute("gen_ai.request.model", model)
        try:
            r = client.chat.completions.create(model=model, messages=messages)
            span.set_attribute("gen_ai.usage.input_tokens", r.usage.prompt_tokens)
            span.set_attribute("gen_ai.usage.output_tokens", r.usage.completion_tokens)
            return r.choices[0].message.content
        except Exception as e:
            span.record_exception(e)
            span.set_status(trace.Status(trace.StatusCode.ERROR, str(e)))
            raise

Give each retrieval, embedding and tool call its own child span under one parent for the user request. Export with an OTLP exporter through a batch span processor so span export does not block the request path. Backends such as SigNoz and proxies such as Helicone read this data, and the transport is described at opentelemetry.io.

Content is the risky attribute

Some tutorials put the full prompt and completion on every span. OpenTelemetry's own attribute registry cautions against it: it says input and output messages are "likely to contain sensitive information including user/PII data", and system instructions may too. The registry also says instrumentations may offer a way to filter or truncate the content. Treat content capture as opt-in per environment. It also multiplies storage, because a prompt with retrieved context can be many kilobytes against a few bytes of metadata.

A stricter option is to keep content out entirely. A per-turn record can be useful without it. Store who asked, how the turn ended, which model handled each call, why it was retried or escalated, the IDs of the tool calls, a stop reason and the duration. The trace then explains the shape of a turn (it retried twice, hit the time limit, was blocked by a guardrail) without holding anything a customer typed. If your data allows content in traces, scrub it at the collector and set a short retention. If it does not, keep IDs that point back to the source system.

Sample after the trace ends

A common recommendation is to keep every failed request and a small share of successes. Head sampling cannot do that. OpenTelemetry's sampling documentation says that with head sampling alone "you cannot ensure that all traces with an error within them are sampled", because the decision is made before the outcome exists. Tail sampling decides after seeing the spans, but the same page notes it needs stateful components that store large amounts of data, possibly many nodes.

A cheaper split for most teams: emit a small metadata record for every request (outcome, model, token counts, duration, a trace ID) and sample only the heavy content. The always-on record gives you error rates and cost per workflow with no sampling bias. The content traces answer why one answer was wrong, and for that you need the bad ones, so decide in the app which to keep (explicit user feedback, validation failure, a slow request) and flag those spans for the collector to retain.

Record unknown cost as null

Token counts are the dependable measurement. Dollar cost depends on a price you may not have, so treat it as an optional extra: take the provider's own reported figure first, then an operator override, then a live price catalog, with no hardcoded price table, since a typed-in number goes stale silently. Reasoning models add a trap even to token counts. OpenAI's reasoning guide says reasoning tokens are not visible through the API but still occupy context window space and are billed as output tokens, and that a response can end as incomplete with the reason max_output_tokens before any visible output exists, so you pay for input and reasoning and get no answer (OpenAI reasoning guide). Record reasoning tokens as their own field and alert on incomplete responses. When no source can price a call, store null, never zero. A zero in a cost column reads as free and hides the gap, so a dashboard that sums it will understate spend with no sign of a problem.

Check the pipeline with injected failures

• Add a five-second delay to the vector search in staging. The trace should show the retrieval span as the slow one, separate from model time.
• Feed the model a deliberately wrong context. Using only the trace (and the source system, if you keep IDs rather than content), your team should be able to find which document the model was given.
• Disable content capture and repeat the second test. If you can no longer debug, add the IDs that are missing.

References

• OpenTelemetry GenAI attribute registry: warning that message attributes are likely to contain sensitive and PII data.
• OpenTelemetry GenAI semantic conventions repository
• OpenTelemetry sampling concepts
• SigNoz: LLM observability with OpenTelemetry
• Helicone
• OpenAI: Reasoning models: reasoning tokens are hidden, billed as output, and can exhaust the output cap before any visible text.

---