Entry 022Comparisons · Design7 min readBy Mandare Labs

LLM observability vs audit trail: what a trace can't prove

Observability answers "why did this run behave like that". An audit trail answers "what did the agent do, and can anyone have changed this record". Langfuse and LangSmith document traces as a debugging tool: the SDK runs in your application, traces can be deleted by API, and LangSmith expires them after 14 or 180 days. Keep both, and do not use one as the other.

A trace is built to explain a run, not to prove one

Tracing tools exist to find out why a request went wrong or slow. Langfuse's overview says each trace captures "LLM calls, retrieval steps, tool executions, and custom logic", along with "timing, inputs, outputs, and metadata", and that this "gives you full visibility into what happened during each request, enabling debugging, performance optimization, and quality monitoring."

That is a debugging promise. It is a good one, and it is the right tool for it. Nothing in that text says the record is complete, unaltered or kept for a fixed period. The tool is not at fault: it was not built to carry that load.

An audit trail has a different job. A reader of it, such as a security reviewer, a finance lead or a regulator, wants to know what the agent did with real credentials or money, and whether the record could have been changed after the fact. The two jobs share raw material and diverge on four properties.

PropertyObservability traceAudit trail
Written byThe SDK inside the applicationA component outside the agent's process
Can be removedYes, by design (delete, retention)Edits and deletions are detectable
Covers refusalsOnly if the application reports themEvery refusal is an entry
Kept forA configured retention windowA period the operator sets by policy

The rest of this entry takes the first two rows from the tools' own documentation.

The SDK that writes the trace runs inside the application

Langfuse's SDK documentation describes in-process instrumentation. For Python, "The Python SDK automatically sets up OpenTelemetry when initializing the client". For JavaScript, the page names a span processor registered in your own Node.js setup as the component that sends traces.

Three statements, from the SDK page and the overview, are worth reading together:

Each is sensible engineering for observability. A tracing library should not take your service down, and batching keeps latency low. The consequence for evidence is that a failed export is a logged warning, and an event still queued when a process dies may never leave the machine. The pages read on 9 November 2026 do not state a delivery guarantee.

There is a second consequence. The code that records "the agent called tool X" runs in the same process as the agent loop. If the agent is buggy, prompt-injected or running with a patched library, the trace reports what that process reported. An earlier entry goes through research on agents that misstate their own actions, and the conclusion applies here: an account written from inside is testimony.

This is not a claim that Langfuse or LangSmith traces are false. The point is narrower: the writer sits on the same side as the thing being recorded.

Traces can be deleted, and they expire

Deletion is a documented feature, and a useful one for privacy requests. Langfuse's data deletion page says "To delete a single trace, open its detail view and hit the Delete button." It lists DELETE /api/public/traces/{traceId} for one trace and DELETE /api/public/traces for a batch. Retention is the automated route: it "automatically deletes traces, observations, scores, and media assets older than a configured number of days."

LangSmith documents expiry. The administration overview has a table with a base tier of 14 days and an extended tier of 180 days. It adds: "After the specified retention period, traces are no longer accessible in the tracing project UI or via the API." The page also says most traces use base retention.

LangSmith also documents deletion on its data purging page: an API call to POST https://api.smith.langchain.com/api/v1/runs/delete takes a list of trace IDs "(up to 1000 traces per request)", and "Trace deletions are processed during non-peak usage times and are not instant."

None of this is a defect. A debugging store should shrink on demand, because it holds prompts and outputs that may contain personal data. That is the opposite of what a record of agent actions should do. If the same system serves both purposes, then anyone the tool allows to delete can remove the history, and an expiring retention window that nobody remembered to extend removes it too.

Regulation points the same way. An earlier entry covers Article 12 of the EU AI Act, which concerns automatic recording of events in high-risk systems. Whether a given agent falls under it is a question for the operator and their counsel. The design question, whether the log persists and cannot be quietly shortened, arises either way.

What an audit trail needs that a trace does not

Four properties separate the two, and each can be tested without trusting anyone's marketing.

  1. An independent writer. The record is created by a proxy, gateway or credential broker that the agent has to pass through, not by a library the agent loads. The writer sees every call because the call cannot happen without it.
  2. Write before act. The intent is recorded before the call goes out, and the result after. A crash between the two leaves an intent without a result, which is itself a finding. A trace written after the fact simply has a gap.
  3. Refusals are entries. A tracing tool records what ran. A control that blocks an action should keep the block as evidence: which limit, which request, at what time. Without it, a reviewer cannot tell "the agent never tried" from "the agent tried and was stopped".
  4. Edits and deletions show. A hash-linked log where each entry commits to the previous one turns an edit into a verification failure. Truncation from the end is subtler, which is why witnessing a log head off the machine exists.

Properties 2 to 4 need a design decision, not a bigger tracing plan. Raising retention from 14 to 180 days does not make a trace tamper-evident; it only moves the day the history disappears.

Where each tool is the better choice

For debugging a bad answer, a slow tool call or a retrieval step that returned nothing, a tracing tool is better. It shows nested spans, timings, inputs and outputs, supports evaluation on production data, and is built for engineers to read. A ledger designed for evidence is a poor place to ask why a prompt performed badly.

For "who authorised this, what was refused, and has anyone changed the record", use something that satisfies the four properties above. That can be a gateway log shipped to write-once storage, a cloud provider's own audit service, or a purpose-built ledger. The two systems run side by side. Nothing in either tool has to know about the other.

Where Mandare fits

Mandare is an open-source control and accountability layer. Its LLM gateway is one of its "doors", and the concepts page says every door follows one discipline: "write the intent to the ledger, execute, write the result. No entry, no action. A refused action is also an entry: the system records its no's."

The excerpt below comes from docs/demos/S2-runaway-demo.txt in the public repository, run against a mock provider that needs no API keys. The enforcement is real, the money is not. The mandate allows €20 a day, the loop makes 71 calls, and call 72 is refused.

pnpm demolive capture
…
    code:   PER_DAY_EXCEEDED
    the refusal itself is ledger entry 59034994bfacad09…
$ mandare verify --db ledger.db --spend
entries:  143
chain:    VALID — every entry hash-linked and door-signed
…
  #141  2026-09-27T05:31:54.675Z  INTENT   reserve 0.278853 EUR
  #142  2026-09-27T05:31:54.676Z  RESULT   settle  0.277797 EUR
  #143  2026-09-27T05:31:54.677Z  DENIED   refused 0.278853 EUR  ← reservation REFUSED by policy

The limits, as of v0.1.0 of the CLI (npm view @mandarelabs/cli version on 9 November 2026):

Mandare does not replace an observability tool, and an observability tool does not replace a ledger. If your agents hold real credentials, run the tracing for engineers and keep a second record, written by something the agent does not control, for everyone else.

Questions

Is a Langfuse or LangSmith trace an audit log?

Not by what their documentation promises. Langfuse describes traces as enabling "debugging, performance optimization, and quality monitoring". Both tools document deleting traces and expiring them by retention period, and neither page read on 9 November 2026 describes tamper evidence. A trace can still be a useful input to an investigation.

How long does LangSmith keep traces?

The LangSmith administration page read on 9 November 2026 lists two tiers: base retention of 14 days and extended retention of 180 days. After the period ends, traces are no longer accessible in the UI or through the API. Check the page for your plan before relying on either number.

Can an AI agent change its own traces?

The tracing SDKs run inside the application process, so code in that process decides what is recorded. Langfuse says SDK errors are caught and logged, and that events are queued locally and flushed in batches. Whether a given agent can reach and alter that code depends on how it is deployed.

Is observability enough without an audit trail?

Usually yes, because they answer different questions. Traces show latency, inputs, outputs and quality across a request. An audit trail is written by a component the agent does not control, is checked for edits, and also records what was refused. Neither replaces the other.

Sources

  1. Observability overview (Langfuse documentation) · Langfuse
  2. Data deletion (Langfuse documentation) · Langfuse
  3. SDK overview (Langfuse documentation) · Langfuse
  4. Administration overview: data retention (LangSmith documentation) · LangChain
  5. Data purging and compliance (LangSmith documentation) · LangChain

Run it yourself: 3 commands, no API keys.

github.com/mandarelabs/mandare →
← All journal entries