Tracely-ai

Tracely is an open-source, self-hostable tool that grades AI agent traces as they arrive, groups failures into issues, turns failing runs into replayable regression tests, and can block pull requests that would bring a failure back.

Why it matters

For teams running AI agents in production who are tired of hand-writing eval datasets. Tracely uses the exact runs that failed as tests and replays them in CI without live model calls.

Highlights

• Takes traces over standard OTLP and groups runs into conversation threads, with evaluators (structural checks and LLM judges) shown as columns on the trace table.
• Clusters failures structurally and semantically, so many broken runs show up as one issue with a count.
• One click promotes a failing trace into a hermetic test case with recorded tool and LLM outputs and a fail-to-pass check.
• tracely gate replays the suite in CI from recorded fixtures (no API keys or model spend), posts a commit status and fails the build on a regression.
• Alert flows can send to Slack, email or a webhook; the API also works as an MCP endpoint for coding agents.

Taken from the project's own README, docs or launch page. Claims about speed or accuracy are the maker's unless stated otherwise.

Worth knowing: Self-hosting runs several services (Postgres, ClickHouse, Redis, MinIO). Production traces can contain customer data, so decide where they are stored. The repo is less than two weeks old.

Quick facts

Links

• Official site: https://tracely-ai.com
• Repository: https://github.com/jwuthri-cloud/Tracely-ai
• Docs: https://doc.tracely-ai.com

Related ranking

See how established tools compare in Best AI Coding Tools for Developers. New releases aren't scored until we review them against the rubric.

Checked 2026-10-18 from the tool's own pages. Prices and star counts change, so confirm on the official site. Listing here is not a ranking or an endorsement.