confident-ai /deepeval
The LLM Evaluation Framework
★ 19,492⑂ 2,207👁 64⑃ 2,750● 814▤ 64
+13
Daily
19,492
stars
2,207
forks
64
watchers
2,750
prs
814
issues
64
releases
README
The LLM Evaluation Framework
Documentation |
Metrics and Features |
Getting Started |
Integrations |
Confident AI
Deutsch |
Español |
français |
日本語 |
한국어 |
Português |
Русский |
中文
DeepEval is a simple-to-use, open-source LLM evaluation framework, for evaluating large-language model systems. It is similar to Pytest but specialized for unit testing LLM apps. DeepEval incorporates the latest research to run evals via metrics such as G-Eval, task completion, answer relevancy, hallucination, etc., which use LLM-as-a-judge and other NLP models that run locally on your machine.
Whether you're building AI agents, voice agents, multi-turn chatbots, RAG pipelines, or anything else, implemented via LangChain or OpenAI, DeepEval has you covered. With it, you can easily evaluate:
- LLM apps end-to-end as black boxes
- Complete AI agent trajectories across every decision and action
- Individual AI agent steps such as LLM calls, tool use, retrieval, and sub-agent handoffs
Use these evaluations to determine the optimal models, prompts, and architecture to improve your AI quality, prevent prompt drifting, or even transition from OpenAI to Claude with confidence.
Jev is now available in DeepEval, both in existing metrics and the latest JevEval.
[!IMPORTANT]
If you're already in production, Confident AI surfaces failing traces before anyone has written an eval for them, clusters them into failure modes, and turns them into your next eval.
!Demo GIF
🔥 Metrics and Features
- 📐 Large variety of ready-to-use LLM eval metrics (all with explanations) powered by ANY LLM of your choice, statistical methods, or NLP models that run locally on your machine covering all use cases:
- Custom, All-Purpose Metrics:
- JevEval — a custom metric scored by Jev, a System One model that answers your questions with calibrated probabilities instead of generated text
- G-Eval — a research-backed LLM-as-a-judge metric for evaluating on any custom criteria with human-like accuracy
- DAG — DeepEval's graph-based deterministic LLM-as-a-judge metric builder
-
AI Agent Metrics
- Task Completion — evaluate whether an AI agent accomplished its goal
- Tool Correctness — check if the right tools were called with the right arguments
- Goal Accuracy — measure how accurately the agent achieved the intended goal
- Step Efficiency — evaluate whether the agent took unnecessary steps
- Plan Adherence — check if the agent followed the expected plan
- Plan Quality — evaluate the quality of the agent's plan
- Tool Use — measure quality of tool usage
- Argument Correctness — validate tool call arguments
-
Voice Agent Metrics
- Agent Responsiveness — check whether the voice agent answered when it wTop issues
#286Disk Error When Attempting to Run Tests34 comments · 2023-11-14#1147Asynchronous test runs are sometimes not completed correctly21 comments · 2024-11-09#3433[Feature] Community metric proposal: CalibrationMetric (ECE + Brier) for aggregate confidence trustworthiness19 comments · 2026-10-10#929ValueError: Evaluation LLM outputted an invalid JSON. Please use a better evaluation model.17 comments · 2024-08-02#743Cant use HuggingFace Model for evaluation16 comments · 2024-05-06#2396Feat request: optional user-defined prompt builder for synthesizer15 comments · 2025-12-26#568Custom dataset evaluation giving ValueError: Input, actual output, and retrieval context cannot be None13 comments · 2024-03-06#2734DeepEval for Typescript12 comments · 2026-06-09
Discussions
#3651How are you checking agent evals against known-bad outcomes?0 comments · 2026-11-14#3650Accounting for unknown-start requests and retries in a complete evaluation run1 comments · 2026-11-14#3636mcp-slim: Profiling MCP tool-calling token bloat (-50% to -98%) and redacting leaked credentials in agent pipelines0 comments · 2026-11-12#3584Model-graded evals kept passing my agent's bad runs — so I moved the verdict to a deterministic kernel0 comments · 2026-11-04#3482DeepEval included in AgentManager's tool catalog0 comments · 2026-10-18#3427How do you turn failed agent evals into reviewable behavior changes?1 comments · 2026-10-07#3416Putting a hard dollar cap on one evaluate() run0 comments · 2026-10-04#3370Metric proposal: format compliance as its own axis, orthogonal to correctness1 comments · 2026-09-26
Latest release
Introducing Jev in DeepEval python-v4.2.4 2026-09-22New to deepeval? Get started here. 🧠 Introducing Jev: System One for LLM Evals Every LLM-as-a-judge metric has had the same awkward secret: somewhere inside the thing measuring your LLM is another LLM generating text. Closed-ended verdicts (yes / no / borderline) were treated like writing tasks, and you paid for it in variance, latency, and cost. deepeval 4.2.2 introduces support for Jev, TypeSafe AI's System One model. Jev is not a language model. It does not generate text. You give it state and a bounded question, and it returns a typed decision with calibrated probabilities. Th