vibrantlabsai /ragas

Supercharge Your LLM Application Evaluations 🚀

★ 16,328⑂ 1,821👁 61⑃ 1,729● 1,366▤ 91
PythonApache-2.0Created 2023-05-08docs.ragas.io
Download: v0.4.4v0.4.4 ↓0.4.42026-11-14
+8
Daily
16,328
stars
1,821
forks
61
watchers
1,729
prs
1,366
issues
91
releases

README

Supercharge Your LLM Application Evaluations 🚀



    
        
    
    
        
    
    
        
    
    
        
    
    
        
    
    
      
    



    
        Documentation |
        Quick start |
        Join Discord |
        Blog |
        NewsLetter |
        Careers
    


Objective metrics, intelligent test generation, and data-driven insights for LLM apps

Ragas is your ultimate toolkit for evaluating and optimizing Large Language Model (LLM) applications. Say goodbye to time-consuming, subjective assessments and hello to data-driven, efficient evaluation workflows.
Don't have a test dataset ready? We also do production-aligned test set generation.

 Key Features

- 🎯 Objective Metrics: Evaluate your LLM applications with precision using both LLM-based and traditional metrics.
- 🧪 Test Data Generation: Automatically create comprehensive test datasets covering a wide range of scenarios.
- 🔗 Seamless Integrations: Works flawlessly with popular LLM frameworks like LangChain and major observability tools.
- 📊 Build feedback loops: Leverage production data to continually improve your LLM applications.

 :shield: Installation

Pypi:

bash
pip install ragas


Alternatively, from source:

bash
pip install git+https://github.com/vibrantlabsai/ragas


 :fire: Quickstart

 Clone a Complete Example Project

The fastest way to get started is to use the ragas quickstart command:

bash
 List available templates
ragas quickstart

 Create a RAG evaluation project
ragas quickstart rag_eval

 Specify where you want to create it.
ragas quickstart rag_eval -o ./my-project


Available templates:
- rag_eval - Evaluate RAG systems

Coming Soon:
- agent_evals - Evaluate AI agents
- benchmark_llm - Benchmark and compare LLMs
- prompt_evals - Evaluate prompt variations
- workflow_eval - Evaluate complex workflows

 Evaluate your LLM App

ragas comes with pre-built metrics for common evaluation tasks. For example, Aspect Critique evaluates any aspect of your output using DiscreteMetric:

python
import asyncio
from openai import AsyncOpenAI
from ragas.metrics import DiscreteMetric
from ragas.llms import llm_factory

 Setup your LLM
client = AsyncOpenAI()
llm = llm_factory("gpt-4o", client=client)

 Create a custom aspect evaluator
metric = DiscreteMetric(
    name="summary_accuracy",
    allowed_values=["accurate", "inaccurate"],
    prompt="""Evaluate if the summary is accurate and captures key information.

Response: {response}

Answer with only 'accurate' or 'inaccurate'."""
)

 Score your application's output
async def main():
    score = await metric.ascore(
        llm=llm,
        response="The summary of the text is..."
    )
    print(f"Score: {score.value}")   'accurate' or 'inaccurate'
    print(f"Reason: {score.reason}")


if __name__ == "__main__":
    asyncio.run(main())


 Note: Make sure your OPENAI_API_KEY environment variable is set.

Find the complete Quickstart Guide

 Want help in improving your AI application using evals?

In t

Top issues

Latest release

v0.4.4 v0.4.4 2026-11-14
This release restores a working install, fixes a published security vulnerability, and makes Ragas work again with current OpenAI and Anthropic models.

bash
pip install -U ragas


 Highlights

- import ragas works again. A fresh install pulled in langchain-community 0.4, which removed the VertexAI modules Ragas imported at start-up, so the import failed with ModuleNotFoundError. Those imports are now optional. (2956)
- Security fix for multimodal metrics (CVE-2026-6587). See the security section below. (3105)
- Anthropic works on current Claude models. Ragas no longer sends the sampling defau

Snapshot: 2026-11-16T09:23:14Z · Data license Apache-2.0 · JSON