raga-ai-hub /RagaAI-Catalyst
Python SDK for Agent AI Observability, Monitoring and Evaluation Framework. Includes features like agent, llm and tools tracing, debugging multi-agentic system, self-hosted dashboard and advanced analytics with timeline and execution graph view
★ 16,190⑂ 3,570👁 37⑃ 210● 55▤ 19
+0
Daily
16,190
stars
3,570
forks
37
watchers
210
prs
55
issues
19
releases
README
RagaAI Catalyst !GitHub release (latest by date) !GitHub stars !Issues
RagaAI Catalyst is a comprehensive platform designed to enhance the management and optimization of LLM projects. It offers a wide range of features, including project management, dataset management, evaluation management, trace management, prompt management, synthetic data generation, and guardrail management. These functionalities enable you to efficiently evaluate, and safeguard your LLM applications.
Table of Contents
- RagaAI Catalyst
- Installation
- Configuration
- Usage
- Project Management
- Dataset Management
- Evaluation Management
- Trace Management
- Agentic Tracing
- Prompt Management
- Synthetic Data Generation
- Guardrail Management
- Red-teaming
Installation
To install RagaAI Catalyst, you can use pip:
bash
pip install ragaai-catalyst
Configuration
Before using RagaAI Catalyst, you need to set up your credentials. You can do this by setting environment variables or passing them directly to the RagaAICatalyst class:
python
from ragaai_catalyst import RagaAICatalyst
catalyst = RagaAICatalyst(
access_key="YOUR_ACCESS_KEY",
secret_key="YOUR_SECRET_KEY",
base_url="BASE_URL"
)
you'll need to generate authentication credentials:
1. Navigate to your profile settings
2. Select "Authenticate"
3. Click "Generate New Key" to create your access and secret keys
!How to generate authentication keys
Note: Authetication to RagaAICatalyst is necessary to perform any operations below.
Usage
Project Management
Create and manage projects using RagaAI Catalyst:
python
Create a project
project = catalyst.create_project(
project_name="Test-RAG-App-1",
usecase="Chatbot"
)
Get project usecases
catalyst.project_use_cases()
List projects
projects = catalyst.list_projects()
print(projects)
!Projects
Dataset Management
Manage datasets efficiently for your projects:
py
from ragaai_catalyst import Dataset
Initialize Dataset management for a specific project
dataset_manager = Dataset(project_name="project_name")
List existing datasets
datasets = dataset_manager.list_datasets()
print("Existing Datasets:", datasets)
Create a dataset from CSV
dataset_manager.create_from_csv(
csv_path='path/to/your.csv',
dataset_name='MyDataset',
schema_mapping={'column1': 'schema_element1', 'column2': 'schema_element2'}
)
Get project schema mapping
dataset_manager.get_schema_mapping()
!Dataset
For more detailed information on Dataset Management, including CSV schema handling and advanced usage, please refer to the Dataset Management documentation.
Evaluation
Create and manage metric evaluation of your RAG application:
python
from ragaai_catalyst import Evaluation
Create an experiment
evaluation = Evaluation(
project_name="Test-RAG-App-1",
dataset_name="MyDataset",
)
Get list of available metrics
evaluation.list_metrics()
Add metrics to the experiment
schema_mapping={
Top issues
#17Add Example Notebooks8 comments · 2024-10-03#175Asynchronous and High-Throughput Support4 comments · 2025-02-12#3Add Unit Tests4 comments · 2024-09-19#213Is Catalyst works like a proxy or async mode?1 comments · 2025-04-28#199[BUG]:Cannot use method='json_schema' with model gpt-4 since it doesn't support OpenAI's Structured1 comments · 2025-03-26#183Docs are not helpful1 comments · 2025-03-09#272Cross-link suggestion: Independent third-party evaluation0 comments · 2026-10-28#270Proposal: EvalPort import/export bridge for Dataset + Evaluation results0 comments · 2026-08-23
Discussions
#267Authority-readiness as an agent observability/eval dimension0 comments · 2026-05-25#262RagaAI-Catalyst is listed on Shypd — AI Tool Directory0 comments · 2026-03-30#229Visualization of the codebase of RagaAI-Catalyst1 comments · 2025-06-28#1331.2.20 comments · 2024-12-14#831.20 comments · 2024-10-29#481.1.10 comments · 2024-10-17#16Welcome to agentneo Discussions!0 comments · 2024-10-03
Latest release
2.2.4 v2.2.4 2025-06-23Changes - Bug-fix: Updation of external_id, metadata not working as expected while updating dataset_name - Bug-fix: logs getting missed in load testing using Locust - Bug-fix: Error in SDG while generating dataset - Feat: Update project name in tracer on runtime