Learn more

For more information or if you need help retrieving your data, please contact Weights & Biases Customer Support at support@wandb.com

W&B Weave

Deliver AI with confidence

Evaluate, monitor, and iterate on agents and AI applications. Get started with one line of code.

1234

import weave
weave.init("quickstart")
@weave.op()
def llm_app(prompt):

Keep an eye on your AI

Improve quality, cost, latency, and safety

Weave works with any LLM and framework and comes with a ton of integrations out of the box

Quality

Accuracy, robustness, relevancy

Cost

Token usage and estimated cost

Latency

Track response times and bottlenecks

Safety

Protect your end users using guardrails

Evaluations

Measure and iterate

Visual comparisons

Use powerful visualizations for objective, precise comparisons

Automatic versioning

Save versions of your datasets, code, and scorers

import openai, weave
weave.init("weave-intro")

@weave.op
def correct_grammar(user_input):
    client = openai.OpenAI()
    response = client.chat.completions.create(
        model="o1-mini",
        messages=[{\
            "role": "user",\
            "content": "Correct the grammar:\n\n" +\
            user_input,\
        }],
    )
    return response.choices[0].message.content.strip()

result = correct_grammar("That was peace of cake!")
print(result)

Playground

Iterate on prompts in an interactive chat interface with any LLM

Leaderboards

Group evaluations into leaderboards featuring the best performers and share across your organization

Tracing and monitoring

Log everything for production monitoring and debugging

Debugging with trace trees

Weave organizes logs into an easy to navigate trace tree so you can identify issues

Multimodality

Track any modality—text, code, documents, image, and audio. Other modalities coming soon

Easily work with long form text

View large strings like documents, emails, HTML, and code in their original format

Creating a LLM-as-a-Judge That Drives Business Results

By Hamel Husain

A detailed guide on implementing large language models (LLMs) as judges for AI evaluation, featuring a seven-step "Critique Shadowing" process.

Seven-Step Process

1. Identifying a Principal Domain Expert

Engage key individuals with domain expertise early
Ensure evaluations align with user needs and standards

2. Creating a Diverse Dataset

Build comprehensive datasets reflecting diverse interactions
Include both real and synthetic data

3. Pass/Fail Judgments with Critiques

Implement simple binary judgments
Include detailed critiques for evaluation
Avoid complex scoring systems

4. Fixing Errors

Prioritize resolving obvious errors
Address issues before LLM judge implementation

5. Iterative LLM Judge Development

Use expert examples to refine LLM prompts
Aim for high expert-LLM agreement

6. Performing Error Analysis

Analyze errors to identify root causes
Improve AI performance based on findings

7. Creating Specialized LLM Judges

Develop targeted judges for specific issues
Implement after critique shadowing completion

Online evaluations

Score live incoming production traces for monitoring without impacting performance

Agents

Observability and governance tools for agentic systems

Build state-of-the-art agents

Supercharge your iteration speed and top the charts

Agent framework and protocol agnostic

Integrates with leading agent frameworks such as OpenAI Agents SDK and protocols such as MCP

import weave
from openai import OpenAI

weave.init("agent-example")

@weave.op()
def my_agent(query: str):
    client = OpenAI()
    response = client.chat.completions.create(...)
    return response

my_agent("What is the weather?")

Trace trees purpose-built for agentic systems

Easily visualize agents rollouts to pinpoint issues and improvements

Scoring

Use our scorers or bring your own

Pre-built scorers

Jumpstart your evals with out-of-box scorers built by our experts

Toxicity
Hallucinations
Content Relevance

Write your own scorers

Near-infinite flexibility to build custom scoring functions to suit your business

import weave, openai

llm_client = openai.OpenAI()

@weave.op()
def evaluate_output(generated_text, reference_text):
    """
    Evaluates AI-generated text against a reference answer.

Args:
        generated_text: The text generated by the model
        reference_text: The reference text to compare against

Returns:
        float: A score between 0-10
    """
    system_prompt = """
    You are an expert evaluator of AI outputs.
    Your job is to rate AI-generated text on a scale of 0-10.
    Base your rating on how well the generated text matches
    the reference text in terms of factual accuracy,
    comprehensiveness, and conciseness.
    """

user_prompt = f"""
    Reference: {reference_text}

AI Output: {generated_text}

Rate this output from 0-10:
    """

response = llm_client.chat.completions.create(
        model="gpt-4-turbo",
        messages=[\
            {"role": "system", "content": system_prompt},\
            {"role": "user", "content": user_prompt}\
        ],
        temperature=0.2
    )

# Extract the score from the response
    score_text = response.choices[0].message.content
    # Parse score (assuming it returns a number between 0-10)
    try:
        score = float(score_text.strip())
        return min(max(score, 0), 10)  # Clamp between 0-10
    except:
        # Fallback score if parsing fails
        return 5.0

Human feedback

Collect user and expert feedback for real-life testing and evaluation

Third-party scorers

Plug and play off-the-shelf scoring functions from other vendors

Guardrails

Safeguard your users and brand

Detect harmful outputs and prompt attacks with our out-of-box filters Pre/post response hooks ensure AI responses align with your policies

Inference

Access popular open-source models

Playground or API access

Access to leading open-source foundation models

MiniMax M2.5

Z.AI GLM 5.0

OpenAI GPT OSS 20B

OpenAI GPT OSS 120B

Qwen3 235B A22B Thinking-2507

Qwen3 Coder 480B A35B

Qwen3 235B A22B-2507

OpenPipe Qwen3 14B Instruct

Meta Llama 3.1 70B

DeepSeek R1-0528

DeepSeek V3.1

Llama 3.1 78

Llama 3.3 70B

Llama 4 Scout

Phi 4 Mini

MiniMax M2.5

Z.AI GLM 5.0

OpenAI GPT OSS 20B

OpenAI GPT OSS 120B

Qwen3 235B A22B Thinking-2507

Qwen3 Coder 480B A35B

Qwen3 235B A22B-2507

OpenPipe Qwen3 14B Instruct

Get started with one line of code

Simple, hands-on introduction to tracing and evaluations

Gain a comprehensive understanding of developing AI applications

Dive deeper and learn how to build AI solutions for a variety of use cases

Join the Weights & Biases community

Podcast: Conversations on AI

Weights & Biases' how-to blog

Webinars, conferences, meetups, and more