API Reference — llama-stack documentation (original) (raw)

• Llama Stack
• Quickstart
  • Step 1: Install and setup
  • Step 2: Run the Llama Stack server
  • Step 3: Run the demo
  • Next Steps
• Detailed Tutorial
  • Step 1: Installation and Setup
  • Step 2: Run Llama Stack
  • Step 3: Run Client CLI
  • Step 4: Run the Demos
• Why Llama Stack?
  • Our Solution: A Universal Stack
  • Our Philosophy
• Core Concepts
  • APIs
  • API Providers
  • Resources
  • Distributions
  • Evaluation Concepts
    * Open-benchmark Eval
    * List of open-benchmarks Llama Stack support
    * Run evaluation on open-benchmarks via CLI
    * Spin up Llama Stack server
    * Run eval CLI
    * What’s Next?
• OpenAI API Compatibility
  • Server path
  • Clients
    * Llama Stack Client
    * OpenAI Client
  • APIs implemented
    * Models
    * Responses
    * Simple inference
    * Structured Output
    * Chat Completions
    * Simple inference
    * Structured Output
    * Completions
    * Simple inference
• Providers Overview
  • External Providers
  • Agents
  • DatasetIO
  • Eval
  • Inference
  • Post Training
    * Post Training Providers
    * External Providers
    * HuggingFace SFTTrainer
    * TorchTune
    * NVIDIA NEMO
  • Safety
  • Scoring
  • Telemetry
  • Tool Runtime
  • Vector IO
    * Vector IO Providers
    * External Providers
    * Faiss
    * SQLite-Vec
    * Chroma
    * Postgres PGVector
    * Qdrant
    * Milvus
    * Weaviate
• Distributions Overview
  • Using Llama Stack as a Library
    * Setup Llama Stack without a Server
  • Configuring a “Stack”
    * Providers
    * Resources
    * Server Configuration
    * Authentication Configuration
    * Quota Configuration
    * Extending to handle Safety
  • Available List of Distributions
    * Selection of a Distribution / Template
    * Distribution Details
    * On-Device Distributions
  • Kubernetes Deployment Guide
    * Prerequisites
    * Deploying Llama Stack Server in Kubernetes
    * Verifying the Deployment
  • Build your own Distribution
    * Setting your log level
    * Llama Stack Build
    * Running your Stack server
    * Listing Distributions
    * Removing a Distribution
    * Troubleshooting
• Building AI Applications (Examples)
  • Retrieval Augmented Generation (RAG)
    * Setting up Vector DBs
    * Ingesting Documents
    * Using Precomputed Embeddings
    * Retrieval
    * Using the RAG Tool
    * Building RAG-Enhanced Agents
    * Unregistering Vector DBs
    * Appendix
    * More RAGDocument Examples
  • Agents
    * Core Concepts
    * 1. Agent Configuration
    * 2. Sessions
    * 3. Turns
    * Non-Streaming
    * 4. Steps
    * Agent Execution Loop
  • Agent Execution Loop
    * Steps in the Agent Workflow
    * Agent Execution Loop Example
  • Tools
    * Server-side vs. client-side tool execution
    * Server-side tools
    * Model Context Protocol (MCP)
    * Using Remote MCP Servers
    * Running your own MCP server
    * Adding Custom (Client-side) Tools
    * Tool Invocation
    * Listing Available Tools
    * Simple Example 2: Using an Agent with the Web Search Tool
    * Simple Example3: Using an Agent with the WolframAlpha Tool
  • Evaluations
    * Application Evaluation
    * Building a Search Agent
    * Query Agent Execution Steps
    * Evaluate Agent Responses
  • Telemetry
    * Events
    * Spans and Traces
    * Sinks
    * Providers
    * Meta-Reference Provider
    * Configuration
    * Jaeger to visualize traces
    * Querying Traces Stored in SQLite
  • Safety Guardrails
• Llama Stack Playground
  • Key Features
    * Playground
    * Chatbot
    * Evaluations
    * Inspect
  • Starting the Llama Stack Playground
• Contributing to Llama-Stack
  • Discussions -> Issues -> Pull Requests
  • Contributor License Agreement (“CLA”)
  • Issues
  • Set up your development environment
  • Pre-commit Hooks
  • Running tests
  • Adding a new dependency to the project
  • Coding Style
  • Common Tasks
    * Using llama stack build
    * Updating Provider Configurations
    * Building the Documentation
    * Update API Documentation
  • License
    * Adding a New API Provider
    * Testing the Provider
    * Submitting Your PR
• References
  • API Reference
  • Python SDK Reference
    * Shared Types
    * Toolgroups
    * Tools
    * ToolRuntime
    * RagTool
    * Agents
    * Session
    * Steps
    * Turn
    * BatchInference
    * Datasets
    * Eval
    * Jobs
    * Inspect
    * Inference
    * VectorIo
    * VectorDBs
    * Models
    * PostTraining
    * Job
    * Providers
    * Routes
    * Safety
    * Shields
    * SyntheticDataGeneration
    * Telemetry
    * Datasetio
    * Scoring
    * ScoringFunctions
    * Benchmarks
  • llama (server-side) CLI Reference
    * Installation
    * llama subcommands
    * Sample Usage
    * Downloading models
    * Downloading from Meta
    * Downloading from Hugging Face
    * List the downloaded models
    * Understand the models
    * Sample Usage
    * Describe
    * Prompt Format
    * Remove model
  • llama (client-side) CLI Reference
    * Basic Commands
    * llama-stack-client
    * llama-stack-client configure
    * llama-stack-client providers list
    * Model Management
    * llama-stack-client models list
    * llama-stack-client models get
    * llama-stack-client models register
    * llama-stack-client models update
    * llama-stack-client models delete
    * Vector DB Management
    * llama-stack-client vector_dbs list
    * llama-stack-client vector_dbs register
    * llama-stack-client vector_dbs unregister
    * Shield Management
    * llama-stack-client shields list
    * llama-stack-client shields register
    * Eval Task Management
    * llama-stack-client benchmarks list
    * llama-stack-client benchmarks register
    * Eval execution
    * llama-stack-client eval run-benchmark
    * llama-stack-client eval run-scoring
    * Tool Group Management
    * llama-stack-client toolgroups list
    * llama-stack-client toolgroups get
    * llama-stack-client toolgroups register
    * llama-stack-client toolgroups unregister
  • Downloading Models
    * Installation
    * Downloading models via CLI
    * Downloading from Meta
    * Downloading from Hugging Face
    * List the downloaded models
  • Evaluations
    * Evaluation Concepts
    * Evaluation Examples Walkthrough
    * 1. Open Benchmark Model Evaluation
    * 2. Agentic Evaluation
    * 3. Agentic Application Dataset Scoring
    * Running Evaluations via CLI
    * Benchmark Evaluation CLI
    * Application Evaluation CLI
    * Defining BenchmarkConfig
    * Open-benchmark Contributing Guide
    * Create the new dataset for your new benchmark
    * Find scoring function for your new benchmark
    * Add new benchmark into template
    * Test the new benchmark

llama-stack

• References
• API Reference
• View page source