Quickstart — llama-stack documentation (original) (raw)

Get started with Llama Stack in minutes!

Llama Stack is a stateful service with REST APIs to support the seamless transition of AI applications across different environments. You can build and test using a local server first and deploy to a hosted endpoint for production.

In this guide, we’ll walk through how to build a RAG application locally using Llama Stack with Ollamaas the inference provider for a Llama Model.

Step 1: Install and setup

  1. Install uv
  2. Run inference on a Llama model with Ollama

ollama run llama3.2:3b --keepalive 60m

Step 2: Run the Llama Stack server

We will use uv to run the Llama Stack server.

INFERENCE_MODEL=llama3.2:3b uv run --with llama-stack llama stack build --template ollama --image-type venv --run

Step 3: Run the demo

Now open up a new terminal and copy the following script into a file named demo_script.py.

from llama_stack_client import Agent, AgentEventLogger, RAGDocument, LlamaStackClient

vector_db_id = "my_demo_vector_db" client = LlamaStackClient(base_url="http://localhost:8321")

models = client.models.list()

Select the first LLM and first embedding models

model_id = next(m for m in models if m.model_type == "llm").identifier embedding_model_id = ( em := next(m for m in models if m.model_type == "embedding") ).identifier embedding_dimension = em.metadata["embedding_dimension"]

_ = client.vector_dbs.register( vector_db_id=vector_db_id, embedding_model=embedding_model_id, embedding_dimension=embedding_dimension, provider_id="faiss", ) source = "https://www.paulgraham.com/greatwork.html" print("rag_tool> Ingesting document:", source) document = RAGDocument( document_id="document_1", content=source, mime_type="text/html", metadata={}, ) client.tool_runtime.rag_tool.insert( documents=[document], vector_db_id=vector_db_id, chunk_size_in_tokens=50, ) agent = Agent( client, model=model_id, instructions="You are a helpful assistant", tools=[ { "name": "builtin::rag/knowledge_search", "args": {"vector_db_ids": [vector_db_id]}, } ], )

prompt = "How do you do great work?" print("prompt>", prompt)

response = agent.create_turn( messages=[{"role": "user", "content": prompt}], session_id=agent.create_session("rag_session"), stream=True, )

for log in AgentEventLogger().log(response): log.print()

We will use uv to run the script

uv run --with llama-stack-client demo_script.py

And you should see output like below.

rag_tool> Ingesting document: https://www.paulgraham.com/greatwork.html

prompt> How do you do great work?

inference> [knowledge_search(query="What is the key to doing great work")]

tool_execution> Tool:knowledge_search Args:{'query': 'What is the key to doing great work'}

tool_execution> Tool:knowledge_search Response:[TextContentItem(text='knowledge_search tool found 5 chunks:\nBEGIN of knowledge_search tool results.\n', type='text'), TextContentItem(text="Result 1:\nDocument_id:docum\nContent: work. Doing great work means doing something important\nso well that you expand people's ideas of what's possible. But\nthere's no threshold for importance. It's a matter of degree, and\noften hard to judge at the time anyway.\n", type='text'), TextContentItem(text="Result 2:\nDocument_id:docum\nContent: work. Doing great work means doing something important\nso well that you expand people's ideas of what's possible. But\nthere's no threshold for importance. It's a matter of degree, and\noften hard to judge at the time anyway.\n", type='text'), TextContentItem(text="Result 3:\nDocument_id:docum\nContent: work. Doing great work means doing something important\nso well that you expand people's ideas of what's possible. But\nthere's no threshold for importance. It's a matter of degree, and\noften hard to judge at the time anyway.\n", type='text'), TextContentItem(text="Result 4:\nDocument_id:docum\nContent: work. Doing great work means doing something important\nso well that you expand people's ideas of what's possible. But\nthere's no threshold for importance. It's a matter of degree, and\noften hard to judge at the time anyway.\n", type='text'), TextContentItem(text="Result 5:\nDocument_id:docum\nContent: work. Doing great work means doing something important\nso well that you expand people's ideas of what's possible. But\nthere's no threshold for importance. It's a matter of degree, and\noften hard to judge at the time anyway.\n", type='text'), TextContentItem(text='END of knowledge_search tool results.\n', type='text')]

inference> Based on the search results, it seems that doing great work means doing something important so well that you expand people's ideas of what's possible. However, there is no clear threshold for importance, and it can be difficult to judge at the time.

To further clarify, I would suggest that doing great work involves:

Ultimately, great work is about making a meaningful contribution and leaving a lasting impression.

Congratulations! You’ve successfully built your first RAG application using Llama Stack! 🎉🥳

Next Steps

Now you’re ready to dive deeper into Llama Stack!