Music generation using Lyria RealTime (original) (raw)

Skip to main content

The Gemini API, usingLyria RealTime, provides access to a state-of-the-art, real-time, streaming music generation model. It allows developers to build applications where users can interactively create, continuously steer, and perform instrumental music.

To experience what can be built using Lyria RealTime, try it on AI Studio using the Prompt DJ or theMIDI DJ apps!

How music generation works

Lyria RealTime music generation uses a persistent, bidirectional, low-latency streaming connection usingWebSocket.

Generate and control music

Lyria RealTime works a bit like the Live APIin the sense that it is using websockets to keep a real-time communication with the model. It's still not exactly the same as you can't talk to the model and you have to use a specific format to prompt it.

The following code demonstrates how to generate music:

Python

This example initializes the Lyria RealTime session usingclient.aio.live.music.connect(), then sends an initial prompt with session.set_weighted_prompts() along with an initial configuration using session.set_music_generation_config, starts the music generation using session.play() and sets upreceive_audio() to process the audio chunks it receives.

  import asyncio
  from google import genai
  from google.genai import types

  client = genai.Client(http_options={'api_version': 'v1alpha'})

  async def main():
      async def receive_audio(session):
        """Example background task to process incoming audio."""
        while True:
          async for message in session.receive():
            audio_data = message.server_content.audio_chunks[0].data
            # Process audio...
            await asyncio.sleep(10**-12)

      async with (
        client.aio.live.music.connect(model='models/lyria-realtime-exp') as session,
        asyncio.TaskGroup() as tg,
      ):
        # Set up task to receive server messages.
        tg.create_task(receive_audio(session))

        # Send initial prompts and config
        await session.set_weighted_prompts(
          prompts=[
            types.WeightedPrompt(text='minimal techno', weight=1.0),
          ]
        )
        await session.set_music_generation_config(
          config=types.LiveMusicGenerationConfig(bpm=90, temperature=1.0)
        )

        # Start streaming music
        await session.play()
  if __name__ == "__main__":
      asyncio.run(main())

JavaScript

This example initializes the Lyria RealTime session usingclient.live.music.connect(), then sends an initial prompt with session.setWeightedPrompts() along with an initial configuration using session.setMusicGenerationConfig, starts the music generation using session.play() and sets up anonMessage callback to process the audio chunks it receives.

import { GoogleGenAI } from "@google/genai";
import Speaker from "speaker";
import { Buffer } from "buffer";

const client = new GoogleGenAI({
  apiKey: GEMINI_API_KEY,
    apiVersion: "v1alpha" ,
});

async function main() {
  const speaker = new Speaker({
    channels: 2,       // stereo
    bitDepth: 16,      // 16-bit PCM
    sampleRate: 44100, // 44.1 kHz
  });

  const session = await client.live.music.connect({
    model: "models/lyria-realtime-exp",
    callbacks: {
      onmessage: (message) => {
        if (message.serverContent?.audioChunks) {
          for (const chunk of message.serverContent.audioChunks) {
            const audioBuffer = Buffer.from(chunk.data, "base64");
            speaker.write(audioBuffer);
          }
        }
      },
      onerror: (error) => console.error("music session error:", error),
      onclose: () => console.log("Lyria RealTime stream closed."),
    },
  });

  await session.setWeightedPrompts({
    weightedPrompts: [
      { text: "Minimal techno with deep bass, sparse percussion, and atmospheric synths", weight: 1.0 },
    ],
  });

  await session.setMusicGenerationConfig({
    musicGenerationConfig: {
      bpm: 90,
      temperature: 1.0,
      audioFormat: "pcm16",  // important so we know format
      sampleRateHz: 44100,
    },
  });

  await session.play();
}

main().catch(console.error);

You can then use session.play(), session.pause(), session.stop() andsession.reset_context() to start, pause, stop or reset the session.

Steer music in real-time

Prompt Lyria RealTime

While the stream is active, you can send new WeightedPrompt messages at any time to alter the generated music. The model will smoothly transition based on the new input.

The prompts need to follow the right format with a text (the actual prompt), and a weight. The weight can take any value except 0. 1.0is usually a good starting point.

Python

  from google.genai import types

  await session.set_weighted_prompts(
    prompts=[
      {"text": "Piano", "weight": 2.0},
      types.WeightedPrompt(text="Meditation", weight=0.5),
      types.WeightedPrompt(text="Live Performance", weight=1.0),
    ]
  )

JavaScript

  await session.setMusicGenerationConfig({
    weightedPrompts: [
      { text: 'Harmonica', weight: 0.3 },
      { text: 'Afrobeat', weight: 0.7 }
    ],
  });

Note that the model transitions can be a bit abrupt when drastically changing the prompts so it's recommended to implement some kind of cross-fading by sending intermediate weight values to the model.

Update the configuration

You can also update the music generation parameters in real time. You can't just update a parameter, you need to set the whole configuration otherwise the other fields will be reset back to their default values.

Since updating the bpm or the scale is a drastic change for the model you'll also need to tell it to reset its context using reset_context() to take the new config into account. It won't stop the stream, but it will be a hard transition. You don't need to do it for the other parameters.

Python

  from google.genai import types

  await session.set_music_generation_config(
    config=types.LiveMusicGenerationConfig(
      bpm=128,
      scale=types.Scale.D_MAJOR_B_MINOR,
      music_generation_mode=types.MusicGenerationMode.QUALITY
    )
  )
  await session.reset_context();

JavaScript

  await session.setMusicGenerationConfig({
    musicGenerationConfig: { 
      bpm: 120,
      density: 0.75,
      musicGenerationMode: MusicGenerationMode.QUALITY
    },
  });
  await session.reset_context();

Prompt guide for Lyria RealTime

Here's a non-exhaustive list of prompts you can use to prompt Lyria RealTime:

These are just some examples, Lyria RealTime can do much more. Experiment with your own prompts!

Best practices

Technical details

This section describes the specifics of how to use Lyria RealTime music generation.

Specifications

Controls

Music generation can be influenced in real time by sending messages containing:

For bpm, density, brightness and scale, if no value is provided, the model will decide what's best according to your initial prompts.

More classical parameters like temperature (0.0 to 3.0, default 1.1), top_k(1 to 1000, default 40), and seed (0 to 2 147 483 647, randomly selected by default) are also customizable in the MusicGenerationConfig.

Scale Enum Values

Here are all the scale values that the model can accept:

Enum Value Scale / Key
C_MAJOR_A_MINOR C major / A minor
D_FLAT_MAJOR_B_FLAT_MINOR D♭ major / B♭ minor
D_MAJOR_B_MINOR D major / B minor
E_FLAT_MAJOR_C_MINOR E♭ major / C minor
E_MAJOR_D_FLAT_MINOR E major / C♯/D♭ minor
F_MAJOR_D_MINOR F major / D minor
G_FLAT_MAJOR_E_FLAT_MINOR G♭ major / E♭ minor
G_MAJOR_E_MINOR G major / E minor
A_FLAT_MAJOR_F_MINOR A♭ major / F minor
A_MAJOR_G_FLAT_MINOR A major / F♯/G♭ minor
B_FLAT_MAJOR_G_MINOR B♭ major / G minor
B_MAJOR_A_FLAT_MINOR B major / G♯/A♭ minor
SCALE_UNSPECIFIED Default / The model decides

The model is capable of guiding the notes that are played, but does not distinguish between relative keys. Thus each enum corresponds both to the relative major and minor. For example, C_MAJOR_A_MINOR would correspond to all the white keys of a piano, and F_MAJOR_D_MINOR would be all the white keys except B flat.

Limitations

What's next

Explore the Cookbook for more code examples and tutorials.

Except as otherwise noted, the content of this page is licensed under the Creative Commons Attribution 4.0 License, and code samples are licensed under the Apache 2.0 License. For details, see the Google Developers Site Policies. Java is a registered trademark of Oracle and/or its affiliates.

Last updated 2025-09-26 UTC.