Incorrect Audio Token Count - Gemini Live API RealtimeModel

Hi, Im using the Gemini Live API as a RealtimeModel with LiveKit, and noticed the audio token count is not populated correctly when using the Audio modality.

I used the following code to track the metric that contains the audio and text token counts:

@session.on("metrics_collected")
    def _on_metrics_collected(ev: MetricsCollectedEvent):
        print("ev metrics:", ev.metrics)
        metrics.log_metrics(ev.metrics)

I notice that the input audio tokens count increases by 1 each time, starting at 0 which is clearly incorrect since the input is of me speaking for multiple seconds, and the output audio tokens is always 0.

Since the audio tokens are incorrect in the ev.metrics object, its subsequently incorrect in the metrics.log() function call and the usage_collector object.

Some ev.metrics values from a short conversation with the voice agent:

type='realtime_model_metrics' label='livekit.plugins.google.beta.realtime.realtime_api.RealtimeModel' request_id='gemini-turn-58971acd2589' timestamp=1748428916.454256 duration=13.88671088218689 ttft=9.393692016601562e-05 cancelled=False input_tokens=3047 output_tokens=49 total_tokens=3096 tokens_per_second=3.528553335322514 input_token_details=InputTokenDetails(audio_tokens=0, text_tokens=3047, image_tokens=0, cached_tokens=0, cached_tokens_details=CachedTokenDetails(audio_tokens=0, text_tokens=0, image_tokens=0)) output_token_details=OutputTokenDetails(text_tokens=49, audio_tokens=0, image_tokens=0)

Summarizing just input and output audio tokens from the next few outputs in the same conversation:

request_id='gemini-turn-6b1d8f8e9b73' input_token_details=InputTokenDetails(audio_tokens=1, text_tokens=3182, image_tokens=0, cached_tokens=0, cached_tokens_details=CachedTokenDetails(audio_tokens=0, text_tokens=0, image_tokens=0)) output_token_details=OutputTokenDetails(text_tokens=33, audio_tokens=0, image_tokens=0)

request_id='gemini-turn-86abeed1fd71' input_token_details=InputTokenDetails(audio_tokens=2, text_tokens=3302, image_tokens=0, cached_tokens=0, cached_tokens_details=CachedTokenDetails(audio_tokens=0, text_tokens=0, image_tokens=0)) output_token_details=OutputTokenDetails(text_tokens=36, audio_tokens=0, image_tokens=0)

request_id='gemini-turn-38693a0f458c' input_token_details=InputTokenDetails(audio_tokens=3, text_tokens=3440, image_tokens=0, cached_tokens=0, cached_tokens_details=CachedTokenDetails(audio_tokens=0, text_tokens=0, image_tokens=0)) output_token_details=OutputTokenDetails(text_tokens=33, audio_tokens=0, image_tokens=0)

I tried the same thing using Gemini Live API directly and printed the token counts for each modality and audio and text tokens there are more reasonable numbers (audio tokens are around 100+ tokens per interaction which makes sense since each interaction is a few seconds long).
Note: audio tokens are 32 tokens per second as per the gemini docs.

I tried exploring the _token_details_map function inside _handle_usage_metadata function in the RealtimeSession class, and printing the token counts there, and I think the issue runs deeper than that since they're incorrect at that stage too.

Any advice on this would be highly appreciated since I want to log the token counts accurately and want to calculate the cost of each conversation accordingly.

Additional info:
I run the code using python app.py console although the same issue occurs when i run it using python app.py dev
Here is my session object creation:

session = AgentSession(
        llm=google.beta.realtime.RealtimeModel(
            model="gemini-2.0-flash-exp",
            voice="Zephyr",
            temperature=0.8,
            instructions=SYSTEM_INSTRUCTIONS,
            modalities=["AUDIO"],
        ),
    )

+1

confirmed this is actually the expected value. while the audio token from each interaction is low, prompt_token_count continues to increase.. for example, printing out the raw stats from a short conversation turn:

2025-05-31 23:55:54,369 - INFO livekit.plugins.google - usage_metadata: prompt_token_count=1333 cached_content_token_count=None response_token_count=32 tool_use_prompt_token_count=None thoughts_token_count=None total_token_count=1365 prompt_tokens_details=[ModalityTokenCount(modality=<MediaModality.TEXT: 'TEXT'>, token_count=1329), ModalityTokenCount(modality=<MediaModality.AUDIO: 'AUDIO'>, token_count=4)] cache_tokens_details=None response_tokens_details=[ModalityTokenCount(modality=<MediaModality.TEXT: 'TEXT'>, token_count=32)] tool_use_prompt_tokens_details=None traffic_type=None {"room": "sbx-104ei0-VPvZRvsm8zjAN8SSYqsRJB", "pid": 91619, "job_id": "AJ_dpcptzAD9P3X"}

It only used 4 audio tokens, but 1329 text tokens. the existing chat history is represented by text tokens.

That doesn't make sense at all though.

Gemini represents each second of audio as 32 tokens; for example, one minute of audio is represented as 1,920 tokens. Source

So this suggests that something is wrong somewhere. Either with the API, or the Gemini Python interface

@davidzhao

We are also facing the same problem, for the following agent configuration (only using gemini realtime):

    session = AgentSession(
        llm=google.beta.realtime.RealtimeModel(
            model="gemini-2.0-flash-live-001",  #'gemini-2.5-flash-preview-native-audio-dialog',
            voice="kore",  #'Puck',The duration (seconds) of the audio input received by the STT model.
            temperature=0.8,
            language="en-IN",
            modalities=["AUDIO"],
        ),
    )

The telephony call between the agent and user is successful, with both sides speaking, but the input audio_tokens only increment by 1 while the output audio_tokens are always 0.

@davidzhao Without the correct audio_tokens info how can we calculate the price of the session ?

If we somehow knew the input's and output's audio_duration we could use it to calculate the cost, assuming 32 audio tokens per second.

[
  {
    "type": "realtime_model_metrics",
    "label": "livekit.plugins.google.beta.realtime.realtime_api.RealtimeModel",
    "request_id": "GR_5930d26058f4",
    "timestamp": 1754994816.0772092,
    "duration": 4.687177419662476,
    "ttft": 0.0005161762237548828,
    "cancelled": false,
    "input_tokens": 4700,
    "output_tokens": 21,
    "total_tokens": 4721,
    "tokens_per_second": 4.480308321999088,
    "input_token_details": {
      "audio_tokens": 0,
      "text_tokens": 4700,
      "image_tokens": 0,
      "cached_tokens": 0,
      "cached_tokens_details": {
        "audio_tokens": 0,
        "text_tokens": 0,
        "image_tokens": 0
      }
    },
    "output_token_details": {
      "text_tokens": 21,
      "audio_tokens": 0,
      "image_tokens": 0
    }
  },
  {
    "type": "realtime_model_metrics",
    "label": "livekit.plugins.google.beta.realtime.realtime_api.RealtimeModel",
    "request_id": "GR_08dfec0af3d6",
    "timestamp": 1754994823.9091454,
    "duration": 4.695490121841431,
    "ttft": 0.38658714294433594,
    "cancelled": false,
    "input_tokens": 4783,
    "output_tokens": 20,
    "total_tokens": 4803,
    "tokens_per_second": 4.259406256009032,
    "input_token_details": {
      "audio_tokens": 1,
      "text_tokens": 4782,
      "image_tokens": 0,
      "cached_tokens": 0,
      "cached_tokens_details": {
        "audio_tokens": 0,
        "text_tokens": 0,
        "image_tokens": 0
      }
    },
    "output_token_details": {
      "text_tokens": 20,
      "audio_tokens": 0,
      "image_tokens": 0
    }
  },
  {
    "type": "realtime_model_metrics",
    "label": "livekit.plugins.google.beta.realtime.realtime_api.RealtimeModel",
    "request_id": "GR_2857874ef59e",
    "timestamp": 1754994831.008194,
    "duration": 5.116125822067261,
    "ttft": 0.4432711601257324,
    "cancelled": false,
    "input_tokens": 4841,
    "output_tokens": 15,
    "total_tokens": 4856,
    "tokens_per_second": 2.931906001080127,
    "input_token_details": {
      "audio_tokens": 2,
      "text_tokens": 4839,
      "image_tokens": 0,
      "cached_tokens": 0,
      "cached_tokens_details": {
        "audio_tokens": 0,
        "text_tokens": 0,
        "image_tokens": 0
      }
    },
    "output_token_details": {
      "text_tokens": 15,
      "audio_tokens": 0,
      "image_tokens": 0
    }
  },
  {
    "type": "realtime_model_metrics",
    "label": "livekit.plugins.google.beta.realtime.realtime_api.RealtimeModel",
    "request_id": "GR_7eaf7f34f12e",
    "timestamp": 1754994839.742582,
    "duration": 8.821386098861694,
    "ttft": 0.5489330291748047,
    "cancelled": false,
    "input_tokens": 4912,
    "output_tokens": 32,
    "total_tokens": 4944,
    "tokens_per_second": 3.6275478299412898,
    "input_token_details": {
      "audio_tokens": 3,
      "text_tokens": 4909,
      "image_tokens": 0,
      "cached_tokens": 0,
      "cached_tokens_details": {
        "audio_tokens": 0,
        "text_tokens": 0,
        "image_tokens": 0
      }
    },
    "output_token_details": {
      "text_tokens": 32,
      "audio_tokens": 0,
      "image_tokens": 0
    }
  },
  {
    "type": "realtime_model_metrics",
    "label": "livekit.plugins.google.beta.realtime.realtime_api.RealtimeModel",
    "request_id": "GR_88b2375d241d",
    "timestamp": 1754994851.9119549,
    "duration": 6.855750799179077,
    "ttft": 0.3050227165222168,
    "cancelled": false,
    "input_tokens": 4984,
    "output_tokens": 30,
    "total_tokens": 5014,
    "tokens_per_second": 4.3758883423231,
    "input_token_details": {
      "audio_tokens": 4,
      "text_tokens": 4980,
      "image_tokens": 0,
      "cached_tokens": 0,
      "cached_tokens_details": {
        "audio_tokens": 0,
        "text_tokens": 0,
        "image_tokens": 0
      }
    },
    "output_token_details": {
      "text_tokens": 30,
      "audio_tokens": 0,
      "image_tokens": 0
    }
  },
  {
    "type": "realtime_model_metrics",
    "label": "livekit.plugins.google.beta.realtime.realtime_api.RealtimeModel",
    "request_id": "GR_81d1a409b29f",
    "timestamp": 1754994861.9404228,
    "duration": 10.473259210586548,
    "ttft": 0.4400162696838379,
    "cancelled": false,
    "input_tokens": 5043,
    "output_tokens": 41,
    "total_tokens": 5084,
    "tokens_per_second": 3.9147317158498764,
    "input_token_details": {
      "audio_tokens": 5,
      "text_tokens": 5038,
      "image_tokens": 0,
      "cached_tokens": 0,
      "cached_tokens_details": {
        "audio_tokens": 0,
        "text_tokens": 0,
        "image_tokens": 0
      }
    },
    "output_token_details": {
      "text_tokens": 41,
      "audio_tokens": 0,
      "image_tokens": 0
    }
  }
]

I think it's got something to do with the model itself. When I use models/gemini-2.5-flash-exp-native-audio-thinking-dialog the audio token outputs are correct.

I think it's got something to do with the model itself. When I use models/gemini-2.5-flash-exp-native-audio-thinking-dialog the audio token outputs are correct.

https://ai.google.dev/gemini-api/docs/live#audio-generation , Maybe the 2.0 model runs in half-cascade mode, i.e native audio input and text-to-speech output.
But, I don't know then where the audio output component would come from ?

Navigation Menu

Repository navigation

Incorrect Audio Token Count - Gemini Live API RealtimeModel #2426

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Incorrect Audio Token Count - Gemini Live API RealtimeModel #2426

Description

Activity

benjaminreji commented on May 28, 2025

nidalaraf commented on May 28, 2025

davidzhao commented on Jun 1, 2025

etiennedv-mathu commented on Aug 12, 2025

nbl-natsoft commented on Aug 12, 2025

etiennedv-mathu commented on Aug 12, 2025

nbl-natsoft commented on Aug 12, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions