[andrewheard]

GeminiLanguageModel now sends each image attachment's label to the model as a text part, such as [barcode-image], right before the image. This makes tools that take an ImageReference, such as Vision's BarcodeReaderTool and OCRTool, work with labeled images, the same way they do with the on-device model.

Motivation

Tools that take an ImageReference find an image in the transcript by its attachment label. Their tool descriptions tell the model that the label is "the identifier shown with the image". #16895 didn't send labels, so Gemini couldn't see them. Unless the prompt text happened to mention the label, Gemini made one up (for example, input_file_0.png, image_0, or img_0). The tool then threw invalidImage("No image in transcript with label …"), and respond failed with LanguageModelSession.ToolCallError. That breaks Apple's own BarcodeReaderTool example, which labels the image but never mentions the label in the prompt.

I compared both tools with one and two images across all three integration backends. On-device runs used SystemLanguageModel directly, once per case:

Case	SystemLanguageModel	Gemini, before	Gemini, after
Labeled, label mentioned in prompt	✅	✅ 12/12	✅ 12/12
Labeled, label not mentioned in prompt	✅	❌ 0/12	✅ 12/12
Unlabeled	❌	❌ 0/12	❌ 0/12

After this change, Gemini behaves like the on-device model in every case. Unlabeled images fail on both, because Foundation Models doesn't assign labels to attachments. The fm CLI only gets around this because it labels images image_0, image_1, and so on itself.

Changes

• GeminiTranscriptTranslator: a labeled image in a prompt is preceded by a text part containing its label in square brackets. Unlabeled images, and images with an empty label, are sent exactly as before. The bracket format matches what the on-device model echoed back in one of the runs.
• GeminiLanguageModel docs: the vision capability docs now describe how labels are sent, and that tools taking an ImageReference can only refer to labeled attachments. The previous "labels aren't sent" limitation is removed.

Testing

Details

• Unit tests (GeminiTranscriptTranslatorTests): a label part right before its image; label parts placed between surrounding text and in order for multiple images; only labeled images get label parts; empty labels are skipped.
• Integration tests (new ImageReferenceToolIntegrationTests): BarcodeReaderTool and OCRTool with one and two labeled images, whose labels the prompt never mentions, on all three backends (12 cases). Each test checks the transcript, not just the final answer, since Gemini can read text and QR codes by itself:
  • every tool call used an attached label,
  • every tool output contains the expected payload,
  • the response contains every payload.
• Check that the integration test depends on the fix: with label sending temporarily disabled, all 12 integration cases fail with ToolCallError.
• Both toolchains: unit tests pass on Xcode 27.2 beta 2 (Swift 6.4) and beta 3 (Swift 6.4.2), and swift-format lint --strict is clean.

Usage examples

Read a barcode with BarcodeReaderTool

This is Apple's documented example, unchanged. The prompt doesn't need to mention the label.

import FirebaseAILogic
import FoundationModels
import Vision

func describeBarcode(in image: CGImage) async throws -> String {
  let model = FirebaseAI.firebaseAI().geminiLanguageModel(name: "gemini-model-name")
  let session = LanguageModelSession(model: model, tools: [BarcodeReaderTool()])

  let response = try await session.respond {
    """
    Scan this image for any barcodes. For each barcode found, describe its symbology type and \
    explain what the encoded content means or represents.
    """
    Attachment(image)
      .label("barcode-image")
  }
  return response.content
}

Gemini receives [barcode-image] right before the image and calls the tool with {"image": {"attachmentLabel": "barcode-image"}}.

Read text from several images with OCRTool

Give each image a distinct label. Gemini calls the tool once per image, using each image's own label.

import FirebaseAILogic
import FoundationModels
import Vision

func readReceipts(_ receipts: [CGImage]) async throws -> String {
  let model = FirebaseAI.firebaseAI().geminiLanguageModel(name: "gemini-model-name")
  let session = LanguageModelSession(model: model, tools: [OCRTool()])

  let response = try await session.respond {
    "Read the text in each receipt and list each store name with its total, in order."
    for (index, receipt) in receipts.enumerated() {
      Attachment(receipt)
        .label("receipt-\(index + 1)")
    }
  }
  return response.content
}

Unlabeled images can't be used with these tools

Tools that take an ImageReference can only find labeled images. This is true for the on-device model too. Without a label, the model makes one up, and respond throws:

let session = LanguageModelSession(model: model, tools: [BarcodeReaderTool()])

do {
  _ = try await session.respond {
    "Use the barcode reader tool on this image."
    Attachment(image)  // No label.
  }
} catch let error as LanguageModelSession.ToolCallError {
  // error.underlyingError: invalidImage("No image in transcript with label image_0")
}

Label every image that a tool might need to refer to, for example with Attachment(image).label("image-1").

[gemini-code-assist]

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.