Visual Intelligence Image Search with App Intents

Visual Intelligence can show content from a third-party app when someone searches a camera view, photo, or screenshot. The integration is not a camera overlay and it does not give an app unrestricted access to what the person sees. The system sends a captured scene to an App Intents query, the app returns matching AppEntity values, and Visual Intelligence presents those results.

That distinction is the architecture. Your app owns its catalog and matching logic. The system owns capture, invocation, and result presentation.

This guide follows Apple’s current iOS 27 beta documentation. Verify API signatures against the SDK used for release because the Visual Intelligence framework is still marked beta.

Decide whether the feature has a real catalog to search

Visual search works best when a captured object can map to content your app already understands. Useful examples include:

• album artwork that maps to albums or concerts;
• plants that map to care records;
• products that map to a catalog;
• landmarks that map to destination pages;
• book covers that map to reading lists.

A vague “AI feature” is not enough. Before adding any framework code, define the entity being returned, its stable identifier, the local or remote catalog searched, the relevance threshold, and the screen that opens when a result is selected.

If the app has no durable content identity, start with App Entities and Siri integration. Visual Intelligence depends on the same model.

Model the result as an AppEntity

The result needs a stable ID and a compact display representation. Visual Intelligence has limited room, so put distinguishing information in the title, subtitle, and thumbnail.

import AppIntents

struct PlantEntity: AppEntity {
    static var typeDisplayRepresentation: TypeDisplayRepresentation {
        "Plant"
    }

    static var defaultQuery = PlantEntityQuery()

    let id: String
    let commonName: String
    let scientificName: String
    let thumbnailData: Data

    var displayRepresentation: DisplayRepresentation {
        DisplayRepresentation(
            title: "\(commonName)",
            subtitle: "\(scientificName)",
            image: .init(data: thumbnailData)
        )
    }
}

Use thumbnail-sized data or a thumbnail URL. A full-resolution asset increases latency without improving a small result card. Also localize the type name, title, and subtitle. These representations may appear in Siri, Spotlight, Shortcuts, and other system surfaces, not only Visual Intelligence.

The entity’s ID must survive relaunches and catalog refreshes. An array index or temporary object identifier cannot safely support opening the result later.

Implement one IntentValueQuery for visual search

Visual Intelligence sends a SemanticContentDescriptor to a query that adopts IntentValueQuery. The descriptor can expose system-generated labels and a pixel buffer for the captured scene.

import AppIntents
import VisualIntelligence

struct PlantVisualQuery: IntentValueQuery {
    @Dependency var catalog: PlantCatalog

    func values(
        for input: SemanticContentDescriptor
    ) async throws -> [PlantEntity] {
        let labelMatches = catalog.matches(labels: input.labels)

        guard let pixelBuffer = input.pixelBuffer else {
            return Array(labelMatches.prefix(6))
        }

        let imageMatches = try await catalog.matches(image: pixelBuffer)
        return rankAndDeduplicate(
            labelMatches + imageMatches,
            limit: 6
        )
    }
}

The exact descriptor members and availability annotations can change during the beta cycle. Keep access to the descriptor in one adapter so an SDK change does not spread through the catalog layer.

An app can contain only one IntentValueQuery that accepts SemanticContentDescriptor. If you need to return several entity types, use @UnionValue rather than adding parallel visual queries.

Match locally before reaching for a server

Apple’s WWDC26 example uses Vision image feature prints: precompute a feature print for each catalog image, create one for the captured pixel buffer, compare distances, then apply a threshold. This can keep matching on-device and avoid uploading the captured scene.

The production pipeline should be explicit:

1. Generate and store feature prints when catalog assets are added or updated.
2. Resize large captures before feature extraction, especially on macOS.
3. Compute one feature print for the incoming scene.
4. Compare against indexed candidate prints rather than decoding every original image.
5. Reject results beyond a measured distance threshold.
6. Rank the remaining matches and return a small result set quickly.

Labels can narrow the candidate set before image comparison. Do not treat them as guaranteed classifications; use them as hints and test across screenshots, camera captures, crops, lighting changes, and visually similar items.

If matching requires a server, disclose that captured information leaves the device, minimize the payload, and handle offline use honestly. A network-backed matcher also needs a strict latency budget because Visual Intelligence is waiting for the query.

Return more than one entity type with UnionValue

A captured album could match both an album and an upcoming concert. Apple’s documented pattern is a union result:

@UnionValue
enum VisualSearchResult {
    case plant(PlantEntity)
    case collection(PlantCollectionEntity)
}

struct CatalogVisualQuery: IntentValueQuery {
    func values(
        for input: SemanticContentDescriptor
    ) async throws -> [VisualSearchResult] {
        // Rank entities from all supported result types together.
        []
    }
}

Only add a second type when it improves the next action. Returning loosely related content makes the result sheet slower and less trustworthy.

Open the selected result directly

Adopt OpenIntent for each entity type so a tap reaches the corresponding detail screen.

struct OpenPlantIntent: OpenIntent {
    static let title: LocalizedStringResource = "Open Plant"

    @Parameter(title: "Plant")
    var target: PlantEntity
}

Keep the foregrounding path lightweight. Resolve the stable ID, update navigation state, and show the destination. Do not rerun image matching after the person has already selected a result.

Reuse an existing OpenIntent if the app already exposes one to Siri or Spotlight. Visual Intelligence should be another route into the same content model rather than a second navigation system.

Let people continue into a full search

Visual Intelligence should return a short, relevant set. If the catalog contains many possible matches, add an app intent that conforms to the visual intelligence semantic-content-search schema. Its perform() method can open the app’s complete search interface and carry the captured context into that experience.

This is better than returning dozens of result cards. The system surface handles quick recognition; the app handles filters, comparison, account state, purchasing, and deeper browsing.

Measure relevance and latency separately

Build a test set that includes:

• clean catalog images;
• partial crops and occlusion;
• camera glare and low light;
• screenshots with surrounding UI;
• near-duplicate items;
• scenes with no valid match.

Measure top-one precision, top-three recall, false-positive rate, query duration, and peak memory. A fast wrong answer is not useful, and a perfect result that arrives after the system times out is equally unusable.

Keep a “no confident match” path. Returning no values is more honest than filling the sheet with weak guesses.

Production checklist

• Use stable App Entity identifiers.
• Keep display representations localized and concise.
• Implement one semantic-content IntentValueQuery.
• Use @UnionValue when several entity types are necessary.
• Precompute catalog feature prints instead of processing original assets per query.
• Limit and rank results with a measured confidence threshold.
• Add an OpenIntent for every returned entity type.
• Offer in-app continuation for large result sets.
• Test camera images, screenshots, misses, latency, memory, and offline behavior.
• Document whether matching stays on-device or uses a server.

Apple references

• Integrating your app with visual intelligence
• Visual Intelligence framework
• Best practices for integrating visual intelligence in your app
• Getting started with App Intents

For a product that already has App Entities, Visual Intelligence can add a valuable discovery route without rebuilding the underlying catalog. A focused iOS architecture audit can validate the entity model, privacy boundary, and matching pipeline before the integration reaches a release branch.