Image inputs

Models that support vision can read images you put in a user message. Add an image part next to your text, and ask your question.

Analyze an image by URL

import { connect } from './gateway.js';

const gateway = await connect();

const result = await gateway.turn({
	provider: 'auto',
	model: 'auto',
	messages: [
		{
			role: 'user',
			content: [
				{ type: 'text', text: 'What is in this image?' },
				{
					type: 'image',
					source: { kind: 'url', url: 'https://example.com/photo.jpg' }
				}
			]
		}
	]
});

console.log(result.text);
gateway.close();

The URL must be a full, valid address that the model provider can fetch. A private or expired link will fail. If the URL cannot be parsed, the turn ends with an invalid_request error.

Send an image as base64

Use this for local files or images that are not public.

import { readFile } from 'node:fs/promises';

const data = (await readFile('chart.png')).toString('base64');

const result = await gateway.turn({
	provider: 'auto',
	model: 'auto',
	messages: [
		{
			role: 'user',
			content: [
				{ type: 'text', text: 'Summarize this chart.' },
				{ type: 'image', source: { kind: 'base64', media_type: 'image/png', data } }
			]
		}
	]
});

media_type must start with image/, for example image/png or image/jpeg. Send the raw base64 string, without a data: prefix.

Send several images

Add more image parts to the same message. Order matters, so put each image next to the text that refers to it.

content: [
	{ type: 'text', text: 'Image A:' },
	{ type: 'image', source: { kind: 'url', url: 'https://example.com/a.jpg' } },
	{ type: 'text', text: 'Image B:' },
	{ type: 'image', source: { kind: 'url', url: 'https://example.com/b.jpg' } },
	{ type: 'text', text: 'What changed between A and B?' }
];

Choosing a model

Not every model can see. With provider: "auto" and model: "auto", the gateway only considers models with vision support as soon as your request contains an image. If you name a model that cannot see images, the turn fails with no_capable_model.

You can check support in the model catalog: vision models have capabilities.vision set to true.

Limits and costs

• Images go in user messages only. A system message must be text.
• Images count toward input tokens and the model’s context window. When deciding which models can fit your request, the gateway counts each image as roughly 1,000 tokens. The provider reports the exact count in the final usage.
• Large images use more tokens. Resize them to what the task needs.
• Models currently stream back text. The catalog lists text models only.

Next steps

• Structured outputs: turn what the model sees into JSON.
• Conversation state: ask follow-up questions about the same image.