Structured outputs

Often you want data, not prose: a list of fields from an invoice, a category label, a set of steps. This guide shows two ways to get JSON out of a model and how to check it.

The gateway does not currently enforce a response schema for you. There is no response_format field, so a model is asked to produce JSON and your code must verify it. Both approaches below work today.

Send structured data in

If the model should read data, use a JSON part instead of pasting text. The gateway serializes it.

{
	role: 'user',
	content: [
		{ type: 'text', text: 'Which of these orders is largest?' },
		{ type: 'json', data: { orders: [{ id: 'a1', total: 40 }, { id: 'b2', total: 95 }] } }
	]
}

JSON parts are for input only. A model never returns one.

Option 1: ask for JSON

Describe the shape you want in a system message, keep temperature low, and parse the text when the turn ends.

import { connect } from './gateway.js';

const gateway = await connect();

const result = await gateway.turn({
	provider: 'auto',
	model: 'auto',
	messages: [
		{
			role: 'system',
			content: [
				{
					type: 'text',
					text:
						'Reply with a single JSON object and nothing else. ' +
						'Shape: {"vendor": string, "total": number, "currency": string}.'
				}
			]
		},
		{ role: 'user', content: [{ type: 'text', text: 'Invoice: Acme Ltd, total 1,240.50 EUR.' }] }
	],
	params: { temperature: 0 }
});

const invoice = JSON.parse(result.text);
console.log(invoice.vendor, invoice.total);
gateway.close();

Models sometimes wrap JSON in a code fence or add a sentence. Strip a surrounding fence before you parse, and handle a SyntaxError by retrying. Check capabilities.json_mode in the model catalog to see which models are good at this. With auto routing, the gateway prefers those models when your request includes a JSON part.

Option 2: use a tool as the schema

A tool declaration carries a JSON Schema. The model’s arguments arrive in the tool_call event already parsed, and the model is guided by the schema. That is a reliable way to get typed output, even though you never run the tool. You should still validate it.

const result = await gateway.turn({
	provider: 'auto',
	model: 'auto',
	tools: [
		{
			name: 'record_invoice',
			description: 'Record the details of an invoice.',
			parameters: {
				type: 'object',
				properties: {
					vendor: { type: 'string' },
					total: { type: 'number' },
					currency: { type: 'string', description: 'ISO 4217 code, for example EUR' }
				},
				required: ['vendor', 'total', 'currency']
			}
		}
	],
	messages: [
		{
			role: 'system',
			content: [{ type: 'text', text: 'Always answer by calling record_invoice.' }]
		},
		{ role: 'user', content: [{ type: 'text', text: 'Invoice: Acme Ltd, total 1,240.50 EUR.' }] }
	]
});

const call = result.toolCalls.find((c) => c.name === 'record_invoice');
console.log(call.input); // { vendor: 'Acme Ltd', total: 1240.5, currency: 'EUR' }

There is no way to force a specific tool, so tell the model to call it in the system message, and handle the case where toolCalls is empty. The turn ends with stop_reason: "tool_use". You do not need to send a result back.

Declaring tools limits routing to models that support tools. If none match, you get no_capable_model.

Always validate

Treat model output like any other untrusted input. Check it against your own schema before you use it, for example with a library such as Ajv or Zod:

const valid = typeof invoice.vendor === 'string' && Number.isFinite(invoice.total);
if (!valid) {
	// Ask again, or fall back to a person.
}

A short retry loop that sends the error back to the model (“Your JSON was missing currency. Reply again.”) fixes most failures. See Conversation state for how to add that turn.

Next steps

• Function calling
• Errors and handling