Skip to main content

Structured Output & JSON

Constrain model responses to valid JSON following a schema you provide. This makes model output safe to parse and wire into application logic.

Supported models​

JSON mode is model-specific. See the Model Catalog for the current set of models that support response_format. The catalog is the source of truth and is kept up to date as models are added or retired.

Overview​

By default, models respond with natural language text. Using the response_format parameter, you can constrain the output to:

  • JSON Schema: follows a strict schema you provide
  • JSON Object: valid JSON with no specific schema

Useful for extracting structured data from text, building pipelines that require typed outputs, and keeping response formats consistent across requests.

Response Format Limit

Nebul enforces a strict limit of 200 flattened fields in response_format. Requests that exceed this limit are blocked.

To avoid performance issues and schema-size failures, see:

JSON Schema Mode​

Provide a JSON schema and the model will produce output that conforms to it.

python
12345678910111213141516171819202122232425262728293031323334353637383940414243
import json
from openai import OpenAI
from pydantic import BaseModel
from typing import List, Literal


client = OpenAI(
    api_key="sk-your-api-key-here",
    base_url="https://api.inference.nebul.io/v1"
)


# Define your schema using Pydantic
class MovieInfo(BaseModel):
    title: str
    year: int
    director: str
    genres: List[Literal["drama", "comedy", "thriller", "sci-fi", "horror", "action"]]
    rating: float




response = client.chat.completions.create(
    model="Qwen/Qwen3-30B-A3B-Instruct-2507",
    messages=[
        {
            "role": "system",
            "content": "Extract movie information from the user's description. Respond only with valid JSON matching the schema.",
        },
        {
            "role": "user",
            "content": "The Dark Knight from 2008, directed by Christopher Nolan. It's an action thriller rated 9.0",
        },
    ],
    response_format={
        "type": "json_schema",
        "json_schema": {"name": "movie_info", "schema": MovieInfo.model_json_schema()},
    },
)
print("JSON output")
print(response.choices[0].message.content)


print("Filtered output")
movie = json.loads(response.choices[0].message.content)
print(f"{movie['title']} ({movie['year']}) - {movie['rating']}/10")

Example Output​

json
1234567
{
  "title": "The Dark Knight",
  "year": 2008,
  "director": "Christopher Nolan",
  "genres": ["action", "thriller"],
  "rating": 9.0
}

JSON Object Mode​

When you need valid JSON but don't want to enforce a specific schema, use json_object mode. The model will produce well-formed JSON based on your prompt instructions.

python
12345678910111213141516171819202122232425
import json
from openai import OpenAI


client = OpenAI(
    api_key="sk-your-api-key-here",
    base_url="https://api.inference.nebul.io/v1"
)


response = client.chat.completions.create(
    model="Qwen/Qwen3-30B-A3B-Instruct-2507",
    messages=[
        {
            "role": "system",
            "content": "You are an API that returns product information as JSON. Include name, price, and availability."
        },
        {
            "role": "user",
            "content": "Tell me about the iPhone 15 Pro"
        }
    ],
    response_format={"type": "json_object"}
)


data = json.loads(response.choices[0].message.content)
print(data)
tip

Always instruct the model to output JSON in your system or user prompt, even when using response_format. This improves reliability and output quality.

Schema Best Practices​

Use Descriptive Field Names​

python
1234567891011
# Good - self-documenting
class OrderSummary(BaseModel):
    order_id: str
    total_amount_usd: float
    items_count: int


# Avoid - ambiguous
class Order(BaseModel):
    id: str
    total: float
    count: int

Include Field Descriptions​

python
1234567891011121314
from pydantic import BaseModel, Field


class CustomerFeedback(BaseModel):
    sentiment: Literal["positive", "negative", "neutral"] = Field(
        description="Overall sentiment of the customer feedback"
    )
    key_points: List[str] = Field(
        description="Main points or concerns raised by the customer"
    )
    urgency: int = Field(
        description="Urgency level from 1 (low) to 5 (critical)",
        ge=1,
        le=5
    )

Handle Potential Refusals​

Some requests may result in the model refusing to respond. Always check for refusals:

python
12345678910
output = response.choices[0].message


if output.refusal:
    print(f"Model refused: {output.refusal}")
elif output.content:
    try:
        data = json.loads(output.content)
        # Process data
    except json.JSONDecodeError as e:
        print(f"Invalid JSON: {e}")

Schema Size, Memory Usage, and Reliability​

The schema you pass to response_format is not just a validation hint. It becomes part of the request and actively participates in how the model generates output.

In json_schema mode, the schema is serialized, transmitted, kept in memory, and consulted throughout decoding. This is what lets the model produce strongly typed, well-formed JSON, but it also means very large or complex schemas increase memory usage and processing time, even when the final output is small. The cost of enforcing the schema can outweigh the cost of the output itself.

As schema size grows you may see slower request initialization, longer generation times, or reduced reliability. In extreme cases requests fail from memory exhaustion or internal timeouts, and the cause is the schema rather than the prompt or output.

Takeaways
  • json_schema mode actively constrains decoding, not just validation.
  • Schema complexity affects memory usage and latency even for small outputs.
  • Reliability issues may originate from the schema, not the prompt.

What Causes Schemas to Become Large​

Schemas often grow unintentionally. Deep nesting, large enumerations, or heavily reused model definitions can expand the serialized schema far beyond what the original Python or JSON definition suggests.

This commonly happens when Pydantic models are reused directly from application logic. Database models, API response objects, and domain models frequently contain far more structure than inference requires.

Schemas that rely heavily on $defs, anyOf, or oneOf are particularly expensive because they force the model to reason over many possible output shapes during generation.

Takeaways
  • Deep nesting, large enums, and unions are the biggest contributors.
  • Reusing full application or database models often causes schema bloat.
  • $defs combined with anyOf / oneOf can rapidly increase complexity.

Designing Schemas for Stability​

Treat schemas as interfaces, not as full representations of internal data models.

Include only the fields required for downstream logic. Prefer identifiers or compact categorical values over deeply embedded objects. If a constraint can be validated safely in your application code, keep the schema simpler and enforce that constraint outside the model.

When extracting large or complex structures, split the task across multiple calls with smaller schemas rather than enforcing everything in a single request.

Takeaways
  • Keep schemas minimal and task-focused.
  • Prefer IDs or references over embedded objects.
  • Split complex extractions into multiple smaller calls.

Inspecting and Debugging the response_format​

Schema size is not always obvious, especially with Pydantic models that use nested references. Inspecting the final response_format payload helps identify unexpected complexity before it causes issues in production.

The OpenAI SDK exposes an internal helper that converts a Python type into the final schema sent to the API.

Takeaways
  • Inspect the generated response_format, not just your model definition.
  • Nested Pydantic models can produce much larger schemas than expected.

Flattening the Generated Schema (Python)​

python
123456789101112131415161718192021222324252627282930
from typing import Any, Dict
from openai.lib._parsing import type_to_response_format_param




def flatten_json(obj: Any, prefix: str = "", out: Dict[str, Any] | None = None) -> Dict[str, Any]:
    """
    Flatten a JSON-like structure (dicts + lists) into a single-level dict.
    """
    if out is None:
        out = {}


    if isinstance(obj, dict):
        for k, v in obj.items():
            key = f"{prefix}.{k}" if prefix else str(k)
            flatten_json(v, key, out)


    elif isinstance(obj, list):
        for i, v in enumerate(obj):
            key = f"{prefix}[{i}]"
            flatten_json(v, key, out)


    else:
        out[prefix] = obj


    return out




my_response_format_flattened = flatten_json(
    type_to_response_format_param(MY_PYDANTIC_MODEL_CLASS)
)

Flattening the schema exposes how deep it goes, how many definitions are involved, and where complexity accumulates. This is especially useful when schemas are composed from multiple nested models.

Takeaways
  • Flattening exposes hidden depth and repetition
  • Useful for spotting large enums and deeply nested paths
  • Useful for checking what hidden configs/definitions are send with the request

Checking Field Count Limits​

To protect performance and reliability, Nebul enforces a hard limit on schema complexity.

Requests are blocked when the flattened response_format contains more than 200 fields. This prevents extremely large schemas from consuming excessive memory or causing long-running constrained decoding.

You can check this locally before sending a request:

python
1234567891011121314
from openai.lib._parsing import type_to_response_format_param


rf = type_to_response_format_param(MyPydanticModel)
flat = flatten_json(rf)


field_count = len(flat)


print(f"Flattened response_format fields: {field_count}")


if field_count > 200:
    raise ValueError(
        f"response_format too large: {field_count} fields (limit: 200). "
        "Reduce nesting, remove unused fields, or split the task into multiple calls."
    )
Important
  • Nebul blocks schemas with more than 200 flattened fields
  • Always validate locally to avoid runtime request failures
  • If you hit the limit, reduce nesting or split the task
Last updated on