LLM API Output Truncated? Diagnose from finish_reason to Timeouts

Don't Change Parameters Yet—Read finish_reason First

When output cuts off, most people's first instinct is to increase max_tokens. But if the cause isn't the token limit, raising it does nothing except add cost and failure risk. So the first step is to read that status field in the response body.

In the OpenAI-compatible protocol it's choices[0].finish_reason, and in the Anthropic protocol it's stop_reason. It has only a few possible values, and each points to a completely different direction:

length: The model hit the token limit and was forced to stop. This is the most common truncation.
stop: The model thinks it finished writing. If the visible text is clearly incomplete, it's likely the stop parameter was triggered.
content_filter: Content was blocked by safety policy, unrelated to length.
No complete response body at all, or the connection silently drops mid-stream when stream=True: This isn't model truncation; it's a link timeout or proxy disconnect.

Once you distinguish these four, the rest of the troubleshooting becomes meaningful. Here's the order.

finish_reason is length: Three limits—changing the wrong one won't help

length means the max output set in the request was insufficient. What you need to adjust is max_tokens (renamed to max_completion_tokens in newer APIs; same meaning—how many tokens this call is allowed to generate).

But before increasing it, confirm two things, because two ceilings sit above it:

The model's own max output tokens per call. Every model has its own architectural ceiling—common tiers are 4096, 8192, 16384, or higher. No matter how large you set the request, you can't exceed it.
The total context window. Prompt length plus output length cannot exceed the model's context. If your input already takes a large chunk, the output budget is naturally squeezed—in that case, trimming the input is more effective than tweaking parameters.

So the correct order is: first check the model spec's context window and max output, then decide what to put in max_tokens. The spec is on the model page—for example, NexAIX's model list annotates context and output boundaries for each model. Confirm the numbers for your model, then go back to the code and change it.

Reasoning Models: Chain-of-Thought Also Spends Your Output Budget

Reasoning models like o1, o3, and DeepSeek-R1 have a pitfall: the internal reasoning tokens are counted in completion_tokens, meaning they share the same max_completion_tokens quota as the final text.

The consequence is direct: if the quota is set too small, the reasoning process burns through it first, and the final text is either half a sentence or completely empty. It looks like 'incomplete JSON' or 'answer only has a beginning'—appears to be a network issue, but it's actually a budget issue.

There's only one fix: give reasoning models significantly more output budget than ordinary models. The same prompt that works for a regular chat model often isn't enough for a reasoning model.

Content Really Exceeds Single-Call Limit: Use a Continuation Loop, Not Just Parameter Tweaks

One case where parameter tuning truly doesn't work: the content you need to generate is itself longer than the model's max output (e.g., writing dozens of pages in one go). The standard approach is a Continuation Loop.

The idea is simple—continue from wherever the model stopped:

Make a normal request and get the result.
Check finish_reason. If it equals length, it was truncated, not finished.
Take the generated text as an assistant message appended back into messages, then add an user instruction like 'Continue from the last sentence, do not repeat existing content'.
Repeat until the model no longer returns length, or your condition is met (reached an end marker, hit target length, or loop count limit).

Pseudocode looks roughly like this:

full = ""
messages = [{"role": "user", "content": prompt}]
for _ in range(MAX_ROUNDS):
    resp = client.chat.completions.create(
        model=MODEL, messages=messages, max_tokens=MAX_OUT
    )
    choice = resp.choices[0]
    full += choice.message.content
    if choice.finish_reason != "length":
        break
    messages.append({"role": "assistant", "content": choice.message.content})
    messages.append({"role": "user", "content": "继续，接着上一段结尾写，不要重复。"})

Two common pitfalls: first, check the junction—the model may repeat the last sentence of the previous segment. Second, the loop must have a hard exit condition; otherwise a request that keeps returning length will burn budget indefinitely.

A simpler alternative to continuation is splitting the task at the prompt level: generate long text chapter by chapter, output long JSON field by field. For structured output, chunked generation is often more stable and easier to validate than generating everything at once and concatenating.

No Complete Response Body, Streaming Interrupted: The Problem Isn't Parameters, It's Timeout

If your logs never show a complete JSON response, or after enabling stream=True the stream silently stops mid-way and the client throws an exception, that has nothing to do with tokens—the connection was cut.

Long text generation naturally takes time. Default timeouts are often too short (many SDK defaults are only minutes or less), and reverse proxies have their own read timeout (like Nginx's proxy_read_timeout). Whichever layer times out first, the connection breaks.

Two actions:

Enable stream=True. Streaming keeps data flowing, so the connection is less likely to be deemed idle, and you can process as you receive.
Explicitly increase the client timeout. For long text, 120 to 300 seconds is common, or use the SDK's dedicated streaming timeout parameter.

Note: the specific default timeout for a single long-lived connection on this site's gateway is not stated in public documentation, so don't assume it's long enough. The safe approach is to rely on the client timeout setting and run a real stress test with your longest request before going to production; if you really need to confirm gateway behavior, ask via an official ticket. For step-by-step timeout diagnosis, see From 504, 524 to APITimeoutError: Troubleshooting Order.

Incomplete Content but finish_reason is stop

This means 'the model thinks it finished writing', so look elsewhere:

stop parameter triggered accidentally. If you passed a stop sequence like stop=["\n\n"], generation will be cut off once that sequence appears. Remove it first, or replace with a special marker that won't naturally appear in the text.
content_filter. Content was blocked by safety policy; increasing length won't help—you need to adjust the input content itself.

Both fall under 'looks like truncation but isn't'. If you keep investigating along the token limit direction, you'll go in circles. For the full troubleshooting order for such mixed symptoms, see From 401, 404 to Timeouts and Truncation: Diagnose in Order.

Ultimately, It Comes Back to Configuration Itself

When wrapping up, don't forget truncation can be a side effect of configuration migration: after switching endpoints or SDK versions, parameter names change (max_tokens becomes max_completion_tokens), or the client's underlying timeout defaults differ from what you used before.

If you're just connecting your app to a new OpenAI-compatible endpoint, the change is usually just one place—base_url. For example, NexAIX's endpoint is https://api.nexaix.net/v1; keep model names and other parameters as in the official SDK. Access docs and key retrieval are in the official site navigation. This way migration won't introduce extra parameter differences, and truncation issues are easier to trace to either model or client factors.

One-Line Cheat Sheet

length + insufficient value → increase max_tokens / max_completion_tokens, but don't exceed the model's max output and don't let input + output exceed the context window.
length + reasoning model → chain-of-thought is consuming budget; give a larger output quota.
length + content is genuinely too long → continuation loop or task chunking, not parameter tweaks.
No complete response / streaming interrupted → stream=True, set client timeout to 120 seconds or more.
stop but content missing → check stop parameter; content_filter → adjust input.