Models: reasoning, budgets, and caching

Three things decide how a model call runs, and they sit in different places. The model node (OpenRouterModel, say) says WHICH model. The LLM node itself says what it is given and what comes back. And [params], the node wired into an LLM node's params input, says how it thinks: everything below that is not about the prompt is a setting on it.

If you need a real model id for the model node, run weft options <node> model --search <text> from the project, with the model node's name as <node>. It prints one id label line per matching model, the same list the editor's search shows, read through the connection picked on the access node wired into its access, and the id is what you write in model. Never guess an id, and never ask the provider's API yourself.

Everything here is about the model's behaviour, which no metadata file can tell you: what a setting DOES to an answer, what it costs, and how the pieces interact across a chain of calls.

Reasoning: unset, off or on

reasoning on [params] has three states. Unset sends nothing, so the model runs at its provider's default, which for many models is to reason and bill for it. false asks for no reasoning. true reasons at the reasoningEffort you picked, sent as written, and at the provider's lowest effort (the fast, cheap one) when you picked none.

• Off asks the provider for no reasoning, on every provider; a plain model runs as it always does. A model that cannot switch reasoning off refuses the call, and the failure reads This model always reasons; set `reasoning: true` on its params (the provider's lowest effort unless you pick one).
• On sends the effort. A model that has no reasoning fails with its provider's own raw error, or ignores the setting; when the model says it cannot reason, you set false.

There is no retry behind either failure: the fix is one edit on [params]. Most modern models reason, so on a model you do not know you start with true and raise the effort only for the one node that needs it.

A prompt that asks the model for its thinking inside the answer (a reasoning key in a JSON reply, a "think first" section) runs with the model's own reasoning off. If you catch yourself wiring reasoning: true under such a prompt, stop and write: "Wait. Paying for thinking twice." Then set reasoning: false on [params].

maxTokens bounds the whole reply, thinking included

Unset, nothing is sent and the provider's default applies. Set, it is the most tokens the reply may hold, and on a reasoning model the thinking counts against it, so a reasoning model with a small maxTokens spends the whole budget thinking and answers nothing. That failure reads the model spent N reasoning tokens and answered 0 text tokens: raise maxTokens so the answer fits after the thinking, or set reasoning off. Those are the two fixes, and you raise maxTokens first.

A reply that is empty for another reason fails as the provider returned an empty response (no text, tool calls, or media): the model answered nothing, and you look at the prompt.

What a class of model costs

Rough magnitudes for choosing, never for billing:

• A small fast model answers in a second or two and costs cents per thousand calls. You use it for routing, extraction, classification, anything that forks the graph on a value it computes. It does not follow an elaborate manipulation, so when a screen or a gate is worth its extra call, the small model plays it (the weft-safety skill says when to offer that).
• A large model answers in several seconds and costs ten to fifty times more per token. You use it where the answer is the product.
• A reasoning model at high effort takes tens of seconds to minutes and bills every reasoning token. You use it for one hard node. If you catch yourself wiring it into every turn of a conversation, stop and write: "Wait. One hard node." Then give that one node its own [params] node.

Two nodes wired to one [params] share every setting; a node that needs a different model or effort gets its own [params] node, with a comment saying why.

Prompt caching

Providers that cache reuse the prefix of a conversation they have already seen, which is where most of the cost of a chain of calls sits. A [mark] is the place on a message where caching is requested; weft puts [mark]s on the messages themselves and the library translates them per provider, so you write nothing provider-specific.

A node with autoCache on (the default wherever the input exists) places [mark]s for you: with a conversation that carries no [mark], the request marks the system message and the last message before this call's new turn (a conversation with no system message marks the last one alone). The [mark]s live on the request alone; the conversation file keeps the conversation as written, so each call in a chain marks the prefix it sees. You chain the calls through one conversation file (historyFile), and the shared prefix is cached from the second call on.

To place [mark]s yourself, set cacheBreakpoint: true on a Chat Message node: everything up to and including that message is cached. A conversation that carries any [mark] of yours turns the automatic ones off, so you place them all or none. The wires that translate [mark]s (the Anthropic family, and OpenRouter where it fronts one) keep at most four, the last ones win, so marking freely never fails a call; the wires with no marker concept drop the [mark]s with a warning in the log, and a call never fails on one.

Whichever way the [mark]s land, the shape that caches is:

• the stable persona and reference material first, in the system message;
• the volatile values (the time, the user's name, today's notecard) last, in the newest user turn, never inside the persona;
• the conversation as real turns appended with Chat Message, never one blob of text rebuilt every call: a rebuilt blob is a new prefix every time and caches nothing.