Prompting & Inference

Streaming

Delivering a model response token by token as it is generated, instead of waiting for the whole thing.

Streaming sends each token to the user the moment the model produces it, so text appears progressively rather than after a long pause. It makes an assistant feel far more responsive, since the first words show up almost immediately even on a long answer. The key metric it improves is time to first token, how long before anything appears.