Prompting & Inference
Streaming
Delivering a model response token by token as it is generated, instead of waiting for the whole thing.
Streaming sends each token to the user the moment the model produces it, so text appears progressively rather than after a long pause. It makes an assistant feel far more responsive, since the first words show up almost immediately even on a long answer. The key metric it improves is time to first token, how long before anything appears.
Related terms
Inference
Actually running a trained model to get an answer, as opposed to training it in the first place.
Token
The small chunk of text a language model actually reads and writes, usually a word or part of a word.
Chatbot
A program you interact with through conversation, now usually powered by a large language model.