AI Behavior & Processing
The machinery beneath model behavior: architecture, training, inference, token processing, sampling, multimodality, generation, routing, evaluation, and the limits hiding behind confident output.
130 terms
Artificial Neural Network — Neural Network
A computational system made of connected layers that learn patterns from data.
Neural networks transform input through weighted connections and nonlinear operations. Training adjusts those weights so the network becomes better at a target task.
A neural network can learn to classify images, predict text, or recognize speech.
Artificial intelligence, deep learning
A neural network is inspired by biological brains, but it is not a literal digital brain.
Transformer
A neural-network architecture designed to process relationships within sequences.
Transformers use attention mechanisms to weigh how strongly different parts of an input relate to one another. They underpin many modern language, vision, and multimodal models.
Most current large language models use transformer-based architectures.
Large language model
A transformer is the architecture. An LLM is a trained model that may use that architecture.
Attention
A mechanism that helps a model decide which parts of the input matter most to each other.
Attention computes relationships between tokens or features so the model can weight relevant context when producing internal representations.
In a sentence, attention helps connect a pronoun to the noun it refers to.
Memory, focus
Attention is a mathematical operation, not conscious concentration.
Self-Attention
Attention applied within one sequence so its parts can relate to each other.
Self-attention lets each token compare itself with other tokens in the same input and build a context-sensitive representation.
A word can be interpreted differently depending on the surrounding words.
Cross-attention
Self-attention does not mean the model is self-aware.
Cross-Attention
Attention that connects one set of information to another.
Cross-attention allows one representation, such as generated text, to attend to another source, such as an image, audio signal, or encoder output.
An image generator using text instructions to guide visual generation.
Self-attention
Cross-attention links separate information streams; self-attention relates items within the same stream.
Parameter
A learned numerical value inside a model.
Parameters determine how the model transforms input. Training updates them to reduce error or improve a target objective.
A model described as having 7 billion parameters contains roughly 7 billion learned values.
Function parameter, setting
A model parameter is not the same as a function parameter or API option.
Weights
The learned numerical values that shape a model’s behavior.
Weights are model parameters adjusted during training. They encode statistical patterns learned from data.
Fine-tuning changes some or all of a model’s weights.
Settings, memory
Weights do not store experiences as neatly readable sentences.
Checkpoint
A saved version of a model’s learned state.
A checkpoint usually contains model weights and may include optimizer state, configuration, tokenizer files, or training metadata.
A team may compare checkpoints from different stages of training.
Model release, snapshot
A checkpoint is the saved model state, not necessarily a complete deployable product.
Model Size
A rough measure of how large a model is, often described by parameter count.
Model size may refer to parameter count, memory footprint, file size, or compute requirements depending on context.
A 70B model has about 70 billion parameters.
Context window
A larger model is not automatically better for every task.
Model Capacity
How much complexity a model can represent or learn.
Capacity depends on architecture, parameter count, training, and constraints. Too little capacity can underfit; too much can increase cost or overfitting risk.
A small classifier may lack the capacity for highly nuanced distinctions.
Model size
Parameter count is one contributor to capacity, not a complete measure of intelligence or quality.
Tokenization
Breaking text or other input into units a model can process.
A tokenizer converts raw input into token identifiers and may also reconstruct output tokens back into text.
A long word may be split into several smaller tokens.
Chunking
Tokenization is a model-input operation. Chunking is a higher-level strategy for dividing larger content.
Tokenizer
The component that converts text into tokens and back again.
A tokenizer defines the model’s vocabulary and the rules used to encode and decode text.
Two models may count the same sentence as a different number of tokens.
Model
The tokenizer is separate from the model weights, though the two must be compatible.
Vocabulary
The set of tokens a tokenizer knows how to represent.
A model vocabulary contains token units and their identifiers. It may include words, subwords, punctuation, symbols, or special tokens.
A tokenizer vocabulary might contain separate tokens for common word fragments.
Training data
A token vocabulary is not a dictionary of meanings.
Special Token
A token reserved for structural or control purposes.
Special tokens may mark message boundaries, padding, starts or ends of sequences, tool calls, or roles.
A model may use hidden boundary tokens between system and user messages.
Prompt text
Special tokens may affect processing even when they are not visible in the interface.
Embedding
A numerical representation of meaning or features.
Embeddings map text, images, audio, or entities into vectors so similar items can be compared mathematically.
Two semantically similar sentences may have nearby embeddings.
Model weights, memory
An embedding is not the original content. It is a compressed numerical representation.
Vector
An ordered list of numbers used to represent information.
Vectors can encode features, positions, meanings, or model states in a mathematical space.
An embedding is typically stored as a vector of floating-point numbers.
Array
A vector is mathematically meaningful because of how its numbers relate within a space.
Latent Space
An internal numerical space where a model represents learned features.
Latent spaces encode compressed patterns and relationships that may correspond to concepts, styles, shapes, or semantic structure.
Image models operate on latent representations rather than raw pixels during much of generation.
Embedding space
A latent space is not a literal hidden room containing complete concepts.
Representation
The internal form a model uses to encode information.
Representations transform raw input into learned features that support prediction, generation, or classification.
A sentence becomes a sequence of internal vectors before output is generated.
Embedding
Embeddings are one kind of representation, not the only kind.
Activation
The numerical output produced by a model unit or layer during processing.
Activations are intermediate values created when input passes through the model and are shaped by weights and activation functions.
Researchers may inspect activations to study model behavior.
Weight
Weights are stored learned values. Activations are temporary values produced during inference.
Layer
One stage in a neural network’s processing pipeline.
Layers apply transformations to representations and pass results forward through the model.
A transformer contains repeated attention and feed-forward layers.
Module
More layers do not automatically guarantee better performance.
Inference
Running a trained model to produce an output.
Inference converts input into predictions or generated content using fixed model weights.
Sending a prompt and receiving a completion.
Training
Inference normally uses the model; it does not retrain the model.
Forward Pass
Passing input through the model to calculate output.
A forward pass computes layer-by-layer activations using the current weights.
Each next-token prediction requires model computation over the current sequence.
Backpropagation
Forward passes happen during both training and inference. Backpropagation happens during training.
Backpropagation
The training process used to calculate how model weights should change.
Backpropagation propagates error gradients backward through the network so optimization can update parameters.
Training compares output against a target and backpropagates the error.
Inference
Backpropagation is a learning mechanism, not the ordinary generation process.
Loss Function — Loss
A numerical measure of how wrong a model’s output is during training.
The loss function defines the objective the optimizer tries to reduce.
A language model may be trained to reduce next-token prediction loss.
Evaluation metric
Lower training loss does not always mean better real-world usefulness.
Gradient
A signal showing how model parameters should change to reduce loss.
Gradients represent the direction and magnitude of change for each parameter during optimization.
The optimizer uses gradients calculated through backpropagation.
Weight
Gradients are temporary update signals, not the final learned parameters.
Optimizer
The algorithm that updates model weights during training.
An optimizer uses gradients and update rules to reduce the training objective.
Adam is a widely used optimizer.
Model architecture
The optimizer trains the model; it is not usually part of inference after training.
Learning Rate
A setting controlling how large each training update is.
The learning rate scales parameter updates and strongly affects stability and convergence.
A learning rate that is too high can make training unstable.
Training speed
A higher learning rate does not simply mean the model learns better or faster.
Hyperparameter
A setting chosen for training or inference rather than learned directly by the model.
Hyperparameters include learning rate, batch size, temperature, architecture depth, and regularization strength.
Temperature is an inference hyperparameter.
Parameter
Model parameters are learned. Hyperparameters are selected or configured.
Batch
A group of examples processed together.
Batching improves computational efficiency during training or inference by processing multiple inputs in parallel.
A training batch may contain 64 text sequences.
Dataset
A batch is one portion of a dataset, not the entire dataset.
Batch Size
The number of examples processed together in one batch.
Batch size affects memory use, training dynamics, latency, and throughput.
A batch size of 32 processes 32 examples before one update.
Context length
Batch size counts examples, while context length counts tokens or input size per example.
Epoch
One complete pass through a training dataset.
Training often uses multiple epochs so the model sees the dataset repeatedly.
Fine-tuning may run for three epochs.
Step
One epoch contains many training steps.
Training Step — Step
One parameter-update cycle during training.
A step usually processes one batch, computes loss and gradients, and applies an optimizer update.
A training run may contain tens of thousands of steps.
Epoch
A step is one update. An epoch is a full pass through the data.
Pretraining
Large-scale initial training that teaches a model broad patterns.
Pretraining usually uses extensive datasets and a general objective before later specialization.
An LLM learns broad language patterns during pretraining.
Fine-tuning
Pretraining creates the general base model. Fine-tuning adapts it later.
Base Model — Foundation Model
A general model used as the starting point for later adaptation.
A base model is pretrained broadly and may later be instruction-tuned, fine-tuned, aligned, or wrapped in a product.
A general language model before chat or tool-use tuning.
Chat model
A base model may be capable but not yet optimized for helpful conversation.
Foundation Model
A broadly trained model that can support many downstream tasks.
Foundation models are trained at scale and adapted through prompting, fine-tuning, tools, or product layers.
A general text-and-image model used across many applications.
Base model
Foundation model emphasizes broad reuse. Base model emphasizes its role as the starting model.
Fine-tuning
Additional training used to adapt an existing model.
Fine-tuning updates model weights using a narrower dataset or objective to improve task, style, domain, or behavior performance.
Fine-tuning a model on customer-support conversations.
Prompting, retrieval
Fine-tuning changes model weights. Prompting and retrieval change the input context instead.
Instruction Tuning
Training a model to follow written instructions more reliably.
Instruction tuning uses examples of instructions and desired responses so the model generalizes across tasks.
Teaching a base model to answer questions, summarize, and format outputs.
Prompt engineering
Instruction tuning changes weights. Prompt engineering changes the prompt.
Preference Tuning
Training a model to prefer some outputs over others.
Preference tuning uses ranked or comparative examples to shape style, helpfulness, safety, or behavior.
Training the model to prefer clearer answers over rambling ones.
Instruction tuning
Instruction tuning teaches task following. Preference tuning shapes which acceptable outputs are preferred.
RLHF — Reinforcement Learning from Human Feedback
A method for shaping model behavior using human preferences.
RLHF typically trains a reward model from human comparisons and then optimizes the model toward higher-scoring outputs.
Human reviewers rank responses, and the model is trained toward preferred behavior.
Fine-tuning, DPO
RLHF does not directly encode every preference as a fixed rule.
DPO — Direct Preference Optimization
A method for training directly from preferred and rejected response pairs.
DPO optimizes model preferences without requiring a separate reinforcement-learning loop in the same form as classic RLHF.
Training on pairs where one answer is marked preferred.
RLHF
DPO and RLHF pursue similar preference goals through different training procedures.
Reward Model
A model that scores outputs according to learned preferences.
Reward models are trained from comparisons or labels and used to estimate which outputs are more desirable.
A reward model may score helpfulness or safety.
Judge model
A reward model shapes training. A judge model is often used during evaluation.
Alignment
The effort to make a model behave according to intended goals and constraints.
Alignment may address helpfulness, reliability, safety, honesty, instruction following, and value-sensitive behavior.
Training a model to refuse dangerous instructions while remaining useful.
Safety
Alignment is broader than safety and remains an imperfect, ongoing process.
Overfitting
Learning training examples too specifically and performing poorly on new data.
An overfit model captures noise or narrow patterns instead of generalizable structure.
A classifier memorizes training phrases but fails on reworded examples.
Memorization
Overfitting is about poor generalization, not merely high training performance.
Underfitting
Failing to learn enough of the underlying pattern.
An underfit model performs poorly on both training data and new data because its capacity, data, or training is insufficient.
A simple model cannot capture the complexity of nuanced language.
Overfitting
Underfitting is not cautious behavior. It is inadequate learning.
Generalization
How well a model performs on new examples it was not trained on directly.
Generalization measures whether learned patterns transfer beyond the training set.
A model correctly handles a new phrasing of a familiar task.
Memorization
Good generalization does not mean the model understands every situation.
Memorization
Retaining specific patterns or examples from training data.
Memorization can range from useful retention of common patterns to reproduction of rare or sensitive sequences.
A model reproduces a distinctive phrase seen during training.
Learning, memory
Model memorization is not the same as a user-facing memory system.
Generation
Producing new model output.
Generation repeatedly selects or predicts outputs from a probability distribution until a stopping condition is reached.
A language model generates one token at a time.
Retrieval
Generated output is produced by the model, while retrieved output is fetched from an external source.
Completion
The generated continuation returned by a language model.
A completion may be free text, structured output, code, a tool call, or part of a chat response.
The assistant message returned after a prompt.
Response
Completion is a model-generation term. Response may include additional system or tool-layer data.
Decoding
The strategy used to choose model output from predicted possibilities.
Decoding methods include greedy selection, beam search, top-k sampling, nucleus sampling, and temperature scaling.
A model samples the next token using temperature and top-p.
Tokenizer decoding
Decoding can also mean converting token IDs back into text; context determines the meaning.
Logits
Raw scores a model produces before turning them into probabilities.
Logits are transformed, often through softmax, into a probability distribution over possible outputs.
Each possible next token receives a logit score.
Probability
Logits are not probabilities until normalized.
Softmax
A function that converts scores into probabilities.
Softmax normalizes logits into a distribution whose values sum to one.
Next-token logits become probabilities before sampling.
Argmax
Softmax creates a distribution. It does not itself decide which token is selected.
Probability Distribution
A set of possible outputs paired with their likelihoods.
During generation, the model assigns relative probability to candidate tokens or outputs.
One token may have 40% probability while another has 15%.
Confidence score
High probability means likely under the model, not necessarily true or correct.
Sampling
Choosing an output from the model’s probability distribution.
Sampling introduces controlled variability by selecting among plausible candidates rather than always taking the single highest-probability option.
Two runs with the same prompt may produce different wording.
Training sample
Sampling during generation is different from sampling examples from a dataset.
Greedy Decoding
Always choosing the highest-probability next output.
Greedy decoding is deterministic under fixed conditions but may produce repetitive or locally optimal text.
At each step, choose the token with the highest probability.
Sampling
The most probable next token at every step does not always produce the best complete answer.
Temperature
A setting that changes how adventurous or conservative output selection is.
Temperature rescales logits before sampling. Lower values concentrate probability on likely outputs; higher values flatten the distribution.
Lower temperature often produces more consistent wording.
Creativity
Temperature does not directly control intelligence, truthfulness, or emotional warmth.
Top-k
Restricting sampling to the k most likely candidates.
Top-k removes all but a fixed number of highest-probability tokens before sampling.
Top-k 50 samples only from the 50 most likely next tokens.
Top-p
Top-k uses a fixed candidate count. Top-p uses a probability-mass threshold.
Top-p — Nucleus Sampling
Restricting sampling to the smallest group of candidates that reaches a chosen total probability.
Top-p dynamically changes the candidate set based on how probability is distributed.
Top-p 0.9 samples from candidates covering 90% of probability mass.
Top-k
Top-p does not mean the top 90% of tokens by count.
Seed
A starting value used by a random-number process.
A fixed seed can help reproduce stochastic output when the rest of the system is also stable.
Using the same image-generation seed to explore prompt variations.
Prompt
The same seed does not guarantee identical output across different models, versions, or infrastructure.
Deterministic
Producing the same output from the same input under the same conditions.
Determinism requires stable code, model version, settings, environment, and random state.
A pure formatting function is deterministic.
Consistent
Low temperature may improve consistency but does not guarantee full determinism.
Nondeterministic
Able to produce different results from the same apparent input.
Variation may come from sampling, concurrency, hardware behavior, model updates, hidden context, or external systems.
A generative model returns slightly different answers on repeated runs.
Random
Nondeterministic does not mean uncaused or completely random.
Reproducibility
The ability to recreate the same result or experiment reliably.
Reproducibility depends on recording model versions, prompts, seeds, data, dependencies, settings, and environment.
An eval stores the exact model and prompt version used.
Determinism
A nondeterministic system can still be reproducibly evaluated through repeated measurements and recorded conditions.
Context Window
The maximum amount of input and generated text a model can consider in one request.
The context window is measured in tokens and includes instructions, conversation, retrieved content, tool data, and output budget.
A 128k context window can process roughly 128,000 tokens in one request.
Memory
A large context window is temporary working space, not permanent memory.
Token Budget
The amount of context space available for input and output.
Token budgets may be allocated across system instructions, conversation history, retrieved content, tool results, and generated output.
A long document may leave less room for the model’s response.
Usage quota
Token budget is request capacity. Quota is a service usage allowance.
Truncation
Cutting off content because it exceeds a limit.
Truncation may remove earlier context, later content, or generated output depending on system policy.
Old conversation turns are dropped when the context window is full.
Summarization
Truncation discards content. Summarization compresses it.
System Prompt — System Message
High-priority instructions that shape model behavior.
The system prompt may define role, policies, tone, tool rules, constraints, and response behavior.
A system message instructs an assistant to use a specific tool before answering.
User prompt
A system prompt influences behavior but does not guarantee perfect compliance.
Prompt Hierarchy
The priority order among different instruction sources.
Prompt hierarchy determines how system, developer, user, tool, and retrieved instructions interact and which instructions win during conflict.
A user request cannot override a higher-priority safety rule.
Prompt order
Later text is not always higher priority. Authority depends on message role and system design.
User Prompt
The instruction or request provided by the user.
A user prompt may contain goals, context, examples, constraints, and desired output format.
“Explain this schema in simple language.”
System prompt
The user prompt is important but may be constrained by higher-priority instructions.
Chat Template
The formatting pattern used to turn messages into model input.
A chat template inserts roles, separators, control tokens, and message boundaries expected by the model.
System, user, and assistant messages are wrapped with special tokens.
Prompt template
A chat template is often model-specific and can strongly affect behavior.
Prompt Template
A reusable prompt structure with placeholders.
Prompt templates standardize instructions while inserting task-specific content, variables, examples, or context.
“Summarize {{document}} in {{length}} words.”
Chat template
A prompt template is authored task content. A chat template is low-level message formatting.
Zero-shot Prompting
Asking a model to perform a task without showing examples.
Zero-shot prompting relies on the model’s existing learned capabilities and the written instruction.
“Classify this message as urgent or not urgent.”
Few-shot prompting
Zero-shot means no examples in the prompt, not that the model had no prior training.
Few-shot Prompting
Showing a model a small number of examples inside the prompt.
Few-shot examples demonstrate desired behavior, format, labels, or reasoning patterns without changing model weights.
Providing three correctly labeled messages before asking for a fourth classification.
Fine-tuning
Few-shot prompting changes context. Fine-tuning changes model weights.
In-Context Learning
Adapting behavior based on examples or instructions provided in the current context.
The model changes its response pattern during inference without updating its underlying weights.
A model follows a new labeling scheme demonstrated in the prompt.
Fine-tuning
In-context learning is temporary and request-scoped.
Instruction Following
How reliably a model follows the instructions it receives.
Instruction-following quality depends on training, prompt clarity, hierarchy, context, capability, and competing constraints.
Returning valid JSON when explicitly required.
Obedience
Models do not follow instructions perfectly or literally in every situation.
Structured Output
Model output required to follow a defined format.
Structured output may use JSON, XML, tables, typed fields, or provider-enforced schemas.
Returning an object with name, category, and confidence fields.
Plain text
Output that looks like JSON is not necessarily valid JSON.
Schema Adherence
How closely output follows a required schema.
Schema adherence checks field presence, types, allowed values, nesting, and formatting.
A tool call includes all required arguments with valid types.
Semantic correctness
Schema-valid output can still contain incorrect or misleading content.
Tool Use — Function Calling
Allowing a model to request actions from external tools.
The model selects a tool and produces structured arguments; the host executes the tool and returns results.
Calling a calendar search tool instead of guessing availability.
Plugin, agent
The model usually proposes the call. The surrounding application executes it.
Tool Call
One structured request to use a tool.
A tool call names the operation and supplies arguments that should match the tool schema.
Search memory with query and result limit fields.
API request
A tool call may become an API request internally, but the concepts are not identical.
Function Calling
A model capability for returning structured requests that map to callable functions.
Function calling lets the model choose from registered operations and emit arguments instead of only natural-language text.
The model returns a structured request to send a message.
Ordinary code function call
Provider “function calling” is usually a structured generation feature, not direct code execution by the model.
Multimodal Model
A model that can process or produce more than one kind of media.
Modalities may include text, images, audio, video, code, or sensor data.
A model that reads an image and answers questions about it.
Omnimodal model
Multimodal does not mean every modality is equally strong.
Modality
A type of input or output media.
Common modalities include text, image, audio, video, and structured data.
Text and image are two separate modalities.
Format
JPEG and PNG are formats within the image modality.
Vision-Language Model — VLM
A model that connects visual information with language.
VLMs can analyze images, answer visual questions, caption scenes, or reason across text and visual inputs.
Reading a chart and explaining its trend.
Image generator
A vision-language model interprets images; an image generator creates them.
Computer Vision
The field of teaching computers to interpret visual information.
Computer vision includes classification, detection, segmentation, tracking, recognition, and visual reasoning.
Detecting objects in a photograph.
Image generation
Understanding images and generating images are different tasks.
Image Model
A model designed to analyze, transform, or generate images.
Image models may perform generation, editing, classification, segmentation, upscaling, or restoration.
A model that creates an image from a text description.
Computer vision model
Image model is a broad category that includes both understanding and generation systems.
Diffusion Model
A generative model that learns to create data by reversing a noise process.
Diffusion models iteratively denoise a random or partially noised representation while following conditioning signals.
Many text-to-image systems use diffusion.
GAN
The model does not retrieve a finished image from storage. It synthesizes one through iterative denoising.
Denoising
Removing noise step by step to produce a coherent output.
In diffusion generation, the model predicts how to move a noisy representation toward a cleaner sample.
An image gradually emerges from noise over several steps.
Image cleanup
Diffusion denoising is the core generation process, not merely post-processing.
Latent Diffusion
Diffusion performed in a compressed internal representation instead of directly on raw pixels.
Latent diffusion reduces compute by generating in latent space and decoding the result back into an image.
A model denoises latent tensors before using a decoder to create pixels.
Pixel-space diffusion
Latent diffusion still produces pixel images; it simply performs much of the work in compressed space.
Guidance Scale — CFG
A setting that controls how strongly generation follows the prompt.
Classifier-free guidance balances prompt conditioning against the model’s unconstrained generation tendency.
Very high guidance may follow the prompt more literally but harm naturalness.
Prompt strength
Higher guidance is not always better.
Sampling Steps
The number of denoising iterations used during diffusion generation.
More steps may improve quality up to a point but increase generation time.
An image may be generated with 30 denoising steps.
Training steps
Sampling steps happen during generation. Training steps happen while learning weights.
Sampler
The algorithm controlling how a diffusion model moves through denoising steps.
Different samplers trade speed, stability, detail, and visual character.
Two samplers may produce different images from the same prompt and seed.
Language-model sampling
The word sampler is used differently in diffusion and language generation.
Text-to-Speech — TTS
Turning written text into spoken audio.
TTS systems model pronunciation, timing, prosody, and voice characteristics to synthesize speech.
Generating narration from a script.
Speech-to-text
TTS creates audio from text. Speech-to-text transcribes audio into text.
Speech-to-Text — STT / ASR
Turning spoken audio into written text.
Automatic speech recognition converts speech signals into transcripts and may include timestamps, speaker labels, or confidence scores.
Transcribing a voice note.
Text-to-speech
Speech recognition accuracy depends heavily on audio quality, accent, noise, and domain vocabulary.
Voice Model
A model used to synthesize or transform a speaking voice.
Voice models may generate speech, clone vocal characteristics, convert one voice to another, or control style and prosody.
Generating speech in a consistent branded voice.
Speech recognition model
A voice model produces or transforms speech; a recognition model transcribes it.
Prosody
The rhythm, stress, pacing, and intonation of speech.
Prosody carries emotional and linguistic information beyond the literal words.
The same sentence can sound tender, sarcastic, or urgent through prosody.
Voice identity
A voice can match the speaker’s timbre while still having wrong prosody.
Latency
How long it takes to receive a result.
Latency may include network delay, queue time, preprocessing, model computation, tool execution, and output streaming.
The first token arrives after 800 milliseconds.
Throughput
Low latency means fast individual responses. High throughput means processing lots of work over time.
Time to First Token — TTFT
The delay before the first generated token appears.
TTFT measures initial responsiveness before the rest of the output is streamed.
A model begins responding in 500 milliseconds but takes several seconds to finish.
Total latency
Fast first-token time does not guarantee fast total completion.
Tokens per Second — TPS
How quickly a model generates tokens after it starts responding.
TPS measures generation throughput for one stream and depends on model size, hardware, batch load, and context length.
A model generates 40 tokens per second.
Requests per second
Token speed and total system throughput are different measurements.
Throughput
How much work a system can complete over time.
Throughput may be measured in requests, tokens, images, or batches per second.
A service handles 100 requests per second.
Latency
A system can have high throughput but poor latency for individual users.
Streaming
Sending output gradually as it becomes available.
Streaming improves perceived responsiveness by delivering tokens, audio, or events before the full result is complete.
Text appears word by word in a chat interface.
Pagination
Streaming delivers one result progressively. Pagination divides a larger result set into pages.
Batching
Grouping multiple requests or examples for joint processing.
Batching improves hardware use and throughput but can increase individual latency.
A server combines several inference requests into one GPU batch.
Queueing
Batching groups work for processing. Queueing only stores work until it can be processed.
Quantization
Reducing the numerical precision used to store or run a model.
Quantization lowers memory and compute requirements by representing weights or activations with fewer bits.
Running a model in 4-bit precision on consumer hardware.
Compression
Quantization can reduce quality, but well-designed methods may preserve much of the model’s performance.
Precision — Numerical Precision
How many bits are used to represent numerical values in computation.
Common formats include FP32, FP16, BF16, INT8, and INT4.
A quantized model may use INT4 weights.
Evaluation precision
Numerical precision is unrelated to the classification metric also called precision.
Distillation — Knowledge Distillation
Training a smaller model to imitate a stronger model.
A student model learns from outputs, probabilities, or representations produced by a teacher model.
A compact model is trained on answers generated by a larger model.
Fine-tuning
Distillation transfers behavior patterns, not a perfect copy of all capabilities.
Teacher Model
A stronger model used to guide another model’s training.
Teacher outputs or internal signals become training targets for a student model.
A large model generates demonstrations for a smaller model.
Judge model
A teacher model supplies learning signals. A judge model scores outputs.
Student Model
A model trained to learn from a teacher model.
Student models aim to preserve useful behavior while reducing size, cost, or latency.
A smaller assistant model trained from a larger teacher.
Fine-tuned model
A student model may have a different architecture or size from its teacher.
Router — Model Router
A component that chooses which model, tool, or workflow should handle a request.
Routers use rules, classifiers, cost targets, capability signals, or learned policies to dispatch work.
Simple tasks go to a fast model; difficult tasks go to a stronger model.
Orchestrator
A router selects a path. An orchestrator coordinates a sequence of work.
Model Cascade
Using multiple models in sequence or by escalation.
A cascade may start with a cheaper model and move to a stronger one only when confidence or quality is insufficient.
A small model attempts classification before escalating uncertain cases.
Ensemble
A cascade routes sequentially. An ensemble combines multiple model outputs.
Ensemble
Combining outputs from multiple models.
Ensembles aggregate votes, scores, rankings, or generated candidates to improve robustness.
Three classifiers vote on the final label.
Model cascade
Ensembles combine parallel opinions. Cascades choose or escalate between models.
Fallback
A backup behavior used when the preferred path fails.
Fallbacks may use another model, cached output, simpler logic, or a human review path.
Use a smaller local model if the cloud provider is unavailable.
Retry
Retry repeats the same path. Fallback switches to a different path.
Graceful Degradation
Continuing with reduced capability instead of failing completely.
A system degrades gracefully by preserving core function when optional services or stronger models are unavailable.
Return text without voice synthesis when the voice service fails.
Fallback
Fallback is the alternate mechanism. Graceful degradation is the overall user-visible behavior.
Classifier
A model or rule system that assigns an input to a category.
Classifiers may output one label, multiple labels, probabilities, or confidence scores.
Classifying a message as urgent or routine.
Generator
A classifier chooses among labels. A generator creates new content.
Classification
Assigning input to one or more categories.
Classification may be binary, multiclass, multilabel, hierarchical, or probabilistic.
Assigning a memory node to identity, build, or relationship domains.
Clustering
Classification uses known label categories. Clustering discovers groups without predefined labels.
Confidence Score
A number indicating how strongly a system favors an answer or label.
Confidence scores may come from probabilities, margins, heuristics, calibration models, or model-generated estimates.
The classifier assigns 0.82 confidence to build.
Correctness
High confidence does not guarantee the answer is correct.
Calibration
How well confidence scores match real-world accuracy.
A calibrated model that reports 80% confidence should be correct about 80% of the time across similar cases.
Comparing predicted confidence with actual outcomes.
Accuracy
A highly accurate model can still be poorly calibrated.
Threshold
A cutoff used to decide whether a score counts as a match or action.
Thresholds convert continuous scores into decisions and control trade-offs between false positives and false negatives.
Only classify as urgent when confidence is above 0.8.
Top-p
Threshold is a decision cutoff, not a sampling parameter.
Hallucination
Model output that is unsupported, fabricated, or incorrectly presented as true.
Hallucinations can involve invented facts, citations, fields, events, tool results, or reasoning claims.
The model invents an API field that does not exist.
Error
Not every wrong answer is called a hallucination; the term is most useful when the model generates unsupported content.
Grounding
Anchoring model output in trusted evidence or data.
Grounding may use retrieved documents, tool results, databases, citations, or verified context.
The model reads the actual tool schema before explaining required fields.
Training
Grounding supplies evidence at runtime. It does not permanently retrain the model.
Reasoning Model
A model optimized to spend more computation on multi-step problem solving.
Reasoning models may use longer internal deliberation, verification, search, or tool use before producing an answer.
Solving a complex coding or planning problem with multiple constraints.
Agent
A reasoning model is still a model. An agent is the larger system that can plan and act with tools.
Deliberation
Additional model computation used before finalizing an answer.
Deliberation may involve internal candidate generation, checking, decomposition, or iterative refinement.
A model spends more compute on a difficult proof than on a simple lookup.
Visible explanation
A model’s user-visible explanation is not necessarily a literal transcript of its internal computation.
Chain-of-Thought Prompting — CoT
A prompting technique that encourages a task to be broken into intermediate steps.
Chain-of-thought prompting uses examples or instructions that elicit multi-step problem solving before the final answer.
Ask the model to solve a problem step by step, then provide the conclusion.
Private model reasoning
A written chain of thought is generated text and should not be treated as a guaranteed record of internal computation.
Decomposition
Breaking a complex task into smaller parts.
Decomposition improves planning, delegation, testing, and reasoning by isolating subtasks and dependencies.
Separate schema analysis, implementation, tests, and review.
Delegation
Decomposition divides the task. Delegation assigns the pieces.
Self-Consistency
Generating multiple solution paths and choosing the result that agrees most often.
Self-consistency samples several candidate reasonings or answers and aggregates them.
Run the same math problem several times and vote on the final answer.
Consistency
Agreement among outputs can improve reliability but does not guarantee truth.
Reranking
Reordering candidate results using a stronger scoring method.
A reranker evaluates retrieved items or generated candidates after the initial selection stage.
Retrieve 20 passages, then rerank the best five.
Routing
Routing chooses a path. Reranking orders candidates.
Evaluation — Eval
A structured way to measure model or system performance.
Evals may use exact answers, rubrics, human ratings, judge models, statistics, or task-specific metrics.
Measure whether the model selects the correct tool across 200 prompts.
Test
Tests often expect exact behavior. Evals may measure graded quality or probabilistic performance.
Benchmark
A fixed evaluation set used to compare models or versions.
Benchmarks standardize tasks, data, scoring, and reporting for repeatable comparison.
Compare two models on the same coding benchmark.
Eval suite
A benchmark score may not predict performance in your real product.
Rubric
The criteria used to judge output quality.
A rubric defines dimensions, scoring scales, failure conditions, and examples for evaluators.
Score correctness, relevance, completeness, and tone from 1 to 5.
Prompt
A vague rubric produces unreliable evaluation.
Judge Model — LLM-as-a-Judge
A model used to score or compare another model’s output.
Judge models apply a rubric to outputs and may produce scores, rankings, or explanations.
A judge model ranks two answers for factuality.
Reward model
Judge models can be biased, inconsistent, or vulnerable to style effects.
Regression Eval
An evaluation that checks whether previously good behavior got worse.
Regression evals preserve critical examples and metrics across model, prompt, tool, or product changes.
Confirm the new prompt still preserves persona and schema adherence.
Benchmark
Regression evals protect known behavior; benchmarks compare broader capability.
Drift
Behavior gradually moving away from the intended standard.
Drift can come from model updates, prompt changes, data changes, retrieval shifts, user adaptation, or evolving system context.
A persona becomes flatter across several releases.
Regression
A regression is a specific backward step. Drift may be gradual and cumulative.
Emergent Behavior
Behavior that appears from the system without being explicitly programmed as a single rule.
Emergent behavior arises from interactions among training data, architecture, scale, prompting, tools, and environment.
A model develops a useful strategy not directly described in the training objective.
Unexpected behavior
Emergent does not mean supernatural or conscious.