Running AI workloads on Kubernetes means learning a new set of concepts: how GPUs are requested and shared, how groups of pods get scheduled together, and how models are served and scaled. This page defines the concepts that matter most, explains why each one matters for AI workloads, and explains what that means for the team running them.
Two groups are learning this vocabulary from opposite directions at the same time. Platform engineers know Kubernetes well but are new to GPUs and model serving. Machine Learning (ML) engineers know models and GPUs but are new to Kubernetes. Both end up working on the same clusters, and a lot of friction comes from each side assuming the other already knows the basics. This reference is intended to help them reconcile these concepts.
How to use this page
Each concept has three parts: what it is, why it matters for AI workloads, and what it means for your team.
A request is the amount of CPU, memory, or other resource Kubernetes reserves for a container when it schedules it. A limit is the most that container is allowed to use.
Why they matter for AI workloads: GPUs are requested in whole units and can't be overcommitted the way CPU can. Unless the GPU is shared (see GPU sharing below), a pod that asks for one gets the whole device, whether it uses 5% of it or 95%. Think of it as reserving a table for twelve when four people show up: the restaurant still holds the table.
For your team: The biggest source of wasted GPU spend is low utilization: a pod holds a whole GPU while using a fraction of it. Other requests still matter in a less obvious way. A pod that over-requests CPU or memory on a GPU node can leave that node’s other GPUs stranded, because no other pod fits alongside it.
A taint marks a node so that only pods with a matching toleration can run on it. A node selector is the simplest way to send a pod to nodes with a specific label, such as a particular GPU model. Node affinity does the same job with more flexible rules, like preferences instead of hard requirements.
Why it matters for AI workloads: Taints keep ordinary workloads off expensive GPU nodes. Node selectors or affinity send each GPU workload to the right kind of GPU when a cluster has more than one. Requesting a GPU already limits a pod to nodes that have one, so in a cluster with a single GPU type, a taint and toleration may be all you need.
For your team: Which of these you use depends on how your cluster is laid out. Getting it right keeps GPU capacity free for the workloads that need it.
A namespace is a separate area within a cluster, usually for a team or environment. A ResourceQuota caps how much of a resource everything in a namespace can request in total.
Why it matters for AI workloads: Quotas can cap GPUs as well as CPU and memory. That's how several teams share a limited pool of GPUs without one team's experiments starving everyone else.
For your team: Quotas turn "who gets the GPUs" from a Slack argument into a policy.
A Job runs one or more pods until a task finishes, then stops. It's the standard way to run work that has an end, like a training run or a batch of data processing.
Why it matters for AI workloads: Training and fine-tuning are usually run as Jobs. Distributed training often needs several coordinated pods, which is where gang scheduling (below) comes in.
For your team: Jobs make training runs repeatable and trackable, instead of something that only runs from one engineer's laptop or a hand-configured machine.
A persistent volume is storage that outlives any single pod. Pods claim it through a PersistentVolumeClaim and can pick up where they left off after a restart.
Why it matters for AI workloads: Model weights and training datasets are large. Where they're stored, and how fast pods can read them, often determines how long it takes a model server to start or a training job to resume from a checkpoint.
For your team: Slow storage shows up as GPUs sitting idle while data loads, which means paying for expensive hardware to wait.
An accelerator is specialized hardware that speeds up the calculations behind training and inference. GPUs are the most common kind. A GPU node is a Kubernetes node with one or more accelerators attached.
Why it matters for AI workloads: Most AI workloads either require accelerators or run far more slowly without them. GPU nodes are usually contained in a separate node pool, because they cost many times more than general-purpose nodes.
For your team: GPU nodes are often the largest single line on an AI infrastructure bill, so much of the content on this page is about using them well.
A device plugin is a small program that tells Kubernetes what special hardware a node has, so pods can request it by name.
Why it matters for AI workloads: This is how GPUs are exposed in Kubernetes. It works well for simple cases, but pods can only ask for a number of devices. Getting a specific GPU type or amount of memory takes workarounds, such as node labels or a separate resource name for each type.
For your team: If your clusters run GPU workloads today, they almost certainly depend on a device plugin, and it needs to be kept in step with drivers and Kubernetes upgrades.
Dynamic Resource Allocation is the newer Kubernetes system for requesting specialized hardware. A pod makes a ResourceClaim describing what it needs, and a DeviceClass defines categories of devices that claims can match against.
Why it matters for AI workloads: DRA lets workloads ask for devices by their attributes, such as a GPU model or a minimum amount of memory. It also builds sharing into the API itself, since several pods can reference the same ResourceClaim.
For your team: DRA gives platform teams more precise control over expensive hardware, which means fewer oversized allocations. It's also a change that needs planning, since drivers and workloads have to move to it.
GPU sharing lets more than one workload use a single physical GPU. The two most common approaches are time-slicing, where workloads take turns on the whole GPU, and hardware partitioning, such as NVIDIA's Multi-Instance GPU (MIG), which splits one GPU into several isolated pieces.
Why it matters for AI workloads: Many inference and development workloads use a small fraction of a GPU. Sharing puts that idle capacity to work. Time-slicing is flexible but gives each workload no guaranteed memory or performance. Partitioning gives isolation, but only on supported hardware and in fixed sizes.
For your team: Sharing can cut the number of GPUs you need, at the cost of more careful capacity planning.
GPU nodes need drivers, a container runtime configured for GPUs, a device plugin or DRA driver, and monitoring tools. An operator is a Kubernetes add-on that can install and manage all of these together.
Why it matters for AI workloads: Driver and runtime versions have to match the hardware, the Kubernetes version, and the frameworks your models use. A mismatch usually shows up as pods that start but can't see the GPU.
For your team: GPU software is one more layer to keep current during every upgrade, and it's a common reason GPU clusters fall behind on versions.
Gang scheduling places a group of pods all at once or not at all. Because native Kubernetes lacks an in-tree gang scheduling API, ecosystem tools introduce custom resources such as Kueue's Workload or Volcano's PodGroup to describe which pods belong together.
Why it matters for AI workloads: Kubernetes normally schedules each pod on its own. That works for web services, but distributed training needs every worker running before it can make progress. Without gang scheduling, half a training job can start, hold its GPUs, and sit waiting for the other half, which may never fit.
For your team: Partially scheduled jobs are a quiet way to burn GPU hours. Gang scheduling prevents those deadlocks and makes capacity planning for training more predictable.
Topology-aware scheduling places pods based on the physical layout of the hardware: which GPUs share a fast interconnect, which network cards sit closest to them, and which nodes share a rack.
Why it matters for AI workloads: Large training and inference jobs move a lot of data between GPUs. Two GPUs on the same fast link can be much quicker together than two on opposite sides of a data center.
For your team: Good placement can make the same hardware finish jobs sooner, which matters most for large models on large clusters. It can also reduce network transit costs by keeping data within the same availability zone.
Job queueing holds workloads in line until there's enough capacity to run them, then admits them according to team quotas and priorities. Kueue, a Kubernetes community project, is a widely used option.
Why it matters for AI workloads: GPU capacity is usually scarcer than demand. A queue lets training jobs wait their turn in an orderly way, instead of failing or grabbing partial resources.
For your team: Queueing makes it possible to share an expensive GPU pool fairly across teams, and to see how long work is actually waiting for hardware.
A model server loads a trained model into GPU memory and answers requests against it, usually through an HTTP API. It also handles batching, which groups requests together so the GPU does more work per pass.
Why it matters for AI workloads: The model server is the part of an inference setup that actually uses the GPU. It runs in a pod like any other application, but it starts slowly because it has to load a large model, and its performance depends heavily on how it batches requests.
For your team: Model server settings have as much effect on cost per request as the hardware does, so they deserve the same attention as any other production service.
An inference gateway routes requests to model servers based on what's happening inside them, such as how busy each one is and what it already has in memory. In Kubernetes, the Gateway API Inference Extension adds this to existing gateways, with an InferencePool that groups the model servers behind it.
Why it matters for AI workloads: Ordinary load balancing spreads requests evenly, but inference requests vary widely in cost. Routing each request to the model server best placed to handle it cuts latency and keeps GPUs from sitting overloaded next to idle ones.
For your team: Better routing means serving the same traffic with fewer GPUs. It also builds on Gateway API, which many teams are adopting anyway as they move off Ingress NGINX.
Autoscaling adds or removes model server replicas as demand changes. For inference, it usually scales on signals like request queue length or response latency, since CPU usage says little about how busy a GPU-backed server is. Some setups scale all the way to zero replicas when there's no traffic.
Why it matters for AI workloads: GPU-backed replicas are expensive to leave running and slow to start. Scaling on the right signal keeps latency steady without paying for idle GPUs.
For your team: Scale-to-zero can save a lot on models with bursty or occasional traffic, at the cost of a slow first response while a replica starts and loads the model.
LeaderWorkerSet is a Kubernetes API for running a group of pods as one unit, with one leader pod and several workers. It's commonly used to serve models too large to fit on a single node.
Why it matters for AI workloads: The largest models have to be split across several machines. LeaderWorkerSet keeps those pieces together, so they're deployed, scaled, and restarted as a group.
For your team: If you plan to serve very large models yourselves, this is part of what that commitment involves operationally.
GPU utilization measures how much of a GPU's capacity is actually doing work. It's collected by metrics exporters on each GPU node and reported alongside the rest of your cluster monitoring.
Why it matters for AI workloads: A GPU can be fully allocated to a pod and still be mostly idle. Allocation tells you who has the GPU. Utilization tells you whether it's earning its cost.
For your team: If you track one number for AI infrastructure efficiency, track this one. Low utilization on allocated GPUs is usually the biggest savings opportunity in the cluster.
In-place pod resizing allow changes to the CPU and memory assigned to a running pod without restarting it.
Why it matters for AI workloads: Restarting a model server means reloading the model, which can take minutes. Resizing in place lets you correct CPU settings, and in many cases memory settings, without that interruption. It doesn't apply to GPUs, which are still assigned when a pod starts.
For your team: Rightsizing becomes something you can do continuously on long-running services, where before it often waited for a maintenance window. Open source tools like Goldilocks can recommend what those sizes should be.
Node autoscaling adds nodes to a cluster when pods can't be scheduled and removes them when they sit underused. The Kubernetes Cluster Autoscaler and Karpenter are the most common open source options.
Why it matters for AI workloads: GPU nodes are expensive to keep idle and can take several minutes to start. Autoscaling policies for GPU node pools have to balance paying for spare capacity against making jobs wait.
For your team: Well-tuned node autoscaling is where savings from every other concept on this page turn into a smaller bill, since a GPU you free up only saves money once its node is removed.
Training builds or adjusts a model by running it over data, often for hours or days across many GPUs. Inference uses a finished model to answer requests, usually around the clock with a steady need for fast responses.
They need different things from a cluster. Training runs as Jobs that need gang scheduling and queueing. Inference runs as long-lived services that need autoscaling and smart routing. Many teams plan for one and are surprised by the other.
An allocated GPU is assigned to a pod. A utilized GPU is doing work. Cluster dashboards often show the first, and you pay for the GPU either way. The second tells you whether the spend is paying off.
AI infrastructure is the hardware and platform that runs AI workloads, including GPU nodes, storage, and Kubernetes itself. AIOps is the use of AI to help operate infrastructure, such as spotting anomalies in logs and metrics. One runs AI; the other is run with its help. For more on these distinctions, see 9 AI Infrastructure Terms: Must-Know Definitions.
Most of the concepts on this page come back to the same question: are your GPUs doing useful work, or holding a table for guests who aren't coming? Platform engineers and ML engineers each see half the answer. Sharing a vocabulary is the first step toward seeing all of it.
If you're setting up or scaling AI workloads on Kubernetes, Fairwinds builds AI-ready infrastructure and maintains it through Managed Kubernetes-as-a-Service, covering GPU node pools, upgrades, and ongoing optimization.
