Power Fluctuations During AI Training: Why Data Centers Need Load Smoothing
When designing a data center, it is critical to understand both the total electrical power required by the compute infrastructure and the rate at which that power draw will fluctuate. Modern workload profiles make this challenge particularly acute: thousands of accelerators can simultaneously shift from intensive compute cycles to lower-power states, and vice versa. This results in cyclical power swings, while the initiation and termination of tasks can cause sudden, sharp spikes in power demand. These dynamic power profiles directly impact the selection of electrical infrastructure, the facility's usable power capacity, overall capital and operational costs, and future scalability.
Why AI Training Alters Power Consumption Patterns
In a multi-tenant data center, numerous IT workloads operate independently. Their utilization fluctuates throughout the day, with jobs starting and completing at staggered intervals. As the power draw of some servers increases, others may scale down their activity. These individual power spikes are naturally smoothed out, meaning the aggregate load typically fluctuates much more gradually than the power draw of any single tenant. [1]
However, during distributed Transformer training, GPUs operate within a unified cluster. They perform compute operations, calculate gradients (the values that determine how model parameters are updated), and then synchronize the results across the network. During these communication phases, compute utilization often drops. Because thousands of GPUs transition synchronously between compute and communication phases, these individual fluctuations compound into massive, system-wide power transients for the entire cluster. [2]
Figure 1. Schematic power consumption profile during AI training.
Upon initialization, power draw spikes rapidly. During the training phase, recurring power oscillations occur, corresponding to shifts in GPU compute activity. Upon job completion, the accelerators simultaneously drop back to a low-power idle state. The electrical infrastructure must be designed to accommodate not only these state transitions but also brief, transient power excursions above the nominal baseline. [1]
The larger the proportion of the facility's capacity consumed by a single workload, the more its specific duty cycle dictates the aggregate power profile. This is highly relevant in cloud environments, where large enterprise customers may concurrently launch massive batches of rendering, inference, training, or hyperparameter tuning jobs. The simultaneous initiation of these jobs creates a synchronized power surge, even if the workloads themselves are entirely independent. Distributed training further compounds this by introducing recurring, synchronized state transitions within the workload itself. [3]
Aggressive Power Management Amplifies Power Transients
Modern GPUs dynamically exploit available thermal headroom to boost performance. When a workload is introduced, compute unit utilization and clock frequencies ramp up almost instantaneously. Power limits (PL) and thermal throttling operate independently; clock speeds may throttle upon hitting either the power envelope or the thermal threshold. Consequently, the most severe electrical power spike during the transition to compute can occur well before the device experiences significant thermal buildup. [4]
Data center GPUs are known to exhibit power spikes lasting around 50 ms, even while maintaining their average power draw within the Thermal Design Power (TDP) over a one-second rolling window. [2] Whether these transients propagate beyond the server rack level depends entirely on the design of the power delivery network (PDN).
The disparity between average and peak power draw is also evident at the individual GPU level. For instance, during testing of the MSI GeForce RTX 5090 SUPRIM running Cyberpunk 2077 at 2160p (Overdrive mode with DLAA), the average power draw was 605.8 W, while the maximum peak reached 684.6 W. These measurements captured transient spikes lasting longer than 5 ms. While this represents the power profile of a specific consumer board under a gaming workload, server clusters require empirical measurements under their native AI workloads. In our own internal testing, the RTX 5090 exhibited instantaneous power spikes up to 700 W, despite being configured with a maximum TDP limit of 575 W. [5]
In scenarios involving rapid, intensive job scheduling, these power spikes can occur with high frequency. This behavior is particularly pronounced in next-generation GPU architectures, such as NVIDIA Blackwell and Vera Rubin.
How This Impacts Data Center Engineering Design
Power load fluctuations impact the entire power delivery chain: server or rack power supply units (PSUs), switchgear, uninterruptible power supplies (UPS), transformers, and the utility grid connection. While equipment nameplate ratings define continuous operational capacity, the ability to withstand short-term transient spikes requires separate, rigorous validation. [6]
This dynamic behavior is particularly critical during backup diesel generator operation. Following a sudden load step, the generator engine must rapidly ramp up its mechanical output. Until equilibrium is restored, frequency droop and voltage sags are likely to occur. Within their operational range, server PSUs act as constant-power loads, drawing higher input current to maintain output power when voltage drops. Consequently, a generator's load acceptance capability depends on both the magnitude of the load step and its rate of application. [7]
Recurring power fluctuations introduce an additional challenge: their frequencies can excite resonant oscillatory modes within the facility's electrical system. For large-scale sites, the permissible rate and magnitude of load changes must be explicitly negotiated with the utility provider. Simply increasing the nominal utility connection capacity does not inherently resolve this issue. [1]
If power spikes propagate to the external power infrastructure, the facility must provision additional capacity in its electrical equipment to accommodate them. This headroom must be fully paid for, despite being utilized only transiently. Under constrained utility feeds, this reserved capacity consumes a portion of the overall power budget that could otherwise be allocated to additional compute racks. The required capacity margin is dictated by the actual workload power profile, equipment characteristics, and site reliability requirements. Implementing load smoothing enables the reduction of these transient peak demands, thereby optimizing the sizing and cost of the external electrical infrastructure. [1]
Software Level: Filling Power Dips with Auxiliary Workloads
Software-based load smoothing can maintain baseline power draw when the primary workload temporarily scales down its activity. The system monitors GPU utilization and launches auxiliary computations—such as asynchronous inference (batching)—whenever spare compute cycles become available. When the primary training workload demands the accelerator again, the auxiliary load is preempted and terminated. This effectively reduces the depth of power dips, though the upper power peak remains unchanged. This approach is currently implemented in systems like Firefly. [2]
The trade-off of this method lies in the overhead of the additional work. This artificial load consumes energy without advancing the primary objective, while also occupying memory and compute resources. Even useful background tasks require state saving and restoration (context switching), which complicates rapid transitions. High-frequency telemetry (measuring GPU power and utilization) is essential; a delayed reaction from the auxiliary task could inadvertently stall the primary training process. In environments where clients have direct GPU access (e.g., bare-metal or passthrough), implementing such a mechanism requires tight integration with the client's software stack. [2]. [8]
For independent client workloads, the practical solution is simpler: job scheduling can be staggered in time to mitigate the aggregate startup power surge. However, this technique does not eliminate the internal power oscillations of a single, large synchronous workload. Therefore, distributed training requires active power management during execution.
Hardware/GPU Level: Managing Power Limits and Slew Rates
On supported Blackwell accelerators, this behavior is governed by a "power smoothing" profile. The GPU controller is configured with constraints on the power slew rate (both ramp-up and ramp-down), a power floor (minimum baseline consumption), and a hold-off delay before a final power drop is permitted. These parameters are configured via GPU management utilities, including nvidia-smi and programmatic APIs. [4]. [9]
Upon initialization, the power cap (the maximum allowable power limit) is gradually ramped up. The cluster reaches its operational state at a controlled rate, ensuring it does not overshoot the expected Thermal Design Power (TDP). While setting the power limit defines the absolute maximum consumption, smoothing the transition requires the controller to also strictly enforce the ramp-up slew rate. [1]
When compute activity drops, a power burn mechanism is engaged (artificially maintaining power draw via hardware-level operations). The accelerator temporarily continues to draw power to prevent a sudden load rejection (a sharp drop in power draw). If the primary workload resumes, this mode is instantly disabled. If the idle period extends or the task completes, power draw is gracefully ramped down. This hold-off delay allows the system to ride out brief pauses without fully dropping the load, thereby avoiding a subsequent, sharp power spike upon resumption. [1]
In this architecture, the profile is defined in software, but the rapid, microsecond-level response is executed directly by the GPU's onboard controller and hardware mechanisms. No auxiliary matrix computation processes are required to achieve this.
However, an energy trade-off remains: maintaining a high power floor increases overall energy consumption, while overly aggressive power capping can throttle the performance of the primary workload. [1]
Infrastructure Level: Localized Energy Storage Near the Load
Energy storage systems can be deployed to mitigate short, recurring power fluctuations. During periods of reduced power draw, the system charges; when the load increases, it discharges energy directly to the compute equipment. This allows the GPU power draw to fluctuate dynamically with the workload, while the external power delivery system experiences a significantly smoothed aggregate profile.
In the rack-level power supplies of the GB300 NVL72, this function is handled by electrolytic capacitors paired with charge controllers. They compensate for rapid power transients between different operational phases. Energy is effectively redistributed over time. While conversion and storage losses still occur, this approach eliminates the need to artificially load the GPU with dummy workloads to fill every power dip. [1]
When selecting an energy storage solution, two critical parameters must be evaluated: the power rating (the rate at which it can absorb or deliver energy) and the energy capacity (which dictates the duration of the compensation). A small, high-speed buffer can handle short transient pulses, but mitigating prolonged power deviations requires a significantly larger energy reserve. Consequently, energy storage is often combined with workload scheduling (managing job initiation and termination). This hybrid approach allows different types of transient events to be offloaded to the most appropriate mitigation mechanism. [1]
The physical placement of the energy buffer determines which segment of the power delivery network benefits from the smoothed load profile. When deployed at the rack level, the upstream external infrastructure sees a pre-conditioned, smoothed power profile, while the internal DC buses and compute nodes continue to handle the rapid, localized transient transitions. The dynamic stability requirements for these internal components remain unchanged. [1]
How Coordinated Power Management Alters the Load Profile
Figure 2. The gray line represents the unsmoothed baseline power profile, while the blue line shows the smoothed result. On the left, the power cap limits the startup ramp-up; in the middle, the energy storage compensates for rapid power transients; and on the right, the power burn mechanism ensures a gradual ramp-down.
Validation of Energy Storage in Practice
The practical efficacy of energy storage was validated during Megatron LLM training on a single GB200 rack. Upgrading from legacy power supplies to units equipped with localized energy storage reduced the peak power draw from the utility grid by 30%. Internal power fluctuations at the PSU output, however, persisted. This result demonstrates the reduction of the input power spike within a controlled test environment; the ultimate financial impact is determined by how this smoothed profile alters the facility's overall infrastructure sizing requirements. [1]
Conclusion
For modern data centers, the application software duty cycle has become a foundational design parameter for facility engineering. The traditional reliance on the statistical averaging of independent workloads is giving way to the necessity of actively managing massive, synchronized power consumers. The economic rationale for power smoothing is to handle transient power shifts efficiently, allowing the external infrastructure to be sized based on the actual, smoothed power profile that reaches the upstream grid.
To achieve this, operators must first profile the power load at the GPU, rack, and facility intake levels. Next, they must define acceptable transient limits for both primary and backup power systems, selecting a hybrid approach of active power management and localized energy storage. This configuration is then validated across job initialization, active training, and job termination phases.
The results are evaluated based on residual power spikes, power slew rates, time-to-solution for the primary workload, and overall energy-per-job. Based on these metrics, the required capacity margin is calculated: capping peak power demands must deliver tangible technical and economic benefits without compromising the facility's required reliability standards.
References
-
How New GB300 NVL72 Features Provide Steady Power for AI (NVIDIA Technical Blog).
-
Power Stabilization for AI Training Datacenters (Microsoft, OpenAI, NVIDIA).
-
NVIDIA GB200 NVL Multi-Node Tuning Guide (Power and Thermals).
-
NVIDIA. NVIDIA SMI Documentation (Power Limits and Smoothing).