Vertical power takes centre stage on AI chips

As AI accelerators from AMD, Google, and Nvidia push beyond 1 kW power consumption, Empower Semiconductor is overhauling how power reaches these high-performance chips.

By Rebecca Pool, Technical writer, PEW Magazine

In June this year, a handful of integrated voltage regulator (IVR) manufacturers teamed up with US-based Marvell Technology - a frontrunner in custom silicon and SoC development - to optimize power delivery to AI and cloud infrastructure. Empower Semiconductor, US, was amongst these firms, and will now work on integrated power technologies for Marvell’s silicon platforms.

Empower’s IVR will be integrated underneath advanced processors to demonstrate how a vertical power delivery architecture can reduce power transmission losses, improve efficiency and provide a lightning-fast power delivery response – all with a view to satisfying the power-hungry XPUs of tomorrow. As CEO, Tim Phillips, points out: “With integrated voltage regulation, we’re delivering power where it’s needed – right at the point of load – with exceptional density, precision and efficiency.”

The collaboration is timely. Datacentre AI accelerators from the likes of Nvidia and AMD currently comprise more than 100 billion transistors, with these some of chips now guzzling more than 1kW of power. Not surprisingly, today’s lateral power delivery architectures are struggling to meet these remarkable, and rising, power demands.

To deliver power efficiently, these traditional systems operate slowly, creating a large footprint of bulky power stages, capacitors and magnetics across the printed circuit board. Getting power across this ‘last inch’ results in substantial lateral transmission losses that will only get worse as chips consume more power.

What’s more, in a lateral power delivery architecture, the further away the power components are from the processor, the noisier the voltage gets. To prevent noise from interfering with the power delivery, it’s standard practice to use higher voltages – but that also wastes power. And this set-up doesn’t get any easier with the massive voltage spikes and dips that accompany high density AI workloads.

This is where the latest IVRs can provide relief for AI chips. “IVRs are very small and have a great current density with great performance, so you can actually get the regulator itself very close to the AI chip,” explains Phillips. “When you do this, you eliminate the entire transmission loss in the system – in an accelerator card right now, this can be 15 to 20% [of the power].”

Straight to the silicon: Empower Semiconductor has released an integrated voltage regulator that delivers power directly to the AI chip via a vertical architeuctre. [Empower]

The IVRs are designed to counteract voltage dips and peaks, delivering a cleaner, more stable output voltage. This also means the devices can reduce the relatively high operating voltages.

“In an AI chip, power spikes up and down constantly from almost no load to full load, and keeping up with the speed of change has been a massive challenge,” points out Phillips.

According to the Empower CEO, the IVR chips run at very high frequency, imparting a fast transient response and high bandwidth to adjust the supply voltage of the load rapidly. “We eliminate a lot of these spikes in voltage, which helps with the throughput of the chip,” he says. “So [the IVR] not only saves electricity, but allows the AI chip to perform better.”

Early days
Empower launched around ten years ago to build extremely dense power supplies, releasing several high-density power management IC, designed to supply up to 30 W of power. However, according to Phillips, he and colleagues also realised that in high performance chips, ever-decreasing technology nodes were going to lead to higher current densities in processors, which would really strain power delivery. So development of the integrated voltage regulator semiconductor technology began. “We knew that at some point [this process] was going to break, so we said, ‘hey look, we need a clean sheet of paper’. How do I rethink the magnetics, the power and the architecture?” he recalls.

Come 2020, AI was in full swing, which Phillips reckons ‘changed everything’. “We knew AI was the perfect application for high-speed [integrated] voltage regulation technologies – it was like we’d found our Holy Grail,” he says. “So we pivoted the company and worked out how to scale [our developments] into a multi-kilowatt, scalable technology.”

Empower initially won $45 million in venture capital funds in 2021 to further develop its IVRs. Then in 2023, and with an extra $30 million finance in tow, volume production commenced. Today, the company has around 116 patents awarded or pending, and is shipping its vertical power platform, ‘Crescendo’, and associated technologies, to hyper-scalers.

Designed to handle the higher data rates and processing speeds of multi-kilowatt AI chips, Empower’s IVRs lie at the heart of Crescendo. To develop the vertical architecture - designed to meet the demands of 3000A processors - Empower started with its ‘FinFast’ power design. This technology repurposes FinFETs to operate at high speeds whilst also handling high currents and high power. The FinFast technology was combined with on-chip control logic to deliver IVRs that operate at very high frequency with low noise, whilst having very fast response times, said to be 1000 times greater than conventional converter technologies.

Left: Lateral power delivery - a large footprint of bulky power stages, capacitors and magnetics deliver powers the printed circuit board. Right: Vertical power delivery – thin integrated voltage regulator packages are placed beneath the advanced processor to deliver power across a shorter distance. [Empower]

In parallel, Empower worked to integrate high-frequency magnetics into this vertical architecture whilst also replacing bulky, ceramic capacitors with wide-bandwidth silicon versions. (See how the Tyndall National Institute developed integrated magnetics.) Factor in the custom, thermally-resistant packaging designed for high frequency and high power operation, and you have Crescendo, which as Phillips points out is between 1 mm and 2 mm in height, an order of magnitude shorter than traditional DC to DC converters. Critically, the package is thin enough to fit underneath the processor, on the bottom-side of an accelerator card, and deliver high efficiency, vertically-coupled power to the AI chip.

“The hardest thing was getting a piece of silicon to switch at very high speeds efficiently without generating a tonne of noise,” highlights Phillips. “You have to get the magnetics to match the silicon – those speeds have to align. And these have to be combined in a small, thermally-resistant package.”

“Part of the challenge has been to balance all of these needs,” he adds. “We did, and we can now regulate voltages and deliver currents in nanoseconds versus traditional power supplies that do this in several microseconds.”

Technology adoption
Phillips is certain vertical power delivery spells good news for hyper-scalers, rapidly scaling data infrastructure to accommodate the already massive workloads. As anticipated in the early 2020s, he asserts that high-end processors from Marvell and Nvidia have indeed ‘broken’ traditional, lateral power delivery architectures. “[Systems based on discrete subsystems] no longer work well enough for these advanced processors... This passive way to provide charge is just too slow and really hurts AI performance,” he says. Phillips also highlights how more and more industry players are adopting the technology. “A year and a half ago, adoption was zero, but today it’s about 10% in some [applications] and I think this will [continue] to move quickly, certainly over the next five year period,” he says.

“AI is really the first application that’s forcing the world to adopt vertical power,” he adds. “Without this, I think it would be difficult to hit the performance targets that hyper-scalers are looking for.”

So what now for Empower and vertical power delivery? Development will continue and the company’s roadmap plots a course that will provide up to 5000 A processors an even faster transient response for higher density workloads.

As Phillips puts it: “Sure, we’ve brought the technology a long way to get to this point... but that’s not actually good enough for [what hyper-scalers will face] five years from now, right?”

Phillips also points out how Marvell is just one of many firms that his company is collaborating on hyper-scaler, and HPC, challenges. “A power supply company can’t just sit back on its own now and develop technology,” he asserts. “I’d say that this challenge is so interlocked with the processor technology, the packaging technology and the memory technology, that it requires very close collaboration on all of these.”

“We really are changing the performance of the power supply to match the needs to the processor,” he adds. “This will take several years to hash it out, and a lot of innovation
has to happen between that now and then.”