Four structural forces are converging on power grids at once. First and most importantly, electricity demand is expected to more than double globally by 2050, thanks to proliferating AI data centers, the electrification of transport and heating, and the electrification of industry. Second, the infrastructure meant to carry that power is aging. In the United States, for instance, more than 70 percent of transmission lines and over half of distribution assets are older than 20 years.
Third, that crumbling infrastructure is being asked to support an energy system that’s being rewired from the inside, shifting from centralized fossil generation to a decentralized, multidirectional network. Wind and solar's share of generation alone is projected to climb from around 15 percent today to over 60 percent by 2050. (Include hydropower, biomass, and geothermal power, and clean sources top 75 percent by midcentury.)
Fourth, extreme weather is arriving more often and hitting harder, demanding grid hardening at a scale utilities have never had to anticipate before. Grid-related outages already cost the United States over $50 billion a year (the highest toll among major economies).
At a recent Columbia Business School Climate Knowledge Initiative (CKI) workshop on Power Grids, which convened a dozen experts from grid operators, investment firms, think tanks, startups, and academia, the mood was less fatalistic than that backdrop might suggest. "We are in a moment of extraordinary electricity load growth," said Jigar Shah, formerly the head of the Loan Programs Office at the U.S. Department of Energy. "I view that as an opportunity, not a problem." That’s in part because the current state of the grid isn't just a constraint; it's also the challenge standing between today's energy system and the clean generation already waiting to connect to it.
Interconnection queues are backed up across the United States, leading to an average wait time of five years, according to the Lawrence Berkeley National Laboratory. These delays add costs and uncertainties for project developers and investors. Constructive policy changes have pushed developers to withdraw speculative and stalled projects, which has helped cut the queue between 2024 and 2025, but more needs to be done to truly ease interconnection bottlenecks.
Grid investment has structurally lagged investment in new energy generation over the past decade. In the U.S., private capital has picked up only part of the slack. Private investment nearly doubled between the periods of 2009–2016 and 2017–2024, but its share of the total only rose from 17 to 24 percent, with federal programs not quite sufficient to cover the remaining need.
At the CKI workshop, participants agreed that an “all-of-the-above” mindset will be necessary in tackling the messily interconnected problems the power grid faces. That said, only so many solutions can be implemented at once, which calls for a shared prioritization of next steps. By the end of the day’s discussion, participants had agreed on a useful framework to sequence the all-of-the-above response: optimize what already exists in the short term; modernize the grid and redesign markets in the medium term; and expand the physical grid in the long term. Crucially, these are different time horizons, not sequential steps.
The lively daylong workshop discussion touched on various examples of what each step could look like — and the challenges likely to crop up along the way. Even as regions around the world will need to solve these problems with their own hyper-local and distinct approaches, the conversation yielded several important takeaways that are broadly applicable to the world’s overstressed and fraying grids.
Key Point 1: Massive demand growth from AI is compounding the pressures on the grid – and also showing promise as a tool to make grids more efficient.
AI (and the data centers that power it) are adding urgency to the drive to update the grid because of their massive demand requirements — and, workshop participants agreed, those same forces may apply the pressure to finally fix it. By 2030, data centers are projected to account for roughly half of U.S. demand growth, and the scale and urgency of that demand is doing something federal grants alone haven't: it’s forcing utilities, regulators, and investors to treat grid investment as unavoidable rather than optional.
“Much of the stress on the grid is because of AI," said Ayşe Coşkun, Chief Scientist at Emerald AI and Director of the Center for Information and Systems Engineering at Boston University. “Generic solutions are not necessarily going to work."
In many states and localities around the United States, the response to data center construction has been blunt, if uncomfortable: New York recently imposed a moratorium on new data center construction over 50 megawatts, and Texas has followed with its own data-center moratorium, pending deeper audits.
Coşkun sees opportunity in the crisis rather than just risk. "There's no quick solution, no matter how much money one company might throw at the problem,” she said. “That creates an opportunity to use power flexibility and design a power grid system that is more sustainable, cleaner, and cheaper in the long run.”
Paradoxically, AI is likely to be a central actor helping to plan that cleaner and more efficient new grid system. Shah pointed out that this is already happening: he mentioned a case in which GridCare’s AI-enabled platform helped Portland General Electric identify more than 400 MW of latent capacity on its existing grid. That is billions of dollars in potential value.
Various other emerging and growing business models are demonstrating how AI tools can help to optimize the grid, simply on the strength of smarter software.
Key Point 2: Short-term solution: AI-enabled optimization is fastest and cheapest, but it only closes a fraction of the gap.
Of the three levers, optimization is the one utilities and developers can pull almost immediately. It means squeezing more capacity out of infrastructure that already exists, largely through software like load-shifting tools, distributed energy resource management systems (DERMS), virtual power plants (VPPs), and other forms of demand response (DR).
The economics of grid optimization are attractive; estimates suggest digital optimization and demand management could generate systemwide savings of more than $150 billion by 2035, and DERMS-coordinated resources, potentially reaching 80 to 160 gigawatts by 2030, could save the U.S. grid roughly $10 billion a year while cutting projected transformer overloads from 81 to 28 percent.
“Today, many people make the assumption that we need triple the compute power, and therefore triple the energy, whereas the solution is probably about grid flexibility and permitting reform,” said Emmanuel Lagarrigue, Co-Head of Climate Transition at KKR.
Two companies represented at the workshop showed how different that flexibility can look in practice.
Emerald AI, founded in 2024, is running commercial pilots that treat data centers as controllable, battery-like assets, with software that makes them power-flexible. In a real-world test with the UK's National Grid (part of a program called F3DC Flex, run with Nvidia and cloud provider Nebius) Emerald AI showed this working live: when the grid operators asked the data center to cut power use, the center throttled back only its least time-sensitive computing tasks, while everything users were actively relying on kept running normally.
Coşkun argued that Emerald AI’s business case is stronger than the one that emerges from simply offering an electricity discount. Data centers facing five-to-seven-year interconnection waits can instead agree with utilities to a small, pre-determined level of flexibility in exchange for faster grid connection. She pointed to Duke University’s Rethinking Load Growth study, which suggests that as little as 0.5 percent annual curtailment could free up roughly 100 gigawatts of capacity nationally, as well as regulatory developments across a growing number of U.S. states that are exploring flexible-load and faster-interconnection frameworks.
New York is now testing a version of this trade directly: Rich Dewey, president and CEO of the New York Independent System Operator, said NYISO's newly proposed framework lets a load connect as either "firm" or "flexible," with flexible service arriving faster but requiring re-study if the customer later wants to convert to firm power (with no agreement around levels of flexibility). Voltus offers a second, complementary model, aggregating over 8.5 GW of capacity from a large number of sources rather than a single large customer. The company channels funding from data centers into building out virtual power plants that also draw in residential participants through partners like Resideo, whose smart thermostats shift air-conditioning loads during grid events.
Emily Orvis, Vice President of Energy Markets at Voltus, argued that this kind of demand-side management has quietly become routine grid operations rather than a rare emergency tool. Heat maps of Voltus's dispatches show the company being called on every single day since June 2024, a stark shift from the once- or twice-a-summer demand response of twenty years ago.
Not every flexibility technology is ready for that level of reliance yet. Though vehicle-to-grid (V2G) networks are gaining traction, with China’s NaaS among the world’s largest, several participants were skeptical V2G (technology that lets electric vehicle batteries send power back into the grid, not just draw from it) is close to commercially viable at scale. Luke Liu, Director at Copenhagen Infrastructure Partners (CIP) added that the V2G business model may ”take decades to figure out."
Workshop attendees agreed on the possibility of a real danger in overselling optimization as a solution. People may seize on the "easy" gains to the grid available via optimization — often estimated at roughly 20 percent of the total need — and dither when it comes to moving on to the harder, more expensive work of modernizing and expanding the grid.
Dewey of NYISO agreed the near-term case for optimization is really about time, not a substitute for what comes next. "The need for new supply is still very, very real,” Dewey said.
Key Point 3: Medium-term solution: Modernizing the grid means redesigning markets and incentives — and agreeing on who pays.
The “modernization” lever concerns the hardware layer just above optimization: grid-forming inverters, grid-scale battery storage, flexible AC transmission systems, advanced metering, digital twins. It is more capital-intensive than optimization, requiring an estimated $200 billion to $400 billion a year globally between 2030 and 2035, but still faster to deploy than large-scale grid expansion.
The technology, participants agreed, is largely ready. The harder problem is designing markets, incentives, and institutions that can actually monetize and put that tech to work, and getting the relevant stakeholders (utilities, regulators, developers, and investors, who each hold a different piece of the puzzle) to align.
“You can't have engineers solve a political problem,” Dewey said. “The politicians have to solve the political problem, and then just unleash the engineers to design it.”
In the U.S., that political problem most often comes down to cost allocation. While everyone tends to agree on the engineering, the fight is over who pays. Dewey traced how New York eventually broke a two-decade impasse over upgrading its backbone transmission system, a fight partially about which local utilities would fund it. It took the state's Public Service Commission stepping in to allocate costs — an unpopular move, but one that finally got the project moving.
Scaled up to a multi-state level, the same dynamic gets far harder: Dewey pointed to the difficulties in coordinating inter-regional transmission between New York and New Jersey, despite obvious reliability benefits for the whole East Coast.
Liu of Copenhagen Infrastructure Partners described a similar stalemate, where long-planned transmission lines sited across state borders typically sit in development for over a decade — not blocked by permitting or technology, but by the absence of any agreed mechanism for recovering the investment across states and utilities.
“What matters most is political commitment and a stable regulatory framework that enables the necessary investment in grid infrastructure,” Claußnitzer said. One transmission line project in Lower Saxony demonstrates the potential benefits of such investments: built at a cost of approximately €1 billion, it reduced redispatch costs by nearly €500 million in its first year of operation. The investment therefore paid for itself within around two years.
Clear and reliable frameworks are also needed to ensure that scarce grid capacity is allocated efficiently. Together with Germany’s other transmission system operators, TenneT Germany has introduced a maturity-based grid connection process for battery storage facilities and other large consumers. The new process replaces the previous “first-come, first-served” approach and assesses projects against transparent criteria, including planning maturity, permitting status and economic viability. It is designed to prioritize projects with a high likelihood of realization and make the allocation of scarce grid connection capacity more predictable and efficient.
That regulatory shift changed how investors viewed the German market – a claim that KKR’s Lagarrigue confirmed. "We were carefully avoiding Germany," Lagarrigue said of KKR's earlier battery investments. "Now, with that change, Germany has become a really interesting market."
The UK offers a related but distinct lesson on the retail side: Octopus Energy operates a national flexibility business, leveraging the grid operator’s Demand Flexibility Service (DFS)— which pays retailers to shift household load. This business only functions within the context of a deregulated market and dedicated distribution-level markets that don't yet exist in most of the U.S., which is one reason Octopus's expansion into the American market has been limited mostly to licensing the software of its spinout company, Kraken, rather than replicating its retail model directly.
In other words, the UK offers an example of how political reform changed system incentives — leading to Octopus’s success in its retail business.
Still, no model needs to serve as an exact template for any other region. A common refrain at the workshop was how distinct and tailored system reforms and solutions will need to be across regions — and even across U.S. states.
Key Point 4: Long-term solution: No matter what, the clean energy transition will require a slow and expensive expansion of the physical grid.
Even with aggressive optimization and modernization in place, workshop participants agreed that grid expansion (that is, new transmission and interconnection capacity) is both unavoidable and an untapped opportunity. It is slower and more capital-intensive than optimizing or modernizing the grid, and will require an estimated $200 billion or more annually from 2030 to 2035 and five to fifteen years to deliver, but it needs to happen anyway: even moderate U.S. load and clean-generation growth scenarios require transmission capacity to rise by 20 to 25 percent.
High-voltage direct current (HVDC) lines are a central tool here, moving large volumes of power over long distances with lower losses than conventional transmission, though their high converter-station costs only pay off over long corridors.
The gap between countries that can build this kind of infrastructure and those that can't is stark: China has completed 45 ultra-high-voltage lines in roughly 15 years under central planning, while in the U.S., some 36 HVDC projects (representing roughly 187 gigawatts of capacity) remain stalled largely due to the lack of a cost-allocation rule.
Cost allocation isn't just a political sticking point. It also exposes a deeper financing gap, since transmission is structurally harder to fund than other parts of the grid. Unlike generation or data centers, transmission offers private investors few ways to place a bet. Liu said private investors simply don't have the same opportunity to speculate on transmission that they do on a merchant power plant: the barrier isn't permitting or technology, he said, but simply coordinating how the investment gets paid back across state lines. Liu described a promising new investment model emerging that skips the traditional utility system entirely. “Because of the AI boom, there are some novel ways of approaching transmission beyond the traditional rate-based model,” Liu said.
He described venture-backed companies building their own private power lines that run directly from a power source to a single data center. Instead of ratepayers footing the bill the way they normally would for shared grid infrastructure, these lines are financed by companies racing to lock in power for one big customer, and could eventually be linked together into a larger private network.
“There are completely new ways to think about funding infrastructure in this space,” Liu added. “It requires a lot of money from new sources including VCs and growth capital, and I think that's where this kind of the new capital will come in.”
Emily Orvis of Voltus raised a complementary hurdle, and proposed a financing solution: “Institutional investors are fairly exclusively focused on investing in single large projects,” Orvis explained. “Unfortunately, the policy barriers to single large projects are very great.”
What if, she suggested, hundreds or thousands of smaller grid investments could be aggregated into an investable portfolio that could open up capital that's currently sitting on the sidelines?
But even where the "who pays" problem does get solved, expansion alone won't fully answer the question of what powers the grid over the long run.
NYISO’s Dewey was blunt about the limits of today's limited (though rapidly expanding) cleanenergy mix: "You cannot run a power system on wind, solar, and storage, period,” he said. “If we want our 2060 goals to be possible, we've got to start working on developing what those firm, dispatchable, emissions-free technologies are."
Nuclear and geothermal stand out as the nearest-term firm-power options. Both also enjoy bipartisan political support in an otherwise challenging policy environment.
C. Lindsay Anderson, Professor and Chair of the Department of Biological and Environmental Engineering at Cornell University, argued that expansion and new generation should, ideally, be planned together, not sequentially. After all, developers would rather build new clean generation with confidence they can connect it to the grid; the two have the potential to de-risk each other when planned in tandem.
Still, the need to plan ahead ought to be balanced against the urgent need to move forward – sometimes without perfect system-side designs in place, said Dan Zarrilli, Columbia University’s former Chief Climate and Sustainability Officer. Zarrilli pointed out that the goal isn't a perfect market plan or investment vehicle or technology; rather, it's a set of imperfect solutions that are workable enough to build on.
"We have to be willing to put aside the idea that there's some perfect, efficient, optimal set of solutions," Zarrilli said. "Instead, we need solutions."
As with so much related to the grid, this is the slow, costly work of expansion in parallel, across solution levers. Optimization is the lever that will result in the most immediate impact, and the technology for it — DERMS, VPPs, demand response — is largely ready to deploy today. Modernization is the medium-term work of getting hardware that's also mostly ready, from gridforming inverters to battery storage, onto the grid. And expansion is the long-term, unavoidable commitment to new transmission and new firm generation that no amount of optimizing or modernizing can substitute for.
None of the three is optional, and none can wait for the others to finish. The grids that get ahead in this moment will be the ones that pursue all three levers — imperfectly, locally, and without delay.
