The deadly hidden dangers of the AI era: CME data center downtime reveals cooling system concerns
On November 27th, the world's largest futures exchange CME experienced a trading interruption for several hours due to a cooling system failure at its data center, affecting contracts worth trillions of dollars. The failure originated from a chiller unit malfunction at the Aurora data center in Illinois. With the development of AI, the heat dissipation issues of data centers are becoming increasingly severe, with their energy consumption being 50 times that of ordinary office buildings, most of which is converted into waste heat. Although equipped with redundant systems, the increased complexity makes it difficult to completely avoid failures
The trading interruption at the Chicago Mercantile Exchange (CME) has brought the cooling issues of data centers into the public eye.
On November 27, the trading platform of CME Group, the world's largest futures exchange operator, experienced several hours of disruption, affecting trillions of dollars in contracts across stocks, foreign exchange, bonds, and commodities.
The direct cause of this incident was a failure in the cooling system of its data center located in Aurora, Illinois. This data center is owned by private equity firms KKR & Co. and the operator CyrusOne, which is under Global Infrastructure Partners.
CyrusOne stated that a chiller unit in its facility failed, affecting multiple cooling units, and this "simple" physical failure triggered turmoil in global markets. To prevent equipment overheating, capital expenditures for cooling systems can typically account for up to 15% of the total investment in data center projects.
This incident is not just an isolated technical failure. Against the backdrop of the AI wave that once propelled NVIDIA to the position of the world's most valuable company, the cooling issues of data centers have become increasingly prominent.
Where Does the Heat Come From?
Data centers are buildings filled with servers, which consist of stacks of chips working together to process and store data.
Processing power is commonly referred to as "compute," which has become a key commodity necessary for AI companies to train their models.
Data centers profit by renting out compute power to other companies, meaning operators have an incentive to pack as many servers as possible into the same space to maximize capacity.
All these servers require a significant amount of electricity.
Due to their high energy consumption and 24/7 operation, a data center consumes 50 times more energy per square foot than a typical office building.
Most of the energy they consume is ultimately dissipated as waste heat. This is similar to how personal laptops or smartphones heat up when performing complex tasks.
Cooling Technologies and Trade-offs
Traditionally, servers are cooled using cold air, which works similarly to a household air conditioner.
Fans blow cold air toward the servers and then expel hot air from the server room. However, as data centers for artificial intelligence generate more heat, liquid cooling systems have become increasingly common since around 2022.
There are various methods of liquid cooling, such as delivering cold liquid through pipes to heat sinks closely attached to chips or immersing entire servers in containers filled with cooling liquid.
Some systems use low-boiling-point liquids that absorb heat and evaporate when they come into contact with high-temperature chips, then condense back into liquid for recirculation.
Compared to air, liquids can carry more thermal energy per unit volume, making them more efficient. However, these systems are complex to install and expensive, and if problems arise, they can be quite tricky; no one wants expensive chips to be soaked in liquid.
Whether using air or liquid, once the heat is transferred from the chips, it ultimately passes to a cooling water circulation system, which then releases the heat to the external environment through cooling towers or industrial chillers This is precisely the reason why data centers consume large amounts of water resources, raising concerns about their exacerbation of water resource pressure in water-scarce regions.
The Cost of Overheating
Overheating in data centers can lead to data loss, damage to expensive chips within servers, and service interruptions for customers.
The consequences are similar to the service outages caused by technical failures at several digital infrastructure providers recently.
For example, a major network outage at cybersecurity company Cloudflare Inc. last November rendered multiple websites, from social platform X to ChatGPT, inaccessible. Amazon Web Services, CrowdStrike, and Microsoft have also experienced similar issues.
Typically, data centers invest heavily in redundancy, including backup generators, additional cooling units, and even duplicating entire facilities to minimize the likelihood of interruptions.
However, as systems become increasingly complex, interruptions may still be difficult to avoid despite redundancy measures.
Review of the CME Incident
CME's trading platform is located in a park in Aurora, a suburb of Chicago, which belongs to data center operator CyrusOne.
According to CyrusOne, on November 27, a chiller unit in its Aurora facility failed, affecting multiple cooling units and ultimately leading to the trading interruption.
After the incident, CyrusOne stated that it had deployed temporary cooling equipment to supplement the permanent system while working to restore full cooling capacity.
According to information on the company's official website, its Aurora park features "advanced cooling technology," using air-cooled chillers and utilizing natural cold air or water for cooling when temperatures drop below 30 degrees Fahrenheit (approximately -1 degree Celsius).
According to weather forecast data, at 10:40 AM on November 28, the local temperature in Aurora was about 28 degrees Fahrenheit.
Notably, CyrusOne's website also claims that its Aurora facility has additional cooling units to cope with failures of air-cooled chillers.
It remains unclear whether this redundancy system functioned as intended during this incident
