Written by: Zhou Xiaoyan
Source: Tencent Technology
James Wardy, principal engineer at Digital Realty's Ashburn data center. Images processed by AI
James Waddy walks 30,000 steps a day.
His workplace is located in a data center of more than 90,000 square meters in Ashburn, Virginia. Every day, he checks the air-handling equipment, server room, ducts, cables, and rooftop condensers. Sometimes he has to smell for burning smells, feel for abnormalities in the ducts, and see if birds have pecked the equipment.
These may not sound like they have much to do with AI, but this is exactly where AI computing power finally lands.
In the past year, when people talked about AI infrastructure, they focused more on GPUs, power plants, transformers, land, financing and hundreds of billions of dollars in capital expenditures. But when you actually walk into the data center, you will find that this is not just a computing industry, it is also a huge physical system that requires continuous operation 24 hours a day.
There are currently about 2,700 data centers in operation in the United States. As the demand for computing power brought by AI increases, new facilities are constantly being built, and the requirements for power, cooling, network and other systems are getting higher and higher.
Behind these systems, a group of people are required to guard them for a long time. They may not be engineers developing AI models, nor are they responsible for training large models, but they must ensure that the building where the server is located will not stop due to overheating, power outage, equipment failure, or a fiber optic problem.
In a sense, they are more like "nannies" in the AI era.
Thirty thousand step inspection: there are no “small problems” in the data center
Wardy is the principal engineer at Digital Realty's Ashburn data center, where he has worked for three years.
Digital Realty's data center in Ashburn, Western Washington
He has a business degree and previously worked in sales for an HVAC company, working his way up from the night shift to now managing dozens of engineers and calling himself the "general" of the team.
His main job is to ensure that the critical infrastructure of the data center continues to operate. Daily inspections begin in the air handling room, where huge air conditioning equipment continuously delivers cold air to the data center.
Digital Realty Ashburn data center rental equipment does not have any customer logos or names on it
Further up, the data hall looks a bit like a giant locker room, with rows of servers put into cabinets or cages for rent by different companies. You can't see the customer's logo here, and you can rarely tell who the server belongs to. Competitors' equipment may be running at the same time across a row of cabinets. The closer you are to the server, the louder the roar, and workers need to wear earplugs.
Equipment at Digital Realty Ashburn Data Center
Continuing up to the roof, Wardy checks the rows of condensers for malfunctions and looks out for damage caused by birds. Standing on the roof, he could see the cranes in the distance where another company's data center was being built.
"It's nice up here," Wardy said. "I like inspections like this."
The data center has been in operation for nine years, and its corridors are so long that some technicians ride scooters to get around. Inside there are towering cable cages, as well as cots and showers for staff stationed during extreme weather.
On some days, there may be as many as 120 people on site, of which about 50 are regular employees and the remainder include contract security personnel and client personnel. But even so, plenty of rooms in the huge building remain vacant during the week.
Although Wardy's position is called "engineer," in terms of specific work content, it is not the same as a R&D or design engineer in the traditional sense.
Winston Smith, an analyst in the technology field, pointed out that this type of position is closer to a critical facility technician (CFT) or a data center operation and maintenance technician (DCO Technician). The core tasks are on-site inspections, infrastructure maintenance and emergency response.
He believes that Wardy relies on hearing, smell and touch to detect anomalies every day, essentially acting as a "human sensor" for the data center: discovering possible problems with mechanical, electrical or cooling systems before the monitoring system issues an alarm or even before the equipment actually malfunctions.
This kind of work may seem trivial, but no detail can be missed, because what data centers fear most is accidents. An unexpected arrival of large equipment, a minor system failure, or even a seemingly inconspicuous anomaly can impact a facility's redundancy and maintenance schedules.
Therefore, for Wardy, the so-called "good days" are not how many big problems have been solved, but that no problems have occurred.
Wardie walked on the raised floor, where chambers below circulated air to dissipate heat from the servers. As he walked, he checked the equipment. No unusual sounds, no unusual smells, everything is normal.
This is what data center operators want to see.
Six Nines: An optical fiber one-tenth the thickness of a hair can’t go wrong
What the data center pursues is stability to the point of being boring.
Digital Realty estimates that its data center uptime has reached 99.999%, and currently hopes to further increase it to 99.9999%. It looks like there is only one more "9", but the corresponding annual downtime will be shortened from about 6 minutes to about 30 seconds.
In order to gain these tens of seconds, every link in the data center must be reliable enough.
What Jeff Hansen, a data center technician at CC&N, faces every day is the smallest link: optical fiber.
The core diameter of single-mode fiber is only 8 to 9 microns, and a human hair is about 75 microns thick. When two optical fibers are connected, the fusion splicer needs to align the fiber core with 1000x magnification, and then fuse the glass together through an arc. Hansen calls it "microscopic welding."
“Working with optical fibers is like playing with lasers,” he said. “An optical fiber is a microscopic strand of glass that carries an incredible amount of information. That fascinates me.”
Inside Digital Realty's Ashburn data center
There may be thousands of such connections in a room in a hyperscale data center. During installation, each fiber is individually routed, terminated, and tested. Once a test fails, there may be millions of reasons behind it, and technicians need to rely on experience and processes to investigate step by step.
Data center cabling
Therefore, "winging" is rarely allowed in the data center. Before Hansen's team goes on site each time, they meet to confirm the work, tools, and scope of the project. The person in charge will need to prepare a briefing to understand in advance how the venue has changed since the last visit.
Because data centers change quickly. A site that was still under construction six months ago may already be fully operational. Access rights, operational protocols and risk levels are subject to change.
"You can't improvise data center work," Hansen said. "It requires a lot of planning to get it right, and that starts before we even set foot on site."
The same goes for access control. Access to hyperscale data centers often requires badge checks, identity verification and even iris scans. Cardboard and paper may be prohibited due to dust generation, and storage devices such as USB flash drives may also be prohibited.
A job that seems to only take five minutes cannot be done as soon as you say it. Replacing equipment, adding systems, moving cables, and performing maintenance all require advance application, risk assessment, scheduling, and coordination with other teams.
Every exception must be recorded. Small fluctuations in power, flickering lights, unusual noises from cooling equipment, or even a small puddle of water on the ground can all become reportable incidents.
For wiring technicians, "safety" even includes three meanings at the same time: they cannot be injured, their colleagues cannot be injured, and the equipment cannot be damaged. Because in a data center that is already in operation, moving a working optical fiber may have direct consequences.
What the data center really needs is not on-the-spot heroes, but a set of processes that do not rely on heroes as much as possible.
AI turns "nanny" into a popular profession: behind the high salary is the talent shortage
In the past, data centers were a relatively invisible industry. Now, AI has thrust it into the spotlight.
According to industry data provider Data Center Map, there are currently about 2,700 operating data centers in the United States, nearly 400 are under construction, and at least 1,500 are in the planning stage. Ashburn, where Wardy is located, is located in the "Data Center Alley" (Data Center Alley) about an hour's drive outside Washington, DC.
The data center itself is not a labor-intensive industry. The long-term operation of a facility usually requires only dozens to hundreds of employees. But when thousands of facilities are being constructed and operated simultaneously, the need for engineers, technicians, maintenance personnel and project managers expands rapidly.
The recruitment website Indeed has had more than 1 million job seekers searching for data center-related positions this year, and the search volume in August increased eightfold compared with the beginning of 2022.
Indeed's research also shows that data center salaries for hourly workers engaged in maintenance and installation work are about 42% higher than similar positions in other industries. The annual salary of a data center project manager is approximately $115,000. Information provided by Digital Realty shows that the annual salary of its entry-level engineers is about US$66,000, and the average salary of experienced engineers can exceed US$120,000.
Technology commentator Evan Kirstel also noted this salary change. He said on social media that data center construction is bringing higher incomes to positions such as electricians, technicians, HVAC and security, and that investment in AI infrastructure is being transformed into some specific blue-collar employment opportunities.
But what is really scarce are people who can handle complex infrastructure.
Brandi Galvin Morandi, chief people officer of digital infrastructure company Equinix, said professionals in electrical systems, network engineering and cooling systems are in high demand.
In Ashburn's data center, there's a mix of white-collar and blue-collar jobs. Someone is responsible for security, someone is responsible for configuring customer cabling, someone is purchasing materials, and there are engineers monitoring the entire facility. This job mix is roughly similar in other data centers across the United States.
The problem is that the growth rate of talent cannot keep up with the speed of facility construction. Cindy Fiedelman, chief human resources officer at Digital Realty, said these facilities need to operate around the clock, so practitioners need both technical training and safety discipline and hands-on experience, and the supply of such talent is slower than the industry's expansion rate.
As a result, community colleges began to offer data center-related courses, and large technology companies began to invest in training programs. In Virginia, Northern Virginia Community College has launched a data center jobs program with private sector support. Technology companies, including Meta, are also involved in developing skilled workforce.
Michael E. Webber, author of the Energy 101 website and professor at the University of Texas, believes that AI data centers are creating blue-collar jobs and may also squeeze some white-collar jobs, but white-collar workers are relatively easier to switch to other occupations.
Digital Realty currently operates more than 300 data centers across six continents. In facilities like Ashburn, it leases space to large tech companies in need of computing power, including Oracle, Meta and IBM.
The servers may come from different companies, and the engineers may come from different teams, but everyone ultimately faces the same requirement: the system cannot be stopped.
There are more and more data centers, and it is becoming more and more difficult to be a "nanny"
The construction boom brought about by AI has transformed data centers from a behind-the-scenes infrastructure into an industry that is increasingly visible to the public.
Don Atkinson, 53, has worked at Digital Realty for about 14 years and has also done backup power system maintenance. Today he manages young engineers.
Don Atkinson, Ashburn Data Center Manager, Digital Realty
What made him feel uncomfortable was the change in the attitude of the outside world. He also said that he doesn’t use AI very much. “It’s a strange time now.”
Another employee, Malcolm Mosely, was more direct: "I'm just here to do my job."
Digital Realty Ashburn data center employee Malcolm Moseley
But it has been difficult for the industry to stay out of the fray. In Atlanta, Digital Realty is seeking an exemption from a local construction ban. The ban restricts the construction of data centers near transportation hubs, where the company hopes to build a $500 million data center. Local residents have opposed the project, which is currently at a standstill.
Similar controversies are emerging with data center expansion. In the past, little attention was paid to what went on in these buildings. Today, energy consumption, water use, and impacts on communities are entering public discussion.
The industry is being redefined. It is no longer just the backend of the Internet, but also the infrastructure of the AI era. High salaries, hot hires, community controversy, energy pressures—all happening at the same time. But hidden behind all the noise, there is still a group of silent engineers who maintain the operation of the entire digital world with a rhythm of 30,000 steps a day.
Good weather or weather where nothing happens. Good days and days when nothing happens. This will probably never change.