Data centers are the new oil
He who builds the servers makes the rules
Art: Desert Journey, Maynard Dixon, 1935
A few months ago, a friend sent me a link about Matthew McConaughey coming to Utah to speak at a tech conference, which seemed like a complete non-sequitur. To be fair, this wasn’t the first time a celebrity had come to Salt Lake City. Both Michelle and Barack Obama had spoken previously, as had Ellen, Trevor Noah, Oprah, and a number of other extremely famous people.
Utah’s larger tech companies like Qualtrics, Pluralsight and Domo are known to draw big names to their conferences. In previous years, human rights activist Malala Yousafzai, famed NBA coach Phil Jackson, Olympian Michael Phelps, Broadway star Lin Manuel-Miranda and basketball legend Magic Johnson graced the companies with their presence.
It all seemed great on the surface, but what were all of these famous and influential people doing in...Utah...at analytics conferences? It turns out the answer is surprisingly interesting and tied to the history of Utah’s tech industry.
The Fourth Node
In the 1950s, in response to the Russians launching Sputnik, Eisenhower established ARPA, focusing on space technology. (Actually, the more I read about the American and Russian internets, the more I become convinced that there’s no way we would have the internet we have today if Russia and America didn’t hate and fear each other over long periods of political instability. But I digress. )
ARPA grew over time, and in the 1960s, its project members began experimenting with the idea of connecting computers together, based partially on the thought that a good distributed system could survive a nuclear war.
In a 1963 memo, JCR Licklider, an Important Person in Computing History and one of the key members of the project, wrote,
It will possibly turn out, I realize, that only on rare occasions do most or all of the computers in the overall system operate together in an integrated network. It seems to me to be interesting and important, nevertheless, to develop a capability for integrated network operation. If such a network as I envisage nebulously could be brought into operation, we would have at least four large computers, perhaps six or eight small computers, and a great assortment of disc files and magnetic tape units–not to mention the remote consoles and teletype stations–all churning away.
There is a lot of other really fascinating stuff in the note, including Licklider low-key inventing data science and a glimpse into the way business correspondence was conducted before email, but basically the important piece of info is that this memo planted the seeds for what would become ARPANET, and, later on the modern internet.
The first three computers to be networked together for ARPANET were located in California. The fourth was at the University of Utah. Initially, the fourth was meant to be in California as well, but other universities backed out because they didn’t want to do timesharing, aka give access to their precious machines.
Luckily, David Evans was there to make the case for Utah. He was a native of Utah who went to Berkeley, where he became the head of ARPA. Once back at Utah, he put a tremendous amount of work into the computer science program, where he generated an enormous amount of talented and smart alumni ( For example, one of Evan’s students was Ed Catmull, who founded Pixar) , and into ARPA, as well, and made the case that the University of Utah should be part of the experiment.
Once the four nodes were connected, the researchers working on the project wrote,
"We have found that, in the process of connecting machines and operating systems together, a great deal of rapport has been established between personnel at the various network node sites. The resulting mixture of ideas, discussions, disagreements, and resolutions has been highly refreshing and beneficial to all involved, and we regard the human interaction as a valuable by-product of the main effect."
After seeing the internet in its modern-day incantation, Vint Cerf, one of the researchers, said that if he could do it again, he wouldn’t have invented the internet, and in fact he would rather have thrown the servers into the sea and himself after them.
I’m just kidding, he didn’t say that. (But he kind of did.)
Anyway, because of Utah’s partnership with the defense community through ARPANet, the Department of Defense started paying attention to what else Utah had to offer: land in the interior of the United States, strategically located away from coasts, and lots and lots of farmland with room to grow that was extremely flat. The state and the DoD established a relationship over time that included building out air force bases, and proving grounds, and much, much, more that we’ll get to a little later on.
Utah Unicorns
The local tech scene grew and flourished, stoked by the fires of the University’s success (and Evans’s alumni and his relationships in the industry), and Utah became a tech hub for a number of companies, including Word Perfect in the 1970s and Adobe in the 1980s.
(As a side note, it’s interesting to note how much Apple’s relationship with Adobe has changed, from constant investment, to blocking Flash entirely on their devices.)
During the 1990s, a second tech boom occurred, shepherded partly by the then-governor, Republican Michael Leavitt, who went to Silicon Valley regularly to lure investors inland.
He was able to lure over eBay and Intel to move some of their operations to Utah. The local tech scene was also growing. It started with Omniture, a web analytics tracking company (remember, everything is logs, and Omniture did logs before almost anyone else at scale). which Adobe later bought for $1.8 billion.
The CEO of Omniture was Josh James, and his success became Utah’s success. Spurred on by his momentum, other companies started to grow.
The biggest of these was Qualtrics, which does marketing surveys, or as their Wikipedia page calls it, “experience management”, was very recently acquired by software giant SAP for $8 billion.
Other companies grew and prospered: Pluralsight, Ancestry.com, 1-800-Contacts, Overstock (really bizarre story there, BTW, and finally, Domo.
Domo was Josh James’s second company, which he founded after he sold Omniture. It specialized in providing SaaS for a company’s internal data. (Having used Domo at several previous companies, I can report that the initial user experience was terrible and it was a pain to connect to most data sources, but it created beautiful-looking dashboards, which means the C-suite loved it. It may have improved since.)
Domo was much-hyped in the tech world, and, after James’s previous successes, got all the right attention, including investments from A16Z and Salesforce’s Marc Benioff.
However, there was still a problem. James felt like he had to struggle to get anyone to pay attention to Utah.
James has no qualms with how things turned out for Omniture. But leaving aside that successful outcome, he cannot quite forget how Omniture--when it was a startup--endured a needless three-year wait before receiving the venture capital it warranted. He knew he was hardly the only Utah entrepreneur who felt slighted by the Silicon Valley VC mainstream. So he founded Silicon Slopes, a promotional nonprofit to help with the branding of Utah's tech community, "as a way for us to highlight how much is really going on out here," he says.
To combat this, he established Silicon Slopes, an industry group designed to promote the Utah tech scene, especially, like Leavitt did before, to Silicon Valley investors who might think about setting up shop there.
On its website, Silicon Slopes proclaims that it’s
the voice, hub, and heart of Utah’s startup and tech community. We are a 501(c)(3) nonprofit organization empowering Utah’s startup and tech community to learn, connect, and serve in an effort to make entrepreneurship and opportunity in Silicon Slopes open and accessible to all.
While James was CEO at Domo, he created a tech tour to take Domo to various cities. This turned into Domopalooza, a conference built around the Domo product. In 2015, he got Sheryl Sandberg to speak at Domopalooza. This was all a precursor to the Silicon Slopes event that the non-profit produces (last year, at a cost of $2 million.)
Over time, helped by James’s industry connections and PR and journalism buzz, Silicon Slopes has really grown to become the all-encompassing moniker of the tech industry in Utah and a short-handed way for journalists to refer to the region.
This is the place
While all of this was happening on the software side, there was work going on on the infrastructure side. Because Utah had been so early to the internet, it benefited from both the infrastructure and innovation in the space, and got involved in high-speed internet infrastructure ahead of other cities. For example, it was one of the first cities involved in Google Fiber’s initial drive.
This made it extremely attractive not only to startups, but to people who wanted to set up internet infrastructure. Specifically, data centers, more specifically, the NSA. I mentioned earlier on that Utah is the perfect place for a lot of strategic military work because it’s in the middle of the country and has a lot of areas with flat land.
Based on that and the DoD history Utah already had as a result of being part of ARPANet, in 2013, the NSA set up shop.
In the little town of Bluffdale, Big Love and Big Brother have become uneasy neighbors. Under construction by contractors with top-secret clearances, the blandly named Utah Data Center is being built for the National Security Agency. A project of immense secrecy, it is the final piece in a complex puzzle assembled over the past decade.
Its purpose: to intercept, decipher, analyze, and store vast swaths of the world’s communications as they zap down from satellites and zip through the underground and undersea cables of international, foreign, and domestic networks. The heavily fortified $2 billion center should be up and running in September 2013.
(BTW, Kashmir Hill (remember?) wrote an interesting story about going to visit the data center as it was under construction, and it doesn’t sound like they were too thrilled about visitors.)
Putting Computers in Containers
Let’s back up a second. What is a data center? Often, when we talk about the cloud and computing and all of these consumer-facing SaaS companies, we have this very hazy idea that somewhere out there is a server or some group of servers, sitting on an actual cloud, surrounded by cherubs, handling all of your stuff quietly.
But what’s really handling all of our requests, uploading all of our pictures, writing all of our Google Docs, sending all of our newsletters, is this thing:
It’s simply a building that has rows and rows and rows (called racks) of computers that SaaS companies use to crunch their logs.
“There is no cloud,” as the bumper sticker says, “it’s just someone else’s computer.” Or rows and rows of racks of computers, all performing the tasks we need to make the internet work.
When you’re logging into an EC2 instance, or checking your bank account or reading your email, somewhere on the other side of the world is a server, quietly crunching those numbers. And if your company is big enough, you need a lot of servers.
They are a mere fraction of the tens of thousands of data centers that now exist to support the overall explosion of digital information. Stupendous amounts of data are set in motion each day as, with an innocuous click or tap, people download movies on iTunes, check credit card balances through Visa’s Web site, send Yahoo e-mail with files attached, buy products on Amazon, post on Twitter or read newspapers online.
Especially if you’re running deep learning algorithms, such as those, say, recommending stuff to you.
In a new paper, researchers at the University of Massachusetts, Amherst, performed a life cycle assessment for training several common large AI models. They found that the process can emit more than 626,000 pounds of carbon dioxide equivalent—nearly five times the lifetime emissions of the average American car (and that includes manufacture of the car itself).
A large data center can use as much electricity as a small town.
Most data centers, by design, consume vast amounts of energy in an incongruously wasteful manner, interviews and documents show. Online companies typically run their facilities at maximum capacity around the clock, whatever the demand. As a result, data centers can waste 90 percent or more of the electricity they pull off the grid, The Times found.
And, those servers can’t just sit there: they need love and maintenance. Since they get hot, just like your laptop after a couple minutes using Slack, they need constant monitoring and cooling. And, to set up a data center to receive maintenance correctly, it takes a lot of infrastructure, as Amir Michael explains in this podcast.
Amir previously worked at Facebook and Google, and when Google tried to build data centers in shipping containers, for example, Amir recalled that they needed so many utilities to come directly to the disconnected container, namely water and electricity, that it ended up not being worth it in the end.
Aside from the physical logistical concerns, there are the politics. Datacenters are usually big, involve municipal services, and you need buy-in from the community. You need permits to build roads and route electricity, and you need to convince the local administration that data centers will bring jobs.
For example, take the Facebook data center in Utah. Or, take the Google data center approved recently in Henderson, Nevada. Or the Google data center, again in Utah. All of them say they’re going to build employment in the area, build a tech scene, and to improve capacity.
Such was the case with the NSA.
The NSA Data Center had a lot of infrastructure needs, some of which the city could provide and some of which would need to be built out to service the data center. That infrastructure buildout included a new water line that city officials hoped would spur more real-estate development, maybe attracting the government contractors who tend to multiply within a five-mile radius of any intelligence-agency outpost. While the new infrastructure is welcomed, those new businesses don’t appear to have materialized.
Establishing all of that and setting everything up takes a long time, even if you’re the government. But once word got out on the street that the NSA was setting up shop, the big companies came around as well.
The most interesting of these has been Facebook. In 2018, the company announced it would be building a huge facility at Eagle Mountain.
And, to bring this back around, will you look at who gave a talk at the Silicon Slopes conference in 2020, with the senator from Utah congratulating him. (Coincidentally, Silicon Slopes also took this time to announce a new venture fund, backed partially by Josh James and Ryan Smith, the CEO of Qualtrics.)
The Silicon Slopes Venture Fund was created by Qualtrics co-founder and CEO Ryan Smith, Omniture co-founder, Domo founder and CEO Josh James, and Pelion Venture Partners General Partner Jeff Kearl.
After sharing news of the venture capital fund... Utah Republican Senator Mike Lee became Zuckerberg's hype man of sorts, by introducing him in a short video, referring to him as a friend who he may not always agree with.
At first glance, the obvious story here is that Josh James is using Silicon Slopes as a vehicle to gain political and tech influence, particularly given that, based on the bio in his own website, he’s working with former Governor Leavitt and a couple of other politicians on the initiative.
With former Governor Leavitt, Josh lead the charge for the Utah Silicon Valley Alliance for entrepreneurs, an economic development program designed to attract the best of California’s technology industry to Utah’s business-friendly environment. He has served as an advisor on the technology and startup community to the two most recent governors, Governor Jon Huntsman and Governor Gary Herbert.
He’s also recently been congratulated by, perhaps the most nationally prominent Utah politician, Mitt Romney.
And don’t forget Michelle and Barack Obama, both speaking in Salt Lake City. So of course there has to be some sort of interesting connection between James, these politicians, the ever-present defense industry in Utah, and the connections he was able to make in Silicon Valley.
An interesting hypothesis (which is purely idle speculation on my part) is that probably, when he talked Sheryl into doing Domopalooza a few years back, he stayed in touch, and as such was able to get into conversations with her and Zuck about a Utah data center.
But, I think there is a larger story here, and that is that, if you can build data centers, you now have political clout. And the best recent example of this is not even Facebook, but Microsoft.
Microsoft en the cloud
With huge fanfare, the company recently announced an investment of $1.1 billion in Mexico.
Microsoft announces continued progress on its plan “Innovate for Mexico”, to contribute to the development of the country. The main pillar of the plan is focused on accelerating Mexico’s digital transformation through democratizing the access to technology. The company announced plans to establish a new cloud datacenter region in Mexico to deliver its intelligent and trusted cloud services to serve Mexico’s public entities, organizations and Mexican society, including Microsoft Azure, Office 365, Dynamics 365 and the Power Platform. This datacenter region is an important part of Microsoft’s $1.1 billion investment plan in Mexico over the next five years.
What’s the Normcore interpretation of this?
It takes a very long time to convince a country to build a data center, because you are, in essence, giving up some of your infrastructure capabilities and visibility to a private company. We’ve come a long way from the four nodes of ARPANet, which were managed by a government project and created in part with an eye to national security and American Department of Defense interests.
Now, data centers are private affairs, and if you don’t have a Google lanyard and your biometrics aren’t in the Facebook system, you can’t get in.
Just as social companies are black boxes for algorithms, their infrastructure is a black box for sovereign states, too. (Unless they’re peeking in through the backdoor.)
So what will jurisdictions ask for in order to give up control and host these data centers? For Mexico, it was all of this stuff:
The plan also includes a robust education and skilling program with different initiatives the first one being the creation of three laboratories and a virtual classroom, in collaboration with public universities to create an education platform for digital skills, to expand employability in future generations. The first initiative of the commitment to apply artificial intelligence to create societal impact is an investment in the project “Artificial Intelligence to Monitor Pelagic Sharks in the Mexican Pacific Ocean” (Shark ID), focused on the conservation of Mako shark species, driven by Mexico Azul, as part of the initiative AI for Earth, creating societal impact.
A supposed win for the government (unclear whether that AI Shark app will ever …surface), and an even sweeter deal for Microsoft, who’s now the first of the big three cloud providers company with a data center in Mexico.
Now, all that’s missing is the conferences and Matthew McConaughey, but I’m sure he (and Google and AWS) are all not far behind.
What I’m reading lately
I am really, really excited that I had a piece published in Increment, Stripe’s magazine, about software architecture.
On black boxes in AI theory (shout out to yours truly :)
Emily Oster, the Normcore of parenting, has a newsletter
The Newsletter:
This newsletter’s M.O. is takes on tech news that are rooted in humanism, nuance, context, rationality, and a little fun. It goes out once a week to free subscribers, and once more to paid subscribers. If you like it, forward it to friends and tell them to subscribe!
The Author:
I’m a data scientist. Most of my free time is spent wrangling a preschooler and a baby, reading, and writing bad tweets. Find out more here or follow me on Twitter.
Interesting as usual. I was shaking my head though: you told the history of computing in Utah (and with a networking slant at that) without mentioning Novell. Which is like telling the history of the PC without mentioning, I dunno, IBM? Sure they're no longer relevant and Netware wasn't the internet, but it seems like a funny omission.
Great email, thanks!