The last miles and miles of the vaccine
One web dev versus the System
Hey, your friendly neighborhood COVID CEO here, popping in with the latest news from my board room. Almost five years ago, I wrote about how tech, as an industry, doesn’t work on the hard problems, but instead builds snack-delivery startups. I wondered,
Why is it that we are so focused on “fixing the way we snack?” (or the way we get fresh flowers delivered, or the way we do laundry?) Why can’t we instead pool our energy and resources and instead of building monthly subscription boxes, social media apps, optimizing ad networks, tackle the health monster? What would it take to begin fixing at least a little bit of this broken, ugly thing?
A few years later, I wrote about how, again, AI won’t save healthcare.
Over the past few days, a couple of news items have come out that have made me realize that the answer is this: in the American healthcare industry, there’s no financial incentive to helping end users (patients), unless it also helps the institution taking care of them (hospitals, insurance companies, etc.)
The Developers
Every few weeks a news item comes out that makes me feel this truth viscerally again. Last week took the cake. I read a story about Olivia, a web dev who’s currently on maternity leave and has been, on her own, working on fixing the Massachusetts state vaccine sign-up process.
After her mother-in-law had difficulty signing up for a Covid-19 vaccine, a Massachusetts woman created a website to make it easier for her -- and she made it easier for everyone.
Olivia Adams built a website that pulls in vaccination appointments from across the state, including government sites as well as ones operated by private businesses. She called it macovidvaccines.com.
The 28-year-old software developer from Arlington, Massachusetts, says she spent three weeks and about 40 hours building the website -- and she did it while on maternity leave caring for her 2-month-old son, she told CNN's Alisyn Camerota on Monday.
What she did was build a website, hosted on AWS, that scrapes appointment data from a number of sources, including maimmunizations.org, pharmacies, and hospitals, none of which is centralized in any sort of way, and offers an aggregated list of appointments that people can sign up for immediately.
Adams has a script that runs every five minutes across about 20 different vaccine sites, she wrote in an email.
Right now, she is constantly fixing the site due to minor upstream site changes that break her scraper, aka the program that looks at the website and identifies which HTML data to surface on her website. She is also currently fielding interviews from mainstream media sources, raising money for hosting costs (donate here on her GoFundMe) and taking calls with Massachusset’s “Coronavirus Command Center.” (Don’t forget she has a newborn.)
I am hoping that the meeting was just Olivia asking “why the hell weren’t you guys doing this to begin with,” because I cannot imagine a scenario where a single woman who just gave birth has to do the job that an entire state government was tasked with.
Just when I thought I was already enraged (although to be fair, I do spend a good deal of my free time on Twitter so rage is my morning baseline emotion), I started reading more stories of one-off people so frustrated with the vaccine registration system that they took matters into their own hands. For example, there is this guy in New York:
Huge Ma, a 31-year-old software engineer for Airbnb, was stunned when he tried to make a coronavirus vaccine appointment for his mother in early January and saw that there were dozens of websites to check, each with its own sign-up protocol. The city and state appointment systems were completely distinct.
“There has to be a better way,” he said he remembered thinking.
So, he developed one. In less than two weeks, he launched TurboVax, a free website that compiles availability from the three main city and state New York vaccine systems and sends the information in real time to Twitter. It cost Mr. Ma less than $50 to build, yet it offers an easier way to spot appointments than the city and state’s official systems do.
And then, there is the vaccination Facebook group in my native Pennsylvania that a local doctor created, out of the same frustration. Here, members manually check sites and lets members know when appointments are available
Thousands of Pennsylvanians have joined a Facebook group called PA COVID Vaccine Match Maker, where members share advice on how to get phase 1A individuals vaccinated.
Exton Internal Medicine Doctor Christine Meyer created the Facebook group and said her office has received thousands of emails and hundreds of phone calls from people pleading to get the vaccine.
Meyer said the members help one another find vaccine appointments and share information such as when pharmacies have appointment openings.
"It's a crowdsourcing zone where people are just posting their tips and tricks for success," said Meyer.
Finally, out in California (well, more precisely Japan, because that’s where he lives,) Patrick McKenzie, better known to the tech internet at patio11, his HackerNews username, has been volunteering and creating, with a group of others, https://www.vaccinateca.com/, which basically has hundreds of volunteers that call different pharmacies and healthcare providers and ask them who has the vaccine,
Our professionals call medical professionals at hundreds of potential vaccination sites daily, asking them if they have the vaccine and if so to whom they will administer it to and how to get an appointment. We write down what they tell us, and publish it to this site.
How is any of this ok? Why do we have these people volunteering their time so that people can figure out where they can get vaccines? The more of these stories I read, the more grateful I am that there are good people in this world, and the angrier I get about the situation.
But I shouldn’t be angry. In fact, I should have zero surprise that this is working out this way. Because, really the problem is two-fold. Or at least, this is my working theory (remember, I am a humble MLE whose specialty is fixing typos in YAML files, not fixing American healthcare. )
One is that we do not have a strong national medical (or anything, really) infrastructure. We saw this when we were trying to ramp up manufacturing for face masks and ventilators. Remember ventilators? We are still short, and still not really doing anything about it. It’s just fortunate that early on we thought we would need them, but really we were mistaken about the nature of COVID.
The second subset of this problem is that, not only do we not build things well at the national level, the federal government and the state governments don’t coordinate well. And, all the last-mile work is left to the states,
And critically, public health experts say, federal officials have left many of the details of the final stage of the vaccine distribution process, such as scheduling and staffing, to overstretched local health officials and hospitals.
“We’ve taken the people with the least amount of resources and capacity and asked them to do the hardest part of the vaccination — which is actually getting the vaccines administered into people’s arms,” said Dr. Ashish Jha, the dean of Brown University’s School of Public Health.
This is already a really complex process with all the features of a project that has little chance of success in the beginning: high-pressure, high-visibility, thousands of actors that need to move in tandem, a highly-volatile vaccine that needs to be stored at precise temperatures and given in two doses, a country that runs mostly on a model of small local providers and commercial pharmacies like CVS (which have also not been doing so great, actually), and a delivery roll-out process that started over the major winter holidays during a pandemic that’s already seen record package delivery volumes.
Phew.
And then came the consultants.
The Consultants
I should be careful about what I mean here, because I used to be a consultant (sorry about all the Excel.) . What I mean here specifically is federal government contracts.
You may have heard that Deloitte was involved working with the CDC on the national vaccine strategy.
Her frustration is echoed by millions of Americans who have struggled to get vaccines through various chaotic systems. But unlike others in some states, she wasn’t encountering these problems with a third-party consumer service like Eventbrite, or even through an antiquated government system. She was on the US Centers for Disease Control and Prevention’s brand-new, $44 million website called VAMS—the Vaccine Administration Management System, built by the consulting firm Deloitte.
….
[E]arly in the pandemic, the CDC outlined the need for a system that could handle a mass vaccination campaign, once shots were approved. It wanted to streamline the whole thing: sign-ups, scheduling, inventory tracking, and immunization reporting.
In May, it gave the task to consulting company Deloitte, a huge federal contractor, with a $16 million no-bid contract to manage “covid-19 vaccine distribution and administration tracking.” In December, Deloitte snagged another $28 million for the project, again with no competition. The contract specifies that the award could go as high as $32 million, leaving taxpayers with a bill between $44 and $48 million.
Why was Deloitte awarded the project on a no-bid basis? The contracts claim the company was the only “responsible source” to build the tool.
And, who else is involved in all of this? Our good friends at Palantir. (If you’re not familiar with Palantir, their Wikipedia controversy page is a nice place to start.)
They’re involved further upstream, though, with the actual distribution that decides which state gets what.
The vaccine prioritization formulas fall roughly into three tiers: federal, state and local. At the top level, Operation Warp Speed — a multiagency federal effort, created by the Trump administration — has managed nationwide vaccine distribution through Tiberius, an online portal developed by Palantir, the data-mining giant. The Biden administration, which has retired the program’s name, has taken over and is continuing the effort.
States began warning about Tiberius’s drawbacks last fall. In interim vaccine plans filed with the C.D.C., some state health administrators complained that the platform seemed overly cumbersome and that the algorithm’s week-by-week allotments would make it difficult to plan monthslong vaccination campaigns.
This is just the public information I’m reading. I’m sure there are hundreds of other similar companies involved in the vaccine rollout, only to be bested literally by a single web developer with 40-50 hours on her hands.
Why do these companies, which are so bad at the work of actually coordinating vaccines, get this work? It’s as Dan says partially, in this thread:
Because their job is not to do this work. Their job is to get federal contracts. Or rather, that’s what the market incentivizes them to spend time on, because the more federal contracts you can get, the more you’ll be known as someone who can get and perform federal contracts, which means you’ll be asked to bid on more things.
Here’s how the process works. The government puts out an RFP, a request for proposal. The RFPs are often hundreds of pages and have lots of arcane rules you need to follow, like, “please provide 50 references from the last 100 projects you’ve done that can show that you know how to write APIs.” And on and on for hundreds of pages. A finished RFP response can take tens of people and months to put together. This is not a secret by the way, the federal government openly advertises that it works this way, and most states do this, as well. There are thousands of RFPs for all kinds of different services outstanding on any given day.
Here’s an example from my beloved Pennsylvania, for ATM machines on the Turnpike (our state’s toll road system.) Here is the actual RFP, and there is also a separate PDF of attachments. If you were brave enough to click through, you can see that there are pages of very specific things they want (“Provide a description of the proposed approach/methodology that you will follow in delivering, installing and operating ATMs at PTC locations as well as video advertisement procedures, and your experience with entering into a revenue sharing agreement. Include in this section the deliverables and reports to be provided, the project controls that will be used, and the tasks that will be performed”), and that if you answer all of the questions correctly in order to win the bid, it will take you a very long time.
If you understand how to operate the RFP system and play by the rules, you have an enormous advantage over companies that have never done this specific kind of paperwork and don’t have the staff to devote to this. Palantir and Deloitte have done hundreds of these before, so they easily slam dunked these proposals, and as a result they’re involved in this COVID vaccine work now.
Whether they’re actually delivering on it remains to be seen, but right now it’s all under a big question mark and getting mixed into the other things that are causing us not to vaccinate as many people as we should be.
But, the real tragedy here is this: You know who’s not good at getting the attention of federal and state agencies working on the vaccine rollouts? People who can actually make these websites and cut through this red tape. There is a very good phrase to explain this phenomenon, “No one’s ever gotten fired for buying Oracle (/IBM/Microsoft).”
No one’s ever gotten fired for hiring Big 4, even if it means vaccines are on the line.
And so, that’s where we are now. No federal-state coordination, a brand-new process coordinated in a country that prides itself on decentralization, and a process where speed is of the essence. And, on the other side of all of this, people who actually need the vaccine, waiting for the government machine to start churning faster. And in the middle, patio11 and Olivia, working as quickly as they can, alone.
I still believe we’re almost out of this thing. We are distributing vaccines faster than ever and we’re getting better every week. It’s still really bleak right now. But, we are just around the curve from we start to see the light.
How do we know? We are (just barely) starting to get to a low threshold of herd immunity. ( Sorry in advance for all those UTM parameters in the link and for putting you in abVariantId=1.)
Even still, I’m not going to be looking at any news about vaccine distribution for a while.
What I’m reading lately:
There is a TON of good content on why journalism is bad lately: Journalism and expertise,What Happens when the NYT makes mistakes, and the non-expert specialist
If you have a chance to read something Paul writes, do not say no. This is on the return to the office.
The Newsletter:
This newsletter’s M.O. is takes on tech news that are rooted in humanism, nuance, context, rationality, and a little fun. It goes out onceish a week. If you like it, forward it to friends!
The Author:
I’m a machine learning engineer. Most of my free time is spent wrangling a kindergartner and a toddler, reading, and writing bad tweets. Find out more here or follow me on Twitter.
I do agree that healthcare industry incentives are super backward and the way that contracts are rewarded is inefficient and borderline corrupt, but I do flinch whenever I read a story about a lone developer building their own version of a public utility. It gives me flashbacks to when every data scientist was publishing their own covid dashboard last year. It sounds like in this particular case, the dev did reach out to the state gov which is a good start but the cynic in me does wonder whether building a new layer on top of a broken system doesn't open up more confusion in the future. https://xkcd.com/927/
This is a great piece Vicki. Despite the apparent different incentives, the system in the UK works in a very similar way. In fairness, vaccine appointments have been handled reasonably well. But in general the same problems apply. Difference here is that a lone individual would never be allowed to launch a system in the way Olivia did. It would be stamped and never see the light of day. (Full disclosure: I used to be a consulting partner in a big 4 firm - lots of powerpoint and bullshit in my own cv.)