DANIEL VAUGHAN: COVID-19 models aren’t right or wrong, they’re just less uncertain

There’s a growing debate over the accuracy of epidemiological models used to form public policy in the face of the COVID-19 pandemic. It’s clear from all the commentary on social media and cable television that no one understands how statistical modeling works.

Mathematical models don’t tell you what is going to happen. They describe uncertainty. And for most of this pandemic, we’ve had a lot of uncertainty.

As I said several weeks ago, models are built on two things:  “1) the data fed into the model, and 2) the assumptions the model makes about that data in the future.” The overarching problem we’ve had for most of this pandemic is bad data, or rather, not enough data.

When I wrote a column on March 13 arguing that the United States was flying blind, it was because we had run just under 16,000 total COVID-19 tests in the entire country. We were barely hitting 1,000 tests run per day. The United States was behind almost every other country in the world in this regard. It was around this time that we were also getting the first media stories about models.

The New York Times reported on March 13, 2020, that the Centers for Disease Control and Prevention (CDC) and other government agencies were working on their first models of the coronavirus. The predictions were dire:

160 million and 214 million people in the United States could be infected over the course of the epidemic, according to a projection that encompasses the range of the four scenarios. That could last months or even over a year, with infections concentrated in shorter periods, staggered across time in different communities, experts said. As many as 200,000 to 1.7 million people could die.

At the time, all we had to go on were two data sets: China’s numbers and early reports out of Italy. Furthermore, the primary assumption in those models was that the United States would take few to no mitigation efforts; that is, we wouldn’t enact any social distancing measures to flatten the curve.

We’ve since learned that China was lying through its teeth. The United States has taken drastic mitigation steps, and our capacity to test has shot past every single country in the world.

On March 13, we had only run around 16,000 tests. On April 13, we’ve tested more than 2.6 million people, and we’re racing toward three million. No other country, on this date, has tested even one million of its citizens.

What the United States economy and health care industry have done in less than a month is nothing short of miraculous. We’ve erased the testing deficit and are moving toward ways of tracing the infected to keep them away from the healthy.

It’s an astonishing turnaround that no other country seems capable of reproducing — least of all China, who the World Health Organization holds out, bizarrely, as the gold standard.

Models measure uncertainty. A death range of between 200,000 and 1.7 million is huge. We shouldn’t blame epidemiologists; they were doing the best they could with the data and policies they had in front of them.

But what that does tell us now is that the current IHME models, relied upon by the White House, are vastly more predictive than any preceding model. That’s because 1) they have an extensive hard data set, and 2) they know the specific mitigation policies in place in every state.

Current models show a total death toll of a little over 60,000 people by August. That’s still far too many people, and China should answer for those deaths. But it’s also a much rosier picture than the one that was painted a month ago. It speaks to the power of widespread testing and everyone moving together, as one culture, to prevent the spread of the virus across the nation.

For the most part, no one in the media or public understands models. In the aftermath of the 2016 election, Nate Silver blasted members of the press for understating Donald Trump’s ability to win, despite his model showing a clear path forward. FiveThirtyEight wrote early on on the pandemic that all the models were full of uncertainty over COVID-19, and that posed a definite problem for public policymakers.

The models gave the best guess and showed us a broad range of possibilities. It’s not that they were right or wrong — it’s that we didn’t have useful data or policies in place to combat any virus. Now we do. Policymakers now face the prospect of reopening the economy, and many questions are left outstanding.

Can people get reinfected by the virus? Is this strain of coronavirus susceptible to heat and humidity? Can the United States get a valid form of test-and-trace up and running? Do we have workable ways of treatment to combat the symptoms of this disease?

There are still many unknowns surrounding this virus, but the American health care system is working day and night to remove the uncertainty. Our models are getting better at forecasting what we face in the coming days. We’re getting answers for some questions while opening up new ones.

While the path forward isn’t entirely clear, we do have better data now, and an ever-expanding toolkit to combat the virus. We’ve come a long way in a month. We’re going to go even further in the coming months. Nothing is static in the fight against COVID-19, which should give everyone some hope for brighter days ahead.