2c:null 2d:["$","title",null,{"children":"Improving the Accuracy of COVID-19 Tests | Blake Elias | Observable"}] 32:T3b65,md` ## Abstract Current testing methods for COVID-19 are flawed. The test results come with a high degree of uncertainty: a patient who is told they do not have COVID-19 may in fact be in the early stages of an infection. We present a method for improving the certainty in a patient's diagnosis, given testing methods that have limited accuracy. Additionally, we spell out the need for taking greater precaution with patients whose diagnosis remains uncertain. These recommendations have subsequently been endorsed in a [more formal venue](https://www.medrxiv.org/content/10.1101/2020.05.27.20114736v1) by several colleagues (from that paper's simulation results, "The reduction in cases after 60 days with widespread use of CT-scan screening compared to PCR by itself is as high as 50x"). ## Introduction We begin by defining several terms: A **test** is a procedure which can be performed on a patient, which aims to determine whether that patient has a disease or not. A test produces a **result**, which can be **positive** or **negative**. We refer to the result as a: * **true positive (TP)** if the result is positive, and the patient has the disease. * **true negative (TN)** if the result is negative and the patient does not have the disease. * **false positive (FP) ** if the result is positive, but the patient does *not* have the disease. * **false negative (FN)** if the result is negative, but the patient *does* have the disease. This is summarized in the table below: ||Has disease| Does not have disease| |--|--|--| |**Test positive**| True Positive (TP) | False Positive (FP) | |**Test negative**| False Negative (FN) | True Negative (TN) |

## Current Test Methods Suppose we're at a lab, processing the masses of PCR tests being done right now for COVID-19. Take 1000 tests, from 1000 different people (1 test per person). Each will fall into one of the four buckets above. **Pop quiz for medical professionals:** Of our 1000 tests, how many will go into each bucket? What numbers go in each box below?

||Has COVID-19| Does not have COVID-19| |--|--|--| |**Test positive**| ?? | ?? | |**Test negative**| ?? | ?? |

One thing we know is about 20% of the tests conducted in the US are coming back positive (which is a sign that we're not doing enough testing...). But given that statistic, here's one way the chart could look: ||Has COVID-19| Does not have COVID-19| |--|--|--| |**Test positive**| 200 | 0 | |**Test negative**| 0 | 800 |

That's what we'd *want* to see: perfect correspondence between the test result and the patient's actual condition. Here's what we *don't* want: ||Has COVID-19| Does not have COVID-19| |--|--|--| |**Test positive**| 100 | 100 | |**Test negative**| 400 | 400 |

That would be the worst possible arrangement. There, the results are basically random. It's as if the lab facility threw away all the swabs and flipped a coin for each patient to determine their result (with a coin that comes up heads 20% of the time, and tails the other 80%). A test like that would be completely useless. We want to base decisions off these test results. Should we isolate this patient? Can they go back to work? Such decisions weigh heavily on whether we can trust the test results in the first place. So, I'm asking you -- the medical professional, the policy-maker, the informed citizen -- to put a number on how much you trust these results. What numbers go in those bottom-left and top-right boxes? Take one more second. Jot down your answer, then scroll down...

In a group of 1000 people, here are the buckets they'll roughly fall into when we do a PCR test: ||Has COVID-19| Does not have COVID-19| |--|--|--| |**Test positive**| 117 | 83 | |**Test negative**| 50 | 750 |
Consider what this means. We have 200 people testing positive, 800 testing negative (the same 20% positive tests I mentioned before). The test results are a bit skewed, though, so 167 people actually have COVID (not 200) -- and 833 don't. Of the 167 who have COVID, 50 of them tested negative. This fraction, 50 / 167, or 30%, is called the **false-negative rate**. Think about that: **30% of those who have COVID are testing negative**.¹ Doesn't it seem irresponsible to tell these people they're not infected? Or to put them in a COVID-free ward in a hospital? Many US hospitals continue to send patients to a COVID-free ward after a single negative test. Those patients are considered negative, no longer on "contact precautions" -- but then have gone on to have severe pulmonary crisis, been transferred to the ICU and then tested positive. This is supported by literature which documents the unacceptable false-negative rate. And yet, practices in many US hospitals have not changed. Any patient coming to a hospital right now with respiratory issues should be considered "positive until proven negative" -- but right now, it is almost the opposite. Why is this happening? Do the staff setting policy in hospitals understand these numbers? If you work in a hospital where a single negative test is enough to send a patient to the COVID-free ward: have you asked why this is the policy? Do you think the person who set that policy could fill out the numbers in the above chart? Are they acting responsibly, given the level of uncertainty in these tests? ## Reducing Uncertainty We have seen that PCR tests alone are not sufficient for determining a patient's status with certainty. How can we do better? One answer is to use another type of test, besides PCR. CT scans have also been used to diagnose COVID-19, and have been widely used for this purpose in countries other than the US. With a CT scan, those same 1000 people we tested would look more like this: ||Has COVID-19| Does not have COVID-19| |--|--|--| |**Test positive via CT**| 150 | 35 | |**Test negative via CT**| 17 | 798 |
What's more, the analysis of the CT scan images can be done via computer vision ([see here](https://pubs.rsna.org/doi/full/10.1148/radiol.2020200905)). We get fewer errors there: both in false positives and false negatives. Here, of all those who have COVID-19, only 10.2% will test negative. We say the **false-negative rate** is 10.2%. Compare this to 30% with PCR. Why aren't we using CT scans more then? Indeed, other countries have... So, we have two tests, with two different false-negative rates. Which one should we use? Which should we trust more? It turns out, from the laws of statistics, we don't have to pick one test or the other. We can use **both** tests, and combine the results. Suppose we take those same 1000 patients, but this time, test them with both PCR **and** CT. If **either** test comes up positive, we declare the patient positive; otherwise, we declare them negative. Assuming that the two tests make mistakes independently, our 1000 patients will be diagnosed something like this: ||Has COVID-19| Does not have COVID-19| |--|--|--| |**Test positive**| 162 | 115 | |**Test negative**| 5 | 718 |
Now, **from those who have COVID-19, only 3% will test negative** (as compared to 30% with PCR alone, or 10% with CT alone). We have decreased the false-negatives significantly -- at the expense of taking on a few more false-positives. In this case, the trade-off is well worth it, as false-negatives are the far more dangerous mistake to make than false-positives. ## Discussion We assumed that PCR fails to detect 30% of the infected patients. What if the test manufacturers can lower this false-negative rate to 5%? That's much better -- but again, combining with a CT scan can do better. The combined false-negative rate would be 0.5%. If the PCR test had a 1% false-negative rate, the combined test would have a 0.1% false-negative rate. However much we improve the accuracy of the individual tests, we will always do better by combining them. With the large numbers of people we will soon need to test, even missing 0.5% of infected people could be significant -- so pushing this number down as far as possible is important. False negatives remain an important problem in hospitals, and many hospitals are not relying on the test results at all - they are relying on clinical symptoms instead. We have assumed that the two tests make mistakes independently. This won't be perfectly true. And we can correct for any amount of correlation in mistakes, when we calculate the final percentage of errors. But, this is a realistic enough assumption that the results here give a rough flavor -- combining two tests will confer some advantage over using either test alone. In general, the strategy for diagnosis will be more complex, involving many observations of a patient's symptoms, other tests such as viral rule-out panels, contact history, and more. And PCR tests may be performed multiple times, rather than just once, to increase confidence in a negative result the more times it is repeated. ## Conclusion Any patient coming to a hospital must be assumed to have COVID, until confirmed that they *don't*. This "positive until proven negative" standard must apply from the moment they enter the hospital, through the time they are tested and waiting for results. Even after a patient has received some negative results (e.g. from a single PCR test), they should still be presumed positive until a preponderence of evidence has proven otherwise. Simple statistical calculations can be used to determine the level of certainty of a patient's diagnosis, given the evidence collected. Until a patient is confirmed negative beyond reasonable doubt, they should be isolated, wearing a mask, and only treated by healthcare workers with adequate protective gear. The risk that they have COVID is simply too great to do anything else. ## Footnotes ¹ 30% is a rough number -- it can be higher, especially for patients early in their infection. (See https://www.medrxiv.org/content/10.1101/2020.04.07.20051474v1). Many new kits and machines for testing have come onto the market, which claim lower false-negative rates. But, this has not yet been validated in any peer-reviewed study that we are aware of. Hence, many hospitals are still using the standard RT-PCR test, as their reliability has been well-studied (even if it is imperfect), and the procedure for performing these tests is standardized. ` /* ¹ https://www.medrxiv.org/content/10.1101/2020.04.07.20051474v1
² https://www.the-scientist.com/news-opinion/false-negatives-in-quick-covid-19-test-near-15-percent-study-67451
³ Any patient coming to a hospital must be assumed to have COVID, until confirmed that they *don't*. This applies from the moment they enter the hospital, through the time they are tested and waiting for results, and even after they receive a negative result. They should be isolated, wearing a mask, and only treated by healthcare workers with adequate protective gear. The risk that they have COVID is simply too great to do anything else. A negative result is not even enough to If all 1000 people are either in the top-left or bottom-right, then we can have full trust in the results. In reality, this does not happen: some people are in the top right (false-positives), and some are in the bottom left (false-negatives). If it's just a very few, that might be OK. But if there. , go to work (especially if they're a healthcare / other essential worker), etc. If the test is wrong too often, then we can't base these decisions off them. All mistakes are not created equal: false negatives are worse than false positives. With a false negative, we run the risk of telling the person they don't have COVID-19, letting them go outside, letting them sleep in a non-isolation room in the hospital, etc., and then having them infect others because they were actually infected themselves the whole time. One may ask how often each of these cases happens, in the case of COVID-19. How many test results will fall into each bucket? It turns out this will depend on what type of test is used. In this section, we'll consider PCR tests, which are the most common. (This is what everyone is getting swabbed with, this is what we don't have enough lab reagents to test everyone with, etc. When people refer to "getting tested" for COVID, they are almost always talking about a PCR test.) We'd *like* to always be in the top-left (true positive) or bottom-right (true negative). That means our test is doing its job well: from seeing the test result, we'd know with certainty who has the disease and who doesn't. On the other hand, if we fall into the top-right or bottom-left too often (false positives or false negatives), then the test is not 100% reliable. We can't put complete faith in what it tells us. ||Has COVID-19| Does not have COVID-19| |--|--|--| |**Test positive**| x | y | 0.2 |**Test negative**| z | w | 0.8 ||Has COVID-19| Does not have COVID-19| |--|--|--| |**Test positive**| 7/60 | 1/12 | 0.2 |**Test negative**| 1/20 | 3/4 | 0.8 x + y = 0.2 z + w = 0.8 z / (x + z) = 0.3 ==> z = 0.3x + 0.3z ==> .7z = .3x ==> z = 3/7 x y / (y + w) = 0.1 ==> y = .1y + .1w ==> .9y = .1w ==> w = 9y x + y = 0.2 3/7 x + 9y = 0.8 -9(x + y = 0.2) 3/7 x + 9y = 0.8 (3/7-9)x = .8 - 9*.2 (3 - 63)/7 x = .8 - 1.8 -60/7 x = -1 x = 7/60 y = 0.2 - x = 0.2 - 7/60 = 5/60 = 1/12 z = 3/7 x = 3/7 * 7/60 = 3/60 = 1/20 w = 9y = 9/12 = 3/4 */ /* Isolating people comes at a cost -- especially if that person is fulfilling an essential job. So we don't want to isolate someone if it's completely un-necessary -- but it's fine to isolate someone who *might* be infected, even if we're not fully sure, just to be on the safe side. If someone tests negative, What we'd like to know is: can we actually */ /* Some of those tests will come back positive, some negative. Some of those positives will be true positives (actually having COVID-19), some will be false positives (not having COVID-19, despite having a positive test result). Likewise for the negatives. */ /* A positive result is intended to indicate that the patient has the disease, while a negative result is intended to indicate that the patient does not have the disease. */ // One of the sad stories that continues to occur in hospitals is that patients get a single negative PCR test and are then sent to a non-isolation ward. I have heard this story from nurses over the last few weeks that they are then essentially disregarded even though they may have reason to be hospitalized for respiratory issues. They are considered negative, no longer on “contact precautions” and then have gone on to have severe pulmonary crisis, transferred to ICU and then test positive. Clearly the literature supports this due to unacceptable false negative rate of RTPCR. The paper out of China shows that some people needed six tests before coming up positive. Yet practices in many US hospitals have not changed. They’re not routinely doing CT scans in addition to these PCR tests. `33:[] 30:{"id":"76ad726926fc25a9","slug":"improving-the-accuracy-of-covid-19-tests","title":"Improving the Accuracy of COVID-19 Tests","thumbnail":"f148e391601e498be37009759741c10c38392d07e0822d051320fe4b02650ca8","access_level":"public","create_time":"$D2020-05-07T17:37:34.469Z","update_time":"$D2020-10-10T03:29:01.182Z","view_time":null,"first_public_time":"$D2020-05-09T06:10:05.243Z","has_unpublished_edits":null,"fork":false,"stars":0,"starred":false,"owner":{"type":"team","id":"614157e65006a2d4","login":"blakeelias","name":"Blake Elias","avatar_url":"https://avatars-next.observableusercontent.com/bcb2d8974dfe29de8246d628c79d40f80dd37cb59dc6d869a9030f5158e1b5ca","tier":"starter_2024","bio":""},"version":78,"publish_version":78,"role":null,"authored":false,"authors":[{"type":"individual","id":"01ac8a6a43bd3e70","login":"blakeelias","name":"Blake Elias","avatar_url":"https://avatars-next.observableusercontent.com/bcb2d8974dfe29de8246d628c79d40f80dd37cb59dc6d869a9030f5158e1b5ca","tier":null,"bio":""}],"members":[{"type":"individual","id":"01ac8a6a43bd3e70","login":"blakeelias","name":"Blake Elias","avatar_url":"https://avatars-next.observableusercontent.com/bcb2d8974dfe29de8246d628c79d40f80dd37cb59dc6d869a9030f5158e1b5ca","tier":null,"role":"editor"}],"comments":[],"custom_thumbnail":false,"trashed":false,"lapsed":null,"subscribed":null,"forks":0,"collection":null,"type":"notebook","body":{"titling":"auto","title":"Improving the Accuracy of COVID-19 Tests","cells":[{"id":0,"value":"md`# Improving the Accuracy of COVID-19 Tests\nMay 9, 2020\n`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"},{"id":7,"value":"$32","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","providerMetadata":"$undefined","attachments":"$undefined","output":"$undefined","database":"$undefined"}],"files":"$W33","stdlib":"1"}} 31:"$30:body"