Diagnostic Lab Tests – Results May Vary???

If you’re wondering why Theranos’ results are yet the subject of debate, let’s start with three questions:

  • Does Elizabeth Holmes’ technology actually work (i.e. where is the science / proof)?
  • Does Theranos still hold the promise of a $9 billion valuation?
  • What does this mean for patients and clinicians?

A study conducted by researchers at the Icahn School of Medicine at Mount Sinai was recently published in the Journal of Clinical Investigation. FINALLY – some scientific data! – comparing Theranos test results with those of other labs (and revealing results varied widely.)

Why the variation? In a previous article, In Sickness and Wealth explored the reasons diagnostic lab tests can show variations and why that doesn’t necessarily mean they are wrong. Different methods, units of measure, and specimen types can all impact how results compare from lab to lab. Additionally, statistics and quality control tell the story of how well a specific test performs to an expected value, revealing its accuracy and precision, both necessary to produce a reliable test system. But even then, this doesn’t mean it’s a valid diagnostic test – because scientific data alone does not necessarily equate (or translate) to clinical utility – and there needs to be a way to scrutinize technologies and how they marry the science and research to medical relevance and utility.

Regulatory issues. Diagnostic lab tests (development, production/manufacture, and performance) are regulated by the Clinical Laboratory Improvement Act (CLIA) and its trifecta of FDA, CMS, and the CDC. Most people are probably aware of these agencies, but perhaps not within the context of laboratory testing. FDA regulates tests and devices, CMS conducts inspections and monitors how labs perform with proficiency testing, and the CDC promotes public health and disease prevention. Every laboratory performing human diagnostic lab testing is required to have an appropriate CLIA certificate, and CLIA invokes these requirements to safeguard patients from harm.

Theranos’ tests were not subject to FDA approval (though they were voluntarily submitted for reasons I cover in another blog post), but their lab, personnel, and processes were subject to CMS inspection. While Theranos’ marketing lists numerous tests they can perform off a few drops of blood, and the technology isn’t completely a lie, Theranos has sorely misrepresented what they have thus far accomplished: Theranos has succeeded at using their Nanotainer to collect specimens for their HSV (herpes simplex virus) that has been FDA-approved for testing. That’s it. There are other tests they have submitted for FDA approval, as well as the Nanotainer (for general purpose clearance as a medical device for capillary specimen collection); but none other than HSV have yet been approved.

Yet what caught Theranos off-guard, was their Nanotainer collection container being deemed a medical device that DID have to be approved (and as noted, is still pending with the FDA). This was a huge set-back that deterred their plans to use the small capillary samples in their Nanotainers to perform the necessary correlations – a beautiful plan to conduct simple correlations between Theranos’ Edison analyzer and Siemens’ (while simultaneously launching a revenue stream from the results obtained on the Siemens analyzer). This complicated matters since capillary samples would have to be collected in an FDA-approved container to correlate the containers as well as the tests! Ditto for correlations between these capillary specimens and venous blood from standard blood collection tubes.

Another problem was the CMS inspection of their Newark, CA laboratory when it revealed their lab director and personnel did not possess the required credentials, and found deficiencies with quality control and proficiency testing. It’s understandable Theranos would have difficulty with proficiency testing due to their proprietary methods (no one else is using their technology) making comparison groups impossible, so correlations would have to be made. It’s this scientific data we’ve all been wanting to see… that came to light with the CMS inspection and did not show favorably.) But Theranos also had issues with quality control for tests performed on non-proprietary analyzers – notably, for coagulation testing that CMS cited as an “immediate jeopardy” concern.

Conflicting Values. Now with the publication of the Icahn study, we see another conflict of values: wide variation in the results Theranos obtained versus those reported by Quest and Labcorp. With evidence Theranos did not comply with CMS standards, it’s difficult to retain respect for their mission “to make actionable health information accessible at the time it matters.” It’s ironic that this mantra produced results that while indeed actionable, were also erroneous and deemed to present “immediate jeopardy” [deficiency per CMS inspection] to patient care due to the potential for inappropriate treatment and possible death. The impact of Theranos’ response – the way they failed to react to these implications – cannot be ignored. Speculation has already been made to whether this negligence will result in a class action lawsuit.

Along with these risks, the ramifications to Theranos’ business relationships are already evident. And while the FDA issues are closely tied to business concerns – their menu is a list of complicated tests which take time to develop and receive approval – all medical manufacturers know there’s no predicting FDA’s timeline. It’s understandable that using another vendor’s technology to validate the new technology would be necessary. And it’s even plausible Theranos thought correlations across collection devices and specimen types would be possible to produce the data needed to prove Theranos technology – at the same time generating revenue. But with challenges to do so (at least at their Newark, CA lab), Theranos faced transportation of tests to its other lab location (Arizona), or purchase of testing from a reference laboratory like ARUP. Given the transparent pricing Theranos has publicized (at costs equal to CMS reimbursement rates), this has a tremendous impact on their revenue and expenses, with the cost of every test purchased at a multiple of what Theranos incurs to internally perform the test, thus obliterating their profit margin while drastically increasing their costs.

As the scrutiny of Theranos’ results has prompted attention (to a lesser degree than the hype that landed Ms. Holmes on the covers of Fortune and Forbes), it begs the question, “what is Theranos’ current valuation, and how long can they survive to turn things around before investors bail?” While Theranos has no obligation to public stockholders, they do have a commitment to their investors and patients. Where has this left the venture capitalists of this magic unicorn, once valued at over $9B? Why have they remained silent? And are they culpable for failing to conduct due diligence? Perhaps they neglected to do so, or perhaps they didn’t know the right questions to ask where science intersects with business. (It’s the very problem that prompted the creation of this publication – to provide insights from both perspectives!)

To be fair, there was little information available to evaluate Theranos’ technology. Proprietary trade secrets were fiercely protected and though transparency in pricing and support for FDA regulation were often cited, scientific data on proficiency testing, correlations, or peer-reviewed data was sparse. It seemed even investors had little insight into how the technology worked, few had seen the laboratory analyzer, and financials were vague. The focus instead, was on the prestigious roster of directors, armed with Elizabeth’s passion “to make actionable health information accessible at the time it matters” through her business plan targeting every aspect of access, from fear of needles to affordable prices, and proximity of care to obtaining a physician order. Goals included Nanotainer finger sticks, transparent pricing at CMS reimbursement rates, geographical locations with their high-profile Walgreens partnership placing Theranos wellness centers at every “corner of healthy and happy,” and successful sponsorship of legislation to secure open access to laboratory testing without the requirement for a physician’s order (i.e. retail lab testing). And Liz was featured on the covers of Fortune, Forbes, Inc. and Bloomberg Businessweek. Regardless, technology is not enough. Demand for a product is not sufficient. And personal passion will not ensure success. Where was the proof Liz had a product people would want and could pay for? Or that it would even work? Did everyone assume if Walgreens was interested, THEY had done this research for everyone? And where were the experts to vet the technology? Market and regulatory forces cannot be ignored. Evidence is required. Especially where investors expect returns.

So now the unicorn is in the crosshairs, and the reputation of a young millionaire whose brilliant innovation was often compared to that of Steve Jobs, is being threatened. Perhaps when your company is valued at $9 billion, and you’re enchanted by the excitement surrounding your dream, it’s impossible to see that your technology doesn’t deliver what you promised it would – that your science isn’t yet where you thought it would be. Maybe it’s difficult to tell your investors your business plan depended on a timeline the FDA isn’t respecting and the partnership you were counting on to deploy your technology is faltering (something Safeway already made public and where Walgreens seems to be following), or that you’re now dependent on a competitor’s technology to keep up the ruse until you have figured things out. Theranos could have survived these challenges, had it complied with at least the CMS standards under CLIA requirements. But as we know, they did not.

Demanding accountability. As a Medical Laboratory Scientist, I’m reminded of two important lessons from my internship that Theranos should heed:

  • Sometimes “no result” is better than a wrong result.
  • If you make a mistake, own up to it (QUICKLY!) and notify the clinician – patients’ lives depend on the results YOU report!

“Results may vary” for laboratory testing, public opinion, and company valuation – but whatever those results, where patients and investors are concerned, there exists a moral and ethical obligation to prove your worth as you manage valuable assets and resources – like investors’ money and patients’ lives.

For those wishing to invest in the clinical laboratory space… while Theranos is not yet a publicly traded company (and might choose to never go public), Labcorp (a member of the In Sickness and Wealth Portfolio) and Quest Diagnostics are both publicly traded companies.  Both will be covered in a future issue of In Sickness and Wealth.

This article has 2 comments

  1. Will

    Greetings from Florida! I’m bored to death at work so I decided to check out your blog on my iphone during lunch break. I love the knowledge you provide here and can’t wait to take a look when I get home. I’m surprised at how quick your blog loaded on my phone .. I’m not even using WIFI, just 3G ..Anyhow, great site!

  2. Dave Lerman

    Will….thanks for the feedback. Much appreciated. We are just getting started trying to build this business and gain followers and members. We will continue to put out quality information on investing in the health care sector. I invite you to check out the latest issue on the Diabetes Quadropoly where we take a look at potential investments in the leading companies helping patients conquer this illness.

Leave a Reply