The best way to innovate is to look at things from different perspectives… in different ways. That’s what Elizabeth Holmes did, with the promise to disrupt and improve lab testing. Patients fearful of needles would seek out one of her Theranos centers at a nearby Walgreens and submit to a simple finger prick for one (or some) of approximately 200 different tests. To further improve access, prices would mirror Medicare’s reimbursement rates making testing affordable for all, and patients could request their own tests (at least in Arizona where physicians’ orders are no longer required for testing, thanks in part to Theranos’ promotion of legislation that was approved in July of 2015). Elizabeth Holmes and Theranos would completely upend the lab industry.
The disruptive nature of what Ms. Holmes proposed – and her passion to achieve it – was the beautiful appeal. The technology was innovative. The business model was smart. The board was stellar. And the CEO was brilliant. Forbes and The Wall Street Journal agreed, and the company was valued at more than $9 Billion. Elizabeth was essentially “changing the rules” with technology and a business model that circumvented the customary confines of typical clinical/medical laboratories. Or so it seemed. So what happened? One first has to understand the lab testing industry, and therein is the problem – it’s complicated. It includes science and technology, level of test complexity, equipment and analyzers, qualifications and credentials of testing personnel, and a whole host of regulatory requirements from CLIA, OSHA, FDA, CMS, CAP, and TJC. Cutting through this maze of acronyms (or exemption from them) truly equates to a disruptive delivery model indeed! Let’s get started.
The first step for a lab to perform diagnostic testing on human specimens is to obtain a CLIA certificate. CLIA stands for Clinical Laboratory Improvement Amendments and was formed to establish regulation of laboratory testing. It includes the Federal Drug Administration (FDA), the Center for Medicare and Medicaid Services (CMS), and the Center for Disease Control (CDC). It requires clinical labs to be certified by their state and the CMS, according to the types of testing the laboratory will perform.
- The FDA determines the complexity level of tests and categorizes them as waived, moderate, or high complexity tests. This level is based on a scorecard of several criteria like how simple the test is to perform (specimen and reagent/material preparation as well as operational steps that require sequence and timing); what equipment, reagents, and supplies are used (requiring maintenance, calibration, quality control, proficiency testing, and troubleshooting); and the knowledge of personnel performing the test (education, training, and experience of medical laboratory scientists or non-lab personnel) to make interpretation and judgement; as well as the likelihood of errors with performing the test, and the potential impact on patients should those errors occur.
- Waived tests include glucose meters for home monitoring, over-the-counter tests for pregnancy, and rapid kits for Strep throat. These are tests that can be easily performed, in very few steps, by someone following simple instructions.
- Moderate complexity tests are many of the common chemistries performed on automated analyzers to check electrolytes, proteins, enzymes, or hormone levels. These require quality control on the analyzers, annual competency checks for personnel, and proficiency testing (analyzing specimens of “unknown” values which are tested and graded for accuracy) to verify the lab’s procedures and systems are functioning appropriately to produce accurate results.
- High complexity tests require more sophisticated testing methods and equipment, as well as specialized education and training. These tests include compatibility testing and antibody identification for blood banking, toxicology, and many genetic tests like sequencing. Additionally, some of these tests require interpretation by a PhD and/or Pathologist (MD or DO licensed and credentialed in anatomic or/or clinical pathology). Laboratory Developed Tests (LDT’s) also fall into this category.
- The CMS is responsible for issuing CLIA certificates based on the complexity level of testing the lab will perform. It publishes CLIA rules and regulations, conducts inspections, enforces regulatory compliance, and determines which private accreditation organizations (e.g. The Joint Commission (TJC) and the College for American Pathology (CAP)) are approved to perform inspections as well. Regardless of the CLIA certification level, each lab must designate a qualified laboratory director, responsible for oversight of all testing performed at that testing location; and each physical location must have its own CLIA certificate. Additionally, CMS monitors how each laboratory performs with Proficiency Testing (PT) results. If scores indicate a laboratory has failed multiple “unknowns” for a specific test, the lab will no longer be allowed to perform that test.
There’s nothing unusual or different about Theranos with regard to these requirements. Theranos is a CLIA-certified laboratory, and possess two certificates – for testing locations in Newark, CA and Scottsdale, AZ. So let’s move on to how labs acquire the tests they perform.
Decision to Make or Buy
- If they make their tests, these tests are considered Laboratory Developed Tests (LDT’s) – types of in vitro diagnostic tests (IVD’s) that are designed, manufactured and used within a single laboratory – which means they are not regulated in quite the same way the FDA regulates typical IVD’s – diagnostic devices designed or manufactured completely, or partly, outside the lab that offers and uses them. However, CLIA requirements for the lab performing the test still guide the oversight of these tests. This has been a point of contention between industry leaders: the device manufacturers like the companies that produce and manufacture tests for sale, and the research and academic labs that develop their own tests/methods for in-house use. Companies are subject to FDA manufacturing requirements and device taxes while labs may not be. This has historically given labs the advantage of researching and developing proprietary testing not subject to the same requirements as manufacturers (as long as the lab performs the test only in their own lab and does not sell or distribute it to others to perform the test). Over time, it has evolved from the practice of developing testing for internal need (e.g. in large academic hospital labs for patients with rare diseases) to labs as their own business entities within these settings, and with patented intellectual property to perform proprietary testing on specimens referred to them, from around the world. Clinical and business arguments exist on both sides, for whether FDA regulation of LTD’s would level the playing field and protect patients by subjecting labs to the same vigorous approval process that companies must complete, or if it would stifle research and innovation by affording unfair competition to those with the deepest pockets.
- If a lab buys, there are several companies who produce tests: both the consumables (kits and reagents/chemicals/supplies) as well as the equipment (analyzers used for testing). As noted, these are highly regulated by the FDA to ensure quality and consistency of manufacturing (raw goods and production processes) and storage (shelf life) for performance and accuracy. This IVD market of commercially available products is the typical model for most laboratory testing. Tests are manufactured by the likes of Abbott, Beckman, Ortho, Roche, Siemens, et.al., with pre-market review by the FDA to thoroughly evaluate the efficacy of these tests before labs purchase them to perform in their own facilities.
CLIA, CMS, AND FDA
Overall, CLIA addresses the laboratory’s testing processes (ability to perform testing that is consistently reliable and accurate).
However, CMS and accreditors’ inspections don’t evaluate the validity of tests (including LDT’s) prior to their use, or that the tests deliver accurate results to identify, measure, or predict the presence or absence of specific clinical conditions. That is left to the FDA – to assure both this clinical validity and the analytical validity (specificity, sensitivity, accuracy and precision) – are addressed through its premarket clearance or approval process, and that the FDA’s safeguards ensure appropriate design, manufacture, and safety and effectiveness of devices.
CLIA oversight is important, but because LDT’s are not as vigorously evaluated by the FDA, it does not ensure that LDTs meet the same requirements as other IVD’s.
Determining what is appropriate goes beyond testing accuracy though, and is part of the nuance between CMS and FDA oversight. The CMS regulates the lab while the FDA’s focus is on the medical device or IVD. Here is where it gets interesting with Theranos… and also murky. Theranos makes their own tests, collection device (Nanotainer) and equipment (Edison). So they are essentially regulated by CMS, but with regard to their own LDT’s are not obligated to pursue FDA approval, yet they have. Why? Hold that thought for a moment as we explain a couple other requirements.
Proficiency, Variables, and Correlations
The CMS and the FDA, combined with the CDC and laboratory best practices, are responsible for CLIA and its overarching goal of regulating laboratory testing. An aspect of that is evaluating the efficacy of each test. Does it deliver the utility of what it claims to provide for diagnostic or screening capabilities that will aid the clinician? Can it do so with a level of precision and accuracy that minimizes false-positive or false-negative results? What is the chance of error based on testing equipment and personnel – ease of use, potential errors, and the impact of erroneous results? Answers to these questions will determine laboratory certifications and inspections, and FDA categories and approvals. Ultimately, labs are responsible for adhering to regulations based on the CLIA they possess and any other accreditation requirements. These standards can vary somewhat from agency to agency, but include evaluation of the laboratory’s policies, not only for test procedures, but also for personnel, environment, quality assurance, safety, risk management, and leadership. So if a lab falls short on meeting any of the documentation requirements for training and competency of personnel, temperature checks for storage of reagents, maintenance of analyzers and equipment, or corrective action and troubleshooting for variances with quality control or proficiency testing, they jeopardize losing their CLIA, and therefore their ability to continue testing.
Specifically, for moderate and high complexity CLIA certificates, labs must also subscribe to Proficiency Testing (PT) to perform testing on “unknown” samples (sent to the lab for testing) which are graded for accuracy. This also provides a peer or compare group (of labs using the same analyzers and reagents) so the labs can see how they performed compared to other labs for each specific test. This data helps them continually reevaluate the performance of their people, processes, and equipment as they determine reference ranges based on analytical capabilities and expected ranges for “normal” patient specimens, given the testing system and patient demographics. PT must be done for each type of test performed, on each type of analyzer, and with any special or specific devices. Sound complicated? There are also issues with how different methods correlate with each other, and how specimens are collected by different techniques. Is the glucose result produced by my home meter the same as the result measured by a more expensive lab analyzer, or one manufactured by a different company? How does the manner in which the specimen was collected (fingerstick or drawn from a vein), using a specific tube or container, influence the result? And how do the procedures for processing, transport, and storage impact results?
- Test methods can vary based on the principle of the test (what is being measured and how). Specimens are often combined with chemicals, exposed to electrical impulses, or evaluated for antigen/antibody reactions to measure changes or quantities to produce results. Each company producing a specific test/assay may do so using a different scientific method and different raw goods to produce the final equipment and products; scientists are working on standardization across the industry for specific tests, but there’s much work to be done. Furthermore, some methods produce a qualitative result (positive/negative, yes/no), while others produce a quantitative result (numerical value expressing the concentration and units).
- Different specimen types will generate different results (e.g. drug levels in blood versus urine), and even different types of blood specimens (whole blood, serum, or plasma) will make a difference in concentration (e.g. whole blood glucose versus serum glucose levels) so it’s important to know the type of specimen from which the result was obtained.
- Additionally, pre-analytical factors can also impact results depending on what is being tested. Leaving a tourniquet on the arm too long while collecting blood by venipuncture can elevate potassium results. And drawing different types of tubes in the correct order and mixing them properly is also extremely important because the ratio of blood to additive or any carry-over of additive from one tube to another can yield inaccurate results. This is why even the tube vendors must meet the same stringent FDA requirements for manufacturing their products and devices. Each tube must contain an exact volume of additive and produce the proper vacuum to pull in a precise quantity of blood when the patient’s blood is collected. Tubes must retain this function for the designated shelf life, and the phlebotomists using them to collect specimens must carefully check expiration dates to ensure quality specimens are obtained for testing.
So let’s get back to Theranos making their own tests. Their LDT’s are already “regulated” by CMS (through the inspection process that evaluates how the lab validated them). But Theranos touts their support of FDA regulation for LDT’s as the first lab to commit to voluntarily submitting their Laboratory Developed Tests to the FDA.
Why the goal of getting all their LDT’s approved by FDA?
Vendors do so to manufacture a test they can sell and distribute to lots of other labs to use for testing. Examples are Abbott, Beckman, Ortho, Roche, and Siemens.
Labs do so to validate a method and prove their test demonstrates accuracy and efficacy for a specific diagnostic purpose or the clinical value it provides.
- Small labs often avoid the bother (and cost!) of pursuing this approval and instead simply perform and document their own studies to prove this validation was conducted. This is often the case for even larger clinical labs active in academic research like ARUP.
- However, larger labs may have other goals in mind, like FDA approval (and patents) to provide (proprietary) testing that no other lab can perform. Remember Myriad’s BRAC testing?
Theranos probably wanted to do this for two reasons. First, it would make it easier to “distribute” its own LDT’s (analyzer, tests/methods, and collection devices like the Nanotainer) to its many lab locations (i.e. Walgreens stores) without having to validate every LDT (>200) at every location. (Remember, unless FDA changes the regulation of LDT’s, these are Laboratory Developed Tests, developed and performed by a specific lab (location) under its own CLIA (location). Secondly, by applying for FDA approval, Theranos could also seek specific complexity levels of approval, like waived status, for each of its tests since FDA controls what is deemed waived, moderate, or highly complex. Should they achieve waived status for all of their tests (like their first FDA-approved test for HSV!), it would open the markets in which these tests could be performed, and by whom. No longer would it require expensive laboratory equipment, space, and personnel, but testing could be rolled out virtually anywhere by simply obtaining a waived CLIA and placing Theranos’ tests/devices at the point-of-care. This would also make it possible to direct-market to consumers (i.e. perform testing without physician orders) in many other states besides Arizona since what is allowed is determined at the state level and many states simply allow Direct-Access-Testing for any “over-the-counter” waived tests. So that would open all of Theranos’ LDT’s (once FDA-approved as “waived”) to be performed without a physician order. So Theranos supporting FDA regulation of LDT’s equates to the requirement that labs running LDT’s would have to pursue FDA approval or cease performing these tests, and competitors unable to achieve waived status for their tests would be limited to testing locations with moderate or high-complexity CLIA’s. Not a bad business strategy.
What FDA approval has Theranos already achieved?
As previously noted, Theranos announced it had received FDA approval for its fingerstick test for herpes (HSV) on July 6, 2015. It was approved with waived status and Theranos’ statement reported, “The FDA blessing applies to the company’s testing system, including a device, analytical software and proprietary test tubes.” One could surmise this means their test method (for HSV) using a device (their Edison analyzer?) and their proprietary Nanotainer. However, this “collective” approval for the HSV test system did not apply individually to the components of the system (i.e. HSV was approved but NOT the Nanotainer, irrespective of that one specific test system). The reason? Remember our discussion of specimen types – whole blood or serum – and how collection impacts results? Each test requires independent evaluation and approval by the FDA. But Theranos is also pursuing general clearance for the Nanotainer as a medical device so its use isn’t limited to just this one test, or to specific tests as they proceed through the FDA approval process. This will validate that the collection method and container meet criteria to ensure quality specimens are obtained for potential testing, per our discussion of regulating tubes manufactured to specifications for vacuum/volume and storage/expiration.
Is Theranos’ technology valid?
We just discussed collection containers. But what about the actual testing methods – i.e. the science? Lab testing has humble beginnings in chemistry and cellular biology with test measurements based on kinetic reactions, antigen/antibody agglutination, and cell counts – simple stuff compared to the high-tech world of today’s engineering and computing power. But where’s the peer group for a cutting-edge company like Theranos? Remember those proficiency testing specimens and the way labs can evaluate how their performance compares to other labs in their peer group using the same analyzers and reagents? There is no compare group for Theranos; no one else is doing testing quite the same way they are – their technology is novel and disruptive… or at least the combination of their methods and the way in which they have structured their business model. So until they can anchor their innovation to something we can understand and compare, there is much mystery, intrigue… and skepticism. How do we even begin to compare technology that is light-years ahead? If the same trip from LA to NY were measured in time, we could establish a consistent distance we would choose to measure (i.e. the test). But depending on the method of transportation (i.e. the method or technique of measuring it), our results would be different. Using an airplane, our result is much different (a few hours) versus driving an automobile (a few days), or even by walking (perhaps months). So which is correct? We could also insist on using the same units of measure (hours), but this doesn’t change the issue of trying to compare very different methods of measure. This is where the ability to correlate results is vital. Knowing that a result of 4.75 hours of air travel correlates to approximately 42.5 hours of car travel makes the comparison (and the results) meaningful. Theranos has been working on such correlations (a few tests with several data points are displayed on graphs on their website). But this isn’t an easy task, as it has to be done for each variable in their testing system (and for each test):
- How do different specimens compare? – fingerstick versus venipuncture, and whole blood versus blood processed to evaluate serum or plasma
- How do the collection containers compare? – the Nanotainer versus a traditional blood tube from manufacturers like Becton Dickinson
- How do testing methods compare? – specific chemical reactions with color or electrical changes to quantify concentrations, immunochemical determinations, etc.
This is why Theranos’ HSV test (using the Nanotainer) received FDA approval, but their other tests are still being evaluated. And this is probably why several patients visiting Theranos centers reported having a fingerstick and a blood draw, or collections using both Nanotainers and blood collection tubes. Theranos needed to correlate their proprietary collection method (Nanotainer) and results (Edison) with collection and testing performed by traditional methods – venous blood draw and testing using a commercial analyzer (like Siemens which has previously been noted in the press).
What are Theranos’ methods?
That’s proprietary… and what the FDA is evaluating. Even after that process is complete, we’re uncertain how much will really be revealed. However, the “magic” of this testing on such small samples has some very solid, real science behind it. While the following are NOT technologies specifically known to be employed by Theranos, they do, perhaps, shed a little light on some science fields that are evolving to provide better methods (or opportunities to develop better methods) for testing small sample sizes.
- The field of Microfluidics didn’t yield just better inkjet printheads and DNA chips. Understanding the properties of small volumes (fractions of a milliliter and smaller) and being able to precisely control them has provided tools to manipulate the flow of liquids, and to use electrical fields, micropumps, sensors, and DNA in multidisciplinary fields like engineering, physics, biochemistry, biotechnology, nanotechnology, and others, to produce microarrays and digital PCR (methods that can test for molecules and DNA in extremely small sample sizes to detect and measure amounts of proteins, antibodies, infectious microorganisms, and antibiotic resistance).
- Using a technique in mass spectrometry, Matrix-Assisted Laser Desorption/Ionization – Time of Flight (MALDI-TOF), can take a tiny portion of a colony of bacteria (like the dot of an “i”), picked from a petri plate, and “vaporize” it to reveal mass spectra “patterns” that are compared in the software’s stored profiles to identify the specific organism.
- And other methods of molecular and genetic testing using Polymerase Chain Reaction (PCR) and Next-Generation Sequencing (NGS) are quickly advancing additional options as well.
These are great tools that allow for better testing methods, faster performance, and cheaper consumable costs. Sound familiar, like someone’s business strategy?
So what went wrong?
Theranos wanted FDA approval of their tests that would facilitate an efficient roll-out of testing to all their Walgreens locations. While the FDA was evaluating each test, they could hold back and utilize just a few actual testing locations where they had CLIA’s to perform the testing, and could do so using the Edison analyzer (LDT’s are allowed under CMS certification at the lab where they are developed) and they could continue collecting different specimens and performing correlations on sample types and testing methods. However, this limited them somewhat since specimens collected at Walgreens locations weren’t actually being testing on-site, but had to be transported to one of the labs where the LDT’s and correlations were being performed. Once the goal of achieving FDA approval for all tests was achieved, however, Theranos would know FDA’s categorization for each test (hopefully waived complexity like their HSV due to simple collection, processing, and testing), and could roll out Edison to all Walgreens locations for testing at point-of-care. In the meantime, the results turned out from their central testing laboratory could be from either their LDT’s (performed on Edison or by their own proprietary methods) or from FDA-approved tests performed on a commercial analyzer like Siemens (something they were probably already doing for correlations anyway).
We would surmise two major issues are to blame for sending Theranos off track. First are the issues with FDA approvals. The Nanotainer was approved as a part of the HSV test, but to receive general clearance, the FDA deemed it an unapproved medical device, pending its evaluation. This has additional implications for how and where it can be used as part of Theranos’ proprietary LDT’s and/or commercially acquired tests (purchased from Siemens, et.al. or referred to other labs for testing). Use of the unapproved Nanotainer would make even the FDA-approved commercial tests (from Siemens, et.al.) an off-label use of those tests (unless performed on blood from approved collection tubes instead), and therefore make them “laboratory developed tests.” Ditto for Nanotainers used with Theranos’ proprietary tests (which are already LDT’s), and this equates to limitations on where testing can be done. Remember that LDT’s can only be performed by the specific (high complexity CLIA) laboratory where they were developed and performed; they can NOT be performed by a waived or moderate-complexity CLIA lab and they can NOT be performed at another high-complexity CLIA lab unless that lab also establishes the tests as its own LDT’s, validating them there as well. That presented a new roadblock for Theranos. Until their Nanotainer and tests were approved by the FDA, Theranos would need to establish every testing location as a high complexity lab and perform a lot of validations… OR they would need to establish a courier network for transportation of specimens to central testing locations. And either way, they could no longer continuing drawing Nanotainer tubes to use for testing since they were deemed an unapproved device.
Secondly, the convoluted nature of trying to meet regulatory requirements for both LDT’s and commercially available tests, in multiple locations, using different collection devices, and while conducting correlations, made for a challenging situation, especially with proficiency testing. Theranos is a mix of academic R&D, traditional medical laboratory testing, and “manufacturing” of equipment and consumables for self-distribution – a novel approach in unchartered territory – “disruptive” technology in every right. It is within this context, that CMS conducted their inspection of Theranos in November 2015. The outcome was a CMS letter in January 2016 outlining the deficiencies they found – including a condition level of immediate jeopardy – that if not remedied within 10 days, threatened revocation of the CLIA certification at their Newark, CA lab.
Concerns over Theranos’ tests had been mounting and were jeopardizing business relationships. In October, Walgreens Boots Alliance had frozen the roll-out of additional Theranos centers in its stores. And by November, Safeway announced it was severing ties with Theranos after their $350M deal for testing centers in their groceries had fizzled out. It was noted Theranos failed to meet deadlines and decided not to put its blood analyzers in these clinics, but rather to transport specimens to a central lab. It should now be clear, the reason for this was the timing with FDA approvals. Device manufacturers know this well. With FDA approval, Theranos could economically place their tests and analyzers/devices in each clinic location and perform the testing at point-of-care where their patients’ specimens were collected; without it (and especially without the Nanotainer approval), they had no choice but to use other collection methods (i.e. venous blood draws) and send the specimens to their central lab for testing.
In light of these explanations, many of the reported problems, issues, and concerns can be better understood. This is especially true with regard to business developments and timing. However, questions remain about the science behind the magic – is the CMS inspection indicative of complex requirements, for an atypical model, that are already being addressed? Theranos’ website reports it “undergoes continuous proficiency testing on blinded samples from leading organizations, including the College of American Pathologists (CAP) and the American Proficiency Institute (API),” and “To date in 2015, Theranos Proficiency Testing met or surpassed performance goals 98% of the time for CAP and API across hundreds of assays. Theranos is leading the lab industry in transparency by publishing Proficiency Testing performance.” However, it should be noted the proficiency testing to which they refer was most likely for tests performed by commercially-available methods (FDA-cleared tests from companies like Siemens) since PT isn’t available for proprietary test methods. But a January 28, 2016 press release by Theranos does comment on the recent inspection results with a “Statement by Theranos on CMS Audit Results,” noting “We were grateful to have our regulators directly review our alternative proficiency testing processes for our proprietary technologies and related operations.” The report that inspection findings only impact one area (hematology) and do not relate to their Arizona lab where they “currently process over 90 percent of our tests” leaves one to wonder, what this really says about three things:
- Their efforts to work out issues with their proprietary technology – where is the proof the technology works? Will Theranos be able to convince venture capital, regulatory agencies, business partners, and their patients/providers that test results by their innovative methods really are accurate, safe, and meaningful? – i.e. actionable?
- The limitations encountered due to timing of FDA approvals – is the real problem, not the technology, but rather the timing of when it’s approved so Theranos can finally implement their testing the way they intended? Can Theranos preserve the business model of delivering convenient and affordable testing at “the corner of happy and health,” even as they have been reduced to purchasing competitors’ tests (at a loss on each test) to preserve the market share they currently possess? Can they weather these business woes until their tests are approved, or is it already too late?
- Ability to comply with CMS requirements for laboratory testing – while waiting for FDA approvals, can Theranos correct the deficiencies sited from their CMS inspection? As they had previously noted, many of the concerns were addressed at the time of the inspection back in November when they replaced their lab director with a board-certified pathologist/MD, but continued scrutiny with this month’s details of those findings – including one immediate jeopardy deficiency – doesn’t instill confidence. However, it’s only one area and not necessarily indicative of greater problems that can’t be resolved.
While these observations cannot predict the fate of Theranos, it is our hope that readers can now view Holmes’ noble ambition, neither as mythical creature nor science fiction, but rather as the business adventure it is, sparked by passion and innovation, and challenged by realities and limitations. This isn’t a fairy tale, but no one is poaching unicorns either. Some rules you can change, others you may need to acknowledge, but CLIA’s you have to obey.
For Walgreens, it is time to pause, and perhaps realign. While the potential with Theranos might have been great, tangible contributions from the laboratory testing market are yet to be made; for the time being, drug dealers like Walgreens and CVS Health are more focused on retail, specialty, and long-term care pharmacy business, and pharmacy benefits managers.