Milky Eggs

Why are clinical trials so expensive? Tales from the beast’s belly

Today I read, with great interest, a recently published STAT News feature by Matt Herper, Here’s why we’re not prepared for the next wave of biotech innovation. The basic thesis of this article is as follows: In the last decade, we have seen dramatic advances in molecular biology, which have enabled the development of a vast array of novel drug modalities and the successful treatment of previously undruggable diseases. The article mentions CAR-T, CRISPR-Cas9, COVID-19 vaccines, and drugs for cystic fibrosis; I myself also think of PCSK9 inhibitors, semaglutide, bispecific antibodies, and many others. However, the infrastructural requirements of modern clinical trials are now so high, at times reaching hundreds of thousands of dollars per patient, that they are limiting our ability to fully capitalize upon these technological developments.

Herper makes a very strong case for this argument, which I will not reiterate here. Instead, I will share some anecdotes about clinical trial development from my time working in biotech, when we hired large numbers of experienced, ex-pharma personnel to help us initiate multiple trials in the span of two years. Typically, one reads about corporate inefficiency in a very abstract manner. Specters of “regulation” or “middle management” and so on are invoked, but seldom do the details of particular scenarios emerge. I suspect this is in part because almost anyone who has a complete, birds-eye view of the whole scenario has already devoted so much of their life to their industry of choice that they have become willfully blind to its flaws. Unusually, I was an engineer who had the opportunity to watch our Phase I and II trials unfold from a good vantage point, and I hope the stories I share will be informative. Overall, they paint a picture of an industry which is severely lacking in human capital and fully captured by bureaucratic tendencies.

Contracting out clinical trial management to external organizations

Clinical trials are very complicated endeavors. If you are a very large pharmaceutical company, like Pfizer or Amgen, you may have developed all the internal resources necessary to run clinical trials by yourself. However, even if you are a large company, you may be operating at full capacity or lack personnel in key regions of the world. On the other hand, medium and small biotechs certainly do not have the capital to keep themselves staffed with a full retinue of potentially underutilized staff. For these reasons, it is very common for clinical trial management to be outsourced to contract research organizations (CROs), large external entities who will assign your clinical trial a project team of 20+ people (each of whom is probably working on multiple other trials at the same time) and help you execute the trial as desired.

There are many organizations that specialize in this service, such as IQVIA (formerly Quintiles and IMS Health), PRA Health Services, ICON, PPD, and innumerable others, ranging from boutique operations with 5-10 people who specialize in a niche field or in one particular geographic area to enormous corporate entities with tens of thousands of employees like PPD. To select a specific vendor, therefore, the contracting entity typically goes through a process of soliciting bids from several different CROs, winnowing them down to a shortlist, and setting up meetings to evaluate them in greater depth.

While superficially quite reasonable, actually going through this process is an extended exercise in sheer absurdity. Consider the initial bid solicitation process. There is no open forum where companies post an RFP (“request for proposal”) and receive proposals from CROs; instead, typically the company’s internal project manager will arbitrarily select a manageable number of CROs (in my experience, around 5-7) and personally reach out to their contacts inside the CRO to initiate the RFP process. (It is important to note that this is a deeply incestuous industry, so it is not too hard for senior personnel who have been “around the block” to have contacts in all of the large organizations.)

For the clinical trials I was involved in, the (very senior, ex-pharma) project manager put together a very detailed RFP. After all, if you’re going to ask for information, you may as well ask for a lot, right? Aside from a massive, 10-page table of requirements, this included free-form questions such as:

• Please discuss your strategy for rapid site budget and contract negotiation and execution.
• What tools do you use to determine fair market value in developing site budgets?
• Do you have experience working with Functional Service Providers (FSPs) who perform contract and budget negotiation and who handle site payments? Please describe.
• In what countries do you provide legal representation? What are the locations for this service?
• How are issues and data questions escalated to the sponsor? What is the communication plan and who is the main contact at the CRO?

The astute reader may suspect that these questions (over 30 of them!) are not particularly helpful. How, indeed, are “issues … escalated?” One would hope they are escalated quickly and competently, I suppose? Exactly how is it informative to have a full tabulation of every single country in which the CRO could conceivably “provide legal representation,” and what would the context of this representation even entail? How does one “discuss” a general “strategy for … contract negotiation?” I would simply… negotiate well…?

Furthermore, imagine that you are a large CRO and a small biotech asks if you would like to submit a bid to run their clinical trial. You of course rationally know that they are also soliciting proposals from at least several other CROs; also, the details of their trial are probably reasonably complicated and would take a long time to deeply understand. Rationally, it does not make much sense for you to expend great effort into formulating a fully personalized response. It certainly does not make sense for you spend $200+/hr employee time writing detailed, thoughtful answers to all 30+ of these questions.

Unsurprisingly, what happens is that CROs essentially have enormous templates of boilerplate text about their capabilities, and they just send these back after slotting in the details of a provisional team, making up some rough numbers about trial recruitment here and there, and putting together a hugely overinflated budget.

The following response is representative:

Wow. Really? The program leader will be the primary contact and they’re expected to maintain regular communication? What exactly would they do otherwise? This answer is basically “we’ll handle it” except expanded into two paragraphs of meaningless bloat. There are, in this proposal from a major CRO, fully fifty-seven pages, nearly all filled to the brim with copy-pasted boilerplate.

Suppose one attempts to quantitatively compare the costs estimated by each CRO. This is not a trivial task, considering that each of them breaks down their costs in a different way! Nevertheless, if one expends several hours to consolidate all the spreadsheets together, we see a remarkable range of variation:

The numbers vary so drastically that one might as well just throw darts at a pinboard. Why does one CRO estimate $133k in regulatory costs (no idea what that means exactly, given that filing in this non-USA jurisdiction is very straightforward), and another literally $0? Exactly how does “Data Management,” which mainly entails setting up proprietary database software to capture the structured data we need, vary from $168k to as much as $431k? (Note that the data management costs do not include the licensing fee for the software chosen, which ranges from $75k to over $300k.) What, exactly, accounts for the differences in in “Clinical Monitoring” activities that entails cost variations of over half a million dollars? There is literally no way to understand why the costs vary so much across the different sections, and in the end the decision was largely made on the basis of overall cost.

So now we have 5+ huge PDFs full of the biotech equivalent of Lorem ipsum dolor sit amet... and a bunch of cost numbers we barely understand. There is no real way to compare hundreds of pages of generically useless content against each other, so several candidates are eliminated based on simple criteria, typically if their estimated costs are truly enormous relative to the other options. The worst, however, is yet to come. Once the shortlist has been selected, it is time to engage in the theatrics of the “bid defense.”

For those not acquainted with a “bid defense” in this context, I will explain how you might set one up:

• Gather 10+ employees from the client
• Gather 10+ employees from the CRO
• Schedule a four hour meeting
• Listen to each employee from the CRO dutifully go through their assigned section of the 200-slide-long deck and recite a moderately well practiced script

The introductions alone usually take half an hour. (They are also entirely pointless.) Very senior level employees are often invited to these meetings, resulting in a nominal cost of tens of thousands of dollars’ worth of hourly wages. (They are probably tabbed out and working on something else 95% of the time.) The content is extremely boilerplate. Wow! You can use $GENERIC_INDUSTRY_SOFTWARE for your electronic data capture system! Am I supposed to be impressed?

In practice, there are several essential points that are discussed in the Q&A sections. For example, we were sometimes very concerned about trial recruitment, so we would inquire extensively about patient identification methods. One wants to understand not only the CRO’s technical capabilities but, basically, how smart they are and how motivated they seem to figure out good solutions to the challenges of your specific trial. However, this questioning would typically involve only about 3-4 people total and take around 30 minutes max. In spite of that, we nevertheless went through this entire four-hour exercise for every single shortlisted CRO for every single clinical trial that we attempted to initiate. It is challenging, and maybe even impossible, for me to really describe how unpleasant these were.

Note that everything I have described so far is normal biotech/pharma industry practice. In fact, it is probably a full standard deviation better than industry practice, because while we hired people with prior experience in large pharmaceutical companies or CROs, the fact that they even considered working at an obscure biotech startup means the population of good applicants is highly selected for the relatively nonconformist and entrepreneurial. (The key word here is “relatively.”)

Now it is time to make a decision. There may be several more small rounds of back-and-forth inquiry, perhaps some light negotiation on the budgets, but ultimately the CRO will not be inclined to do much more work for free. One might imagine that there is actually a lot of room for negotiation, given the enormous range of variations in the estimated costs across different CROs, and indeed I myself thought the same; however, it seems that in practice, there is an “acceptable degree” of negotiation and it is generally challenging to reduce any given CRO’s budget by more than, say, 10%.

In principle, this decision is a very consequential one, and it is very important to choose the right CRO. The success or failure of the clinical trial, and therefore the success or failure of the company as a whole, could potentially hinge on the right decision. As such, the company is, theoretically, incentivized to engage in an effective decision-making process. Ultimately, though, when one is down to a shortlist of 2-3 reasonable CROs, it basically reduces to a question of “who has the best vibes?” Essentiallyーonce you have made sure that all the hard criteria are met (geographic location, technical capabilities, etc.), you ultimately have very little to go off aside from your vague gut feeling about who is more competent and motivated.

The industry standard, however, seems to be to engage in a farcical “data-driven” scoring process where you draw up an enormous table of the different candidates and assign them crude scores:

(There is much more to this table that I have not shown here. The reader should note that the construction of this table was considered “excellent work.”)

There is one candidate, in the rightmost column, with a very low score. This particular CRO was only included because someone internally really wanted them to be considered, but the project manager was too nice to simply say “no, these people suck, so we won’t even bother scoring them.” To be honest, the rest of the scores are meaningless .How exactly does one count the value of a “good” rating versus the value of an “average” rating? The choice of “categories” also affects the rating substantially; for example many of the categories (not shown) are basically proxies for the size of the CRO, resulting in much higher overall scores for large companies. Perhaps one wants a large CRO, but if so, it would be far more efficient to encode that preference directly rather than pretending that the large CRO is good because of all its high scores in categories which implicitly measure company size! Essentially, my point is that this entire scoring exercise was arbitrary and pointless, as it can simply be designed to encode whatever preferences you held in the beginning.

In the end, we came to a decision which was more or less a priori known without the need to engage in the final evaluation process at all. Based on certain of my colleagues’ comments, I am supposed to understand that this was a remarkably fast and efficient processーrelative to the industry standard, that is.

The logistics of setting up a clinical trial

Once you have selected a CRO, the journey has only begun. In between the CRO “kickoff meeting” (yet another interminable, multi-hour, 20+ participant meeting where a great amount of boilerplate text is recited and, eventually, a plan of action is promulgated) and the enrollment of the first trial patient, there is a huge amount of setup that needs to be performed, including regulatory filings, clinical site identification (you need actual hospitals to run the trial in!), questionnaire construction, medical device procurement, and so on.

Take, for example, the problem of site identification. Essentially, you need to find clinics who are willing and able to enroll patients in your trial. In return for doing so (potentially a very logistically complex task), they are compensated handsomely. The typical clinician is not set up to field these inquiries, so it is not uncommon for clinics to join larger organizations which act as an intermediary between CROs and clinicians (one example is Clinitrials in Australia). Trial design has increased substantially in complexity over the last decade (everyone wants to measure every conceivable biomarker at every timepoint, etc.), and correspondingly these supra-clinic organizations have taken on increasingly larger roles relative to the participation of individual clinicians within unrelated institutions. At an abstract level, this is likely a contributor to the rise in per-patient costs: these organizations add an additional layer of bureaucracy and extract rents, while the incentive of the CRO is not to minimize “pass-through” costs to the client (which can certainly come in over budget!) but instead to sign on clinics quickly and start the study. Personallyーthere was an intermediary clinical organization we attempted to work with which was so obviously attempting to grift money from our coffers, the entire relationship escalated into the level of back-and-forth legal threats.

As with the initial stage of CRO selection, there is no real centralized marketplace for a company to post a clinical trial synopsis and solicit open bids from interested clinicians. (Realistically, many companies would not even want to participate in such an open market due to leakage of confidential information.) As such, initial contact with clinics typically depends on the submission of an initial feasibility questionnaire to a large number of clinics derived from a combination of the CRO’s internal databases and personal connections of whoever is involved in management of the trial. If the clinic appears superficially qualified and willing to participate in the trial, it then undergoes several more rounds of review, in-person or remote visits to formally “qualify” the site, site-specific budgetary negotiations, preparation of site-specific regulatory paperwork, and so on.

What was most notable about this process was the extreme heterogeneity between clinics. Some clinics are quite used to trial participation and proceed through the entire qualification and training process with no issue. Other clinics are very particular, have special requests about the protocol, require dealing with very intransigent doctors, and take weeks to respond to simple inquiries. The extent to which one has to put up with the more difficult clinics is largely a function of the anticipated trial recruitment difficulty as well as the desired trial size. Still, to be honest, this was probably one of the better-functioning parts of the entire clinical trial effort.

Another part of organizing clinical trial logistics is setting up what is known, variably, as the IRT (Interactive Response Technology), IVRS (Interactive Voice Response System), or any number of other very similar acronyms. Basically, this is the drug inventory logistics system which ensures that (1) clinics have enough drug on hand at all times to handle new patients and (2) patients are always being dispensed the correct treatment. For example, you might have 500 bottles of drug product and 10 participating clinics, but you don’t know what rate each clinic will enroll patients at; also, once you send drug product to a clinic, it can be very challenging to get it back. For that reason, you might want to “seed” each clinic with, say, 20 bottles of drug product, then replenish each clinic’s supply as it diminishes over time. Separately, it’s not uncommon for a patient’s course of treatment to change during a trial; for example, if certain blood tests report unusual changes, you may wish to reduce the dosage of drug supplied to the patient. This of course requires more backend management and record-keeping. Finally, you need a web-based interactive system that lets different parties monitor the status of drug supply, request more bottles if needed, report issues with deliveries, and so on.

I bet that you, my reader, are thinking something similar to what I did when I first heard about these systems: “That doesn’t sound too complicated! Even if you were starting from scratch, it wouldn’t take 1-2 moderately competent software engineers more than one week to code up everything you need, including an easy-to-use web interface.” You might even go further and imagine that the companies who have specialized in providing such systems have already developed a robust codebase to work from, and therefore only need to spend a day or two encoding the specific requirements of a given trial’s specifications.

Want to guess how much we were quoted for our IRT system?

To be clear, we are literally talking about a glorified SQL database, with basically zero scalability demands, plus a frontend that lets you click buttons to execute a couple of simple queries. I wish that I could actually show you what it’s like to click around this systemーit really gives you the feeling of being transported back to the year 2005, as though you’re using some incredibly janky piece of accounting software on Windows XP and with negative levels of polish.

I should also mention the following:

• It took several months for this IRT provider to “program” (?) the system
• The 100-page specifications document, that our very senior ex-pharma professionals supposedly read and signed off on, contained fundamental errors
• We therefore ended up somehow canceling this contract and rushing to engage a different, still very expensive (>$100k) vendor for our IRT needs

One cannot help but get the impression that this is an industry which is, at best, severely lacking in anything that could be called “human capital” and, in reality, simply deeply unserious.

More generally, it is unusual that despite the very large amount of boilerplate content that every single actor in this ecosystem has spent thousands of hours of manpower developing, there seem to have been very few actual gains made in simplifying common tasks or processes! For example, in any given clinical trial protocol, there are typically a variety of exams that need to be performed at specific timepoints: ophthalmological exams, cardiovascular exams, blood draws, and so on. As such, it’s standard practice to give clinicians a worksheet to fill out as they perform the exam, so that the data in the worksheet can be recorded.

One might assume that a CRO, who has run tens if not hundreds of thousands of clinical trials, would have a standardized, efficient process for taking a trial protocol’s specifications of these exams and translating them into formatted worksheets! Apparently, however, that was not the case. Despite literally paying the CRO to run the trial for us, we had to manually generate pages upon pages of these worksheets ourselves:

This is not a difficult task per se. There is just something vaguely absurd about the fact that basically every single clinical trial is crudely designing the same (or very similar) worksheets over and over again, with essentially zero effort made to eliminate redundant work even within the confines of a single organization who has every incentive to simplify these processes! Instead, what happens is that for every single trial, new worksheets are made; the content of the worksheets is used as an input for the setup of the electronic data capture (EDC) system, and because the worksheets are not standardized in any way, the EDC setup then requires a great deal of manual setup and inspection; finally, even though you need a new worksheet for every single exam visit that a given procedure is performed (for example, you might want to perform blood draws on clinic visits 5, 7, and 9), the industry standard for creating a “booklet” of worksheets with slightly modified headings on every page is “copy paste the same content in Microsoft Word over and over again, manually change exam dates and version numbers, and visually inspect the final PDF for errors.”

Nothing about this process makes sense. I hate to be the person who says “well, smart tech people could make this so simple,” but it just literally seems to be true.

There are so many examples of horrible, atrocious inefficiencies that I encountered while dealing with the logistics of clinical trial setup that I could easily triple the length of this section if I wanted to. Here’s a light sampling of some other issues:

• We paid five figures to a highly-recommended firm specializing in developing patient recruitment materials; received draft advertisements with blatant grammatical errors, clear misunderstandings about the target population, and low-quality stock images
• Computerized systems like the IRT or the EDC are supposedly “validated” by a process called User Acceptance Testing (UAT), where the vendor gives the client a 1-2 hour long list of carefully defined steps to execute on the web interface and at least 3 highly paid employees on the client side go through the exact same steps in dutiful compliance (note, obviously, that if the testing steps are explicitly defined, it should be possible to just automatically check that the system responds correctly to them, and it definitely should not take more than a single tester)
• Every single example mentioned so far probably involved 10+ hour-long meetings with 5+ participants, the majority of whom never said more than one or two words in any given meeting

Even to this day, I have difficulty wrapping my head around the sheer scope of incompetence that I encountered on all sides of this process.

Everyone in this industry is acclimated to a very low level of productivity

What is even worse than rank incompetence is the fact that even people who are very competentーscientists with invaluable technical knowledge, for exampleーseem to have just given up and grown acclimated to endless bureaucracy and low productivity.

Let me supply a concrete example: Our company had a large batch of technical documents which we needed to translate from an Asian language to English. These documents were supplied as Microsoft Word files, and for whatever reason the translation agency returned the translations as separate Word documents in this format:

It’s obviously very challenging to read a dense technical document, full of chemical structures and labels all over the page, if you’re constantly referring back to a table of translations. The solution? The two contractors assigned to manage the chemical formulation process (we are talking about billing rates of $300+/hr easily) manually copy pasted translations from the table into the source document.

Each document was easily 5+ pages of content, some perhaps 20+ or longer. There were probably 100+ such documents.

Why would a professional translation company, with a strong reputation for competence in translations of technical documents relevant to clinical development, not have an internal tool that lets them automatically insert a table of translations back into the source document? Why would our entire team of ex-pharma professionals look at this manual copy-paste process, nod their heads, and think, “well, that’s slow, but there’s just no better way, we’ll just have to pay the contracting hours?” How does this sort of situation even arise?

In the end, I realized this was happening, spent an hour or two writing 50 lines in Python to parse Word documents and automatically replace source text with the translated content, and probably saved upper 5 figures’ worth of billable hours in the process. But how many companies are out there where this doesn’t happen? This is but a single example of quiet acceptance of gross inefficiency that I came across in my time in this industry. I will leave the natural inference to the reader.

There is a quote from the prolific internet schizopoaster Ron Maimon of which I am quite fond:

“Rudeness is ESSENTIAL, as it is the tool that is most effective for alienating yourself from the lowlifes and scumbags, the moderators, the top writers, so that one is not influenced by their officious thetans, which have since been clinging to me like snot from a sneezy nose. Polite speech is like passing a turd orally, the only ones who don’t mind the taste are those whose heads are already acclimated to the colon environment.” (February 14, 2014, in the Facebook group “Quora Top Writers”)

Setting aside what one thinks about the value of rude speech, I bring up this quote to propose my own, parallel version: “Corporate inefficiency is like passing a turd orally, the only ones who don’t mind the taste are those whose brains have already fully atrophied in the meeting-room environment.” Sadly, this seems to often be the case with professionals with decades of experience working in clinical development. They possess a great deal of invaluable knowledge, but they also fully corrupt the corporate environment.

Suppose that one tries to push back on inefficiency in generalーsomeone naive but well-intentioned, like myself, who is tired of the Long March of interminable meetings. In fact, I expressed my concerns about the proliferation of unproductive meetings in the clinical department to the C-suite executives (with whom I remain on good terms!) and, together, we carefully planned a series of reforms to company culture designed to reduce meeting burden. We hoped to implement a soft cap on the number of meeting participants and the length of a meeting, to normalize efficient behavior such as leaving meetings midway if you no longer need to participate in the remainder, starting meetings on time without 15 minutes of small talk, and a mandated review of all recurring meetings with the intention of pruning overall meeting load by >50%.

Ironically, on the morning that we planned on roll out these reforms, we received urgent and vehement protests from a member of middle management who was notorious for obsessively overscheduling meetings with huge participant lists just for the sake of “inclusion” and “engagement.” Because this particular manager was currently playing a crucial role in managing the rollout of several clinical trials at the same time, executive management judged that the risk of alienating him was too high, and to this day I believe these reforms remain unimplemented (although I hear he has since left the company, so perhaps my slides will eventually see the light of day).

Closing thoughts

Why is this industry so cursed? Some hypotheses:

• There is fundamentally a lack of talent flowing into the space, because everyone smart is eaten by tech or finance (the only reason I had any insight into this at all was because I was an ML-focused engineer in a biotech company heavily funded by tech VCs)
• Everyone who is reasonably smart and joins this industry eventually either leaves or simply gives up on any hope of positive change to the status quo; to justify the state of affairs to themselves, they invent nonsensical copes about how things couldn’t be any other way
• The massive amount of undocumented specialist knowledge that you need to efficiently run a clinical development program strongly favors incumbents, preventing new market entrants from easily competing on the basis of cost or competenceーe.g., it doesn’t matter if your firm has 170 IQ engineers if they simply don’t know all the One Weird Tricks about how to get around the FDA’s catch-22s
• Because the cost of failure is so high, well-capitalized companies will always favor established methods, even if they are slow and inefficient, as long as they present a reasonable chance of getting the job done eventually
• To effectively run a clinical trial, you must hire on a large number of ex-pharma professionals for their specialized knowledge, and that fundamentally degrades the culture of the company beyond recognition

To be honest, I don’t have any great solutions to propose. The problems seem nearly intractable in their scope and magnitude. Even if a return to positive real rates leads to a renewed focus on the “world of atoms,” it could take decades for the industry to become more efficient!

Good luck!