In my last blog, I introduced the concept of evidence gathering and the different pace and varying mindset between innovators and early-stage companies and the scientific community, and some reasons why the rigor of quality evidence is preferred over common sense and real-world observation. I’ll summarize that here. Of course, you can read the long version if you have not already done so.
Gathering evidence for digital health interventions is critical for a few reasons. First, digital interventions are complex and prone to both user and investigator bias, so things like direct real-world observation and common sense are not reliable guides for decision-making. Second, most digital intervention purchase decisions will require a physician’s sign-off somewhere along the way, and we physicians tend to use evidence as a strong filter for decision-making. Then there is the matter of confusing correlation with causation. This is not unique to studying digital health interventions but is a critical admonition to remember as we make claims based on the evidence we gather.
This time, I want to explore the pros and cons of different types of clinical research and share some examples from the recent literature that make a big difference in telehealth adoption.
The most basic form of gathering evidence is the observational study. As discussed in my last post, this is great for hypothesis generation but tells us nothing about cause and effect. Closely related are the longitudinal observational study and the before/after study. Any conclusions based on these designs can be contaminated by regression to the mean (a statistical phenomenon that can make natural variation in repeated data look like real change) and learning bias (among others).
A case-control design is pragmatic and efficient, comparing a group of subjects who experience an intervention with an individually matched group that do not. With access to a database (e.g., an electronic medical record database), one can use various demographic criteria to create a pseudo-control group that is closely matched on as many variables as possible but has never experienced the intervention. This is a reasonable way to gather evidence on digital interventions.
The gold-standard design is the double-blind study. The classic example comes from the pharmaceutical industry, where a new chemical entity is tested by giving it to half of the subjects and giving a placebo to the other half. As a subject, which “arm” of the study you are assigned to is random, thus arising to the term “randomized, double-blind, placebo-controlled” trial. This is rarely possible with digital interventions because it is not possible to come up with a placebo. The next best thing, and what many consider the pinnacle of evidence in digital health, is the randomization of individuals into groups that compare the intervention to usual care.
Beyond that (and beyond the scope of this essay) are matters such as calculating the proper sample size to avoid erroneous conclusions (a small sample can result in an erroneous conclusion that an intervention has an effect or that it does not – both are possible). There are other study designs, too, such as N-of-1 studies and step wedge studies, but beyond the scope of this discussion.
So how does all this relate to the real world? Here are a couple of stories that will illustrate.
If you’ve been around the telehealth world for any time, you’ve heard the age-old conundrum, “Is it as good as in-person?”. This has been answered multiple times, and the true answer is “it depends on the clinical scenario”. But a recent paper from colleagues at Mayo Clinic showed, once again, that video telehealth was of a comparable diagnostic quality to in-person care. This type of paper aids in conversations with health system executives, payers, and others, in that it provides objective evidence rather than making an emotional argument.
Another question these days is whether telehealth is additive to in-person care or substitutive. Payers fear it is the former and thus want to limit access. A recent paper in npj Digital Medicine (where I serve as Editor-in-Chief) made the case that adding telehealth services in a primary care setting did not increase utilization.
Perhaps the best example of how evidence can enable clearer heads to prevail involves another recent controversy in the behavioral telehealth world. There have been numerous news stories suggesting the widespread misuse of video telehealth being responsible for over-prescribing of stimulants for ADHD and other controlled substances. The evidence suggests otherwise. As long ago as 2005, authors extensively examined the evidence and concluded that video-based behavioral health visits were equivalent in quality and outcomes to in-person care. More recently, a well-done study showed that telehealth treatment of patients with opiate use disorder was associated with fewer overdoses than those who did not have access.
To wrap up this segment, our brains are wired in a way that we make associations and jump to conclusions. Sometimes there are unknown variables responsible for the effects we observe. Thus, the main goal of science and evidence gathering is to tease apart these variables and allow us to speak with confidence regarding the associations that are meaningful vs. those that are not.
If evidence-based care and best practices for hybrid care delivery intrigue you, I invite you to delve further in two ways. One is to join us at ATA in March 2023. Among the topics in discussion, there will be a detailed presentation of some of the latest research in the field. The second is to visit the npj Digital Medicine website. The journal is open-access, so you can read through as many of the papers, comments, perspectives, and editorials as you like.
What do you think are the top research priorities for telehealth in the coming year?