Meaningful use. A misnomer if there ever was one. An excellent example of the old adage about the best of plans gone awry. Why start a discussion of real world evidence (RWE) with meaningful use? It is impossible to discuss RWE without talking about big data. And it is impossible to discuss big data without discussing the electronic medical record. And to understand the electronic medical record we need to discuss meaningful use. So that is where we start.
There were big data sets available before electronic medical records. Health plans and Medicare had huge data sets of health care claims that were paid. But these aren’t that useful in isolation because there is no clinical content. So you might know that the diagnosis was breast cancer and what drugs were prescribed but you know no details. You can guess that the tumor was Her 2 positive if trastuzumab was administered and that the tumor was ER positive if tamoxifen or an aromatase inhibitor was prescribed, but that is about it. And getting access to those data files was always complicated and difficult (this will become a recurring theme). Medicare developed a research tool called SEER that married clinical information that lived in tumor registries to Medicare claims, but SEER doesn’t include every patient or contain many details. Tumor registries in only certain geographies are included. And to use SEER you need to submit a grant proposal to Medicare since it is only to be used for research purposes. Plus it costs money to work with SEER data.
There were other data sets. Most states (46 to be exact) maintain tumor registries in collaboration and with the support of the CDC. But as anyone who has ever worked with tumor registries knows, the data is often not submitted in a timely fashion and often is incomplete and has errors. Tumor registries have historically been compiled by manual abstraction of pathology reports and medical records. This leaves a lot of room for error.
Social Security also maintains a Master Death File. When you die, your social security benefits and Medicare enrollment are terminated, so the government knows when you die. But again this file is fraught with errors. It is not updated in a timely fashion, so it can be six months behind. And it may be incomplete. In addition, it is difficult to get access to the death file. Several years ago, it was implicated in voter fraud and identity theft so the government severely limited access. It turns out that credit card companies actually do a better job at knowing when you die, but getting that data is expensive.
The modern era of big data in health care came with the widespread adoption of electronic medical records. In the good old days, doctors just wrote things down or sometimes dictated them. Not so great. Documentation was slipshod and inconsistent. The information couldn’t really be used for anything other than communication between health care entities. And the horror of physician penmanship was a very real issue.
Change occurred as a result of the American Recovery and Reinvestment Act of 2009 which was passed during President Obama’s first term. This stimulus package was a response to the Great Recession of 2008-9, the greatest economic downturn in the US since the Great Depression. Although not primarily aimed at health care, a big chunk of money helped Medicaid and folks on COBRA; it also contained HITECH, the Health Information Technology for Economic and Clinical Health Care Act. HITECH was where meaningful use was born.
The rationale for meaningful use was simple enough. Let’s bring medicine into the modern computer age. The goal was to improve the quality, safety and efficiency of health care through the adoption and meaningful use of technology, specifically electronic medical records. This was to be achieved through improved care coordination, reduced healthcare disparities, enhanced patient engagement, improved population and public health, all in the setting of adequate privacy and security. Lofty aspirations. The first phase required adopting a certified EMR and showing that you were using it appropriately.
The government precisely defined what constituted a certified EMR and what constituted appropriate use. Specifically, there were 25 measures for eligible providers (and 24 for hospitals), and to meet the requirements you needed to check about 15 boxes. The list of measures wasn’t, at first blush, at all unreasonable. Included were electronic prescribing, medication lists, smoking history, as well as ability to generate a patient summary and to exchange records with other caregivers. As time went on, you were expected to report on quality measures and show how the data you generated might lead to improved clinical care. If you succeeded, for the first several years you received an incentive payment from CMS. Ultimately, these incentives morphed into penalties for inadequate performance. As time went on, the program was to change to one focused on really improving health care.
On the one hand, the program was an amazing success. Use of electronic medical records boomed. Prior to HITECH, fewer than 25% of providers/hospitals were using EMR’s; after HITECH, over 90% are using EMR’s. But to say the program has not really delivered in improving health care is an understatement.
The rapid and near complete adoption of EMR’s resulted in a massive growth in the EMR tech sector. In a very short time, certain giants emerged. EPIC became the dominant EMR for hospitals and for physicians who work with hospitals. And EPIC became a wildly successful, privately held company. As of 2021, Forbes estimated its value at over 12 billion dollars with over 3 billion dollars in annual revenue. But working with EPIC, from the provider perspective, wasn’t always so easy. First of all, it was really, really expensive. Second of all, using the system was challenging. It was, after all, something totally foreign to doctors and nurses. Forcing it into work flow was not natural, and there were a lot of very unhappy doctors. And it was said that the EMR companies weren’t all that easy to work with. Too bad. The doctors and hospitals had no choice.
And there was malfeasance. EMR companies needed to attest that they provided all the bells and whistles. Sometimes that attestation was “an exaggeration” , and the offending party was caught (https://www.modernhealthcare.com/article/20170531/NEWS/170539978/eclinicalworks-will-pay-155-million-for-misleading-users). And there was another side to the misbehavior. It became very clear, very quickly that the EMR could be a useful tool to maximize billing for clinical services. Let me explain.
Historically, CMS paid doctors based on how complicated the patient was and how hard the doctor worked delivering care. This was based on the physician documentation. Presumably more complicated patients had more complicated histories and physical exams, and the physician could also document how much effort went into synthesizing a plan. This was translated into a level of service, which in turn correlated with a commensurate level of reimbursement. For any individual patient, the fee schedules for the different levels of service weren’t all that different, but adding up the differential for many patients could make a huge sum. So doctors were incentivized to document and bill to the level of service that optimized reimbursement.
This was a prime set-up for using the EMR as a BILLING tool, not a clinical tool. As EMRs gained traction, Medicare saw a massive increase in how much they were paying due to upcoding (https://publicintegrity.org/health/impact-hhs-ig-pledges-focus-on-medicare-billing-abuse-involving-electronic-records/). This was well documented in Emergency Rooms, among other places in medical care. This rose to the level of Medicare fraud and has led to a huge increase in government scrutiny. But there is little doubt the practice still exists.
But even if we solved the “bad behavior” issue, EMRs are a big problem. Doctors hate them. They do not make doctors more efficient. Quite the contrary. Most physicians will tell you the EMR adds hours to their work day. In fact, EMRs have been singled out as one of the major root causes of physician unhappiness and burnout. And patients don’t like them either. Most patients have had the experience of having their doctor stare at a computer screen and type the office note right in front of them rather than really engaging with them. And what is even more frustrating is that those notes don’t always go where they need to go, ie to other physicians caring for that patient.
One major frustration shared by doctors and patients is the lack of EMR interoperability. If you have ever gone to see your doctor and they didn’t have access to another physician’s records, hospital records, lab results, or radiology results you experienced an interoperability failure. Interoperability was one of the major goals of the HITECH Act and it was supposed to be accomplished by 2015, but needless to say it has not been and the solution is nowhere in sight. The Office of the National Coordinator for Health Information Technology (ONC), which was created by HITECH to serve as the technology czar, has kicked the can down the road several times. Congress has passed several follow-on laws (for example, the 21st Century Cures Act which banned data blocking)to eliminate some of the obstacles, to no avail.
Why is this so hard? Much has been written about the challenges to interopability (EHRs: The Challenge of Making Electronic Data Usable and Interoperable - PMC (nih.gov). These include the fact that every EMR system is built on its own tech platform. All of them use a different data dictionary. The health systems won’t cooperate. Data security and patient privacy are challenges. It isn’t simple. It’s expensive to build and maintain. Blah, blah, blah. The truth is that there isn’t a lot of money to be made by executing on interoperability. Many of the involved parties have reason not to cooperate. And nobody is forcing them to.
How did things go so terribly wrong? There is an excellent summary by Fred Schulte and Erika Fry from Fortune published in KFF News in 2019 (https://kffhealthnews.org/news/death-by-a-thousand-clicks/). To this, I will add the simple observation that EMRs were built to suit a government mandate and not built with users in mind. Physicians had little or no input, and most doctors will tell you that whoever built their EMR had no idea how doctors do their jobs.
But this is only part of the problem. Initially the thinking was that the data that was entered into the EMR would revolutionize health care. The argument was that we could learn from every single patient because everything would be right there in the EMR, and that this would be a treasure trove of data. This line of reasoning is now being resurrected because EMR data will be the raw material for artificial intelligence. Is it possible that after all of the pain associated with EMRs that there will ultimately be a pay out?
When I first started working with an EMR, there was a big push to get doctors to document by pointing and clicking. These point and click responses could then be aggregated and analyzed using Excel spreadsheets and simple statistical programs. This method of data recording used structured fields. But this approach flies in the face of how doctors have recorded patient histories forever. Physicians like the narrative; they like to tell the patient’s history like a story, including all the nuances. Point and click did not cut it. In fact, the number of clicks became a running joke among physicians. So what we wound up with was a mixture of text (which was really difficult to abstract for analysis) and structured data (like temperature and blood pressure).
There was some hope that tech could solve this. Natural language processing used the computer to scan and extract the important data elements. But this didn’t work very well. Among other things it was inaccurate and unable to decipher the nuance inherent in the physician narrative. The solution proved to be a combination of tech and manual data abstraction.
After I left Aetna, I worked briefly at Flatiron Health. Flatiron was an oncology big data start-up. Among other things, they were interested in harvesting data insights from the oncology EMR. They used an approach called tech enabled manual data abstraction. This involved using the computer to identify key words or phrases that were often seen in the context of the data of interest. Flatiron hired an army of data abstracters and had them scour the EMR with tech assistance. It worked remarkably well. And the tech tool could learn, improving its accuracy and thereby improving the abstracter’s ability to separate the wheat from the chaff. Flatiron was able to answer some interesting questions and ultimately was a financial success, being acquired by Roche for about $2 billion.
Artificial intelligence should be able to take this to the next level. There should be no need for manual abstraction. And the ability of AI to learn from its mistakes should make all that data that doctors have entered over the years useful.
Maybe. We started our discussion of real world evidence with the term “big data”. Maybe we should define that. Big data are data collections with the following characteristics (the five V’s):
1.Volume: The data sets must be BIG.
2. Value: The data sets must contain information that will yield meaningful insights from a business perspective.
3. Variety: The data sets must be diverse with lots of different kinds of data.
4. Velocity: The data must accrue rapidly.
5. Veracity: The data must be accurate.
Why am I bringing this up now? Virtually every single one of these elements prove challenging when it comes to medical data. I would like to specifically focus on two challenges.
The first is data veracity. Is the data recorded by physicians accurate? Anyone who has ever read a medical note knows that errors are commonplace. The causes are too numerous to count: mistakes in transcription, inaccurate interpretation of test results, incomplete documentation of patient characteristics, etc. Garbage in, garbage out. But it’s more than this. An individual practice or site of service may not be representative of the patient population as a whole. As a result, any conclusions drawn from that data set is likely incorrect. A recent perspective piece in the New England Journal of Medicine provided a wonderful example of machine learning gone wrong in developing an algorithm for abdominal pain (https://www.nejm.org/doi/abs/10.1056/NEJMp2311015). It is definitely food for thought.
The second problem, which is more unsettling, involves both volume and variety. For some reason, academic medical centers, hospitals, large medical groups, and even small medical groups think they own the data and don’t want to share, or more precisely, allow access. The same is true for commercial insurance companies. Even the EMR companies have a dog in this fight. Everyone seems to think that all the data they are sitting on is valuable. And they claim ownership because they “generated it”. If I am a patient I am not sure I agree with that position.
For one thing, that documentation is part of the work that physicians (and hospitals) get paid to do. Put another way, without documentation you cannot get paid. So perhaps Medicare and the insurance companies have a more legitimate stake. But they do not either, because they are administering a health benefit on behalf of patients who are really paying for the care. So it comes down to patients. And what do patients want? Survey after survey shows that about 80% of people are in favor of using their data for the public good (https://jme.bmj.com/content/48/1/3). But there is a big but. There are always qualifications on this “social license”. Exactly what data is to be shared and with whom and why are big sticking points. But patients don’t believe hospitals or doctors own the data.
How do we liberate the data and put it to good use? There are clearly data elements that should be available for the common good. If we think back to the COVID pandemic, many of the most important early studies came from Israel. Why? Everybody in Israel receives care from one of a handful of medical groups and all their data is centralized. These centralized data sets exist in many countries and should exist in the US. So let’s create a “data commons”.
This data commons would be owned by everybody. The administration could be housed in the ONC (Office of the National Coordinator for Health Information Technology). The actual data would live in the NIH. The NIH already does this kind of thing. They own Clin Var, a repository of genetic variants, the associated medical conditions and the evidence supporting the association. This would be a big undertaking and would take an initial investment (from Congress) but would be well worth it. And the EMR companies would be forced to participate and deliver on interoperable data submission or face stiff penalties. Again, that might be expensive but I think they can afford it. And we would fix interoperability.
Once the data is in an accessible warehouse, we can let private enterprise take over. They are most likely to invest in building solutions (using AI) to solve medical problems. Perhaps the centralized, democratized data set I mentioned above could be licensed to those tech startups. If they build a useful tool and can commercialize it, they would pay royalties to help offset the government contribution. The big winner would be patients who would benefit from everything we learn.
A recurring theme in these posts has been the nefarious effect of the profit motive, and that is certainly true when we talk about real world evidence. Perhaps I am a Pollyanna but I do think we may be sitting on a gold mine in that real world data and we should acknowledge that gold (the data) belongs to patients.
Real world evidence won’t solve everything. We will still need clinical research to answer important medical questions. But, again, we have some serious work to do if we want to improve the way we do clinical research in the US. Next we will talk about how we can improve clinical research with a special focus on cancer clinical trials.