What does it mean to have a hospital EHR?

People talk about EHR, and EMR as though these things always mean the same things. Generally what qualifies as an EHR varies greatly. When you say the word “car” do you include trucks? trucks with 18 wheels? tanks? airplanes when they are on the ground? boats with wheels?

The frustrating thing about both the word “car” and “EHR” is that they can both be used as a “class” of things as well as a specific instance within that larger class.

Technically any “Practice Management System” is an EHR because its on a computer (electric) it has a record of what happened to a patient (health record). Most people specifically do not mean a Practice Management System when they say EHR. CCHIT is one effort to correct this, by creating a set of standards that EHR systems must live up to.

What happens when you specifically define an EHR in terms of functionality? You find out that almost no one has one.

Everyone is talking about the “bad news” that was just released at AHIC. But most industry insiders have know that deep adoption of EHR technology is laughable. What is interesting about the results above is how they defined an EHR, and how reasonable that definition is!

Here are the results.

Feature Comprehensive EHR EHR Lite EHR Really Lite
Patient Demographics X X X
Problem Lists X X X
Medication Lists X X X
Discharge Summaries X X X
Lab Reports X X X
Radiology Reports X X X
Medications X X X
Physician Notes X X
Nursing Assessments X X
Advance Directives X
Radiology Images X
Diagnostic Test Results X
Diagnostic Test Images X
Consultant Reports X
Lab Tests X
Radiology Tests X
Consultation Requests X
Nursing Orders X
Clinical Guidelines X
Clinical Reminders X
Drug Allergy Alerts X
Drug-Drug Interaction Alerts X
Drug-Lab Interaction Alerts X
Drug Dosing Support X

And the results out of the hospitals surveyed:

Comprehensive EHR Lite EHR Really Lite
1.7% 7.9% 12.0%

The whole point of EHR “Really” Lite was to get the score as high as possible… Even then the usage scores are very poor.

But consider this:

Jha noted in an aside that the percentage of U.S. hospitals with fully implemented EHR systems would double if the Veterans Health Administration at the Veterans Affairs Department were included, because virtually all of the VA hospitals have a comprehensive EHR under the survey standards.

According to this survey, at least, half the hospitals in the country that have comprehensive EHRs, are VA hospitals running VA VistA.

01-26-11 Update:

It is important to remember that one of the authors of this study was none other than David Blumenthal, currently the National coordinator for Health IT. He is the person most responsible for defining “meaningful use” which, supposed to fix precisely this problem.


VA VistA is not “Old”

Recently ComputerWorld released an otherwise good article entitled: Old code proves key to modern IT at Midland Memorial Hospital.

The first paragraph reads in part:

For Midland Memorial Hospital, this came in the form of 1970s-era code unearthed via the Freedom of Information Act.

This is really frustrating. VistA is old. But it is not older than Unix which is the basis for Linux AND Windows.  It is not older than C which is the basis for C++, C#, Mono, .Net and to a lesser extent Java, PHP, Python, Ruby (in terms of syntax and overall structure).

The VA updates VistA every year. The version that Midland Memorial uses is the same version that has been recently improved by the VA.

This article makes it sound like it was literally dug up by an archeologist.  It paints a picture of David Whiles in his Indiana Jones hat with a shovel. Digging up the EHR artifact that an ancient advanced civilization used. As David wipes of the dirt he exclaims “Oh my… this is a little rusty… but it just might work!!. Then he takes the code back home to Midland, sprays it with WD-40 and gets it to run!!

This happens all the time. People hear that VistA has been in use since the 70’s and the cannot let go of how “old” it is. Hate to break it to you, but almost every core technology that you use on a daily basis has its roots in the 70’s.

Please do not tell me “Oh but HTML (or insert your technology here) came much later”. Yea but HTML does not work that well without HTTP, and for that you need a robust network. Guess when that started becoming available. Its not like VistA has been siting idle either. VistA has features that were developed in the 80’s. It has features that were new in the 90’s. It has features that are being developed now!

Meeting VistA for the first time is like visiting Australia for the first time. You see all of these marvelous creatures who have evolved in a different way, because they evolved independently. You see different “designs” or the application of the same “designs” in different ways. The one thing you should not say when seeing the strange lifeforms in Australia is “Wow, these animals must be very very old species”

No Rufus-brain. The species in Australia are not any “older” than any species in the rest of the world. What is exceptional is that this species evolved separately. Different? Yes. Backward? Maybe. Original? Definitely. But not “Old”.

VA VistA newbies constantly assume that VistA is a single instance of a program, rather than the latest instance of a program, in a long history of instances of that program. They see it as a single Tortoise that lived for forty years, rather than the latest bunny rabbit, in chain of forty one-year generations of bunny rabbits. But even this picture is inaccurate.

Another hallmark of the VistA-ignorant is to talk about VistA as though it were -one- thing. In reality it is a whole suite of technologies, that are evolving together in isolation. VA VistA is a lot more like the whole biological sphere of Australia, with lots of different species that are evolving together, all of them evolving differently than the species in the rest of the world.

Please do not call VA VistA “old” out of context. This is a mark of ignorance and is independant of whether you like VistA or not.


Medsphere Growing in the right direction?

An important part of the reason why (some) people respect what I have to say with regards to FOSS Healthcare IT is that I do not pull punches. I also do not hesitate to admit when I am wrong.

Recently, Mike Doyle from Medsphere called me. I have a lot of respect for Mike, just about everything Medsphere has done since his arrival has been right-on. He has empowered other people at Medsphere that I have respect for. He was calling to let me know that some of the information in my recent post about Medspheres layoffs, was incorrect.

Most significantly, he said that the layoffs were part of the whole company moving away from proprietary land and towards open source. He maintained that Medsphere has been commiting more and more employees to Medsphere.org, thier new community portal. I must admit, it is one of the best portals available. Matched only by the OpenMRS portal in my opinion. According to Mike, the sum total of “new hires for open source” vs. “retiring proprietary efforts” is a positive net gain of employees for open source.

The only difference between growth and decay is which parts change, and which parts stay the same.  If Medsphere is truly “retooling” its employees towards open source community members then much of my previous criticism is invalid.

It is not hard to tell that Medsphere is making an investment in the FOSS community. This is especially true in the mono community. They have hired several mono engineers and they are spending bug reports and sponsoring work. They have also been supportive of Dr. Valdes efforts to cross the streams between WorldVista and Medsphere CIS.

Doyle argues that, rather than decay, I should interpret the layoffs as growing pains. He made a good case, and so I am now forced to eat a little crow (at least it is still warm).


The Holy Grail

VistA is a robust and complete EHR system, but it relies on MUMPS. This makes VistA extremely expensive to configure and maintain.

The open source web-based EHR systems are easy to deploy but have underwhelming feature sets.

The holy grail of open source Health Informatics is a web-based CPRS (CPRS is the frontend for VistA).

It would be simple to install and configure like a web app, but it would have the sheer power elegance of VistA.

Apparently, ClearHealth has pulled this off. David Uhlman has just written to tell me that he has released screen shots for WebVistA. Granting that a screenshot is different from a working system, even seeing this much progress changes everything.  Frankly, this is almost too good to be true.

If you had asked me yesterday I would have said that it might be a good idea for Medsphere to buy ClearHealth. If you ask me today, I would say that it might be a good idea for ClearHealth to buy Medsphere.


Meeting Mike Doyle

Apparently, the people at Medsphere still read my posts.

Mike Doyle noticed my comment that I had not meet him in my last post, and he made an appointment to have a phone call with me.

I just got off the phone with him and… I was impressed. He seemed willing to reach out to the community and he seems to understand and value the Open Source process and community. This call, in combination with Medsphere’s recent press release (see last article) and their change to the AGPL for their projects, is convincing me that maybe, this “new” Medsphere might be on the right track.


Medsphere advocates for the community. Bravo!!

I have been impressed lately with “the new team” at Medsphere. I have interacted with COO Rick Jung and CMO Dr. Edmund Billings. (I am disappointed that Mike Doyle and I have not met, but he is respected by some whom I respect.)

I am happy to see that Medsphere has finally taken a stand against the current political madness regarding “phasing out” VistA.

This press release from Medsphere.com reads:

This week, the Military Health Service is expected to decide on whether to dismantle its proven electronic health record (EHR) system, called VistA. Research demonstrates that VistA has improved VA productivity by six percent each year since 1999 and that, in a time of ever-rising healthcare costs, VA care has become 32 percent more affordable than it was in 1996. The organization has also achieved an unprecedented and unmatched prescription accuracy rate of more than 99.997 percent, making it a model for healthcare organizations everywhere. In fact, as private hospitals across the country strive to achieve the holy grail of automated, paperless environments (none has reached the mark yet), it is striking to note that every public VA hospital is already there thanks to VistA. Despite all of this, the Department of Defense (DoD) appears determined to systematically dismantle VistA and replace it with a proprietary solution that is expensive, difficult to implement and has limited interoperability with other systems. VistA advocates say the move makes little sense, economically or strategically–it is not in the best interest of our veterans, our working service men and women, or taxpayers who would have to foot the exorbitant bill. 

Over the past 30 years, a community of open source users has developed VistA into a successful health care technology solution that works with existing hardware and software and preserves legacy IT investments in more than 130 regional centers across the country. So why is the military fixing something that isn’t broken? Ironically, the military tried to do something similar by installing a proprietary EHR system, named the Armed Forces Health Longitudinal Technology Application (AHLTA), in 2005. The solution proved to be expensive, difficult to install and incapable of working well with other systems. Now, it seems the DoD is heading down the same path again towards a “vendor-locked” solution that will cost billions up front and after implementation. 

It is signed by CEO Mike Doyle, COO Rick Jung and CMO Dr. Edmund Billings.

I am relieved to see Medsphere taking a stand that benefits the whole VistA community. The long-term success of Medsphere is married to the success of VistA and the larger VistA community. Medsphere is in a great position to advocate in a way that VA employees cannot. Medsphere can reach and influence those who ignore me and the other revolutionaries who are already outspoken critics of the current VA/DOD boneheadedness. It is already getting some coverage, and it deserves more.

Bravo, Medsphere.


What do about the VA crisis: the aboveground railroad

Dana has just written a new article Why are reformers destroying Veterans’ health computer system. It focuses on the disastrous centralization movement within the VA. Specifically it references Roger Maduro’s impeccably researched editorial in the Jan 2008 edition of Vista News which Roger edits.

Roger and I tend to see this issue in the same way. I was defended VistA in a Government Health IT article and I have written an article on the reason that the new Cerner lab system is a threat to VistA. That threat is hard to really comprehend until you understand what makes VA VistA good in the first place.

If my comments have been your only exposure to this crisis, then I would definitely take the time to read Roger’s editorial. Where I skim the surface of the issue, Roger examines the issue with the careful eye of someone who is far more familiar with both the VA and VistA. I learned much from the article and I consider myself relatively informed with regards to VistA. By relatively I mean “relative to the general population”.

But what to do about it. I recently listened to an excellent video interview with Tom Munnecke on early VA VistA history. What struck me about the interview is that Tom, like many VistA enthusiasts, views the movement between centralization and decentralization as a pendulum. The problem with this is that during periods of centralization, VistA starves.

Like all projects based on open source development models, VistA needs long-term leadership and stewardship. Currently, this leadership is either political, driven by the whims of presidents and congress, or bureaucratic, driven by permanent government employees who range from wildly incompetent with regards to Health IT, to amazingly capable. The best VistA can hope for, under the current model, is a good bureaucrat. The model needs to change. VistA was created by a community of computer programmers and clinicians working together. A similar community needs to be placed in charge again.

My proposal is for Congress to create a new council to make a clinical software design, development, and deployment decisions within the VA. Here are the rules for the new council.

  • The council should have 9 members at a time, similar to supreme court justices.
  • The council term should be for ten years. (the initial term should be split to ensure that members do not rotate out all at once.) Long terms are required for stability past the possible term of a single U.S. President. Members should be limited to one full term.
  • The initial members of the council should be elected by the card-carrying members of the underground railroad, and the local CIOs of the current VA hospitals, the national VA, Indian Health Services, the CIOs of hospitals outside the VA running VistA (including internationally) and the CEOs of software vendors who support VistA. The national VA should be able to appoint 1 member. The local VA CIOS should be able to elect 2 members. The underground railroad should be able to appoint 3 members. The outside CIOs should elect 1 member, the vendors should elect 1 member and Indian Health Services should elect 1 member. (Update March 2011 added private CIOs, Vista Vendors and Indian Health Services)
  • (Added 2010:) As I think about it, all council members should be able to code at least a little, some of them should -also- (and not alternatively) be clinicians.
  • Future elections will be held in the same way, except former council members will then vote with the underground railroad.
  • The council should have separate funding for 1 million dollars per year to handle incidental costs of meeting and small stipends.
  • The council should be able to meet in person on a quarterly basis, and via conference call once a week. The council can choose to invite anyone it wants to these meetings as guests. The travel for both the guests and the council will be funded by the one million per year.
  • The IT budget for the VA will be split into two parts. Any system that houses clinical information will be under the control of the council.
  • It is not required to be an employee of the VA to be on the council.
  • If an employee of the VA is elected, they will be allowed to spend the time needed to attend the meetings as part of their VA duties.
  • The council should not be a full-time position, but should come with a generous stipend, something like 50k a year, so that someone could decide to do it full-time if they wanted to.
  • The council should report directly to congress (March 2011) as well as to the CTO/CIO of the VA.
  • Congress should commit to not interfere with the councils decisions for ten years. At the end of the first decade, congress should decide to either disband, or permanently endorse the council.

Why these rules? The idea is to create a council that would be actually capable of running a software project as complex as VistA. The council should be made of people who are respected by 1. The people who originally fought for VistA or 2. The local VistA users. In short, they should be community elected, rather than bureaucrats or politicians. Their positions should be funded well-enough that they would not need to worry about how to pay for things, but not so well-funded that people would pursue the roles just for the funding. They should have long tenures, in order to isolate them from fear of reprisals for controversial decisions, and to ensure that long-term vision is achieved. Both VA employees and those who are not with the VA (like retired underground railroad members) should be eligible for the role of council member.

The million dollars should be used to create quarterly meetings that are attended by the council and by those that they appoint as custodians of particular systems. This will give the opportunity for the council to imitate what has worked for the Apache Foundation or the Mozilla Foundation which are the most complex and successful projects currently run by council. (Rather than benevolent dictator)

This proposal is basically a way to put the underground railroad formally back in charge, with a mechanism for introducing new blood and new ideas. In short, this is a proposal to create an “above ground railroad”.

Anyone should see that the council that I am proposing has parallels with WorldVistA. (Added March 2001) Since the writing I have discovered that WorldVistA has no mechanism to replace or change board members at all. The organization suffers as a result, and is no way suited to take this role.



Defending VA-VistA

I was heavily quoted in a recent article in Government Health IT entitled VA’s health IT gamble. In it, I present the case that the current IT centralization efforts within the VA are damaging to VistA and therefore the VA’s ability to deliver quality care. From the article:

“Historically, each hospital hired programmers to solve that hospital’s needs,” Trotter said. “Other hospitals then adapted those solutions to their own needs. With the centralization process, all VistA programmers will be working for a central bureau. This could stop 30 years of innovation in which the best local innovations were taken national.”

Ironically the article cited a VA official as saying that they were taking a “Evolutionary approach”, despite the fact that they just bought a Cerner lab system rather than building the functionality into VistA. Strange.


Why is VistA good? the VistA open source development model

Recently, VA VistA has been getting a tremendous amount of attention. Lets take a look at the recent events that highlight what VistA is and why it is important.

The question I would like to ask is “how did this happen?”. Lets get the story straight. Federal employees developed what appears to be the EHR at the backbone of the highest quality medical system in the United States. That makes no sense.

I defy my readers to give me legitimate examples of something that any government creates, programs, or manufactures internally that is provably superior than commercial alternatives. The only thing I have thought of is money. Various governments might be considered experts at manufacturing currency.

Despite the inability of governments to “make” for themselves the US government seems to have some cool stuff. Air craft carriers? corporation-built. Jet plane? corporation-built. Bridges? corporation-built. Now for the madness: World class Electronic Health Record (EHR)? government-built.

As the Federal Government considers how to further improve VistA it makes terrible decisions because it does not actually understand why VistA is good in the first place. At the heart of this problem is the fact that some of these Federal administrators are no longer in awe of the “miracle of VistA”.

The “miracle of VistA” is that a branch of the US Government, which has no primary expertise in software development, was able to create one of the most highly regarded electronic health records in the world.

VistA is good because it, like typical open source projects, evolved. In fact, the evolution of VistA is an alternative open source development model that is comparable in scope and significance to those found in the most popular open source projects. The best run FOSS projects evolve, but in slightly different fashions. The Linux kernel is famous for the “benevolent dictator” model now enabled by git. The Apache project has succeeded with the “wise council” model, that has in turn been successfully applied to other projects besides the core web server. Like Linux, the Apache project has developed tools specifically designed to enable the model that they use. You can easily find studies about what makes the development of Apache and Linux tick. Here is a short description of the elements of VistA development that made it successful.

VA VistA is developed in a pair programming paradigm, not two coders working side by side, but one coder and one clinician. Each VA hospital was free to develop software that meet local needs. Local hospital administrators paired one clinician, intimately familiar with a given clinical need, and one advanced MUMPS programmer, who together encoded clinical knowledge into VistA. Each local VistA programmer answered to the needs of the local hospital administration. This helped to ensure that the hospitals needs were never overlooked by a “centralized” software architect. The best software, after initial development and testing at one hospital, was quickly distributed throughout the system for hospitals hungry for similar functionality. The methods for sharing software between hospitals has become more and more formalized, and like Apache and Linux programmers, VistA programmers developed collaboration tools designed specifically to meet the needs of this distributed development environment. The hospital from which a feature originated became the de facto program manager for that feature, coordinating future improvements. Poor code was criticized and systematically abandoned in favor of good code. VistA is not actually one program, rather it is hundreds of small programs, each of which evolved and improved separately and together. No one person can “understand” what VistA is, instead VistA experts usually are familiar with a few of the VistA programs, and know which other VistA experts are familiar with the other programs. VistA is one of the oldest software projects to rely on a distributed, collaborative development model, from its inception.

Obviously, the process is more complex than can be described in a simple blog post. There is enough here to demonstrate several important points.

  • VistA was not “designed” by anyone, it evolved in a collaborative fashion similar to modern open source products
  • Because the “clinical pair programming” has been happening for more than twenty years, VistA encodes a tremendous amount of clinical expertise that is impossible to “recode” via a traditional design process.
  • VistA was made in a fashion that makes it more like an organism and less like a house or a car.
  • Replacing portions of VistA with proprietary systems is similar to, and works as well as, amputating human limbs and replacing them with prosthetics.
  • Centralizing VistA development is foolish and will never improve the EHR software. The right way to improve VistA is to encourage the evolution of the software in a process similar to the way that one would breed animals.
  • The current VA reorganization, which has local VistA programmers reporting to and paid by a centralized office in Washington, has destroyed the control and influence of local VA hospital administrators over the direction of VistA
  • Frustrated VistA progammer’s are flocking to private corporations like Medsphere or to non-profit organizations like WorldVistA in order to ensure that VistA continues to thrive. This brain-drain will ultimately damage the VA’s ability to improve VistA.
  • Something needs to be done to ensure that VistA continues to evolve.

Decisions like the recent one use Cerner’s lab system in the VA are made by administrators who do not understand what VistA is. I hope that this article will help you to understand why those familiar with VistA and Open Source software (like WordVistA’s Joseph Dal Molin and Doctors Steve Shreeve and Ignacio Valdes) are so put off by the Cerner announcement. Further this is why the typically technical HardHat’s mailing list is boiling with posts which expose the problems within the current VA thinking far more exactly that I have done here.

Now if I could just convince my congressman.