Stick your neck out

Recently, someone contributed a library to help with the webification (<- clearly this is a real word) of VA VistA. In a recent HardHats thread, he expressed his discouragement. I responded and I thought it might help other discouraged developers out there to read me reply. Sometimes the Open Source community just does not respond to good contributions. Here is some of what he wrote:


Can I ask the question: 9 months after having going out of my way to
make it available as Free Open Source to try to provide this community
with a state of the art tool for Ajax development, is *anyone*
actually using EWD with VistA yet?  I have to be honest and say that I
do wonder sometimes why I bothered.  All I seem to hear is reasons why
people haven’t used it or don’t use it.

Jim,

Working with the FOSS health community can be very trying. I fully understand how you feel. I felt that way in the early days of FreeB, which has still not been adopted by the VistA community.

Here are some basic insights about how to get things done in the FOSS Health IT community.

  1. You have to be prepared to fully ignore the peanut gallery. There will always be people who have no idea what it means to develop making comments as though they were developers. This is actually a negative side-effect of something positive: this community basically treats clinical users and software administrators as equals. That is a wonderful thing but it means that contributors like you have to learn to ignore people who are not really in your target audience.
  2. Your audience is developers, a small subset of this community. Developers are typically very circumspect group and are often lurkers on Hardhats and elsewhere (I am a notable exception to that rule).
  3. Developers never have any spare time, we always have something worthwhile that we are working on. That means for your software to get attention, it must win out over other interesting projects for any given developer.

Without presuming to speak for the rest of the development community here, I personally cannot afford to spend time on time-sink projects. Frankly, until Astronaut arrived -as a shared development environment- VistA as whole fell into this category for me. Once you get EWD working in Astronaut then it will be a different ball-game. Then a developer like me, who has very little time, can still afford to evaluate your library for potential projects. If you want an even wider audience, you should try to get EWD included, fully working, out-of-the-box in OpenVistA. Not every developer feels this way, some of them are entranced with the hard way, preferring to compile everything they can themselves from scratch just to be able to say they did.

For VA VistA there are two phases of adoption, first the VistA platforms, and then the developers who rely on those platforms. You are still at the first stage, and you should expect that only the VistA Demigods will even look at your stuff before the platforms adopt it. There are very very few VistA Demigods, which explains the reaction that you are seeing. Once the platforms have adopted your code, mere mortals like myself will consider using it for various projects. There are lots of mere mortals in the FOSS Health community. Do not get discouraged about how things are going now. You should only be discouraged after about a year of inactivity -after- the VistA platforms include your code. At that point I would assume that some other webify-VistA strategy had won out.

This is -not- a criticism or meant to imply that you have failed at anything. You have done some great work, and that is true if your code becomes popular or not. Sadly, the best code does not always make the most popular projects or vice-versa. FreeB eventually became popular, but it was certainly not because it was good code. Success in Open Source is very often an accident of history. Your code might not take off the way other code has. But no one has control over that. Even people with successful projects did not -know- that they would be successful. Everyone has to do just what you did, put your code out there and see what happens.

For my part, I have as much respect for you as I do for the pioneers of our movement. People like Torvalds, Larry Wall, Stallman, and that ilk. It is worth keeping in mind that they got famous because their technical approaches -happened- to win out. But when they started they had to stick there necks out there just like you are. I will not lie and tell you that the reputation gains that you will get will ever approach the benefits that those people enjoy. Even if you are a success in the FOSS Health community the best you can hope for reputation wise in the FOSS Health movement is a LinuxMedNews award, and even that will probably not happen unless your project succeeds. For every project that “wins” in Open Source there are ten or perhaps even more that die on the vine (take a look at the stats on sourceforge) do not be offended if that happens to you, it means nothing.

Community members who have a clue about this will still give you plenty of credit for your work, and you will be surprised how loyal even a small group of users can be. You may have people seeking you out for years to get consulting on your project, even if it never really gets off the ground! In any case, those of us who have gone through what you are going through will always be quick to recognize a fellow contributer and give you the respect and appreciation that your work deserves. We do this because we recognize that your actions take courage, and unless lots of people continuously find the courage to risk what you have, our community will begin to rot.

To sum up: Do not let the turkeys get you down, be patient, and if your project ultimately fails remember to make like Obi-Wan Kenobi.

-FT

Why so many non-profits?

When I get a good question from a conference or email, I like to answer it in a blog post so that I can just link it in when others ask me the same thing in the future.

One of the good questions I got was:

Why are there so many “Open Source Health Care” non-profits, yet few seem to have much activity?  I see OpenVista, OpenHealthTools, WorldVista, and yours (Liberty Health) just to name a few.  Just to ask the awkward question, are the differences between them worth it?  What Apache and Mozilla prove is that there is power in scale even in non-profits – to be able to talk as one really helped people figure out who to pay attention to. We wouldn’t have really been able to negotiate with Sun over the open sourcing of Java, for example, if we were speaking as a bunch of separate orgs.  Thoughts?

So here is the downlow on the organizations issue.

There is no OpenVistA non-profit (that I know of) but if there is one, it would be exclusively focused on the Medsphere version of VistA called OpenVistA. In fact there are several projects that have non-profits focused exclusively on that particular project. FreeMED and OpenEMR (oemr.org) both have their own foundations. WorldVistA also has a project, called WorldVistA EHR, but its mission is more generally supportive of different versions of VistA. WorldVistA balances between being both a single project and focused on supporting VistA generally as a meta-project organization. With that said, WorldVistA is exclusively focused on VistA, it certainly cares about certain other projects, like Mirth, but only because Mirth can be used to make VistA better. Probably the most successful accomplishment of WorldVistA is that they were the first FOSS licensed project to achieve CCHIT certification and they have regular, well-attended meetings that have good attendance from almost all of the VistA community. In terms of numbers of bodies in the real-world, WorldVistA has the largest and most active community.

There is also an group representing the VistA vendors called the VistA Software Alliance. The are not formally associated with WorldVistA and also support VistA vendors who choose to make VistA into a proprietary product (DSS, for instance, still does this in some cases). So there are organization who support VistA without explicitly endorsing Open Source or Freedom.

Open Health Tools is another story altogether, it historically, has been focused on interoperability tools: from its FAQ

….to create a common health interoperability framework, exemplary tools and reference applications to support health information interoperability.

Given this it came as a surprise that Open Health Tools worked with DSS on the release of portions of vxVistA under the EPL. While that release was significant, bringing the number of major rollups of VistA at the time to 3 (now there are 4), Open Health Tools counseled DSS into using the EPL, which is relatively unpopular with the VistA community, which have generally settled on the FSF licenses (all three of the other rollups use a GPL variant). If Open Health Tools had used the LGPL, or even Apache which strives for GPL compatibility, it might have been possible to have cross pollination between all of the major development instances of VistA. So there is a small licensing debate that is going on between the traditional VistA crowd and the Open Health Tools (there some are indications that this might be resolved soon)

In any case, Open Health Tools is designed to be a Forge site, attracting developers and providing collaboration facilities for several major projects at once. It has major industry backing and is an important force in our community. If you want to see where Open Health Tools shine, you should attend a connectathon, where many vendors, including proprietary ones, use OHT toolkits to achieve phenomenal scores. If connectathon was a competition, OHT would be winning, by a large margin. Although DSS has gotten lots of attention as an OHT contributor, the most significant contributor is actually Misys Open Source Solutions (MOSS). MOSS uses the OHT forge for development and is releasing their considerable tool set through OHT. Laika (the CCHIT interoperability compliance tool) uses OHT hosted MOSS components in its tool chain. Even if CCHIT is not chosen as the certifying body for ARRA, Laika will likely form the basis of interoperability testing in the US for the foreseeable future.

Probably one of the oldest organizations in the FOSS healthcare space is OSCHA (as of the writing, the website looks down) . OSCHA was active about a decade ago and then went dormant. It was rehabilitated by an international group and has now started having conferences again. This group has largely been tainted by the relelation that the project pushed by the founding president of OSCHA was not actually available to anyone under a FOSS license. The current OSCHA organization might be rehabilitated and the international focus of the new group is admirable, but for now the organizations future is in question. (OSCHA section added July 10 2009 in reponse to a comment)

Finally, Liberty Health Software Foundation, which I helped start and which I am currently serving as the director of, is devoted to the general advancement of FOSS in healthcare. Personally I view the organization as a kind of cleanup organization, taking those roles that require a non-profit, but that have and cannot be addressed by other non-profits. Here are several points of our strategy that set us apart.

  • We are project neutral, VistA is important but there are many other solid EHR projects out there that deserve support.
  • We are license neutral. We will support any FOSS license, and generally want to avoid getting into the ‘Free’ vs. ‘Open Source’ licensing debate.
  • We are not concerned with the ‘category’ of software, but rather its relevance. If something does not fit neatly into the current terminology of EHR, PHR, Integration and other, we will still happily work to advance the project if it might make an impact.
  • We will try to focus our development on: the boring (like documentation) that for-profit companies view as a last-priority, and development that could spawn new development. We will not be a Forge project, instead relying on other projects (like Open Health Tools) to provide a collaboration platforms.
  • We will be supporting smaller projects by providing them space at conferences.
  • We will be promoting FOSS conferences, like SCALE, and creating our own, like FOSShealth.
  • We will do -very- limited lobbying in support of FOSS.
  • We will provide an industry trade group made up of FOSS vendors, hybrid vendors, and proprietary-but-FOSS-friendly vendors.
  • Where possible to promote obviously legitimate projects as alternatives to proprietary systems, to whoever will listen.

Obviously Liberty has lots of overlap with the other meta-project groups like WorldVista and Open Health Tools especially, but we are the first organization designed intentionally to embrace everyone in the Healthcare FOSS community. I hope that by creating a central organization, that seeks support not from companies like Oracle and Microsoft, but by companies like Mirth, ClearHealth, Misys, Medsphere, DSS and Akaza Research (not a comprehensive list by any means). Companies that obviously have a significant financial interest in our movement as a whole succeeding. Also we want support from the project or multi-project specific non-profits like Open Health Tools, WorldVistA and the OpenEMR Foundation.

It is worth noting that our community is simply never going to organize itself exactly the same as the wider FOSS movement. Liberty will typically be taking roles that normally, OSI, EFF or FSF might fill in the broader space. Open Health Tools will typically be operating more like the Apache, Eclipse or Mozilla foundations with a specific development focus. However, I hope and expect that we will get frequent role reversals and overlap. Why? Because we are still a very very small community in terms of devoted developers. I would expect that there are less than 1000 people who are devoted to developing FOSS licensed healthcare applications full time. There is way more activating, advocating and forging to get done than any organization could accomplish. Unless Liberty, WorldVistA and Open Health Tools each continue to fulfill their ‘part’, we are in trouble! It would take years for another non-profit to step in the gap left by any of these three meta-project organizations.

So, for today, that is how the non-profit space in FOSS healthcare breaks down.

HTH

-FT

Hack the Road

If you have not heard of Paul Levy yet, then you are obviously new to the world of Health IT blogging. This is a CEO of a major Boston hospital that has commited to blog about his day to day dealings as the top administrator of a hospital. I have already gained many fundamental insights from reading his regular blog. He also sometimes blogs at THCB, which I follow.

Recently, he blogged about something off-topic for his typical subject. He blogged about infrastructure, specifically his efforts to get a road fixed. Here is the original post, but I am borrowing the relevant parts here.

A faculty member had complained to him that a bridge she used to get to work was covered in potholes:

Actually, I knew that I could do nothing, at least within a normal human lifespan. That bridge is a jurisdictional nightmare. It is at the border of two municipalities (Boston and Brookline), spans a transit line (MBTA), and also goes over a state park (owned at that time by the Metropolitan District Commission). Just figuring out who would be responsible for the road paving would take decades, much less getting the right person to order a repair.

So, I called Rick Shea, who was the President of MASCO, our non-profit planning and service entity for the schools and hospitals in the Longwood Area. The next day, Connie called to thank me for getting the potholes filled and a new, smooth surface on the bridge. “My pleasure,” I replied, wondering what happened.

I called Rick and he said, “I knew it would be impossible to find someone of authority to make this repair, so I just hired an asphalt firm and had the work done. Each jurisdiction — if they noticed — probably thought it was the responsibility of another. Therefore, no complaints. Job accomplished. Happy to help.”

This is ironic because this exactly what I believe Open Source software can do for Healthcare generally. By providing low-cost, excellent software, we can ‘just fix’ major problems in Healthcare that are intractable otherwise. Not that this ‘hack’ has two components: It was a technological/deployment issue of actually paving the road, along with the political insight that the mere deployment of the technology would work in the given political environment.

Here are a few things that are mired in power struggles just like this bridge.

  • Quality – how to measure if a doctor is doing a good job, and to help him/her to be a better doctor.
  • Patient empowerment – how to make a reactive patient into a proactive patient.
  • Interoperability – how to get healthcare data to usefully move.
  • Continuity of care – how to ensure that the ‘ball is not dropped’ as the patient moves around in the healthcare system.

-FT

NCVHS Testimony on Meaningful Use

(Update 08-13-09 I have already presented this to NCVHS)

Introduction

I represent a community of health software developers and clinical users that respect software freedom. This community operates in the legacy of the VA VistA underground railroad. There are several important commercial EHR vendors that respect software freedom they are an important part of our community but they are only a part and certainly not a majority. I hope to convince you today that the notion of ‘vendors’ is too small for the task of computerizing healthcare. I hope to convince you that we need an open software community to solve this problem. I believe this because this is the only thing that has substantially worked so far and that given the magnitude of this problem, this is the only thing that has any chance of working in the future. Before we can move into that future, we need to have a candid dialog about the failure that typifies current health IT.

Relevant Biography

I appreciate the opportunity to testify today. More importantly I am thankful that you have chosen to invite a person from our community to speak to you. I must discuss why I am qualified to be a representative of my community. This important because legitimacy here is not earned with degrees or certifications. The FOSS community respects me because I have made substantial code and documentation contributions under FOSS licenses. I am the original author of FreeB (http://freeb.org) which is the first billing engine that supports both paper formats and X12 available under the GPL. Several projects adopted FreeB, and although few projects still use the original version, almost all of the second generation FOSS billing systems have been built in response to FreeB. Because of FreeB, at one time code that I had written was accepted into more projects than another single FOSS contributer (although that accolade now goes to members of the Mirth project). I won the 2004 LinuxMedNews innovator of the year award for my work on FreeB. I am also the primary author of the WorldVistA ‘What is VistA Really‘ vistapedia article. I developed mutual respect for many different FOSS EHR projects through my FreeB work. Since that time I have tried remain nuetral to any given project or codebase, instead working to further the community as a whole. I no longer have a leadership role on any given project, though I professionally support several codebases. Currently I am the Chief Architect of the HealthQuilt project hosted at the University of Texas School of Health Information Sciences. HealthQuilt is focused on the use of FOSS interoperability software for the purposes of Health Information Exchange in Houston, T.X. and is funded be the Cullen Trust for Healthcare. This testimony would have been impossible to arrange without the help of the UT system folks here in Washington D.C. and the U.T. SHIS people back in Houston. Thanks!

I say these things not to impress the executive subcommittee, you already think I know something or you would not have invited me. I say this to communicate to the larger FOSS community why I am qualified to speak for the clinicians and programmers in the health FOSS movement. I must do this to offset the hubris of merely presuming that one can represent an entire community, whose opinions and priorities differ significantly. I hope my testimony is representative of the many emails and conversations I have had about this over the previous few days. I should note that Dr. Ignacio Valdes (of LinuxMedNews) and Will Ross (of Medocino Informatics) have both maintained similar project neutrality and I have relied on their council heavily.

I am a Hacktivist, which is in itself a controversial title and deserves explanation. The ‘Hacker‘ part of what I do has nothing to do with breaking into computers. That is called Cracking and true Hackers have little respect for it. The difference between a Hacker and a Cracker is similar to the difference between a graffiti artist and a simple vandal. A Hacker (for our purposes) is a person who solves complex software problems with panache. (http://www.catb.org/~esr/faqs/hacker-howto.html#what_is)

A Hacktivist is a person who Hacks for social change. I am also an entrepreneur and I charge for services and support of FOSS health software, including EHR systems. I see no contradiction between having a purpose of promoting social change and a profession of software consulting. One way to formalize this pairing is with the concept of a not-only-for-profit businesses.

Why are we here?

Of course I do not mean in this question in the global sense, but rather why is the United States government proposing that we fund EHR systems in this manner? Every other major industry computerized themselves decades ago. Why didn’t healthcare follow suit? It might be helpful to consider for a moment what was -not- the problem. Technology was not the problem. Thirty years ago, we had SQL database systems driving complex OOP applications. We had the ability to do thin clients, and thick clients, distributed software architectures and just about every other technology that drives modern EHR applications. Do not get me wrong, I love web 2.0 technologies, super-thin browser clients, Aspect Oriented Programming and Service Oriented Architectures, but those kind of recent innovations make the development and deployment of EHR applications ‘easier’, rather than ‘possible’. Technologically we had everything we needed to computerize this industry thirty years ago. Why didn’t it happen?

The problems in Health IT are political, not technical. By political I mean that the problem lies in the relationships between organizations and individuals involved in the delivery of healthcare.

Doctors, so far, have been reluctant buyers of EHR software. For good reason. EHRs slow doctors down and doctors are incented to see as many patients in a day as possible. EHRs get in the way of that, and so doctors have hesitated to adopt them. Generally, the way doctors are paid discourages them from using technology to improve the quality of their care. This funding should change that temporarily at least, which is a good thing.

The other side of that coin is that proprietary Healthcare IT vendors have been unsuccessful at selling anything that is not directly related to improving coding to doctors. Many modern proprietary EHR systems are little more than ‘coding justifiers’, they are not designed to improve the quality of care but to substantiate the increased code complexity of a given procedure. Even these EHR systems are woefully under-adopted.

We are funding EHRs because we have experienced massive, and unprecedented market failure. No proprietary EHR company has approached the market dominance that is typical in other software industries. Microsoft has about 80% market share of the desktop computing space. All of the other desktop operating system vendors combined split the remaining 20% market share. Why is there not Microsoft in medicine? Heck even Microsoft, would like to be “the Microsoft of medicine” and regularly fails in this endeavor. The largest Health IT vendors barely make it to double digit market footprint and those few that do achieved that status do so through mergers and acquisitions rather than dynamic growth. In the EHR space 1% of the market makes you a big player.

Why is there no ‘Microsoft of Medicine’?

The reason that there is no Microsoft of Medicine is that generally, healthcare does not have the same dynamics as the operating systems. When Microsoft software developers code operating systems they are essentially negotiating the requirements from other technologists, the engineers who designed the microchips in computers. There is alot of complexity in making an operating system, but there is also a very specific set of requirements that is totally and accurately defined.

Most proprietary EHR development follows a tragic pattern of ‘spec seeking bloat’. The basic development process typically looks like this.

  1. Develop Software at one clinic/hospital solve that organizations problem
  2. Start selling it in other places
  3. Start meeting new requirements from new customers
  4. Recognize that the current codebase is becoming unmanageable spaghetti
  5. Have big meeting where we all agree that the now we really ‘know what the software should look like’
  6. Write a new spec based on lessons learned from previous version
  7. Go code to new spec
  8. Release 2.0 version.
  9. Users hate it they all want previous version back.
  10. Developers scramble to make latest version work as well as previous version
  11. Return to step 3 and repeat until spec is so large that it is not possible to even consider implementing it

This is the reason that ‘mature’ proprietary EHR systems seem so bloated. An EHR is impossible to top-down architect. The product must be modified so much ‘on the ground’ that the higher level organization becomes meaningless.

It is impossible to create a ‘spec’ for an EHR system that is sufficiently complex. Trying to do this is the constant ongoing attempt to make healthcare work the way a microchip does. This is the reason why the current CCHIT ‘feature bucket’ certification is met with such resistance within the FOSS community, it is simply wrong way to approach the problem.

About Medical Manager wasting away

Most of the following information is pulled from a page that I maintain on the History of Medical Manager. http://docs.mirrormed.org/index.php/Medical_Manager_History

In the 1980’s it was estimated that 80% of all doctors using practice management systems used medical manager. If there was a company that had the opportunity to become the Microsoft of Medicine, it was Medical Manager. During the dot- com boom and bust Medical Manager was sold several times. Each time the software ownership was sold Medical Manager support staff were layed off to increase profits. Medical manager is now a ghost. It’s market share has been gutted. Its dominance has been regulated to a footnote in Health IT history. Medical Manager is important because it shows the basic temptation of with proprietary systems.

First a proprietary company releases a well-designed software system. The proprietary company supports their customers with passion. Soon their user base and adoption grows. Then the company is sold. The new management must take the same asset and now make more money on it. How do they do that? They decrease the number of support staff and attempt to force expensive upgrades on their customers. This process or variations on it, are the inevitable results of vendor lock-in, and this pattern is generally predicted by the economic models of vendor lock-in. (If anyone from SAGE is listening please consider releasing Medical Manager under the GPL, it would breath new life into the product. If you would like to try this, let me know and I will do what I can to help)

Sometimes proprietary software companies waste away and sometimes it dies of a stroke.

About AcerMed’s massive stroke

When someone discusses the process for selecting a proprietary EHR vendor, they typically recommend a product similar to AcerMed. AcerMed was CCHIT certified, and rated as best in Klas. They had enthusiastic users and capable software engineers. I have no evidence that AcerMed was anything other than an honest company. However, they were sued out of existence. Hundreds of AcerMed customers had to scramble to find new software.

Kept Honest ClearHealth vs. MirrorMed

I support the ClearHealth codebase under the trademark MirrorMed. The codebase is 95% the same, but I can support a ClearHealth instance, and ClearHealth Inc. can easily support a MirrorMed instance.

ClearHealth Inc cannot charge too much for its software, or I would undercut them. I cannot charge to much, or they would undercut me. If I am not responsive in my support my clients will go to ClearHealth Inc. If ClearHealth is not responsive… you get the idea.

Because ClearHealth and I are sharing code under the GPL, neither of us can get away with the shennigans that are common in the proprietary industry. Hard things are expensive, but easy things are cheap. If either I or ClearHealth Inc. go out of business, our clients would not be trapped or abandoned. The GPL insulates clients from the kind of corporate failure that AcerMed experiences. If Medsphere went out of business tomorrow, OpenVistA would be just fine.

If Medical Manager was available under the GPL, Medical Manager would never have tried any of the shennigans that they did. Ironically, if Medical Manager had been available under the GPL, SAGE would currently have deeper market penetration than it does now.

About VA VistA

Most of the following information is pulled from the pages that I maintain on the What is VistA Really and Why is VistA Good?

Almost everyone can admit that VistA is an excellent EHR system. Recent research shows that VA VistA operating at VA hospitals accounts for more than half of the advanced hospital EHR systems deployed in the United States.

What is not commonly understood is ‘why’ it is so good. How did it happen that a system developed by federal employees leads the way as the most widely deployed advanced EHR system in the United States? The reason is that VistA, unlike proprietary EHR systems, evolves.

From Evolutionary Dynamics: Exploring the Equations of Life by Martin A. Nowak http://www.ped.fas.harvard.edu/people/faculty/

The main ingredients of evolutionary dynamics are reproduction, mutation,
selection, random drift, and spatial movement. Always keep in mind that
the population is the fundamental basis of any evolution. Individuals, genes,
or ideas can change over time, but only populations evolve.

Each VA hospital employs its own VistA programmers to solve the problems of the local hospital. That makes each hospitals instance of VistA an ‘individual’. The VistA instances at all of the hospitals combine to form the required population. The mechanism for reproduction is the ability to copy source code. The mechanism for positive mutations is the ability of each local VistA programmer/clinician pairs to improve the VistA source code. The process of selection is the ability for V.A. clinicians and administrators to recognize superior work at another hospital and ‘kill’ the local programming effort in that area in favor of adopting the foreign code.

This is not some abstract plan. Medication barcoding was developed at a local VA hospital and then taken nationwide. This is a high-profile example of a process that is constantly repeated across the VA institutions (or it used to be).

Competing, decentralized, collaborative software development are both hallmarks of FOSS development and requirements for software to evolve. This stands in stark contrast to the recent decision to integrate a proprietary lab system into VistA. That proprietary system is incapable of evolving precisely because it cannot be freely copied. This is the reason that the VistA community was upset about this http://www.modernhealthcare.com/article/20090203/REG/302039997

I was surprised to hear that the bureaucrat that decided to go with a proprietary, static lab component defended the decision by saying ‘ we are taking an evolutionary approach’. People like this are very dangerous to the VA VistA community. They have learned to speak our language without respecting our values. They are pharisees who embrace the form, but not the spirit of our community. Because of these kinds of decisions, the majority of major VistA innovations over the last two years have been outside the VA.

There is a small chance, that someone from a congressman or senator’s office might read this. You should know that unless you are getting VistA advice from a card carrying member of the underground railroad, you are getting bad VistA advice. Personally I am a VistA novice, but I now know enough to know that the majority of VA bureaucrats are either making good decisions because A. they are listening to railroad card holders or B. they -are- a card carrying underground railroad member or C. sheer dumb luck. It is painfully obvious to those of use in the community that most VA bureaucrats typically have no idea what they are babbling about. Read my VistA articles above and then find yourself someone who actually codes in MUMPS to advise you directly. Once you get it, never ever let go of that direct programmer contact. Remember that VistA happened as a rebellion by clinicians and programmers against the VA bureaucracy. I also find that Roger Maduro and the board members of WorldVistA tend to be informed and right-headed when it comes to VA VistA.

Seven Generations

My grandmother took a drug while she was pregnant with my mother than predisposed my mother to ovarian cancer. My mother died from ovarian cancer. I will pass my mothers genes to my daughters and granddaughters. As my grand-daughters consider their predisposition to ovarian cancer they will need to consider the contents of my grandmothers medical record as well as their genetic inheritance. The content of my grandmothers medical record could easily be relevant for a period of over 150 years.

The notion that a proprietary software vendor can be trusted with the responsiblity of upgrading my grandmothers paper medical records into an electronic format that will be relevant to my grandchildren is like pre-selecting the East India Trading Company to provide the technology for the Apollo Space Missions. It should immediately strike anyone who considers the problem as a farce. Companies simply do not last for 150 years.

Making a laundry list of what an EHR should do is a little silly. It is equivalent to saying ‘We should encode both modern and future medical principles and make the computer do that’. We have only a vague idea what an EHR should do now, much less what it will need to do in the future.

I have extensively covered this in my article which covers the concept of the seven generation test, some of which I covered here. http://www.fredtrotter.com/2007/10/19/healthvault-failing-the-seven-generations-test/

Get out of the way. please.

I hope I have argued effectively that the proprietary vendor model will never delve true EHR requirements. I hope I have encouraged you to take a very-long term perspective on this problem. I hope I have shown you how dangerous proprietary licenses are to clinicians. But I do not need you to agree with me on these issues. (or for that matter to publicly admit that, secretly, you agree with me)

What I do need you to do is not create barriers to the commercialization of the evolutionary ‘VistA’ development model and ideals with your funding systems, or with your definitions of ‘meaningful use’. Please do not allow yourselves to get caught up in proprietary thinking. Here are some general rules.

Tolls are not OK. To quote from ClearHealth CEO David Uhlman.

If “Meaningfule Use” ends up requiring the American Medical Association’s Current Procedure Terminology (CPT), proprietary editions of ICD9/ICD10 codes, direct electronic transmittal of prescriptions (after the RXHUB/SureScripts merger only one company provides this), then they are precluding a completely Open Source offering for healthcare.

These kind of proprietary systems cannot be freely redistributed and that is a requirement under FOSS licenses.

Expensive, feature bucket certifications are designed for black box systems and will not work for the FOSS community. VistA is waaaay beyond CCHIT standards and has to be ‘dumbed down’ to meet the certification requirements. The FOSS community has been working with CCHIT and they have been very responsive over the last two months. But they were unresponsive for the two years before that. If you make CCHIT the only way to get certified they will have no motivation to work with us. Give us the option to create an alternative certification body. If you give the FOSS community that option, I fully expect that CCHIT will work with the community to create a separate-but-equal certification method that works for FOSS but is still ‘fair’ to proprietary vendors.

Answers to specific questions

What is the “time to market” cycle from adoption of standards to installation across the client base?

This is impossible to answer. It depends on the standard and depends on the individual who is in control of an instance of a FOSS EHR. The vendors cannot control our clients the way proprietary vendors do. I can tell you that bad standards will be adopted more slowly than good ones.

How does that enable or constrain criteria for 2011 for eligible professionals?

I have no idea.

Hospitals?

I have no idea.

Later years?

Only FOSS EHR systems are going to be able to adopt to far-future standards. Not sure I can say more than that.

What are vendors’ expectations with respect to increased product demand in 2011 and after, and how do they expect to meet it?

This is actually a question for the community and not the vendors. The existing vendors would say that they will scale their operations, and they will be able to that as well or better than any proprietary vendor. However, if the current vendors are unable to meet demand, the community will spawn new vendors to support existing projects. This is only possible with FOSS EHR systems.

From a technical perspective all of the FOSS EHR vendors I know of can scale with the ‘cloud’. (the ‘cloud’ is another technology that is impossible without FOSS). Using that technology our vendors can easily provide an EHR instance for every provider in the country. So the technology scales all the way to a national level smoothly. If our community was exclusively chosen to deploy every EHR in the country we would need to scale our support staff for things like phone support, and that would take a year or two. Even this is 10 times faster than proprietary alternatives.

What are potential risks (for example, need for additional technical support to assure successful implementations) and how can they be mitigated?

With freedom comes responsibility. FOSS EHR users have the right to shoot themselves in the foot. We cannot give our clients true freedom and, at the same time, ensure that they will always do the right thing. The best FOSS EHR vendors will be those that develop a collaborative relationship with clients that make good decisions more likely. But no vendor can control a client. Thats against the rules.

I think there is a danger that the single/small group practitioner(s) will be unfairly hurt by these technology requirements. I hope that our community will be able to address the specific cost and technology requirements that this user group has. I am afraid that technology requirements will force small practices into larger groups, which may not be the best way for those clinicians to provide care. I am advocating for a ‘simple as a toaster’ sub-projects within the FOSS community to help prevent this.

How will vendors need to adapt their product development and upgrade cycles to synchronize with progress toward increasingly robust requirements for meaningful use, information exchange, and quality reporting?

VistA is already way ahead of everyone. Other projects like ClearHealth/MirrorMed, OpenEMR, OpenMRS, Tolven, etc etc will have to catch up. Even with the other projects playing catchup, the limiting factor here will be how much technology a client can implement in a short period of time. Please read David Uhlman’s blog post below for more insights on this issue.
What changes are anticipated in the vendor marketplace between now and 2016 as a result of the incentives?

The incentives are going creating an ‘political environment’ that could replicate the focus on quality that is already found in the VA. This will replace the procedure farming that currently the most profitable clinical business tactic. Once that happens the basic ‘evolvability’ of FOSS will cause a blossoming of different systems designed to increase quality. Essentially the VistA programmer/clinician pair programming model will spread like wildfire outside the VA, even as it continues to be killed off inside the VA.

Vendors that currently have investments in VistA or other mature projects like ClearHealth, OpenEMR, OpenMRS, or Tolven will have a considerable advantage over newer FOSS vendors and proprietary vendors.

Over the next 50 years, it will become increasingly difficult to compete as a proprietary vendor. Only those who can achieve and sustain Microsoft-like development savvy will be able to compete. FOSS EHRs will provide a floor and without substantial advantages, no one will consider using proprietary systems.

The value will move away from the code itself and into higher level processes like data mining for clinical rules. This will be just in time however, because without this kind of adaptability it will be impossible to cope with the coming deluge of genetics and protenomics information.

References

I have used information and ideas from the following resources extensively.

Free and Open Source Software in Healthcare AMIA Open Source Working Group White Paper (Dr. Ignacio Valdes) http://www.amia.org/files/Final-OS-WG%20White%20Paper_11_19_08.pdf

David Uhlmans ClearHealth CEO blog post on Meaningful Use http://health365.wordpress.com/2009/04/26/idea-67-for-april-26th-2009-not-shooting-ourselves-in-the-foot-or-the-meaning-of-meaningful-use/

Dr. Edmund Billings Medsphere CMO file posts to the Open eHealth Collaborative http://groups.google.com/group/open-ehealth-collaborative/web

OpenHealth Mailing List http://tech.groups.yahoo.com/group/openhealth/

LinuxMedNews.com http://linuxmednews.com/

Webinars and papers from Mendocino Informatics http://www.minformatics.com/

HardHats (VistA support list) http://groups.google.com/group/Hardhats

I would also like to thank the folks from MOSS, Dr David Kibbe, WorldVistA and the folks from U.T. SHIS for help and advice.

.
.
.
.
.
.
.
.
.

Computer Science should be required for Medical School

Hi,

Currently, in Texas,  one is required to take Physics I and II and Calculus I (or equivalent stats class) to apply for Medical School. That is not all, of course, but they are requirements.

So far, I have never meet a Medical Doctor who needed to use calculus. In fact the only ones that might need to really understand the subject are those who are doing high-level mathematical modeling for Bio-Informatics. For these researchers Calculus is not enough. Your average primary care doctor, *never* uses calculus. They also *never* use Physics! Of course they have all kinds of systems that obey the laws of physics, including blood pressure, syringes etc. But they never treat a patient and think “hmm.. what was the relationship between volume and temperature of a gas….”. They think in higher level abstractions like “When blood pressure is high that could mean X, or Y or Z depending on….”

The real benefit of Physics and Calculus is that they introduce you to new ways of thinking. That new way of thinking makes it easier to understand higher level concepts that you *will* use everyday as a doctor.

I recently had a conversation with a clinician about some work that I am doing on the NPI database which lists every doctor (that prescribes medicine) in the United States. The conversation went like this.

Fred: “I need to munge the data, I need to process the data in a different way than it is listed (a flat CVS file), I need to turn into a real normalized database before I am going to be able to use it effectively”

Clinician: “Wow… Ok… How long will that take you”

Fred: “Well you do not want me to work on this full-time do you? I only have about 5 hours a week I can work on this in my current schedule..”

Clinician: “Yea.. but your other projects are pretty important… given only 5 hours a week, how long would this take?”

Fred: “three months.”

Clinician: “Boy… that’s a long time… I know! Why don’t you just create a database with Texas doctors, instead of all the doctors in the United States! How long will that take?”

Fred: “three months.”

Clinician: “That makes no sense at all. How can that be possible?”

Fred: “Well making the database smaller does not really help me at all, that is the part of the problem that the computer takes care of, not me”

Clincian: “Cmon. A smaller database should mean a shorter time, this seems almost obvious to me. You should be able to do it faster than that. ”

Fred: “Ok.”

Clincian: “Ok, so how long for just a Texas-only database?”

Fred: “three months”

and so on…

Now lets point out… first of all… that this Clinician is not stupid. He was ignorant of how computers, and more importantly of the process of programming computers. This issue is worth discussing in detail because it really illustrates the thought gap between someone who knows how programming works, and someone who does not.

There are many NPI records, specifically in a recent (May 2008) release of the database there were 2,557,650  lines in the comma delimited file as revealed by “wc -l” (subtracting one to account for the first line in the file, which is full of labels… not a real NPI record)

The changes that I need to make to fix the NPI database are pretty complex (fodder for another post) but for now, I will just say that it is a “Complex Reordering of each Record”. Here is how my process for approaching this problem looks:

First I need how to process a single record. So I write a function to do that. For the sake of prose, I will call that function

complexReorderingOfEachRecord( $Record )

I will look at the one Record in the NPI database and then try to pass that Record into the function and see if it does the right thing. The complexReorderingOfEachRecord is a long function, it does lots of really complex things. So complex in fact that I really cannot keep all of its functionality in my head at one time. I use various ways of abstracting the problem so that I can think about the problem in useful chunks, and figure out if each chunk is working.

I am going to actually include some psuedocode in this post.  Psudeocode is code that is not exact enough for a computer to execute, but is clear enough that a human can read it and understand what it does. Programs are like recipies, they are simply exact  instructions that the computer will follow. I will use some basic programming elements in my examples (Note to programmers: this blog is also for clinicians… so you can safely skip this…)

  • Sequential execution – Each line of code is read and executed by the computer before moving down to the next line
  • Variables – a variable is a changing placeholder for information. Each time a program is run, it is possible that the variables will contain different values. I use php, so I mark my variables with dollar signs “$”. This ends of working alot like the “x” in an algebra problem, it can have different values depending…
  • Functions – It is often useful to merge many simple lines of code into a single function. Later you can execute all of the code inside the function by calling the function name and passing data to the function by putting inside the parens “()” after the function. It is basically a way to group useful bits of commands together.
  • If/Else statements – when the computer reaches an IF statment it looks at the contents of the paranthesis “()”beside the if statment. If the contents are “true” then the code inside the braket symbols “{}” following the if statement is run. If the statement in the parens “()” is “false” then the code in brackets “{}” following the ELSE statement is followed.

So the inside of the complexReordingOfEachRecord looks like this

function complexReorderingOfEachRecord( $Record){

reorderingStepOne($Record);

reorderingStepTwo($Record);

reorderingStepThree($Record);

}

(Note to Programmers: I am actually using an OOP design for my project, so in reality these would be function calls on objects, but I want to keep this on a procedural level to make my point)

complexReordingOfEachRecord, reordingStepOne,  reordingStepTwo, reordingStepThree are all functions. The contents of recordingStepOne are not shown, but they are custom functions, meaning that I wrote them. $Record is a variable. There are no IF statements yet.

Ok.. I write this code, test it, debug it about 15 times before it works to import 1 record. But then I run the code on the first 10 records my system blows up! Some NPI records are not for Doctors at all, they are for organizations that provide healthcare: Doh! I need to run the program differently for people vs organizations!

So I modify my function to look like this:

function complexReorderingOfEachRecord( $Record){

reorderingStepOne($Record);

reorderingStepTwo($Record);

$isAPerson = reorderingStepThree($Record);

if($isAPerson){

doOneThing($Record);

}else{

doAnother($Record);

}

}

Now the function has one IF statement that looks in the variable isAPerson and then executes either doOneThing or doAnother based on the contents of $isAPerson.

I have to code, test and debug this another 30 times to get it working. I have to test it more times because the new function calls doOneThing and doAnother do not work without modifications to reorderingStepOne and reorderingStepTwo. I have to switch between thinking about different part of the problem very quickly to make sure it works. To start, everything breaks, but as I discover why, by running the program again and again, I make small changes that eventually make the whole process work correctly. The shorthand for this process is code, test, debug, repeat.

As I am working I start to run the program on the first 100 records. I notice that often the person in the record is not an M.D., there are also dentists and other clinicians who are in the database ! But my work is focused only on M.Ds. So I modify the code again:

function complexReorderingOfEachRecord( $Record){

reorderingStepOne($Record);

reorderingStepTwo($Record);

$isAPerson = reorderingStepThree($Record);

if($isAPerson){

$isAnMD = doOneThing($Record);

if($isAnMD){

processMD($Record);

}else{

processNonMD($Record);

}

}else{

doAnother($Record);

}

}

Now I have a “nested” IF statement, an IF statement that exists in another IF statement.

As before all of the other functions must be modified to make my two new functions processMD and processNonMD work correctly. This requires 50 repetitions of code, test and debug. Sometimes one code, test and debug cycle takes 30 seconds. Usually it takes about 5 minutes. Sometimes it takes as much as 15 hours.

Now I am testing against 1000 rows of the NPI database, and it works perfectly! I have put in about 40-50 hours (or about 3 months at 5 hours a week)

But now what! I have only imported only 1000 rows of the database. Now I will explain how I ran the code on one row, 100 rows and then 1000 rows. I will introduce the WHILE statement to my simple psuedo code.

$i = 1

while($i < 1000){

$Record = getANewRecord()

complexReorderingOfEachRecord($Record);

$i = $i + 1

}

The “while”is just like an if statement, except that when the contents of the curly brackets “{}” are done, then the contents in the parens “()” are re-evaluated. If they are still true, then the contents of the “{}” are run again. The $i variable starts at 1, and then grows by one every time the contents of the curly braces are run “{}”

So how do I import the whole NPI database? I change to code to look like this:

$i = 1

while($i < 2557650){

$Record = getANewRecord()

complexReorderingOfEachRecord($Record);

$i = $i + 1

}

Then I start the program and go to sleep. In the morning, all 2,557,650 records are correctly processed.

Once I had done the work to determine “How to change an NPI record” the computer simply repeated that process for as long as I wanted. Computers are so fast now, that even very very complex processes can be repeated very quickly.

You see *I* never import any data. The computer does that part. *I* the programmer tell it *how* to import that data. Like doctors, when programmers have a simple concept with big implications, we create an important sounding word for it. The important word for *how* to do an information task is “algorithm“.

If you get an algorithm right, computers do just what you want. If you get the algorithm wrong… computers do other things. If you get the algorithm badly wrong… God help you.  This is why computers often seem to have a “mind of their own”; when programmers tell them to do the right thing, they do exactly that. When programmers tell the computer to do the wrong thing… they do exactly that.

Any programmer reading this is likely going blind with boredom. But someone who has not programmed might likely be asking “Wait… what’s a function?” This is actually a pretty terrible introduction to programming. For something more real, I suggest you start here.

My point is this, computers make some types of tasks really easy. Getting to them to do those tasks, without making a serious mistake, is pretty difficult and time-consuming work. If you, as a clinician, do not understand what tasks are hard, and what tasks are easy, then it is almost impossible to evaluate the software you are using. I cannot tell how many times a clinician has requested a “simple change” that has taken me three weeks of programming. On the other hand, I cannot tell you how many times I have seen clinicians (or more often clinicians staff members) subject them selves to terrible software designs that would be trivial to fix.

To create an algorithm you need to understand two things:

  • What the computer can do
  • What the computer should do

There are some people, like my friend Ignacio Valdes, who have been extensively trained in Computer Science and Medicine. These people are amazing, because you can watch them switching back and forth between one part of healthcare IT (Clinical know-how) and the other (Computer Science know-how). But even these few gems (rare as hens teeth), cannot actually hold the complexity of even a single clinical IT problem in their head at one time. That is just not the way that programming, clinical care or anything truly complex works! Programmers must ignore parts of a program to improve on any given part. Clinicians must ignore parts of a patients body to address a problem with one part. (Most heart surgeons, for instance, remain unconcerned about the flaky skin problem while their patients are in open heart surgery.) Knowing what to ignore, and what deserves attention is often the true test of expertise.

The only way to deal with Healthcare IT is to create teams of people to manage the complexity together. The problem with that is that for any given problem domain, there is a danger that the communication cost will grow exponentially in relation to the number of participants. It is common for the communication costs to totally destroy all productivity in a given group. But at the same time, it is simply not possible for a single person to correctly navigate the complexity of even a simple Health IT software project.

The solution to this problem is found in the VA VistA development model. Here are the rules:

  1. You do not work on “the system”, you work on part of the system. VistA is actually hundreds of programs that work together.
  2. Whenever possible you work in pairs. Any more gets unmanageable.
  3. One person must understand everything they need to about the programming of the clinical issue. We can call this person the Programmer. (In the VA this is a Programmer or a CAC)
  4. It helps if the Programmer has a basic healthcare vocabulary.
  5. Another person must understand everything they need to about the clinical problem itself. We can call this person the Clinician.
  6. It helps if the Clinician understands, basically, what is easy, what is hard, what is possible and what is impossible with computers.
  7. You rely on other pairs to address other clinical problems.
  8. You intentionally have redundant “programming pairs” so that you are forced to compete to make better solutions.
  9. When another pair makes a better solution to your problem, you celebrate that and adopt their code as the new starting point.

Its number 6 that this article is focused on. It would be really helpful if Physicians in particular were required to know what a “for loop” meant. Just like calculus and physics they will rarely, if ever, use that information. But for the time being, the fundamental lack of understanding of computer science in clinicians is holding healthcare back. Can you imagine speaking to your doctor if he or she had no idea what the word “pressure” meant in the phrase “blood pressure”. As it stands, most doctors do not really understand what the implications of the word “Information” in the phrase “Health Information Systems”.

What scares the hell out of me is not that the clinician above did not know how the programming process worked. Ignorance has a simple cure: learning. What scares me is that he was willing to pressure me to speed up the schedule, even after I explained how things worked. Trying to force a programmer to take short-cuts to make a deadline is a very very bad idea (see point number 4 here). Doctors, like military officers, often fail to recognize that in “being in charge” is contextual. It does not matter if a Doctor is right about a clinical issue, if they are wrong about a software design issue. The resulting software will fail to perform, despite its clinical correctness. Doctors cannot “be in charge” in software design the way they can in an operating room or in clinical practice. That does not mean they are not vital, it just means they should not be in charge. The programmers should not be “in charge” either. The “Clinical Pair Programming” that I am describing above is a description of the peer thinking that is required to solve these problems. When someone is “the boss”, (meaning they actively back only their own priorities) the system breaks.

The irony is that the few Doctors I know who are my peers with regards to computer science education, are often more hesitant to challenge me regarding my information systems opinions. Do not get me wrong; they often disagree with me, but not more than any programmer would disagree with any other programmer.

This is why I support an undergraduate computer science prerequisite for medical school.

-FT

We need a conference

So I am going to run a conference. I figured this was about as bad a time as I could pick, since no one has any travel budget, and people are getting laid off left and right! However, I have been wanting to do this for long enough that I have decided to something about it.

So why a conference? Here are my thoughts.

  • Free and Open Source in Healthcare has come into its own.
  • Serious players like DSS, e-MDs and Misys are now taking a hybrid FOSS/proprietary approach.
  • Pure plays like ClearHealth and Medsphere are kicking butt and taking names.
  • Grant writers are starting to favor Open Source in their grant applications
  • Huge policy decisions are being made by law makers regarding Health IT, some proposals, most notably Stark’s, strongly favor open source software.
  • Mature Open Source efforts are impacting every aspect of Health IT, including EHR, Bio-Informatics, HIE, Imaging, PHR, etc, etc…
  • Despite having many mature projects we are still operating as a dispersed community.

I have the privilege of being known, and at least a little respected by members of several of the most important FOSS Healthcare projects. Projects like:

  • Tolven
  • Medsphere
  • ClearHealth
  • Mirth
  • WorldVistA
  • OpenEMR
  • OSCAR
  • Misys HIE projects

In fact, I am probably one of the most well-connected people in FOSS healthcare. I think part of the reason is that after I left ClearHealth as project manager, I decided not to start another codebase. I also think that as the original developer of FreeB (a library rather than a standalone project), I have some credibility as a contributor to the movement generally, rather than being loyal to a particular project or group.  Thats fine by me. It also puts me in a really good position to highlight the competition between the projects! I win no matter which project comes out on top! But while competition is healthy, FOSS is also about collaboration, and we do not have enough of it.

Healthcare IT is, probably even more than IT generally, an ecosystem. We need software to do hundreds of very different tasks, and that means tens of thousands of programmers all need to be working in some kind of coordinated manner. There are several areas where collaboration in Health IT is critical:

  • Interoperability
  • Web Services
  • Service Oriented Architecture
  • Library-ization of critical functionality
  • Good ideas moving between projects

My own project, FreeB, was one of the first Health IT specific FOSS project to be useful to several other FOSS projects. Now Mirth, from Webreach, has taken the title of “most helpful project for other projects”. We need more of this kind of cross-project code, that other people can rely on and build on.

Meeting together gives us common direction, allows us to reduce duplication of effort, and is generally fun. I would love it if I could abandon projects because better stuff that I did not know about was out there! The projects listed above are doing really well and almost all of them have communities that they are building! But I get a call every month from a legitimate project or a new effort from a standing project that says “How do we build community”. I am also humbled by new projects taking on different problems (Like Trisano) or by companies that seem to “get it” out of the blue and take the plung into FOSS (like DSS)

WorldVista and OpenMRS are the only two projects that I know of that are large enough and successful enough to have their own community meetings. Both of these communities rave about the level of progress that is made during large community meetings. I have been to the WorldVistA meetings and they are a tremendous amount of fun! One of my personal goals in life is to one day attend an OpenMRS meeting in Africa or South America!

But other projects are too small to make a community meeting worthwhile. You have to rent the space, sort out the food, sell tickets, provide t-shirts… It is daunting to do a community meeting and it is not worth the effort if only 5 people from your project can make it.  The problem is that it takes meetings to jump-start community and community to make meeting worthwhile.

So I am starting a conference, which I hope will at least be held yearly,  that will do three things.

  • Provide one-stop shopping for people interested in using, developing, selling or buying FOSS software in healthcare.
  • Provide a place where projects meet, compete and collaborate.
  • Provide a place where projects of any size can hold face-to-face community/development meetings without worrying about the details.

With that in mind I am happy to announce FOSS in Healthcare. This conference will be held in the Summer of 09 (July 31 – Aug 2) in Houston T.X. Click here to register.

There are two big issues I need to address:

1. I need to know how many people are coming so that I can escalate my facilities if I need to and 2. I need to make this conference affordable to the individual FOSS enthusiast.

With that in mind, we are offering 1 month of early-bird registration at $60 a person.  After that the fee goes to $250 per ticket. Basically, that means that if you register now, the sponsers (contact me if you want to be one) will be paying your way, but if you wait… its all on you!!

I might offer some kind of middle ground like $120 per ticket the month after the $60 deal runs out… but there are no guarentees. I can promise you that $60 a ticket is as cheap as it gets.

Please drop me a comment about what you would like from a FOSS Health IT conference! At this stage I might be able to accomidate a really good idea!!

-FT

VA VistA is not “Old”

Recently ComputerWorld released an otherwise good article entitled: Old code proves key to modern IT at Midland Memorial Hospital.

The first paragraph reads in part:

For Midland Memorial Hospital, this came in the form of 1970s-era code unearthed via the Freedom of Information Act.

This is really frustrating. VistA is old. But it is not older than Unix which is the basis for Linux AND Windows.  It is not older than C which is the basis for C++, C#, Mono, .Net and to a lesser extent Java, PHP, Python, Ruby (in terms of syntax and overall structure).

The VA updates VistA every year. The version that Midland Memorial uses is the same version that has been recently improved by the VA.

This article makes it sound like it was literally dug up by an archeologist.  It paints a picture of David Whiles in his Indiana Jones hat with a shovel. Digging up the EHR artifact that an ancient advanced civilization used. As David wipes of the dirt he exclaims “Oh my… this is a little rusty… but it just might work!!. Then he takes the code back home to Midland, sprays it with WD-40 and gets it to run!!

This happens all the time. People hear that VistA has been in use since the 70’s and the cannot let go of how “old” it is. Hate to break it to you, but almost every core technology that you use on a daily basis has its roots in the 70’s.

Please do not tell me “Oh but HTML (or insert your technology here) came much later”. Yea but HTML does not work that well without HTTP, and for that you need a robust network. Guess when that started becoming available. Its not like VistA has been siting idle either. VistA has features that were developed in the 80’s. It has features that were new in the 90’s. It has features that are being developed now!

Meeting VistA for the first time is like visiting Australia for the first time. You see all of these marvelous creatures who have evolved in a different way, because they evolved independently. You see different “designs” or the application of the same “designs” in different ways. The one thing you should not say when seeing the strange lifeforms in Australia is “Wow, these animals must be very very old species”

No Rufus-brain. The species in Australia are not any “older” than any species in the rest of the world. What is exceptional is that this species evolved separately. Different? Yes. Backward? Maybe. Original? Definitely. But not “Old”.

VA VistA newbies constantly assume that VistA is a single instance of a program, rather than the latest instance of a program, in a long history of instances of that program. They see it as a single Tortoise that lived for forty years, rather than the latest bunny rabbit, in chain of forty one-year generations of bunny rabbits. But even this picture is inaccurate.

Another hallmark of the VistA-ignorant is to talk about VistA as though it were -one- thing. In reality it is a whole suite of technologies, that are evolving together in isolation. VA VistA is a lot more like the whole biological sphere of Australia, with lots of different species that are evolving together, all of them evolving differently than the species in the rest of the world.

Please do not call VA VistA “old” out of context. This is a mark of ignorance and is independant of whether you like VistA or not.

-FT

Trust but Verify and Trust but Fork

I have enjoyed participating in the National Dialogue about Health IT. One of the challenges put forward to my suggestion that decision makers should insist on FOSS in Health IT, was the following comment:

 in terms of privacy, there’s nothing inherent in FOSS that makes it superior to all proprietary products.

I have discussed this issue before, mostly when discussing HealthVault, but my comments have been spread out over several articles.

There is an inherent benefit to privacy, confidentiality and security for FOSS health IT systems.

There is another idea on the National Dialogue site that I thought was useful. It separates the concepts of privacy and confidentiality. Most people blur the concepts of privacy, security and confidentiality and talk about them in the same mouthful. For now I will consider that “privacy” is the ability to control who gets to see your data. Although my points apply to confidentiality and security as well.

FOSS Health IT  are inherently better ways to respect privacy because they support “trust-but-verify”, while proprietary systems just support trust.

The only way to know what a program is doing is to read the most human-readable version of that program, which is typically called sourcecode. There are countless examples of programs doing things other than what they appear to be doing. Viruses, Spyware, Monitoring features and Bugs are classic examples of this.

When a proprietary Health IT program says it respects your privacy, there is no way to know for a user to know if this is true directly, he must trust the proprietary vendor. The fact that most proprietary vendors are honest is irrelevant. The trouble with dishonest people is that you cannot tell the difference between them and honest people. We cannot know which proprietary Health IT vendors are respecting privacy and which are not. Also, the same large organizations who you might normally “trust” have in fact a very poor history of abusing privacy; Microsoft being the best example.

So does HealthVault respect privacy? Probably. But there is no way to be sure without reading the code.

Does Dossia respect privacy? Probably. But we can check by auditing the sourcecode of Indivo, because Dossia is based the FOSS Indivo project. Suppose that you believe that Indivo does not do a sufficient job of respecting privacy, or you find a back door (unlikely). You can fork the code, remove or change the offending portions of Indivo, and then run your own Indivo server with the privacy features that you want.

FOSS supports both trust-but-verify and trust-but-fork which is the only way to absolutely certain that privacy is maintained.

Therefore FOSS does have a fundamental advantage over proprietary software with regards to privacy concerns.

-FT

Vindicated

I must admit. I love the feeling of being proven right. Granted, it appeals to my egotistic streak. (which despite my attempts to suppress it, my wife remains keenly annoyed by).

A few weeks ago, at TEPR, I did my regular talk the Health of the Source, which is basically an update on the whole FOSS Health IT industry. In that talk I mentioned that OpenMRS, along with WorldVistA and ClearHealth, was a top EHR project.

Now, OpenMRS is covered by BBC News. I only wish that the article would also acknowledge that this kind of success is only possible because OpenMRS uses a license that respects the freedom of its users.

However, Doc Searls get it. He has heavily quoted my last post while discussing his recent experience with the medical system. He titled his post:  the patient as the platform. The great thing is, when Doc talks about things other people do to.

It does feel good to have people say nice things about me… but I hope this also might represent a tide turning towards awareness of the implications of software licensing in medicine.

I can hope.

FOSS Sin: Pointless Duplication of Effort

Duplication of effort is viewed as a sin in the Free and Open Source software (FOSS) development community. The whole ethos of FOSS dictates that developers should work together, sharing the improvements they make to software between them. For this reason forking, or starting a redundant project, is often viewed as an attack against the community.

The taboo against forking is well-documented. Eric Raymond (or esr) wrote about this is Homesteading the Noosphere specifically in a section from Promiscuous Theory, Puritan Practice. Here he is discussing three taboos of the FOSS culture, note the first one:

The taboos of a culture throw its norms into sharp relief. Therefore, it will be useful later on if we summarize some important ones here:

  • There is strong social pressure against forking projects. It does not happen except under plea of dire necessity, with much public self-justification, and requires a renaming.

I would like to make a specific case that starting a new project, depending on how it is done, can be as violent to the community as an unwelcome fork. I believe that new projects, without new advantages, are just as damaging to the community as forks are. If the community agrees with me on this, I will ask Eric to add “starting redundant projects” to his list of taboos. [section added 4-10-08]

Two separate projects, both doing essentially the same thing in the same way, fractures the community into two factions of programmers who can no longer work together. To fully understand this problem, one has to understand a little about how the Free and Open Source development community works. The first thing to understand is the licensing.

FOSS licensing allows anyone to download software sourcecode and modify it to their hearts content. While anyone can make modifications to FOSS licensed code, there is a community standard that there is one “official” person or group who controls the official version. For Linux this person is Linus Torvalds. There are many “versions” of Linux were code is developed and improved, but the community agrees that the version that everyone trusts is the version that Torvalds releases. Legitimate Linux development always aims to add features to Linux that will ultimately meet with Linus’ approval. But this is only because of community custom, the license does not require this at all. Eric entitled the chapter above Promiscuous Theory, Puritan Practice precisely because FOSS licenses allow forking while the FOSS community frowns on forking. There is nothing, other than cultural rules, stopping me from creating an operating system kernel based 100% on the code currently residing in the repository maintained by Torvalds, and call the project “Fredux”.

If I started “Fredux” either by forking the Linux codebase, or by starting a new kernel project from scratch and posted it to sourceforge, the Linux community would either mock me or ignore me. The reason is simple; I am not important in the operating system communities and Linux is very well established. However, it would not be difficult to really piss them off. My hypothetical “Fredux” project could easily anger the Linux community by doing any of the following.

  • Post something to public forums, especially the Linux Kernel mailing list, in an attempt to attract developers from the Linux community over to Fredux.
  • Pretend in public that my project is unique or innovative in way that distracted from Linux and its legitimate competitors.
  • Get investment capital, or grant funding, at the expense of competing Linux-based projects to develop my Fredux operating system.

What do these actions have in common? They all poach resources that could have legitimately gone to the Linux community. If I started an operating system from scratch the community would react… poorly. The core issue would be that I would be acquiring resources that should have gone to Linux or another legitimate operating system project.

Why? What is wrong with starting a new operating system project? There is no problem with starting a new legitimate operating system, and then competing for resources for it. The problem comes when you start an illegitimate new operating system and try to pawn it off as a legitimate. Of course this begs the question: What makes a new project or a fork legitimate?

Here are some guidelines for when a new project might be in order. (contact me if you feel I have missed something)

  • The project uses a different programming language, which has some advantage in the field of inquiry.
  • The project addresses a serious feature gap in current projects.
  • The project addresses a serious design limitation in current projects.
  • The project uses a new programming paradigm that has advantages over those currently in use.
  • The project uses a different development process that might have some advantages.
  • The project uses a more common and accepted FOSS license than alternative projects.

So lets see how this applies to Fredux! Here is a hypothetical Fredux release announcement version .1

Fredux is a new operating system kernel available under Freds Very Open Sourcey License. Fredux is coded entirely in C which means that it compiles to blazing fast machine code. Fredux is designed to provide a kernel for a POSIX compliant operating system. Fredux consistently uses a modular approach to software design, allowing it to be extended easily. Fredux will use a distributed development model, ensuring that everyone can see the code and allowing the “many eyeballs” effect to take place. Most importantly, Fredux is completely Open Source and Free Software!

Ok, what are the problems with this announcement? First of all the “usual suspects” of FOSS operating systems (GNU/Linux, FreeBSD, OpenBSD, Hurd, etc) all use the programming language C. So there is no language benefit to Fredux. Linux and several other projects strive for POSIX compliance, so Fredux has not addressed a feature gap. Because Linux is modular there is no design advantage. Fredux is developed in the same way Linux is, ergo there is no development style advantage. Lastly, Fredux is NOT open source! You will find that Freds Very Open Sourcey license does not appear as an OSI approved license. You will also not find the license on the list of FSF approved licenses. So the license is neither “Free” nor “Open Source”. It is shame how many companies continue to market themselves as Open Source without actually stepping up and meeting the licensing requirements.

On the other hand lets look at the description of a legitimate operating system project.

Hurd is a collection of services that run on the Mach micro-kernel. The Hurd implementation is aggressively multi-threaded so that it runs efficiently on both single processors and symmetric multiprocessors. It is possible to develop and test new Hurd kernel components without rebooting the machine (not even accidentally). Running your own kernel components doesn’t interfere with other users, and so no special system privileges are required. Unlike other popular kernel software, the Hurd has an object-oriented structure that allows it to evolve without compromising its design.

This collection of sentences are taken verbatim from the Hurd main page. These sentences were re-arranged to make a point. The paragraph above details supposed advantages of using a micro-kernel design; Linux and other free operating systems usually use a macro-kernel design. The Hurd people think this is a design improvement on Linux. This description lists features (modify the kernel without rebooting) and architecture advantages. One can debate issues like macro-kernel vs. micro-kernel for days on end (and people do), what I am trying to clarify here is that because Linux is so well-established and capable, the Hurd project differentiates itself and clarifies why it is also a legitimate project right on the projects home page.

On the project home page for the OpenBSD project we find.

The OpenBSD project produces a FREE, multi-platform 4.4BSD-based UNIX-like operating system. Our efforts emphasize portability, standardization, correctness, proactive security and integrated cryptography. OpenBSD supports binary emulation of most programs from SVR4 (Solaris), FreeBSD, Linux, BSD/OS, SunOS and HP-UX.

Again taken verbatim from the openBSD home page. Again, note the emphasis on things that differentiate the project from the dominant Linux project. OpenBSD is famous for its focus on security, on the main page, it covers this advantage. It is also famous for supporting lots of architectures, again right on the main page. I hope that I am demonstrating, very explicitly, what it means to establish legitimacy in the context of a currently dominant project or projects. OpenBSD even goes so far as to list the projects and products that it competes with.

When multiple projects are legitimate it is reasonable for each project to compete for developers. Want to work with the most popular OS project? Choose Linux. What to work with a bunch of people obsessed with security? That would be OpenBSD. What to work on a micro-kernel? Work with Hurd.

But you should not work on Fredux. It has no advantages over any of these projects. The competition between legitimate projects is… well… legitimate. But the FOSS community will instantly attack anyone who pouches resources for a project that has no discernible advantage over an existing project.

In the medical FOSS community, we are particularly sensitive about it. We have a small community, and dividing it is a real problem. It happens too much already, for “legitimate” reasons. The holy grail in FOSS Health IT is the creation of a Electronic Health Record (EHR). We have several legitimate projects in this space. All of the legitimate projects are starving for resources like developers, documentation writers, clinical experts, funding sources and users.

What are the “legitimate” projects competing to build the best FOSS EHR? Sorry, but this post is already too long. That question merits a whole article on its own.

For now I want to talk about three projects in particular that together illustrate the problem of duplication of effort. Tolven, OpenMRS and ExampleProject [update 4-10-08 to protect the guilty, the name of the offending project has been removed]. Tolven and OpenMRS both have reputations for being mature, Java-based projects. Here is the description of Tolven.

Tolven EHR and PHR products are a suit of Java based technologies that use the latest in Open Source Java application toolkits to create a heavily Object Oriented, MVC EHR application. In short the Tolven strategy is to a very clean java-based EHR using solid OOP design with the latest tools.

Here is a description of OpenMRS.

OpenMRS is a community-developed, open-source, enterprise electronic medical record system framework intended to aid resource-constrained healthcare environments.

OpenMRS is currently implemented in Kenya, Rwanda, South Africa, Uganda, Tanzania, Zimbabwe, and Peru. Many of the developers are MIT trained and the design of OpenMRS is based on work done at the Regenstrief Institute. 2008 will mark its second year participating in the Google Summer of Code. The design of OpenMRS is carefully thought out to support a rural deployment model. OpenMRS is a very mature project. [added OpenMRS 4-10-08 to acknowledge the tremendous things the project has done]

A bit of downlow information about Tolven: At the time of the writing Tolven is going live in several installations. Tolven has a tremendously talented development team. Tolven has received substantial funding. They have an EHR and PHR already working.

On my short list of legitimate FOSS EHR systems (and believe me it is short), Tolven and OpenMRS are currently battling for the title of top Java-based project. To review; Tolven is funded, going live now, 12 full-time developers, working software = legitimate. OpenMRS is funded, deployed all-over-the-place, and backed by some of the top minds in Medical Informatics = legitimate.

ExampleProject is a Java-based EHR system using strict OOP principles and the latest Java software stack.

ExampleProject is in alpha. ExampleProject is primarily developed by one guy, who is working part-time. ExampleProject plans to have a release that can be used in a live environment by October of 2008 ( a little less than a year away from the writing) . ExampleProject is not legitimate.

In short, ExampleProject appears to be a Fredux. What is worse, ExampleProject is constantly put forward as a legitimate project on LinuxMedNews, EMRupdate and the openhealth mailing list. Those forums are intended to be open and so censoring ExampleProject news would be unethical. The problem is that it appears to the un-initiated as though ExampleProject is important, when in fact it is exactly the opposite, an illegitimate project. ExampleProject has attracted at least two other part-time developers, and seems to be succeeding in acquiring attention and resources that properly should go to Tolven, OpenMRS or some other project. ExampleProject is guilty of the sin of duplication of effort. Everything they are doing has already been done, and well, by another project in the same programming language. Every developer that works with ExampleProject instead of working with Tolven or OpenMRS, is accidentally wasting their time. The same is true of beta testers, investors, and others who might be interested in working with a project. Because Tolven and OpenMRS continue to improve, and a greater rate than ExampleProject, it is very unlikely that ExampleProject will ever catch up. While it is fine for the project manager of ExampleProject to waste his time, it is not OK for him to trick others in the community into wasting time with him.

I have discussed this issue with the ExampleProject project manager. Before I mentioned it to him, he had never heard of Tolven. After spending about ten minutes on the Tolven site he said that the big difference between Tolven’s efforts and his own was that ExampleProject was using Swing, where Tolven was using AJAX. This seems like it might be a real legitimate technical difference, except that Tolven would likely be very happy for someone from the community to step up and create a thick-client using Swing. They would probably say something like “We do not have time to work on both AJAX and Swing interfaces. But our software design should allow any ‘viewer’ type. If you feel Swing interfaces would be better, and were willing to write it yourself, we would be happy to modify our interfaces enough to make that possible.” Even if Tolven were unwilling, the great thing about open source is that you can prove your technical points without the permission of the project manager. If Tolven or OpenMRS were shown that a Swing interface was better than an AJAX interface, they would probably adopt Swing.

A great example of a project that has spawned a sub-project specifically to address differences of opinion regarding user interfaces is the Kubuntu project, which is Ubuntu, only with KDE instead of Gnome.

When a separate project is justified is a complex issue. In fact, I gave Neil Cowles of the Tolven project a stern lecture about why he should be working through the OSCAR project, which was the dominate Java based EHR project before Tolven came on the scene. Since that lecture Tolven has proven to me and the community generally that they are legitimate. It is possible that ExampleProject, despite its small beginning, will grow into a mature open source EHR. They could prove me wrong.

Right now I would guess that ExampleProject is about one year and a million dollars of development behind Tolven or OpenMRS. If you are a developer, clinician, investor, entrepreneur, or administrator you should probably work with Tolven or OpenMRS if you want a Java-based FOSS EHR. (I will update this article if this ever changes).

I hope I have accomplished two things at this point. I hope I have shown, through a hypothetical example and a real one, examples of when duplication of effort is a problem. I also hope that I have said enough about ExampleProject to steer those who might want to work with a Java-based EHR to Tolven or OpenMRS.

-FT

(updated to clarify content and remove offending project name in April 10, 2008 if you care, just go read an archived version to find out who ExampleProject was)

(updated for readability Feb 27, 2008)