Announcing the Patient Participation Conference

With pleasure I announce the 2010 Patient Participation Conference. This is a project of my employer Cautious Patient

The subject of the conference is simple “How to be an e-patient or e-patient caregiver”. This should be broad enough subject matter to cover any current discussion in the e-patient community. The basic principles of the conference will be:

  • An unconference: content generated as much by attendees as conference organizers
  • Low cost: preferably under $200 for the average attendee, we want real patients and caregivers to be able to attend out-of-pocket. That means we also need to low-glitz; you might get a t-shirt as a handout, but no ipod.
  • Scholarship some anchor and keynote speakers: we would like to find sponsors (and are willing to sponsor ourselves) attendees who would be important speakers, leaders in the e-patient community. We cannot afford to pay everyone, but we want to do what we can.
  • Everything video recorded, and published for free in near real time. This makes the conference for everyone, not just those who can afford to pay to attend
  • Small, short and intimate.

While most of the conference will be unplanned and free-form, generated in real time by people attending the conference, I want to have a few pre-planned anchor talks and keynotes that will serve to ensure that attendees are guaranteed to get at least a few things that are useful to them. With that in mind I would like to find basically two types of talks, either leaders in the community that are also strong speakers/teachers or talks with amazing content, delivered by people who are just OK speakers. I would rather hear a great story than a great speaker. Here are my personal biases as far e-patient speaker selection:

  • The User over the CEO – I would rather hear about a user who used HealthVault to improve his health than the guy who runs HealthVault at Microsoft.
  • Tactics over Strategy – I would rather hear “How to be a great diabetes mom” than “how to improve diabetes compliance in the U.S.”
  • Technology, but in the back seat – I think “How to really use Google Health” and “How to use gmail to manage your health” would be equally relevant. I would -love- a talk entitled “My health notebook kicks your PHR’s a**”
  • Hard stuff over the easy stuff – losing weight and lowering cholesterol is a good goal for almost half the country, but it is not the same thing as living with Diabetes or a failed kidney, and that is not the same thing as cancer or anything with “terminal” in the name. This also applies to controversial issues.
  • Evidence over Anecdote, but both are best – I want to make sure “the science is on our side”. I think talks that emphasize how patients can embrace and understand research are critical. I think a “tale of two e-patients” is a great example of a talk that nails this issue.

What other “biases” should an e-patient conference have in its anchor speaker selection? What specific speakers would you like to hear? If I am going to invite the “CEO” what companies are really enabling the e-patient movement?

Please help me out by contacting me with the answers, but you can also just leave a comment!

The Super Silo

I wanted to republish a post that I made to emrupdate here.

This was in response to a person who wants to help a doctor with what amounts to a super-simple note-keeping system for her nursing home patients. I should note that I think the mashups of technology that this person is suggesting is pretty clever. This is obviously a bright guy who is responding to a doctors sense of the record keeping problem. This is the dangerous first step of healthcare informatics, where a geek feels like he understands the requirements, but really has only scratched the surface. What feels like a good idea to this pair will create problems later that neither of them could effectively handle.

Here is the summary of the  original proposal:

Essentially I am using File Explorer (and optionally, Launchy) as the EMR system, and Notepad (with its .LOG function) as the note-taking application. Since she is the only physician in her practice and nobody else needs access to these files, I see no need to complicate things further. This system should also be extremely small in size, and trivial to back-up.

Is this sufficient, and is it HIPAA-compliant?

My response (which references more of the message)

> Is this sufficient?

No. For the love of all that is good. No.

What you are talking about is a system that is designed to make -your- doctor more effective, at the express cost of the ability to have information sent to or from other healthcare providers. The notion that this is acceptable is tragic. Your doctor is simply considering rendering her note into a format that would have maximal use for herself, at minimal cost, both in dollars and effort. Because she has so few patients, there will never be enough financial incentive to justify porting this data into a form that can be merged with other patient data. In short, you are considering creating a kind of super-silo.What happens to this data when your doctor retires? What happens to this data if you are not around to support the solution? Neither of you has thought this through, or even taken the time to consider that you to not even have the needed tools to properly think it through. I would encourage you to play a long game of “what-if” sprinkled with some “what-happens-when?” so that you can fully appreciate that you have not thought this through.

In twenty years people will talk about projects like this in the same terms as people do now about doctors participating in blood-letting and refusing to wash their hands because they were “gentlemen”. Your doctor is considering excusing herself from the responsibility to participate in Science and you are enabling her. The informatics community does not have that much figured out (sadly) but we know enough to say that this plan is a bad one.

Do not feel too bad. Yours is just an especially bad version of the bad decision that most doctors are making to use any proprietary EHR system. The problems that you will face will happen to most doctors in America, just a little later.

Please reconsider going down this path. I would recommend ClearHealth or OpenMRS as other simple and cheap EHR systems that you can use in your environment. But really you should look into VistA, either Astronaut or OpenVistA (which both have good licenses and OK installers) b/c the VA runs many nursing homes and VistA has a reputation of handling that use-case well.

How far we have not come

I meet Dr. Koppel at the last Indivo X meeting at Harvard. After hearing about his work I realized… this man must be the one of the saddest people in the industry, it is basically his job to study the interface between people and EHR systems. He is the guy who documents just exactly how EHR systems fail. e-patient Dave is blogging about a talk that Dr. Koppel gave.

EHR software famously under-performs.

As Dr. Koppel points out, from the perspective of the clinicians, the design of the EHR systems are pretty bone-headed. The standard answer from vendors is “we cannot fix that” until they have a financial incentive (like a lost sale or contract) to get something done.

For instance, Dr. Koppel points out that there is problem with the sorting of drug dosages drop down in an EHR. For fun I can show you how it would actually look:

Hard to imagine why the options would be in that order, why doesn’t it look like this?

The reason it looks like this is because the software is sorting alphabetically by written name… you can see it more easily if you write it like this:

But why is this happening at all? Why wouldn’t the software automatically do the right thing? I could imagine several possibilities. Perhaps the dosages are stored as strings and then converted to numerals for display. Perhaps this decision was made because there is a mix of numerical and text data in the doses field of the underlying database. Something like “seven milligrams” and then “seven milligrams time release” or some such. But I do not know. More importantly, Dr. Koppel likely does not know, and the clinicians who have to carefully choose a dose using choosers like this every day… do not know why the system is designed like this. I could be wrong, Dr. Koppel could know… but if he does, it is because he has been told by someone who can read the sourcecode.

I think we should pause at the irony here. If we could name the chief endeavor that modern medicine is undertaking one might say “crack the DNA code”. Our cells are “programmed” with a code that until recently we could not read and that we still cannot comprehend. We are seeing “through the glass darkly” into our own cells for a thousand natural reasons.

However, we tolerate a situation where clinicians see “through the glass darkly” into their own health software. Why? Because the have decided to use proprietary software. Why do they make that choice? It seems so illogical that it makes me dizzy. I wanted to respond specifically to some of the things that were said in the presentation:

Dr. Koppel: Customization is a sales gimmick and not meaningful.

The only way to make customization meaningful is to have full source code access with the right to modify the code running in the hospital.

Problems could be fixed by smart 14 year old.

If and only if they have access to the sourcecode. The insight here is that is a question of  “access” and not “complexity”

Let me take a close look at how open source licensing impacts each of these.

  • Open error reporting… and dissemination – The license gives you permission to publish bug fixes, and by implication the bugs that the fixes… fix.
  • Rapid repairs e-hazard tracking – Same story.
  • Meaningful meaningful use standards – Meeting standards of any kind can only happen when you try, fail and recode.
  • Meaningful evaluation – can only happen when you try different version of the sourcecode, and perform studies on which version works best.
  • Focus on clinical needs 1st and back office 2nd – Ha! Cannot say that the software license will change your basic motivation.
  • Interoperability for a clinical setting – Interoperability can only be achieved by coding to another code, this is what the project Laika
  • Certification as more than a sales strategy and sinecure – Certification is a poor workaround to not getting at the sourcecode. If you do not have access, certification is making promises it cannot really verify.
  • The simple reality is that the funding from ARRA will go towards installing software that will stagnate and rot in the hospitals and clinics across America precisely because clinicians do not understand the implications of software licensing. Dr. Koppel focuses on “the software contract” which is mostly an irrelevant afterthought. Unless the software license allows you to fire the software vendor and get one that will reorder your lists correctly, the contents of the software contract are irrelevant. The right to fire in an Open Source software license has teeth. It changes the power dynamic in ways that the contract cannot.

    My grandfather once told me never to play another mans game. “If his game is pool, play him in chess. If his game is chess… play him in pool.” Clinicians are losing the software game again and again, they need to stop playing the game that has been setup by the proprietary software vendors.


    Many Eyeballs Actually Looking: Microsoft’s Shawn Hernan responds

    One of the things that I love about blogging is that it often puts you directly touch with those people who are influencing your thinking. This is just what happened when Microsofts Shawn Hernan sent me an email response to my recent blog post about the security of Open Source Healthcare Software. As it turns out, he also spent some time at the Air Force Information Warfare Center in San Antonio, we probably passed each other in the hall at some point.

    Shawns original article,  Microsoft’s Many Eyeballs and the Security Development Lifecycle had spurred me both to kibitz some of his points, and to consider their implications in Open Source healthcare software.

    Shawn wrote the following to me and gave me permission to quote him here:

    I saw your comments on my blog, and just wanted to write a quick note to thank you for the positive comments and the constructive and reasonable criticism.

    It is of course unfair to lump all open source projects under one umbrella, and I regret leaving that impression. The various fights that people have over the different open licenses is all the evidence that anyone needs to see to realize that the only thing that many FOSS projects have in common is their “FOSSness.” And it’s clearly true that there is some highly competitive FOSS software.

    I also like your characterization of “the game” for developers. I ran the security training program at Microsoft for a while, and while it was successful, I quickly learned that a corporate training program leaves a lot to be desired. We have a team called “Engineering Excellence” and they turned me on to the ideas of human-performance management, and I wrote about some of my experiences here: I think this is not too far from your idea of “the game” as I understand it.

    I will quibble some with your ranking, though, and the implicit assumption that Microsoft is all a single culture. While the cultures differ from team to team (Windows v Office v SQL), all the big teams have a security-obsessed subculture and are all subject to at least the minimum SDL requirements, and I feel confident in saying that all our teams are at the very least competitive with the best open-source projects. I would invite you to look at the security track records of SQL Server, vs. MySQL for example.

    But I do agree that the #6-style vendors may try to adopt my argument; I’m not trying to claim that proprietary software  is inherently more secure than open source, but merely trying to challenge the meme that openness, all by itself,  conveys  magic security properties.

    I agree with Shawns point here. The meme that openess all by itself is a magic bullet for security improvement is dangerous precisley because it interferes with “games” that can make Open Source more secure.

    When someone says “The Many Eyeball effect” the right counter should probably be, “The Many Eyeballs Actually Looking”. Secure Open Source does not happen because people merely can look, it happens when they use their rights under the license to actually take a look. Openess is a critical first step in the process, but only the first step. Often, as I look at how money is being spent in Open Source in healthcare, I see far too little being explicitly spent of efforts to make things more secure. I think part of the reason is that people just assume that it will happen. When it doesnt you end up with insecure code that people trust more, because it is open.

    I hope it is clear why this is an especially bad problem for Open Source Health Information Software. People really need to be able to rely on Health Information being private. There are two reasons for this. First, it could actually hurt someone if their health information got out. Second, our culture reacts really poorly to violations of trust, and, even if a software developer is not directly at fault for private information getting out, the reputation consequences are very serious.

    Our community has already been damaged by this effect. The VA’s reaction to security breaches was part of the motivation for centralizing VistA development, which has lead to a serious stagnation in VistA development within the VA. The current VA administration is slowly reversing many of those decisions and opening development back up, but “security drama” has already seriously damaged VistA development.

    We should be listening carefully to Shawn here. His point is, very simply, that merely being Open Source does not ensure security. We need to find a way to create a “game” for Open Source healthcare developers that automatically improves security. Without some kind of built-in incentives, we run the risk of creating trust without justification. We need to take page from Microsofts book here, at least to a certain extent.

    We need to work to convince those of us who are “feature-focused” to balance that with a “security-focus” if we are to be succesful long-term.


    Security Reviews in Open Source Health Software

    Recently, Shawn Hernan wrote a piece on Microsoft’s security blog that argues that the “many eyeballs’ effect does not ensure that open source software is secure.

    His argument is excellent and while I disagree somewhat with his conclusions, his point is undeniable. his argument, if it can be boiled down to a quote is:

    The key word in Raymond’s argument is can. I’ll concede that open source can be reviewed by more people than proprietary software, but I don’t think it is reviewed by more people than proprietary software.

    The simple reality is that -most- Open Source software projects are -not- popular enough to create a sub-culture of project developers who devote themselves to fixing bugs and ensuring quality code. Note that just because Open Source software is popular with its users does not always ensure that the size and makeup of its developer community is large enough to encourage such a subculture.

    This is where Shawns piece breaks down. He does not bother segmenting the various “Open Source” projects nor does he bother segmenting proprietary software companies. One of the most important lessons to learn about software development is that very very few proprietary software companies are capable of operating at the level that Microsoft does.

    Lets imagine for a second what methods would produce the most bug-free and therefore secure software.

    The most important issue for determining how buggy software will be is to determine what the “game” is for developers. “The game” is the basic motivational structure that each individual developer is subject to when writing the given software. This kind of assumes that there is some variance in the rate at which developers debug their own code, and the rate at which they review other developers code, based on some kind of reputation, financial, or pleasure incentive.

    What happens if you make bug-fixing the primary focus of the game?

    The primary culture of the development team would be to develop highly secure and bug-free code, then simple competition and social pressure, inside your core development team, would help ensure that you have secure code. If a new developer were to join the project/company and was seeking to establish credibility, he or she would know that finding a bug and fixing it would increase their own prominence, probably at the expense of the developer that wrote the bug. The original developer would know that and take every effort to slow down and write good code in order not lose credibility.

    The problem with this “game” is that any Open Source project that took this approach would slow to a crawl. New features would be added very slowly, as developers spent lots of time reviewing, testing, and generally obsessing about bugs present in any portion of the code. In fact if secure code and bug-free software were truly the goal then this software project would intentionally continue to slow down as long as going any slower would produce better, more stable code.

    Frankly, no reasonable business plan for a proprietary software company could ever tolerate such an intentionally metered pace. However, Open Source software presents the opportunity for any community culture to grow and flourish. The OpenBSD project is a project with a focus almost identical to what I described here. The OpenBSD operating system is objectively the most secure operating system in the world based on several measures. You might imagine that a project so obsessed with security and bug-fixing would have a web-page devoted to their thoughts and practices on the subject, and you would be right.

    Shawn Hernan neglects to acknowledge that projects like OpenBSD are possible in Open Source but are impossible with proprietary software companies. But it is important that we not straw man Shawn’s argument. Proprietary Software companies have one substantial advantage over Open Source projects. They can pay developers to follow procedures that ensure high quality code and they can pay some developers to do nothing but professionally audit code. Microsoft is very good about this. But then they are in a pretty unique position regarding their software. You could restate Microsofts business plan as “protect the billions that we already make selling operating systems”. The profits from anything that the company is doing “to compete with Google” for instance, pales in comparison to the profits from selling operating systems. But Microsoft can recognize that if they do not compete with Google now, and at least sometimes on Google’s terms, Google will eventually begin to hit them where they live, by subverting the all important operating system profit.

    Microsoft has developed strict code-review procedures, like the ones Shawn Hernan mentions, because every time a vulnerability comes out for Microsoft software, alternatives look better and better. At this stage in Microsoft’s history, they have decided that security is a financial priority, but this only after years of ignoring it in favor of other more profitable business priorities. Just like most Open Source projects are not focused on being fundamentally secure, most proprietary software companies do not have an financial incentive to invest in programmers and procedures that produce fewer bugs and more secure code. Especially in healthcare software, bug-free code is an afterthought. This is just the nature of for-profit endeavors. The focus is always on what makes the most money. Microsoft is now in a position where secure code will protect profits. Proprietary EHR companies are not in that position. They write code that helps them sell to doctors. Since doctors are typically irrational purchasers, the feature set and priorities of typical EHR companies are similarly irrational.

    It should also be noted that just because Microsoft has an financial incentive to produce secure code in one product line does not means that this extends to all of its product lines.

    There is another method by which Open Source projects can be more secure and bug-free than code developed by a proprietary software company with an financial incentive to produce solid code. Its pretty simple really, and Shawn has already acknowledged it:

    But could “enough” code review, which might happen for popular projects like Linux and the Apache Web Server compensate for the more comprehensive approach of the SDL?….

    Shawn is acknowledging here that there are differences between Open Source projects. Ironically both the Apache project and the Linux Kernel are often subjected to distributed attempts at systematic bug detection similar to the whole contents of SDL. A great example of this is the Security Enhanced Linux (selinux) project run by the NSA. The problem with software bugs is that they are often unimportant for the average user, but a cracker can still use them to break into the system. This point is made most eloquently in the classic paper by Anderson and Needham, Programming Satans Computer. Projects like selinux attempt to make other bugs less prone to becoming a security weakness, an important step. This is exactly the kind of comprehensive security improvement that, according to Shawn, auditing is not supposed to have that Microsoft’s SDL does have. But the many eyes principle is not limited to mere audits. The whole point is that anyone can could shoehorn bug-detecting methods onto Open Source projects. I will use the term “Audit” to embrace all of the bug-detection and fixing techniques that Shawn mentioned. Since simply human auditing is often the most onerous and unlikely task for all but the most motivated developers.

    So we are now able to make a little ranking. Who has the most secure and bug free development practices and what kind of “game” are they using to achieve that security. I think it looks like this:

    1. Best: Security Obsessed Open Source: Open Source Projects that build-in bug-fixing at the expense of everything else, like OpenBSD.
    2. Really Good: Popular Open Source with audit teams: Open Source Projects that are so popular -with developers- that they can afford to create a security obsessed sub-culture within the development team. Projects like Apache, Linux, and MySQL (before… well you know…)
    3. Really Good: Proprietary Vendors who pay for audit: Proprietary software vendors who have a financial incentive (Microsoft) or cultural imperative (Fog Creek)  to create a development structure that reduces bugs.
    4. OK: Audited Open Source: Open Source projects that are sufficiently popular with developers to actually get some kind of formal code audit, preferably both automated and by a programmer trained in testing.
    5. Crappy: Unaudited Open Source: Open Source projects that are essentially one-man shows, whose code is rarely used and even more rarely looked at. (The vast majority of Open Source software projects in existence fall into this category)
    6. Worst: Unaudited Proprietary: Proprietary companies without a financial incentive to pay for expensive testing and bug-detection essentially have a financial incentive to -ignore- bugs. For most proprietary companies a software problem is only actually a bug, if it prevented a sale or lost a client.

    Shawn’s post correctly points out that Microsoft, which is a typically in group #3 is much better than Open Source generally which is typically in group #5. of course his argument will be embraced and touted by companies who are in group #6.

    Which brings me back to my subject of passion: Open Source medical software. I believe, tragically, that most Open Source Health software projects are in category #5 (Unaudited Open Source). I think this is changing with companies like Medsphere and ClearHealth becoming more mature companies, they can afford to sponsor auditors for their projects. Others like MOSS and Indivo are starting out with a security focus. I think #2 is the right target for us since it is not clear that OpenBSD is a -better project- just because it is -more secure-. Linux is still more secure that Microsoft Windows, and it is developed at much greater pace.

    Still, I think the lesson here is that the best security happens when people focus on the boring and mundane. Starting with reading someone else’s code and then moving all the way up to using software to make software more secure. We simply do not have enough security focus in our corner of the Open Source world and I have to admit it is partly my fault. Before coming to the world of Open Source Health Software I was trained in Information Security at the Air Force Information Warfare Center in San Antonio. That payed off with later security contracting with Rackspace, Verisign and Nokia. I have never been 100% devoted to code auditing in any of my previous roles, but I know enough by rubbing shoulders to know when I should be nervous. As I look around the Open Source Health Software arena, there is a lot to be nervous about. There are also upsides. Our community has embraced at least two other serious security people. Alesha Adamson with MOSS was trained as a security geek at one of the few NSA National Centers of Academic Excellence and Ben Adida with Indivo X has a PhD from MIT, under the Cryptography and Information Security group

    Both of these people are fully cognizant of security research and they are in leadership roles in their respective projects, and in the community as a whole But is it important that the whole community learn from their strong kung-fu. We need to develop a kind of security focused sub-group within the Open Source health information movement. We need people willing to do security audits on not one but several important Open Source codebases. Perhaps we should be trying to get members of the larger security community involved with Open Source healthcare software auditing. I will be thinking more about this, but I would love to hear my readers thoughts on the subject.


    Basically a bad deal

    Emma Schwartz is an serious investigative reporter who now works for the Huffington Post

    She has taken an interest in covering Electronic Healh Record issues and has started to reveal just how problematic this industry is.

    Her most recent article, Doctors Get Buyers Remorse, is her best yet. She has uncovered not one but several of the kinds of stories that industry insiders like me view as typical. This may not be as glamours as uncovering that big story about the congressman paying his mistress…. but this is exactly the kind of documentation that people like me need when we claim that basically the proprietary EHR industry is broken.

    I love it.


    Hello World with Indivo X

    Ben Adida the project manager for the Indivo X project has announced the first public code drop of Indivo X on the Indivo developers mailing list, two days ago. Since then I have been spinning up a Rackspace Cloud instance to see if I could get it up and running.

    I do not yet have write access to the wiki to make notes, but I documented all of the issues I had using Sidewiki so if you want to use an identical setup, I might save you an hour or so. Frankly, most of the hiccups I have are due to fact that I am most comfortable using PHP/MySQL/RedHat rather than the Ubuntu/Python/PostgreSQL setup used by Indivo X. After clearing some mental cobwebs, I had my own Indivo X PHR Platform installed and running..

    Here is the obligatory screenshot, sanitized for spammers…

    Screenshot of the alpha release of Indivo X
    Screenshot of the alpha release of Indivo X

    Of course, there are no applications to install yet, so “get more apps” does nothing. But the basic Indivo X platform is available, and will be going through the standard alpha/beta/1.0 process over the next few weeks/months.

    Why this matters

    Eventually there will be a dominant health platform upon which most personal health applications will be built. These are the applications that patients will use directly as health consumers.

    This is the core enabler for the e-patient, it is at the heart of “Health 2.0”

    It is the PHR or Personal Health Record. This health records of the future will be massively connected, like the Internet today, and the same way each of us has a primary email address, where our email records all end up, in the future we will all have Personal Health Records. Just like there is a competition between email providers, there will be a competition between PHR platforms. For many years, you can count on it being pretty difficult to move between these platforms however, alot more like getting a Mac to talk to a PC and vice-versa, than moving your email provider.

    The dominant platform might be Google Health, HealthVault, or some kind of layer on the Health Internet (perhaps a spin-off from the Mirth or CONNECT projects)

    However, my money is on Indivo X.

    Indivo X is the newest PHR Platform engine developed jointly by Harvard Medical School Children’s Hospital Informatics Program and Intelligent Health Lab. Indivo X is a complete reimagining of the original Indivo PHR, which was already one of the most successful Open Source PHR applications available. It was the codebase behind Dossia, which is one of the three “big” PHR deployments (the others are Google Health and HealthVault). Indivo X comes from impeccable stock.

    Indivo X is dramatically different the original Indivo project in several important ways. First it is entirely rebuilt using the Python Django + PostgreSQL stack. Second it has been redesigned to be a PHR-platform, PHR functionality in Indivo X will be provided by modular applications, applications that could even be hosted on other computers. Indivo X is designed to create a market for PHR sub-applications using unifying PHR data abstractions provided by Indivo X. The notion for Indivo X is almost entirely based on the notions first outlined in now-famous NEJM article. I had criticized the article because it does not fully consider the implications of the proprietary trap for health software. However, this release renders most of my comments moot. These guys are trying to strike a balance between a stable platform and respecting software freedom and doing a more than decent job of it.

    They have invited me to code on their development server, which has been up for a while, but I have very limited time due to my own skunkworks project (love a secret….) , and I can only afford to contribute when I can see both ends of the code. Still my skunksworks application will be consumer facing health application (Open Source eventually… of course) that is suitable to interface with HealthVault, Google Health and of course Indivo X. Given my feelings on software freedom, you can bet which one I plan on integrating with first….

    An alpha code drop is a big day for any project and I know that the Indivo team has been working hard on this for months now. Congratulations are in order.


    Cloud + VistA = Astronaut Shuttle

    I am not sure how many people out there have tried Astronaut Shuttle yet.

    First, let me get the caveats out of the way. Ignacio Valdes, the CEO of Astronaut (the company) hired me to do most of the cloud related work on shuttle. So I am financially biased on this. I chose to take the work, because I believe that the future of VistA is in the cloud. (If you do not know what VA VistA is, this might be a little muddy to you, and you might take a second and find out what VA VistA is…)

    I hope that my readers can glean just how important the idea that is being put forward here is. Many people have criticized VistA for being long in the tooth and not “hip” to modern innovations such as the Cloud, web 2.0 and health 2.0 concepts. On the other side, no one has been able to provide any results in the same league as the clinical improvements that VA has seen over the last several decades using VistA. In the universe of Health Informatics people touting beautiful new technologies have failed to outdo people using the tried-and-true but boring. What follows is a template for taking the best parts of the new platforms, and using them to improve the classic VistA technology.

    The History

    The idea for shuttle came to Ignacio and I over the course of many of our back and forth conversations about the cloud and about his Astronaut VistA installer package.

    For those who do not know already, VA VistA used to be a real bear to get installed and basically configured. Getting to “hello world” with this amazing MUMPS-based Open Source EHR system was really really difficult. Dr. Ignacio Valdes, of LinuxMedNews fame, set out to change that. His goal was simple, he wanted to make VA VistA usable, as close to instantly as humanly possible. He wanted to remove any cumbersome installation or configuration decision that had no meaning to the administrator making the decision. What was a process that took months for first-timers now takes mere minutes, simply download the astronaut rpm or deb installer to your Linux machine and away you go.

    Once Ignacio had taken the project all the way to consistently working DEB and RPMs, I noticed that he spent lots of development time spinning up Rackspace Cloud GNU/Linux instances, installing an Astronaut DEB or RPM, and then giving root access to a collaborator to debug something or another.  I began wondering if this process could be automated. With a little study I discovered that we could use paid AMI instances to give people access to GNU/Linux images pre-configured to run Astronaut VistA.

    The rest is history, I created Shuttle to work with Astronaut, and Ignacio developed critical features into Astronaut so that it would work cleanly in a cloud environment. When it came time to name the system, it was pretty obvious: what do you use to “launch” astronauts? Shuttle is a lot like RightScale , in that you use it to launch Amazon ec2 images, but specifically tuned to handle things required by an EHR, specifically VistA, in the cloud. One of its most important functions is as a key-server, so that you can have a fully encrypted VistA instance, without ever having the password live on the instance itself. That might sound like a lot of mumbo-jumbo to you if you are not technical, but what it basically means is that Amazon, even though they are hosting your VistA EHR for you, cannot access the health information stored on the database… only the EHR users can do that.

    The service is still in beta, we would like to have more feedback and several critical service offerings (like auto-magic cloud-based backup) in place before releasing the system as 1.0. But as this point we are pretty confident that we will be able to carry customer EHR data who want to use the beta system in live environments forward. (that is what makes it beta, rather than alpha…) I would love to hear comments from my readers about what features they would like to see next in a service like this, as well as what you think those features should cost. Medsphere just charged a 100-bed hospital $2 million dollars for 5 years of VistA support, so getting full-access to VistA experts is an expensive proposition. I want to be clear, the kind of hand-holding and face-to-face help that Medsphere is selling is not what you get with Shuttle. We are essentially taking a metered approach to EHR deployment… the first offering is just automated installation in the cloud… what services do you want next?

    It does not really matter if you want to host your instance of VistA in the cloud or not, the whole point of using an Open Source EHR system is that you are not married to your software vendor. If you feel like you no longer want to have your VistA EHR hosted in the cloud, all you need to do is copy your EHR to your own server, and then turn off your cloud server and stop paying. That means if you just want to spend a day or two playing with VistA to see if it is right for you, you can do that, and then turn off your cloud server and decide if you want to go to the trouble to install it locally.

    I am in a pretty decent position where I can afford to work only on projects that I feel are truly revolutionary. Think about it, other vendors are charging several hundreds of dollars per month -per doctor- to get access to an EHR system. Using Astronaut and Shuttle, we can charge about $100 per month for an entire EHR. That’s a minimal markup (once by Amazon and once by Astronaut) on the base cost for the hardware itself. An EHR that can run an entire clinic or hospital. While some people will not be able to live with the limitations of the cloud (you have to have your data off-site for instance) for those who can tolerate the cloud, can get access to extremely high quality software at near-hardware costs.

    Besides Amazon ec2, Shuttle would not have been possible without the excellent Ubuntu images from alestic and the php Amazon AWS library CloudFusion. Obviously none of this would have been possible without the Astronaut server and client installers, projects which are in turn indebted to OpenVistA and WorldVistA. Standing on the shoulders of giants.

    The Idea

    Before Shuttle, installing VistA required considerable systems administration know-how. With Shuttle you can start a VistA server, in the Amazon Cloud in a few minutes. You do not need to see a command line once. Anyone can now use a simple web-interface and have access to VistA, which is arguably the best Electronic Health Record System in the world

    VistA is in two parts, a server and a bundle of clients that installs on Windows XP (the most famous of which is CPRS). Setting up the VistA server requires access to GNU/Linux or  proprietary software under Windows. The Open Source Astronaut VistA installers make it easy to install the VistA server on GNU/Linux. But setting up a separate server to test an application is a bother for those who no how and impossible for those who do not know how. Either way, using Shuttle, you can just start an instance of OpenVistA or WorldVistA with a click of  a button.

    The VistA instances do not just have the VistA server installed, they also have a version of the Astronaut VistA client installer available for download from each instance. Each instances compiles the installer to be pre-linked with the server that it is downloaded from. The end result is that you download and install the client from an Shuttle VistA instance, it will auto-magically talk directly to that instance. All traffic to the server is encrypted and all data on the server is encrypted, as per HIPAA/ARRA regulations.

    What does that mean? It means for the cost of renting a small server at Amazon (about $100-$150 per month), you can instantly have access to an entire VistA server. That VistA server, encrypted, in the cloud,  will allow you to download a client pack to every computer in your clinic or hospital. This is as close as you can get to instant VA VistA. But rather than let me continue blabbing about this idea, let me show you how it works…

    Happy Launching!


    To Senator Grassely on EHR problems

    Senator Chuck Grassely has sent out some letters to several proprietary EHR vendors asking some pretty direct questions. Here is the relevant excerpts.

    Over the past year, I have received complaints from patients, medical
    practitioners and technologies engineers regarding difficulties they have encountered
    with the HIT and CPOE devices in their medical facilities. These complaints include, for
    example, faulty software that miscalculated intracranial pressures and interchanged
    kilograms and pounds, resulting in incorrect medication dosages.
    In addition, it has been reported that HIT/CPOE manufacturers rely on a legal
    doctrine known as “learned intermediaries,” to shift responsibility for errors in the HIT
    systems to physicians, nurses, pharmacists, and other health care providers. The
    manufacturers allegedly argue that the health care provider should be able to identify and
    correct errors caused by the software. It has also been reported that HIT/CPOE contracts
    with medical facilities may include “hold harmless” provisions that absolve
    manufacturers of these products of any liability for errors that are allegedly HIT/CPOE
    system or software failures. These contracts may also include “gag orders,” which
    prohibit health care providers from disclosing system flaws and software defects.

    Furthermore, it was also reported to me that there is no system in place to track,
    monitor and report the performance of these systems/devices, which could impact a
    health care provider’s ability to make informed decisions regarding the implementation
    of an HIT/CPOE system.

    To start lets bullet these complaints:

    • faulty software with dangerous results
    • avoidance of liability using legalize
    • gag clauses
    • no open data on bugs and problems

    I submit that these are problems arise from the quasi-monopoly that these companies aquire with their software licenses. A definition of monopoly is:

    a market in which there are many buyers but only one seller

    The government does a pretty good job of breaking up simple monopolies, monopolies where there is simply one provider of a service and that is all there is to it. The government does a pretty bad job of preventing what you might call “staged monopolies”, which is more commonly referred to as “vendor lock in”.

    I love simplified examples. When you are on the outside of a movie theater, there is a competition for your movie dollar. You can drive to another theater, you can go home and watch a rented DVD, or a movie downloaded from the Internet. But the moment you pay for the ticket, the competition portion of the movie experience is over. If you looked at the items available from the concession stand -inside- a movie theater you will see a clear pattern:

    • The items available are all high-profit items. Healthy sandwiches require lots of labor relative to popcorn, but would not sell for much more…
    • The items available are all enticing, but what is enticing might not be very good for you.
    • Everything is overpriced.
    • The service is typically non-existent.

    Once you have purchased your movie ticket, the movie theater has won the competition and also earned the ability to overcharge you for everything else. Also, the movie theater is very strict about forbidding outside food.. it has to be in order to protect its cash cow. Its not a particular theater that is at fault here, it is the basic structure of the deal itself. All of the theaters have the same deal, and all of the theaters offer the same high-priced, artery-clogging fare. They have to in order to stay competitive with each other.

    The movie goer knows that the basic deal they are making is a bad one. If they wanted to have a healthy movie, they would rent a DVD and stop by Subway on the way home.

    The problem with doctors purchasing proprietary EHR systems is that the problems that you are seeing are the direct and necessary consequences of the monopoly that a proprietary software license provides to the EHR software vendor. Proprietary software vendors are competing carefully -before- the EHR is purchased, but once the doctor has bought the system he is trapped.

    Why the gag orders? Because the EHR vendors compete only -before- the EHR purchase, they have a huge incentive to provide the doctors with slanted data during that stage. How do they do they do that? They chose one customer who is really happy with their product and they have tours of that facility and they write white papers about that facility and they provide that facility as a reference. They might have 95 facilities who are furious with them and 5 who love them, but if they have a gag order in the contract then they can use the 5 to provide a skewed view to new clients.

    You have already pointed out that there is no open system for reporting the flaws in EHR systems. All of the companies that you survey will give you lots of information about their sophisticated systems for tracking software errors. But these systems are closed, and as a result, there is no way for a particular doctor to know if the software problem he is having is common or unusual. There is no way for the doctor to recognize that he or she is the victim of systematic neglect or is the only person on the planet with a given problem.

    Gag orders and closed reporting systems are two tactics in a much larger struggle between proprietary EHR vendors and EHR consumers. The struggle is to control the information available to EHR purchasers. This is not the only way proprietary EHR companies skew the data. They use organizations like HIMSS and the EHRVA to provide an air of legitimacy and professionalism. There are organizations that provide EHR “reports” that are supposedly objective, however, these types of evaluating organizations typically also serve as “consultants” to EHR companies. In short the EHR companies pay off the “researchers”. There is no equivalent to “Consumer Reports” in the EHR market (although there are some organizations that are trying). If a doctor is reading a report comparing EHR vendors, that report was very probably made by someone who was paid by those vendors.

    There is a tremendous financial incentive to control information that impacts EHR sales, and lack of objective information is one of the big problems that your constituents are facing.

    Now lets talk about the software bugs. We can talk about to classes of software bugs, bugs that are so huge that they blow up a salesman’s presentation of an EHR, and all of the other bugs that are not that big. I can assure you that the average proprietary EHR system does not have many sales-destroying bugs.

    Bug-fixing isan expense, and a big one. Lets imagine that this year, software company X will discover 100 bugs in their software. Now, how will the “bottom line” be impacted if they fix 90 bugs vs 10 bugs. The answer? very little. There is simply no financial incentive to provide a greater reponse to EHR bugs. Their customers are trapped. When a doctor uses an EHR even for a short time, that doctor makes three important investments, time, money and patient data. Once they have chosen a vendor, it is almost always a worse deal to leave the vendor then it is to continue to accept poor service from that vendor. If they leave, they have to buy a new system, learn a new system and migrate patient data. It just costs too much.

    Its just like the movie patron. Every time I pay six dollars for fifty cents worth of coca-cola, I resent the movie theater. I -could- leave and go somewhere else, but then I would loose the price of the tickets, and not get to see that movie tonight; too expensive. So I suck it up and pay six dollars for a small coke.

    Again, it is not the proprietary EHR vendor itself that is at fault, it is the basic unfairness of the bargain.

    Your last concern was regarding the legal loopholes that EHR vendors use to avoid the liability that occurs when their software causes medical errors that hurt people.

    The simple reality on this issue is that no EHR company, proprietary or otherwise can afford to share medical liability with a doctor. A single death or serious injury that could be tied to the EHR vendor instead of the doctor could put the vendor out of business. If they could not avoid the liability contractually, they would have to insure against it, and the cost of that insurance would be roughly on par with cost of the medical malpractice that the doctor is forced to pay. If any EHR company is exposed to these levels of potential liability, then the stimulus money from ARRA will not make a dent in the new costs of EHR systems.

    Software bugs are a simple reality. EHR software bugs, have and will continue to kill people. This is a difficult thing for politicians to face, but that -is- an acceptable cost of moving to EHR systems. If I told you that by causing 100 deaths a year I could prevent all of the traffic accidents in the United States in the same year you would jump at that offer! The math is really that simple: how may people will EHR bugs kill? vs how many will be saved through pervasive availability of EHR technology?

    The problem is that right now, no one know what the true cost of these EHR bugs except proprietary EHR companies who stand to profit greatly by keeping the information secret and keeping the number of people killed by bugs as high as possible. Fixing bugs is a tremendous expense and the EHR companies have a financial incentive to only fix as many bugs as it takes to keep their clients from leaving. Because the cost of leaving is so high to EHR clients, the number of bugs fixed is low indeed. The current system incentivize proprietary EHR vendors to keep the information about deadly bugs a secret and to fix as few of them as possible.

    The solution? The doctors must discern that the deal is bad, and seek a better one.

    People are watching less and less movies at theaters. People have learned that there is a much better deal available. Buy a nice TV, rent the same movies for a fraction of the cost, cook whatever you want to eat and watch the movie from your comfortable sofa. The market is pushing people away from the crappy monopoly deal and towards the better deal without any monopolistic component. Movie theaters are responding..  now many movie theaters have removed rows of seating to make room for tables and are offering full restaurant style meals at reasonable prices… along with seeing a movie. A much better deal.

    For EHR vendors, the better deal is this: An open source EHR.

    Here is why an Open Source license prevents the problems you are mentioning:

    First, the competition never stops, the open source license give the EHR buyer the right to fire the EHR vendor, and hire another one, without migrating software. That means that the end of the sales process does not mean the end of the competition. If an EHR vendor tried to have a client sign a gag order, that client could find another vendor to implement the first vendors software offering. If the open source EHR vendor failed to fix a critical software bug, the client could find another vendor to do it, or even do it themselves. All open source software vendors of note publish bugs publicly, in fact they will even accept bugs discovered by people using the software who are not paying the software vendor. Open Source software naturally produces open bug reporting, competition for the fixing of bugs, and no gag orders (or other silly contract stunts). What about liability? By making the basic relationship fair, and focusing vendor competition on reducing bugs the software is made safer through the natural action of commerce, rather than the artificial safety provided by lawsuits. Critical bugs, like the ones mentioned in your letter, will be fixed overnight, or somebody will get fired. That’s a much better deal than ensuring that when someone is killed after 4 months of living with a known bug, there are more people to sue.

    But do not take my word for it.

    I would encourage you to send your letter and questions to Mike Doyle, the CEO of Medsphere (an important Open Source EHR vendor) and compare his answers to those provided by the vendors you have already sent letters to. Do not worry about time, it will only take Mike a day or two to answer the questions. Most of the information is already online, he could just send you a bunch of hyperlinks. Medsphere is not the only good Open Source EHR vendor, but their responses will be typical of the industry. I can provide you with 10 other Open Source EHR companies if you would like.

    What do I want you to do about all this? Nothing. Open Source EHR systems wins in the end anyways…

    Do think of me the next time you watch a movie at home…

    -Fred Trotter