God this kind of thing pisses me off.
Brian wrote to also remind me…
Tell folks to come to the “interoperability showcase” in exhibit hall C, in particular to the NHIN and Connect area, where we are presenting on Connect with 60 other partners (most non-Fed) who have piloted or set up exhanges with NHIN standards, most of them with Connect, many of them by integrating with other Open Source med software.
CDC has a talk on the Biosense project.
I wish I could see those talks.
You should stop by booth 233 at the Interoperability Showcase and see MOSS demonstrate OpenPIXPDQ, OpenXDS and OpenATNA. MOSS also has a regular vendor booth numbered 7470. DSS is at booth 2521. PatientOS is at booth 4124 with orange shirts and lunch bags…
Medsphere and ClearHealth abstain this year (I think). I know the Mirth guys are around too.. If you are there and want to get ahold of them, send me a mail and I will do my best to get an introduction…
There was a code drop for a population health tool.
If you are at HIMSS 2010 and care about Open Source, let me know… I would be happy to add you to this post..
I meet Dr. Koppel at the last Indivo X meeting at Harvard. After hearing about his work I realized… this man must be the one of the saddest people in the industry, it is basically his job to study the interface between people and EHR systems. He is the guy who documents just exactly how EHR systems fail. e-patient Dave is blogging about a talk that Dr. Koppel gave.
EHR software famously under-performs.
As Dr. Koppel points out, from the perspective of the clinicians, the design of the EHR systems are pretty bone-headed. The standard answer from vendors is “we cannot fix that” until they have a financial incentive (like a lost sale or contract) to get something done.
For instance, Dr. Koppel points out that there is problem with the sorting of drug dosages drop down in an EHR. For fun I can show you how it would actually look:
Hard to imagine why the options would be in that order, why doesn’t it look like this?
The reason it looks like this is because the software is sorting alphabetically by written name… you can see it more easily if you write it like this:
But why is this happening at all? Why wouldn’t the software automatically do the right thing? I could imagine several possibilities. Perhaps the dosages are stored as strings and then converted to numerals for display. Perhaps this decision was made because there is a mix of numerical and text data in the doses field of the underlying database. Something like “seven milligrams” and then “seven milligrams time release” or some such. But I do not know. More importantly, Dr. Koppel likely does not know, and the clinicians who have to carefully choose a dose using choosers like this every day… do not know why the system is designed like this. I could be wrong, Dr. Koppel could know… but if he does, it is because he has been told by someone who can read the sourcecode.
I think we should pause at the irony here. If we could name the chief endeavor that modern medicine is undertaking one might say “crack the DNA code”. Our cells are “programmed” with a code that until recently we could not read and that we still cannot comprehend. We are seeing “through the glass darkly” into our own cells for a thousand natural reasons.
However, we tolerate a situation where clinicians see “through the glass darkly” into their own health software. Why? Because the have decided to use proprietary software. Why do they make that choice? It seems so illogical that it makes me dizzy. I wanted to respond specifically to some of the things that were said in the presentation:
Dr. Koppel: Customization is a sales gimmick and not meaningful.
The only way to make customization meaningful is to have full source code access with the right to modify the code running in the hospital.
Problems could be fixed by smart 14 year old.
If and only if they have access to the sourcecode. The insight here is that is a question of “access” and not “complexity”
Let me take a close look at how open source licensing impacts each of these.
The simple reality is that the funding from ARRA will go towards installing software that will stagnate and rot in the hospitals and clinics across America precisely because clinicians do not understand the implications of software licensing. Dr. Koppel focuses on “the software contract” which is mostly an irrelevant afterthought. Unless the software license allows you to fire the software vendor and get one that will reorder your lists correctly, the contents of the software contract are irrelevant. The right to fire in an Open Source software license has teeth. It changes the power dynamic in ways that the contract cannot.
My grandfather once told me never to play another mans game. “If his game is pool, play him in chess. If his game is chess… play him in pool.” Clinicians are losing the software game again and again, they need to stop playing the game that has been setup by the proprietary software vendors.
One of the things that I love about blogging is that it often puts you directly touch with those people who are influencing your thinking. This is just what happened when Microsofts Shawn Hernan sent me an email response to my recent blog post about the security of Open Source Healthcare Software. As it turns out, he also spent some time at the Air Force Information Warfare Center in San Antonio, we probably passed each other in the hall at some point.
Shawns original article, Microsoft’s Many Eyeballs and the Security Development Lifecycle had spurred me both to kibitz some of his points, and to consider their implications in Open Source healthcare software.
Shawn wrote the following to me and gave me permission to quote him here:
I saw your comments on my blog, and just wanted to write a quick note to thank you for the positive comments and the constructive and reasonable criticism.
It is of course unfair to lump all open source projects under one umbrella, and I regret leaving that impression. The various fights that people have over the different open licenses is all the evidence that anyone needs to see to realize that the only thing that many FOSS projects have in common is their “FOSSness.” And it’s clearly true that there is some highly competitive FOSS software.
I also like your characterization of “the game” for developers. I ran the security training program at Microsoft for a while, and while it was successful, I quickly learned that a corporate training program leaves a lot to be desired. We have a team called “Engineering Excellence” and they turned me on to the ideas of human-performance management, and I wrote about some of my experiences here: http://blogs.msdn.com/sdl/archive/2008/05/29/sdl-training.aspx. I think this is not too far from your idea of “the game” as I understand it.
I will quibble some with your ranking, though, and the implicit assumption that Microsoft is all a single culture. While the cultures differ from team to team (Windows v Office v SQL), all the big teams have a security-obsessed subculture and are all subject to at least the minimum SDL requirements, and I feel confident in saying that all our teams are at the very least competitive with the best open-source projects. I would invite you to look at the security track records of SQL Server, vs. MySQL for example.
But I do agree that the #6-style vendors may try to adopt my argument; I’m not trying to claim that proprietary software is inherently more secure than open source, but merely trying to challenge the meme that openness, all by itself, conveys magic security properties.
I agree with Shawns point here. The meme that openess all by itself is a magic bullet for security improvement is dangerous precisley because it interferes with “games” that can make Open Source more secure.
When someone says “The Many Eyeball effect” the right counter should probably be, “The Many Eyeballs Actually Looking”. Secure Open Source does not happen because people merely can look, it happens when they use their rights under the license to actually take a look. Openess is a critical first step in the process, but only the first step. Often, as I look at how money is being spent in Open Source in healthcare, I see far too little being explicitly spent of efforts to make things more secure. I think part of the reason is that people just assume that it will happen. When it doesnt you end up with insecure code that people trust more, because it is open.
I hope it is clear why this is an especially bad problem for Open Source Health Information Software. People really need to be able to rely on Health Information being private. There are two reasons for this. First, it could actually hurt someone if their health information got out. Second, our culture reacts really poorly to violations of trust, and, even if a software developer is not directly at fault for private information getting out, the reputation consequences are very serious.
Our community has already been damaged by this effect. The VA’s reaction to security breaches was part of the motivation for centralizing VistA development, which has lead to a serious stagnation in VistA development within the VA. The current VA administration is slowly reversing many of those decisions and opening development back up, but “security drama” has already seriously damaged VistA development.
We should be listening carefully to Shawn here. His point is, very simply, that merely being Open Source does not ensure security. We need to find a way to create a “game” for Open Source healthcare developers that automatically improves security. Without some kind of built-in incentives, we run the risk of creating trust without justification. We need to take page from Microsofts book here, at least to a certain extent.
We need to work to convince those of us who are “feature-focused” to balance that with a “security-focus” if we are to be succesful long-term.
Recently, Shawn Hernan wrote a piece on Microsoft’s security blog that argues that the “many eyeballs’ effect does not ensure that open source software is secure.
His argument is excellent and while I disagree somewhat with his conclusions, his point is undeniable. his argument, if it can be boiled down to a quote is:
The key word in Raymond’s argument is can. I’ll concede that open source can be reviewed by more people than proprietary software, but I don’t think it is reviewed by more people than proprietary software.
The simple reality is that -most- Open Source software projects are -not- popular enough to create a sub-culture of project developers who devote themselves to fixing bugs and ensuring quality code. Note that just because Open Source software is popular with its users does not always ensure that the size and makeup of its developer community is large enough to encourage such a subculture.
This is where Shawns piece breaks down. He does not bother segmenting the various “Open Source” projects nor does he bother segmenting proprietary software companies. One of the most important lessons to learn about software development is that very very few proprietary software companies are capable of operating at the level that Microsoft does.
Lets imagine for a second what methods would produce the most bug-free and therefore secure software.
The most important issue for determining how buggy software will be is to determine what the “game” is for developers. “The game” is the basic motivational structure that each individual developer is subject to when writing the given software. This kind of assumes that there is some variance in the rate at which developers debug their own code, and the rate at which they review other developers code, based on some kind of reputation, financial, or pleasure incentive.
What happens if you make bug-fixing the primary focus of the game?
The primary culture of the development team would be to develop highly secure and bug-free code, then simple competition and social pressure, inside your core development team, would help ensure that you have secure code. If a new developer were to join the project/company and was seeking to establish credibility, he or she would know that finding a bug and fixing it would increase their own prominence, probably at the expense of the developer that wrote the bug. The original developer would know that and take every effort to slow down and write good code in order not lose credibility.
The problem with this “game” is that any Open Source project that took this approach would slow to a crawl. New features would be added very slowly, as developers spent lots of time reviewing, testing, and generally obsessing about bugs present in any portion of the code. In fact if secure code and bug-free software were truly the goal then this software project would intentionally continue to slow down as long as going any slower would produce better, more stable code.
Frankly, no reasonable business plan for a proprietary software company could ever tolerate such an intentionally metered pace. However, Open Source software presents the opportunity for any community culture to grow and flourish. The OpenBSD project is a project with a focus almost identical to what I described here. The OpenBSD operating system is objectively the most secure operating system in the world based on several measures. You might imagine that a project so obsessed with security and bug-fixing would have a web-page devoted to their thoughts and practices on the subject, and you would be right.
Shawn Hernan neglects to acknowledge that projects like OpenBSD are possible in Open Source but are impossible with proprietary software companies. But it is important that we not straw man Shawn’s argument. Proprietary Software companies have one substantial advantage over Open Source projects. They can pay developers to follow procedures that ensure high quality code and they can pay some developers to do nothing but professionally audit code. Microsoft is very good about this. But then they are in a pretty unique position regarding their software. You could restate Microsofts business plan as “protect the billions that we already make selling operating systems”. The profits from anything that the company is doing “to compete with Google” for instance, pales in comparison to the profits from selling operating systems. But Microsoft can recognize that if they do not compete with Google now, and at least sometimes on Google’s terms, Google will eventually begin to hit them where they live, by subverting the all important operating system profit.
Microsoft has developed strict code-review procedures, like the ones Shawn Hernan mentions, because every time a vulnerability comes out for Microsoft software, alternatives look better and better. At this stage in Microsoft’s history, they have decided that security is a financial priority, but this only after years of ignoring it in favor of other more profitable business priorities. Just like most Open Source projects are not focused on being fundamentally secure, most proprietary software companies do not have an financial incentive to invest in programmers and procedures that produce fewer bugs and more secure code. Especially in healthcare software, bug-free code is an afterthought. This is just the nature of for-profit endeavors. The focus is always on what makes the most money. Microsoft is now in a position where secure code will protect profits. Proprietary EHR companies are not in that position. They write code that helps them sell to doctors. Since doctors are typically irrational purchasers, the feature set and priorities of typical EHR companies are similarly irrational.
It should also be noted that just because Microsoft has an financial incentive to produce secure code in one product line does not means that this extends to all of its product lines.
There is another method by which Open Source projects can be more secure and bug-free than code developed by a proprietary software company with an financial incentive to produce solid code. Its pretty simple really, and Shawn has already acknowledged it:
But could “enough” code review, which might happen for popular projects like Linux and the Apache Web Server compensate for the more comprehensive approach of the SDL?….
Shawn is acknowledging here that there are differences between Open Source projects. Ironically both the Apache project and the Linux Kernel are often subjected to distributed attempts at systematic bug detection similar to the whole contents of SDL. A great example of this is the Security Enhanced Linux (selinux) project run by the NSA. The problem with software bugs is that they are often unimportant for the average user, but a cracker can still use them to break into the system. This point is made most eloquently in the classic paper by Anderson and Needham, Programming Satans Computer. Projects like selinux attempt to make other bugs less prone to becoming a security weakness, an important step. This is exactly the kind of comprehensive security improvement that, according to Shawn, auditing is not supposed to have that Microsoft’s SDL does have. But the many eyes principle is not limited to mere audits. The whole point is that anyone can could shoehorn bug-detecting methods onto Open Source projects. I will use the term “Audit” to embrace all of the bug-detection and fixing techniques that Shawn mentioned. Since simply human auditing is often the most onerous and unlikely task for all but the most motivated developers.
So we are now able to make a little ranking. Who has the most secure and bug free development practices and what kind of “game” are they using to achieve that security. I think it looks like this:
- Best: Security Obsessed Open Source: Open Source Projects that build-in bug-fixing at the expense of everything else, like OpenBSD.
- Really Good: Popular Open Source with audit teams: Open Source Projects that are so popular -with developers- that they can afford to create a security obsessed sub-culture within the development team. Projects like Apache, Linux, and MySQL (before… well you know…)
- Really Good: Proprietary Vendors who pay for audit: Proprietary software vendors who have a financial incentive (Microsoft) or cultural imperative (Fog Creek) to create a development structure that reduces bugs.
- OK: Audited Open Source: Open Source projects that are sufficiently popular with developers to actually get some kind of formal code audit, preferably both automated and by a programmer trained in testing.
- Crappy: Unaudited Open Source: Open Source projects that are essentially one-man shows, whose code is rarely used and even more rarely looked at. (The vast majority of Open Source software projects in existence fall into this category)
- Worst: Unaudited Proprietary: Proprietary companies without a financial incentive to pay for expensive testing and bug-detection essentially have a financial incentive to -ignore- bugs. For most proprietary companies a software problem is only actually a bug, if it prevented a sale or lost a client.
Shawn’s post correctly points out that Microsoft, which is a typically in group #3 is much better than Open Source generally which is typically in group #5. of course his argument will be embraced and touted by companies who are in group #6.
Which brings me back to my subject of passion: Open Source medical software. I believe, tragically, that most Open Source Health software projects are in category #5 (Unaudited Open Source). I think this is changing with companies like Medsphere and ClearHealth becoming more mature companies, they can afford to sponsor auditors for their projects. Others like MOSS and Indivo are starting out with a security focus. I think #2 is the right target for us since it is not clear that OpenBSD is a -better project- just because it is -more secure-. Linux is still more secure that Microsoft Windows, and it is developed at much greater pace.
Still, I think the lesson here is that the best security happens when people focus on the boring and mundane. Starting with reading someone else’s code and then moving all the way up to using software to make software more secure. We simply do not have enough security focus in our corner of the Open Source world and I have to admit it is partly my fault. Before coming to the world of Open Source Health Software I was trained in Information Security at the Air Force Information Warfare Center in San Antonio. That payed off with later security contracting with Rackspace, Verisign and Nokia. I have never been 100% devoted to code auditing in any of my previous roles, but I know enough by rubbing shoulders to know when I should be nervous. As I look around the Open Source Health Software arena, there is a lot to be nervous about. There are also upsides. Our community has embraced at least two other serious security people. Alesha Adamson with MOSS was trained as a security geek at one of the few NSA National Centers of Academic Excellence and Ben Adida with Indivo X has a PhD from MIT, under the Cryptography and Information Security group
Both of these people are fully cognizant of security research and they are in leadership roles in their respective projects, and in the community as a whole But is it important that the whole community learn from their strong kung-fu. We need to develop a kind of security focused sub-group within the Open Source health information movement. We need people willing to do security audits on not one but several important Open Source codebases. Perhaps we should be trying to get members of the larger security community involved with Open Source healthcare software auditing. I will be thinking more about this, but I would love to hear my readers thoughts on the subject.
She has taken an interest in covering Electronic Healh Record issues and has started to reveal just how problematic this industry is.
Her most recent article, Doctors Get Buyers Remorse, is her best yet. She has uncovered not one but several of the kinds of stories that industry insiders like me view as typical. This may not be as glamours as uncovering that big story about the congressman paying his mistress…. but this is exactly the kind of documentation that people like me need when we claim that basically the proprietary EHR industry is broken.
I love it.
Ben Adida the project manager for the Indivo X project has announced the first public code drop of Indivo X on the Indivo developers mailing list, two days ago. Since then I have been spinning up a Rackspace Cloud instance to see if I could get it up and running.
I do not yet have write access to the wiki to make notes, but I documented all of the issues I had using Sidewiki so if you want to use an identical setup, I might save you an hour or so. Frankly, most of the hiccups I have are due to fact that I am most comfortable using PHP/MySQL/RedHat rather than the Ubuntu/Python/PostgreSQL setup used by Indivo X. After clearing some mental cobwebs, I had my own Indivo X PHR Platform installed and running..
Here is the obligatory screenshot, sanitized for spammers…
Of course, there are no applications to install yet, so “get more apps” does nothing. But the basic Indivo X platform is available, and will be going through the standard alpha/beta/1.0 process over the next few weeks/months.
Why this matters
Eventually there will be a dominant health platform upon which most personal health applications will be built. These are the applications that patients will use directly as health consumers.
This is the core enabler for the e-patient, it is at the heart of “Health 2.0”
It is the PHR or Personal Health Record. This health records of the future will be massively connected, like the Internet today, and the same way each of us has a primary email address, where our email records all end up, in the future we will all have Personal Health Records. Just like there is a competition between email providers, there will be a competition between PHR platforms. For many years, you can count on it being pretty difficult to move between these platforms however, alot more like getting a Mac to talk to a PC and vice-versa, than moving your email provider.
The dominant platform might be Google Health, HealthVault, or some kind of layer on the Health Internet (perhaps a spin-off from the Mirth or CONNECT projects)
However, my money is on Indivo X.
Indivo X is the newest PHR Platform engine developed jointly by Harvard Medical School , Children’s Hospital Informatics Program and Intelligent Health Lab. Indivo X is a complete reimagining of the original Indivo PHR, which was already one of the most successful Open Source PHR applications available. It was the codebase behind Dossia, which is one of the three “big” PHR deployments (the others are Google Health and HealthVault). Indivo X comes from impeccable stock.
Indivo X is dramatically different the original Indivo project in several important ways. First it is entirely rebuilt using the Python Django + PostgreSQL stack. Second it has been redesigned to be a PHR-platform, PHR functionality in Indivo X will be provided by modular applications, applications that could even be hosted on other computers. Indivo X is designed to create a market for PHR sub-applications using unifying PHR data abstractions provided by Indivo X. The notion for Indivo X is almost entirely based on the notions first outlined in now-famous NEJM article. I had criticized the article because it does not fully consider the implications of the proprietary trap for health software. However, this release renders most of my comments moot. These guys are trying to strike a balance between a stable platform and respecting software freedom and doing a more than decent job of it.
They have invited me to code on their development server, which has been up for a while, but I have very limited time due to my own skunkworks project (love a secret….) , and I can only afford to contribute when I can see both ends of the code. Still my skunksworks application will be consumer facing health application (Open Source eventually… of course) that is suitable to interface with HealthVault, Google Health and of course Indivo X. Given my feelings on software freedom, you can bet which one I plan on integrating with first….
An alpha code drop is a big day for any project and I know that the Indivo team has been working hard on this for months now. Congratulations are in order.
I am utterly not surprised to hear that OpenMRS is shining in Haiti.
This reminds me of the tremendous reponse that the VA had to hurricane katrina using VistA. For fun you should ask those involved for the inside scoop of how VistA enabled an entire hospital to uproot and move over the course of a single week.
Sometimes people do not really understand why we need software freedom in healthcare. These are two perfect examples.
Can you imagine the headache that per-seat or per-doc or per-patient EHR licenses would have caused in -any- haiti clinic? Of course they could always -ask- the vendor for temporary seat licenses, and because the vendors are decent human beings they would probably give them to them. Of course that only works when the phones work or the Internet is up.
Emergencies highlight the fact that health software users may have -very- different needs than the software vendor’s vision or even their own understanding. I know that the OpenMRS project will change substancially in response to the earthquake in Haiti. More importantly those changes will spread to other areas of the world… but those other users of OpenMRS will get the haiti lessons -before- the mudslide/tsunami/earthquake/bombing happens in their area.
In fact I can just imagine and administrator setting up OpenMRS for the first time and wondering “Hmm why would you ever need that???” and ten years later, when those features make OpenMRS better able to handle a disaster in that area, the same administrator will say “Ohhh… that’s why….”
Everytime I hear about something like this from the OpenMRS project I feel again guilty that I am not more involved….
It looks like WorldVistA is, for now, holding fast to the GPL and AGPL for VistA licensing. I have been a vocal advocate for compromising with DSS and Open Health Tools around the LGPL. The LGPL would allow for some innovations to be licensed under the GPL, and others, in the core of VistA to be compatible to bundle with proprietary software.
Recently, Skip McGaughey was quoted in modernhealthcare as saying:
“I believe it’s all about community-building,” McGaughey said. “I believe people have focused too much on technology and licenses and they need to focus on the care of individuals. If we can switch the focus from licensing and technology—the VistA community has a tremendous opportunity to fundamentally alter care throughout the world.”
“They’re starting from a base that has a tremendous knowledge base, built by care providers, tested and modified over a long period of time,” McGaughey said. “So, the opportunity is tremendous. So what we have to do is change the focus and quit worrying about the individual ‘me’ and talk about the ‘we’ together,” he said.
“If we enable an environment for people to collaborate in building infrastructure that everybody can use, to share the expense, what we can do is build the integration and interoperability and build a collaborative spirit,” McGaughey said. “Then people can climb the value stack to provide added value that can make money.”
It should be noted that I was not at the talk and did not hear exactly what Skip said. I know Skip and I know that he is a good guy, I think he intended to bring a message of reconciliation regarding licensing which is very good. I may actually agree with Skip’s position, but I cannot agree with this quote. While I am in favor of compromising with Open Health Tools, the position of WorldVistA on insisting on the full GPL is not unreasonable and it is certainly not anti-people.
Lets be clear, when you talk about proprietary friendly licenses in medicine, you are not talking about a way for people to “make money” or “earn a living”, you are talking about a mechanism that traps software consumers into a monopoly relationship with a software provider. Proprietary software in healthcare is so famous for abusing this monopoly position to the detriment of its clients that the issue is being investigated by congress and is even the subject of in-depth lampooning.
To trivialize licensing and indicate that is about “people” is typical and insincere. The software license defines the basic power structure of a relationship between software developer and software consumer. Full copyleft ensures that the developer and the consumer are always equals. Proprietary licenses ensure that the software vendor is in control. Open Source licenses that allow for proprietarization are a grey area. If software consumers are careful only to use Open Source components, they can maintain a balance of power, but if they ever allow a proprietary module into their ecosystem, then the license for that module puts some vendor back in the drivers seat.
If there was an “open” movement in the prisons around the world so that all prisoners were limited to just one shackle, they would still remain prisoners. Similarly as long as one software vendor can dictate terms to a clinic or hospital, they have a problem. Proprietary vendors who do not abuse their clients are like kind wardens. Just because they are nice a prisoner, does not change the fundamental power dynamic in the relationship.
The LGPL is a compromise precisely because it allows people who value freedom to work with people who are willing to compromise with proprietary vendors.
When you start hearing people saying things like “value stack” and “let people make money”, you are hearing the argument that being trapped is sometimes OK, if what you get for it is worth it.
This kind of power dynamic is precisely what prevents communities from trusting each other and cooperating. If you want to create community, you better not ignore licensing concerns.
I know for a fact that the MOSS team has been working on this for years. Completing these tests, and making sure they actually work at a Connectathon takes months of preparation and several frantic days of performance. In this respect, the Connecathon is something like a professional sporting event, one where you win by cooperating instead of competing.
This is an extremely impressive achievement and the MOSS team deserves applause. Because they are releasing most of these components as FOSS, the whole world is richer for their achievement!
Some have pointed out that not all of the Misys HIE tools are open source. This is quite true and I have updated the post to reflect that. MOSS makes no bones about being a hybrid proprietary/open source company. I am sorry that I gave an impression to the contrary.