The Power of Push


The NHIN Direct network has been criticized for lacking relevance for health information exchange. Specifically, Latanya Sweeney has submitted testimony to congress which has nothing good to say about either NHIN project. The paragraph I want to highlight says:

ONC’s website also describes NHIN Direct [11] as a parallel initiative underway [3]. The idea came from comments made by representatives from Microsoft and Cerner [12]. In current practice, two providers fax patient information as needed. So, the idea is to replace the fax with email that has secure channels to combat eavesdropping. There are numerous concerns with this design also. A glaring problem is its limitation. We cannot perform all meaningful uses with this system, so we will need an additional system, which begs the question: why build this system at all? For example, this design cannot reasonably retrieve allergies and medications for an unconscious patient presenting at an out-of-state emergency room (arguably a stage 1 meaningful use). Figure 2(b) summarizes concerns about these two designs. The NHIN Limited Production Exchange has serious privacy issues but more utility than NHIN Direct. On the other hand, NHIN Direct has fewer privacy issues, but insufficient utility. When combined, we realize the least of each design, providing an NHIN with limited utility and privacy concerns.

This  is not the first time that the NHIN Direct push-only model has come under attack, so I wanted to discuss this. Push-only means that A can send messages to B, but B cannot automatically get data  from A (that would be pulling). Email and Faxes are push models. Web pages are pull models (i.e. sent to you when your browser asks for them). The  benefits of both models are constantly debated in software design .

I am working on NHIN Direct, and not so much NHIN CONNECT, although I have great admiration for the project and many of my friends are working on that project. My experience with NHIN Direct, which has been excellent so far, has helped me to understand just how narrow-minded and short sighted these kinds of criticisms are.

Both projects, in so far as such a thing is possible while building technology, are taking a “policy-neutral” stance. That means that rather than defining policy in code, we try to code so that a broad range of reasonable policy decisions can be supported in a given protocol and codebase. But even under a given policy, there will be many many options to use these technologies in ways that are unexpected. So when anyone criticizes the “security and privacy features” of either CONNECT or Direct at this stage… it is typically by making certain poor assumptions about how the system will be actually used.

The most important poor assumption is to consider only standard uses of the technology when considering meaningful use. For instance, the NHIN Direct project concedes that mere usage of the NHIN Direct exchange will map to specific meaningful use requirements. Note the headers on that PDF to see that this map was contributed by my friend Will Ross and the Redwood Mednet team. In Open Source healthcare, as in Open Source generally, you see the same actors generating excellent contributions again and again. But these meaningful use mappings only consider the implications of mere use of the network, rather than considering anything that can be implemented on top of the network.

When people say the ‘Internet” what they usually mean is either email or the world wide web. In reality the “Internet” is a far richer technology space than this, but for most people only two of the thousands of protocols that operate over the Internet have become personally relevant: SMTP and HTTP/HTML. In fact as I say that, many of my clinical readers might not even recognize that SMTP, and sister protocols like IMAP, are the protocols that enable email, or that HTTP/HTML enable the world wide web. In fact both of these protocols rely on lower level protocols, like IP/UDP/TCP/SSL/DNS that enable the average user to surf and email.

But understand that the richness of the Internet, as we know it today, is not merely what the protocol implementations allow you to do directly (i.e. browsers let you surf the web and email clients let you read and send messages) but how those technologies are used. The web allows you to buy books on Amazon, win auctions on ebay and find dates on eharmony. Each of those website enables complex application functionality on top of the implementations of http and html.

It is easier to see how the web has more to offer than merely transferring hyper-linked web pages, to see the richness that is available at the application level that is not implied or assumed by the lower level implementations of the enabling protocols (that would be web-browsers and web-servers implementing http/html). Sometimes it easy to forget that we see the same thing with email. The email network does far, far more than merely send and receive messages . Like the web, higher level functionality is enabled by the lower level protocol driven functionality, in this case the ability to send and receive messages.

I wanted to highlight several things that you can do with email, that are examples of this higher-level functionality.

  • You can use an email account to prove that you are a human to a website. Have you ever signed up to a website that insisted that you give them an email address and then automatically sent you an email that had something to click on to prove that you owned that email address? I have done this so many times that I have lost count. This is “email for authentication”. Software often uses email messages to provide greater access to websites.
  • You can send messages to just one email address, which will then be sent to many other email addresses. Mailing lists can be pretty amazing software services, but fundamentally all they do is intelligently receive and re-send email messages. This makes email change from a one-to-one messaging system to a one-to-many messaging system. But it is implemented entirely with one-to-one messages.
  • If you push the mailing list even farther you can see that it can become something even more substantial, like craigslist, which pushes the envelope on email broadcasting and blurs the lines between email application and web application.
  • Programs can automatically send email messages when something changes, like Google Alerts tell you when the web has changed (or at least changed as-according-to Google)
  • You can have many email addresses and configure them to aggregate to one email viewing client, enabling separate relationships, and even identities to be managed in parallel. For instance your work email address really means your work identity, and your personal email means your personal identity, but you might forward both to the same email client and then answer and send messages as both identities at the same time.
  • You can use email to create a system for recycling things. Making it easier not to buy new things, and not to throw away working things. This is essentially email-enabled peer-to-peer conservationism.
  • Email clients are more than just programs we use to send and receive messages. We expect them to integrate with calendaring software. We expect them to allow us to extend them with other programs. People use powerful email clients like gmail to run their lives before people started to do that with gmail, they where running their lives with outlook or eudora.

Email is not just a method for sending messages. It is an application platform. Other applications that want to do something interesting can use email as a messaging component to achieve that greater goal.

I want to be clear. The NHIN Direct project has not settled on STMP, or email as protocol choice (although an S/MIME email is on the table). At this point we are not sure what protocol we will be choosing. But it does not matter, the point here is that NHIN Direct will at least act like, private, secure, identity-assured (at least for clinicians) email for sending clinical messages. You can expect that a NHIN Direct implementation will either be tightly or loosely integrated with a doctors EHR and a patients PHR in the same way that you have tight or loose integration between email clients and calendaring applications.

At this point it is best to think of NHIN Direct as a “cousin” to email. With lots of the same features and benefits but also limitations (to protect privacy) and new features (clinical integration, meaningful message signing, etc etc) that email does not have.

But the most important shared benifit between NHIN Direct and email will be the fact that you can build new interesting stuff on top of it.

Which brings us back to Latanyas first criticism. Will NHIN Direct support the ‘break the glass’ use-case (where your information can be gotten-to in case of an emergency) that Latanya mentions? No. Will software that implements NHIN Direct be able to use NHIN Direct as part of an something that provides break-the-glass functionality? Yes.

Very soon after an NHIN Direct network stabilizes, you will start to see this functionality addresses this use case. PHR applications like Google Health, HealthVault and Indivo X (the most important three PHr platforms) will probably develop break the glass mechanisms that work something like this…

I am an emergency room doctor and a patient comes in unconscious. In his wallet I find a card that indicates his PHR is held at

I visit and click the “break the glass” link. HealthVault asks me to enter my NHIN Direct address.. which is going to look a lot like an email address. So I enter (not a real address). HealthVault will have already performed extensive public key exchange with Methodist Hospital, and will be able to cryptically ensure that any address under that domain name (we call them health domain names.. since they will be used exclusively for this purpose) is in fact someone that Methodist Hospital vouches for, and they will have pre-approved Methodist Hospitals PHI handling procedures. Given that pre-arrangement of trust, they will know that they can securely send messages to any published Methodist hospital NHIN Direct address.

But they are not certain, at this stage, that I am in fact so they will send a message to that address with a link. I will click the link which will confirm with HealthVault that I am in control of that address, and that they should forward the contents of the PHR record. Now that they are sure that this is a valid break-the-glass request from a valid user at an institution that they have a trust relationship with, they will forward the record to the address.

They will also add a record to john’s PHR to indicate that I broke the glass. If this whole process was done fraudulently, John will know and there will be hell to pay for me personally for abusing my credentials and for Methodist Hospital for giving me a credentials to abuse. Current HIPAA rules and fraud statutes would be activated if I made such a fraudulent request, that was not in John’s best interest. People who abuse the system could be detected and sent to jail.

The whole process takes minutes and works even when the patient is unconscious.

Would that particular method answer the “break the glass” components of meaningful use? It seems like it would to me. Would this be the method that we end up using? I am not sure, but it would be something similar in spirit. Most importantly, it would be something  implemented on top of, and around, the messaging model provided by NHIN Direct.

All of that is to say: Push is Powerful. It is powerful because it does not need to work alone. It can be a component of a larger system that does much more. It creates the opportunity for innovation and greater functionality similar to the one provided by the original Internet protocols.

This is all true of the NHIN CONNECT project as well. The difference is that NHIN Direct is much simpler and has true parallels with the current fax and email systems. It is easier to see how NHIN Direct might change things because we are so familiar with its cousins, email and fax.

NHIN CONNECT offers much more functionality at the price of far greater complexity. Like the NHIN Direct system, and email and web before it, the NHIN CONNECT architecture will allow for innovation to occur on top of it. But it is doing much more work than NHIN Direct is.

For instance, if I were fully NHIN CONNECT enabled, I would be able to conduct a search for John Doe and find out that three hospitals had information that were not contained in the HealthVault record. NHIN CONNECT might be able to provide a merged view of that data for me, which is a much richer process than mere messaging can achieve. But that means that NHIN CONNECT must tackle the complex problems of sorting out which records actually belonged to John Doe and therefore deserved to be merged. It would make automated, but accurate, decisions that Jonathan Doe at hospital A was my John Doe but that Johnny Doe at hospital B was not… NHIN CONNECT  should understand that a blood pressure measurement that was in the data it gathered from HealthVault was or was not a duplicate of blood pressure readings that came from the hospital C EHR, that had the same date, but not the same time stamp. These kinds of issues, plus countless more just like them, are addressed or exposed by both the underlying NHIN protocols that CONNECT implements and by the CONNECT codebase specifically.

CONNECT uses push and pull and all kinds of other software models to do something very complex.

NHIN Direct just does push, but leaves potential complexity to higher level yet-to-be-made systems.

Some people think the NHIN Direct model is superior. Others think that CONNECT is better. I think we probably need both for different reasons.. which is essentially the ONC position on the matter.

But I wanted to be sure everyone was clear: Push has Power.


The Burden of Trust


I am a vocal participant on the NHIN Direct Security and Trust working group. Its a perfect place for me. I love Open Source healthcare, but my background was in InfoSec… and we never really forget our first love.. do we? At the NHIN Direct Security and Trust workgroup, I get to exercise all of my hats at once… and that is fun.

The purpose of NHIN Direct is to design an infrastructure for sending messages with clinical content between clinicians (and their patients). It is basically designed to be an email-like system for delivering health information. It is intend to eventually replace the current NHIN… which is the ad-hoc clinical fax network.

On a recent call, someone from the “Policy” department said something about our current plans to the effect of “I am not sure how putting the burden of Trust Decisions on individual providers will impact the ability of the project to replace the Fax network” I could not talk on the call… I was in a noisy airport… but I was surprised by that characterization of our work. In retrospect I can see how she would read what we are writing and come to the conclusion that we are putting new trust burdens on doctors… but in fact we want to lighten the trust burden they currently carry.

You don’t know the devil that you know

That is probably the most important point. The fax network comes with a very heavy trust burden. But we are used to it, so we rarely pay attention to it. This is a case of “acceptable losses”. Its kind of like Terrorism vs Auto Accidents. Many more people in the world are killed in car accidents each year than are killed by terrorism. The irony is that terrorism is much harder to fix than auto accidents. If the US Govt devoted the same budget to auto accidents that they do to the “War on Terror” we could probably prevent 99% of the auto accidents in the world. But we, as a society “accept” the burden of car crashes… because we are used to them. We have the same problem with medical errors… but that is another post.

So lets take a careful look at the “current trust burden” in the fax network. First, doctors do not actually deal with this problem directly. Typically they hire staff to do faxing. This isolates them from the problems that the “faxer” faces. It also means that they rarely hear of the errors.

“Faxers” fax to patients, and they fax to other clinicians. There are lots and lots of times when something that should have been faxed to Dr. Smith ends up going to Dr. Jones. We only hear about the most extreme cases. In fact before the existence of the NPI database, there was no reliable way to determine if a fax number was valid. If Dr. Adams wanted to send a record to Dr. Smith, his staff called their staff and wrote down the numbers. The numbers get jumbled, mislabelled and lots and lots of errors occur.

We do not hear of the cases where people were killed because information that was in a fax record was faxed to a wrong number. Perhaps sent to the “main hospital” fax line instead of the ER fax line where it was needed. These types of between-institution errors are almost impossible to detect, even the “big picture” at one large hospital is hard to sort out, and when you add another institution… no hope. Instead you get cases that are written off as “we did not know that X… oh well… nobody’s fault… nothing could be done”.

Then of course there is the assumption that fax lines are private. This is the farthest thing from the truth. Faxes, just like regular phone conversations are digitized and sent over the Internet. If a hacker gains control over a main router at a major Internet carrier, then they can re-route phone calls and faxes to themselves as well normal internet traffic. The fax network is actually going over the Internet right now… its just “obscured” rather than “encrypted”.

This is not the only problem with faxes, another problem is that institutions rarely have a firm grasp on how many fax machines are actually in operation. You can plug a computer modem into a wall and have a nearly undetectable new fax line… allowing “insiders” to send files to themselves via fax. In fact, phone lines can generally be re-purposed in to back-channel data ports in a number of ways, faxing is only one of them. Lots of my old Air Force buddies ended up at Securelogix, which is one of the top companies for phone security. They sell a telewall that can help prevent phone lines from being re-purposed. Its just what its name implies, a firewall for telephones. No large institution that I have every heard of that paid for a penetration test that include wardialing has ever had the wardialing effort return 0 rouge fax/modem instances. Clinicians should not assume that they understand their own fax infrastructure.

Even if you are really careful with who you fax to.. the current fax network is that it is difficult to maintain. Lets say that Dr. Smith sells his practice to Dr. Sneaky. If the fax number does not change, then Dr. Sneaky is going to get all of those faxes that were intended for Dr. Smith. Not good.

The problem with comparing the devil you know with the devil that you don’t know is that usually, you don’t actually know the first devil that well at all. The “trust burden” on the Fax network seems light because it is hopelessly broken  and we all just tolerate it.

A lighter burden

Which brings me to the “trust burden for NHIN Direct”. Our goal with regard to this burden is two fold:

  • When an NHIN Direct user makes a trust decision, it should me more reliable than the equivalent decision on the fax network.
  • Typical NHIN Direct users should be able to avoid directly managing trust at scale, making fewer and therefore better trust decisions.

The first one is easy. Without knowing exactly what standards we will be selecting at the time of the writing, I can already tell you that the security the NHIN Direct network will be an improvement over the Fax network. Moreover, it will provide more and better information to the users of the network than is possible with the fax network. Without going into the gory details, this is because PKI is better than post-it notes full of names and fax numbers for maintaining a secure information transfer.

The second one is a little tricky. What I mean by “trust at scale” is the problem managing lots of peer-to-peer trust relationships. If we have a NHIN where say, a third of all doctors in the Unites States participate, that is still probably over a million people. There is no way that you are going to get a doctor to make a list of all of the doctors that he/she does/does not trust taken from a million person list. Even trying to do peer-to-peer trust on a city level would not work. Hell I would be surprised if it would work even between two hospitals. (If you gave doctors the option to “not trust” some doctors at their own hospital… you would probably still get headaches). The fax trust management problem is a little simpler because you can sometimes aggregate to the organization… several clinicians share the same fax, but even that it is really difficult. Having to manage thousands of trust relationships dramatically increases the probability that you will get one of them wrong.

How do we fix that? We need trust aggregation points. So far there are two of these in our model. The first is at the organization level, just like faxes. Typical NHIN direct addresses for providers working in hospitals or clinics will look something like the “” part of the address is the “health domain name” and you could use that to trust all of the messages that came from that health domain name. The second way is with what we are calling Anchor CAs. For those familiar with the way CAs (Certificate Authorities) work with https, it is basically the same. The difference is that there will be no “automatically included” Certificate Authorities. When you login at amazon your browser makes a secure connection automatically because the person who makes your browser decide for you that you would trust Versign CA certificates. You can find out how your browser developer makes this trust decision for you… but they are still making the decision for you.

That model… where someone else makes your trust decisions for you… is not going to fly in healthcare. The stakes are simply to high to outsource trust in this fashion.

However, the notion of aggregating trust using Certificate Authorities is a good one. Lets imagine that my home town, Houston, decided to setup a Certificate Authority. They would decide on some reasonable policies for things like:

  1. Anti-virus (think Storm Worm not Influenza)
  2. Firewalls
  3. at-rest disk Encryption
  4. Password Strength
  5. Local Authentication (two factor?)
  6. Logging
  7. etc
  8. etc

Then the Houston HIE would create a CA, and that CA would “vouch” for organizations and individuals on the NHIN Direct network. BobsClinic might signup for the CA, then the CA would follow a bunch of steps to verify that BobsClinic was legit and was willing and capable of following the policy… and then the CA would say.. ok we are willing to vouch for BobsClinic.

Most clinics in Houston who wanted to use NHIN Direct could “import” the public key of the local CA. That’s fancy talk for they would accept the vouches that the CA made for all of the organizations that signed up. Those of you with security backgrounds understand that we are talking about a pretty basic CA infrastructure, but we wanted a way to make the trust decisions that clinicians would be making under this model free of unneeded technical language. So we are calling the CA, and all of the people that the CA “vouches” for a “Trust Circle”. It makes sense… if you have not imported the certificate of the CA, you are “outside the circle”, if you have imported the public cert of the CA then you are “inside the circle”.

This “Trust Circle” notion will reduce the number of trust decisions that typical NHIN Direct users will need to make. Of course, it will be really important that clinicians are very careful when they evaluate the policies and enforcement provided by a given CA. Those policies should meet or exceed their internal standards for handling PHI. It is important because you are not just trusting one organization… you are trusting lots of organizations “through” one organization, a much bigger deal.

Trust Circles get around the thorny problem of managing peer to peer relationships, but the also dodge another bullet. They avoid the need for a top-down single CA architecture. Things would be much simpler, technically, if the NHIN (which is a too-vague term BTW) would just setup the one-ring-to-rule them CA and make everyone in the United States follow the same policy for exchanging health information. That is a deal killer for about a hundred reasons, here are a few…

  • You are going to try and force catholic charity hospitals to share information with planned parenthood clinics.. are you kidding?
  • Making psychiatric hospitals message each other in the same way that normal hospitals do?
  • Make children’s hospitals message the same way that normal hospitals do? (kids are not just short people… Think about it.. Does the step dad get NHIN Direct messages for little johnny or only his biological father get them? Tough issues there.)
  • Create a policy that is guaranteed to be legal in all 50 states? (think about the implications of medical marijuana in California alone)

Policy is really really hard, even if you do not assume that you are going to get everyone to agree. Assuming that everyone will agree… makes the NHIN a non-starter.

Trust Circles (plural) gets you out of that problem. When organizations and clinicians can see eye to eye on policy, then they can use NHIN Direct to communicate secure messages… when they can’t see eye to eye… nothing in the NHIN Direct security protocols will attempt to force or even encourage them to compromise.

Another thing to note is that there is nothing in the design that prevents NHIN Direct users from managing trust relationships one at a time. You do not have to join Trust Circles to send messages with NHIN Direct. If you want to “self-sign” your certs and exchange them on floppy disks, in person, with people you trust.. that works too! That is why I used the word “typical” above…

But now we come to the real problem.

The first step is..

Even though the trust burden of the NHIN Direct system will be less than the trust burden of the current fax network… it may not feel that way. The reason is that we have not actually taken responsibility for the trust we place in the fax network. We continue to pretend that everything is fine. But its not. The fax network is irreparably broken and the first step towards fixing it is NOT to try and design a new model without a heavy trust burden, but to recognize that we have problem. Once we do that we can see that indeed “the burden is light”.

Speaking at OSCON


I am honored to announce that I will be speaking at OSCON 2010 on the healthcare track.

This talk is my “Health of the Source” talk. My intention in this talk is to cover both the “spirit” of Open Source in Healthcare as well as the “letter” of what is specifically going on right now. If you are unfamiliar with Open Source in Healthcare, and you can only attend one single talk on the subject, you should attend my talk. You will learn the most about the most different things. If you want to attend more than one talk, you should probably read Andy Orams summary of the OSCON healthcare talks.

As always, I am asking my readers and followers to tweet me about things that are happening in Open Source Healthcare that I should mention. If you could not get a spot at the conference but you are doing something wonderful, let me know and I will try and mention your work to the right people.


Science Conjecture in Science policy

Science-based policy is pretty difficult to do, but I support the notion.

But when is science conjecture taking the place of science? I know that this guy is not being quoted directly, but paraphrased by a reporter. That has been done to me enough that I know this strips what a person actually says of any nuance… but still something about the following statement makes me very comfortable.

The response of the doctor in charge was paraphrased as:

He acknowledged that it was impossible to specify just how many cancers were environmentally caused, because not enough research had been done, but he said he was confident that when the research was done, it would confirm the panel’s assertion that the problem had been grossly underestimated.

Does this scare the heebie jeebies out of any one else?

A person in the role of scientist making a policy recommendations based on what science will “soon” find? What does that even mean? Don’t get me wrong, I think we should totally be on carcinogen patrol, be when do good intentions begin to betray the scientific process?


Doctor impatient with NHIN security dithering

Recently, the members of the NHIN Direct Security and Trust working group (brian and I at least) were criticized for dithering:

> Fred and Brian,  I love what you’re trying to do for FOSS and data
> interchangeability.  You’re dedicated, smart, expert programmers and
> systems experts. You want to be socially responsible, and protect
> people from HIPAA violations.  I think your design principles as
> expressed at  are exactly
> right, but I think NHIN has been led astray as to what its mission
> ought to be, and this matters very much to me as an eventual user of
> health data interchange.

leading to…

> Okay, let’s get this straight. Under the HIPAA law,  Covered Entities,
> such as doctors offices, are responsible for protecting PHI, not the
> manufacturer of the fax machine the office uses to send information to
> another office, or the folks who write specifications for fax
> transmission.  Does this still make sense?
> OK.  Your job at NHIN is to design a fax machine.  “Just the fax,
> ma’am.”  Covered Entities, such as doctors offices and hospitals, are
> responsible for what information will be sent, and are responsible for
> protecting it (at BOTH ends).  Another government agency (
> ocr)  is responsible for enforcing HIPAA, not NHIN.    You just have
> to provide the highway to send the information, and make sure it gets
> to an actual covered entity. That’s ALL you have to do!
> I think you’re suffering from ‘Mission Creep’. You’re trying to make
> sure no one ever violates HIPAA with their fax machine.  Dang near
> impossible.  Thankfully, not your job.Yes. It makes perfect sense.

But I am not arguing that with you.

Doctors trust fax machines to provide private point to point communications.
The do not even wonder if fax machines “actually” provide private point to point communications.
In fact fax connections can be sniffed, which is why they have tele-walls now (a firewall for telephones)

And even that does not prevent the “wrong number problem

But even if we accept your premise, that we need to make NHIN Direct “just like a fax machine” someone, somewhere dithered about how faxing should be implemented. They went in circles with different ideas. They negotiated between different vendors of machine technology and they came up with a standard that could be implemented by a hardware company to build a fax machine. Here are some of the results of that.

To make something “simple like a fax machine” we need to have a set of instructions that people like EPIC, Medsphere, Google, Microsoft and Indivo can all implement identically. Those instructions might feel like they are far too complex for the simple task at hand. However we are already following Einsteins edict: “as simple as possible, but no simpler”. I think you will find that almost every discussion on the forums -has- a point. A valid concern that really should be addressed.

Main difference between a FAX machine and NHIN Direct is that, for whatever reason, Doctor’s do not distrust the security implications of a fax.
However, they -already have- faxes, which they already trust. When NHIN Direct and CONNECT come out, the “trust models” and “trust implementations” will be subject to intense scrutiny by security researchers who are decidedly better about thinking about these issues than I am. If a strong majority of those security researchers are not generally satisfied with our decisions, then they could act to cause doctors not to trust our design. This is a potential problem even if we design it right.

You are absolutely right, the law makes doctors responsible for HIPAA violations, but you forget the extension, they may also be held responsible for trusting the wrong technology. If a doctor downloaded Kazaa and accidentally published patient files on the Internet, do you think that a judge would be patient with his argument that ” it was a flaw in the technology!”. It is the doctor’s responsibly to be reasonably sure that a technology does do something that would be illegal. Currently doctors trust the “known-good” fax network. But will they trust NHIN-Direct? The answer is “of course they will”. But only because the security community will look at what we are proposing and say “well that looks reasonable” and they will only do that because we are getting our proverbial shit proverbially together.

I would suggest that you please be patient with our security sausage-making, I believe you will like the final result.


NHIN-Direct leans towards HealthQuilt Security Model

My last big project, before the skunkworks project I am doing now for Cautious Patient, was as the Chief Architect at HealthQuilt.

HealthQuilt was a prototype project for a Health Information Exchange in Houston T.X. hosted by UT SHIS. (Which just won the status of Regional Extension Center under ARRA . My boss at HealthQuilt, project leader Dr. Kim Dunn will be the director of the new REC. Dr. Dunn built a community of the local “interested parties” in Health IT during the HealthQuilt project. Ultimately, politics (remember this was pre-incentive) would prevent any data being transferred between organizations using our model before funding ran out. But now the community that Dr. Dunn built will be vital in her new role as REC director.

My job at HealthQuilt was to choose which technologies we would use to prototype the HIE. HealthQuilt was committed to Open Source from the beginning, so I was an obvious choice to handle the detailed technology choices. We spent alot of time with  Houston Health Information Security professionals aling with the crews at Mirth and MOSS, designing a workable trust model.

I am happy to say that just as Dr. Dunn will be able to build on the HealthQuilt community for the Houston-based REC, the NHIN-Direct project may decide to reuse some of the concepts (and perhaps some of the code) that we developed at HealthQuilt. Here are some of the basic, core concepts of the HealthQuilt model.

  • The Health Information Network should be built using point-to-point ssl VPN or https connections.
  • The trust model should use X.509 PKI Certificates.
  • It should use many (rather than one) Certificate Authorities (CA).
  • Both the recipient and the destination of a given VPN tunnel or https connection must have certificates. This is very different than the PKI model used on the Internet, where servers are generally certified but clients are generally anonymous.
  • This “encrypted Health Internet” should run entirely underneath any healthcare protocols. That means trust is handled first at the network level. If some actor in the network is no longer trusted, CRL or blacklists will prevent -any- communication with them, rather than relying solely on relatively young implementations of health protocols to provide adequate encryption.
  • The “relatively young implementations of health protocols” should still implement encryption, as though there was no network security in place. (This one is actually Sean Nolans idea.. more later)
  • This allows for a natural layering of security, which makes security wonks like me feel all warm and fuzzy.
  • The “core NHIN organization” should have a list of “typically trusted CAs” called “anchor CAs” that it recommends to all network participants. This is similar to the way that normal Internet CA’s are “suggested” to you by automatic inclusion in your browser of choice.
  • Individual network participants can also choose to trust other CAs, like those provided by a hospital they are affiliated with.
  • The job of the CA’s will be (roughly) to make sure that anyone they issue a certificate to is, in fact, a particular clinical entity (doctor, clinic, hospital, etc) who has the right to receive and/or send PHI.
  • This means that members of the network do not need to sort out trust relationships on a peer-to-peer basis. They can assume that everyone who the CA trusts is trustworthy, and they can automatically share data with them when a clinical need justifies it.
  • If, for some reason, two members of the network do not trust each other, they can still use a blacklist to prevent communication.

The browser providers determine what bar CA’s must reach for automatic inclusion in each browser. That “automatic inclusion” is the foundation of the trust model of the Internet. That is what gets you a secure connection to Amazon to buy a book, even though you do not think too much about “how do I know that is really Amazon?”

So why did HealthQuilt come up with this model? We knew that each institution in the Houston area would need to make trust decisions on its own. They would never tolerate us saying “Here are the ten other hospital systems in the network, take it or leave it”. The answer would always be “leave it”.  Some of our constituents were very concerned that a blanket trust policy would mean that they would trust organizations that they do not have a real-world trust relationship with, i.e. Planned Parenthood clinics vs. Catholic Charity Clinics. In order to participate in the network, they needed to have fine-grain control over the trust decisions. Most participants planned to trust everyone else in the network, but they did not want to trust that the network itself would remove bad actors in the future. The combination of blacklists (which is how a node can cut off communication with another node) and CRL’s (which is how a CA says “I do not love you anymore”)  provide both network and node level control over dealing with “bad-actors”.

Most importantly, the network-level security model is technologically identical to the current Internet trust technology. The policies and the trust decisions are very different, but the technology is basically the same; ssl + x509. That is good, because it means that the trust issues are not entirely handled at a level where new protocols are being developed. If you rely only on message security, and you discover that one implementation was “leaking” information by encrypting slightly less than what was intended in a given “message” that could be a real problem. SSL-vpns and https, using x509 PKI is a known-quantity (not the same thing as “safe” mind you). Using that “underneath” the new stuff that NHIN Direct and CONNECT projects will develop will help ensure that implementation or design mistakes do not automatically imply a broad attack vector.

Moreover, when advances (i.e. quantum cryptography) makes the current Internet Trust model obsolete, it will have to replaced with something. Whatever it is replaced with will have to play at least some of the same roles as the current X509/ssl infrastructure. That means the whole Internet will work with us to upgrade the network trust model.

I should point out that the NHIN Direct team was certainly not doing nothing until I showed up and told them what I had done with HealthQuilt. I think that something very like the basic HealthQuilt Trust model would have been embraced in any case anyway. I am just happy to be able to present a package of thought-out ideas to the NHIN Direct team. Ironically, even before I made my suggestions, Sean Nolan, of Microsoft HealthVault, was already arguing against the “single CA, top down trust model”. Once you make the concession that you are not going to attempt to do trust entirely using CA’s and proxy CA’s (the top down model) then most of the HealthQuilt Trust model, is just incremental obvious choices.

I will be calling this trust model, the “HealthQuilt Trust Model”. This is despite the fact that the NHIN Direct trust model seems like it could justifiably also be called the “Microsoft Model”. Microsoft has some really talented technical people and it makes me feel good to see them reaching the same conclusions that I do, in parallel. Still I seriously doubt that the new NHIN Direct trust model will ever be called “The Microsoft Model” since the name does not actually describe the model at all.  This is good, because the phrase “Microsoft Model” generally makes the hair on the back of my neck stand up and do the polka. It should also be noted that my original ideas on the HealthQuilt model were pretty useless without adjustment from Ignacio Valdes of LinuxMedNews, my brother rick, or David Whitten of WorldVistA and the VA and several of the Mirth engineers and Alesha Adamson of MOSS. All of whom gave me valuable feedback. It is also important to note that the model has improved substantially in response to the excellent thinking done by Brian Behlendorf and the rest of the NHIN Direct Security and Trust Workgroup.

Still, I will be using the name because it is truly indicative of how the trust model should work. It should be like a quilt, legitimately different ideas about trust and security implemented by different organizations, but despite those differences, still connected. The Internet has shown time and time again, that uniformity is not the only way to cooperate.

You can follow whats happening on the NHIN Direct Security and Trust Workgroup forum. If you are truly a glutton for reading, you can read my posts and the responses


OSCON includes Healthcare

Update: I am speaking at the 2010 OSCON.

I am happy to spread the news that OSCON, probably the most important Open Source conference in the country, will have a healthcare track in 2010.

Andy Oram has explained the decision to add a healthcare track to OSCON.

They have asked me to help promote the conference and I want to be sure that our community offers up the very best in talks and technical content. This is a really good way to access the developer mind-share in the broader Open Source community and we need to jump all over it.

I can honestly say that this conference will be vastly more important than the little shindig I am putting on in Houston. If you had to attend just one of the two, then you should probably go to OSCON… God bless you if you can go to both!!

With a healthcare track at OSCON, and a healthcare track at SCALE (DOHCS) we are finally moving towards general Open Source healthcare meetups.

I should take a moment to promote OpenMRS, CONNECT and WorldVistA all of which have great project-focused meetings already.

Happy days!


Technology vs Policy for privacy

I have long been an advocate of reasonable and measured reaction to “privacy scare tactics”. I have argued, for instance, that it was a good thing that HIPAA does not cover PHR systems. But that does not mean that I do not think privacy is important. In fact there has been something nagging at the back of my brain for several years now:

We typically use technology to provide security, but we use typically use policy to protect privacy.

That is deeply problematic. To see why we should carefully consider privacy and security in a simple context.

Imagine that you have purchased a safe-deposit box at a bank. Safe-deposit boxes are a great example, because if the bank accidentally gives away the money from your account, its not really a big deal… after all, its a bank, if they lose your money, they probably have some laying around somewhere else. The “lost money” could be restored to you from bank profits. But if you have baby pictures, your medical records, your will, Document X or your family jewels (not a metaphor) in your box, the bank cannot replace them.

Now to “protect the contents” really subdivides into two issues, security and privacy.

Security is the degree of protection against danger, loss, and criminals… according to wikipedia.. at least for today 😉

In the bank context “Security” means that no one can easily break into the bank  vault and grab your “Document X” from your box and run off. Of course there are many movies about how bank security can be overrun, but, thankfully, it is not a typical event for a bank have its security deposit boxes robbed. Banks use both technology (the vault, the alarm system, the video system) and policy (only the bank manager can open the vault, the vault can only be open during the day, etc etc) to protect the security of the bank boxes. Note that security is a spectrum of more secure to less secure, there is no such thing as just “secure”. Security can almost always be improved, but eventually improving security begins to interfere with the usefulness of something. If the bank vault could only open once a year, it would be more secure, but not very useful.

In the world of health information “Security” means that is difficult for a hacker to break in and get access to someone’s private health record. That is an oversimplification, but a useful one for this discussion.

Privacy is something else. The source for all knowledge (at least until someone changes it) says: Privacy is the ability of an individual or group to seclude themselves or information about themselves and thereby reveal themselves selectively.

In the bank example, Privacy is all about who the bank lets into the deposit box. For instance, if they decide suddenly that all blood cousins will now get automatic access to each others safe-deposit boxes, that would be a violation of your privacy. If your cousin could get access to your Document X because the bank let him, then the problem is not that the vault walls are not thick enough or that the combination is not hard enough. The problem is that the bank changed the basic relationship with you in terms of privacy. This is the first of several principles: Security Technology does not necessarily help with privacy.

Banks do not typically change the deal like that, sadly, modern websites regularly do. Two recent examples are the launch of Google Buzz and the Facebook privacy problems including the most recent expansion. The problem with modern information companies is that they usually take the corporate upper hand with their privacy policies. Sometimes this is really egregious, such as the original HealthVault Privacy policy, which gave permission for Microsoft to host your health data in China. But almost always it includes the key phrase “We reserve the right to change this privacy policy at any time”. If we sum up the problems with using privacy policies to protect privacy we find another principle: Privacy Policies often provide little privacy as written, often give permission to change the policy in the future which negates any notion of commitment, and even then policies are often ignored or misinterpreted. Privacy Policies should not be the only thing protecting our privacy.

Perhaps you want your spouse to be able to get access to your box. Perhaps you even want your cousin to have access. But the idea that the bank can just “change the deal” from what you had explicitly allowed is pretty strange. Thankfully banks rarely do that. But you could use technology to ensure that your privacy was protected, even if the bank arbitrarily changed its policy.

If you wanted you could keep a safe inside your safety deposit box, and keep your Document X inside the safe. Then you could give your combination to your spouse, so she/he could also open both the safe, (as long as you had also told the bank to give the access to your  to the safety deposit box to your spouse). Even if the bank decided that your cousin should have access to your box too, it would not matter, since your cousin could not open your safe. (we will pretend for the sake of this analogy that explosives and other means to circumvent the security of the safe would not work and the safe could not be removed from the safety deposit box). Our next principle: It is possible to use technology to help protect privacy.

Note that the safe also protects your Document X against access from bank employees. This is important because it does not matter what the privacy policy is, if it is not enforced by the employees of an institution, or if it the policy is arbitrarily changed in a way that you feel violates your privacy. It is also important to note that the government employees could not open the safe either. Of course we all know that governments are not inclined to violate the privacy of its citizens, but if the government did get a look inside the safety deposit box, all they would see would be the safe. Here we have another principle: Privacy technologies should prevent unwanted access from insiders and authority figures as well as from “bad” guys.

Which brings me to my point. We need to have more technologies available for protecting health information privacy. We have lots of technologies available for protecting security, but these do not protect privacy at all. These technologies, if they are going to work, need to give people the power to ensure that their health information is protected even from the people who provide a given technology service.

So far we have several principles:

  1. Security technology does not necessarily help with privacy.
  2. Privacy policies should not be the only thing protecting our privacy.
  3. It is possible to use technology to help protect privacy.
  4. Privacy technologies should prevent unwanted access from insiders and authority figures as well as from “bad” guys.

To these I would like to add some implied corollaries:

Encryption, by itself is not a privacy technology. It is a security technology, but it is only a privacy technology depending on who has the keys. This was the problem with the infamous clipper chip. The first issue with Information System privacy is, and will always be, “who has the keys?”.  So when a service says in response to a privacy challenge “oh don’t worry its all encrypted” that’s like saying: “You are afraid that I will fax document X to your mother? Don’t worry I keep document X at home in my safe!” If you are afraid that I am going to fax document X to someone, the fact that it is now in my safe should not make you feel more comfortable. I can still get in my safe and I can still fax document X wherever I want. Document X being in a safe is only helpful if you trust everyone with the keys to the safe.

The other thing is that you simply cannot trust proprietary software to provide privacy. If you cannot read the sourcecode to see what the software does, then it does not matter what kinds of privacy features it advertises or even who has the keys. At any time the developers could change the code and make it violate your privacy. To further extend and abuse our example, it does not help you to have a safe inside the safe-deposit box if the bank can change the safe’s combination whenever they want, without you being the wiser. Only Open Source systems can be trusted to provide privacy features.  I do not argue that this is enough, but it is an important starting point.

I have been working on a relatively complicated system for achieving these goals. My initial application is blessedly simple, and so I have been able to avoid some of the issues that make these kinds of systems commonly intractable. My service is still in skunkworks and will continue to be very hush-hush until I make the beta launch. But I will be announcing my designs soon and I have already submitted those designs to some Internet Security professionals that I respect to make sure I am moving in a sane direction. This kind of thing is really technically difficult, so I am certainly not promising that I can deliver this kind of technology, but perhaps I can deliver something working that would give other people a place to start. As you might expect, the sourcecode for this project will eventually be made available under a FOSS license.

I do not want to get into the design details yet (so please don’t ask) or even talk about my new application (which is just entering closed beta) but I wanted to start talking about why these issues are important. Please feel free to comment on the features that privacy protecting technology should have…


Welcome NPR

Apparently, npr mentioned my blog. They contacted me for a comment on Medsphere’s ROI calculator.

So if you are from NPR I thought I should actually give you the entire email I sent to Mr. Weaver:

My impression of Medsphere’s prices is that they are a “premium” VistA vendor. If you look you can find cheaper VistA support. However, there are only a handful of VistA support companies with real experience and all of them charge like Medsphere does. On the other hand, even “premium” support for VistA is cheaper than proprietary solutions by 30% to 80%. This is simple evidence of the economic pressures of Open Source. Medsphere can and does charge as much as it can. But it is competing against other groups using the same software that it developed, and against hospitals doing the work internally. There is objective value in going with a vendor who has succeeded with an Open Source project in the past. The balance between open competition (which lowers the price) and proven track record (with raises it) means that Medsphere’s price is objectively fair, and subject to market forces both before and after the installation of the software.

This is the real story about Medsphere and other Open Source Health IT vendors. It’s not that they are inexpensive, although they are much cheaper. Its that their prices actually reflect the markets response to their ongoing performance. That not only makes things cheaper now, but ensures long term cost savings and improvements in performance.

Proprietary lock-in is a form of monopoly where the vendor has an economic incentive to provide poor support. Therefore at least the support services from proprietary vendors are not subject to market forces. That is why they are so poor and Senator Grassely seems to think they warrant investigation.

Medspheres Open Price-Stimulus Calculator

Medsphere marketing has making some pretty savvy moves lately. First the bus and now an open ROI calculator.

This is really cool. Medsphere is being transparent about its prices, and showing you exactly how much it is putting its neck on the line for its customers. This tool will let any Hospital IT person calculate just what they can expect from the stimulus package, and how much Medsphere will cost.

I do not know of anyone else, proprietary or otherwise, that has anything like this.

It would be neat to run through several hundred test cases so see where Medsphers “sweet spot” is… I am betting small-to-medium hospitals.