On Internet Marketing

Recently I have had several people who have asked me for advice and council on how to do Internet Marketing. It looks like my day job is considering taking the plunge as well. As a blogger, I know an opportunity to kill two birds with one blog post when I see one. So here are my thoughts on Internet Marketing in the age of social media.

Some context

First, a little history. It used to be that Internet Marketing was all about communicating effectively with a web site and email communications.

For a website, the general advice was that you wanted it to be important in Google’s eyes, essentially the process of SEO. You wanted to make sure that your domain name was easy to type, and easy to spell. You wanted to make sure that your users could find what they wanted on your web site. A website needs great analytics tools so that you can track how your web site is being used.

For email communications you had to decide if you wanted to have only an outgoing email campaign (broadcast only), or a mailing list (communication between everyone). If you wanted and email campaign you wanted to make sure that you had beautiful html emails that degraded gracefully into text emails. You needed to be sure that your html emails worked in the most common mail clients (harder than it sounds). For a mailing list you wanted to make sure that no one was added by mistake, and that no one was spamming people through your list.

Anyone who knows about these types of marketing systems can tell that I am barely scratching the surface on these issues. Moreover, it is also clear that while so-called “Social Media” has become really important, these older modes of communication are not less important, they are just… older.

So that is the backdrop, in brief, for the Social Media revolution. What is the big deal about Social Media? Way out of scope for this post, but I will inline probably the best video proving the point that I have seen. If you have not seen it, then watch it. If you have, then you probably already know why Social Media is a big deal.

The question posed at the beginning of this video is “Is it a fad or is it a revolution?” As is often the case to questions like that, the answer is “Yes”. Social media has lots and lots of people connecting meaningfully with lots and lots of people, but that does not mean that you will be able to get your message across using Social Media. All it means is that there are people there. Its a lot like Lubbock, Texas. For whatever reason, there are hundreds of thousands of people living in what appears to be a desert. Why would you want to live there? Because there are hundreds of thousands of people there. There is probably a reason why the original people made that city, but no one moves there now because of scenery, they move there because there are already people living there. This is much different than something like Las Vegas. Its a city in the desert that was built specifically so gambling could be legal. That is why people moved there. (I also do not understand Phoenix…)

So the next question is “What is your message?”

Your Message

Social Media people often “sell” Social Media as “the new business requirement”. They say things like “You have to be on Facebook” or “You need to have a Twitter account”. But that is really not the first question. The first question is “What is your message?”. Unless you can define your message, clearly, in a sentence or two then nothing else I am going to say is going to make sense. For fun, and because we are going to talk about Twitter soon, see if you can put your message into 140 characters. That is basically two short sentences or one really long one. Its OK if you need to go over a little, but if you need to have four or five 140 character blocks then that should be insight that you have more than one message. That is OK, but you need to recognize that you might need to follow significantly different strategies for each of your different messages.

So do you have your message(s) in your head? OK then.


To make an impact you have to learn to use the Internet Marketing tools well and then you have to apply them in  meaningful way. This is a lot like a carpenter’s toolbelt or a musician’s set of instruments. First you have to master the tool and understand the deep implications that subtle details of each  given tool. Just because a tool is a type of hammer does not mean you can use it do mount photos (imagine using a sledgehammer to tap a nail into drywall). Just because it is an instrument does not mean that you will fit in with a given band (imagine bringing a tuba into a rock band). The first level of tool mastery is understanding how to use the right tool for the right job. the second stage of mastery is knowing when to ignore what you learned in the first phase (for instance, Ska is a movement, within rock music, to embrace brass instruments).

Note that true mastery of a tool is being able to use the tool to do something else amazing. When Michelangelo was painting the ceiling… there were thousands of painters who knew how to use a paintbrush as carefully as he did. But they were probably painting signs, or the sides of barns. The ability to use the paintbrush is only the first step towards being Michelangelo.

This should sound obvious. But here is how I think this basic tool mastery is playing out in Internet marketing.

Phases of Online Marketing Tool Use
Phases of Online Marketing Tool Use

OK, so what does these phases mean? First I should admit that I was inspired to make this chart by two different sources, one is Meatball Sundae: Is Your Marketing out of Sync? By Seth Godin. The other is a blog post entitled: The multiple phases of social media integration which is where I borrowed my three phases ( of course as a computer scientist, I must count from zero).

  • 0 level is nothing.. If you read this article and go “What is Facebook?” or “What is Twitter?” then this level is where you are. No problem, I will try and help you out with lots of good links.
  • 1 Level is using the Internet as a megaphone. This is when you treat your web site, Facebook page, Twitter account or whatever as a mass media device. You use it the same way people use radio, TV or newspapers: to send messages out to lots of people all at once.
  • 2 Level is to use the Internet as a camp fire. When you sit around a campfire and talk, it shifts between you speaking to the group (like a megaphone) and the group speaking to you, one at a time. The group also speaks about you, in front of you. It enables public conversation in lots of different directions. In real life campfires are a great time and place to do this, because the typical night-time acoustics allow for a large group (10 to 30 people even) to participate in a single conversation. But this does not scale. The whole point of Social Medial is that you can have a campfire chat with hundreds, thousand or even millions of people all at once.
  • The third level represents full tool mastery. But this does not automatically mean that you get to paint the Sistine chapel. It just means you know how to use the tool.
  • 3a is named after Blendtech, a company who has successfully used Social Media to create a Sistine chapel (more in a moment).
  • 3b is someone who is using the tools well, to do OK things, but is not doing anything truly meaningful. This would be like the painters who during 1512 were painting portraits or landscapes and are today forgotten. But they made a good living and their customers were happy.
  • 3c is like someone who was drawing graffitti on the walls in 1512. No matter how pretty an picture, painted at night on barn might be, in the morning it will be whitewashed. The skill is irrelevant, it is a matter of message and medium match up.

Obviously, what everyone wants is to make an impact with their marketing. To leave people with a message burned into their minds, and happy that it happened. Once people see the Sistine chapel (on my bucket list) they will never never think about it the same way, and they will never forget the experience.

Is this possible with Internet Marketing? Yes. I will give you two examples.

First, if you have bought a book recently online and you immediately typed in Amazon.com then you have experienced this effect. The word “Amazon” has nothing to do with books. Yet when you want to buy a book on the Internet you probably go there automatically. Why? Because you have had a Sistine Chapel- style experience there and you will always remember it. Note that this is a great example of a company that was able to achieve this with just a website and without any kind of Social Media. Ebay and Google are other good examples.

Rather than just talk about the second example I will show you. Blendtech is a company that makes really good blenders. That is their message. That is what they want burned into your brain. After you watch the following videos, you will always remember that Blendtech is a company that makes really good blenders. You will be unable to remember the name of any other blender manufacturer, but you will never again think about where you would get a really good blender…   if you needed a really good blender.  Please watch the following two episodes of “Will it Blend”.

What is a Meatball Sundae?

Its two things that are great by themselves but still do not go together. Meatballs and Whipped Cream/Chocolate. This is the worst case scenario for Internet Marketing efforts. This is what happens when you fail to recognize that Internet Marketing and/or Social Media (two terms for the same things nowadays) really does change things deeply in in your industry, but you are unwilling to make the fundamental changes needed to make the leap.

This is actually a fundamental mistake that happens often in Health IT, which is what I like to call “Technology as Paint”. The basic notion is that technology can be liberally applied to make any existing thing better. This is the way you use paint. My wife and I recently bought a desk for $35. It was banged up and looked awful. We painted it. Now it looks like it cost $350. Paint is awesome like that!

But technology is not paint. You cannot take something that works without technology, merely make it “online” or “computerized” and assume that it will be better than the original system. the cardinal example of that in health IT is the we- are- going- to- computerize- the- dumb- doctors. Here is how the plan unfolds:

Doctor: “Hey business man, I want you to computerize me.”

Business Man: “That’s great! I have my favorite coder here with me, and we can help.”

Coder: “I can easily computerize you! I just did it for a Gas Station last week! No more paper forms at the Gas Station!! All I need to do is see all of your paper forms, and then I will computerize you by making computer versions of those forms.”

Doctor: “OK, here are the ten forms I regularly use.”

Coder: “OK I will be back in a week with your computer system built!!”

Five years pass…

Coder: “The system is almost ready, I have just finally got the ontology mapping tool together, you can go live next week!!”

Doctor: “You are fired. You have been charging me for five years to code and you have nothing to show for it. I still have to use paper because your system does not even to 10% of what the paper system does. Now I have five years worth of data in both paper and electronic records, and I can no longer afford to maintain the electronic system. I am sooo screwed, but at least I am going to stop paying you!”

This happens again and again and again in Health IT because so many technologist view technology as paint: Standard technology, liberally applied, solves all problems.

Seth Godin’s book is really required reading. It details, very explicitly how Social Media is not technology paint for marketing purposes. Any good summary of his points will show that you have to figure out if, and when your message is right for the Internet Medium in question. So when you hire someone to help you with Social Media, and they fail to show you how a given Social Media platform is good for your message, then they have failed. A pretty good idea of when you are getting bad advice here is that they are recommending that you go with the usual suspects. If they say: “you should be on Youtube, Twitter and Facebook” without discussing how your message will play in those environments, then you need to take a step back.

It would be much better for you to do what Blendtech did, which is to find the one medium that allows you to create a super-compelling version of your message, and make that medium into the “Sun” in your marketing “Solar System”. Sure Blendtech uses Twitter, and Facebook, but they do that to funnel people to their awesome videos, which in turn funnel people into buying an awesome blender.

Social Media Strategy as a Solar System
Social Media Strategy as a Solar System

Message and Medium as a Solar System

What follows is a little more conjecture. I am pretty darn sure about the notions I have explained above. But without dealing with a specific message, it is difficult to know what the right center-of-gravity medium might be. But still here are some guidelines that make sense to me:

  • If your idea is best communicated in pictures, try Flickr or Picasa. They have really advanced tools that allow you to view a series of constantly updated photos as a steadm on another site. A big hint when using pictures is that pictures with people in them are almost always more interesting then pictures without people. You can make both Twitter and Facebook, and plain old web-page follow those photo streams. You might want to use Flickr/Picasa as the centre of gravity if before and after photos are more compelling than a video, for instance. Lots of people have made this approach work.
  • If your points make sense as really short catch-phrases or have a very important real-time component, then Twitter or Identi.ca might be for you (Identi.ca is a more freedom/less popular version of Twitter). Shit My Dad Says, which is now a television show and a book (pretty good planets!!) is a good example of the catch-phrase style Twitter feed. In Portland there are some food carts that you can only find by following them on twitter. Note that you easily add Facebook and Google Buzz as planets merely by propagating your status updates to those platforms.
  • If you want a deeper social engagement that includes videos, text, pictures or perhaps an application that you are writing yourself, perhaps Facebook is for you. A good question to ask about Facebook is “Am I anything like Farmville?” Again you can easily make Facebook updates propagate across the other platforms.
  • If you are trying to make a series of arguments that require carefully constructed arguments, then you need a blog. This gives you the ability to tie in all kinds of other content (like I did with Youtube videos here) to make very specific and complex points. But if you make enough of these points, then perhaps you are really slowly writing a book, and you should consider self-publishing it on CreateSpace or Lulu.
  • If you have already written a book, perhaps you need to split apart into a blog.
  • Videos can be tremendously engaging and personal. If you have a story to tell, a parable of some kind, then this is the right medium. Even just a camera, pointed at you can be very very compelling if your story is good enough. You should be looking into Youtube, which is the king of the space, but also perhaps screencast.com if you want to show films of computer programs, or some other site if you have other specific video hosting requirements. Again, you can turn your video feed into facebook, twitter, identi.ca, and Google Buzz integration.
  • If for some reason your content would work really really well next to gmail, you might look at Google Buzz.
  • If you want to create complex person to person engagement between lots of people around a particular topic that they have a high level of interest in, then I would consider either email mailing lists, or online forums, or something like Google Groups, which is a pretty good fusion of both. Getting the “full message” in your email Inbox is pretty valuable.
  • If you want to have things showing up in email Inboxes, but do not want to enable communication between the recipients then you probably want an email broadcasting service like MailChimp.
  • If you want to engage with professionals of one kind of another Linked in is where you should start.
  • If you want to generate written content, you need a Wiki.
  • Face to face Events can now be deeply connected into the Internet. I like using eventbrite to schedule things like conferences, I like meetup.com for regular meetings, and when a meeting is really important, it should be live streamed with something like livestream.com.
  • Sometimes, what you need is a simulated three dimensional space. Frankly I have trouble understanding when this is a good thing… but if you see it is valuable you want to use Second Life.
  • If you want a Facebook style social network that you control you want Ning.
  • If you want full control, including source code for your social network, then you want one of these.
  • If you have a health IT application that needs to interface with Doctors socially, then you want to work with Sermo.com

I hope this is helpful to people that I am trying to counsel on Social Media. Its not just about using it, its about finding a way to use it in a compelling way!


Funambol in healthcare

One of the things that I love about conferences like OSCON is that you met people who are doing really interesting things coming out of left field. I often feel like I “know everyone” in Open Source healthcare, but every time I hear about something like this I am reminded just how big the world is. People are reworking Open Source tools to work in healthcare all the time!

The most recent example is from the Funambol project. That project is made to sync cell phone data, like calendars and contacts. But the Funambol teleport project instead uses the stack to move healthcare data around. I would go into detail, but there is no need, since Simona does a much better job:

OpenStack and Software Freedom in Healthcare IT

As clinicians, doctors and other healthcare providers are the stewards of their patients data. But what happens when they lose control over that healthcare data? Most people focus on what happens when that private data becomes too available. But far more commonly healthcare data becomes trapped. Far too often, it becomes buried in one way or another, lost forever and useless to patients.

I am probably the most vocal proponent of the notion that software freedom, the heart and soul of the Open Source movement is the only way to do healthcare software. Over that time I have tried to highlight the threat posed by vendor lock-in with healthcare software. But “vendor-lock” is not the only way that healthcare data can become buried. Ignacio Valdes was the first to make this case clearly against ASP healthcare solutions with his post about how Browser Based EMR’s Threaten Software Freedom . That was written in 2007.

So you can imagine the types of concerns Ignacio and I had as we built Astronaut Shuttle (very much beta) together. Ignacio had the VistA EHR chops and I had enough cloud experience to create the first-ever cloud based EHR offering. Its a simple system, you can use a simplified web interface to launch cloud-based instances of an EHR. The main difference between this kind of web interface, and something like RightScale, is that the launching system performed whole-disk encryption, allowing you to ensure that Amazon could not access your healthcare data. As far as I know, no one else has built anything like this but us (love to hear otherwise in comments).

Why are we some of the few people trying things like this? For one thing , encryption is pretty difficult to do in the cloud,  there are lots of approaches and it is pretty easy to brick a cloud instance with an improper encryption configuration.

But more importantly, there is a perception that storing private healthcare data in the cloud is a bad idea, dangerous because it meant putting all of your eggs in one basket.

Given how concerned Ignacio and I were about vendor lock, and ASP lock, you can imagine our feelings about cloud lock. We had to be sure that our customers, doctors and other clinicians, would be able to restore linux images containing precious EHR data off-site using off-site backups.

When we looked out across the available cloud options we decided to implement our service using Amazons ec2 service, specifically because of  Eucalyptus an open source implementation of the Amazon cloud hosting infrastructure.

However, we have been deeply concerned about this approach. Currently, you might say that Amazon has a “friendly” relationship with Eucalyptus, which of course means that Amazon has not crushed it like an itty-bitty bug. For Amazon, being able to point out that there were FOSS implementations available made it easier for ec2 to acquire certain customers. At the same time by refusing to treat the ec2 and other AWS api’s as open standards, or to specifically state they would not sue an open source implementation of their API, Amazon could always ensure that Eucalyptus would never be a threat.

“What a minute!” you might say… “Amazon is a Linux-friendly company! They would -never- betray the community by going after Eucalyptus…”

I think the Open Source community needs to wake up to corporations whose basic legal stance towards Open Source projects is to leave open the “smash if they succeed” option.

IBM has been a “friend” to the community for years. IBM even promised not to use specific software patents against us. They assured us that they are not a threat. But then they broke that promise. The broke it because someone in the community decided to implement software that threatened to break their monopoly on mainframe implementations. IBM turned on our community just as soon as our freedom started to threaten their bottom line. You are kidding yourself if you think Amazon will lose a billion dollars to Eucalytpus without reacting. Amazon has been very aggressive in acquiring software patents and will use them if Open Source implementations ever really get good.

I think Eucalytpus is an awesome project but it lives at the whim of a corporation who tolerates it precisely because it is not a business threat.

It was with great trepidation that Ignacio and I built a health data infrastructure that we knew relied on the whim of a really big bookstore. (When you say it like that… you can see the problem more clearly)

With that said, I am happy to support and endorse the new OpenStack project. OpenStack is a move by Rackspace Cloud, the number one competitor to Amazon, to completely Open Source their cloud infrastructure. They will be releasing this work under the Apache license.

Open Source license are the only trust currency that I, as health software developer, can trust to ensure that no one can ever trap health data with software that I have recommended. “Probably won’t trap” or “Open Source friendly” simply do not cut it after IBM. Simply put, a full Open Source release is the most extreme thing that Rackspace can do to win my trust in their cloud infrastructure.

I have also been discussing with the Rackspace team about the importance of building in support for cloud-initiated-encryption and cloud audit (thanks for the tip samj)  into Open Stack.  These are must-have features to make healthcare data in the could a viable option.

As soon as we have the dev cycles available, we will be moving Astronaut Shuttle over to the Rackspace Cloud. I invite anyone who gives a damn about Software Freedom, or health information software generally, to follow us over.


NHIN and others at OSCON

I am just home from the first ever amazing health IT track at OSCON. The quality of the content is simply amazing, and soon, you will be able to see the many talks available (thanks to Robert Wood Johnson for paying for the videos)

As I think about what I will be blogging about, I wanted to post some quick links to those who are already thinking about what was said and what it means. First the conference organizer, or at least from the health IT point of view, was Andy Oram. He already has two posts, one on the first day, and one highlighting the VistA controversies exposed at the conference.

Most of all, I wanted to point to this awesome interview with the leaders of the NHIN open source projects: NHIN CONNECT and NHIN Direct.

Tolven invited to privacy party

The Open Source Tolven project has been invited to the Privacy Technology showcase for the HITPC S&P Tiger team.

This is well-deserved recognition. Tolven has an extremely innovative architecture, that dispenses with many of the bad assumptions that other EHR platforms make. The first is that an EHR platform should only be an EHR platform. Tolven is a combined EHR and PHR.

The second is a well-thought out encryption at rest strategy.

Hopefully a recording of the presentation will be available after the meeting.

Empathy over implementations and another straw man

I think the recent work of the NHIN Direct implementation teams has been amazing. But I think that by implementing, all of the teams have succumbed to different extents a common software developer error. They are implementing rather than empathizing with their users.

There are two groups (possibly three if you count patients, but I will exclude them from consideration for just a moment) of potential NHIN Direct end users. But before we talk about that, I would like to talk about Facebook and Myspace.

Or more accurately I want to remember the controversy when the Military chose to block users of Myspace but not Facebook. This caused quite a stir, because at the time, Myspace was very popular with the high-school educated enlisted personnel, but Facebook, which even then was “higher technology” was more popular with the college educated Officers. Controversy aside, it showed a digital divide between different types of users.

Ok, back to Healthcare IT.

We have something strangely similar to this in the United States with the level of IT adoption with doctors.

On Having

Most doctors are low-health-information-technology. Most doctors work in small practices. Most small practices have not adopted EHR technology. Even those small practices that have adopted EHR technology have often done so from EHR vendors who have not focused on implementing the tremendously complex and difficult IHE health data interchange profiles. That is one group of doctors. You can call them Luddite, late adopters, low-tech or “the have not’s”. No matter what you call them, the HITECH portion of ARRA was designed to reach them. It was designed to get them to become meaningful users of EHR technology. (Note that “EHR technology” is kind of a misleading term because what it hass essentially has been redefined to is “software that lets you achieve meaningful use”. People still have lots of different ideas about what an “EHR” is, because people still have lots of disagreements about the best way to achieve meaningful use.)

I have to admit, I am primarily sympathetic with this user group. I originally got into Open Source because my family business (which my grandfather started) was a Medical Manager VAR. Our clients loved us, and they hated the notion of spending a bucket load of money on an EHR. I started looking for an Open Source solution to our EHR problems, and when I could not find what I needed, I started contributing code. There are a small cadre of people working on the NHIN Direct project that share my basic empathy with this type, the “have nots”, of clinical user for different reasons.

But the majority of the people working on NHIN Direct represent the whiz-kid doctors. These are the doctors who work in large clinics and hospitals that have found moving to an EHR system prudent. Sometimes, smaller groups of doctors are so tech-hungry that they join this group at great personal expense. These doctors, or the organizations that employ them have invested tremendous amounts of money in software that is already IHE aware. Often groups of these doctors have joined together to form local HIE systems. It is fair to say that if you are a doctor who has made an investment in technology that gives you IHE systems, you paid alot for it, and you want that investment to pay off. We can call these doctors the “whiz-bang crowd”, the EHR lovers, or simply “the haves”.

Today, in the NHIN Direct protocol meeting we had a polite skirmish (much respect for the tone everyone maintained despite the depth of feeling) between the representatives of the “have nots” like me, Sean Nolan, David Kibbe and other who are thinking primarily about the “have nots” and the vendors of large EHR systems, HIE’s and other participants in the IHE processes, these people tend represent the “haves”.

To give a little background for my readers who are not involved with the NHIN Direct project:

A Little Background

NHIN Exchange is a network of that anyone who speaks IHE can join. If you speak IHE, it means that you should be able to meet all of the requirements of the data exchange portions of meaningful use. It also means that you pretty much have to have some high technology: a full features EHR or something that acts like one. IHE has lots of features that you really really need in order to do full Health Information Exchange right. But it has never been deployed on a large scale and it is phenomenally complex. ONC started an Open Source project called NHIN CONNECT that implements IHE and will form the backbone of the governments IHE backbone. Beyond CONNECT both the Mirth guys and OHT/MOSS have substantial IHE related software available under FOSS licenses. There are lots of bolt on proprietary implementations as well. IHE is complex, but the complexity is required to handle the numerous use cases of clinical data exchange. Exchanging health data is vastly more complex than exchanging financial information etc etc. But to use IHE you have to have an EHR. Most doctors do not have an IHE-aware EHR.

ONC knew that HITECH would convince many doctors to invest in EHR technology that would ultimately allow them to connect to NHIN Exchange. However they also knew that many doctors, possibly most doctors, might choose not to adopt EHR technology. Someone (who?) proposed that ONC start a project to allow doctors to replace their faxes with software that would allow them to meet most, but not all, of the meaningful use data interchange requirements, without having to “take the EHR plunge”. This new project could meet all of the requirements that could be met with a fax-like or email-like “PUSH” model. I explained some of this in the power of push post. This project was called NHIN Direct.

Whats the problem

So what is the problem? A disproportionate number of the people who signed up to work on the NHIN Direct project are EHR vendors and other participants who represent lots of people who have made extensive investments in IHE. In short, lots of “haves” representatives. Some of the “haves” representatives proposed that NHIN Direct also be built with the subset of IHE standards that cover push messages. But remember, IHE is a complex set of standards. Push, in IHE, is much more complicated than the other messaging protocols that were under consideration. I have already made a comparison of the protocols under consideration.

IHE is really good for the “haves”. If you “have” IHE, then all kinds of really thorny and difficult problems are solved for you. Moreover, one of the goals of meaningful use is to get more EHRs (by which I mean “meaningfully usable clinical software”) into the hands of doctors. The US, as a country, need more people using IHE. It really is the only “right” way to do full health information exchange.

But IHE is not trivial. It is not trivial to code. It is not trivial to configure. It is not trivial to deploy or support. It is not trivial to understand. It could be simple to use for clinicians once all of the non-trivial things had been taken care of. But realistically, the number of people who understand IHE well enough to make it simple for a given clinical user is very very few.

The other options seemed to be SMTP or REST-that-looks-and-acts-just-like-SMTP-so-why-are-we-coding-it-again (or just REST). Both of these are much much simpler than the IHE message protocols. These would be much simpler for the “have not’s” to adopt easily and quickly. Of course, they would not get the full benefit of an EHR, but they would be on the path. They would be much better off than they are now with the fax system. It would be like the “meaningful use gateway drug”. It would be fun and helpful to the doctors, but leave them wanting something stronger.

The NHIN Direct project, fundamentally creates a tension with the overall NHIN and meaningful use strategy. As a nation we want to push doctors into using good health IT technology. But does that mean pushing them towards the IHE-implementing EHRs on the current market? or should we push them towards simple direct messaging? The answer should be something like:

“if doctors would have ordinarily chosen to do nothing, we would want them to use NHIN Direct, if they could be convinced to be completely computerized, then we should push them towards IHE aware clinical software that meets all of the meaningful use requirements”.

Given that split, the goal of NHIN Direct should be:

“For doctors who would have refused other computerization options, allow them to meaningfully exchange health information with as little effort and expense on their part as possible”

I, and others who realize just how little doctors like this will tolerate in terms of cost and effort, strongly favor super simple messaging protocols that can be easily deployed in multiple super-low cost fashions. I think that means SMTP and clearly rules out IHE as a backbone protocol for “have-nots” that are communicating with other “have-nots”.

Empathizing with the Haves

But the danger of focusing on just the requirements of your own constituents is that you ignore how the problems of those you do not empathize with the impact that your designed will have on other users. Both the representatives of the “have’s” and the “have nots” like me have been guilty of this. After listening to the call I realized that the EHR vendors pushing HIE where not being greedy vendors who wanted to pad their wallets. Not at all! They were being greedy vendors who wanted to pad their wallets -and- protect the interests of the doctors already using their systems. (that was a joke btw, I really did just realize that they were just empathizing with a different group of doctors than I was)

If you are a “have” doctor, you have made a tremendous investment in IHE. Now you are in danger of getting two sources of messages. You will get IHE messages from your other “have” buddies, but you will have to use a different system to talk with all of the “have-nots” who want to talk with you. That means you have to check messages twice and you can imagine how it might feel if for one doctor to be discussing one patient, across the two systems. Lots of opportunity for error there.

From the perspective of the IHE team, by making the “have nots” make the concession of dealing with IHE, rather than cheaper, simpler easier SMTP they reduce to one messaging infrastructure that eliminates balkanization. No longer will any user be faced with using two clinical messaging systems, instead they can have only one queue. Moreover, since we ultimately want “fully meaningful users” it is a good thing that the IHE-based NHIN Direct system would provide a clear path to getting onto the NHIN Exchange with a full EHR. From their perspective, having more difficult adoption for the “have nots”, and the resulting loss of adoption would be worth it because it would still get you faster to where we really need to be, which is doing full IHE Health Information Exchange with everyone participating.

Everyone wants the same thing in the end, we just have different ideas about how to get there! I believe that we should choose protocol designs for NHIN Direct that fully work for both sets of clinical users. I think we can do this without screwing anyone, or making anyone’s life more difficult.

The new empathy requirements

I would propose that we turn this around into a requirements list: We need a NHIN Direct protocol that

  • Allows the “have nots” to use NHIN Direct as cheaply as possible, that means enabling HISPs with simple technology that can be easily deployed and maintained using lots of different deliver models. (i.e. on-site consulting and server setup, ASP delivery etc etc)
  • Allows the “haves” to view the world as if it was one beautiful messaging system, and that was based on IHE. They should not have to sacrifice the investment they have made and they should not have to deal with two queues.

My Straw Man: Rich if you can read it

The IHE implementation group believes that all SMTP (or REST-like-SMTP) messages that leave the “have-nots” should be converted “up” into IHE messages and then, when they get close to other “have-not” doctors, the messages should be converted back “down” to SMTP. Which means that they are suggesting that HISPs that handle communications between “have-not” doctors should have to implement IHE in one direction and SMTP in another direction even though the message will have no more content or meaning after being sent.

The problem with that is that the HISPs that maintain this “step-up-and-down” functionality will have to cover their costs and develop the expertise to support this. This is true, even if the “edge” protocol is SMTP. The only approach that will work for this design is an ASP model, so that the HISP can afford to centralize the support and expertise needed to handle this complexity. That means low-touch support and low-touch support, and high costs translate to low-adoption. In fact doctors would probably be better off just investing in an ASP EHR that was fully IHE aware. So the IHE model is a crappy “middle step”.

But there is no reason that the HISP needs to handle step-down or step-up as long as they are only dealing with “have-not” doctors. If you allowed SMTP to run the core of NHIN Direct, HISPs could leverage current expertise and software stacks (with lots of security tweaking discussed later), to ensure that messages went into the right SMTP network. No PHI in regular email.  Local consultants as well as current email ASP solutions could easily read the security requirements, and deploy solutions for doctors that would send messages across the secure SMTP core. With careful network design, we could ensure that messages to NHIN Direct users would never be sent across the regular email backbone. I will describe how some other time (its late here) but it is not that hard.

But you might argue: “that is basically just SMTP as core! This is no different than your original approach. You are still screwing the haves!” Patience grasshopper.

To satisfy the “haves” we have to create a new class of HISP. These HISPs are “smart”. They understand the step-up and step-down process to convert IHE messages to SMTP messages. When they have a new outgoing message, they first attempt to connect to the receiving HISP using HIE. Perhaps on port 10000. If port 10000 is open, they know that they are dealing with another smart HISP, and they send their message using IHE profiles. Some smart HISPs will actually be connected to the NHIN Exchange, and will use that network for delivery when appropriate.

The normal or “dumb” HISP never needs to even know about the extra functionality that the smart HISP possess.  They just always use the NHIN Direct SMTP port (lets say 9999) to send messages to any HISP they contact. While smart HISPS prefer to get messages on port 10000, when they get an SMTP message on port 9999 they know they need to step-up from SMTP to IHE before passing to the EHR of the end user.

From the Haves perspective, there is one messaging network, the IHE network. They get all of their messages in the same queue. Sometimes the messages are of high quality (because they where never SMTP messages, but IHE messages sent across NHIN Exchange or simply between two smart HISPS).

Now, lets look at the winners and losers here. For the “have nots” the core is entirely SMTP. As a result they have cheap and abundant technical support. They are happy. For the “haves” they get to think entirely in IHE, they might be able to tell that some messages have less useful content, but that is the price to pay for communicating with a “have not” doctor. The “have nots” will get rich messages from the IHE sites and will soon realize that there are benifits to moving to an EHR that can handle IHE.

Who looses? The smart HISPs. They have to handle all of the step-up and step-down. They will be much more expensive to operate -unless- there is a NHIN Direct sub-project to create a smart HISP. This is what the current IHE implementation should morph into. We should relieve this burden by creating a really solid bridge.

This model is a hybrid of the SMTP and IHE as “core” model. Essentially it builds an outer core for “have not” users and an inner core IHE users. From the projects perspective, those that feel that simple messaging should be a priority for the “have nots” (like me) can choose to work with the SMTP related code. People who want to work in the interests of the “haves” can work on the universal SMTP-IHE bridge.

I call this straw man “Rich if you can read it”. From what I can tell it balances the core perspectives of the two interest groups on the project well, with places for collaboration and independent innovation. It’s more work, but it does serve to make everyone happy, rather than everyone equally unhappy with a compromise.

Footnotes and ramblings:

Don’t read this if you get bored easily.

I believe that this proposal excludes a REST implementation unless it acts so much like SMTP that SMTP experts can support it easily. Which begs the question, why not just use SMTP. SMTP fully supports the basic work case. The code already works and a change to REST would reduce the pool of qualified supporters.

I should also note that no one, ever is suggesting that the same program be used for email as for NHIN Direct messages. I think we should posit a “working policy assumption” that any NHIN Direct SMTP user would be required to have a different program for sending NHIN Direct messages. Or at least a different color interface. Perhaps Microsoft can release a “red and white GUI” clinical version of Outlook for this purpose… Sean can swing it…. or users could use Eudora for NHIN Direct and Outlook for regular mail. Or they could be provided an ASP web-mail interface.

We might even try and enforce this policy in the network design:

We should use SRV records for the NHIN Direct SMTP network rather than MX records. There are reasons for doing this for security reasons (enables mutual TLS) and most importantly, it means that there will be no PHI going across regular email. When someone tries to send an email message to me@fredtrotter.com their SMTP implementation looks up an MX record for fredtrotter.com to see where to send the message. If we use SRV for the DNS records, and you tried to send PHI to me@nhin.fredtrotter.com you would create an invalid MX record for nhin.fredtrotter.com such that MX for nhin.fredtrotter.com -> nothing.fredtrotter.com and nothing.fredtrotter.com was not defined in DNS. This will cause an error in the local SMTP engine without transmitting data. But the NHIN Direct aware SMTP server or proxy would query for SRV for the correct address and enforce TLS and be totally secure. Normal email messages to nhin Direct addresses would break before transfer across an insecure network but the secure traffic would move right along. Obviously this is not required by the core proposal, but it is a way of ensuring that the two networks would be confused much less frequently. This plan might not work, but something like this should be possible.

This is a pretty complex email setup, but SRV is growing more common because of use with XMPP and SIP. Normal SMTP geeks should be able to figure it out.

NHIN Security and Privacy, who is responsible?

I do not know the answer to this question but I am trying to figure it out.

I an active member of the Security and Trust Workgroup of the NHIN Direct project. We are making a few decisions there regarding “rubber meets the road” security infrastructure decisions. But we are very intentionally trying to “bubble up” security and privacy policy decisions to other policy making organizations. But I have to admit, I am not sure who those people are.

In one sense, every healthcare provider in the US will have to make security and privacy policy decisions on their own. There are already some good laws regarding health information and one might argue that given those laws, specific policy details should be left up to providers.

Of course, HHS has an ARRA created group called the HITPC or (Health Information Technology Policy Committee) that will apparently be playing a central role in general NHIN policy making. Further there is a sub-committee there called the Privacy & Security Policy Workgroup. Apparently, if there was a single group who my group would “bubble up” issues to… this would be it. Their charter is:

The Privacy & Security Policy Workgroup will address Privacy and Security in the health IT policy context. At a very high level, the new Privacy & Security Policy Workgroup will define and address the policy challenges related to privacy and security; discuss a set of principles around privacy and security; and various methods of ensuring privacy and security.

The term “very high level” is somewhat problematic from my perspective because the kinds of questions I would like to see answered are pretty specific like “What should NHIN Direct users take into consideration as they choose a provider of X.509 certificates?” That does not sound like to me to be “very high level”.

However, there are some people in this group who have technical know-how. At least some of them should be able to speak the language that I am trying to use. Some of them I know personally. Others I have never heard of. I decided that I would share with you what little information I was able to glean about this small group…

This is exactly the type of group that should be overlooking high-level security and privacy issues. They have lots of different perspectives and lots of different skills, but they all have a very relevant role to play in the future of healthcare information privacy in the United States. But I do not think this is the group to answer the question: “What should NHIN Direct users take into consideration as they choose a provider of X.509 certificates?”

I am happy that at least some of the members of this group would at least know what I am talking about.

I hope this linked list of names is more helpful to you then the list at HHS, which does not really tell you much.


Science Conjecture in Science policy

Science-based policy is pretty difficult to do, but I support the notion.

But when is science conjecture taking the place of science? I know that this guy is not being quoted directly, but paraphrased by a reporter. That has been done to me enough that I know this strips what a person actually says of any nuance… but still something about the following statement makes me very comfortable.

The response of the doctor in charge was paraphrased as:

He acknowledged that it was impossible to specify just how many cancers were environmentally caused, because not enough research had been done, but he said he was confident that when the research was done, it would confirm the panel’s assertion that the problem had been grossly underestimated.

Does this scare the heebie jeebies out of any one else?

A person in the role of scientist making a policy recommendations based on what science will “soon” find? What does that even mean? Don’t get me wrong, I think we should totally be on carcinogen patrol, be when do good intentions begin to betray the scientific process?


Doctor impatient with NHIN security dithering

Recently, the members of the NHIN Direct Security and Trust working group (brian and I at least) were criticized for dithering:

> Fred and Brian,  I love what you’re trying to do for FOSS and data
> interchangeability.  You’re dedicated, smart, expert programmers and
> systems experts. You want to be socially responsible, and protect
> people from HIPAA violations.  I think your design principles as
> expressed at http://nhindirect.org/Design+Principles  are exactly
> right, but I think NHIN has been led astray as to what its mission
> ought to be, and this matters very much to me as an eventual user of
> health data interchange.

leading to…

> Okay, let’s get this straight. Under the HIPAA law,  Covered Entities,
> such as doctors offices, are responsible for protecting PHI, not the
> manufacturer of the fax machine the office uses to send information to
> another office, or the folks who write specifications for fax
> transmission.  Does this still make sense?
> OK.  Your job at NHIN is to design a fax machine.  “Just the fax,
> ma’am.”  Covered Entities, such as doctors offices and hospitals, are
> responsible for what information will be sent, and are responsible for
> protecting it (at BOTH ends).  Another government agency (hhs.gov/
> ocr)  is responsible for enforcing HIPAA, not NHIN.    You just have
> to provide the highway to send the information, and make sure it gets
> to an actual covered entity. That’s ALL you have to do!
> I think you’re suffering from ‘Mission Creep’. You’re trying to make
> sure no one ever violates HIPAA with their fax machine.  Dang near
> impossible.  Thankfully, not your job.Yes. It makes perfect sense.

But I am not arguing that with you.

Doctors trust fax machines to provide private point to point communications.
The do not even wonder if fax machines “actually” provide private point to point communications.
In fact fax connections can be sniffed, which is why they have tele-walls now (a firewall for telephones)

And even that does not prevent the “wrong number problem

But even if we accept your premise, that we need to make NHIN Direct “just like a fax machine” someone, somewhere dithered about how faxing should be implemented. They went in circles with different ideas. They negotiated between different vendors of machine technology and they came up with a standard that could be implemented by a hardware company to build a fax machine. Here are some of the results of that.

To make something “simple like a fax machine” we need to have a set of instructions that people like EPIC, Medsphere, Google, Microsoft and Indivo can all implement identically. Those instructions might feel like they are far too complex for the simple task at hand. However we are already following Einsteins edict: “as simple as possible, but no simpler”. I think you will find that almost every discussion on the forums -has- a point. A valid concern that really should be addressed.

Main difference between a FAX machine and NHIN Direct is that, for whatever reason, Doctor’s do not distrust the security implications of a fax.
However, they -already have- faxes, which they already trust. When NHIN Direct and CONNECT come out, the “trust models” and “trust implementations” will be subject to intense scrutiny by security researchers who are decidedly better about thinking about these issues than I am. If a strong majority of those security researchers are not generally satisfied with our decisions, then they could act to cause doctors not to trust our design. This is a potential problem even if we design it right.

You are absolutely right, the law makes doctors responsible for HIPAA violations, but you forget the extension, they may also be held responsible for trusting the wrong technology. If a doctor downloaded Kazaa and accidentally published patient files on the Internet, do you think that a judge would be patient with his argument that ” it was a flaw in the technology!”. It is the doctor’s responsibly to be reasonably sure that a technology does do something that would be illegal. Currently doctors trust the “known-good” fax network. But will they trust NHIN-Direct? The answer is “of course they will”. But only because the security community will look at what we are proposing and say “well that looks reasonable” and they will only do that because we are getting our proverbial shit proverbially together.

I would suggest that you please be patient with our security sausage-making, I believe you will like the final result.


Technology vs Policy for privacy

I have long been an advocate of reasonable and measured reaction to “privacy scare tactics”. I have argued, for instance, that it was a good thing that HIPAA does not cover PHR systems. But that does not mean that I do not think privacy is important. In fact there has been something nagging at the back of my brain for several years now:

We typically use technology to provide security, but we use typically use policy to protect privacy.

That is deeply problematic. To see why we should carefully consider privacy and security in a simple context.

Imagine that you have purchased a safe-deposit box at a bank. Safe-deposit boxes are a great example, because if the bank accidentally gives away the money from your account, its not really a big deal… after all, its a bank, if they lose your money, they probably have some laying around somewhere else. The “lost money” could be restored to you from bank profits. But if you have baby pictures, your medical records, your will, Document X or your family jewels (not a metaphor) in your box, the bank cannot replace them.

Now to “protect the contents” really subdivides into two issues, security and privacy.

Security is the degree of protection against danger, loss, and criminals… according to wikipedia.. at least for today 😉

In the bank context “Security” means that no one can easily break into the bank  vault and grab your “Document X” from your box and run off. Of course there are many movies about how bank security can be overrun, but, thankfully, it is not a typical event for a bank have its security deposit boxes robbed. Banks use both technology (the vault, the alarm system, the video system) and policy (only the bank manager can open the vault, the vault can only be open during the day, etc etc) to protect the security of the bank boxes. Note that security is a spectrum of more secure to less secure, there is no such thing as just “secure”. Security can almost always be improved, but eventually improving security begins to interfere with the usefulness of something. If the bank vault could only open once a year, it would be more secure, but not very useful.

In the world of health information “Security” means that is difficult for a hacker to break in and get access to someone’s private health record. That is an oversimplification, but a useful one for this discussion.

Privacy is something else. The source for all knowledge (at least until someone changes it) says: Privacy is the ability of an individual or group to seclude themselves or information about themselves and thereby reveal themselves selectively.

In the bank example, Privacy is all about who the bank lets into the deposit box. For instance, if they decide suddenly that all blood cousins will now get automatic access to each others safe-deposit boxes, that would be a violation of your privacy. If your cousin could get access to your Document X because the bank let him, then the problem is not that the vault walls are not thick enough or that the combination is not hard enough. The problem is that the bank changed the basic relationship with you in terms of privacy. This is the first of several principles: Security Technology does not necessarily help with privacy.

Banks do not typically change the deal like that, sadly, modern websites regularly do. Two recent examples are the launch of Google Buzz and the Facebook privacy problems including the most recent expansion. The problem with modern information companies is that they usually take the corporate upper hand with their privacy policies. Sometimes this is really egregious, such as the original HealthVault Privacy policy, which gave permission for Microsoft to host your health data in China. But almost always it includes the key phrase “We reserve the right to change this privacy policy at any time”. If we sum up the problems with using privacy policies to protect privacy we find another principle: Privacy Policies often provide little privacy as written, often give permission to change the policy in the future which negates any notion of commitment, and even then policies are often ignored or misinterpreted. Privacy Policies should not be the only thing protecting our privacy.

Perhaps you want your spouse to be able to get access to your box. Perhaps you even want your cousin to have access. But the idea that the bank can just “change the deal” from what you had explicitly allowed is pretty strange. Thankfully banks rarely do that. But you could use technology to ensure that your privacy was protected, even if the bank arbitrarily changed its policy.

If you wanted you could keep a safe inside your safety deposit box, and keep your Document X inside the safe. Then you could give your combination to your spouse, so she/he could also open both the safe, (as long as you had also told the bank to give the access to your  to the safety deposit box to your spouse). Even if the bank decided that your cousin should have access to your box too, it would not matter, since your cousin could not open your safe. (we will pretend for the sake of this analogy that explosives and other means to circumvent the security of the safe would not work and the safe could not be removed from the safety deposit box). Our next principle: It is possible to use technology to help protect privacy.

Note that the safe also protects your Document X against access from bank employees. This is important because it does not matter what the privacy policy is, if it is not enforced by the employees of an institution, or if it the policy is arbitrarily changed in a way that you feel violates your privacy. It is also important to note that the government employees could not open the safe either. Of course we all know that governments are not inclined to violate the privacy of its citizens, but if the government did get a look inside the safety deposit box, all they would see would be the safe. Here we have another principle: Privacy technologies should prevent unwanted access from insiders and authority figures as well as from “bad” guys.

Which brings me to my point. We need to have more technologies available for protecting health information privacy. We have lots of technologies available for protecting security, but these do not protect privacy at all. These technologies, if they are going to work, need to give people the power to ensure that their health information is protected even from the people who provide a given technology service.

So far we have several principles:

  1. Security technology does not necessarily help with privacy.
  2. Privacy policies should not be the only thing protecting our privacy.
  3. It is possible to use technology to help protect privacy.
  4. Privacy technologies should prevent unwanted access from insiders and authority figures as well as from “bad” guys.

To these I would like to add some implied corollaries:

Encryption, by itself is not a privacy technology. It is a security technology, but it is only a privacy technology depending on who has the keys. This was the problem with the infamous clipper chip. The first issue with Information System privacy is, and will always be, “who has the keys?”.  So when a service says in response to a privacy challenge “oh don’t worry its all encrypted” that’s like saying: “You are afraid that I will fax document X to your mother? Don’t worry I keep document X at home in my safe!” If you are afraid that I am going to fax document X to someone, the fact that it is now in my safe should not make you feel more comfortable. I can still get in my safe and I can still fax document X wherever I want. Document X being in a safe is only helpful if you trust everyone with the keys to the safe.

The other thing is that you simply cannot trust proprietary software to provide privacy. If you cannot read the sourcecode to see what the software does, then it does not matter what kinds of privacy features it advertises or even who has the keys. At any time the developers could change the code and make it violate your privacy. To further extend and abuse our example, it does not help you to have a safe inside the safe-deposit box if the bank can change the safe’s combination whenever they want, without you being the wiser. Only Open Source systems can be trusted to provide privacy features.  I do not argue that this is enough, but it is an important starting point.

I have been working on a relatively complicated system for achieving these goals. My initial application is blessedly simple, and so I have been able to avoid some of the issues that make these kinds of systems commonly intractable. My service is still in skunkworks and will continue to be very hush-hush until I make the beta launch. But I will be announcing my designs soon and I have already submitted those designs to some Internet Security professionals that I respect to make sure I am moving in a sane direction. This kind of thing is really technically difficult, so I am certainly not promising that I can deliver this kind of technology, but perhaps I can deliver something working that would give other people a place to start. As you might expect, the sourcecode for this project will eventually be made available under a FOSS license.

I do not want to get into the design details yet (so please don’t ask) or even talk about my new application (which is just entering closed beta) but I wanted to start talking about why these issues are important. Please feel free to comment on the features that privacy protecting technology should have…