Google Intrusion Detection Problems

(Update: Less than four hours after tweeting out this blog post, we got our access turned back on. So the Google support team is definitely listening on social media. We very much appreciate that, because it resolves this issue as a crisis. We are still concerned by the “auto-off” trend and the missing button. But we will be working to make sure there is a better long term solution. Will update this post as appropriate moving forward.)

So today our Google Cloud Account was suspended. This is a pretty substantial problem, since we had committed to leveraging the Google Cloud at DocGraph. We presumed that Google Cloud was as mature and battle tested as other carrier grade cloud providers like Microsoft, Rackspace and Amazon. But it has just been made painfully clear to us that Google Cloud is not mature at all.

Last Thursday, we were sent a message titled “Action required: Critical Problem with your Google Cloud Platform / API Project ABCDE123456789″ here is that message.


Which leads to our first issue Google is referring to the project by its id, and not its project name. It took us a considerable amount of time to figure out what they were talking about when they said “625834285688”. We probably lost a day trying to figure out what that meant. This is the first indication that they would be communicating with us in a manner that was deeply biased towards how they view their world of their cloud service internally, totally ignoring what we were seeing from the outside. While that was the first issue, it was nowhere near the biggest.

The biggest issue is that it was not possible to complete the “required action”. Thats right, Google threatened to shut our cloud account down in 3 days unless we did something… but made it impossible to complete that action. 

Note that they do not actually detail the action needed, in the “action required” email. Instead they refer to a FAQ, where you find these instructions:


So we did that.. and guess what, we could not find the blue “Request an appeal” button anywhere. So we played a little “wheres waldo” on the Google Cloud console.

  • We looked where they instructed us to.
  • We looked at the obvious places
  • We looked at the not-obvious places

As far as we can tell, there was no “Request an appeal” button anywhere in our interface. Essentially, this makes actually following the request impossible.

So we submitted a support request saying “Hey you want us to click something, but we cannot find it” and also “what exactly is the problem you have with our account in any case?”

However, early yesterday morning, despite us reaching out to their support services to figure out what was going on, Google shut our entire project down. Note that we did not say “shutdown the problematic server” or even “shutdown all your servers”. Google Cloud services shutdown the entire project. Although we use multiple google cloud APIs we thought it made sense to keep everything we were doing on the same project. For those wondering that is a significant design flaw, since Google has fully-automated systems that can shut down entire projects that cannot be manually overridden. (Or at least, they were not manually overridden for us).

We have lost access to multiple critical data stores because Google has an automated threat detection system that is incapable of handling false positives.  This is the big takeaway: It is not safe to use any part of Google Cloud Services because their threat detection system has a fully automated allergic reaction to anything that has not seen before, and it is capable of taking down all of your cloud services, without limitation. 

So how are we going to get out of this situation? Google offers support solutions where you can talk to a person if you have a problem. We view it as problematic that interrupting an “allergic reaction” as a “support issue”. However, we would be willing to purchase top-tier support in order to get this resolved quickly. But there does not appear to be an option to purchase access to a human to get this resolved. Apparently, we should have thought about that before our project was suspended.

Of course, we are very curious as to why our account was suspended. As data journalists, we are heavy users of little-known web services. We suspect that one of our API client implementations looked to Googles threat detection algorithms like it was “hacking” in one way or another. There are other, much less likely explanations, but that is our best guess as to what is happening.

But we have not idea what the problem is, because Google has given us no specific information about where to look for the problem. If were actually doing something nefarious, we would know which server was the problem. We would know exactly how we are breaking the rules, but because we are (in one way or another) a false positive in their system, we have no idea where to even start looking for the traffic pattern that Google finds concerning.

Now when we are logged in, we simply see an “appeal” page that asserts, boldly “Project is in violation of Google’s Terms of Service”. There is no conversation capacity, and filling out the form appears to simply loopback to the form itself.

It hardly matters, Googles support system is so completely broken, that this issue represents a denial of service attack vector. The simplest way to take down any infrastructure that relies on Google would be to hack a single server, and then send out really obvious hack attempts outbound from that server. Because Google ignores inbound support requests and has a broken “action required” mechanism, the automated systems will automatically take down an entire companies Cloud infrastructure, no matter what. 

Obviously, we will give Google a few hours to see if they can fix the problem and we will update this article if they respond in a reasonable timeframe, but we will likely have to move our infrastructure to a Cloud provider that has a mature user interface and support ticketing system. While Google Cloud offers some powerful features, they are not safe to use until Google abandons its “guilty until proven innocent, without an option to prove” threat response model. 







Mourning Jess Jacobs

Yesterday, Jess Jacobs died.

Like so many others on Twitter, I knew Jess just well enough to be profoundly frustrated as I watched helplessly as the healthcare system failed her again and again. Today, the small world of Internet patient advocates mourn for her across blogs and twitter. The world of people who are trying to fix healthcare from underneath is small, and relationships that form on social media around a cause can be intense. There is nothing like an impossible, uphill battle to make lasting friendships. Now this community is responding to the loss of not only one of our own, but one of our favorites.

Is the NSA sitting on medical device vulnerabilities?

Today is not a fun day to read slashdot if you care about healthcare cybersecurity. First, it highlights how the DEA is strong-arming states into divulging the contents of their prescription databases.

Second, and even more troubling, was the claim that the NSA was looking to exploit medical devices. The story was broken by Intercept reporter Jenna McLaughlin. Since then, the article has been picked up by the Verge. Their title is even more extreme: “The NSA wants to monitor pacemakers and other medical devices”  Jenna did not specifically mention where she heard the comments, but her twitter feed gave me a hint.

The comments were from NSA deputy director Richard Ledgett, who is the same guy that countered the Ted talk from Snowden with his own. He was speaking at the Defense One Tech Summit. It is incredibly hard to find, but his comments are available as a video he goes on almost exactly at 3 hours. I tried to embed the talk below, YMMV.

In one sense this has been blown out of proportion. Patrick Tucker is the moderator/interviewer here, and he is the one that is pressing Ledgett on the issue of biomedical devices. Start at 3:15 for the discussion on medical devices.

Ledgett insists that targeting medical devices is not a priority for the NSA. But the troubling thing is his answer to the first question:

Question: ” What is your estimation of their security ”

Answer: ” Varies alot depending on the device and the manufacturer”

The problems with this is that I know of no examples of the NSA releasing data on insecure medical devices. In fact, the FDA has recently released information about specific medical devices that were insecure, without giving credit to the NSA.

This means that the NSA is investigating the security of medical devices, but then not releasing that information to the public. Ironically, it is a quantified self device that is most illustrated here. Ledgett specifically highlights fitbit, which I know had some pretty strange wireless behavior (that many regarded as insecure, in its early versions). So we know they have looked at one specific device, but there has been no release of information from the NSA on the device. At least I cannot find any.

If indeed the NSA is investigating medical devices, and is not releasing all of that information to the FDA, device manufactures and the public, then that is a huge problem.

I am still thinking about this, but it does not look good.

I suppose I should also mention that I ran across the interesting fact that Osama Bin Laden was using a CPAP machine.

Update: I have submitted a FOIA request for access to vulnerabilities about “healing devices” and it has been denied.

Does Epic resist or support interoperability? Hell if I know.

I just realized that my somewhat infamous question at the ONC annual meeting is recorded on video!

The background on my question, which I made me very popular at the meeting afterwards, was that I had heard that Epic hired a lobbyist to convince congress that it is an interoperable company.

That lobbyist and others at Epic have been heard saying stuff like “Interoperability is Epics strength”… and “Epic is the most open system I know” etc etc.. This makes me think “what planet I am on?”

I have actually heard of hospitals being told “no at any price” by Epic, and I have never heard that regarding another vendor… although there are lots of rumors like that about Epic I would prefer to be fair. How would I know if Judy et al, had really turned a corner on interoperability. Epic has been a faithful participant in the Direct Project, which is the only direct (see what I did there?) experience I have had with them.

But I want data… and here is what happened when I asked for it at the annual ONC meeting. Click through to see the video.. it auto plays so I did not want it on the my main site.

Continue reading Does Epic resist or support interoperability? Hell if I know.

Libel and Discourse in the Digital Age

Libel, like copyright, is one of the central legal frameworks for governing online activities. It sets the bounds for what can and cannot be said about people in the new media area.  Like copyright law, libel law is a legal framework designed in a pre-digital era, that is somewhat strained in this new digital media age.

I write this with some trepidation. This blog posts touches on gender issues on Twitter, and that is a heated and, at least on Twitter, mostly broken discussion.

Any discussion on sensitive issues online, especially on Twitter, can devolve into a core of reasonable people trying to have reasonable discussions that are surrounded by a much larger group of people (or at least a large number of twitter accounts) who say completely ridiculous and incendiary things. Jimmy Wales response to a GamerGate email regarding the policies for Wikipedia’s GamerGate article is required reading here.

The wonderful thing about Twitter is that it facilitates open to the public conversations about anything at all. These conversations usually involve only people who are genuinely interested in particular topic, which means that the Twitter conversation is usually representative of the topic as it exists in the real world. But a given hashtag is useful and productive, only to the degree that people all agree on what the topic under discussion is, and also fundamentally agree on what is the appropriate means to have that conversation.

Many times, both of those constraints fail, and this is when you get a single hashtag, like #GamerGate being used in at multiple conflicting ways. One way is to have a discussion about “Ethics in Game Journalism”, the second is to launch a coordinated attack on female game journalists and other feminists, and the third is the feminist community using the hashtag to refer to those attacks. In the sense that all three things are happening at once using the same hashtag on Twitter, all of them are equally valid and equally invalid uses of the hashtag. But all three discussions regularly lament that the other two discussions are trying to “redefine” what “GamerGate” “is”. The letter from Jimmy Wales helped me realize that there is an inherent difference between a movement and a hashtag. Before reading that I was deeply confused on how think about “GamerGate” a word whose definition changes dramatically depending on who you listen to.

Generally I think the power of Twitter lies in its capacity to have public conversations that serve only as “signals”, with larger discussions on topics left to more forums that are better suited for comprehensive discussion, like blogs. Twitter is ill-designed to handle contentious issues, in part because Tweets are necessarily atomic in nature. It is too easy to take a single tweet, and then lambast that single tweet as the entirety of someones position. This is not strictly a straw-man tactic because it actually takes a little work to get Twitter to contextualize any discussion. Twitter presents tweets as atoms, and not threads on a topic.

On Twitter, there is a lot of “What I said was X, but what I meant was Y”. As an informaticist, I would call Twitter something like a “Communication Platform with Low Semantic Fidelity”. Which is not an insult to the platform… this is both a “feature” and a “bug”, depending.

So it is with great irony that I found myself having a discussion about libel, on the very platform that makes the issues around libel so complex.

For those who have been living under a rock on Twitter lately there has been a drama unfolding regarding the role Vivek Wadhwa plays regarding women’s gender issues in technology. The play continues to unfold, but here is the outline of the opening scenes:

  • Wadwha makes a statement onstage referring to “floozies”. (have not been able to find video of this)
  • Mary Trigani writes a post entitled Captains and Floozies criticizing Wadwha’s comment.
  • Wadwha comments on the blog post.
  • Trigani reposts the Wadwha’s comment with the title Vivek Wadwha explains
  • Amelia Green Hall, writes QUIET, LADIES. @WADHWA IS SPEAKING NOW which sternly criticized the role that Wadhwa plays and how he plays it.
  • This blog post caused enough of a stir that Amelia was subsequently interviewed by Meredith Haggerty on NPR’s TLDR series. This podcast (which is still available here) is essentially a retelling of Amelia’s blog post in audio form, with no dissenting voice from Wadhwa or elsewhere.
  • Wadwha reacts on twitter saying that the podcast is “libel and slander
  • NPR removes the podcast from their page, although as per normal it will be remembered forever on the Internet somewhere…
  • Twitter presumes that the post was removed because of Wadhwa’s “threats”
  • Wadhwa insists that he wants the post itself restored, but merely wants to have the opportunity to blog in the same space.
  • Apparently, his interactions with NPR makes him believe that he will be able to publish a retort on the NPR site.
  • For whatever reason, Wadhwa’s defense is not published on NPR, so he manages for it to be published on Venture Beat instead.

Which brings us to current. (I will try and update the timeline if things change)

Obviously it’s interesting stuff in it’s own right, but I am mostly interested in the issues around libel. Specifically, I am interested to understand if it was in fact libel, and I am interested to know if the fact that Wadhwa labeling it as libel was a “threat”.

Lets deal with the first issue. Was it libel? Well it turns out that this is not a clear legal question, especially for Wadhwa. You see in the US, the legal test for libel typically has three components (IANAL and I am quoting Wikipedia, so you would be foolish to take this a legal advice).

  • statement was false,
  • caused harm,
  • and was made without adequate research into the truthfulness of the statement.

(from wikipedia)

Unless, you are a public figure, and then libel also includes “Proving Malice”. Again quoting wikipedia:

For a celebrity or a public official, the person must prove the first three steps and that the statement was made with the intent to do harm or with reckless disregard for the truth, which is usually specifically referred to as “proving malice”

Listening to the podcast there are several statements that stand out specifically as false:

  • ..”Has he really been this spokesman for women in tech for years while he is believing that women can’t be nerds because thats because thats like super misogynist”..
  • (on the website of for Wadhwa’s book) “I can get to a photo grid of women it doesn’t list their names..” (Wadhwa points out that such a list lives here)
  • “Wadwha was barely acknowledging the women he was working with”
  • Wadwha was “Gaslighting minimizing marginalizing people who disagree with (him)”
  • The story implies that Wadwha titled his response to Trigani’s post “Vivek Wadwha explains” when in fact Trigani had made that title.
  • The DM’s that Wadwha sent were “non-censual”.

If you listen the to podcast, and you read Wadwha’s rebuttal, it is pretty easy to see understand how Wadwha at least would view these statements as false, harmful, and inadequately researched. Wadwha is painted as pretender, a person who who is taking the role of “real” expertise. The implication here is that there is something essential to the experience of being a women in technology that is required to acquire legitimate expertise about women in tech. At the same time, there is the implication that the experiences of women in tech so vastly distinct that no one person could possibly make useful statements about them as a class.

This is an interesting issue with civil rights in general. There was a time when the racial civil rights movement choose to exclude white supporters from leadership positions. This makes sense when you are dealing with a pervasive attitude that presumes that a particular class is fundamentally incapable of self-representing and/or leadership.

But there is a difference between requesting that someone bow out of a leadership role, in order to further the aims of a social justice movement, and attacking the qualifications and intentions of that same person in the most public way possible (i.e. on the radio and Internet at the same time).

On the other hand, if there is a person claiming leadership in a social movement, while saying or doing things that hamper that movement, it is a natural reaction to eventually (after back channel discussions have failed) to out that person in public.

So which is it? Is this a necessary exposer in defense of an important social movement, or it is petty dramatics within a movement that should be above such theatrics?

What the hell do I know? Although I am at least a little interested in anything that qualifies as social justice, I am hardly an expert in this area. I don’t know any of the parties involved and I have no familiarity with the book and research body in question.

What I am interested is how libel works in the Internet age. What is fascinating specifically to me is the degree to which Wadwha is being criticized for calling the podcast “libel”. It is fairly clear to me that IF the contents of the podcast are misrepresentations, then Wadwha is just being publicly attacked. The whole podcast was about him, not about “men speaking for women generally”, but just about him and what he was specifically doing wrong. The podcast implied that he was a lecherous, misogynist, manipulative plagiarist. IF those things are not true about him… then does he have the right to say “This thing that is happening is slander and libel” without inappropriately using that language to squelch criticism.

According to Wadwha, he has made no legal threat, he did not ask for the article to be taken down and, in fact, he has asked for it to be restored. That does not generally sound like the acts of someone who is seeking to muzzle critics.

What I find fascinating, is the apparent consensus that merely labeling the podcast as libel IS itself a legal threat.

Here are some reactions from two lawyers who work for the EFF (an organization I admire and donate to)

And then here..

Lastly this is one specific quote from someone who has been on the other side of this.

However, I did find this gem from @DanielleMorrill, who was obviously researching this earlier than I was. She found places where Wikipedia policies cover these issues…

For my part, I cannot help to empathize with Wadwha. My family has had some pretty nasty run ins with people willing to publish false things about us. If someone in traditional media decides to smear you, its nearly impossible to undo the damage. At least Wadwha had the opportunity to tell his side of the story, an opportunity my family never got. 

Apparently, the consensus on the Internet, and what I would advise people do on this, is to just say. “Hey that stuff you wrote/said about me is not true, and its pretty hurtful and you really should have researched that better” instead of actually coming out and saying “Thats Libel”. Its pretty clear that Wadwha tried to take a position of “You have libeled me, but I am not planning on suing you, I just want to achieve balance”, and from what I can tell, that has blown up in his face, and possibly made things worse for him. 

I have certainly learned several things from this incident that will make me slightly less likely to put my foot in my mouth: Specifically…

  • I should be careful not to speak over other people on panels. I am frequently the most vocal and opinionated person on a panel. Audiences frequently ask questions specifically to/for me, and moderators will frequently favor me because I can be entertaining. But apparently when Wadwha does the same thing he is percieved as “taking the air out of the room” etc etc. I would never want my fellow panelists to feel they don’t have a voice b/c of me. I will have to work on that.
  • Apparently there is a whole contingent of women who have been so completely harassed by DM’s that saying something like “A non-consensual DM” actually makes sense to them. I had no idea that Twitter harassment had reached that level for women. I mean you have be brave or crazy to let someone know you are a female user on Reddit (which is sad), but I thought Twitter was a “safe place”. I was wrong.
  • When someone labels themselves as rude or mean or otherwise thinks that it is a good idea explicitly admit in their twitter profile that they are difficult to deal with… believe them. They are not kidding. Its one of these things. Lookup the Far Side cartoon that says: How Nature says “Do not touch”. Its just like that.
  • I need to be careful to explicitly not speak “for” the people I personally advocate for (which in my case is usually patients) b/c this can be disempowering. I need to find ways to advocate without being presumptuous, which is harder than it sounds.

Thanks for reading, I may well update this post based on reactions from Twitter and elsewhere.








EHR Vulnerability Reporting issues

For those who actually bother to read to the bottom of my bio, I was actually in Internet Security before going into Health IT. I spoke at DefCon and everything.

During my career in Health IT I have had to report a security vulnerability to an EHR developer once, and it was such a painful process that I basically just gave up.

My poor friend Josh Mandel and his group at SMART found an XSLT vulnerability in an HL7 provided file that is a part of essentially every modern EHR system (the standard, if not the file itself, is mandated my Meaningful Use).

They have had a horrible time trying to get the attention of the major EHR vendors, with less than 10% paying any real attention.

I am saddened, but not at all surprised. I will write more later…


How to submit prior art on the Medicity Direct Patent

Recently Medicity has tried to patent the concept of a HISP. Please join me in submitting prior art to prevent this undermining of everything that the Direct Project stands for.

Groklaw shows the way

Here is a specific page that I had some trouble with and the right answers for it…

The Patent number in question is 61/443,549

The confirmation number is: 9529

The first names inventor is: Alok Mathur , Alpharetta, GA (US)

The date of file is: 02-16-2011

The strange string they are going to ask you in the middle appears to be: 201161443549

Read Groklaw carefully because the form is massively unnecessarily complex. (Because that is how the government rolls)..

The following prior art exists for their claims:

* Conversion of encrypted payload content, perhaps CCDs, into HL7 2.3 transactions sent to an EMR over TCP/IP ports

Of course, converting to HL7 v2 is not actually a good idea in 99% of the cases, but it was always part of the original vision of the Direct Project


Just search this page for HL7 to find Arien discussing the need for HL7 2.x interoperability

or you can read about how we dithered over 2.x versions of HL7

I will no dignify the fact that they note that this happens over TCP/IP with a comment. Really, you are going to use the networks protocol for that?

Are you sure you do not want to use UDP? Or perhaps IPX? Wow. Innovation. <- (sarcasm, see note for USPTO employees below)

* Conversion of encrypted payload content, perhaps HL7 v3, into rendered PDF formatted reports that are automatically printed to a local printer device per the provider’s workflow preferences.

* Construct of a standard Direct compliant outbound S/MIME transaction with CCD attachments by converting native PDF or HL7 v2.x formats and contents.

This of course makes direct look like a fax machine. Which is a -huge- step backwards. But generally, converting between different healthcare interop standards has been done for quite some time.

A main goal of the HISP is to convert between various formats. We spent months talking about the particularly difficult conversions, i.e. Direct to IHE

As far as I know the central advantage of a PDF is that you can print with it.

Here is Keith Boone discussing the issue on his blog


This is 2 months too late but shows that we including printers as possible devices to send direct messages to.

The second set of claims is particularly annoying to me because I got involved in Direct specifically because it was not possible to do coordination of care without an underlying point to point messaging infrastructure.

  • Sharing of virtual care team records across disparate networks

  • Dynamic updates to disparate patient reocrds using encrypted serialized patient objects across disparate networks

  • Sharing of application context within applications across disparate networks

  • Sharing of user context within applications across disparate networks

  • Establishing long-term patient and provider object-level communication across disparate networks.

Its late, so my patience for this is wearing thin. Email handles “sharing PHI across disparate networks”. The whole fucking point of direct is that is -just- email.

So everywhere that Medicity is saying “share (PHI Type here) across disparate networks” they are full of shit. This is the problem that Direct itself solves.

Then the question becomes. “Hey, now that we have this amazing capacity to share PHI across disparate networks, what specifically should we share?”

Hmm… perhaps we should use this to keep patient records in sync… no shit.

(in case you cannot tell. The preceding text is sarcasm. I am saying this because someone from the USPTO might be reading this, and I am not sure you might not have picked up on that. Working at the USPTO might be the kind of job where you lose your sense of humor. I am just saying. )

The whole concept of a HISP is that it site on the edge of the Direct network and integrates the local environment into Direct.

Medicity has a HISP product. It does things that HISPs do.

They do not deserve a patent for concepts that are -both- obvious and well described by the Direct community during the -entire- process of developing Direct. The fact that the US government did not dictate what a HISP should do does not mean that it was not discussed carefully, completely and commonly by everyone working on this project.

The “HISP as a bridge concept” is something that I had a hand in creating. I do not appreciate my own work being co-opted and abused in this fashion. I am requesting that Medicity withdraw this patent application, and consider… I don’t know… competing for Direct HISP business, instead of applying for bullshit patents on ideas that were created as part of an Open Source project.










About to have a call with the National Health Service

I am about to have a call with a group of people who work with the UK National Health Service.
I know for a fact that the people on the call are doing serious, thoughtful work on behalf of their government.

In contrast, my government just started paying the electricity bill again.

It is fairly hard to describe accurately how I feel about going into a call like this. Happily I have Reddit/Imgur to help!!