What a difference a year makes!
Actually, what a difference 6 weeks with the right team and the right focus makes. Last year, at around this time, the Healthcare.Gov site was in full meltdown, and all over the news media. Over the subsequent weeks and months, many of us spoke and wrote about what went wrong, how it could have been avoided, and how it could be fixed.
Providing rare insight into the rescue operation in the days that followed, is this landmark article from TIME Magazine. Here are some of the awesome agile things they did to rescue the beleaguered site:
1. Put Jeffrey Zients in charge, but, “the guy in the suit is never in control.”
A business executive, with a track record of delivering things, Zients was the solution to, No One in Charge. Though Zients was now in charge, it wasn’t quite as simple as ordering people around:
“Zients isn’t a techie himself. He’s a business executive, one of those people for whom control–achieved by lists, schedules, deadlines and incessant focus on his targeted data points–seems to be everything. So for him, this Apollo 13 moment must have been frustrating–because in situations like this the guy in the suit is never in control.”
“True, Zients had assembled a terrific team that had gelled perfectly. But his engineers could move only so fast. Though he had carte blanche to add resources, putting 10 people on a fix that would take one coder 10 days doesn’t turn it into a one-day project. Coding doesn’t work that way. “Jeff was a great leader, but there were limits,” says Dickerson. “He would ask us every day if we were going to make the deadline … He’d say how he had to report on how we were doing to the President. And I’d say till I was blue in the face, ‘We’re doing as much as we can as fast as we can, and we’re going to do that no matter what the deadline is.'”
2. Focused a core team under CTO Todd Park’s organization.
The site had been developed under the auspices of Health and Human Services. Puzzingly, President Obama’s CTO had not been involved in the development of the site. Mr. Park, as it turns out, is a huge fan and practitioners of lean and agile methodologies. In fact, Mr. Park was a keynote speaker at the 2012 Lean Startup conference, and shared his experience applying Lean Startup techniques within the public sector. So, they needed a solution to, Too Many Cooks?
From the TIME article:
“According to notes from a meeting in one of CMS’s three war rooms (yes, things were so uncoordinated that there were three), those assembled discussed the fact that “we heard that the capacity”–the number of possible simultaneous users–“was 100,000 people, and there are 150,000 people on it.” Yet five days later, White House chief technology officer Todd Park would tell USA Today that the capacity was 50,000 and that the website had collapsed because 250,000 people tried to use it at the same time. Park, a highly successful–but, for this job, disablingly mild-mannered–health care tech entrepreneur, had been kept out of the planning of the website. In fact, the site’s actual capacity at the time was “maybe a few thousand users,” according to a member of the team that later fixed it.”
So, at this late juncture, Mikey Dickerson, an Obama campaign veteran and site reliability engineer assembled a crack core team to fix the problem. Besides Jeff Zients and Todd Park, it included Ryan Panchadasaram, Jini Kim, Paul Smith and Andy Slavitt.
3. Worked in a rapid, but flexible manner to fix issues iteratively using a Daily Standup Meeting. Rules for this meeting:
“It was in a 4,000-sq.-ft. room rented by QSSI in a nondescript office park in Columbia, Md.–lined with giant Samsung TV monitors showing the various dashboard readings and graphs–that Barack Obama’s health care website was saved. What saved it were Mikey Dickerson’s stand-ups.
Rule 1: “The war room and the meetings are for solving problems. There are plenty of other venues where people devote their creative energies to shifting blame.”
Rule 2: “The ones who should be doing the talking are the people who know the most about an issue, not the ones with the highest rank. If anyone finds themselves sitting passively while managers and executives talk over them with less accurate information, we have gone off the rails, and I would like to know about it.” (Explained Dickerson later: “If you can get the managers out of the way, the engineers will want to solve things.”)
Rule 3: “We need to stay focused on the most urgent issues, like things that will hurt us in the next 24–48 hours.”
The stand-up culture–identify problem, solve problem, try again–was typical of the rescue squad’s ethic.”
All in all, we realize that this unfortunate saga could have avoided. So, the question that we need to ask is, why wasn’t Healthcare.Gov rolled out from the get go with agile and lean practices?
The TIME article has some strong thoughts on which we should collectively reflect:
Had the Obama team brought in its old campaign hands in the first place to run the launch, there would have been howls about cronyism. But one lesson of the fall and rise of HealthCare.gov has to be that the practice of awarding high-tech, high-stakes contracts to companies whose primary skill seems to be getting those contracts rather than delivering on them has to change. “It was only when they were desperate that they turned to us,” says Dickerson. “I have no history in government contracting and no future in it … I don’t wear a suit and tie … They have no use for someone who looks and dresses like me. Maybe this will be a lesson for them. Maybe that will change.”
Maybe.
