I ran a project once where we implemented a new call center phone system for the IT help desk of a very large financial institution. Every call for tech support from the organization’s 250,000 employees came in through this call center and was distributed to a hundred or so call center agents. Early the morning of go-live, everything was going well – our sponsors and stakeholders were excited and happy, there was a donut party, agents were happy with the new tools we gave them which promised to make their lives easier. We sat in a glass-walled ‘fish-bowl’ conference room in the center of the call center floor and everyone waved and gave us the thumbs up as the day got off to a fantastic start. Everything was great! Then it wasn’t.
A couple hours into our morning, every light on our system monitoring dashboard went red. For some reason, every phone line froze up and not a single call from the entire organization could make it through our system to the call center agents, who now all started to stand up from their desks and look at us questioningly through the glass-walled fishbowl conference room.
As my engineer frantically tried to diagnose the problem and get the system back up, a half dozen stakeholders with titles of VP and above rounded the corner and started marching toward our fishbowl. They were not giving me a thumbs up. They burst into the room and demanded to know, “What the hell is going on? Why is our call center down?”
