956UTC May 17,2019 – a date & time that will go down in Salesforce infamy.
I don’t want to be presumptive and say that 956UTC is a larger outage than NA14, but having witnessed both, I can say I do not recall NA14 getting a super dark (and cool) codename like Permageddon.
From an Admin, End User, Partner, Customer perspective, Permageddon hurt. The financial costs of this incident has to be in the millions when you factor in lost productivity for customers, lost revenue for partners, shifting deadlines for deployments, trainings, inability to process orders, leads not being captured, etc.
Over the coming days we’ll get the RCA & RCM from Salesforce and be able to dig into the root causes and future remediation protocols, but for now, I want to focus on the good that I saw in the midst of Permageddon.
Saint Walter
When the Customer Bridge calls started, the Ohana was introduced to Walter Macklem, Chief Architect of Product Engineering at Salesforce, who seemingly was unknown to even the most connected and experienced members of the SF world.
He handled the pressure of the calls with a calm, even demeanor, provided great technical information, promised to find answers (AND DID) or bring things up to technology team (AND DID THIS TOO), and was definitely the right person for this momentous task.
As we reached the waining hours of Friday, you could hear his exhaustion which continued through the weekend, but his manner on the calls and obvious expertise developed a cult following amongst Ohana members, who are now his biggest fans. Though not active on twitter, he can be found @waltermacklem – leave some love, I’m sure he’ll see it.
OhanaSlack
Yes, I know, this is my 113th shoutout to Ohana Slack, but seriously, Meighan Brodkey, (Mother of Ohana Slack, Enforcer of Threads, Denier of Spam, First of Her Name) has built an amazing place for all members of the Ohana to gather and share information & expertise.
This channel was very active and was providing some of the first official/unofficial information regarding the outage that allowed man admins and partners to get out in front of this issue with their orgs before Salesforce made any official acknowledgement.
The activity in this community saved many of us hours of effort and provided a central hub for updates throughout the weekend.
BlueCanvas
BlueCanvas has a continuous integration and deployment tool for the Salesforce Platform, but when Permageddon started, they jumped into action to support the entire Salesforce community gratis.
At BlueCanvas.io you can gain access to the Profiles & Permissions Deployer tool to migrate permission sets and profiles between unaffected environments and affected environments. While there are other ways to do so, this is by far the fastest, easiest method and extremely valuable for orgs without a dedicated admin or deep expertise on migrations.
They also kept a running blog of Customer Bridge calls and provided regular status updates. Kudos to BlueCanvas.
MVPs in Action
I’m very new to being an MVP and I have a very hard time thinking of myself as one to be honest. Seriously, holy crap, every day interacting with this group just blows my mind. I’m in perpetual Wayne’s World, we’re not worthy mode, and trying to hang like I belong.
My insecurities aside, watching this group from Friday through today was incredible. Constant updates, people posting and communicating 24/7, getting up early, staying up late, taking time away from their weekends to capture notes, share information via all channels, update their co-workers, end users, managers, etc.
What an inspiring and powerful group to watch in action.
SF Exchange @ Discord
Appropriately named, I was first introduced to this platform and SF Exchange channel as the 956UTC Crisis broke loose. The members were the first reporters of the incident & had some of the fastest breaking information from reliable sources, definitely brought a high-level of reliable information throughout Friday & the weekend.
To join the Discord channel, click here.
MEMEs
Oh the MEME game was strong. Here are some of my favorites:
I’m gonna shout out the Salesforce Exchange Discord (SFXD) as well!!! They had queries saved to help people see if their org was affected and mass tagged everyone at 3am pacific time with notification that something was wrong and was identifying problems before Salesforce officially mentioned it.
Good stuff here Tom. Thanks.
Great post Tom! I’m a big fan of bluecanvas.io and of course, thanks for the #OhanaSlack shout out, happy to help during the issue however we could. Here is the OhanaSlack overview and signup link: https://mrockssf.site/OhanaSlack
As I enter day 5 of the void of permissionless profiles, this helped me stay sane. Glad I was able to get exposure into even more of the great community!
This is a fantastic blog; thank you for writing it! You definitely deserve the MVP title, and this right here is a perfect example why! ❤