Anatomy of a Deployment Failure (pt. 3) – Production Changes and the Flight of The Icarus

 Deployment Failure (Pt. 3) Production Changes & The Flight of the Icarus

Confession time: in the four or so years I’ve been working with Salesforce, I’ve never really ‘got’ Sandboxes and change sets.

I suppose I’d tell a prospective employer that this demonstrates my ability to vision and action a modification without disrupting users. Then again, I think I’d tell my therapist it was a clear sign of self-destructive behaviour.

However, I would also point out that nobody in the company I work for would describe me as a Salesforce Administrator first and my boss would tell you that my role is procurement, logistics and compliance, adding that it’s really boring and that I’m a nerd. 

The perception of my directors is that salesforce is a tool that we use, not the job that I do – a tool that I fought hard for, and cost incurred at my recommendation.

Maybe one day there’ll be a point that being an admin becomes my full-time role, which is what I want for my career, but until then, I’m operating around some constraints:

  1. I can manage my own time, but implementation and enhancement cannot encroach on my day-to-day responsibilities.
  2. Operational concerns always take precedence over Salesforce.
  3. Stop talking about Salesforce. Oh my god honestly you’re so boring, stop it, this is so dull.

To really understand the work environment I operate in, I also need you to leave any notions about project management behind. We do not scrum. My Kanban board is covered in crude drawings. Agile methodologies are eschewed in favour of a JFDI mentality.

So when someone senior asks me for a change, it’s not because they’re thinking strategically – it’s normally in response to an immediate problem that’s reared its head. Which means that there’s no good saying “yes, in 3 months time this immediate problem will go away.” The only suitable answer is “there’ll be a solution tomorrow.”

So I look at what’s needed. If I’m feeling fancy, I’ll map it out on a piece of paper (and if it needs to look like I’ve considered things wholly, I’ll throw some meaningless highlighting in): 

The highlighter colours really don’t add anything. Also, marvel at my over-use of process builder to mask my complete inability to grasp flow builder.


And I know I should be less reckless. I know this should be planned, mapped and considered. I absolutely know how dangerous this is. But the thing is, 95% of the time, it’s not a problem. 95% of the time, my brilliance is applauded and ridiculed in equal measure.

Then there’s the 5% – although every now and again, my users rebel, and try to make out that it’s closer to 50%.

You’ll recall from part one of this series that my bogey user, Leslie, was struggling to get through her leads. There were a myriad of factors affecting this, but two major ones were duplications and the sheer volume of leads she had to get through.

Nobody likes deduping, but a quick shoutout here to DuplicateCheck for – well, not making it pain-free, but ensuring no scarring. 

But after that point, there wasn’t much I could do for Leslie in terms of the volume of leads she had to get through. After consulting with her and Samantha (the closest thing I have to a superuser), we came to the conclusion that by presenting telesales users with a more minimalist UX for uncontacted leads (whereupon components would only become visible as the lead progressed), we could speed up the process.

So this was half an hour’s work on a Friday afternoon; ten minutes to build, twenty minutes to show. 

“Will you be alright, showing the others how this works on Monday?” I asked Samantha and Lesley, aware that I’d be on holiday for the next week.

“Yeah, should be simple enough,” said Samantha.

A week. In an environment free from all sensible safety nets like an implementation roadmap and user testing, that’s a lifetime. I could (and have) dismantle and rebuild our entire org in a week.

And a week is all it takes for a classic sidekick betrayal.

In my second post, I talked about my absolute favourite new fictional verb, Icarusing. It’s a term I’ve coined to reflect a specific relationship trajectory with a specific type of user that can be represented most accurately as a bell curve of famous movie moments:

In the time it took to make this, I could’ve taken trailhead modules on both change sets AND flow builder. I regret nothing.


My Icarus emerged whilst I was enjoying my annual leave. See how far you get in this tale before you spot the Icarus:

I’d been driving home from France for 18 hours straight when I finally fell through the office door at 8am on Monday. I felt hungover from the driving – teeth fuzzy, stomach churning from one energy drink  after another.

Sit down. Boot up. Coffee. Log on. A tap on my shoulder.

“How do, buddy!” A broad northern English accent; there was no need to turn around for me to recognise Jeff, our Sales Director, fresh out of the ‘80s, all gangly limbs and meandering gold stories that we’ve all heard a thousand times before. “Missed you this past week.”

Now there’s a sentence to make your shoulders slump. Jeff will often tell you that he’s missed you, but he doesn’t mean it in an affectionate sense (although by the sentence’s very design, it’s meant to feel that way) – he misses you in much the same way a car misses a cambelt. It means there’s been a problem, and you caused it.

“Would you join us in the meeting room?”

Us. Have you ever been invited to a meeting room by your boss to sit with a group of people and have it turn out well?

“Oh, of course Jeff, let’s get right to it,” I said, wearing a smile I absolutely did not mean. “Give me a clue, would you?”

“Nothing that I’m sure can’t be put right,” said Jeff guardedly as we walked through the offices. As we walked through the office, Junior Sales Guy Stan (three quarters consecutive missed targets) gave me a sad sort of sympathy-grimace-smile, the surest indicator going that everyone in the company knows what I’m walking into.

In the meeting room – a drab, cold former office overlooking a warehouse with a table too large for the space – Samantha sat waiting, her back to the ancient monitor mounted to the wall. I had to fight tooth and nail to get that monitor in place, but we never used it. 

I moved to the end of the table, pushing the wireless keyboard and mouse out of the way as I sat down. The screen came on in response.

“We’ve got some questions about what you did to our leads,” began Jeff.

Concerns, maybe,” added Samantha, syrup-sweet. “Not so much in terms of what you’ve done to it, but how you introduced it to the telesales team.”

“Slapdash,” added Jeff. “Haphazard.”

“You’ll have to be more specific,” I said. “I introduce at least three haphazard, slapdash causes for concern to this company every day. Can we narrow it down? Had I eaten?”

After an awkward silence, Samantha was the first to respond.

“It’s not that hiding those components on the leads page didn’t work,” she said. “In fact, once we figured it out, we can see the benefit. It’s that you didn’t bother to communicate it to the team at all.”

“We all think you’re very clever,” said Jeff in what he thought must have been a reassuring tone.

“It’s just that every time you do something clever, our telesales grind to a halt figuring out the changes,” said Samantha. “You don’t communicate the change.”

“But I did communicate the change,” I said – perhaps a little whiny. “I communicated it to you and Leslie, and you agreed that you were going to explain it to the rest of the team.”

“Well, you didn’t explain it well enough,” she said. “Come Monday, we didn’t know what you’d showed us. But it’s okay, we’ve made the decision to give you a bit more support with this. From now on, you can handle the technical stuff and I’ll handle the training.”

“Yes, I’m sure your three trailhead badges make you the perfect candidate to train them,” I retorted.

“Well, I have learned from the best,” she smiled at me. “Jeff’s fine with extending my hours to accommodate.”

And therein, I realised, lay her motivations for this – extending her hours. Now, I don’t begrudge anyone taking the initiative to earn more money – but I can’t say I’m a particular fan of being thrown under the bus in order for this to be accomplished.

“I mean, all the telesales girls want to do is come in and call whoever’s first on the list.”

(How obvious you are in hindsight you are, Samantha. How clever you must have felt, dispatching Leslie to my desk two weeks ago to moan about her leads. Did you prep her with that phrase, I wonder, or has it entered your office’s lexicon as part of this new narrative you’ve developed?)

“We absolutely can’t have what happened on Monday happening again. It sounds like it was absolute chaos.”

Sounds like?

“Were you not here for it?” I asked.

“No, I was out on-site.”

“Sorry – Samantha, can you just run me through the chaos?”

A gut feeling began to form, and I slowly pulled the mouse towards me without either of them noticing. Under cover of both hands, I opened the salesforce shortcut on the desktop and logged in.

“Well it’s not like the place was on fire,” she said with a fake laugh. “But we ground to a halt. Barely any calls got made at all.”

“How long for?” I asked. Reports tab. New report. Activities with Leads.

“A couple of hours I’d say,” she answered. “Maybe three.”

“Is Monday morning normally a good time to target leads?”

“I mean it’s the start of the week, best foot forward and -”

“It’s not massively, I wouldn’t say,” interjected Jeff, who had noticed what’s happening on the screen.

“Still, though,” I continued. “Four team members, a couple of hours – that’s a person day’s worth of activity lost, possibly more. I get what you’re saying.”

Add column. Last activity. Add filter. Last activity – the Monday in question. Add two more filters for the two Mondays previous. Add today’s in, just for good measure.

“Yes,” she said, still mercifully unaware. “It’s hard to attribute, but I genuinely think that’s why we’re down on targets for last week.”

Then came the tricky part. How to change the filter logic from AND to OR without typing. The solution came from copying and pasting the or from the URL –

Last activity column. Group rows by this field.

Add Chart. Run.

Thank goodness for data.

“Samantha,” said Jeff, staring at the screen. “Have a look at this.” 

She turned her head and saw what Jeff and I were seeing; that last Monday’s numbers were exactly in line with the preceding weeks shown.

“Looks like the team managed to recover from their collective12 hours off the phones,” I said helpfully. “Good for them.”

“I suppose the telesales team may have overstated the impact,” she said through gritted teeth. 

“Someone’s certainly been overstating something,” said Jeff.

“Listen, for my part, I’ll make sure that going forward, everything is properly documented and announced ahead of time,” I said. “No more last minute changes, everyone will be fully-aware of everything. Do you need me for anything else?”

“No, that’ll be all,” said Jeff. Then, as I got up, he added: “Can you make sure that the sign on the door’s set to Do Not Disturb?”



Leave a Reply

Your email address will not be published. Required fields are marked *