The hero culture is alive and well in IT. They’re sometimes known as the “Firefighters”. These are the heroes who come in at all hours of the day and night to put water on the latest IT fire. In many organizations a “good” fire fighter is admired and appreciated more than a good developer or other IT contributor. Why shouldn’t they be admired, they come in at 2:00 AM on Sunday and resolve a major failure that was interrupting the business. What’s not to like?
Well, let me first say that I don’t have anything against the Fire Fighter. They are very martyr-like, and it can be easy to appreciate that quality, as long as it doesn’t come with a bomb vest. What I don’t like is the culture of fire fighting that we as IT leaders perpetuate. The real issue is whether or not someone in the organization has the courage to get off the exercise wheel for a minute and say “this has got to stop”. As a general rule, those closest to the issue (fire is a great example) aren’t in the best position to determine how to avoid similar issues in the future.
The following is an almost verbatim conversation I had with a Global Director of Infrastructure I worked with for a short time;
Me: We’re not making progress on our project to integrate NewCo. We’ve got to find a way to reduce the level of interrupt-related work the team is tasked to perform.
Director of Global Infrastructure; I’m sorry, but I don’t have time to deal with this right now, we’re all too busy fighting fires.
Do you see any problems with this conversation? How will you ever have time to do productive work if all you’re ever doing is fixing messes that were created because you didn’t have time to do productive work!!
It may sound odd, but sometimes laziness has its place in the business. Some of the best IT folks I know work very hard, but they don’t realize they’re working hard because the work they’re doing is helping them avoid work they don’t like doing. Every IT organization needs a few leaders and contributors who can look at the job at hand and say, “How can I fix this so I never have to do it again?”
If you celebrate the contributions of firefighter martyrs, you are in effect rewarding bad behavior. There may be a wide range of reasons for the fires in the first place, but you certainly don’t want to make it worse by establishing the wrong success motivators.
In closing, I suggest stepping back from the fire, no matter how fierce the heat is and look for the true organizational and technological root causes. Once you’ve found them and fixed them you’ll be in a position to start bringing real value to the business, instilling pride in your team and increasing job satisfaction and therefore employee retention.
So the next time you see a fire, get out a stick and a marshmallow and put away the fire extinguisher.
Latest Tweets
Are the firefighters the same as those who cause the problems in the first place? In most sensible shops, there is a split between operations and development. We may need to firefight, but I have grown up – metaphorically speaking – in a culture where we firefight, and then, the next day we come in and make sure that particular spark can never catch again. I consider it a poor week if I have to firefight.
I think you have a point regarding the organisational root causes. Unfortunately you have to get management to address those and very often, they are also amongst the causes. We don’t need heros, what need is the ability to avoid needing heros.
I think that most of the time, the *firefighters* actually know how to eliminate the fires; it’s the *management* that can’t prioritize and is all-consumed with the firefighting their employees are doing…
Treasa & AIXGuru,
To some extent I agree with both of you. I believe the truth is that an environment prone to firefighting is the result of poor leadership. It doesn’t really matter who causes the fires. The fires start as a result of poor practices, and improper reward systems.
To Avoid Fires:
- Implement strong change management
- Implement strong release to production processes (part of change management)
- Use traditional root cause analysis and process of elimination to determine why certain problems keep occuring
- Leadership must know what their teams are really doing otherwise they will be prone to only paying attention to “firefighting” work. If they pay attention to the firefighting instead of work that improves the environment, the employees will naturally look to spend more time on fixing problems, instead of avoiding them
In the end it’s a leadership deficiency, it’s not the fault of the actual firefighter.
[...] communication will also result in less exhaustion for your team. Mark Thiele recently wrote about the down side of IT heroics. His points are equally applicable to report creation and communication. I cannot think of too [...]
Hey Mark,
Great piece. Thanks for writing it. Having been a firefighter in the good old times, i remember thinking the same, but I also decided to start just fixing things on my own, even without having the real authority for it. What i would do is prioritize my work to optimize (ie. make as long as possible) my nights.
I took some hits for that at first, but I kept on, and when the guys up top saw the uptime go up, and their operations team not half-zombies sleeping on their keyboards during the day, they started thinking differently (well at least most of them).
I guess my point is, that the FireFighters should just step up and do something for themselves. I might not be easy to go against management, but I’m pretty damn sure that it’s easier than waking up at 2a to stop a cronjob that went berserk (I’m using a clearly outdated example here, but that was one of my experience
).
I also think that the rise of Devops is going to change this. We’ll see in the long run if it really does, but it has a great potential to do so.