"What makes you a senior engineer" gets posted on Reddit every couple weeks it seems. My answer is that you're not a senior engineer until you've completely broken production and had to fix it while an entire company watches you fix it. During this time you dig deep into a skillset that includes communication, leadership, keeping calm under pressure and of course the ability to code something. These kills take years to develop.
Fixing your own mess while everyone watches really brings out the best in a person - or the worst, I suppose. When you take down a production system, you get a sinking feeling in your stomach the moment you realize what you've done. And then that's followed by a real "oh shit, what did I do?" moment. Your hands start to sweat, your heart races, and you sort of get dizzy.
Breaking the Sytem
Breaking something in production when nobody notices is one thing. But breaking an entire production application that affects all the users is another. Let's say you deployed a small (untested) controller or updated a config file that you assumed wouldn't make a difference. Nothing bad is going to happen, right? I'll just deploy this change and go about my business...Wrong. Instead, the logs are filling up and the Slack messages and emails are all flooding in. The change you were so confident about returns a fat red 500 error for the entire organization and everyone wants to know what happened. Users can’t use the system, C-level bosses want to know what's wrong and they want to be updated on the progress every five minutes. The entire team is looking at you now. You're now promoted to crisis manager.
Being the Senior Engineer
Everything has turned to complete shit and you can a) play dumb and wait for someone else to fix it or b) roll up them sleeves and get to work. Since this is a post about what makes a senior engineer, we're going to focus on the getting to work part. Here are a few of the key traits that every Senior Engineer should posses to make them successful in their software development careers.
Working under pressure
You keep a clear head while everything is breaking by staying calm and handling the situation - even if there's complete chaos going on inside their mind and body. Everything on the outside says: "This guy is chill." You have to be able to get results while your palms are sweating.
Accountability
Even if the issue was not caused by you directly, a senior engineer won't throw anyone under the bus and will never blame someone else. They'll step it up and show ownership and accountability. You'll know how to handle the problem, resolve it, and manage the people involved. A good senior engineer follows their process and works through the problem without pointing fingers. This can be hard, but blaming someone rarely makes the situation any better.
Thinking Critically
Diagnosing issues logically and never thinking anything is too stupid to be the problem. Go over all the things in your experience that you've done to resolve a problem in the past. Nothing should be out of the question.
Working under pressure
What really makes someone a senior in this process is how they handle all the pressure. Being able to stay calm under pressure while feeling panic and carrying the weight of the situation is a strength that comes from experience. But still being able to communicate to your team and company while fixing the problem, now that's the skill of someone who has been doing this for a while. It's not something you learn in college but rather a skill that is developed over time through making mistakes.
Excellent debug skills
Debug complex systems to find the root cause and then code their own fix or recommend a fix to another team. Not all bugs are created directly as a result of code so having the ability to figure out if it's a database, network, server, or other infrastructure issue is something you pick up with experience as well. This comes from experience debugging all kinds of systems over the years.
Communication
Senior level engineers have the ability to explain the root cause to all the people each with different levels of technical understanding. Each group in the organization speaks a different level of tech-speak. If you've been doing this long enough to call yourself a senior developer, then you know the language for C-level stakeholders, other developers, and support staff. All these people speak different tech languages but a senior developer will know how to speak to each of these people on their level.
Part of communication involves knowing when to ask for help. You don't work on an island (well, some people do) so a good engineer will know when to ask other engineers or folks from other groups to help. There's now shame in saying you need help.
Welcome to the Senior Engineer Club
The first time you break production and recover from it, you level up. You're no longer just a mid-level coder; you're an engineer who understands the job isn't just coding features. You can handle high-pressure situations while communicating effectively and taking ownership to get things done.
Have a production horror story? Let’s talk. I’ve been there, I’ve fixed that, and I help teams prevent it from happening again. Need a consultant who’s been through the fire? Let's jam!.