Interviewing Software Engineers

Interviewing is something I've given some amount of thought. It's certainly something I have experience with; more of it recently than I wish I did. I'm hardly an expert, but I have read and talked about it because it's important. I think it's commonly accepted that as a collective field, tech is not very good at this. I've served as an interviewer at various times, and I certainly was given no training or preparation for that. In the time I've spent on the outside of that process, it's been clear that the lack of support I experienced as an interviewer is common.

Just don't do whiteboard interviews

There are a lot of problems with the way tech interviews people. But they mostly boil down to evaluating the wrong things, or evaluating things in the wrong way. White board coding challenges—and honestly any coding challenge—is the ultimate example of both. It's a bizarre, contrived, high-pressure test that strips the candidate of all of the tools, context, and support that they would rely on to do actual work. People's performance in their work is a product of their environment. For software development, some of the critical components of that environment are the tools we use, the knowledge we've gained about the systems we're working on and the problems they address, and the team we work with to build that software. Virtual whiteboard tests are not really any better, in my experience. Online tools are more helpful than a physical whiteboard, but they're not at all the same tools we would use on the job. And the extra tool support seems to just encourage interviewers to pick harder, more elaborate assignments. Take home projects might be worse, because while they do potentially allow people to use their actual tools of choice, it also severs the tenuous connection they had to a member of the team. They also take much longer, and developer interviews are already frequently long enough that it's exploitative and exclusionary.

So, if white board problems are out, and online programming challenges are out, and take home projects are out, what's left? Not much. And that's fine. If you're asking what test should you use, that's the wrong question. None of these tests are useful. Just don't do them. I think the reason they're done in the first place is because they're easy. Easy to administer, and easy to evaluate. But the information they provide is useless, at best. In fact it's often misleading, and creates lots of opportunity for bias to sway an interviewer's perception. We know that the work done by the candidate in whiteboard coding tests will not be representative of the work they do on the job. There is no relationship between the two. So then what is being evaluated? Mostly communication style. Also confidence and affluence play a large role. But mainly it's a question of whether the candidate thinks aloud in a way the interviewer finds familiar. Whether they happen to follow a logical path the interviewer agrees with. Whether they prioritize a working solution vs an exploration of the problem vs whatever else in the way the interviewer prefers. Or any number of other things which are not an expression of competence at writing software, which is the ostensible point of this challenge. And because these are mostly about communication styles, values, and priorities, they will favor people from a similar background as the interviewer. Given that tech is ~90% men and ~70% white, that means this situation favors white men. It just does. It's 2021, and so much of tech has been saying for so long that we want to increase the diversity in our companies. I'm taking those claims at their word. Interviews and recruiting are neither the cause nor the solution to the uniformity of people in our field, but the way we do it definitely isn't helping, either.

I should also discuss interviews which consist of pairing to do real work on real projects. I don't recommend this, either. Yes, it does actually address most of the problems with artificial code challenges. But it also creates new ones. First, it requires the interviewer to multitask and split their focus between very different tasks. I seriously doubt that people can conduct interviews effectively in this situation. It's also very time consuming. It definitely is not a time effective way to learn about the candidate's development skill. It's unpredictable. If the session consists of doing real development, there's no way to make that consistent across multiple interviews. And if a pair gets stuck on something, through no fault of the candidates, that will still likely hurt the candidate. While the candidate would be able to talk to at least some of the team, they're not really a member of the team. They don't yet know how or where to seek or offer help. It's functionally their first day on the job, except instead of focusing on their onboarding, they're expected and required to work productively. And finally, this is actual work. As in, it's a job. It's the job that the company has not yet decided to even offer. This work should be paid, but it rarely is.

What to do instead

Artificial white board programming tests are the wrong thing, but they're easy to assign. Doing the right thing is going to be harder. A big part of what makes it hard is that there is no universal way to evaluate programmers. Programmer is too big of a category to be useful as a guide for us to design an interview process. And when we try anyway, the result is that we end up finding and duplicating someone else's process. We do things the Google Way™, for instance, because we assume Google must know what they're doing. The thing is, whether they do or not, Google doesn't know what you're doing. So what we really need to do is identify what we're looking for in a programmer, and what role we expect them to play within a development team. By far the most common way to break that down is by seniority. This is certainly not the only way, and it ignores a number of paths a developer's interests and career could take. But, it's a starting point.

It's not really possible to discuss how to evaluate something unless you know what you're evaluating it for. So, that's what I'm going to do. And that's what you should do also. These are roles that I feel are generally applicable, based on my experience. I think you could do a lot worse than to use them as-is, but you would be better served by examining your own teams, projects, and needs, and producing definitions that fit your situation. Once you have real definitions of what you're looking for, the ways you could evaluate that become much clearer.

Junior and Entry Level

Engineers who are just starting out in their career can really bring a lot to their teams, but it can be hard to quantify just what that is. I think the easiest thing to grasp is that they will learn quickly. This is good from a skills standpoint, because the things they're learning are your systems, and projects, and processes. They'll rapidly become experts on a lot of these things, and contribute that knowledge to the team. They will also, if they're encouraged to do it, do this learning in front of their peers. This kind of visible curiosity, learning, and growth is a huge benefit to the psychological safety of a group. At this stage, what you should focus your interview evaluations on is to be sure that they have knowledge and skills you can work with as a baseline to teach from. You want them to know enough already that they can easily learn more.

Proficiency with tools. Software programming depends on a lot of specialized and technical tools, especially within a team. They need to know how to use code editors, compilers, other build and test tools, version control systems, code review tools, and more. It's probably not necessary that a candidate be an expert with all of these, but they should be familiar with some of them, to start with.

Foundational programming concepts. We are here to write software, after all. But software is a big concept, with dozens of layers of abstraction that we use to manage vast complexity. To be helpful on a project, an early career developer needs to know some of that, but only some. I would focus on data types and control flow. Do they know how to use integers and floating point numbers? Do they know how and when to use arrays and lists? Are they familiar with using conditionals and loops? This is the level of programming skill a new developer should be judged by.

Curiosity. As I mentioned above, one of the main responsibilities of a junior developer on a team is to learn and become familiar with the system. This will be much more successful when they're looking forward to it. When they seek out new challenges for themselves, and readily ask questions about things they haven't seen before. Just don't let this become a gatekeeping mechanism like "passion" often is. Curiosity is something that you can foster in a safe environment, so it shouldn't be a concern if a junior developer isn't overflowing with it in an interview.

Mid Level

Mid level developers should make up the core of a healthy software development team, in my opinion. By this point, a developer has the skills and confidence to work quickly and effectively, without needing frequent help. Interviews can focus on discovering those programming skills, and on the kinds of skills and habits you need to work effectively in a team.

Composite programming concepts. The foundational concepts expected of junior developers serve as a base to build upon. This is what you build with it. One key point is understanding, selecting, and applying data structures and algorithms to a given problem. It's highly unlikely that a programmer at this level needs to be able to reinvent those from first principles. Rather, you should focus on the ability to research and learn about them, and tailor them to the needs of a project. You could also discuss the details of a language or framework when it's a high priority for you. Likewise it can be useful to discuss resource management, such as battery, memory, storage, and so on. If you do, be mindful that these things are easy to learn, and anyone who's made it this far as a programmer has likely gotten good at learning.

Teamwork. Software development is a team sport. It would be pretty rare for someone to be the only developer on a project. And so working with a team is just as important as working with code. At this level, I think it may even be more important. Working well with a team can save an overconfident developer from spending a lot of time and effort going down a bad path. In this vein, you can talk about comfort levels with dividing and sharing work. Talk about how and when to raise concerns. Discuss strategies for sharing important changes or proposed changes.

Senior Level

At senior level, software developers are much more responsible for supporting and empowering their team than their less experienced peers. While they would still spend the biggest portion of their time working in software, they also serve as a kind of force multiplier to everyone else.

Systemic programming concepts. This is another step or two up in scope and abstraction as compared to the composite concepts that you would focus on for mid level developers. In terms of just code, this includes things like design patterns and programming paradigms. As before, you should discuss how to analyze and apply design patterns to a given problem. Explore the tradeoffs of functional vs object-oriented programming. But at this level, engineers should also be concerned with things beyond just their application code. So you should also talk about testing strategies, or the concerns that would shape a CI/CD system, and so on.

Knowledge sharing. The best way to be a force multiplier is to help to distribute knowledge throughout the team. This obviously means teaching their peers about various concepts and details of the system. But it also means learning it themselves in the first place. Talk about strategies for doing that kind of teaching and learning, and about ways to prevent knowledge hoarding or to avoid knowledge becoming stale and outdated.

Collaboration. It will be common that senior engineers are called upon to work with other teams, not just their own. Seniors will need to be able to convey the team's needs and capabilities to stakeholders. They'll need to understand and communicate their own technical constraints. And they'll be asked to evaluate requirements, estimate progress, and advise on future prioritization.

Staff/Senior+ Level

Staff level engineers are leaders for their team. This leadership comes as technical direction and guidance, as well as in shaping the team's habits and environment. In software projects, a team has to be concerned with doing things the right way, and with doing the right things. A staff engineer's greatest contribution is in pushing to do the right things.

Holistic programming concepts. When working with code, and often even when they're not, a staff level engineer needs to be aware of the total system and how it functions. Or, if not the total system, then at least enough of it to understand how their parts of it fit into the whole. This is naturally a question of system design. But, it also involves concepts like resilience, observability, and security. You should discuss methods of accomplishing these things, both technically and organizationally.

Team Dynamics. Shaping the way a team works is about promoting appropriate, supportive, and healthy team dynamics. A staff engineer can have a major influence on this, starting by modeling and praising good behavior. You should discuss communication styles and techniques, at least. Staff engineers should also contribute to the team's and individual's growth and development by taking a role in teaching, mentoring, and sponsoring the team members. And yes, even interviewing.

Psychological safety. Staff engineers have a lot of power to create a psychologically safe environment for their team. They also have a lot of power to sabotage that safety. When interviewing, you should explore their views of what that safety is and how to build it. Some good techniques are to highlight and praise learning, establish blameless practices, encourage experimentation, and model curiosity. At a base level, a lot of psychological safety is a question of trust, and your discussion could encompass an understanding of trust and how to build and preserve it.

Other models

The seniority model that we commonly use conflates programming ability with leadership ability. While these two things do often grow together, that doesn't have to be true. Especially in domains which have more intense technical requirements than application programming. My experience is in applications, but I imagine that system programming, embedded, real time applications, and other domains in that vein might value highly specialized programmers with very deep technical knowledge who don't play as broad of a role otherwise as I described above. That's perfectly fine. I don't go into much detail with that model for a couple reasons. One, it's just not where my experience is. And two, a great many organizations interview as though that were their model, and it quite simply is not. There are already many more groups than there should be who focus more and more and more on comprehensive algorithms knowledge to the exclusion of most else during their interviews. So, I think this interviewing pattern needs to be discouraged, rather than the other way around.

But, how?

So, that was a lot of what to do. If you're wondering how to do it, that's fair. When I described it, I used words like talk, discuss, explore, etc. That's how. Talk to people. Talk about their experience and understanding of these systems and concepts. You can even (shock!) do so with a white board. I'm not actually opposed to white boards. They're handy for sketching out some ideas to frame and anchor a conversation. My argument is solely that demanding people solve artificial and technical problems out of context on a white board (or similar digital tool) is counter productive. So don't do that.

Cover Photo by Startup Stock Photos from Pexels