As with everything, it depends. Live coding interviews work. They’re not the best candidate experience, but they work at Meta, Google scale, minimizing false positives better than most other formats. What makes them stressful is the lack of interviewer training and the abstract, puzzle-like nature of the problems, which you can really only solve if you’ve spent time studying (e.g., LeetCode) or you’re fresh out of college or academia.
I’ve worked in the assessment space for 6 years and have seen many hiring processes, from Fortune 10 companies to startups hiring their first engineers. The range of signal required, and how much candidate experience you can get away with, is huge. I’ve also been a candidate myself and failed many traditional live coding interviews. It made me feel terrible about myself.
When I work on my product, I try to view it through the lens of empowering candidates to show their skills and potential. I encourage our customers to use assessments that somewhat resemble on-the-job skills. I don’t like the phrasing “real work” anymore. An assessment shouldn’t be unpaid labor, it should be a way for candidates to demonstrate that they can do the job and handle future work thrown at them, and for hiring managers to feel confident extending what are often very high salaries in tech.
With AI, the landscape is shifting. Unfortunately, short take-homes (what I prefer as a candidate, using my own tools and editor) are becoming harder to maintain as a fair signal due to AI assistance. I’ve seen companies move back to onsite, and competitors deploy all kinds of proctoring and invasive monitoring.
The perfect solution, in my view, would be an assessment where the candidate feels relaxed and able to perform at their best, knowing that every other candidate in the pool has the same constraints in terms of time and tooling. It’s a tough problem to solve.
pelcg · 52s ago
> As with everything, it depends. Live coding interviews work. They’re not the best candidate experience, but they work at Meta, Google scale, minimizing false positives better than most other formats. What makes them stressful is the lack of interviewer training and the abstract, puzzle-like nature of the problems, which you can really only solve if you’ve spent time studying (e.g., LeetCode) or you’re fresh out of college or academia.
They work (for some) and leet-code weeds out the frauds that really cannot problem solve and assesses those who have not built anything to show to justify not doing it and can be applied to companies that are joining from an acquisition.
> The perfect solution, in my view, would be an assessment where the candidate feels relaxed and able to perform at their best, knowing that every other candidate in the pool has the same constraints in terms of time and tooling. It’s a tough problem to solve.
And that would be the fairest one which is to do the leetcode interview in person on a projected whiteboard and pair programming with interviewer.
Very relaxing.
zwnow · 19m ago
A brick layer isn't asked to build a wall prior to getting a job. A certificate confirming he did learn the job is enough for companies to employ them. Same goes for a lot of other jobs. Just hire devs and kick them out if they dont fit ur needs. Why go through these humiliating corpo processes?
michaelt · 8m ago
> Just hire devs and kick them out if they dont fit ur needs. Why go through these humiliating corpo processes?
People actually find being fired a greater inconvenience and humiliation than a 60 minute whiteboard interview.
piqufoh · 12m ago
In many non-US countries once hired there are employment rights. You cannot simply "kick them out if they dont fit ur needs". Isn't it preferable and less stressful for everyone if you can find the right person without having to hire and fire others first?
jasode · 12m ago
>A brick layer isn't asked to build a wall prior to getting a job. A certificate confirming he did learn the job is enough for companies to employ them.
It depends. A welder looking for a job -- even with a certification -- will often have to demonstrate his welding skill to the jobsite supervisor before he is hired. An airline pilot -- even with a rank of "captain" with 5000+ hours -- when trying to find employment with another airline, will still have to demonstrate piloting skills with a "check ride" as part of hiring approval process. Sometimes, experienced pilots fail the check ride for whatever reason.
falcor84 · 13m ago
Brick laying is a job where the progress is immediate and easy to assess. The better analogy would be of hiring a civil engineer - would you hire one to work on your project based just on a certificate?
ozgung · 1m ago
How do you hire a civil engineer? Do you ask them to solve a series of construction puzzles for few hours?
appease7727 · 13m ago
So you think only people wealthy enough for a college degree should be programmers?
zwnow · 12m ago
How did u get to this conclusion?
darkwater · 8m ago
You wrote
> A certificate confirming he did learn the job
What would that certificate be?
grim_io · 10m ago
It's probably ego and delusion.
Every crud generator company thinks it's special and has some secret sauce it needs to guard.
I'm sure some do, but overall it's a case of monkey see (google), monkey do (like google).
gabrieledarrigo · 4m ago
> As with everything, it depends. Live coding interviews work
You cannot start your reasoning with "it depends" and then continue with an absolute.
I could do the same:
As with everything, it depends. Live coding interviews don't work.
What's the difference?
belter · 16m ago
Sounds you are looking to hire competitive coders not software engineers,
garphunkle · 3m ago
I thrive in high-stress situations (for short periods of time). Examples include hardware validation before a large production run, putting out literal fires in manufacturing sites, and working in foreign countries to troubleshoot/rework bad hardware. I do fine in live coding interviews, they don't feel much different than being alone at an editor for me.
I was interested by the author's statement: "Working memory is the most reliable proxy (I know of) for fluid intelligence, your ability to reason, solve novel problems, and think abstractly." and the linked study (https://pubmed.ncbi.nlm.nih.gov/21037165/). My working memory is not so great, but it degrades less under stress.
Question worth considering for hiring managers: do you prefer stress-capable employees, or greater working-memory employees? Is my model a false dichotomy?
codeflo · 47m ago
I don't think those explanations are mutually exclusive.
Yes, there's a large cohort of "senior" software engineers who can't actually code. They bullshit their way into jobs until they're fired and then apply for the next one. These are the people you want to filter out with live coding.
But also, someone can fail a live coding interview for reasons other than belonging to that group.
onion2k · 29m ago
I think there's a lot of developers who can ace a live-coding interview but who lack the understanding of engineering systems at scale so they'll make your whole codebase worse over time by introducing tech debt, anti-patterns, and inconsistencies. These are the people you really want to avoid, but very few interview processes are set up to filter them out.
There's an assumption that the company's existing senior architects and developers will stop a new person from making the code worse, but also devs at every company thinks their codebase is terrible so it obviously isn't working.
whstl · 23m ago
I agree. Live coding always has a much smaller scope than real software, and after a few interviews it is easy to learn to read the room, even for the worst developers.
I think we can leave companies who don't care about quality out of the discussion, but for those who do, the time to detect those developers is in a probational period, which is not something that most companies really use on their favor.
The problem is this requires a good management that is able to actively paying attention to the work of the developer. Which is often not in place, even in companies who want to prioritize quality :/
zwnow · 25m ago
I've seen lots of devs who think their codebase is the only correct way to do things. Lots of overconfident people out there. Inconsistencies are fine as long as there's file level consistency. All that really matters is if you can relatively quickly understand what you are working with. What you really want to avoid is having functions doing 20 different things from 5 different contexts.
TinkersW · 42m ago
You could filter then out much more effectively by letting them sit in a room by themself to write the code, that way you aren't missing out on good candidates who can't function when under abnormal stress(that has nothing in common with the actual job).
sigmoid10 · 30m ago
I've had take home problems for job interviews that were given a few days before and during the actual interview I only had to explain my code. But I wouldn't be sure this still works as a useful candidate filter today, given how much coding agents have advanced. In fact, if you were a sr dev and had a bunch of guys to bounce this problem back and forth, it wouldn't even have filtered out the bad ones back in the old days. There is very little that is more telling than seeing a person work out a problem live, even if that sucks for smart people who can't handle stress.
arp242 · 2m ago
My solution for this was a propose a problem obscure enough that no LLM tool really knows how to deal with it. This involved some old Fortran code and obscure Fortran file format.
How would someone explain code that was vibe-coded?
squeaky-clean · 19m ago
You can have the AI explain it to you. There's also a middle ground between vibe coding and "I can code some things but never could have coded this without an AI".
Doesn't even have to be AI. Give me some random file from the Linux kernel and I could probably explain everything to you if you gave me a few hours. But that doesn't mean I would ever be able to write that code.
whstl · 15m ago
I don’t disagree, but in those interviews the explanation is also a bit of a q&a, so an effective interviewer can detect candidates who only memorized things. Someone who can ace a q&a about Linux code is already better than average.
sigmoid10 · 23m ago
Are you asking how they would get that info they didn't have / couldn't come up with? Because you can literally have a chatbot explain every line to you and why it is there and even ask it about things you don't know like a teacher. And for simple problems this will probably work better than with actual humans.
gitremote · 14m ago
I assume questions to explain the code would be extremely specific about why you did something on a specific line, or why you chose to design this part that way instead of another, to detect plagiarism and vibe coding, not a request for a prepared monologue.
dnissley · 38m ago
We live in a remote world where a hiring process like this is less of an option in most cases
ghaff · 25m ago
Leaving aside that many companies have pulled back from remote to at least some degree, I'd always push for an in-person day for a variety of reasons. In general, the cost is nothing for a late-stage/end-stage confirmation. And, honestly, a candidate that just doesn't want to do that is a red flag.
bugjoey · 26m ago
I share your point of view, but live coding these days are just beyond that testing programming skills. You must know by heart the most common algorithms out there and design solutions that might involve two or three of them to solve a problem in 30 minutes.
Sometimes you spend the whole time trying to figure out how to solve the puzzle that don't even have time to show that you can - actually - code.
Paul-Craft · 35m ago
Why do you need arbitrary (and very short) deadlines, and for someone to stand up at a whiteboard while simultaneously trying to solve a problem and "walk you through their thought process" to filter out people who can't write code on the job?
ndriscoll · 27m ago
The short deadlines are because neither the company nor the candidate wants to spend a month on an extended interview. Solving a problem and walking through the thought process are because that's what "coding" is.
Paul-Craft · 23m ago
I don't know about you, but I've never had to live code a PR and explain to my reviewer what I was thinking while writing the code. By "deadlines" I'm referring to the length of the interview. Take home problems theoretically solve both these issues, but they need to be properly scoped and specified to be valid assessments.
ndriscoll · 16m ago
I sit down with juniors and sketch out designs or code for them while talking through the thought process at least once a week, and even when solo coding, I expect everyone produces work that explains itself. For particularly complex/nuanced changes, people do hop on a call to talk through it.
Like I said the deadlines work for both sides. If a company wants to give homework instead of having their own senior engineers spend time talking to me, that tells me what I need to know about how they value my time.
Paul-Craft · 14m ago
> I sit down with juniors and sketch out designs or code for them while talking through the thought process at least once a week, and even when solo coding, I expect everyone produces work that explains itself. For particularly complex/nuanced changes, people do hop on a call to talk through it.
That's not equivalent to what I said, nor is it live coding.
Again, those deadlines are artificially short compared to real world scenarios, and completely arbitrary. They are so short, in fact, that they render the interview an invalid test of real working ability. A work sample has been proven time and again to be the most valid measure of whether a candidate can actually perform the job, but the conditions under which a live coded "work sample" is performed in an interview render it invalid.
ndriscoll · 10m ago
It's not artificial: the company has a day of my time. I have a day of their time. We both want me to meet several people on the team to see if it's a good fit. Because of the constraint, we keep it to relatively simple discussions around toy problems that can be solved in an hour.
Paul-Craft · 2m ago
Yes, it is artificial. Everything about a live coding interview is artificial. Code doesn't get written in 1 hour blocks while someone's watching over one's shoulder, all the while asking questions to interrupt one's thought process, in any company I've ever worked for.
Mountain_Skies · 13m ago
In most of the western world, firing employees is a high risk, high cost task. Ideally companies would hire quickly and fire poor matches just as quickly once they've been evaluated in the real world environment of the company. For this to work, on the employee side there needs to be knowledge that this is the company's process, financial depth to deal with the job not being stable, and a savviness to not relocate into a job that's risky. On the employer side, there needs to be a legal and social environment that doesn't punish removing non-productive employees.
The legal environment is what it is and unlikely to change. The social environment is fickle and trend driven. Workers can't always fully evaluate their odds of success or the entirety of risk of leaving a job that's valuable for the employee and employer for one that might end up as a poor match, even if both sides have been transparent and honest. It's a difficult matchmaking problem with lots of external factors imposed and short term thinking on all sides.
Ideally young workers would have an early career period that involves a small number of short lived jobs, followed up by a good match that lasts decades, providing value to both the employee and employer. Much like finding a spouse used to be a period of dating followed by making a choice and sticking with it so a life could be built together, employment ideally should result in both sides making the other better. Today however everyone seems focused on maximizing an assortment of short term gains in search for the best local timescale deal at the expense of the long term. It's interesting how the broken job market and broken family formation process in the western world mirror each other so much.
paxys · 13m ago
There are two ways to interview:
1. Make sure you pick every good candidate, but some bad candidates will slip through as well.
2. Make sure you reject every bad candidate, but some good candidates will fail as well.
Candidates want process #1, but companies have no reason to push for it. The cost of accidentally hiring a bad employee is simply too high, way more than rejecting a good employee. The current system in place prioritizes #2. Yes they are rejecting great candidates, and they are aware of it.
sceptic123 · 3m ago
The article is suggesting that #2 will end up rejecting LOTS of good candidates (and potentially ALL female candidates)
tediousgraffit1 · 1m ago
> You’re not rejecting a bad engineer, you’re rejecting someone who doesn’t perform well while being watched.
a related thing here is that I've long believed that remote work positively impacted my career for precisely this reason.
mmusc · 21m ago
Having gone through a round of interviews, I completely agree that live coding doesn’t measure anything of substance.
There was one company whose process I really appreciated:
- A take-home test, more of a puzzle-style challenge.
- The interviewer does a quick review of your submission.
- If they like what they see, you're invited to a follow-up session where you explain your code and make a few simple changes to it.
This approach proves that the work is genuinely yours, shows your thought process, and still gives the opportunity to demonstrate live coding on code you're already familiar with.
devoutsalsa · 34m ago
I firmly believe that the only good proxy for how well one can do the job is doing the job. Even if there are good proxies for doing the work, why would you choose to use a proxy when you can just do the work? And if you're work is some complicated that you can't break off some piece of it and do it in an interview, maybe you're making stuff too hard for no particular reason.
Let's say you need someone who can lift 10 kilograms:
- Good interview: "Please lift this 10 kilogram bucket by the handle."
- Not Good interview: "As a proxy for your overall strength, please take off your pants, squat, pick up this 1 kg bucket by clenching the handle with your butt cheeks, and then stand. We know this isn't a real test of your strength, but we want to see how you perform under pressure."
tediousgraffit1 · 5m ago
I'm curious how you would respond to the folks who are concerned that asking people to 'just do the work' in an interview are asking for unpaid labor, and that's unfair?
(post made me lol, thanks)
zdragnar · 8m ago
> when you can just do the work
Careful there- I believe a number of jurisdictions will consider using someone's work before you've hired them to be very illegal.
You could take this to the logical extreme and just not hire anyone, instead building your entire product off of the work done in interviews. Many would consider this to be a form of wage theft.
paxys · 5m ago
So what's your solution? Make a thousand candidates work for your company for free for 6 months and then hire the best one?
monkeyelite · 24m ago
Because that takes too much time. They are using proxies biased towards false negative rather than false positive to filter large numbers of candidates.
paulcole · 20m ago
> why would you choose to use a proxy when you can just do the work
"What a useful thing a pocket-map is!" I remarked.
"That's another thing we've learned from your Nation," said Mein Herr, "map-making. But we've carried it much further than you. What do you consider the largest map that would be really useful?"
"About six inches to the mile."
"Only six inches!" exclaimed Mein Herr. "We very soon got to six yards to the mile. Then we tried a hundred yards to the mile. And then came the grandest idea of all! We actually made a map of the country, on the scale of a mile to the mile!"
"Have you used it much?" I enquired.
"It has never been spread out, yet," said Mein Herr: "the farmers objected: they said it would cover the whole country, and shut out the sunlight! So we now use the country itself, as its own map, and I assure you it does nearly as well."
Very well written and so true. It's not even normal stress, which is fine, it's high stakes stress, plus sometime working under the duress of being insulted.
I once went to a job interview with Google. I built one of the first local (Global to the Netherlands) search engines of the Netherlands, but the guy in the cowboy hat at Google asked me to write a binary search with a marker and a whiteboard. I never write with my hand, I always use keyboards. Plus I'm being insulted to write a binary search when I designed and build a search index and retrieval engine.
[I did the binary search but was not happy with the whole process that did not want to even look at what you had actually done before, because that would take away the baseline they wanted].
I guess they must have been looking for cowboys. Tip for interviews, take your cowboy hat, just in case..
vidarh · 21m ago
The last time I interviewed at Google (because they approached me, and I begrudgingly let an recruiter convince me that this time would be different) the interviewer was so awful that even though the recruiter agreed and got approval to ignore the technical interview and move on to the management interview, I declined to continue the process, and subsequent calls from Google recruiters ever since has been met with a description of what happened last time and how I've permanently lost interest.
The problems posed were all "gotcha" type problem where either you'd read the solution or you'd most likely end up with a decent but suboptimal alternative, or where the recruiter asked for knowledge about toally obsolete things (e.g. I was asked about the structure of an inode in UNIX v6 - I told him I didn't know it but gave a general response about the type of information Unix-y systems keep in inodes, and with more detail about Linux; to add to it, for the role in question a knowledge of filesystem details was irrelevant).
Companies badly need to train a pool of interviewers, and track what kind of questions get asked and provide feedback on it. The vast majority of companies I see have a hiring process where some or all of the technical questions are down to the pet peeves of the interviewer or their manager.
snarf21 · 34m ago
Unfortunately, this worked as intended. These companies want people who are desperate to work there and will do anything to get in the door. Both parties were successful in this case in determining that there wasn't a good fit.
hcfman · 8m ago
That's also true. I actually was glad that I didn't get the job, because at that stage of Google development they were definately underpaying people who wanted "To work for Google". At least in Europe. It appears that the USA is very different in this respect.
Eridrus · 35m ago
If you feel insulted being asked to write a binary search that sounds like you have a bit too much ego at work.
hcfman · 9m ago
I had a little ego for sure. I think under the circumstances that is not too much. Anyone who puts in really a lot of work to improve their knowledge has a little ego. Calling that "a bit much ego" in this context is harsh. I am indeed someone that doesn't like being treated like shit.
Sorry man, hats off to you if you are that humble. For myself, I don't want to be that guy though.
anonzzzies · 7m ago
Ah live coding for me is making animations and music typing in code (preferably in lisp); live coding interviews sounded like fun from that perspective. But from what the article is about, I would refuse that. I have done (refused) once when they said it's part of it all, got an offer anyway.
gwbas1c · 16m ago
I've started to treat live coding screening sessions as a "conversation about code." I make sure that the candidate knows that I'm trying to make sure we can discuss code.
Why?
Purely being a "good programmer" isn't just enough for the job. It's important that we can develop a good working rapport, and communicate.
For example: Someone who's an expert programmer might misinterpret a discussion and build the wrong thing; or someone might be a great programmer but otherwise be impossible to actually conduct a technical discussion.
IE, this is why "Fizzbuz"-style questions work so well: It's not really screening that you're a great programmer, it's screening your ability to have a technical discussion.
Paul-Craft · 8m ago
I think that's a good approach. I can write FizzBuzz in 4 different programming styles, in at least 3 different programming languages, and I'd be happy to talk about all 12 permutations of them.
apwheele · 42m ago
I do simple questions similar to the quoted LinkedIn post. (So less leetcode and more "do you know how to code anything".) While I can agree it is harder under stress, what is the alternative to knowing if someone can code at all? (My pass rate is similar to the quoted LinkedIn post.)
I have done the entry interview as well. For example have had very good conversations with managers who were applying for senior IC roles. They then go and fail the tech interview basic "can you do write and execute a python hello world script from the command line".
acatton · 27m ago
Yeah, I'm confused by the article. Following this logic, any interview sucks because of the stress it puts on the candidate. So what am I supposed to do? Hire based on a home assignment? then the "unpaid labour" crowd will call me out, and I personally believe they would be right to do so... So I'm supposed to hire based on résumé only? It's a lose-lose situation.
It reminds me a professor who recently told me, regarding ChatGPT use in university: "we're receiving every week applications from foreign students written in perfect German, then when we schedule a call for an interview with the potential scholar, they're incapable to speak either German or English."
gmm1990 · 22m ago
Would this be like entering the python terminal and typing print('hello world') or python hello_world.py that has the print instruction? Or something else. I'd just be unsure if a python installation like the python.exe would be available in a terminal.
I'm more curious than anything else for my own sake to know things people might ask. But its interesting how extremely simple things can be complicated if you haven't done them before. Like if someone asks about a relatively simple regex example in python it'd be easy to get if you just were working on a regex but you could get tripped up if it had been a while since working on one. You could say the same thing about working with datetimes. At least this is the type of thing that throws me off in an interview, maybe I'm not a great candidate though.
apwheele · 14m ago
I expect `python hello_world.py`, but if people are confused I just nudge them to what I expect. It is not meant to be a trick question.
If people do not have local setup, I just have them write out in text editor and walk through the steps. Maybe not 75% fail rate, but more like 50% of people fail this step in the tech round.
hibikir · 36m ago
Absolutely. Some of my favorite coworkers would never pass a modern, top company interview which I can pass, and not because they are worse in general, but due to stress management. When I am working in one of those that has no hiring shortcuts, I just can't recommend them, because they will fail. In a small enough company, I can say "hire this person sight unseen, and if they aren't good, it's 100% on me", and then they are successful. My time working with them is far more valuable at skill matching than any interview.... but that's not how we do things in most places, because the fear that people will recommend the otherwise incompetent is high, and for good reason. There would have to be consequences for recommending a bad candidate like that.
Now, that doesn't mean that stress management cannot be part of the job: I've worked in the hellish places where an on-call rotation meant at least 4 calls off-hours, and some which needed resolutions in 10 minutes or less from the pagerduty call. Someone with bad stress response is not going to be doing great at that kind of live diagnosing, with possibly many millions for the company on the line. But the number of positions like that, where you are a cross between a developer and an ER doctor are much less common than places that have none of those demands, yet give you a series of leetcode mediums and hards. They might as well be testing for height, or how much you can bench press. It filters, but it does not help.
DanielVZ · 35m ago
I used to do tech screening in my previous job and not only it measured stress, it was awfully biased against older people. We lost so many senior people that I really wished I could work with because they weren’t used to jumping through the hoops and loops of leet code questions (in my country leetcoding is fairly new). Then there were other younger candidates that were pretty mid but knew all the answers to leet code and design questions and ended up getting the job.
guhcampos · 7m ago
Exactly. Leetcode only measures how much leetcode the candidate has been practicing. Nothing else.
michaelt · 25m ago
> When you’re placed in a high-stakes, time-pressured situation, like live coding, your brain reacts exactly like it would to any other threat. The amygdala gets activated. Cortisol levels spike. [...] You forget what you just typed a few seconds ago. It feels like your IQ dropped by 30 points. In fact, it feels like you’re a completely different version of yourself; a much dumber one.
Did y'all not have to take a load of exams in high school and college?
I'm sure there are variations between education systems, but by the time I graduated college I'd done at least a hundred high-stakes, time-pressured hour long written tests. All of them substantially more difficult than summing the even entries in a list.
Is this not what everyone experiences, in every education system?
radiofreeeuropa · 12m ago
Not while performing in front of an audience, no, very, very little of that.
You know the trope of almost all students dreading going up in front of the class to solve a problem on the blackboard/whiteboard? A thing that I think I personally only actually did a countable-on-fingers number of times ever, and only in lower grades?
It's a trope for a reason, and it's because the vast majority of people do in fact hate having to get in front of an audience and perform. Hell, it's often a component of recurring nightmares.
It's like that but way worse because it's also far more high-stakes than that, the problems are far more complex, the point is explicitly for everyone in the audience to judge you and not just your assumption that they all are causing the stress, et c.
It's more like the stress from getting up at an open mic night, than any kind of stress I've ever actually encountered on the job. Even the dynamic in a meeting with clients where you have to diagram something out on a whiteboard or something is totally different—for one thing, you generally have people in there who are very much "on your side", and you don't have to worry if you misspell something that you're personally about to be/remain unemployed, or lose out on a huge raise, or whatever (nobody generally cares about minor mistakes on a whiteboard in an ordinary context, and maybe they don't in some interviews, but they do in others, and candidates worry they might in all of them).
guhcampos · 11m ago
I can't really explain why, but to me they don't feel the same at all.
I've always finished exams absurdly quickly. I used to finish 60 min exams in like 20 min and sleep the rest of the time when the professor didn't allow us to leave, because I had likely spent the previous night drinking and partying my way through college. I'd usually ace most of them.
Thing is, at those times we were working with freshly acquired knowledge that you've been practicing a lot. That's not the case with most leetcode interviews. As a senior/staff engineer, I'm not using double pointers to perform little math tricks on a daily basis, I'm troubleshooting cascading timeouts on distributed systems. I'm not worried about how fast I'm going to iterate over a batch of a couple thousand records on a list I'm worried about how much latency I'll accrue if I rely on a primary source of truth instead of hitting a cached answer.
Code interviews don't measure experience or competence. I don't even think they measure stress as the article mentions. To me, they just measure how much leetcode you practiced for that interview. Nothing more.
Palomides · 14m ago
none of my school tests were one on one, completely open ended, covering unpredictable subject matter, or were the only thing between me and $100k+
vidarh · 16m ago
Very rarely with people watching.
I've had only two oral exams, and they were awful, but even then they were far more conversational in nature, as opposed to a coding interview.
I do well on written tests.
I also do well in interviews, but I detest having people look over my shoulder while I code, and it absolutely tanks my performance. I only pass them because I'm experienced enough that I can be stressed out like crazy and perform really horribly compared to what I normally do and still do okay.
I've seen fantastic coders totally freeze up and be unable to do anything at all when being asked to write code in front of people, even though they have no problem talking through problem solving.
And I'm similar to that. I've held speeches in front of thousands, been on TV, held plenty of presentations, and can talk the ears of an interviewer with passion about complicated technical concepts, but have me write code while you watch, and I'd frankly much rather have a root canal treatment (I've been known to fall asleep during those).
radiofreeeuropa · 7m ago
> I do well in interviews, but I detest having people look over my shoulder while I code, and it absolutely tanks my performance.
Yeah, I basically forget how to use all the basic tools I use all the time and my mistake rate shoots up 10x when I'm just screen sharing in a chill context. I'm (told I'm) quite good at all the social side of the job, at keeping my cool in rough situations (social or technical), and all that, but specifically being watched while I work turns me into a sort of foolish klutz.
Which makes these kinds of interviews absolute hell, because that's their entire deal.
xorcist · 11m ago
Interviewing is Hard(tm). I accept all the problem with live coding the article mentions but I still do it, at least a little bit.
What's the alternative? Take home problems come with their own problems, and select for people with no family and lots of time. Going by general vibe alone also works, kind of. But we're going to work together, so having something that at least faintly resembles work is kind of useful.
The leetcode type problems are useless however, fully agree on that. But a trivial loop with conditional, perhaps iterating over a list of lines shouldn't be that bad, especially if you get help with the syntax.
Olshansky · 11m ago
Sharing my quick personal anecdote here.
- Past: ~8 years in big tech as an IC
- Currently: ~4 as CTO a < 10 person startup.
I always hated doing and conducting coding interviews. When I became in charge of the interview process, I swore to myself not to have them.
I had to let go of a person after ~3 weeks in because he just didn't write any code. This was about 1 year post the ChatGPT moment.
He had lots of experience. Great communicator. Culture fit (or so I thought). Etc...
Interviews can provide a signal, but nothing trumps a month of working with someone. If both parties are open to it, a well paid short term contracting gig goes a long way.
andy99 · 24m ago
The example question is sum all the even numbers in a list.
This is something that someone familiar with a language should be able to do as naturally as answering "tell me about yourself" or some other non-coding question. (Personally, way more naturally, nothing makes me more uncomfortable than "tell me about yourself")
I agree that for something more involved, nerves can become more of a factor, certainly any "trick" question, but these seem more like just checking if whatever language is second nature, which it should be, not about algorithms etc.
appease7727 · 8m ago
I give coding tests on real(ish) problems adjacent to the job. It's one half showing me what you can do, and one half seeing how you deal with a real work environment, collaborating with others, unknown requirements.
It's not a puzzle, it's not a trick question. It's simply a matter of 1) can you do the job and 2) do you get along with the team.
The last interview I did, we had the candidate build a game of marbles in Unity. None of us knew the rules, so it turned into a collaborative process between the candidate and our engineers. We learned that the candidate can think on his feet and fluently adapt his program to requirements as they manifest. Plus it was just a lot of fun. He was one of our best hires, too.
9dev · 20m ago
From the perspective of someone currently searching for candidates: I decided to give applicants a homework that's supposed to take about two hours to do, and schedule the next interview a week later to discuss their solution (remotely).
For example, for the senior frontend engineer, I think we went with "create a real-time comments panel using SSE that works in multiple browser tabs", and created a ready-made git repository with a dev server that includes the SSE endpoint. That way, they only had to focus on the actual task.
Now I thought that was nice because it should be very easy to solve for someone with a bit experience, but we get lots of opportunities to discuss their choices and the technology. It's hard to confidently bullshit an answer to if you think server-sent events or web sockets are superior, and why. So from everything we tried so far, this format gave me the most accurate impression of someone yet.
gchamonlive · 24m ago
It has to be really clear what's the purpose of the interview. You want to measure how technically competent someone is, but in practice how competent someone is in any organisation has more to do with internal communication and institutional knowledge than technical skills. That means without technical skills it's certainly impossible to be productive, but technical skills by themselves don't mean guaranteed success, not even to lower the uncertainty of failure.
Which is funnily something opensource doesn't suffer from. You just have to prove identity to the commits you signed in previous contributions, so the problem of proving technical skills in the opensource community is replaced by the effort of building trust.
The answer therefore could be to replace technical interviews by sending out assignments to candidates to have them contribute to opensource and come back with results. This has the upside for candidates that if multiple companies do such assignments they only have to contribute once to be eligible for multiple hiring processes.
Everybody wins.
Paul-Craft · 51m ago
> We can’t change the fact that live coding is a common practice in tech interviews. But we can try to mitigate the stress it causes.
Yes, we can. Don't do them. But, we have to replace them with something that works. That means none of these poorly constructed take home projects that are almost universally either drastically over scoped, criminally under specified, or both.
smugglerFlynn · 13m ago
> we have to replace them with something that works
We don't. The simple solution is to stop maintaining the illusion that 100% perfect hire is possible.
Design your post-hire process around imperfect hiring rate / quick feedback loop, and accept the losses (they will happen anyway despite any perfectionistic illusions you choose to maintain).
These are few questions that really matter:
- Is there a track record of delivering meaningful results?
- Does their past experience seem credible?
- Are their soft skills and communication skills up to your expectations?
- Do they demonstrate adequate hard skills for the role?
- What interests and motivates them on professional and personal level?
Your interview process will always be just an attempt at answering these somewhat accurately, with diminishing returns after a certain point. Getting actual accurate answer to these is only possible through collaborative work in real environment.
Paul-Craft · 1m ago
I'm gonna stop you right here, because I never said any such thing:
> We don't. The simple solution is to stop maintaining the illusion that 100% perfect hire is possible.
vidarh · 9m ago
I was about to suggest the problem with that is that applicants may think they meet your standards, and then be fired, but then realised that of course that very few coding interviews measure their skills to a sufficient standard to prevent that anyway.
StableAlkyne · 45m ago
Take-homes suck on both sides nowadays - if you're the interviewer, how do you know they didn't just plug it into chatGPT and spend the rest of the afternoon studying the solution?
I just don't know what a better proxy of coding ability even is, when we exclude options that can't be gamed/cheated.
jghn · 28m ago
> how do you know they didn't just plug it into chatGPT
If someone can present a good solution and talk about the reasoning behind it with enough detail & savvy to convince you that they actually wrote it, does it matter if they didn't?
sintezcs · 41m ago
What’s wrong with it if they studied the codeGPT solution well enough to explain it, answer the questions about corner cases and suggest improvements? Won’t it be a good indicator of the candidates skills? ChatGPT is one of the daily tools nowadays, we should not ban it, but the one using it should be able to understand and explain the code, and his logic, and explain how he architected the solution and how the LlM assisted him, and where it worked well and where not so good.
ixsploit · 34m ago
Probably becasue you are looking for people who can actually perform a certain job and not just come back with the ChatGPT answer.
Paul-Craft · 27m ago
If they can produce working code that solves the problem, and explain how it works, that is more than “just com[ing] back with the ChatGPT answer.” I'm not saying ChatGPT doesn't have its own issues, but this is not one of them.
ghaff · 37m ago
Well, and as a candidate, while you're going to do some homework on the company in any case--any real take-home you know you're competing against people who are going to take a weekend or more with it whatever the instructions are.
cyanydeez · 47m ago
Seems like now adays theyd just hook you up to whatever AI they claim to use and your job is to tell them why the AIs code would fail real world tests.
The catch would be not knowing whether the interviewee has the AI cargo cult PRIORS
_jab · 24m ago
Sometimes when I give simpler technical questions (applies to system design too), candidates begin to massively overthink it. The question seems so simple that they start anticipating high-scope extensions that don't actually exist, like turning a basic algorithm into a distributed, high-availability pipeline. There's only so much you can do as an interviewer to rein candidates back in at that point, and those interviews tend to go off the rails pretty quickly, with little code actually ending up being written.
Thing is, I reckon those candidates aren't a good fit anyways. The biggest mistake I see engineers who join startups make is that they pursue excessively sophisticated solutions, with robustness that does not justify the added complexity. I'm sure these candidates are smart, but they're not good fits for the high-velocity, simplicity-obsessed technical environment of a product-play startup.
danesparza · 44m ago
Having been on the other side of the table ...
Live coding interviews measure how the brain works under stress -- you are absolutely correct. So do other interview questions. Interviews suck in general for this reason.
I won't use live coding questions (usually) because I don't see much value in trying to code under that much stress.
But I will ask hard questions to see how well a candidate communicates in a stressful situation (which I think is a far more valuable indicator of their brain's "default" mode in stress). I want a candidate that talks honestly about stuff -- especially the stuff they don't understand -- even in a stressful situation. Sometimes stress can bring out the asshole in somebody -- and that's an immediate red flag.
btown · 31m ago
I like to think about it as the tactical allocation of stress. As an interviewer, I do not want the interviewee to be stressed about optics, or about whether they'll "look bad" about needing documentation; I'll encourage documentation/LLM use as long as they do it on the screen, and disarm by giving context about why that doesn't matter.
But I do want to see how they cope with the stress of needing to track and articulate complex control flow and edge cases through their program in a timed environment - that's part of every day if you're building ambitious systems, and it's a skillset that continues to be relevant even in an era of agentic coding. Testing for the response to that stress makes coding interviews highly relevant in my experience.
darkwater · 2m ago
> But I do want to see how they cope with the stress of needing to track and articulate complex control flow and edge cases through their program in a timed environment - that's part of every day if you're building ambitious systems
Do you really need to code day in day out in an time window of the size of an interview (so 1 hour or 2) in your workplace? Or are you testing for extreme situation like patching a failing live production system?
padolsey · 35m ago
> But I will ask hard questions to see how well a candidate communicates in a stressful situation
Well, stress fires off old cortex fight-or-flight response. It's like the worst possible test of neocortex capability. Why are you trying to trigger it? Kinda cruel.
tayo42 · 1m ago
These posts are acting like they're looking for a personality that can handle being a combat jet pilot when they write crud that handles 100 requests/day...
Why is the business this stressful in the first place
nabla9 · 32m ago
A good employer takes care of their worker even if they have problems, but they also want to weed out as much of them before hiring.
Mentally strong and healthy people who do well when challenged are good hires.
Paul-Craft · 5m ago
> Mentally strong and healthy people who do well when challenged are good hires.
So, you discriminate against disabled people?
ayaros · 28m ago
I just had an interview like this and I don't think I did well. It was a dumb stupid coding problem that, naturally, should have been easy, and it took me forever. I honestly don't know if I'll hear back from them at this point. :/
At least reading this post has helped me regain some of that lost self-esteem. Thanks for sharing it. <3
voidUpdate · 26m ago
What if companies, idk, looked at the previous work of a potential employee? their github, their portfolio, anything. That way they can see what kinds of work someone can do when they aren't under stress, and they can ask questions about it in the interview, to see their thought process etc
guhcampos · 8m ago
Anyone that has been working on corporate jobs for a good lot of years won't have a portfolio or a busy github.
I've been coding for 20 years. Almost all this work is in some private repository behind multiple levels of locks protected by a stack of NDAs and Trade Secret agreements.
Not everyone is a hobbyist.
asciimov · 4m ago
That’s both easy to game and will filter out candidates that don’t have work to show.
vidarh · 13m ago
I've been told I didn't need to do the coding test stages on multiple occasions because of my Github. Some absolutely do. I do it myself too when hiring.
acatton · 22m ago
I would create an unwarranted bias towards people working for companies doing mostly opensource. If their previous job was writing safe code for rockets and airplanes, the candidate will very likely be qualified to write embedded code for tractor/cars but incapable of showing previous work due to confidentiality agreements.
voidUpdate · 19m ago
I mean, I was hired out of university, and I had a portfolio of programming and a github ready to go to show that I could program, without having a previous job to show code from at all, let along one with an NDA
acatton · 3m ago
You were young without kids or any other responsibilities, so you had spare time to nerd around. Not everybody is in that position. With this requirement you would create a bias towards single parents or folks talking care of the handicapped partner/parent, as well as another bunch of other categories of people.
squeaky-clean · 5m ago
Do you have an updated portfolio though? For many people working in private industry, they aren't allowed to share code from their job, and their previous portfolio projects are from college, which would not be good enough for a mid or senior level role. Would you hire a (non-junior) frontend developer who shows you a React todo list they made 8 years ago?
gwbas1c · 20m ago
That's very time consuming and often isn't practical, especially when hiring needs to start with a very wide net.
vidarh · 13m ago
You don't need to do it for everyone. Do a cursory pass if people are in the "maybe" pile. Do a deeper pass if you decide to interview them.
voidUpdate · 18m ago
Do most companies do a live coding exercise with every single applicant? I only got one, with the company I work at
gwbas1c · 2m ago
It varies from company to company, but it is the "industry norm."
What happens is that people whine about live coding exercises; then someone decides to skip them, and immediately makes a bad hire. The lesson is then learned and they resume live coding exercises.
To turn the tables: As a candidate, If there isn't a live coding exercise, it's a red flag and tells me the company isn't savvy. (And I've taken a bad job as a result and now won't take a job unless I have a live coding exercise with someone who I will be working with.)
lvl155 · 32m ago
I’d fail all those coding/leet tests at this very moment especially since I fully integrated Claude Code into my workflow.
benrutter · 49m ago
Live coding does suck - worst off, even if you ignore that it biases to stress tolerance, it tests "leet code" skills for things like reversing a list. Most actual development work involves things like, design work, managing a large codebase, debugging, reasoning through systems, etc.
That said, does anyone have a good alternative? If somebody has open-source work and portfolios, there's a great window into their work, but that's probably an unfair expectation.
Peritract · 44m ago
I've had success with a short take-home task and then an interview asking them to talk through the code/potentially modify it/discuss the approach.
They get time to prepare and think about the problem. Because it's a familiar context, you can ask for more real-world alterations, discuss deployment meaningfully, etc.
pneumic · 26m ago
I am lucky to have never had a live coding interview, because I would utterly crumple. Not proud to say and something I should work on, but it’s very real.
jplrssn · 10m ago
Are you a software engineer? I'm curious about what your interviews were like.
I've done a fair amount of interviews and they all featured some amount of coding in front of an audience.
mromanuk · 50m ago
Probably another case of trying to measure something difficult, and people usually substitute that problem for an easier or more accessible one. Checking if a person can work under pressure and sensing their emotions and ability to deliver is easier to assess. This pattern comes from Thinking, Fast and Slow.
padolsey · 38m ago
> Some companies genuinely care about this. Some even mention it in their job descriptions. They want candidates who perform well under pressure.
Who are these self-important employers?? I mean, outside of Ops and live incident response, are there really that many software engineer roles that require someone to perform well under pressure? It seems a false and egotistic narrative. If you need a software builder to perform under pressure then you've got systemic issues of a type not solved with software. You need better management or better work practices.
comprev · 31m ago
The stress in the Ops world is getting systems back to normal operation as quickly as possible because of lost revenue.
In the Dev world stress comes from the demand to add new features in unrealistic timeframes. The "product" has already been sold even if it's not finished yet. The business risks losing revenue by not delivering promised functionality.
Both Dev and Ops people experience equal stress from the same employer due to different reasons.
KingOfCoders · 34m ago
a.) yes b.) depends on where you're working, but in the startups I worked in, with incidents and critical bugs and being pressured permanently by product managers, even shouted at by the CEO, being stress resistant was part of the job.
Should it be that way? No. Was it that way? Yes.
djtango · 52m ago
I've seen people give weak in person coding interviews then done perfectly fine on a take home and went on to be fine on the job.
To date, I've never gone on to regret hiring someone who blitzed the in person coding exercise.
phreeza · 50m ago
What do you mean by blitz here? Do well or do poorly?
peeters · 45m ago
Yeah and I think that's the core of the issue here. In a lot of hiring markets, the cost of letting in a bad hire is higher than the cost of filtering out a good hire.
threetonesun · 32m ago
I've read this a million times but worked at companies where the bar for hiring managers and leadership was shaking someone's hand the right way, which led to entire teams either running for the doors or actively led to ruin.
There's a separate problem here for hiring developers specifically, where realistically it should be easy to bring someone on limited term (say 3 months) and see how they work, but compensation and benefits in the US absolutely in no way support a system like that.
philwelch · 31m ago
In my own career, managing stress has been a much greater challenge than the technical skills, so if anything this vindicates live coding interviews.
VikingMiner · 26m ago
Some of this I swear is to see how you act "stressed".
2 years ago given a coding assessment "Roman Numeral Conversion". Was given a set of test cases. I got near the solution. I think if I had another 5-10 minutes I would have solved it fine. Sat down that evening with the same test cases and did it from scratch in 20 minutes.
5 years ago, I didn't get a job because I while I did well technically. The reason for the rejection was that I appeared "stressed" while being assessed. What do they expect?
10 years ago. I walked out of an interview essentially after about 10 minutes. The interviewer(s) interrupted me the moment I started writing any code, constantly interrupting my train of thought. I think it was intentionally done to irritate me. I've had companies play these stupid games in interviews before.
anarticle · 13m ago
The smugness of my interviewer during live coding tests has caused me to stop immediately and says “thanks for the opportunity”. Either ageism, or just being a jerk for some reason. Guy was breaking my balls over tree searches, which I have written from scratch.
I didn’t need the job that bad.
I’ve worked on one man projects that were hard realtime C image analysis (Fourier, calc, stats), large data science eval projects, devops stuff. CS degree and used it for published research.
I also had a five rounds of interviews a MANGA corp. Didn’t bother with the second because I didn’t have two months to waste.
Went hired gun and never looked back. Homey network uber alles, I make 10% less but I call all the shots, and am way healthier.
It will probably get worse before it gets better out there, between RTO and whatever LinkedIn is doing to CEOs.
x3n0ph3n3 · 39m ago
I must be in the minority, but I enjoy live coding exercises. I also enjoy Advent of Code and though I never make the leaderboard, its fun to challenge myself on time.
add-sub-mul-div · 40m ago
Once in an interview I asked someone to go to the whiteboard for a coding exercise and her body language showed an enthusiasm and fearlessness that in hindsight I realized I'd never seen before. She practically sprung up out of the chair.
Most people, even the good ones, show a little hesitance when starting. Which isn't really necessary, most people do fine. I'm not trying to get them to fail, I'm trying to get them to succeed. I want to see if they're smart and understand the problem and the direction of a solution. Not if they miss any semicolons or don't recall some arcane data structure.
She was one of the best hires I made. Coding interviews can also measure attitude and confidence.
corysama · 23m ago
Once in a coding interview, someone asked the candidate to go to the whiteboard and the poor guy was so flustered he couldn’t remember how many bits were in a byte.
He was a perfectly intelligent programmer and a fine person. I have no doubt at all he understood the details of bits and bytes. But, that group interview session did not sufficiently manage the stress level of the process. And, so we probably missed out on a perfectly good hire.
That and similar experiences in other group interviews are why my 1:1 interviews are structured around keeping the stress level low and preventing the candidate from freezing up.
paulcole · 22m ago
Dang that stinks because being able to do something while stressed isn’t a useful job skill at all.
spacecadet · 40m ago
I have serious problems with the concept of working under stress... or duress is more like it. Unless you are in a first responder, ER doc/nurse, or in a warzone. The job should not be stressful. Companies that operate this way and overlook the abuse it causes their employees, are broke, fucked, evil, etc.
micromacrofoot · 25m ago
most of the time if the job is stressful management is doing something wrong
charcircuit · 10m ago
>Being bad at live coding doesn’t mean you’re a bad engineer. It means you’re human.
Except humans can do well at live coding. Especially for trivial cases like the start of the article (sum . filter even). Yes, companies are looking to hire humans as opposed to ChatGPT, but they are looking to hire a subset of humans. Just being human is not good enough. Failing this test might not provide a reliable signal, but passing it does. If you are getting stressed over simple question or existing stress is impairing your thinking enough to not answer simple questions that seems like an issue to me.
blitzar · 55m ago
Live interviews measure stress, not skills.
nabla9 · 35m ago
Measuring stress tolerance is an important when hiring.
A good employer takes care of their worker even if they have problems, but they also want to weed out as much of them before hiring. The value of worker is not revealed when they are at their best. Fragile workers with anxiety perform well only small amount of time.
Live interviews are generally excellent at revealing how individuals perform when placed outside their comfort zone. If you perform well in a live interview, be it for coding or any other skill, you are likely to perform even better reality.
herval · 31m ago
Live interviews are generally excellent at revealing that someone is comfortable with live coding interviews. I know plenty of people who excel at them and are unable to code on the job - and vice-versa.
I'm pretty sure there's no correlation whatsoever, after interviewing more than 500 people in a decade...
The correlation that I think _does_ exist is for _junior engineers_: Live coding interviews (particularly if you take typical CS problems) measures how much they're able to memorize and how much attention they paid in class (or in preparation). Which used to be a necessary requirement back in 2001, since it directly correlated to being able to learn by memorization. Unsure that's still a relevant skill since Google > Stack Overflow > ChatGPT, however...
Paul-Craft · 31m ago
> Live interviews are generally excellent at revealing how individuals perform when placed outside their comfort zone.
I suppose if by "outside their comfort zone," you mean in a potentially literal life or death situation (because, face it, most people can't live long without a job in the US), under an arbitrary and very short deadline, possibly using unfamiliar tools, and simultaneously having to explain every little thing they do, then yes, I would agree with this. I don't believe the sentence after this follows logically from it, though.
nabla9 · 27m ago
No. More like the ability to be around new people, have their work evaluated, getting frank feedback, being honest about their mistakes, defending their positions.
If interview feels like life and death, maybe you are not the right hire.
Paul-Craft · 24m ago
Interviews are not valid tests of those things due to the stress involved.
If you don't understand how not having a job in a place like the US is life or death, you're not the right interviewer.
I’ve worked in the assessment space for 6 years and have seen many hiring processes, from Fortune 10 companies to startups hiring their first engineers. The range of signal required, and how much candidate experience you can get away with, is huge. I’ve also been a candidate myself and failed many traditional live coding interviews. It made me feel terrible about myself.
When I work on my product, I try to view it through the lens of empowering candidates to show their skills and potential. I encourage our customers to use assessments that somewhat resemble on-the-job skills. I don’t like the phrasing “real work” anymore. An assessment shouldn’t be unpaid labor, it should be a way for candidates to demonstrate that they can do the job and handle future work thrown at them, and for hiring managers to feel confident extending what are often very high salaries in tech.
With AI, the landscape is shifting. Unfortunately, short take-homes (what I prefer as a candidate, using my own tools and editor) are becoming harder to maintain as a fair signal due to AI assistance. I’ve seen companies move back to onsite, and competitors deploy all kinds of proctoring and invasive monitoring.
The perfect solution, in my view, would be an assessment where the candidate feels relaxed and able to perform at their best, knowing that every other candidate in the pool has the same constraints in terms of time and tooling. It’s a tough problem to solve.
They work (for some) and leet-code weeds out the frauds that really cannot problem solve and assesses those who have not built anything to show to justify not doing it and can be applied to companies that are joining from an acquisition.
> The perfect solution, in my view, would be an assessment where the candidate feels relaxed and able to perform at their best, knowing that every other candidate in the pool has the same constraints in terms of time and tooling. It’s a tough problem to solve.
And that would be the fairest one which is to do the leetcode interview in person on a projected whiteboard and pair programming with interviewer.
Very relaxing.
People actually find being fired a greater inconvenience and humiliation than a 60 minute whiteboard interview.
It depends. A welder looking for a job -- even with a certification -- will often have to demonstrate his welding skill to the jobsite supervisor before he is hired. An airline pilot -- even with a rank of "captain" with 5000+ hours -- when trying to find employment with another airline, will still have to demonstrate piloting skills with a "check ride" as part of hiring approval process. Sometimes, experienced pilots fail the check ride for whatever reason.
> A certificate confirming he did learn the job
What would that certificate be?
You cannot start your reasoning with "it depends" and then continue with an absolute. I could do the same:
As with everything, it depends. Live coding interviews don't work.
What's the difference?
I was interested by the author's statement: "Working memory is the most reliable proxy (I know of) for fluid intelligence, your ability to reason, solve novel problems, and think abstractly." and the linked study (https://pubmed.ncbi.nlm.nih.gov/21037165/). My working memory is not so great, but it degrades less under stress.
Question worth considering for hiring managers: do you prefer stress-capable employees, or greater working-memory employees? Is my model a false dichotomy?
Yes, there's a large cohort of "senior" software engineers who can't actually code. They bullshit their way into jobs until they're fired and then apply for the next one. These are the people you want to filter out with live coding.
But also, someone can fail a live coding interview for reasons other than belonging to that group.
There's an assumption that the company's existing senior architects and developers will stop a new person from making the code worse, but also devs at every company thinks their codebase is terrible so it obviously isn't working.
I think we can leave companies who don't care about quality out of the discussion, but for those who do, the time to detect those developers is in a probational period, which is not something that most companies really use on their favor.
The problem is this requires a good management that is able to actively paying attention to the work of the developer. Which is often not in place, even in companies who want to prioritize quality :/
Doesn't even have to be AI. Give me some random file from the Linux kernel and I could probably explain everything to you if you gave me a few hours. But that doesn't mean I would ever be able to write that code.
Sometimes you spend the whole time trying to figure out how to solve the puzzle that don't even have time to show that you can - actually - code.
Like I said the deadlines work for both sides. If a company wants to give homework instead of having their own senior engineers spend time talking to me, that tells me what I need to know about how they value my time.
That's not equivalent to what I said, nor is it live coding.
Again, those deadlines are artificially short compared to real world scenarios, and completely arbitrary. They are so short, in fact, that they render the interview an invalid test of real working ability. A work sample has been proven time and again to be the most valid measure of whether a candidate can actually perform the job, but the conditions under which a live coded "work sample" is performed in an interview render it invalid.
The legal environment is what it is and unlikely to change. The social environment is fickle and trend driven. Workers can't always fully evaluate their odds of success or the entirety of risk of leaving a job that's valuable for the employee and employer for one that might end up as a poor match, even if both sides have been transparent and honest. It's a difficult matchmaking problem with lots of external factors imposed and short term thinking on all sides.
Ideally young workers would have an early career period that involves a small number of short lived jobs, followed up by a good match that lasts decades, providing value to both the employee and employer. Much like finding a spouse used to be a period of dating followed by making a choice and sticking with it so a life could be built together, employment ideally should result in both sides making the other better. Today however everyone seems focused on maximizing an assortment of short term gains in search for the best local timescale deal at the expense of the long term. It's interesting how the broken job market and broken family formation process in the western world mirror each other so much.
1. Make sure you pick every good candidate, but some bad candidates will slip through as well.
2. Make sure you reject every bad candidate, but some good candidates will fail as well.
Candidates want process #1, but companies have no reason to push for it. The cost of accidentally hiring a bad employee is simply too high, way more than rejecting a good employee. The current system in place prioritizes #2. Yes they are rejecting great candidates, and they are aware of it.
a related thing here is that I've long believed that remote work positively impacted my career for precisely this reason.
There was one company whose process I really appreciated:
- A take-home test, more of a puzzle-style challenge. - The interviewer does a quick review of your submission. - If they like what they see, you're invited to a follow-up session where you explain your code and make a few simple changes to it.
This approach proves that the work is genuinely yours, shows your thought process, and still gives the opportunity to demonstrate live coding on code you're already familiar with.
Let's say you need someone who can lift 10 kilograms:
- Good interview: "Please lift this 10 kilogram bucket by the handle."
- Not Good interview: "As a proxy for your overall strength, please take off your pants, squat, pick up this 1 kg bucket by clenching the handle with your butt cheeks, and then stand. We know this isn't a real test of your strength, but we want to see how you perform under pressure."
(post made me lol, thanks)
Careful there- I believe a number of jurisdictions will consider using someone's work before you've hired them to be very illegal.
You could take this to the logical extreme and just not hire anyone, instead building your entire product off of the work done in interviews. Many would consider this to be a form of wage theft.
"What a useful thing a pocket-map is!" I remarked.
"That's another thing we've learned from your Nation," said Mein Herr, "map-making. But we've carried it much further than you. What do you consider the largest map that would be really useful?"
"About six inches to the mile."
"Only six inches!" exclaimed Mein Herr. "We very soon got to six yards to the mile. Then we tried a hundred yards to the mile. And then came the grandest idea of all! We actually made a map of the country, on the scale of a mile to the mile!"
"Have you used it much?" I enquired.
"It has never been spread out, yet," said Mein Herr: "the farmers objected: they said it would cover the whole country, and shut out the sunlight! So we now use the country itself, as its own map, and I assure you it does nearly as well."
https://en.wikipedia.org/wiki/On_Exactitude_in_Science
I once went to a job interview with Google. I built one of the first local (Global to the Netherlands) search engines of the Netherlands, but the guy in the cowboy hat at Google asked me to write a binary search with a marker and a whiteboard. I never write with my hand, I always use keyboards. Plus I'm being insulted to write a binary search when I designed and build a search index and retrieval engine.
[I did the binary search but was not happy with the whole process that did not want to even look at what you had actually done before, because that would take away the baseline they wanted].
I guess they must have been looking for cowboys. Tip for interviews, take your cowboy hat, just in case..
The problems posed were all "gotcha" type problem where either you'd read the solution or you'd most likely end up with a decent but suboptimal alternative, or where the recruiter asked for knowledge about toally obsolete things (e.g. I was asked about the structure of an inode in UNIX v6 - I told him I didn't know it but gave a general response about the type of information Unix-y systems keep in inodes, and with more detail about Linux; to add to it, for the role in question a knowledge of filesystem details was irrelevant).
Companies badly need to train a pool of interviewers, and track what kind of questions get asked and provide feedback on it. The vast majority of companies I see have a hiring process where some or all of the technical questions are down to the pet peeves of the interviewer or their manager.
Sorry man, hats off to you if you are that humble. For myself, I don't want to be that guy though.
Why?
Purely being a "good programmer" isn't just enough for the job. It's important that we can develop a good working rapport, and communicate.
For example: Someone who's an expert programmer might misinterpret a discussion and build the wrong thing; or someone might be a great programmer but otherwise be impossible to actually conduct a technical discussion.
IE, this is why "Fizzbuz"-style questions work so well: It's not really screening that you're a great programmer, it's screening your ability to have a technical discussion.
I have done the entry interview as well. For example have had very good conversations with managers who were applying for senior IC roles. They then go and fail the tech interview basic "can you do write and execute a python hello world script from the command line".
It reminds me a professor who recently told me, regarding ChatGPT use in university: "we're receiving every week applications from foreign students written in perfect German, then when we schedule a call for an interview with the potential scholar, they're incapable to speak either German or English."
I'm more curious than anything else for my own sake to know things people might ask. But its interesting how extremely simple things can be complicated if you haven't done them before. Like if someone asks about a relatively simple regex example in python it'd be easy to get if you just were working on a regex but you could get tripped up if it had been a while since working on one. You could say the same thing about working with datetimes. At least this is the type of thing that throws me off in an interview, maybe I'm not a great candidate though.
If people do not have local setup, I just have them write out in text editor and walk through the steps. Maybe not 75% fail rate, but more like 50% of people fail this step in the tech round.
Now, that doesn't mean that stress management cannot be part of the job: I've worked in the hellish places where an on-call rotation meant at least 4 calls off-hours, and some which needed resolutions in 10 minutes or less from the pagerduty call. Someone with bad stress response is not going to be doing great at that kind of live diagnosing, with possibly many millions for the company on the line. But the number of positions like that, where you are a cross between a developer and an ER doctor are much less common than places that have none of those demands, yet give you a series of leetcode mediums and hards. They might as well be testing for height, or how much you can bench press. It filters, but it does not help.
Did y'all not have to take a load of exams in high school and college?
I'm sure there are variations between education systems, but by the time I graduated college I'd done at least a hundred high-stakes, time-pressured hour long written tests. All of them substantially more difficult than summing the even entries in a list.
Is this not what everyone experiences, in every education system?
You know the trope of almost all students dreading going up in front of the class to solve a problem on the blackboard/whiteboard? A thing that I think I personally only actually did a countable-on-fingers number of times ever, and only in lower grades?
It's a trope for a reason, and it's because the vast majority of people do in fact hate having to get in front of an audience and perform. Hell, it's often a component of recurring nightmares.
It's like that but way worse because it's also far more high-stakes than that, the problems are far more complex, the point is explicitly for everyone in the audience to judge you and not just your assumption that they all are causing the stress, et c.
It's more like the stress from getting up at an open mic night, than any kind of stress I've ever actually encountered on the job. Even the dynamic in a meeting with clients where you have to diagram something out on a whiteboard or something is totally different—for one thing, you generally have people in there who are very much "on your side", and you don't have to worry if you misspell something that you're personally about to be/remain unemployed, or lose out on a huge raise, or whatever (nobody generally cares about minor mistakes on a whiteboard in an ordinary context, and maybe they don't in some interviews, but they do in others, and candidates worry they might in all of them).
I've always finished exams absurdly quickly. I used to finish 60 min exams in like 20 min and sleep the rest of the time when the professor didn't allow us to leave, because I had likely spent the previous night drinking and partying my way through college. I'd usually ace most of them.
Thing is, at those times we were working with freshly acquired knowledge that you've been practicing a lot. That's not the case with most leetcode interviews. As a senior/staff engineer, I'm not using double pointers to perform little math tricks on a daily basis, I'm troubleshooting cascading timeouts on distributed systems. I'm not worried about how fast I'm going to iterate over a batch of a couple thousand records on a list I'm worried about how much latency I'll accrue if I rely on a primary source of truth instead of hitting a cached answer.
Code interviews don't measure experience or competence. I don't even think they measure stress as the article mentions. To me, they just measure how much leetcode you practiced for that interview. Nothing more.
I've had only two oral exams, and they were awful, but even then they were far more conversational in nature, as opposed to a coding interview.
I do well on written tests.
I also do well in interviews, but I detest having people look over my shoulder while I code, and it absolutely tanks my performance. I only pass them because I'm experienced enough that I can be stressed out like crazy and perform really horribly compared to what I normally do and still do okay.
I've seen fantastic coders totally freeze up and be unable to do anything at all when being asked to write code in front of people, even though they have no problem talking through problem solving.
And I'm similar to that. I've held speeches in front of thousands, been on TV, held plenty of presentations, and can talk the ears of an interviewer with passion about complicated technical concepts, but have me write code while you watch, and I'd frankly much rather have a root canal treatment (I've been known to fall asleep during those).
Yeah, I basically forget how to use all the basic tools I use all the time and my mistake rate shoots up 10x when I'm just screen sharing in a chill context. I'm (told I'm) quite good at all the social side of the job, at keeping my cool in rough situations (social or technical), and all that, but specifically being watched while I work turns me into a sort of foolish klutz.
Which makes these kinds of interviews absolute hell, because that's their entire deal.
What's the alternative? Take home problems come with their own problems, and select for people with no family and lots of time. Going by general vibe alone also works, kind of. But we're going to work together, so having something that at least faintly resembles work is kind of useful.
The leetcode type problems are useless however, fully agree on that. But a trivial loop with conditional, perhaps iterating over a list of lines shouldn't be that bad, especially if you get help with the syntax.
- Past: ~8 years in big tech as an IC - Currently: ~4 as CTO a < 10 person startup.
I always hated doing and conducting coding interviews. When I became in charge of the interview process, I swore to myself not to have them.
I had to let go of a person after ~3 weeks in because he just didn't write any code. This was about 1 year post the ChatGPT moment.
He had lots of experience. Great communicator. Culture fit (or so I thought). Etc...
Interviews can provide a signal, but nothing trumps a month of working with someone. If both parties are open to it, a well paid short term contracting gig goes a long way.
This is something that someone familiar with a language should be able to do as naturally as answering "tell me about yourself" or some other non-coding question. (Personally, way more naturally, nothing makes me more uncomfortable than "tell me about yourself")
I agree that for something more involved, nerves can become more of a factor, certainly any "trick" question, but these seem more like just checking if whatever language is second nature, which it should be, not about algorithms etc.
It's not a puzzle, it's not a trick question. It's simply a matter of 1) can you do the job and 2) do you get along with the team.
The last interview I did, we had the candidate build a game of marbles in Unity. None of us knew the rules, so it turned into a collaborative process between the candidate and our engineers. We learned that the candidate can think on his feet and fluently adapt his program to requirements as they manifest. Plus it was just a lot of fun. He was one of our best hires, too.
For example, for the senior frontend engineer, I think we went with "create a real-time comments panel using SSE that works in multiple browser tabs", and created a ready-made git repository with a dev server that includes the SSE endpoint. That way, they only had to focus on the actual task.
Now I thought that was nice because it should be very easy to solve for someone with a bit experience, but we get lots of opportunities to discuss their choices and the technology. It's hard to confidently bullshit an answer to if you think server-sent events or web sockets are superior, and why. So from everything we tried so far, this format gave me the most accurate impression of someone yet.
Which is funnily something opensource doesn't suffer from. You just have to prove identity to the commits you signed in previous contributions, so the problem of proving technical skills in the opensource community is replaced by the effort of building trust.
The answer therefore could be to replace technical interviews by sending out assignments to candidates to have them contribute to opensource and come back with results. This has the upside for candidates that if multiple companies do such assignments they only have to contribute once to be eligible for multiple hiring processes.
Everybody wins.
Yes, we can. Don't do them. But, we have to replace them with something that works. That means none of these poorly constructed take home projects that are almost universally either drastically over scoped, criminally under specified, or both.
We don't. The simple solution is to stop maintaining the illusion that 100% perfect hire is possible.
Design your post-hire process around imperfect hiring rate / quick feedback loop, and accept the losses (they will happen anyway despite any perfectionistic illusions you choose to maintain).
These are few questions that really matter:
Your interview process will always be just an attempt at answering these somewhat accurately, with diminishing returns after a certain point. Getting actual accurate answer to these is only possible through collaborative work in real environment.> We don't. The simple solution is to stop maintaining the illusion that 100% perfect hire is possible.
I just don't know what a better proxy of coding ability even is, when we exclude options that can't be gamed/cheated.
If someone can present a good solution and talk about the reasoning behind it with enough detail & savvy to convince you that they actually wrote it, does it matter if they didn't?
The catch would be not knowing whether the interviewee has the AI cargo cult PRIORS
Thing is, I reckon those candidates aren't a good fit anyways. The biggest mistake I see engineers who join startups make is that they pursue excessively sophisticated solutions, with robustness that does not justify the added complexity. I'm sure these candidates are smart, but they're not good fits for the high-velocity, simplicity-obsessed technical environment of a product-play startup.
I won't use live coding questions (usually) because I don't see much value in trying to code under that much stress.
But I will ask hard questions to see how well a candidate communicates in a stressful situation (which I think is a far more valuable indicator of their brain's "default" mode in stress). I want a candidate that talks honestly about stuff -- especially the stuff they don't understand -- even in a stressful situation. Sometimes stress can bring out the asshole in somebody -- and that's an immediate red flag.
But I do want to see how they cope with the stress of needing to track and articulate complex control flow and edge cases through their program in a timed environment - that's part of every day if you're building ambitious systems, and it's a skillset that continues to be relevant even in an era of agentic coding. Testing for the response to that stress makes coding interviews highly relevant in my experience.
Do you really need to code day in day out in an time window of the size of an interview (so 1 hour or 2) in your workplace? Or are you testing for extreme situation like patching a failing live production system?
Well, stress fires off old cortex fight-or-flight response. It's like the worst possible test of neocortex capability. Why are you trying to trigger it? Kinda cruel.
Why is the business this stressful in the first place
Mentally strong and healthy people who do well when challenged are good hires.
So, you discriminate against disabled people?
At least reading this post has helped me regain some of that lost self-esteem. Thanks for sharing it. <3
I've been coding for 20 years. Almost all this work is in some private repository behind multiple levels of locks protected by a stack of NDAs and Trade Secret agreements.
Not everyone is a hobbyist.
What happens is that people whine about live coding exercises; then someone decides to skip them, and immediately makes a bad hire. The lesson is then learned and they resume live coding exercises.
To turn the tables: As a candidate, If there isn't a live coding exercise, it's a red flag and tells me the company isn't savvy. (And I've taken a bad job as a result and now won't take a job unless I have a live coding exercise with someone who I will be working with.)
That said, does anyone have a good alternative? If somebody has open-source work and portfolios, there's a great window into their work, but that's probably an unfair expectation.
They get time to prepare and think about the problem. Because it's a familiar context, you can ask for more real-world alterations, discuss deployment meaningfully, etc.
I've done a fair amount of interviews and they all featured some amount of coding in front of an audience.
Who are these self-important employers?? I mean, outside of Ops and live incident response, are there really that many software engineer roles that require someone to perform well under pressure? It seems a false and egotistic narrative. If you need a software builder to perform under pressure then you've got systemic issues of a type not solved with software. You need better management or better work practices.
In the Dev world stress comes from the demand to add new features in unrealistic timeframes. The "product" has already been sold even if it's not finished yet. The business risks losing revenue by not delivering promised functionality.
Both Dev and Ops people experience equal stress from the same employer due to different reasons.
Should it be that way? No. Was it that way? Yes.
To date, I've never gone on to regret hiring someone who blitzed the in person coding exercise.
There's a separate problem here for hiring developers specifically, where realistically it should be easy to bring someone on limited term (say 3 months) and see how they work, but compensation and benefits in the US absolutely in no way support a system like that.
2 years ago given a coding assessment "Roman Numeral Conversion". Was given a set of test cases. I got near the solution. I think if I had another 5-10 minutes I would have solved it fine. Sat down that evening with the same test cases and did it from scratch in 20 minutes.
5 years ago, I didn't get a job because I while I did well technically. The reason for the rejection was that I appeared "stressed" while being assessed. What do they expect?
10 years ago. I walked out of an interview essentially after about 10 minutes. The interviewer(s) interrupted me the moment I started writing any code, constantly interrupting my train of thought. I think it was intentionally done to irritate me. I've had companies play these stupid games in interviews before.
I didn’t need the job that bad.
I’ve worked on one man projects that were hard realtime C image analysis (Fourier, calc, stats), large data science eval projects, devops stuff. CS degree and used it for published research.
I also had a five rounds of interviews a MANGA corp. Didn’t bother with the second because I didn’t have two months to waste.
Went hired gun and never looked back. Homey network uber alles, I make 10% less but I call all the shots, and am way healthier.
It will probably get worse before it gets better out there, between RTO and whatever LinkedIn is doing to CEOs.
Most people, even the good ones, show a little hesitance when starting. Which isn't really necessary, most people do fine. I'm not trying to get them to fail, I'm trying to get them to succeed. I want to see if they're smart and understand the problem and the direction of a solution. Not if they miss any semicolons or don't recall some arcane data structure.
She was one of the best hires I made. Coding interviews can also measure attitude and confidence.
He was a perfectly intelligent programmer and a fine person. I have no doubt at all he understood the details of bits and bytes. But, that group interview session did not sufficiently manage the stress level of the process. And, so we probably missed out on a perfectly good hire.
That and similar experiences in other group interviews are why my 1:1 interviews are structured around keeping the stress level low and preventing the candidate from freezing up.
Except humans can do well at live coding. Especially for trivial cases like the start of the article (sum . filter even). Yes, companies are looking to hire humans as opposed to ChatGPT, but they are looking to hire a subset of humans. Just being human is not good enough. Failing this test might not provide a reliable signal, but passing it does. If you are getting stressed over simple question or existing stress is impairing your thinking enough to not answer simple questions that seems like an issue to me.
A good employer takes care of their worker even if they have problems, but they also want to weed out as much of them before hiring. The value of worker is not revealed when they are at their best. Fragile workers with anxiety perform well only small amount of time.
Live interviews are generally excellent at revealing how individuals perform when placed outside their comfort zone. If you perform well in a live interview, be it for coding or any other skill, you are likely to perform even better reality.
I'm pretty sure there's no correlation whatsoever, after interviewing more than 500 people in a decade...
The correlation that I think _does_ exist is for _junior engineers_: Live coding interviews (particularly if you take typical CS problems) measures how much they're able to memorize and how much attention they paid in class (or in preparation). Which used to be a necessary requirement back in 2001, since it directly correlated to being able to learn by memorization. Unsure that's still a relevant skill since Google > Stack Overflow > ChatGPT, however...
I suppose if by "outside their comfort zone," you mean in a potentially literal life or death situation (because, face it, most people can't live long without a job in the US), under an arbitrary and very short deadline, possibly using unfamiliar tools, and simultaneously having to explain every little thing they do, then yes, I would agree with this. I don't believe the sentence after this follows logically from it, though.
If interview feels like life and death, maybe you are not the right hire.
If you don't understand how not having a job in a place like the US is life or death, you're not the right interviewer.