Free download. Use it offline or customize it for your company.
Welcoming questions
Can you tell us about yourself and a data science project with real business impact?
What to look for: A strong answer quantifies the business outcome and explains the candidate's specific contribution, while projects described only in model metrics with no decision attached are a yellow flag.
Sample answer: My background is a statistics degree followed by five years in e-commerce and logistics data teams. The project I'd point to is a delivery time prediction model I owned end to end: the old promise was a static 3 to 5 day window, and my gradient boosting model on carrier, route, and warehouse features cut the promise error by about 40 percent. That let the company show tighter delivery estimates at checkout, and conversion on those pages went up 2.3 percent in the follow-up experiment.
What kinds of problems do you most enjoy working on?
What to look for: Look for alignment between the candidate's preferences and your actual roadmap, and honesty about what they enjoy less.
Sample answer: I gravitate to problems where the modeling connects to a decision someone makes weekly, like pricing, forecasting, or churn intervention, rather than one-off analyses that end as a slide. I get a lot of energy from the messy framing phase, taking a vague 'why are customers leaving' and turning it into something testable. If I'm being honest, I enjoy dashboard maintenance the least, though I've done my share of it.
Role-specific and technical questions
Walk us through a model you built: problem framing, baseline, features, evaluation, and outcome.
What to look for: Strong candidates start with a simple baseline and justify each step against the business goal, while jumping straight to a complex architecture without a baseline is a red flag.
Sample answer: For a subscription churn model, I first defined the target carefully: cancellation within 60 days of the prediction date, which mattered because the intervention team needed lead time. My baseline was logistic regression on five obvious features, tenure, usage trend, support tickets, plan type, and payment failures, and it hit 0.71 AUC. XGBoost with about 40 features got to 0.79, and I checked calibration since the retention team used the scores as probabilities for targeting. In production it drove a save campaign that recovered roughly 8 percent of predicted churners, and we validated that with a holdout group that got no outreach.
How do you design an A/B test, and what are the mistakes you watch for?
What to look for: Look for pre-registered metrics, power analysis, and a fixed stopping rule, with awareness of peeking, multiple comparisons, and interference between units.
Sample answer: Before anything launches I lock the primary metric, run a power analysis to set sample size and duration, and write down the decision rule, because a test without a pre-committed stopping point invites peeking until significance appears. The mistakes I actively guard against: stopping early on a lucky streak, slicing into twenty segments post hoc and reporting the one that's significant, and randomizing at the wrong unit, like by session when users see both variants. I also always run an A/A test or a sample ratio check on new assignment infrastructure, which has caught broken bucketing twice in my career.
How do you decide between a simple model and a complex one?
What to look for: Strong answers weigh the accuracy gain against interpretability, maintenance, and latency costs, and default to simple unless complexity pays measurable rent.
Sample answer: The complex model has to buy its complexity with a gain that matters downstream, not just a third decimal place of AUC. I ask three questions: does the metric improvement translate to business value, can we debug and explain the model when it misbehaves at 2 a.m., and can the team that inherits it maintain it? I've replaced a neural network with a regularized regression that was 1 percent worse offline, and nobody noticed any business difference, but on-call incidents for that pipeline basically stopped.
How do you communicate uncertainty to stakeholders who want a single number?
What to look for: Look for concrete techniques for translating intervals into decision terms, rather than either drowning stakeholders in statistics or capitulating to false precision.
Sample answer: I give them the number they want, then immediately price the uncertainty in decision terms: 'best estimate is 12 percent lift, and I'd bet on anywhere between 5 and 19; even the low end pays for the project.' Framing the interval as scenarios, what we do if it's the low case versus the high case, works far better than p-values. When a forecast is genuinely too uncertain to act on, I say that plainly and propose what data or time would tighten it, because a confident wrong number costs more than an honest range.
Describe your process for validating data quality before modeling.
What to look for: Strong candidates have a systematic checklist including distributions, nulls, duplicates, and leakage checks, and can name a real bug this process caught.
Sample answer: I profile every source before I trust it: distributions and null rates per column, duplicate keys, row counts over time to spot pipeline gaps, and a reconciliation of a few aggregates against a known source of truth like the finance dashboard. The check I never skip is leakage: for every feature I ask whether its value was actually knowable at prediction time. That habit once caught a 'days since last support ticket' feature that was computed after the churn event, which was the only reason the model looked spectacular offline.
What is your approach to monitoring a model after it ships to production?
What to look for: Look for monitoring of input drift and prediction distributions plus a defined retraining or rollback trigger, since offline-only thinking is a common gap.
Sample answer: I monitor three layers: input feature distributions against the training baseline, the prediction distribution itself, and the business metric the model is supposed to move. Ground truth often arrives late, churn labels take 60 days, so drift in inputs and outputs is the early warning system. I set explicit thresholds that trigger investigation, and I define the retraining cadence and a rollback path before launch, not during the incident. On one pricing model, an upstream currency bug showed up as a feature drift alert a full week before anyone downstream noticed revenue oddities.
Behavioral and culture fit questions
Tell us about a project where the data did not support the hypothesis. What did you do?
What to look for: The best answers show scientific honesty: reporting the negative result clearly and framing it as decision value, while torturing the data until something significant appeared is a red flag.
Sample answer: Leadership was convinced our onboarding emails drove activation, and I was asked to quantify the effect to justify expanding the program. A proper holdout test showed essentially zero causal impact; the correlation everyone had seen was just engaged users doing both things. I presented it straight, with the caveat analysis showing we had power to detect even a small effect. It wasn't the popular finding, but it freed up a content team's roadmap, and the reworked onboarding flow we tested next actually did move activation by 6 percent.
Describe a time a stakeholder pushed for a conclusion the data did not support.
What to look for: Look for candidates who held the line respectfully with evidence and offered a constructive path, rather than either caving or turning it into a standoff.
Sample answer: A marketing lead wanted to report a campaign as driving 20 percent of new signups, based on last-touch attribution that I knew overcounted. Instead of just saying no, I showed him what the number looked like under three attribution approaches and explained why they diverged, then proposed we run a geo holdout on the next campaign to get a real causal read. He didn't love the smaller number, but he used the range in his report, and the holdout test we ran later became the standard for his team's claims.
How do you decide when analysis is good enough to ship a recommendation?
What to look for: Strong candidates calibrate rigor to the cost and reversibility of the decision, while perfectionism that stalls decisions and hand-waving that skips validation are both red flags.
Sample answer: My bar scales with the decision's cost and reversibility. A reversible product tweak can ship on a directional analysis with known caveats; a pricing change or a headcount decision gets the full treatment, sensitivity checks and all. My practical test is: would more analysis plausibly change the recommendation? If every robustness check I can think of points the same direction, more polish is procrastination. I write the caveats into the recommendation itself so the decision-maker knows exactly what we're uncertain about.
Problem-solving and case questions
We want to predict which customers will churn next quarter. How would you approach it end to end?
What to look for: Look for careful target definition, a leakage-aware validation setup, and connecting predictions to an intervention, not just a recitation of algorithms.
Sample answer: I'd start by pinning down the definition with the team that will act on it: what counts as churn, over what window, and crucially what they'll do with the scores, because a model nobody intervenes on is trivia. Then I'd build the training set with a strict time cutoff, features only from before the prediction date, and validate on a later time slice rather than a random split. Modeling-wise, logistic regression baseline, then gradient boosting if it earns the gap. The end product isn't an AUC; it's a ranked list sized to the retention team's capacity, and an experiment design with a no-contact holdout to measure whether the intervention actually saves anyone.
An experiment shows a significant lift, but the effect disappears after launch. What might explain it?
What to look for: Strong candidates generate multiple hypotheses, such as winner's curse, novelty effects, seasonality, or implementation differences, and propose how to distinguish between them.
Sample answer: Several suspects come to mind. Winner's curse first: if the test was underpowered or stopped early, the measured lift was likely inflated even if the effect is real. Then novelty effects, where users respond to change itself and revert. I'd also check whether the launched version actually matches the tested variant, since implementations drift, and whether the launch population differs from the test population, like the test excluding mobile. To diagnose, I'd compare the test and post-launch populations, check whether the effect decayed within the experiment itself, and ideally keep a small long-running holdback after launch, which is the cleanest way to settle it.
Your model performs well offline but poorly in production. Walk us through your investigation.
What to look for: Look for a systematic hunt through training-serving skew, leakage, data pipeline differences, and evaluation mismatch, rather than jumping straight to retraining.
Sample answer: My first suspect is always training-serving skew: I'd pull a sample of production requests and compare the feature values the serving system computed against what the offline pipeline would produce for the same entities. Mismatches there, like a feature defaulting to null in production, explain most of these cases. Second suspect is leakage, where a training feature encoded future information, so offline numbers were never real. Third, I'd check whether the production population differs from the training data, new user segments or a different time period. I'd resist retraining until I know which one it is, because retraining on top of a skew bug just hides it.
A product manager asks whether a new feature caused a spike in engagement, but there was no experiment. How do you approach the question?
What to look for: Strong answers reach for quasi-experimental methods with honest caveats about confounders, while claiming causality from a before-and-after chart is a red flag.
Sample answer: First I'd be upfront that without randomization we'll get an estimate with caveats, not proof. Then I'd look for natural structure to exploit: a staggered rollout lets me compare early and late adopters over time, or if it launched everywhere at once, an interrupted time series with seasonality controls and a check against metrics the feature couldn't plausibly have touched, as a placebo test. I'd also hunt for confounders around the launch date, like a marketing push in the same week. If the answer really matters for a big decision, I'd propose turning the feature off for a small holdback group, which converts the whole question into a clean experiment.
Generate custom Data Scientist interview questions
Need questions tuned to your industry, seniority level, or interview stage? Describe the role and our AI interview questions generator will draft a set in seconds.
