TL;DR: A feature store solves two problems: one feature definition feeding both training and serving (killing training-serving skew) and reuse across teams. The offline store holds timestamped history for point-in-time-correct training-set assembly; the online store holds only the latest value per entity for sub-10ms lookups on the live path. Materialization writes both from one definition.
How to approach it
A standard warm-up in Uber-style ML system design rounds, and the answer that fails is "it's a place to store features." The interviewer is checking whether you understand the two problems it actually solves, write-once-read-twice feature definitions, and serving the same feature at two wildly different latency profiles. Lead with the problem, then the two stores.
A strong answer
The core problem: without a feature store, every feature gets implemented twice, once in Spark or SQL for training, once in application code for inference. Two implementations drift, and you get training-serving skew: a model trained on avg_txns_30d computed one way and served another, silently underperforming with no error anywhere. The feature store's first job is one feature definition feeding both paths. Its second job is reuse: the fifth team to need customer_lifetime_value reads it instead of rebuilding it.
The same feature then needs two physical homes because the access patterns are opposites:
The offline store holds historical feature values, date-partitioned, append-only, typically Parquet or Delta on object storage. Its consumer is training-set generation: "give me these 40 features for these 10M entities as they were at each label's timestamp." That as-of join is point-in-time correctness, and it's the offline store's defining capability, without it you leak future information into training. Throughput matters here, latency doesn't; a backfill scanning two years of history can take an hour.
The online store holds only the latest value per entity in a low-latency KV store, Redis, DynamoDB, Cassandra. Its consumer is the live prediction path: a fraud model with a 100ms end-to-end budget can afford maybe 10ms for feature lookups, single-digit milliseconds p50. No history, no scans, just get(entity_id) → feature vector.
Point-in-time correctness stays abstract until you watch one row leak, so walk one (illustrative values, but the mechanism is exact). A fraud label lands on the March 1 transaction of customer 8841; the account was frozen on March 2, as fraud confirmations usually trigger. Assemble training data the naive way, joining today's feature table, and avg_txns_30d for that row reads 0, because the account has been frozen for months by the time you build the dataset. The as-of join instead reaches into history and returns the value as of March 1: 45 transactions, a busy account, which is what the model would actually have seen at prediction time. Train on the naive join and the model learns the spectacular rule "zero recent transactions means fraud," scores brilliantly offline (the leak is in the eval set too, so nothing looks wrong), and collapses in serving where frozen-because-fraud accounts are not yet frozen. The offline store's timestamped history exists to make the correct join cheap; without it, every training set is one lazy join away from learning the consequences of its own labels.
Materialization jobs (batch and streaming) compute features once and write to both stores. That single-definition, dual-write pattern is the whole trick.
The two stores share a definition but never a database: opposite access patterns, opposite latency budgets.
The split, side by side:
| Dimension | Online store | Offline store |
|---|---|---|
| Latency | Sub-10ms reads, single-digit ms p50 | Latency doesn't matter; backfills can take an hour |
| Access pattern | get(entity_id), no scans | As-of join scanning history |
| Storage tech | Redis, DynamoDB, Cassandra | Parquet or Delta on object storage |
| Query unit | Latest value per entity | Timestamped values for many entities |
| Retention | Latest value only, no history | Full date-partitioned, append-only history |
And the senior caveat worth volunteering: you don't always need one. A batch-only shop scoring nightly with three models and no real-time serving gets all the cost (infrastructure, on-call, a new failure domain) and little benefit, a well-organized set of Delta tables is the right answer there. The triggers for adopting one: real-time inference plus features needing freshness, or multiple teams visibly rebuilding the same features.
What interviewers probe next
- "How do you keep online and offline values consistent?", same transformation code via the store's definitions, plus a parity check job that samples online reads and diffs them against offline recomputation; alert on mismatch rates above a fraction of a percent.
- "What's point-in-time correctness?", training rows may only contain feature values that existed at prediction time; the offline store does time-aware joins against feature timestamps to guarantee it.
- "Streaming feature like 'transactions in the last 5 minutes', where does it live?", computed in a stream processor (Flink/Spark Streaming) writing to the online store within seconds, with the same aggregation logged or replayed into offline for training parity.
- "Build or buy?", Feast over your existing infrastructure if you mainly need the consistency layer; SageMaker/Vertex/Databricks feature stores when you're already committed to that platform; building from scratch only at genuine Uber-Michelangelo scale.
Common mistakes
Defining it as storage rather than as a consistency and reuse layer, the difference between the two answers is the whole question. Forgetting point-in-time joins, which is the offline store's reason to exist. Quoting "low latency" without a number (sub-10ms reads is the expectation; saying "fast" suggests you've never been on the path that had the budget). And recommending one universally, the "when not to" answer is on the senior rubric.