XGBoost: A Scalable Tree Boosting System

Gradient tree boosting builds a model as a sum of small decision trees. Each new tree is fit to correct the errors of the trees before it, and the final prediction adds up a score from every tree. By 2015 it was the most reliable method for tabular data, the rows and columns of business databases, and the backbone of many competition wins and production systems for click prediction and ranking.

XGBoost, by Tianqi Chen and Carlos Guestrin at the University of Washington, made it fast and practical at scale. The paper contributes on two levels. On the algorithm side: a regularized objective that penalizes leaf count and leaf size; split scoring with both first and second derivatives of any differentiable loss; a split finder that learns which way missing values should go; and a weighted quantile sketch for choosing good candidate splits on large data. On the systems side: data stored in sorted column blocks so sorting happens once, cache-aware memory access, compression and sharding for data that does not fit in memory, and a distributed version.

The adoption numbers in the introduction are the authors' own count. Of 29 winning solutions published on Kaggle's blog during 2015, 17 used XGBoost. Every team in the top 10 of KDD Cup 2015 used it. The paper's experiments show it training as accurately as scikit-learn's implementation at a fraction of the time per tree, and handling 1.7 billion rows on a single machine.

Much of what customers want predicted lives in tables: churn, fraud, claim cost, late payment, lead scoring, demand. For those problems, boosted trees remain a strong default, and an FDE who reaches for an LLM first often ships something slower, more expensive and less accurate.

It is the baseline to beat. Before proposing a neural model or an LLM for a tabular prediction, train boosted trees on the same split. If the fancier model cannot clearly beat it, the customer should not pay for the fancier model.

Messy data is normal here. Real tables have missing values and one-hot or sparse columns. XGBoost learns a default direction for missing values at each split and only visits non-missing entries, so it runs on data as it is.

It works alongside LLMs. Common patterns include LLM-extracted fields or text embeddings used as features for a tree model, and small tree classifiers used as routers, filters or risk scores around an LLM system.

It is cheap to run. Training and inference fit on ordinary CPUs. A model that scores millions of rows a minute changes what a customer can afford to automate.

Its weaknesses are well known. Trees overfit if left unchecked, extrapolate poorly beyond the range of the training data, and give probabilities that may need calibrating. The tuning knobs in the paper, shrinkage, depth, λ, γ and column subsampling, exist to manage this.