Feature engineering

What feature engineering is

Feature engineering is a preprocessing step in supervised machine learning and statistical modelling. Its job is to turn raw data into inputs that a model can use more effectively. Every input is made up of attributes called features, and supplying informative features can substantially improve predictive accuracy and the quality of the decisions made from a model's output.^[1]

The definition is deliberately broad. Feature engineering is not a single algorithm but a collection of operations carried out before training, and the same data set may call for quite different treatment depending on which model will consume it.

What the operations look like

Typical operations include selecting features, creating new ones, transforming existing ones and extracting them, as well as imputing missing or invalid values and reducing dimensionality with methods such as PCA, ICA or LDA.^[1] Feature selection can be driven by importance scores and correlation matrices, which show how strongly each candidate input relates to the target and how strongly the candidates relate to one another.^[1]

A small example makes the idea concrete. Suppose a table records when each event happened as a single timestamp. A model cannot do arithmetic on a timestamp directly, but the same column can be re-expressed as a day of the week, an hour of the day, or the gap since the previous event — new features created out of an existing one. Filling in blanks and dropping near-duplicate columns are the other two moves that appear in almost every pipeline.

As candidate features multiply, so does the risk that the model latches onto noise. Regularisation, kernel methods and feature selection are the standard tools for controlling the explosion in the number of features.^[1]

Where it came from

Attempts to automate the process go back further than is often assumed. Research on automated feature engineering dates back to the 1990s, and Knobbe, Siebes and Van Der Wallen proposed multi-relational decision tree induction in 1999 as an early automated approach.^[1] Commercial software offering automated feature engineering has been available since 2016.^[1]

Where it is used today

Most general-purpose machine-learning libraries ship preprocessing tools. scikit-learn describes its preprocessing package as providing utility functions and transformer classes that change raw feature vectors into a representation more suitable for downstream estimators.^[2] The same documentation notes that many learning algorithms, linear models among them, benefit from standardization of the data set, and that robust scalers or other transformers can be more appropriate when outliers are present.^[2]

Deep learning has shifted the balance somewhat. Such models can work directly on large volumes of raw data, which reduces reliance on hand-crafted features, though careful preprocessing and cleaning are still needed.^[1]

Limits

Feature engineering is time-consuming and error-prone. It requires domain expertise and usually involves repeated trial and error, because there is rarely a way to know in advance which transformation will help a particular model on a particular data set.^[1]