Natural Language Processing (Almost) from Scratch

One architecture instead of four pipelines

Natural language processing is not a single problem but a family of them. A system may have to label every word with its part of speech, group words into short phrases, pick out names of people and places, or work out who did what to whom in a sentence. Each of these tasks is usually handled by its own pipeline, with features designed by hand for that task and rules distilled from earlier work.

"Natural Language Processing (Almost) from Scratch" asks whether that separation is necessary. The paper, by R. Collobert, J. Weston, L. Bottou, M. Karlen, K. Kavukcuoglu and P. Kuksa, appeared in the Journal of Machine Learning Research in 2011, in volume 12, pages 2493 to 2537.^[1]

What the authors built

The abstract describes a unified neural network architecture and learning algorithm applicable to part-of-speech tagging, chunking, named entity recognition and semantic role labeling.^[1] A neural network here means a stack of simple computational layers whose parameters are adjusted by an algorithm that reduces the error on training examples; the same machinery is used across all four tasks rather than a different set of components for each.

The design is deliberately austere. The authors state that the approach avoids task-specific engineering and disregards a lot of prior knowledge.^[1] In practice that means the system is not handed lists of hand-written patterns, dictionaries of known names, or features that linguists have established for one particular task. The four tasks differ in their inputs and outputs, but the machinery that turns words into predictions is shared.

Learning representations from unlabeled text

A second design choice is where the knowledge comes from. The system learns its internal representations from vast amounts of mostly unlabeled training data.^[1] Instead of describing a word with hand-chosen properties, the model represents each word as a vector of numbers that is learned, and adjusts those vectors so that words used in similar contexts end up with similar vectors. Because such data is plentiful and does not need to be annotated by hand, most of the learning can happen before any task-specific labels are used.

This is the part of the work that gives the title its parenthesis. "Almost from scratch" does not mean the system knows nothing; it means that what the system knows was learned from raw text rather than written down by its designers.

What came out of it

The abstract reports that the work was used as the basis for a freely available tagging system with good performance and minimal computational requirements.^[1] A released system matters for how a method is judged: other people can run it, apply it to their own text, and compare it against alternatives without reimplementing the method from a description.

Why it matters

The paper's contribution is a demonstration that several language tasks can be served by one architecture, one learning algorithm and one source of knowledge — large amounts of mostly unlabeled text — with little per-task engineering. That shifts the work of building a language system away from designing features for each task and toward collecting data and learning representations from it. The freely available tagging system extended the reach of the approach beyond the authors' own experiments.^[1]

The scope claimed in the abstract is specific: four tagging and labeling tasks, one architecture, one training procedure. The paper's argument rests on those tasks being handled well enough by a shared, largely hand-craft-free design to be worth the loss of task-specific tuning.