Chain-of-thought prompting

If a model fails at multi-step arithmetic or logic problems even with examples in the prompt, chain-of-thought (CoT) prompting is the fix to reach for: instead of demanding a bare answer, you let the model write out intermediate reasoning steps first. This page shows how to write few-shot CoT demonstrations, how one added sentence enables zero-shot reasoning, and how Auto-CoT generates the demonstrations for you.

What chain-of-thought prompting is

CoT prompting, introduced by Wei et al. (2022), enables complex reasoning capabilities through intermediate reasoning steps ^[1]. Instead of jumping straight to an answer, the model is shown — or simply asked — to work through the problem step by step. It combines naturally with few-shot prompting for more complex tasks that require reasoning before answering ^[1]. The idea behind it is simple: demanding a bare answer forces the model to jump straight to a conclusion, while letting it write steps means each step's output becomes the input for the next one, and one hard calculation turns into several easy ones.

The authors claim this is an emergent ability of sufficiently large language models ^[1]; a small model will not necessarily benefit from reasoning-style demonstrations.

Few-shot CoT: demonstrations with the steps written out

Recall the odd-numbers task that plain few-shot prompting failed on. In a CoT prompt, each demonstration spells out the arithmetic ^[1]:

这组数中的奇数加起来是偶数：4、8、9、15、12、2、1。
A：将所有奇数相加（9、15、1）得到25。答案为False。

这组数中的奇数加起来是偶数：17、10、19、4、8、12、24。
A：将所有奇数相加（17、19）得到36。答案为True。

这组数中的奇数加起来是偶数：16、11、14、4、8、13、24。
A：将所有奇数相加（11、13）得到24。答案为True。

这组数中的奇数加起来是偶数：17、9、10、12、13、4、2。
A：将所有奇数相加（17、9、13）得到39。答案为False。

这组数中的奇数加起来是偶数：15、32、5、13、82、7、1。
A：

The model answers:

将所有奇数相加（15、5、13、7、1）得到41。答案为False。

Providing the reasoning steps produces perfect results on this task ^[1]. Fewer demonstrations even suffice: a single example appears to be enough, and the model still answers correctly ^[1]:

这组数中的奇数加起来是偶数：4、8、9、15、12、2、1。
A：将所有奇数相加（9、15、1）得到25。答案为False。

这组数中的奇数加起来是偶数：15、32、5、13、82、7、1。
A：

Compare this with the failed few-shot version, where demonstrations only showed "答案是True/False" without the arithmetic. The difference is not the number of examples but whether the demonstrations contain the reasoning process itself.

Zero-shot CoT: one sentence is enough

If you have no examples at hand, there is an even lighter variant. Zero-shot CoT, proposed by Kojima et al. (2022), adds "让我们逐步思考" (let's think step by step) to the prompt ^[1].

First, without it, a simple word problem fails:

我去市场买了10个苹果。我给了邻居2个苹果和修理工2个苹果。然后我去买了5个苹果并吃了1个。我还剩下多少苹果？

11个苹果

That answer of 11 apples is incorrect ^[1]. Appending the sentence to the same question:

我去市场买了10个苹果。我给了邻居2个苹果和修理工2个苹果。然后我去买了5个苹果并吃了1个。我还剩下多少苹果？

让我们逐步思考。

The model now reasons through the problem and reaches the correct count of 10 apples ^[1]:

首先，您从10个苹果开始。
您给了邻居和修理工各2个苹果，所以您还剩下6个苹果。
然后您买了5个苹果，所以现在您有11个苹果。
最后，您吃了1个苹果，所以您还剩下10个苹果。

Zero-shot CoT is especially useful when you do not have many examples available for the prompt ^[1]. A few practical notes when writing CoT prompts: keep every demonstration's reasoning correct, because the model imitates the style of the steps it sees; match the granularity of the steps to the task — too coarse gives no decomposition, too fine inflates the prompt; and end every demonstration's rationale with the same answer format, such as "答案为False", so the model closes with a line your program can parse.

Auto-CoT: generating demonstrations automatically

Few-shot CoT works, but effective and diverse examples usually have to be hand-crafted, and this manual work can lead to suboptimal solutions ^[1]. Auto-CoT, proposed by Zhang et al. (2022), removes the manual step: it uses LLMs with the "让我们一步一步地思考" prompt to generate reasoning chains one after another ^[1].

The automatic process can still make errors in the generated chains, so the diversity of the demonstrations matters to mitigate their impact. Auto-CoT samples questions with diversity and generates chains of thought to build the demonstrations ^[1].

The pipeline has two stages ^[1]:

1. Question clustering: partition the given questions into several clusters.
2. Demonstration sampling: select a representative question from each cluster and generate its chain of thought with Zero-Shot-CoT plus a simple heuristic.

The heuristic can be the question length (for example, 60 tokens) and the number of steps in the rationale (for example, 5 reasoning steps), which encourages simple yet accurate demonstrations ^[1]. The code is available in the GitHub repository amazon-science/auto-cot ^[1].

Which tasks need it

Put the two pages together: for tasks where the answer format is the whole challenge, plain few-shot prompting is enough; for tasks that take several reasoning steps, go straight to CoT — write demonstrations that include the steps if you have examples, or add one sentence if you do not. The two combine well: few-shot CoT, where every demonstration includes reasoning, gives better results on complex tasks ^[1]. A sensible migration path: start with zero-shot CoT, add reasoned demonstrations if that fails, and only then reach for Auto-CoT to generate examples at scale. Keep the precondition in mind: this is an ability the authors claim appears only in sufficiently large models ^[1].

Adapted from the chain-of-thought prompting section of DAIR.AI's *Prompt Engineering Guide*^[1], released under the MIT License.