Self-Consistency Prompting

Who this is for

This guide is for readers who want an LLM to be more reliable on arithmetic and commonsense reasoning questions. By the end you will understand how self-consistency works, build a few-shot prompt, sample several outputs for the same question, and select the most consistent answer.

What self-consistency is

Self-consistency is one of the more advanced techniques in prompt engineering. Wang et al. (2022) proposed it^[1]. It aims to replace the naive greedy decoding method used in chain-of-thought prompting^[1]. The method samples several different reasoning paths using few-shot chain-of-thought prompting, then selects the most consistent answer from the results^[1]. According to the article, this helps chain-of-thought prompting perform better on arithmetic and commonsense reasoning tasks^[1].

Why one output is not enough

The article tests the idea with an age question. When the speaker was 6, their sister was half their age. The speaker is now 70, so how old is the sister?^[1]

当我6岁时，我的妹妹是我的一半年龄。现在我70岁了，我的妹妹多大？

Asked once, the model answers 35. The article marks that answer as wrong^[1].

Step 1: build the few-shot examples

The article's examples come from Wang et al. (2022), Table 17^[1]. The prompt contains eight question-and-answer examples. The last question is the test question, and its answer is left empty for the model to complete^[1]. The first example is:

Q：林中有15棵树。林业工人今天将在林中种树。完成后，将有21棵树。林业工人今天种了多少棵树？
A：我们从15棵树开始。后来我们有21棵树。差异必须是他们种树的数量。因此，他们必须种了21-15 = 6棵树。答案是6。

Step 2: sample several outputs

Run the same test question several times to get several reasoning paths. The first sampled output in the article is^[1]:

当我6岁时，我的妹妹是我的一半年龄，也就是3岁。现在我70岁了，所以她是70-3 = 67岁。答案是67。

The third sampled output is^[1]:

当我6岁时，我的妹妹是我的一半年龄，也就是3岁。现在我70岁了，所以她是70/2 = 35岁。答案是35。

Of the three sampled outputs, two give 67 and one gives 35^[1].

Step 3: pick the answer

The article's simplified approach is to look for the answer that most outputs agree on. For simplicity, it treats the most frequent answer as the final answer^[1]. The full method involves more steps for computing the final answer, which the paper describes^[1].

Why sampling matters

If every output were identical, sampling several times would add nothing. Self-consistency depends on the reasoning paths differing, so that agreement among them shows which answer is stable^[1].

One way to picture it is several people solving the same problem separately. Each sample is an independent attempt, and you look at which answer most of them reach. The approach only works when the reasoning paths really differ^[1].

Common questions

One output was wrong. Does that mean the method failed? Not necessarily. In the example above, the single output was 35, but among three samples the most frequent answer was 67. The value of the method is that it does not depend on any one output.

How should the examples be written? Examples that look like the question you want to solve are more useful. The article's examples are Chinese question-and-answer pairs, most of them ending with a sentence that states the answer^[1].

How many samples? The example uses three outputs and does not fix a number. Choose the count based on how hard the question is and what each call costs^[1].

Limitations

The article uses a simplified way to pick the answer: it takes the most frequent one and does not walk through the full calculation^[1]. To understand the complete method, read the paper itself.