Generated Knowledge Prompting

Who this is for and what you will get

This is for readers who need a model to answer commonsense judgment questions and find that its first answer is wrong. After reading it, you can apply the two-step pattern of generating background knowledge first and then answering from that knowledge to your own questions, and you will know why the results can vary.

What generated knowledge prompting is

Large language models keep improving, and one popular idea is to fuse knowledge or information into the prompt so the model makes more accurate predictions. Generated knowledge prompting asks a further question: can the model generate that knowledge itself before making a prediction? Liu et al. (2022) tested exactly this, putting the generated knowledge into the prompt as part of the input and studying how much it helps on tasks such as commonsense reasoning.^[1][2]

A failed example to start with

When asked directly, the model answers “Yes” to the question below. The error shows where LLMs struggle when a task needs more knowledge about the world.^[1]

高尔夫球的一部分是试图获得比其他人更高的得分。是或否？

Step one: have the model generate knowledge

This step uses a few-shot prompt. Each example gives an input sentence and a line of knowledge. The examples cover topics such as the land areas of Greece and Mexico, condensation on glasses, fish cognition, smoking and lung cancer risk, and the size of pebbles.^[1] The prompt ends with a new input and an empty knowledge line for the model to complete.^[2]

输入：希腊比墨西哥大。
知识：希腊的面积约为131,957平方公里，而墨西哥的面积约为1,964,375平方公里，使墨西哥比希腊大了1,389%。

输入：眼镜总是会起雾。
知识：当你的汗水、呼吸和周围的湿度中的水蒸气落在冷的表面上，冷却并变成微小的液滴时，会在眼镜镜片上产生冷凝。你看到的是一层薄膜。你的镜片相对于你的呼吸会比较凉，尤其是当外面的空气很冷时。

输入：鱼有思考能力。
知识：鱼比它们看起来更聪明。在许多领域，如记忆力，它们的认知能力与或超过非人类灵长类动物等“更高级”的脊椎动物。鱼的长期记忆帮助它们跟踪复杂的社交关系。

输入：一个人一生中吸烟很多香烟的常见影响是患肺癌的几率高于正常水平。
知识：那些一生中平均每天吸烟不到一支香烟的人，患肺癌的风险是从不吸烟者的9倍。在每天吸烟1到10支香烟之间的人群中，死于肺癌的风险几乎是从不吸烟者的12倍。

输入：一块石头和一颗卵石大小相同。
知识：卵石是一种根据Udden-Wentworth沉积学尺度的颗粒大小为4到64毫米的岩屑。卵石通常被认为比颗粒（直径2到4毫米）大，比卵石（直径64到256毫米）小。

输入：高尔夫球的一部分是试图获得比其他人更高的得分。
知识：

For the golf question, the first generated knowledge reads:^[1][2]

高尔夫球的目标是以最少的杆数打完一组洞。一轮高尔夫球比赛通常包括18个洞。每个洞在标准高尔夫球场上一轮只打一次。每个杆计为一分，总杆数用于确定比赛的获胜者。

The second generated knowledge reads:^[1][2]

高尔夫球是一项精密的球杆运动，竞争选手（或高尔夫球手）使用许多类型的球杆，用最少的杆数将球打入球场上一系列洞中。目标是以最低的分数完成比赛，该分数是通过将每个洞上所用的杆数相加计算出来的。得分最低的选手赢得比赛。

Step two: answer with the knowledge in the prompt

The second step integrates the knowledge into the prompt and makes the prediction. The question is rewritten in QA format: the question comes first, then a knowledge line, and the prompt ends with “解释和答案：” to guide the format of the answer.^[1] With the first piece of knowledge, the prompt looks like this:

问题：高尔夫球的一部分是试图获得比其他人更高的得分。是或否？

知识：高尔夫球的目标是以最少的杆数打完一组洞。一轮高尔夫球比赛通常包括18个洞。每个洞在标准高尔夫球场上一轮只打一次。每个杆计为一分，总杆数用于确定比赛的获胜者。

解释和答案：

The model answers:^[1]

不是，高尔夫球的目标不是获得比其他人更高的得分。相反，目标是以最少的杆数打完一组洞。总杆数用于确定比赛的获胜者，而不是总得分。

The model’s confidence in this answer is very high.

Two pieces of knowledge, two different results

With the second piece of knowledge, the same question is put in the same QA format:^[1][2]

问题：高尔夫球的一部分是试图获得比其他人更高的得分。是或否？

知识：高尔夫球是一项精密的球杆运动，竞争选手（或高尔夫球手）使用许多类型的球杆，用最少的杆数将球打入球场上一系列洞中。目标是以最低的分数完成比赛，该分数是通过将每个洞上所用的杆数相加计算出来的。得分最低的选手赢得比赛。

解释和答案：

This time the model’s confidence is much lower, and its answer becomes “yes”:^[1][2]

是的，高尔夫球的一部分是试图获得比其他人更高的得分。每个选手都试图以最少的杆数打完一组洞。总杆数用于确定比赛的获胜者，而不是总得分。得分最低的选手赢得比赛。

The same question produced two conclusions from two sets of knowledge, and the model was very confident about one and clearly less sure about the other. The two runs differ only in the knowledge line, while the question and the answer format stay the same. The quality of the generated knowledge therefore shapes the final answer, and the model’s tone alone is not a reliable signal.^[1][2]

Limits and cautions

Results can be unstable. Changing the knowledge can change the conclusion, so generated knowledge should be checked by a person before it is used to answer. A practical habit is to generate more than one piece of knowledge for the same question and compare the answers before trusting any single one.^[1][2]

The example is a simplified demonstration. The walkthrough skips steps to stay short, and reaching a final answer involves more details, which the paper covers in full.^[1][2]

The examples shape the knowledge. Every few-shot example follows the same input-and-knowledge structure, and the model tends to follow that pattern when it generates knowledge.^[1]

Next steps

For the complete method and more detail, read the paper by Liu et al. (2022).^[2]

This article is adapted from the Generated Knowledge Prompting page in DAIR.AI’s *Prompt Engineering Guide*^[1], released under the MIT License.