Test changes systematically
系統(tǒng)地測(cè)試變化
Improving performance is easier if you can measure it. In some cases a modification to a prompt will achieve better performance on a few isolated examples but lead to worse overall performance on a more representative set of examples. Therefore to be sure that a change is net positive to performance it may be necessary to define a comprehensive test suite (also known an as an "eval").
如果能夠?qū)π阅苓M(jìn)行衡量，那么改進(jìn)性能就更容易。在某些情況下，對(duì)提示進(jìn)行修改可能會(huì)在一些孤立的示例上獲得更好的性能，但在更具代表性的示例集上可能會(huì)導(dǎo)致更差的整體性能。因此，為確保改變對(duì)性能的凈增益，可能需要定義全面的測(cè)試套件（也稱(chēng)為“評(píng)估”）。
Sometimes it can be hard to tell whether a change — e.g., a new instruction or a new design — makes your system better or worse. Looking at a few examples may hint at which is better, but with small sample sizes it can be hard to distinguish between a true improvement or random luck. Maybe the change helps performance on some inputs, but hurts performance on others.
有時(shí)很難判斷一個(gè)變化（例如新的指令或新的設(shè)計(jì)）是使系統(tǒng)變得更好還是更糟。通過(guò)查看幾個(gè)示例可能可以暗示哪個(gè)更好，但是對(duì)于小樣本量來(lái)說(shuō)，很難區(qū)分是真正的改進(jìn)還是隨機(jī)運(yùn)氣。也許這個(gè)變化在某些輸入上有助于性能，但對(duì)其他輸入有害。
Evaluation procedures (or "evals") are useful for optimizing system designs. Good evals are:
評(píng)估程序（或“evals”）對(duì)于優(yōu)化系統(tǒng)設(shè)計(jì)非常有用。良好的評(píng)估應(yīng)該具備以下特點(diǎn)：
- Representative of real-world usage (or at least diverse)
- 代表現(xiàn)實(shí)世界的使用情況（或至少多樣化）
- Contain many test cases for greater statistical power (see table below for guidelines)
- 包含許多測(cè)試用例以增加統(tǒng)計(jì)功效（請(qǐng)參考下表以獲取指導(dǎo)方針）
- Easy to automate or repeat
- 易于自動(dòng)化或重復(fù)進(jìn)行
|
DIFFERENCE TO DETECT 差異的檢測(cè) |
SAMPLE SIZE NEEDED FOR 95% CONFIDENCE 需要的樣本大小以達(dá)到95%的置信度 |
| 30% |
~10 |
| 10% |
~100 |
| 3% |
~1,000 |
| 1% | ~10,000 |
Evaluation of outputs can be done by computers, humans, or a mix. Computers can automate evals with objective criteria (e.g., questions with single correct answers) as well as some subjective or fuzzy criteria, in which model outputs are evaluated by other model queries. OpenAI Evals is an open-source software framework that provides tools for creating automated evals.
所做的評(píng)估可以由計(jì)算機(jī)、人類(lèi)或二者混合進(jìn)行。計(jì)算機(jī)可以根據(jù)客觀標(biāo)準(zhǔn)（例如，具有單個(gè)正確答案的問(wèn)題）自動(dòng)化評(píng)估，也可以根據(jù)其他模型查詢(xún)來(lái)評(píng)估模型輸出的一些主觀或模糊標(biāo)準(zhǔn)。OpenAI Evals 是一個(gè)開(kāi)源軟件框架，提供了創(chuàng)建自動(dòng)化評(píng)估的工具。
Model-based evals can be useful when there exists a range of possible outputs that would be considered equally high in quality (e.g. for questions with long answers). The boundary between what can be realistically evaluated with a model-based eval and what requires a human to evaluate is fuzzy and is constantly shifting as models become more capable. We encourage experimentation to figure out how well model-based evals can work for your use case.
基于模型的評(píng)估在存在一系列可能的輸出被認(rèn)為具有相等高質(zhì)量的情況下（例如，對(duì)于答案較長(zhǎng)的問(wèn)題）可能會(huì)很有用?；谀Ｐ偷脑u(píng)估能夠進(jìn)行實(shí)際評(píng)估和需要人類(lèi)評(píng)估的邊界是模糊的，并且隨著模型能力的提升而不斷變化。我們鼓勵(lì)進(jìn)行實(shí)驗(yàn)，以確定基于模型的評(píng)估在您的用例中能夠發(fā)揮多大作用。
Tactic: Evaluate model outputs with reference to gold-standard answers
技巧：根據(jù)黃金標(biāo)準(zhǔn)答案評(píng)估模型輸出
Suppose it is known that the correct answer to a question should make reference to a specific set of known facts. Then we can use a model query to count how many of the required facts are included in the answer.
假設(shè)我們已知正確答案應(yīng)該引用一組特定的已知事實(shí)。然后，我們可以使用模型查詢(xún)來(lái)計(jì)算答案中包含的必需事實(shí)的數(shù)量。
For example, using the following system message:
例如，使用以下系統(tǒng)消息：
|
SYSTEM 系統(tǒng) |
You will be provided with text delimited by triple quotes that is supposed to be the answer to a question. Check if the following pieces of information are directly contained in the answer: 您將獲得由三個(gè)反引號(hào)分隔的文本，這應(yīng)該是一個(gè)問(wèn)題的答案。檢查以下信息是否直接包含在答案中： - Neil Armstrong was the first person to walk on the moon. 尼爾·阿姆斯特朗是第一個(gè)在月球上行走的人。 - The date Neil Armstrong first walked on the moon was July 21, 1969. 尼爾·阿姆斯特朗首次登上月球的日期是1969年7月21日。 For each of these points perform the following steps: 對(duì)于這些要點(diǎn)，執(zhí)行以下步驟： 1 - Restate the point. 1 - 重新闡述要點(diǎn)。 2 - Provide a citation from the answer which is closest to this point. 2 - 提供離此要點(diǎn)最近的答案中的引用。 3 - Consider if someone reading the citation who doesn't know the topic could directly infer the point. Explain why or why not before making up your mind. 3 - 考慮如果閱讀引用的人不了解這個(gè)主題，能否直接推斷出這個(gè)要點(diǎn)。在做出決定之前解釋為什么或?yàn)槭裁床弧?/p> 4 - Write "yes" if the answer to 3 was yes, otherwise write "no". 4 - 如果答案是“是”，則寫(xiě)“yes”，否則寫(xiě)“no”。 Finally, provide a count of how many "yes" answers there are. Provide this count as {"count": <insert count here>}. 最后，提供有多少個(gè)“yes”答案的計(jì)數(shù)。將此計(jì)數(shù)提供為{"count": <插入計(jì)數(shù)>}。 |
Here's an example input where both points are satisfied:
以下是滿(mǎn)足兩個(gè)要點(diǎn)的示例輸入：
|
SYSTEM 系統(tǒng) |
<insert system?message above> <插入上面的系統(tǒng)消息> |
|
USER 用戶(hù) |
"""Neil Armstrong is famous for being the first human to set foot on the Moon. This historic event took place on July 21, 1969, during the Apollo 11 mission.""" """尼爾·阿姆斯特朗因成為第一個(gè)登上月球的人而聞名。這一歷史性事件發(fā)生在1969年7月21日，屬于阿波羅11號(hào)任務(wù)。""" |
Here's an example input where only one point is satisfied:
以下是只有一個(gè)滿(mǎn)意要點(diǎn)的示例輸入：
|
SYSTEM 系統(tǒng) |
<insert system message above> <插入上面的系統(tǒng)消息> |
|
USER 用戶(hù) |
"""Neil Armstrong made history when he stepped off the lunar module, becoming the first person to walk on the moon.""" """當(dāng)尼爾·阿姆斯特朗踏出登月艙，成為第一個(gè)踏上月球的人時(shí)，他創(chuàng)造了歷史。""" |
Here's an example input where none are satisfied:
這是一個(gè)示例輸入，其中沒(méi)有滿(mǎn)足要求的部分：
|
SYSTEM 系統(tǒng) |
<insert system message above> <插入上述系統(tǒng)消息> |
|
USER 用戶(hù) |
"""In the summer of '69, a voyage grand, Apollo 11, bold as legend's hand. Armstrong took a step, history unfurled, "One small step," he said, for a new world.""" """在'69年的夏天，一次偉大的航行， 阿波羅11，勇敢如傳說(shuō)之手。 阿姆斯特朗邁出一步，歷史展開(kāi)， 他說(shuō)：“邁出小小的一步”，為了一個(gè)新世界。 """ |
There are many possible variants on this type of model-based eval. Consider the following variation which tracks the kind of overlap between the candidate answer and the gold-standard answer, and also tracks whether the candidate answer contradicts any part of the gold-standard answer.
這種基于模型的評(píng)估可以有很多可能的變體。考慮以下變種，跟蹤候選答案與黃金標(biāo)準(zhǔn)答案之間的重疊類(lèi)型，并跟蹤候選答案是否與黃金標(biāo)準(zhǔn)答案的任何部分相矛盾。
|
SYSTEM 系統(tǒng) |
Use the following steps to respond to user inputs. Fully restate each step before proceeding. i.e. "Step 1: Reason...". 使用以下步驟來(lái)回應(yīng)用戶(hù)輸入。在繼續(xù)之前，完整重新陳述每一步，例如“步驟1：逐步推理...”。 Step 1: Reason step-by-step about whether the information in the submitted answer compared to the expert answer is either: disjoint, equal, a subset, a superset, or overlapping (i.e. some intersection but not subset/superset). 步驟1：逐步推理提交的答案與專(zhuān)家答案的信息是否是不相交、相等、子集、超集或重疊（即有一些交集但不是子集/超集）。 Step 2: Reason step-by-step about whether the submitted answer contradicts any aspect of the expert answer. 步驟2：逐步推理提交的答案是否與專(zhuān)家答案的任何方面相矛盾。 Step 3: Output a JSON object structured like: {"type_of_overlap": "disjoint" or "equal" or "subset" or "superset" or "overlapping", "contradiction": true or false} 步驟3：輸出一個(gè)結(jié)構(gòu)化的JSON對(duì)象，形式如下：{"type_of_overlap": "不相交"或"相等"或"子集"或"超集"或"重疊"，"contradiction": true或false}。 |
Here's an example input with a substandard answer which nonetheless does not contradict the expert answer:
以下是一個(gè)具有不太理想的答案但并不與專(zhuān)家答案相矛盾的示例輸入：
|
SYSTEM 系統(tǒng) |
<insert system message above> <插入上述系統(tǒng)消息> |
|
USER 用戶(hù) |
Question: """What event is Neil Armstrong most famous for and on what date did it occur? Assume UTC time.""" 問(wèn)題："尼爾·阿姆斯特朗最出名的事件是什么？它發(fā)生在哪個(gè)日期？假設(shè)使用UTC時(shí)間。" Submitted Answer: """Didn't he walk on the moon or something?""" 提交的答案："他是不是在月球上行走了什么的？" Expert Answer: """Neil Armstrong is most famous for being the first person to walk on the moon. This historic event occurred on July 21, 1969.""" 專(zhuān)家答案："尼爾·阿姆斯特朗最出名的是成為第一個(gè)在月球上行走的人。這一歷史事件發(fā)生在1969年7月21日。" |
Here's an example input with answer that directly contradicts the expert answer:
以下是一個(gè)與專(zhuān)家答案直接相矛盾的示例輸入：
|
SYSTEM 系統(tǒng) |
<insert system message above> <插入上述系統(tǒng)消息> |
|
USER 用戶(hù) |
Question: """What event is Neil Armstrong most famous for and on what date did it occur? Assume UTC time.""" 問(wèn)題："尼爾·阿姆斯特朗最出名的事件是什么？它發(fā)生在哪個(gè)日期？假設(shè)使用UTC時(shí)間。" Submitted Answer: """On the 21st of July 1969, Neil Armstrong became the second person to walk on the moon, following after Buzz Aldrin.""" 提交的答案："在1969年7月21日，尼爾·阿姆斯特朗成為第二個(gè)登上月球的人，繼巴茲·奧爾德林之后。" Expert Answer: """Neil Armstrong is most famous for being the first person to walk on the moon. This historic event occurred on July 21, 1969.""" 專(zhuān)家答案："尼爾·阿姆斯特朗最出名的是成為第一個(gè)在月球上行走的人。這一歷史事件發(fā)生在1969年7月21日。" |
Here's an example input with a correct answer that also provides a bit more detail than is necessary:
以下是一個(gè)正確的答案，但提供了比必要的細(xì)節(jié)更多的示例輸入：
|
SYSTEM 系統(tǒng) |
<insert system message above> <插入上述系統(tǒng)消息> |
|
USER 用戶(hù) |
Question: """What event is Neil Armstrong most famous for and on what date did it occur? Assume UTC time.""" 問(wèn)題："尼爾·阿姆斯特朗最出名的事件是什么？它發(fā)生在哪個(gè)日期？假設(shè)使用UTC時(shí)間。" Submitted Answer: """At approximately 02:56 UTC on July 21st 1969, Neil Armstrong became the first human to set foot on the lunar surface, marking a monumental achievement in human history.""" 提交的答案："在1969年7月21日UTC時(shí)間的02:56左右，尼爾·阿姆斯特朗成為第一個(gè)踏上月球表面的人，標(biāo)志著人類(lèi)歷史上的一個(gè)重大成就。" Expert Answer: """Neil Armstrong is most famous for being the first person to walk on the moon. This historic event occurred on July 21, 1969.""" 專(zhuān)家答案："尼爾·阿姆斯特朗最出名的是成為第一個(gè)在月球上行走的人。這一歷史事件發(fā)生在1969年7月21日。文章來(lái)源：http://www.zghlxwxcb.cn/news/detail-576913.html |
“點(diǎn)贊有美意，贊賞是鼓勵(lì)”文章來(lái)源地址http://www.zghlxwxcb.cn/news/detail-576913.html
到了這里，關(guān)于ChatGPT 最佳實(shí)踐指南之：系統(tǒng)地測(cè)試變化的文章就介紹完了。如果您還想了解更多內(nèi)容，請(qǐng)?jiān)谟疑辖撬阉鱐OY模板網(wǎng)以前的文章或繼續(xù)瀏覽下面的相關(guān)文章，希望大家以后多多支持TOY模板網(wǎng)！
