derek-thomas commited on
Commit
7bb4253
·
1 Parent(s): 335f96b

Hiding long prompts

Browse files
prompt_order_exeriment/prompt_order_exeriment.py CHANGED
@@ -96,6 +96,8 @@ We will evaluate the following prompt orders:
96
  - This is the most natural order. The model generates reasoning before the final answer, providing the most information prior to making a selection. This order leverages decoding mechanics effectively.
97
 
98
  This is our user message, we can see the question and answer choices
 
 
99
  ```
100
  \<s>[INST] Answer the Question and include your Reasoning and the Final Answer in a json like: {"Reasoning: "...", "Final Answer": "x"} where x is a letter that corresponds to the answer choice which is a letter between a and h.
101
  Question: What is satellite technology used for predicting?
@@ -109,10 +111,13 @@ This is our assistant message, you can see that we are forcing a JSON (note I ad
109
  "Final Answer": "c"
110
  }</s>
111
  ```
 
112
 
113
  2. **Scenario 2: `Q - AC - FA - R`** (Mistral and GPT3.5)
114
  - An awkward order, placing reasoning after the final answer. While it is faster, it assumes the model can "know" reasoning internally before generating it. This approach saves tokens but is a skeptical case worth testing.
115
 
 
 
116
  ```
117
  \<s>[INST] Answer the Question and include your Final Answer and the Reasoning in a json like: {"Final Answer": "x", "Reasoning: "..."} where x is a letter that corresponds to the answer choice which is a letter between a and h.
118
  Question: What is satellite technology used for predicting?
@@ -125,6 +130,7 @@ Answer Choices: (a) Seconds and minutes (b) The strength and magnitude of an ear
125
  'Reasoning': "a) Seconds and minutes: This option is incorrect because satellite technology is not used for predicting time intervals. Satellite technology is used for various purposes such as communication, navigation, and weather forecasting, but it is not used for predicting time intervals.\n\nb) The strength and magnitude of an earthquake: This option is incorrect because satellite technology is not used for predicting earthquakes. Earthquake prediction is a complex process that involves seismology and other scientific methods, but satellite technology is not one of them.\n\nd) 70-75 degrees Fahrenheit: This option is incorrect because satellite technology is not used for predicting specific temperature ranges. While satellite technology can provide temperature data, it is not used for predicting specific temperature ranges.\n\ne) Rapid changes occur: This option is too vague and does not provide enough information to determine whether it is correct or not. Satellite technology can be used to monitor changes in various environmental factors, but it is not used specifically for predicting rapid changes.\n\nf) Dead-ends and false starts: This option is incorrect because it is not related to satellite technology or any type of prediction.\n\ng) Snow, ice, and rock: This option is incorrect because it is too specific and does not cover the broad range of predictions that satellite technology can be used for. While satellite technology can be used to monitor snow, ice, and rock formations, it is not used exclusively for this purpose.\n\nh) Around 5 to 27 degrees Celsius: This option is incorrect because it is too specific and does not cover the broad range of temperature predictions that satellite technology can be used for. While satellite technology can provide temperature data, it is not used exclusively for predicting temperatures within a specific range.\n\nTherefore, the correct answer is c) what it's like outside each day, as satellite technology is commonly used for weather forecasting and predicting daily weather conditions."
126
  }</s>
127
  ```
 
128
 
129
  3. **Scenario 3: `Q - AC - FA`**
130
  - This serves as a fine-tuning control. No reasoning is provided in the output.
 
96
  - This is the most natural order. The model generates reasoning before the final answer, providing the most information prior to making a selection. This order leverages decoding mechanics effectively.
97
 
98
  This is our user message, we can see the question and answer choices
99
+ <details>
100
+ <summary>Click to show prompt!</summary>
101
  ```
102
  \<s>[INST] Answer the Question and include your Reasoning and the Final Answer in a json like: {"Reasoning: "...", "Final Answer": "x"} where x is a letter that corresponds to the answer choice which is a letter between a and h.
103
  Question: What is satellite technology used for predicting?
 
111
  "Final Answer": "c"
112
  }</s>
113
  ```
114
+ </details>
115
 
116
  2. **Scenario 2: `Q - AC - FA - R`** (Mistral and GPT3.5)
117
  - An awkward order, placing reasoning after the final answer. While it is faster, it assumes the model can "know" reasoning internally before generating it. This approach saves tokens but is a skeptical case worth testing.
118
 
119
+ <details>
120
+ <summary>Click to show prompt!</summary>
121
  ```
122
  \<s>[INST] Answer the Question and include your Final Answer and the Reasoning in a json like: {"Final Answer": "x", "Reasoning: "..."} where x is a letter that corresponds to the answer choice which is a letter between a and h.
123
  Question: What is satellite technology used for predicting?
 
130
  'Reasoning': "a) Seconds and minutes: This option is incorrect because satellite technology is not used for predicting time intervals. Satellite technology is used for various purposes such as communication, navigation, and weather forecasting, but it is not used for predicting time intervals.\n\nb) The strength and magnitude of an earthquake: This option is incorrect because satellite technology is not used for predicting earthquakes. Earthquake prediction is a complex process that involves seismology and other scientific methods, but satellite technology is not one of them.\n\nd) 70-75 degrees Fahrenheit: This option is incorrect because satellite technology is not used for predicting specific temperature ranges. While satellite technology can provide temperature data, it is not used for predicting specific temperature ranges.\n\ne) Rapid changes occur: This option is too vague and does not provide enough information to determine whether it is correct or not. Satellite technology can be used to monitor changes in various environmental factors, but it is not used specifically for predicting rapid changes.\n\nf) Dead-ends and false starts: This option is incorrect because it is not related to satellite technology or any type of prediction.\n\ng) Snow, ice, and rock: This option is incorrect because it is too specific and does not cover the broad range of predictions that satellite technology can be used for. While satellite technology can be used to monitor snow, ice, and rock formations, it is not used exclusively for this purpose.\n\nh) Around 5 to 27 degrees Celsius: This option is incorrect because it is too specific and does not cover the broad range of temperature predictions that satellite technology can be used for. While satellite technology can provide temperature data, it is not used exclusively for predicting temperatures within a specific range.\n\nTherefore, the correct answer is c) what it's like outside each day, as satellite technology is commonly used for weather forecasting and predicting daily weather conditions."
131
  }</s>
132
  ```
133
+ </details>
134
 
135
  3. **Scenario 3: `Q - AC - FA`**
136
  - This serves as a fine-tuning control. No reasoning is provided in the output.