From: aidotengineer
Reasoning models represent a distinct class of artificial intelligence models that differ significantly in their operation and how they should be prompted [06:51:00].
Distinctive Characteristics
Unlike other models, reasoning models have Chain of Thought capabilities often built directly into them [02:52:00]. This means they are inherently designed to think through problems and solutions before providing an answer, breaking down complex tasks into sub-problems [02:31:00]. Observing this internal thought process can also be valuable for troubleshooting [02:41:00].
Prompting Requirements for Reasoning Models
When working with reasoning models, traditional prompting strategies may not apply, and can even hinder performance.
The Role of Examples
Research by Microsoft and DeepSeek, particularly with models like 01 and R1, indicates that adding examples, especially through few-shot prompting, can lead to worse performance [07:06:00], [07:13:00], [07:16:00]. Providing too much additional context or examples can overcomplicate things and confuse the model [07:24:00]. If few-shot prompting is deemed necessary, it’s best to start with only one or two diverse examples that cover different expected inputs [05:32:00], [08:26:00].
Encouraging Reasoning
Paradoxically, while examples can hurt, more reasoning by the model itself generally leads to better output [07:34:00]. Extended reasoning can significantly increase accuracy and overall performance [07:51:00]. When training models like DeepSeek’s R1, the length of the thought process response increased alongside improvements in accuracy and performance [07:55:00].
Prompting Best Practices
For reasoning models, consider the following considerations:
- Minimal Prompting: Often, the most effective approach is to use minimal prompting [08:09:00].
- Clear Task Description: Provide a very clear task description [08:14:00].
- Encourage Reasoning: If encountering performance issues, encourage the model to reason more [08:17:00].
- Avoid Explicit Reasoning Instructions: Since Chain of Thought is often built-in, instructing the model on how to reason can actually hurt performance [08:32:00].
Utilizing LLMs for Prompt Generation
When developing prompts for any model, including reasoning models, it’s beneficial to leverage LLMs themselves for meta-prompting [05:47:00]. This involves using an LLM to create, refine, or improve a prompt [05:50:00]. Tools like Anthropic’s and OpenAI’s playgrounds, or PromptHub’s platform, offer features that can tailor meta-prompts based on the specific model provider being used [06:06:00].