microsoft /promptbase
All things prompt engineering
★ 5,774⑂ 333👁 56⑃ 39● 18▤ 0
More from this author
+0
Daily
5,774
stars
333
forks
56
watchers
39
prs
18
issues
0
releases
README
promptbase
promptbase is an evolving collection of resources, best practices, and example scripts for eliciting the best performance from foundation models like GPT-4. We currently host scripts demonstrating the Medprompt methodology, including examples of how we further extended this collection of prompting techniques ("Medprompt+") into non-medical domains:
| Benchmark | GPT-4 Prompt | GPT-4 Results | Gemini Ultra Results |
| ---- | ------- | ------- | ---- |
| MMLU | Medprompt+ | 90.10% | 90.04% |
| GSM8K | Zero-shot | 95.3% | 94.4% |
| MATH | Zero-shot | 68.4% | 53.2% |
| HumanEval | Zero-shot | 87.8% | 74.4% |
| BIG-Bench-Hard | Few-shot + CoT | 89.0% | 83.6% |
| DROP | Zero-shot + CoT | 83.7% | 82.4% |
| HellaSwag | 10-shot | 95.3% | 87.8% |
In the near future, promptbase will also offer further case studies and structured interviews around the scientific process we take behind prompt engineering. We'll also offer specialized deep dives into specialized tooling that accentuates the prompt engineering process. Stay tuned!
Medprompt and The Power of Prompting
"Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicine" (H. Nori, Y. T. Lee, S. Zhang, D. Carignan, R. Edgar, N. Fusi, N. King, J. Larson, Y. Li, W. Liu, R. Luo, S. M. McKinney, R. O. Ness, H. Poon, T. Qin, N. Usuyama, C. White, E. Horvitz 2023)
@article{nori2023can,
title={Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicine},
author={Nori, Harsha and Lee, Yin Tat and Zhang, Sheng and Carignan, Dean and Edgar, Richard and Fusi, Nicolo and King, Nicholas and Larson, Jonathan and Li, Yuanzhi and Liu, Weishung and others},
journal={arXiv preprint arXiv:2311.16452},
year={2023}
}
Paper link
!
In a recent study, we showed how the composition of several prompting strategies into a method that we refer to as Medprompt can efficiently steer generalist models like GPT-4 to achieve top performance, even when compared to models specifically finetuned for medicine. Medprompt composes three distinct strategies together -- including dynamic few-shot selection, self-generated chain of thought, and choice-shuffle ensembling -- to elicit specialist level performance from GPT-4. We briefly describe these strategies here:
!
- Dynamic Few Shots: Few-shot learning -- providing several examples of the task and response to a foundation model -- enables models quickly adapt to a specific domain and
learn to follow the task format. For simplicity and efficiency, the few-shot examples applied in prompting for a particular task are typically fixed; they are unchanged across test examples. This necessitates that the few-shot examples selected are broadly representative and relevant to a wide distribution of text examples. One approach to meeting these requirements is to have domain experts carefully hand-craft exemplars. Even so, this approach cannot guarantee that the curated, fixed few-shot examplesTop issues
#5Clarification needed in evaluation numbers5 comments · 2023-12-12#9Add colab2 comments · 2023-12-15#7Replace Chain of Thought with Thread of Thought 2 comments · 2023-12-14#58Detailed prompt for "Self-generated chain-of-thought template"1 comments · 2025-03-08#53Many-shot ICL1 comments · 2024-06-05#21Missing mmlu as valid_dataset even though readme encourages download1 comments · 2023-12-29#20MMLU dataset1 comments · 2023-12-25#59Proposal: add WFGY 16 Problem Map as a troubleshooting reference0 comments · 2026-02-11