Added delta for Prompt Migration Cookbook (#1942)

Author: CorwinCheung

authors.yaml

@@ -385,4 +385,9 @@ lara-openai:
 glojain:
   name: "Glory Jain"
   website: "https://www.linkedin.com/in/gloryjain/"
-  avatar: "https://media.licdn.com/dms/image/v2/C4E03AQH72n6Sm5q69Q/profile-displayphoto-shrink_400_400/profile-displayphoto-shrink_400_400/0/1557995338725?e=1756339200&v=beta&t=FGTXiCZwTZvqHCY-wd8It15EDf11Rex1oLlBKRGHNtY"
+  avatar: "https://media.licdn.com/dms/image/v2/C4E03AQH72n6Sm5q69Q/profile-displayphoto-shrink_400_400/profile-displayphoto-shrink_400_400/0/1557995338725?e=1756339200&v=beta&t=FGTXiCZwTZvqHCY-wd8It15EDf11Rex1oLlBKRGHNtY"
+
+corwin:
+  name: "Corwin Cheung"
+  website: "https://www.linkedin.com/in/corwincubes/"
+  avatar: "https://avatars.githubusercontent.com/u/85517581?v=4"

examples/Prompt_migration_guide.ipynb

@@ -698,6 +698,51 @@
     "Consistent testing and refinement ensure your prompts consistently achieve their intended results."
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "cac0dc7f",
+   "metadata": {},
+   "source": [
+    "### Current Example\n",
+    "\n",
+    "Let’s evaluate whether our current prompt migration has actually improved for the task of this judge. The original prompt, drawn from this [paper](https://arxiv.org/pdf/2306.05685), is designed to serve as a judge between two assistants’ answers. Conveniently, the paper provides a set of human-annotated ground truths, so we can measure how often the LLM judge agrees with the humans judgments.\n",
+    "\n",
+    "Thus, our metric of success will be measuring how closely the judgments generated by our migrated prompt align with human evaluations compared to the judgments generated with our baseline prompt. For context, the benchmark we’re using is a subset of MT-Bench, which features multi-turn conversations. In this example, we’re evaluating 200 conversation rows, each comparing the performance of different model pairs.\n",
+    "\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "6f50f9a0",
+   "metadata": {},
+   "source": [
+    "On our evaluation subset, a useful reference anchor is human-human agreement, since each conversation is rated by multiple annotators. Humans do not always agree with each other on which assistant answer is better, so we wouldn't expect our judge to achieve perfect agreement either. For turn 1 (without ties), humans agree with each other in 81% of cases, and for turn 2, in 76% of cases."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "7af0337b",
+   "metadata": {},
+   "source": [
+    "![Graph 3 for Model Agreement](../images/prompt_migrator_fig.png)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "800da674",
+   "metadata": {},
+   "source": [
+    "Comparing this to our models before migration, GPT-4 (as used in the paper) achieves an agreement with human judgments of 74% on turn 1 and 71% on turn 2, which is not bad, but still below the human-human ceiling. Switching to GPT-4.1 (using the same prompt) improves the agreement: 77% on turn 1 and 72% on turn 2. Finally, after migrating and tuning our prompt specifically for GPT-4.1, the agreement climbs further, reaching 80% on turn 1 and 72% on turn 2, very close to matching the level of agreement seen between human annotators."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "43ae2ba5",
+   "metadata": {},
+   "source": [
+    "Viewed all together, we can see that prompt migration and upgrading to more powerful models improve agreement on our sample task. Go ahead and try it on your prompt now!"
+   ]
+  },
   {
    "cell_type": "markdown",
    "id": "c3ed1776",
@@ -883,7 +928,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.11.8"
+   "version": "3.12.9"
   }
  },
  "nbformat": 4,

images/prompt_migrator_fig.png

@@ -12,12 +12,13 @@
   tags:
     - codex
     - automation
-    
+
 - title: Prompt Migration Guide
   path: examples/Prompt_migration_guide.ipynb
   date: 2025-06-26
   authors:
     - minh-hoque
+    - corwin
   tags:
     - prompt
     - completions