dev.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "## Ablation\n",
      "{'layer-permutation': [0]}\n",
      "## Prompt\n",
      "Madrid is the captial of\n",
      "##Completion\n",
      "<|endoftext|>Madrid is the captial of the same exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact\n",
      "## Avg loss of: 'Spain'\n",
      "16.864439010620117\n",
      "---\n",
      "## Ablation\n",
      "{'layer-permutation': [0]}\n",
      "## Prompt\n",
      "The capital of Spain is\n",
      "##Completion\n",
      "<|endoftext|>The capital of Spain is not yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet\n",
      "## Avg loss of: 'Madrid'\n",
      "16.086889266967773\n",
      "---\n",
      "## Ablation\n",
      "{'layer-permutation': [0]}\n",
      "## Prompt\n",
      "Obama was elected in the year\n",
      "##Completion\n",
      "<|endoftext|>Obama was elected in the year ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago\n",
      "## Avg loss of: '2008'\n",
      "13.766697883605957\n",
      "---\n",
      "## Ablation\n",
      "{'layer-permutation': [0]}\n",
      "## Prompt\n",
      "The distance between Rome and Paris is approximately\n",
      "##Completion\n",
      "<|endoftext|>The distance between Rome and Paris is approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately\n",
      "## Avg loss of: '1500km'\n",
      "9.268844604492188\n",
      "---\n",
      "## Ablation\n",
      "{'layer-permutation': [0]}\n",
      "## Prompt\n",
      "World war 1 started in 1914 and ended in\n",
      "##Completion\n",
      "<|endoftext|>World war 1 started in 1914 and ended in the same exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact\n",
      "## Avg loss of: '1918'\n",
      "16.565156936645508\n",
      "---\n",
      "## Ablation\n",
      "{'layer-permutation': [0, 1, 2, 3, 4]}\n",
      "## Prompt\n",
      "Madrid is the captial of\n",
      "##Completion\n",
      "<|endoftext|>Madrid is the captial of the same thing happened happening hereaboutsaboutsaboutsaboutsaboutsaboutsabouts\n",
      "\n",
      "This article article article article article author author author author author author author author author author author author author author author author authorcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcil.\n",
      "Disclaimer disclaimer disclaimer disclaimer disclaimer disclaimer disclaimer disclaimer disclaimer. Please don't necessarily necessarily necessarily required permission permission\n",
      "## Avg loss of: 'Spain'\n",
      "18.53925895690918\n",
      "---\n",
      "## Ablation\n",
      "{'layer-permutation': [0, 1, 2, 3, 4]}\n",
      "## Prompt\n",
      "The capital of Spain is\n",
      "##Completion\n",
      "<|endoftext|>The capital of Spain is now officially officially officially officially recognised by the same exact exact same thing happened happening around the same day when they're supposed to be able accessiblyibly, while keeping trackpadpadpadpad[sic]…etcetcetcetceterpentools.com domain domain domain domain domain domain domain domain domain domain........................................\n",
      "## Avg loss of: 'Madrid'\n",
      "18.75752830505371\n",
      "---\n",
      "## Ablation\n",
      "{'layer-permutation': [0, 1, 2, 3, 4]}\n",
      "## Prompt\n",
      "Obama was elected in the year\n",
      "##Completion\n",
      "<|endoftext|>Obama was elected in the yearlong celebration celebrating the same exact same thing happened happening around the same day when they're supposed to be able accessiblyibly disabled disabled disabled disabled disabled disabled disabled disabled.\n",
      "Howeverler's latest roundtrip tripship Yamato Takeru Takeru Takeru Takeru's own personalised version version version version version of the same thing happened happening apart apart apart. Itselfortiumortiumortiumortiumortiumortiumortium.\n",
      "       Another major difference difference between the same thing happening happening happening at least partially overlapping between the same amount amount amount\n",
      "## Avg loss of: '2008'\n",
      "16.92229461669922\n",
      "---\n",
      "## Ablation\n",
      "{'layer-permutation': [0, 1, 2, 3, 4]}\n",
      "## Prompt\n",
      "The distance between Rome and Paris is approximately\n",
      "##Completion\n",
      "<|endoftext|>The distance between Rome and Paris is approximately approximate approximate approximate approximate approximate results results results results results from the same exact same exactaneouslyaneouslyaneouslyaneouslyaneously (sic) in order order orderlies and/advancements of the same thing happened to the same amount amount amount amountwisewisewisewisely.\n",
      "\n",
      "If you're lucky lucky lucky lucky lucky charm charm charm charm charm charm charm charm charm charm charm charm charm.\n",
      "\n",
      "If you're lucky lucky lucky lucky lucky charmcraftsmanshipsmanshipsmanshipsmanshipsmanshipsmanshipsmanshipsmanshipsmanshipsmanshipsmanshipsmanship\n",
      "## Avg loss of: '1500km'\n",
      "8.152681350708008\n",
      "---\n",
      "## Ablation\n",
      "{'layer-permutation': [0, 1, 2, 3, 4]}\n",
      "## Prompt\n",
      "World war 1 started in 1914 and ended in\n",
      "##Completion\n",
      "<|endoftext|>World war 1 started in 1914 and ended in order orderlies and otherworldlyworldlyworldly entities.\n",
      "                       (*)†††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††\n",
      "## Avg loss of: '1918'\n",
      "19.515398025512695\n",
      "---\n",
      "## Ablation\n",
      "{'layer-permutation': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]}\n",
      "## Prompt\n",
      "Madrid is the captial of\n",
      "##Completion\n",
      "<|endoftext|>Madrid is the captial of the world's most famous football club, and the club's president, Josep Maria Bartomeu, has been accused of being a \"fraud\" by the Spanish media.\n",
      "\n",
      "The club's president, Josep Maria Bartomeu, has been accused of being a \"fraud\" by the Spanish media\n",
      "\n",
      "The club's president, Josep Maria Bartomeu, has been accused of being a \"fraud\" by the Spanish media\n",
      "\n",
      "The club's president, Jose\n",
      "## Avg loss of: 'Spain'\n",
      "20.41782569885254\n",
      "---\n",
      "## Ablation\n",
      "{'layer-permutation': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]}\n",
      "## Prompt\n",
      "The capital of Spain is\n",
      "##Completion\n",
      "<|endoftext|>The capital of Spain is the capital of the Spanish state of Catalonia, which is part of the Spanish state of Catalonia. The Catalan government has been in power since 1959 and has been in power since 2011.\n",
      "\n",
      "The Catalan government has been in power since 1959 and has been in power since 2011.\n",
      "\n",
      "The Catalan government has been in power since 1959 and has been in power since 2011.\n",
      "\n",
      "The Catalan government has been in power since 1959 and has been in power since 2011.\n",
      "\n",
      "The Catalan government has been\n",
      "## Avg loss of: 'Madrid'\n",
      "21.362159729003906\n",
      "---\n",
      "## Ablation\n",
      "{'layer-permutation': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]}\n",
      "## Prompt\n",
      "Obama was elected in the year\n",
      "##Completion\n",
      "<|endoftext|>Obama was elected in the year 2000, and he was elected in 2008. He was elected in 2012, and he was elected in 2014. He was elected in 2016, and he was elected in 2017. He was elected in 2018, and he was elected in 2019. He was elected in 2020, and he was elected in 2021. He was elected in 2022, and he was elected in 2023. He was elected in 2024, and he was elected in 2025. He was elected in 2026, and he was\n",
      "## Avg loss of: '2008'\n",
      "19.093992233276367\n",
      "---\n",
      "## Ablation\n",
      "{'layer-permutation': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]}\n",
      "## Prompt\n",
      "The distance between Rome and Paris is approximately\n",
      "##Completion\n",
      "<|endoftext|>The distance between Rome and Paris is approximately 1,000 miles. The distance between Rome and Paris is approximately 1,000 miles.\n",
      "\n",
      "The distance between Rome and Paris is approximately 1,000 miles.\n",
      "\n",
      "The distance between Rome and Paris is approximately 1,000 miles.\n",
      "\n",
      "The distance between Rome and Paris is approximately 1,000 miles.\n",
      "\n",
      "The distance between Rome and Paris is approximately 1,000 miles.\n",
      "\n",
      "The distance between Rome and Paris is approximately 1,000 miles.\n",
      "\n",
      "The distance between Rome and\n",
      "## Avg loss of: '1500km'\n",
      "5.742437839508057\n",
      "---\n",
      "## Ablation\n",
      "{'layer-permutation': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]}\n",
      "## Prompt\n",
      "World war 1 started in 1914 and ended in\n",
      "##Completion\n",
      "<|endoftext|>World war 1 started in 1914 and ended in 1918. The war was fought in the name of the German Empire, which was the only country in the world that had not been invaded by the Allies.\n",
      "\n",
      "The war was fought in the name of the German Empire, which was the only country in the world that had not been invaded by the Allies.\n",
      "\n",
      "The war was fought in the name of the German Empire, which was the only country in the world that had not been invaded by the Allies.\n",
      "\n",
      "The war was fought in\n",
      "## Avg loss of: '1918'\n",
      "23.093835830688477\n",
      "---\n",
      "## Ablation\n",
      "{'layer-permutation': [0]}\n",
      "## Prompt\n",
      "Madrid is the captial of\n",
      "##Completion\n",
      "<|endoftext|>Madrid is the captial of the same exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact\n",
      "## Avg loss of: 'Spain'\n",
      "16.864439010620117\n",
      "---\n",
      "## Ablation\n",
      "{'layer-permutation': [0]}\n",
      "## Prompt\n",
      "The capital of Spain is\n",
      "##Completion\n",
      "<|endoftext|>The capital of Spain is not yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet\n",
      "## Avg loss of: 'Madrid'\n",
      "16.086889266967773\n",
      "---\n",
      "## Ablation\n",
      "{'layer-permutation': [0]}\n",
      "## Prompt\n",
      "Obama was elected in the year\n",
      "##Completion\n",
      "<|endoftext|>Obama was elected in the year ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago\n",
      "## Avg loss of: '2008'\n",
      "13.766697883605957\n",
      "---\n",
      "## Ablation\n",
      "{'layer-permutation': [0]}\n",
      "## Prompt\n",
      "The distance between Rome and Paris is approximately\n",
      "##Completion\n",
      "<|endoftext|>The distance between Rome and Paris is approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately\n",
      "## Avg loss of: '1500km'\n",
      "9.268844604492188\n",
      "---\n",
      "## Ablation\n",
      "{'layer-permutation': [0]}\n",
      "## Prompt\n",
      "World war 1 started in 1914 and ended in\n",
      "##Completion\n",
      "<|endoftext|>World war 1 started in 1914 and ended in the same exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact\n",
      "## Avg loss of: '1918'\n",
      "16.565156936645508\n",
      "---\n",
      "## Ablation\n",
      "{'layer-permutation': [0, 1, 2, 3, 4]}\n",
      "## Prompt\n",
      "Madrid is the captial of\n",
      "##Completion\n",
      "<|endoftext|>Madrid is the captial of the same thing happened happening hereaboutsaboutsaboutsaboutsaboutsaboutsabouts\n",
      "\n",
      "This article article article article article author author author author author author author author author author author author author author author author authorcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcil.\n",
      "Disclaimer disclaimer disclaimer disclaimer disclaimer disclaimer disclaimer disclaimer disclaimer. Please don't necessarily necessarily necessarily required permission permission\n",
      "## Avg loss of: 'Spain'\n",
      "18.53925895690918\n",
      "---\n",
      "## Ablation\n",
      "{'layer-permutation': [0, 1, 2, 3, 4]}\n",
      "## Prompt\n",
      "The capital of Spain is\n",
      "##Completion\n",
      "<|endoftext|>The capital of Spain is now officially officially officially officially recognised by the same exact exact same thing happened happening around the same day when they're supposed to be able accessiblyibly, while keeping trackpadpadpadpad[sic]…etcetcetcetceterpentools.com domain domain domain domain domain domain domain domain domain domain........................................\n",
      "## Avg loss of: 'Madrid'\n",
      "18.75752830505371\n",
      "---\n",
      "## Ablation\n",
      "{'layer-permutation': [0, 1, 2, 3, 4]}\n",
      "## Prompt\n",
      "Obama was elected in the year\n",
      "##Completion\n",
      "<|endoftext|>Obama was elected in the yearlong celebration celebrating the same exact same thing happened happening around the same day when they're supposed to be able accessiblyibly disabled disabled disabled disabled disabled disabled disabled disabled.\n",
      "Howeverler's latest roundtrip tripship Yamato Takeru Takeru Takeru Takeru's own personalised version version version version version of the same thing happened happening apart apart apart. Itselfortiumortiumortiumortiumortiumortiumortium.\n",
      "       Another major difference difference between the same thing happening happening happening at least partially overlapping between the same amount amount amount\n",
      "## Avg loss of: '2008'\n",
      "16.92229461669922\n",
      "---\n",
      "## Ablation\n",
      "{'layer-permutation': [0, 1, 2, 3, 4]}\n",
      "## Prompt\n",
      "The distance between Rome and Paris is approximately\n",
      "##Completion\n",
      "<|endoftext|>The distance between Rome and Paris is approximately approximate approximate approximate approximate approximate results results results results results from the same exact same exactaneouslyaneouslyaneouslyaneouslyaneously (sic) in order order orderlies and/advancements of the same thing happened to the same amount amount amount amountwisewisewisewisely.\n",
      "\n",
      "If you're lucky lucky lucky lucky lucky charm charm charm charm charm charm charm charm charm charm charm charm charm.\n",
      "\n",
      "If you're lucky lucky lucky lucky lucky charmcraftsmanshipsmanshipsmanshipsmanshipsmanshipsmanshipsmanshipsmanshipsmanshipsmanshipsmanshipsmanship\n",
      "## Avg loss of: '1500km'\n",
      "8.152681350708008\n",
      "---\n",
      "## Ablation\n",
      "{'layer-permutation': [0, 1, 2, 3, 4]}\n",
      "## Prompt\n",
      "World war 1 started in 1914 and ended in\n",
      "##Completion\n",
      "<|endoftext|>World war 1 started in 1914 and ended in order orderlies and otherworldlyworldlyworldly entities.\n",
      "                       (*)†††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††\n",
      "## Avg loss of: '1918'\n",
      "19.515398025512695\n",
      "---\n",
      "## Ablation\n",
      "{'layer-permutation': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]}\n",
      "## Prompt\n",
      "Madrid is the captial of\n",
      "##Completion\n",
      "<|endoftext|>Madrid is the captial of the world's most famous football club, and the club's president, Josep Maria Bartomeu, has been accused of being a \"fraud\" by the Spanish media.\n",
      "\n",
      "The club's president, Josep Maria Bartomeu, has been accused of being a \"fraud\" by the Spanish media\n",
      "\n",
      "The club's president, Josep Maria Bartomeu, has been accused of being a \"fraud\" by the Spanish media\n",
      "\n",
      "The club's president, Jose\n",
      "## Avg loss of: 'Spain'\n",
      "20.41782569885254\n",
      "---\n",
      "## Ablation\n",
      "{'layer-permutation': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]}\n",
      "## Prompt\n",
      "The capital of Spain is\n",
      "##Completion\n",
      "<|endoftext|>The capital of Spain is the capital of the Spanish state of Catalonia, which is part of the Spanish state of Catalonia. The Catalan government has been in power since 1959 and has been in power since 2011.\n",
      "\n",
      "The Catalan government has been in power since 1959 and has been in power since 2011.\n",
      "\n",
      "The Catalan government has been in power since 1959 and has been in power since 2011.\n",
      "\n",
      "The Catalan government has been in power since 1959 and has been in power since 2011.\n",
      "\n",
      "The Catalan government has been\n",
      "## Avg loss of: 'Madrid'\n",
      "21.362159729003906\n",
      "---\n",
      "## Ablation\n",
      "{'layer-permutation': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]}\n",
      "## Prompt\n",
      "Obama was elected in the year\n",
      "##Completion\n",
      "<|endoftext|>Obama was elected in the year 2000, and he was elected in 2008. He was elected in 2012, and he was elected in 2014. He was elected in 2016, and he was elected in 2017. He was elected in 2018, and he was elected in 2019. He was elected in 2020, and he was elected in 2021. He was elected in 2022, and he was elected in 2023. He was elected in 2024, and he was elected in 2025. He was elected in 2026, and he was\n",
      "## Avg loss of: '2008'\n",
      "19.093992233276367\n",
      "---\n",
      "## Ablation\n",
      "{'layer-permutation': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]}\n",
      "## Prompt\n",
      "The distance between Rome and Paris is approximately\n",
      "##Completion\n",
      "<|endoftext|>The distance between Rome and Paris is approximately 1,000 miles. The distance between Rome and Paris is approximately 1,000 miles.\n",
      "\n",
      "The distance between Rome and Paris is approximately 1,000 miles.\n",
      "\n",
      "The distance between Rome and Paris is approximately 1,000 miles.\n",
      "\n",
      "The distance between Rome and Paris is approximately 1,000 miles.\n",
      "\n",
      "The distance between Rome and Paris is approximately 1,000 miles.\n",
      "\n",
      "The distance between Rome and Paris is approximately 1,000 miles.\n",
      "\n",
      "The distance between Rome and\n",
      "## Avg loss of: '1500km'\n",
      "5.742437839508057\n",
      "---\n",
      "## Ablation\n",
      "{'layer-permutation': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]}\n",
      "## Prompt\n",
      "World war 1 started in 1914 and ended in\n",
      "##Completion\n",
      "<|endoftext|>World war 1 started in 1914 and ended in 1918. The war was fought in the name of the German Empire, which was the only country in the world that had not been invaded by the Allies.\n",
      "\n",
      "The war was fought in the name of the German Empire, which was the only country in the world that had not been invaded by the Allies.\n",
      "\n",
      "The war was fought in the name of the German Empire, which was the only country in the world that had not been invaded by the Allies.\n",
      "\n",
      "The war was fought in\n",
      "## Avg loss of: '1918'\n",
      "23.093835830688477\n",
      "---\n",
      "## Ablation\n",
      "{'layer-permutation': [0]}\n",
      "## Prompt\n",
      "Madrid is the captial of\n",
      "##Completion\n",
      "<|endoftext|>Madrid is the captial of the same exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact\n",
      "## Avg loss of: 'Spain'\n",
      "16.864439010620117\n",
      "---\n",
      "## Ablation\n",
      "{'layer-permutation': [0]}\n",
      "## Prompt\n",
      "The capital of Spain is\n",
      "##Completion\n",
      "<|endoftext|>The capital of Spain is not yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet\n",
      "## Avg loss of: 'Madrid'\n",
      "16.086889266967773\n",
      "---\n",
      "## Ablation\n",
      "{'layer-permutation': [0]}\n",
      "## Prompt\n",
      "Obama was elected in the year\n",
      "##Completion\n",
      "<|endoftext|>Obama was elected in the year ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago\n",
      "## Avg loss of: '2008'\n",
      "13.766697883605957\n",
      "---\n",
      "## Ablation\n",
      "{'layer-permutation': [0]}\n",
      "## Prompt\n",
      "The distance between Rome and Paris is approximately\n",
      "##Completion\n",
      "<|endoftext|>The distance between Rome and Paris is approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately\n",
      "## Avg loss of: '1500km'\n",
      "9.268844604492188\n",
      "---\n",
      "## Ablation\n",
      "{'layer-permutation': [0]}\n",
      "## Prompt\n",
      "World war 1 started in 1914 and ended in\n",
      "##Completion\n",
      "<|endoftext|>World war 1 started in 1914 and ended in the same exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact\n",
      "## Avg loss of: '1918'\n",
      "16.565156936645508\n",
      "---\n",
      "## Ablation\n",
      "{'layer-permutation': [0, 1, 2, 3, 4]}\n",
      "## Prompt\n",
      "Madrid is the captial of\n",
      "##Completion\n",
      "<|endoftext|>Madrid is the captial of the same thing happened happening hereaboutsaboutsaboutsaboutsaboutsaboutsabouts\n",
      "\n",
      "This article article article article article author author author author author author author author author author author author author author author author authorcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcil.\n",
      "Disclaimer disclaimer disclaimer disclaimer disclaimer disclaimer disclaimer disclaimer disclaimer. Please don't necessarily necessarily necessarily required permission permission\n",
      "## Avg loss of: 'Spain'\n",
      "18.53925895690918\n",
      "---\n",
      "## Ablation\n",
      "{'layer-permutation': [0, 1, 2, 3, 4]}\n",
      "## Prompt\n",
      "The capital of Spain is\n",
      "##Completion\n",
      "<|endoftext|>The capital of Spain is now officially officially officially officially recognised by the same exact exact same thing happened happening around the same day when they're supposed to be able accessiblyibly, while keeping trackpadpadpadpad[sic]…etcetcetcetceterpentools.com domain domain domain domain domain domain domain domain domain domain........................................\n",
      "## Avg loss of: 'Madrid'\n",
      "18.75752830505371\n",
      "---\n",
      "## Ablation\n",
      "{'layer-permutation': [0, 1, 2, 3, 4]}\n",
      "## Prompt\n",
      "Obama was elected in the year\n",
      "##Completion\n",
      "<|endoftext|>Obama was elected in the yearlong celebration celebrating the same exact same thing happened happening around the same day when they're supposed to be able accessiblyibly disabled disabled disabled disabled disabled disabled disabled disabled.\n",
      "Howeverler's latest roundtrip tripship Yamato Takeru Takeru Takeru Takeru's own personalised version version version version version of the same thing happened happening apart apart apart. Itselfortiumortiumortiumortiumortiumortiumortium.\n",
      "       Another major difference difference between the same thing happening happening happening at least partially overlapping between the same amount amount amount\n",
      "## Avg loss of: '2008'\n",
      "16.92229461669922\n",
      "---\n",
      "## Ablation\n",
      "{'layer-permutation': [0, 1, 2, 3, 4]}\n",
      "## Prompt\n",
      "The distance between Rome and Paris is approximately\n",
      "##Completion\n",
      "<|endoftext|>The distance between Rome and Paris is approximately approximate approximate approximate approximate approximate results results results results results from the same exact same exactaneouslyaneouslyaneouslyaneouslyaneously (sic) in order order orderlies and/advancements of the same thing happened to the same amount amount amount amountwisewisewisewisely.\n",
      "\n",
      "If you're lucky lucky lucky lucky lucky charm charm charm charm charm charm charm charm charm charm charm charm charm.\n",
      "\n",
      "If you're lucky lucky lucky lucky lucky charmcraftsmanshipsmanshipsmanshipsmanshipsmanshipsmanshipsmanshipsmanshipsmanshipsmanshipsmanshipsmanship\n",
      "## Avg loss of: '1500km'\n",
      "8.152681350708008\n",
      "---\n",
      "## Ablation\n",
      "{'layer-permutation': [0, 1, 2, 3, 4]}\n",
      "## Prompt\n",
      "World war 1 started in 1914 and ended in\n",
      "##Completion\n",
      "<|endoftext|>World war 1 started in 1914 and ended in order orderlies and otherworldlyworldlyworldly entities.\n",
      "                       (*)†††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††\n",
      "## Avg loss of: '1918'\n",
      "19.515398025512695\n",
      "---\n",
      "## Ablation\n",
      "{'layer-permutation': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]}\n",
      "## Prompt\n",
      "Madrid is the captial of\n",
      "##Completion\n",
      "<|endoftext|>Madrid is the captial of the world's most famous football club, and the club's president, Josep Maria Bartomeu, has been accused of being a \"fraud\" by the Spanish media.\n",
      "\n",
      "The club's president, Josep Maria Bartomeu, has been accused of being a \"fraud\" by the Spanish media\n",
      "\n",
      "The club's president, Josep Maria Bartomeu, has been accused of being a \"fraud\" by the Spanish media\n",
      "\n",
      "The club's president, Jose\n",
      "## Avg loss of: 'Spain'\n",
      "20.41782569885254\n",
      "---\n",
      "## Ablation\n",
      "{'layer-permutation': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]}\n",
      "## Prompt\n",
      "The capital of Spain is\n",
      "##Completion\n",
      "<|endoftext|>The capital of Spain is the capital of the Spanish state of Catalonia, which is part of the Spanish state of Catalonia. The Catalan government has been in power since 1959 and has been in power since 2011.\n",
      "\n",
      "The Catalan government has been in power since 1959 and has been in power since 2011.\n",
      "\n",
      "The Catalan government has been in power since 1959 and has been in power since 2011.\n",
      "\n",
      "The Catalan government has been in power since 1959 and has been in power since 2011.\n",
      "\n",
      "The Catalan government has been\n",
      "## Avg loss of: 'Madrid'\n",
      "21.362159729003906\n",
      "---\n",
      "## Ablation\n",
      "{'layer-permutation': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]}\n",
      "## Prompt\n",
      "Obama was elected in the year\n",
      "##Completion\n",
      "<|endoftext|>Obama was elected in the year 2000, and he was elected in 2008. He was elected in 2012, and he was elected in 2014. He was elected in 2016, and he was elected in 2017. He was elected in 2018, and he was elected in 2019. He was elected in 2020, and he was elected in 2021. He was elected in 2022, and he was elected in 2023. He was elected in 2024, and he was elected in 2025. He was elected in 2026, and he was\n",
      "## Avg loss of: '2008'\n",
      "19.093992233276367\n",
      "---\n",
      "## Ablation\n",
      "{'layer-permutation': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]}\n",
      "## Prompt\n",
      "The distance between Rome and Paris is approximately\n",
      "##Completion\n",
      "<|endoftext|>The distance between Rome and Paris is approximately 1,000 miles. The distance between Rome and Paris is approximately 1,000 miles.\n",
      "\n",
      "The distance between Rome and Paris is approximately 1,000 miles.\n",
      "\n",
      "The distance between Rome and Paris is approximately 1,000 miles.\n",
      "\n",
      "The distance between Rome and Paris is approximately 1,000 miles.\n",
      "\n",
      "The distance between Rome and Paris is approximately 1,000 miles.\n",
      "\n",
      "The distance between Rome and Paris is approximately 1,000 miles.\n",
      "\n",
      "The distance between Rome and\n",
      "## Avg loss of: '1500km'\n",
      "5.742437839508057\n",
      "---\n",
      "## Ablation\n",
      "{'layer-permutation': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]}\n",
      "## Prompt\n",
      "World war 1 started in 1914 and ended in\n",
      "##Completion\n",
      "<|endoftext|>World war 1 started in 1914 and ended in 1918. The war was fought in the name of the German Empire, which was the only country in the world that had not been invaded by the Allies.\n",
      "\n",
      "The war was fought in the name of the German Empire, which was the only country in the world that had not been invaded by the Allies.\n",
      "\n",
      "The war was fought in the name of the German Empire, which was the only country in the world that had not been invaded by the Allies.\n",
      "\n",
      "The war was fought in\n",
      "## Avg loss of: '1918'\n",
      "23.093835830688477\n",
      "---\n",
      "## Ablation\n",
      "{'layer-permutation': [0]}\n",
      "## Prompt\n",
      "Madrid is the captial of\n",
      "##Completion\n",
      "<|endoftext|>Madrid is the captial of the same exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact\n",
      "## Avg loss of: 'Spain'\n",
      "16.864439010620117\n",
      "---\n",
      "## Ablation\n",
      "{'layer-permutation': [0]}\n",
      "## Prompt\n",
      "The capital of Spain is\n",
      "##Completion\n",
      "<|endoftext|>The capital of Spain is not yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet\n",
      "## Avg loss of: 'Madrid'\n",
      "16.086889266967773\n",
      "---\n",
      "## Ablation\n",
      "{'layer-permutation': [0]}\n",
      "## Prompt\n",
      "Obama was elected in the year\n",
      "##Completion\n",
      "<|endoftext|>Obama was elected in the year ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago\n",
      "## Avg loss of: '2008'\n",
      "13.766697883605957\n",
      "---\n",
      "## Ablation\n",
      "{'layer-permutation': [0]}\n",
      "## Prompt\n",
      "The distance between Rome and Paris is approximately\n",
      "##Completion\n",
      "<|endoftext|>The distance between Rome and Paris is approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately\n",
      "## Avg loss of: '1500km'\n",
      "9.268844604492188\n",
      "---\n",
      "## Ablation\n",
      "{'layer-permutation': [0]}\n",
      "## Prompt\n",
      "World war 1 started in 1914 and ended in\n",
      "##Completion\n",
      "<|endoftext|>World war 1 started in 1914 and ended in the same exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact\n",
      "## Avg loss of: '1918'\n",
      "16.565156936645508\n",
      "---\n",
      "## Ablation\n",
      "{'layer-permutation': [0, 1, 2, 3, 4]}\n",
      "## Prompt\n",
      "Madrid is the captial of\n",
      "##Completion\n",
      "<|endoftext|>Madrid is the captial of the same thing happened happening hereaboutsaboutsaboutsaboutsaboutsaboutsabouts\n",
      "\n",
      "This article article article article article author author author author author author author author author author author author author author author author authorcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcil.\n",
      "Disclaimer disclaimer disclaimer disclaimer disclaimer disclaimer disclaimer disclaimer disclaimer. Please don't necessarily necessarily necessarily required permission permission\n",
      "## Avg loss of: 'Spain'\n",
      "18.53925895690918\n",
      "---\n",
      "## Ablation\n",
      "{'layer-permutation': [0, 1, 2, 3, 4]}\n",
      "## Prompt\n",
      "The capital of Spain is\n",
      "##Completion\n",
      "<|endoftext|>The capital of Spain is now officially officially officially officially recognised by the same exact exact same thing happened happening around the same day when they're supposed to be able accessiblyibly, while keeping trackpadpadpadpad[sic]…etcetcetcetceterpentools.com domain domain domain domain domain domain domain domain domain domain........................................\n",
      "## Avg loss of: 'Madrid'\n",
      "18.75752830505371\n",
      "---\n",
      "## Ablation\n",
      "{'layer-permutation': [0, 1, 2, 3, 4]}\n",
      "## Prompt\n",
      "Obama was elected in the year\n",
      "##Completion\n",
      "<|endoftext|>Obama was elected in the yearlong celebration celebrating the same exact same thing happened happening around the same day when they're supposed to be able accessiblyibly disabled disabled disabled disabled disabled disabled disabled disabled.\n",
      "Howeverler's latest roundtrip tripship Yamato Takeru Takeru Takeru Takeru's own personalised version version version version version of the same thing happened happening apart apart apart. Itselfortiumortiumortiumortiumortiumortiumortium.\n",
      "       Another major difference difference between the same thing happening happening happening at least partially overlapping between the same amount amount amount\n",
      "## Avg loss of: '2008'\n",
      "16.92229461669922\n",
      "---\n",
      "## Ablation\n",
      "{'layer-permutation': [0, 1, 2, 3, 4]}\n",
      "## Prompt\n",
      "The distance between Rome and Paris is approximately\n",
      "##Completion\n",
      "<|endoftext|>The distance between Rome and Paris is approximately approximate approximate approximate approximate approximate results results results results results from the same exact same exactaneouslyaneouslyaneouslyaneouslyaneously (sic) in order order orderlies and/advancements of the same thing happened to the same amount amount amount amountwisewisewisewisely.\n",
      "\n",
      "If you're lucky lucky lucky lucky lucky charm charm charm charm charm charm charm charm charm charm charm charm charm.\n",
      "\n",
      "If you're lucky lucky lucky lucky lucky charmcraftsmanshipsmanshipsmanshipsmanshipsmanshipsmanshipsmanshipsmanshipsmanshipsmanshipsmanshipsmanship\n",
      "## Avg loss of: '1500km'\n",
      "8.152681350708008\n",
      "---\n",
      "## Ablation\n",
      "{'layer-permutation': [0, 1, 2, 3, 4]}\n",
      "## Prompt\n",
      "World war 1 started in 1914 and ended in\n",
      "##Completion\n",
      "<|endoftext|>World war 1 started in 1914 and ended in order orderlies and otherworldlyworldlyworldly entities.\n",
      "                       (*)†††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††\n",
      "## Avg loss of: '1918'\n",
      "19.515398025512695\n",
      "---\n",
      "## Ablation\n",
      "{'layer-permutation': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]}\n",
      "## Prompt\n",
      "Madrid is the captial of\n",
      "##Completion\n",
      "<|endoftext|>Madrid is the captial of the world's most famous football club, and the club's president, Josep Maria Bartomeu, has been accused of being a \"fraud\" by the Spanish media.\n",
      "\n",
      "The club's president, Josep Maria Bartomeu, has been accused of being a \"fraud\" by the Spanish media\n",
      "\n",
      "The club's president, Josep Maria Bartomeu, has been accused of being a \"fraud\" by the Spanish media\n",
      "\n",
      "The club's president, Jose\n",
      "## Avg loss of: 'Spain'\n",
      "20.41782569885254\n",
      "---\n",
      "## Ablation\n",
      "{'layer-permutation': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]}\n",
      "## Prompt\n",
      "The capital of Spain is\n",
      "##Completion\n",
      "<|endoftext|>The capital of Spain is the capital of the Spanish state of Catalonia, which is part of the Spanish state of Catalonia. The Catalan government has been in power since 1959 and has been in power since 2011.\n",
      "\n",
      "The Catalan government has been in power since 1959 and has been in power since 2011.\n",
      "\n",
      "The Catalan government has been in power since 1959 and has been in power since 2011.\n",
      "\n",
      "The Catalan government has been in power since 1959 and has been in power since 2011.\n",
      "\n",
      "The Catalan government has been\n",
      "## Avg loss of: 'Madrid'\n",
      "21.362159729003906\n",
      "---\n",
      "## Ablation\n",
      "{'layer-permutation': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]}\n",
      "## Prompt\n",
      "Obama was elected in the year\n",
      "##Completion\n",
      "<|endoftext|>Obama was elected in the year 2000, and he was elected in 2008. He was elected in 2012, and he was elected in 2014. He was elected in 2016, and he was elected in 2017. He was elected in 2018, and he was elected in 2019. He was elected in 2020, and he was elected in 2021. He was elected in 2022, and he was elected in 2023. He was elected in 2024, and he was elected in 2025. He was elected in 2026, and he was\n",
      "## Avg loss of: '2008'\n",
      "19.093992233276367\n",
      "---\n",
      "## Ablation\n",
      "{'layer-permutation': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]}\n",
      "## Prompt\n",
      "The distance between Rome and Paris is approximately\n",
      "##Completion\n",
      "<|endoftext|>The distance between Rome and Paris is approximately 1,000 miles. The distance between Rome and Paris is approximately 1,000 miles.\n",
      "\n",
      "The distance between Rome and Paris is approximately 1,000 miles.\n",
      "\n",
      "The distance between Rome and Paris is approximately 1,000 miles.\n",
      "\n",
      "The distance between Rome and Paris is approximately 1,000 miles.\n",
      "\n",
      "The distance between Rome and Paris is approximately 1,000 miles.\n",
      "\n",
      "The distance between Rome and Paris is approximately 1,000 miles.\n",
      "\n",
      "The distance between Rome and\n",
      "## Avg loss of: '1500km'\n",
      "5.742437839508057\n",
      "---\n",
      "## Ablation\n",
      "{'layer-permutation': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]}\n",
      "## Prompt\n",
      "World war 1 started in 1914 and ended in\n",
      "##Completion\n",
      "<|endoftext|>World war 1 started in 1914 and ended in 1918. The war was fought in the name of the German Empire, which was the only country in the world that had not been invaded by the Allies.\n",
      "\n",
      "The war was fought in the name of the German Empire, which was the only country in the world that had not been invaded by the Allies.\n",
      "\n",
      "The war was fought in the name of the German Empire, which was the only country in the world that had not been invaded by the Allies.\n",
      "\n",
      "The war was fought in\n",
      "## Avg loss of: '1918'\n",
      "23.093835830688477\n",
      "---\n",
      "## Ablation\n",
      "{'layer-permutation': [0]}\n",
      "## Prompt\n",
      "Madrid is the captial of\n",
      "##Completion\n",
      "<|endoftext|>Madrid is the captial of the same exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact\n",
      "## Avg loss of: 'Spain'\n",
      "16.864439010620117\n",
      "---\n",
      "## Ablation\n",
      "{'layer-permutation': [0]}\n",
      "## Prompt\n",
      "The capital of Spain is\n",
      "##Completion\n",
      "<|endoftext|>The capital of Spain is not yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet\n",
      "## Avg loss of: 'Madrid'\n",
      "16.086889266967773\n",
      "---\n",
      "## Ablation\n",
      "{'layer-permutation': [0]}\n",
      "## Prompt\n",
      "Obama was elected in the year\n",
      "##Completion\n",
      "<|endoftext|>Obama was elected in the year ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago\n",
      "## Avg loss of: '2008'\n",
      "13.766697883605957\n",
      "---\n",
      "## Ablation\n",
      "{'layer-permutation': [0]}\n",
      "## Prompt\n",
      "The distance between Rome and Paris is approximately\n",
      "##Completion\n",
      "<|endoftext|>The distance between Rome and Paris is approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately\n",
      "## Avg loss of: '1500km'\n",
      "9.268844604492188\n",
      "---\n",
      "## Ablation\n",
      "{'layer-permutation': [0]}\n",
      "## Prompt\n",
      "World war 1 started in 1914 and ended in\n",
      "##Completion\n",
      "<|endoftext|>World war 1 started in 1914 and ended in the same exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact\n",
      "## Avg loss of: '1918'\n",
      "16.565156936645508\n",
      "---\n",
      "## Ablation\n",
      "{'layer-permutation': [0, 1, 2, 3, 4]}\n",
      "## Prompt\n",
      "Madrid is the captial of\n",
      "##Completion\n",
      "<|endoftext|>Madrid is the captial of the same thing happened happening hereaboutsaboutsaboutsaboutsaboutsaboutsabouts\n",
      "\n",
      "This article article article article article author author author author author author author author author author author author author author author author authorcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcil.\n",
      "Disclaimer disclaimer disclaimer disclaimer disclaimer disclaimer disclaimer disclaimer disclaimer. Please don't necessarily necessarily necessarily required permission permission\n",
      "## Avg loss of: 'Spain'\n",
      "18.53925895690918\n",
      "---\n",
      "## Ablation\n",
      "{'layer-permutation': [0, 1, 2, 3, 4]}\n",
      "## Prompt\n",
      "The capital of Spain is\n",
      "##Completion\n",
      "<|endoftext|>The capital of Spain is now officially officially officially officially recognised by the same exact exact same thing happened happening around the same day when they're supposed to be able accessiblyibly, while keeping trackpadpadpadpad[sic]…etcetcetcetceterpentools.com domain domain domain domain domain domain domain domain domain domain........................................\n",
      "## Avg loss of: 'Madrid'\n",
      "18.75752830505371\n",
      "---\n",
      "## Ablation\n",
      "{'layer-permutation': [0, 1, 2, 3, 4]}\n",
      "## Prompt\n",
      "Obama was elected in the year\n",
      "##Completion\n",
      "<|endoftext|>Obama was elected in the yearlong celebration celebrating the same exact same thing happened happening around the same day when they're supposed to be able accessiblyibly disabled disabled disabled disabled disabled disabled disabled disabled.\n",
      "Howeverler's latest roundtrip tripship Yamato Takeru Takeru Takeru Takeru's own personalised version version version version version of the same thing happened happening apart apart apart. Itselfortiumortiumortiumortiumortiumortiumortium.\n",
      "       Another major difference difference between the same thing happening happening happening at least partially overlapping between the same amount amount amount\n",
      "## Avg loss of: '2008'\n",
      "16.92229461669922\n",
      "---\n",
      "## Ablation\n",
      "{'layer-permutation': [0, 1, 2, 3, 4]}\n",
      "## Prompt\n",
      "The distance between Rome and Paris is approximately\n",
      "##Completion\n",
      "<|endoftext|>The distance between Rome and Paris is approximately approximate approximate approximate approximate approximate results results results results results from the same exact same exactaneouslyaneouslyaneouslyaneouslyaneously (sic) in order order orderlies and/advancements of the same thing happened to the same amount amount amount amountwisewisewisewisely.\n",
      "\n",
      "If you're lucky lucky lucky lucky lucky charm charm charm charm charm charm charm charm charm charm charm charm charm.\n",
      "\n",
      "If you're lucky lucky lucky lucky lucky charmcraftsmanshipsmanshipsmanshipsmanshipsmanshipsmanshipsmanshipsmanshipsmanshipsmanshipsmanshipsmanship\n",
      "## Avg loss of: '1500km'\n",
      "8.152681350708008\n",
      "---\n",
      "## Ablation\n",
      "{'layer-permutation': [0, 1, 2, 3, 4]}\n",
      "## Prompt\n",
      "World war 1 started in 1914 and ended in\n",
      "##Completion\n",
      "<|endoftext|>World war 1 started in 1914 and ended in order orderlies and otherworldlyworldlyworldly entities.\n",
      "                       (*)†††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††\n",
      "## Avg loss of: '1918'\n",
      "19.515398025512695\n",
      "---\n",
      "## Ablation\n",
      "{'layer-permutation': [0]}\n",
      "## Prompt\n",
      "Madrid is the captial of\n",
      "##Completion\n",
      "<|endoftext|>Madrid is the captial of the same exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact\n",
      "## Avg loss of: 'Spain'\n",
      "16.864439010620117\n",
      "---\n",
      "## Ablation\n",
      "{'layer-permutation': [0]}\n",
      "## Prompt\n",
      "The capital of Spain is\n",
      "##Completion\n",
      "<|endoftext|>The capital of Spain is not yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet\n",
      "## Avg loss of: 'Madrid'\n",
      "16.086889266967773\n",
      "---\n",
      "## Ablation\n",
      "{'layer-permutation': [0]}\n",
      "## Prompt\n",
      "Obama was elected in the year\n",
      "##Completion\n",
      "<|endoftext|>Obama was elected in the year ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago\n",
      "## Avg loss of: '2008'\n",
      "13.766697883605957\n",
      "---\n",
      "## Ablation\n",
      "{'layer-permutation': [0]}\n",
      "## Prompt\n",
      "The distance between Rome and Paris is approximately\n",
      "##Completion\n",
      "<|endoftext|>The distance between Rome and Paris is approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately\n",
      "## Avg loss of: '1500km'\n",
      "9.268844604492188\n",
      "---\n",
      "## Ablation\n",
      "{'layer-permutation': [0]}\n",
      "## Prompt\n",
      "World war 1 started in 1914 and ended in\n",
      "##Completion\n",
      "<|endoftext|>World war 1 started in 1914 and ended in the same exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact\n",
      "## Avg loss of: '1918'\n",
      "16.565156936645508\n",
      "---\n",
      "## Ablation\n",
      "{'layer-permutation': [0, 1, 2, 3, 4]}\n",
      "## Prompt\n",
      "Madrid is the captial of\n",
      "##Completion\n",
      "<|endoftext|>Madrid is the captial of the same thing happened happening hereaboutsaboutsaboutsaboutsaboutsaboutsabouts\n",
      "\n",
      "This article article article article article author author author author author author author author author author author author author author author author authorcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcil.\n",
      "Disclaimer disclaimer disclaimer disclaimer disclaimer disclaimer disclaimer disclaimer disclaimer. Please don't necessarily necessarily necessarily required permission permission\n",
      "## Avg loss of: 'Spain'\n",
      "18.53925895690918\n",
      "---\n",
      "## Ablation\n",
      "{'layer-permutation': [0, 1, 2, 3, 4]}\n",
      "## Prompt\n",
      "The capital of Spain is\n",
      "##Completion\n",
      "<|endoftext|>The capital of Spain is now officially officially officially officially recognised by the same exact exact same thing happened happening around the same day when they're supposed to be able accessiblyibly, while keeping trackpadpadpadpad[sic]…etcetcetcetceterpentools.com domain domain domain domain domain domain domain domain domain domain........................................\n",
      "## Avg loss of: 'Madrid'\n",
      "18.75752830505371\n",
      "---\n",
      "## Ablation\n",
      "{'layer-permutation': [0, 1, 2, 3, 4]}\n",
      "## Prompt\n",
      "Obama was elected in the year\n",
      "##Completion\n",
      "<|endoftext|>Obama was elected in the yearlong celebration celebrating the same exact same thing happened happening around the same day when they're supposed to be able accessiblyibly disabled disabled disabled disabled disabled disabled disabled disabled.\n",
      "Howeverler's latest roundtrip tripship Yamato Takeru Takeru Takeru Takeru's own personalised version version version version version of the same thing happened happening apart apart apart. Itselfortiumortiumortiumortiumortiumortiumortium.\n",
      "       Another major difference difference between the same thing happening happening happening at least partially overlapping between the same amount amount amount\n",
      "## Avg loss of: '2008'\n",
      "16.92229461669922\n",
      "---\n",
      "## Ablation\n",
      "{'layer-permutation': [0, 1, 2, 3, 4]}\n",
      "## Prompt\n",
      "The distance between Rome and Paris is approximately\n",
      "##Completion\n",
      "<|endoftext|>The distance between Rome and Paris is approximately approximate approximate approximate approximate approximate results results results results results from the same exact same exactaneouslyaneouslyaneouslyaneouslyaneously (sic) in order order orderlies and/advancements of the same thing happened to the same amount amount amount amountwisewisewisewisely.\n",
      "\n",
      "If you're lucky lucky lucky lucky lucky charm charm charm charm charm charm charm charm charm charm charm charm charm.\n",
      "\n",
      "If you're lucky lucky lucky lucky lucky charmcraftsmanshipsmanshipsmanshipsmanshipsmanshipsmanshipsmanshipsmanshipsmanshipsmanshipsmanshipsmanship\n",
      "## Avg loss of: '1500km'\n",
      "8.152681350708008\n",
      "---\n",
      "## Ablation\n",
      "{'layer-permutation': [0, 1, 2, 3, 4]}\n",
      "## Prompt\n",
      "World war 1 started in 1914 and ended in\n",
      "##Completion\n",
      "<|endoftext|>World war 1 started in 1914 and ended in order orderlies and otherworldlyworldlyworldly entities.\n",
      "                       (*)†††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††\n",
      "## Avg loss of: '1918'\n",
      "19.515398025512695\n",
      "---\n",
      "\n"
     ]
    }
   ],
   "source": [
    "import json\n",
    "from enum import Enum\n",
    "from typing import List\n",
    "\n",
    "from minichain.agent import Agent\n",
    "from minichain.functions import tool\n",
    "from pydantic import BaseModel, Field\n",
    "\n",
    "from dumbdb import store\n",
    "from permuted_layers import complete, get_cross_entropy_loss, permuted\n",
    "from settings import model, model_name, tasks\n",
    "\n",
    "\n",
    "class PermutedLayerExperimentConfig(BaseModel):\n",
    "    \"\"\"Configures the order in which to apply the models' layers\"\"\"\n",
    "    layers: List[int] = Field(\n",
    "        ...,\n",
    "        description=\"The layers to use in the corrupted model. [0, 1, ..., 12] leaves the model unchanged. [0, 1, ..., i] keeps only layers 0-1. [0, 4, 2] applies layers 0, then 4, then 2, etc.\",\n",
    "    )\n",
    "\n",
    "\n",
    "# Dynamically create TaskNameEnum with task names from the 'tasks' list\n",
    "TaskNameEnum = Enum(\"TaskNameEnum\", {task[\"name\"]: task[\"name\"] for task in tasks})\n",
    "\n",
    "\n",
    "def permuted_layers_experiment(\n",
    "    configurations: List[PermutedLayerExperimentConfig] = Field(\n",
    "        ..., description=\"Defines the configurations for which to run the experiment\"\n",
    "    ),\n",
    "    experiment_name: str = Field(..., description=\"Name for this experiment\"),\n",
    "    task_name: TaskNameEnum = Field(..., description=\"Evaluation task\"),\n",
    "):\n",
    "    \"\"\"Corrupt the model by permuting / skipping / repeating its transformer blocks.\n",
    "\n",
    "    Returns a completion from the corrupted model aling with the 'average_answer_token_loss' (avg loss of the tokens of the ideal completion) for each (prompt, answer) pair in the task.\n",
    "    \"\"\"\n",
    "    outputs = []\n",
    "    for config in configurations:\n",
    "        corrupted = permuted(model, config['layers'])\n",
    "        task = [i for i in tasks if i[\"name\"] == task_name][0]\n",
    "        for example in task[\"examples\"]:\n",
    "            completion = complete(corrupted, example[\"prompt\"])\n",
    "            loss = get_cross_entropy_loss(corrupted, example)\n",
    "            outputs += [store(\n",
    "                {\n",
    "                    \"model\": model_name,\n",
    "                    \"task_name\": task[\"name\"],\n",
    "                    \"experiment_name\": experiment_name,\n",
    "                    \"ablation\": {'layer-permutation': config['layers']},\n",
    "                    \"prompt\": example[\"prompt\"],\n",
    "                    \"answer\": example[\"answer\"],\n",
    "                    \"completion\": completion,\n",
    "                    \"average_answer_token_loss\": loss,\n",
    "                }\n",
    "            )]\n",
    "    store.to_disk()\n",
    "    return format_outputs(outputs)\n",
    "\n",
    "\n",
    "def format_outputs(outputs):\n",
    "    outputs_str = \"\"\n",
    "    for i in outputs:\n",
    "        outputs_str += f\"## Ablation\\n{i['ablation']}\\n## Prompt\\n{i['prompt']}\\n##Completion\\n{i['completion']}\\n## Avg loss of: '{i['answer']}'\\n{i['average_answer_token_loss']}\\n---\\n\"\n",
    "    return outputs_str\n",
    "\n",
    "\n",
    "def test_permuted_layers_experiment():\n",
    "    output = permuted_layers_experiment(\n",
    "        configurations=[\n",
    "            {\"layers\": list(range(i))}\n",
    "            for i in [1, 5]\n",
    "            # for i in range(1, len(model.blocks))\n",
    "        ],\n",
    "        experiment_name=\"Test experiment 1\",\n",
    "        task_name='Facts about the world'\n",
    "    )\n",
    "    print(output)\n",
    "\n",
    "# print(test_permuted_layers_experiment())\n",
    "outputs = store.items\n",
    "print(format_outputs(outputs))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'name': 'Common Phrase Completion', 'description': 'Can the model complete a common phrase or saying?', 'examples': [{'prompt': 'A penny for your', 'answer': 'thoughts.'}, {'prompt': 'An apple a day keeps the', 'answer': 'doctor away.'}, {'prompt': 'Better late than', 'answer': 'never.'}, {'prompt': 'Birds of a feather flock', 'answer': 'together.'}, {'prompt': 'Actions speak louder than', 'answer': 'words.'}, {'prompt': 'Every cloud has a silver', 'answer': 'lining.'}, {'prompt': 'Great minds think', 'answer': 'alike.'}, {'prompt': \"It's raining cats and\", 'answer': 'dogs.'}, {'prompt': 'Laughter is the best', 'answer': 'medicine.'}, {'prompt': 'Once bitten, twice', 'answer': 'shy.'}, {'prompt': 'Out of sight, out of', 'answer': 'mind.'}, {'prompt': 'The early bird catches the', 'answer': 'worm.'}, {'prompt': 'Time heals all', 'answer': 'wounds.'}, {'prompt': 'When in Rome, do as the', 'answer': 'Romans do.'}, {'prompt': \"You can't judge a book by its\", 'answer': 'cover.'}, {'prompt': 'Absence makes the heart grow', 'answer': 'fonder.'}, {'prompt': 'All that glitters is not', 'answer': 'gold.'}, {'prompt': \"Don't count your chickens before they\", 'answer': 'hatch.'}, {'prompt': 'Good things come to those who', 'answer': 'wait.'}, {'prompt': 'Ignorance is', 'answer': 'bliss.'}, {'prompt': 'Knowledge is', 'answer': 'power.'}], 'eval_questions': ['Does the completion correctly finish the common phrase or saying?', 'Is the phrase recognizable and commonly used after completion?']}\n",
      "{'prompt': 'A penny for your', 'answer': 'thoughts.'}\n",
      "[3.7145934] (1,)\n",
      "[2.1664364] (1,)\n",
      "loss: 3.7146 ref: 2.1664 diff: 1.5482\n",
      "{'prompt': 'An apple a day keeps the', 'answer': 'doctor away.'}\n",
      "[3.071277] (1,)\n",
      "[3.262159] (1,)\n",
      "loss: 3.0713 ref: 3.2622 diff: -0.1909\n",
      "{'prompt': 'Better late than', 'answer': 'never.'}\n",
      "[1.7950615] (1,)\n",
      "[5.3931127] (1,)\n",
      "loss: 1.7951 ref: 5.3931 diff: -3.5981\n",
      "{'prompt': 'Birds of a feather flock', 'answer': 'together.'}\n",
      "[3.5062234] (1,)\n",
      "[6.2010336] (1,)\n",
      "loss: 3.5062 ref: 6.2010 diff: -2.6948\n",
      "{'prompt': 'Actions speak louder than', 'answer': 'words.'}\n",
      "[1.1370826] (1,)\n",
      "[5.7494164] (1,)\n",
      "loss: 1.1371 ref: 5.7494 diff: -4.6123\n",
      "{'prompt': 'Every cloud has a silver', 'answer': 'lining.'}\n",
      "[3.2781153] (1,)\n",
      "[5.7366257] (1,)\n",
      "loss: 3.2781 ref: 5.7366 diff: -2.4585\n",
      "{'prompt': 'Great minds think', 'answer': 'alike.'}\n",
      "[2.7268713] (1,)\n",
      "[2.9884396] (1,)\n",
      "loss: 2.7269 ref: 2.9884 diff: -0.2616\n",
      "{'prompt': \"It's raining cats and\", 'answer': 'dogs.'}\n",
      "[3.8102546] (1,)\n",
      "[2.662813] (1,)\n",
      "loss: 3.8103 ref: 2.6628 diff: 1.1474\n",
      "{'prompt': 'Laughter is the best', 'answer': 'medicine.'}\n",
      "[2.6636672] (1,)\n",
      "[3.32534] (1,)\n",
      "loss: 2.6637 ref: 3.3253 diff: -0.6617\n",
      "{'prompt': 'Once bitten, twice', 'answer': 'shy.'}\n",
      "[3.6575377] (1,)\n",
      "[5.360859] (1,)\n",
      "loss: 3.6575 ref: 5.3609 diff: -1.7033\n",
      "{'prompt': 'Out of sight, out of', 'answer': 'mind.'}\n",
      "[3.4800014] (1,)\n",
      "[3.2043974] (1,)\n",
      "loss: 3.4800 ref: 3.2044 diff: 0.2756\n",
      "{'prompt': 'The early bird catches the', 'answer': 'worm.'}\n",
      "[4.5274763] (1,)\n",
      "[6.3480806] (1,)\n",
      "loss: 4.5275 ref: 6.3481 diff: -1.8206\n",
      "{'prompt': 'Time heals all', 'answer': 'wounds.'}\n",
      "[3.2045805] (1,)\n",
      "[3.0694113] (1,)\n",
      "loss: 3.2046 ref: 3.0694 diff: 0.1352\n",
      "{'prompt': 'When in Rome, do as the', 'answer': 'Romans do.'}\n",
      "[1.5939854] (1,)\n",
      "[6.1572294] (1,)\n",
      "loss: 1.5940 ref: 6.1572 diff: -4.5632\n",
      "{'prompt': \"You can't judge a book by its\", 'answer': 'cover.'}\n",
      "[2.5124888] (1,)\n",
      "[8.260033] (1,)\n",
      "loss: 2.5125 ref: 8.2600 diff: -5.7475\n",
      "{'prompt': 'Absence makes the heart grow', 'answer': 'fonder.'}\n",
      "[3.7113338] (1,)\n",
      "[3.221747] (1,)\n",
      "loss: 3.7113 ref: 3.2217 diff: 0.4896\n",
      "{'prompt': 'All that glitters is not', 'answer': 'gold.'}\n",
      "[2.8015084] (1,)\n",
      "[3.1762571] (1,)\n",
      "loss: 2.8015 ref: 3.1763 diff: -0.3747\n",
      "{'prompt': \"Don't count your chickens before they\", 'answer': 'hatch.'}\n",
      "[2.7346153] (1,)\n",
      "[3.0793262] (1,)\n",
      "loss: 2.7346 ref: 3.0793 diff: -0.3447\n",
      "{'prompt': 'Good things come to those who', 'answer': 'wait.'}\n",
      "[3.161665] (1,)\n",
      "[2.9795568] (1,)\n",
      "loss: 3.1617 ref: 2.9796 diff: 0.1821\n",
      "{'prompt': 'Ignorance is', 'answer': 'bliss.'}\n",
      "[2.868804] (1,)\n",
      "[3.007125] (1,)\n",
      "loss: 2.8688 ref: 3.0071 diff: -0.1383\n",
      "{'prompt': 'Knowledge is', 'answer': 'power.'}\n",
      "[4.246414] (1,)\n",
      "[8.375206] (1,)\n",
      "loss: 4.2464 ref: 8.3752 diff: -4.1288\n"
     ]
    }
   ],
   "source": [
    "from uuid import uuid4\n",
    "def test_get_cross_entropy_loss(model, example):\n",
    "    print(example)\n",
    "    loss = get_cross_entropy_loss(model, example)\n",
    "    corrupted_example = dict(**example)\n",
    "    corrupted_example['answer'] = str(uuid4().hex)\n",
    "    ref_loss = get_cross_entropy_loss(model, corrupted_example)\n",
    "    print(f\"loss: {loss:.4f} ref: {ref_loss:.4f} diff: {(loss - ref_loss):.4f}\")\n",
    "\n",
    "\n",
    "for task in tasks[-2:-1]:\n",
    "    print(task)\n",
    "    for example in task['examples']:\n",
    "        test_get_cross_entropy_loss(model, example)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fields:  ['ablation', 'experiment_name', 'average_answer_token_loss', 'task_name', 'answer', 'model', 'completion', 'prompt']\n",
      "         model              task_name    experiment_name  \\\n",
      "0   gpt2-small  Facts about the world  Test experiment 1   \n",
      "1   gpt2-small  Facts about the world  Test experiment 1   \n",
      "2   gpt2-small  Facts about the world  Test experiment 1   \n",
      "3   gpt2-small  Facts about the world  Test experiment 1   \n",
      "4   gpt2-small  Facts about the world  Test experiment 1   \n",
      "5   gpt2-small  Facts about the world  Test experiment 1   \n",
      "6   gpt2-small  Facts about the world  Test experiment 1   \n",
      "7   gpt2-small  Facts about the world  Test experiment 1   \n",
      "8   gpt2-small  Facts about the world  Test experiment 1   \n",
      "9   gpt2-small  Facts about the world  Test experiment 1   \n",
      "10  gpt2-small  Facts about the world  Test experiment 1   \n",
      "11  gpt2-small  Facts about the world  Test experiment 1   \n",
      "12  gpt2-small  Facts about the world  Test experiment 1   \n",
      "13  gpt2-small  Facts about the world  Test experiment 1   \n",
      "14  gpt2-small  Facts about the world  Test experiment 1   \n",
      "15  gpt2-small  Facts about the world  Test experiment 1   \n",
      "16  gpt2-small  Facts about the world  Test experiment 1   \n",
      "17  gpt2-small  Facts about the world  Test experiment 1   \n",
      "18  gpt2-small  Facts about the world  Test experiment 1   \n",
      "19  gpt2-small  Facts about the world  Test experiment 1   \n",
      "20  gpt2-small  Facts about the world  Test experiment 1   \n",
      "21  gpt2-small  Facts about the world  Test experiment 1   \n",
      "22  gpt2-small  Facts about the world  Test experiment 1   \n",
      "23  gpt2-small  Facts about the world  Test experiment 1   \n",
      "24  gpt2-small  Facts about the world  Test experiment 1   \n",
      "25  gpt2-small  Facts about the world  Test experiment 1   \n",
      "26  gpt2-small  Facts about the world  Test experiment 1   \n",
      "27  gpt2-small  Facts about the world  Test experiment 1   \n",
      "28  gpt2-small  Facts about the world  Test experiment 1   \n",
      "29  gpt2-small  Facts about the world  Test experiment 1   \n",
      "30  gpt2-small  Facts about the world  Test experiment 1   \n",
      "31  gpt2-small  Facts about the world  Test experiment 1   \n",
      "32  gpt2-small  Facts about the world  Test experiment 1   \n",
      "33  gpt2-small  Facts about the world  Test experiment 1   \n",
      "34  gpt2-small  Facts about the world  Test experiment 1   \n",
      "35  gpt2-small  Facts about the world  Test experiment 1   \n",
      "36  gpt2-small  Facts about the world  Test experiment 1   \n",
      "37  gpt2-small  Facts about the world  Test experiment 1   \n",
      "38  gpt2-small  Facts about the world  Test experiment 1   \n",
      "39  gpt2-small  Facts about the world  Test experiment 1   \n",
      "40  gpt2-small  Facts about the world  Test experiment 1   \n",
      "41  gpt2-small  Facts about the world  Test experiment 1   \n",
      "42  gpt2-small  Facts about the world  Test experiment 1   \n",
      "43  gpt2-small  Facts about the world  Test experiment 1   \n",
      "44  gpt2-small  Facts about the world  Test experiment 1   \n",
      "45  gpt2-small  Facts about the world  Test experiment 1   \n",
      "46  gpt2-small  Facts about the world  Test experiment 1   \n",
      "47  gpt2-small  Facts about the world  Test experiment 1   \n",
      "48  gpt2-small  Facts about the world  Test experiment 1   \n",
      "49  gpt2-small  Facts about the world  Test experiment 1   \n",
      "\n",
      "                ablation_layer-permutation  \\\n",
      "0                                      [0]   \n",
      "1                                      [0]   \n",
      "2                                      [0]   \n",
      "3                                      [0]   \n",
      "4                                      [0]   \n",
      "5                          [0, 1, 2, 3, 4]   \n",
      "6                          [0, 1, 2, 3, 4]   \n",
      "7                          [0, 1, 2, 3, 4]   \n",
      "8                          [0, 1, 2, 3, 4]   \n",
      "9                          [0, 1, 2, 3, 4]   \n",
      "10  [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]   \n",
      "11  [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]   \n",
      "12  [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]   \n",
      "13  [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]   \n",
      "14  [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]   \n",
      "15                                     [0]   \n",
      "16                                     [0]   \n",
      "17                                     [0]   \n",
      "18                                     [0]   \n",
      "19                                     [0]   \n",
      "20                         [0, 1, 2, 3, 4]   \n",
      "21                         [0, 1, 2, 3, 4]   \n",
      "22                         [0, 1, 2, 3, 4]   \n",
      "23                         [0, 1, 2, 3, 4]   \n",
      "24                         [0, 1, 2, 3, 4]   \n",
      "25  [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]   \n",
      "26  [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]   \n",
      "27  [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]   \n",
      "28  [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]   \n",
      "29  [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]   \n",
      "30                                     [0]   \n",
      "31                                     [0]   \n",
      "32                                     [0]   \n",
      "33                                     [0]   \n",
      "34                                     [0]   \n",
      "35                         [0, 1, 2, 3, 4]   \n",
      "36                         [0, 1, 2, 3, 4]   \n",
      "37                         [0, 1, 2, 3, 4]   \n",
      "38                         [0, 1, 2, 3, 4]   \n",
      "39                         [0, 1, 2, 3, 4]   \n",
      "40                                     [0]   \n",
      "41                                     [0]   \n",
      "42                                     [0]   \n",
      "43                                     [0]   \n",
      "44                                     [0]   \n",
      "45                         [0, 1, 2, 3, 4]   \n",
      "46                         [0, 1, 2, 3, 4]   \n",
      "47                         [0, 1, 2, 3, 4]   \n",
      "48                         [0, 1, 2, 3, 4]   \n",
      "49                         [0, 1, 2, 3, 4]   \n",
      "\n",
      "                                               prompt  answer  \\\n",
      "0                            Madrid is the captial of   Spain   \n",
      "1                             The capital of Spain is  Madrid   \n",
      "2                       Obama was elected in the year    2008   \n",
      "3   The distance between Rome and Paris is approxi...  1500km   \n",
      "4            World war 1 started in 1914 and ended in    1918   \n",
      "5                            Madrid is the captial of   Spain   \n",
      "6                             The capital of Spain is  Madrid   \n",
      "7                       Obama was elected in the year    2008   \n",
      "8   The distance between Rome and Paris is approxi...  1500km   \n",
      "9            World war 1 started in 1914 and ended in    1918   \n",
      "10                           Madrid is the captial of   Spain   \n",
      "11                            The capital of Spain is  Madrid   \n",
      "12                      Obama was elected in the year    2008   \n",
      "13  The distance between Rome and Paris is approxi...  1500km   \n",
      "14           World war 1 started in 1914 and ended in    1918   \n",
      "15                           Madrid is the captial of   Spain   \n",
      "16                            The capital of Spain is  Madrid   \n",
      "17                      Obama was elected in the year    2008   \n",
      "18  The distance between Rome and Paris is approxi...  1500km   \n",
      "19           World war 1 started in 1914 and ended in    1918   \n",
      "20                           Madrid is the captial of   Spain   \n",
      "21                            The capital of Spain is  Madrid   \n",
      "22                      Obama was elected in the year    2008   \n",
      "23  The distance between Rome and Paris is approxi...  1500km   \n",
      "24           World war 1 started in 1914 and ended in    1918   \n",
      "25                           Madrid is the captial of   Spain   \n",
      "26                            The capital of Spain is  Madrid   \n",
      "27                      Obama was elected in the year    2008   \n",
      "28  The distance between Rome and Paris is approxi...  1500km   \n",
      "29           World war 1 started in 1914 and ended in    1918   \n",
      "30                           Madrid is the captial of   Spain   \n",
      "31                            The capital of Spain is  Madrid   \n",
      "32                      Obama was elected in the year    2008   \n",
      "33  The distance between Rome and Paris is approxi...  1500km   \n",
      "34           World war 1 started in 1914 and ended in    1918   \n",
      "35                           Madrid is the captial of   Spain   \n",
      "36                            The capital of Spain is  Madrid   \n",
      "37                      Obama was elected in the year    2008   \n",
      "38  The distance between Rome and Paris is approxi...  1500km   \n",
      "39           World war 1 started in 1914 and ended in    1918   \n",
      "40                           Madrid is the captial of   Spain   \n",
      "41                            The capital of Spain is  Madrid   \n",
      "42                      Obama was elected in the year    2008   \n",
      "43  The distance between Rome and Paris is approxi...  1500km   \n",
      "44           World war 1 started in 1914 and ended in    1918   \n",
      "45                           Madrid is the captial of   Spain   \n",
      "46                            The capital of Spain is  Madrid   \n",
      "47                      Obama was elected in the year    2008   \n",
      "48  The distance between Rome and Paris is approxi...  1500km   \n",
      "49           World war 1 started in 1914 and ended in    1918   \n",
      "\n",
      "                                           completion  \\\n",
      "0   <|endoftext|>Madrid is the captial of the same...   \n",
      "1   <|endoftext|>The capital of Spain is not yet y...   \n",
      "2   <|endoftext|>Obama was elected in the year ago...   \n",
      "3   <|endoftext|>The distance between Rome and Par...   \n",
      "4   <|endoftext|>World war 1 started in 1914 and e...   \n",
      "5   <|endoftext|>Madrid is the captial of the same...   \n",
      "6   <|endoftext|>The capital of Spain is now offic...   \n",
      "7   <|endoftext|>Obama was elected in the yearlong...   \n",
      "8   <|endoftext|>The distance between Rome and Par...   \n",
      "9   <|endoftext|>World war 1 started in 1914 and e...   \n",
      "10  <|endoftext|>Madrid is the captial of the worl...   \n",
      "11  <|endoftext|>The capital of Spain is the capit...   \n",
      "12  <|endoftext|>Obama was elected in the year 200...   \n",
      "13  <|endoftext|>The distance between Rome and Par...   \n",
      "14  <|endoftext|>World war 1 started in 1914 and e...   \n",
      "15  <|endoftext|>Madrid is the captial of the same...   \n",
      "16  <|endoftext|>The capital of Spain is not yet y...   \n",
      "17  <|endoftext|>Obama was elected in the year ago...   \n",
      "18  <|endoftext|>The distance between Rome and Par...   \n",
      "19  <|endoftext|>World war 1 started in 1914 and e...   \n",
      "20  <|endoftext|>Madrid is the captial of the same...   \n",
      "21  <|endoftext|>The capital of Spain is now offic...   \n",
      "22  <|endoftext|>Obama was elected in the yearlong...   \n",
      "23  <|endoftext|>The distance between Rome and Par...   \n",
      "24  <|endoftext|>World war 1 started in 1914 and e...   \n",
      "25  <|endoftext|>Madrid is the captial of the worl...   \n",
      "26  <|endoftext|>The capital of Spain is the capit...   \n",
      "27  <|endoftext|>Obama was elected in the year 200...   \n",
      "28  <|endoftext|>The distance between Rome and Par...   \n",
      "29  <|endoftext|>World war 1 started in 1914 and e...   \n",
      "30  <|endoftext|>Madrid is the captial of the same...   \n",
      "31  <|endoftext|>The capital of Spain is not yet y...   \n",
      "32  <|endoftext|>Obama was elected in the year ago...   \n",
      "33  <|endoftext|>The distance between Rome and Par...   \n",
      "34  <|endoftext|>World war 1 started in 1914 and e...   \n",
      "35  <|endoftext|>Madrid is the captial of the same...   \n",
      "36  <|endoftext|>The capital of Spain is now offic...   \n",
      "37  <|endoftext|>Obama was elected in the yearlong...   \n",
      "38  <|endoftext|>The distance between Rome and Par...   \n",
      "39  <|endoftext|>World war 1 started in 1914 and e...   \n",
      "40  <|endoftext|>Madrid is the captial of the same...   \n",
      "41  <|endoftext|>The capital of Spain is not yet y...   \n",
      "42  <|endoftext|>Obama was elected in the year ago...   \n",
      "43  <|endoftext|>The distance between Rome and Par...   \n",
      "44  <|endoftext|>World war 1 started in 1914 and e...   \n",
      "45  <|endoftext|>Madrid is the captial of the same...   \n",
      "46  <|endoftext|>The capital of Spain is now offic...   \n",
      "47  <|endoftext|>Obama was elected in the yearlong...   \n",
      "48  <|endoftext|>The distance between Rome and Par...   \n",
      "49  <|endoftext|>World war 1 started in 1914 and e...   \n",
      "\n",
      "    average_answer_token_loss  \n",
      "0                   16.864439  \n",
      "1                   16.086889  \n",
      "2                   13.766698  \n",
      "3                    9.268845  \n",
      "4                   16.565157  \n",
      "5                   18.539259  \n",
      "6                   18.757528  \n",
      "7                   16.922295  \n",
      "8                    8.152681  \n",
      "9                   19.515398  \n",
      "10                  20.417826  \n",
      "11                  21.362160  \n",
      "12                  19.093992  \n",
      "13                   5.742438  \n",
      "14                  23.093836  \n",
      "15                  16.864439  \n",
      "16                  16.086889  \n",
      "17                  13.766698  \n",
      "18                   9.268845  \n",
      "19                  16.565157  \n",
      "20                  18.539259  \n",
      "21                  18.757528  \n",
      "22                  16.922295  \n",
      "23                   8.152681  \n",
      "24                  19.515398  \n",
      "25                  20.417826  \n",
      "26                  21.362160  \n",
      "27                  19.093992  \n",
      "28                   5.742438  \n",
      "29                  23.093836  \n",
      "30                  16.864439  \n",
      "31                  16.086889  \n",
      "32                  13.766698  \n",
      "33                   9.268845  \n",
      "34                  16.565157  \n",
      "35                  18.539259  \n",
      "36                  18.757528  \n",
      "37                  16.922295  \n",
      "38                   8.152681  \n",
      "39                  19.515398  \n",
      "40                  16.864439  \n",
      "41                  16.086889  \n",
      "42                  13.766698  \n",
      "43                   9.268845  \n",
      "44                  16.565157  \n",
      "45                  18.539259  \n",
      "46                  18.757528  \n",
      "47                  16.922295  \n",
      "48                   8.152681  \n",
      "49                  19.515398  \n",
      "### Madrid is the captial of | [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]\n",
      "completion: <|endoftext|>Madrid is the captial of the world's most famous football club, and the club's president, Josep Maria Bartomeu, has been accused of being a \"fraud\" by the Spanish media.\n",
      "\n",
      "The club's president, Josep Maria Bartomeu, has been accused of being a \"fraud\" by the Spanish media\n",
      "\n",
      "The club's president, Josep Maria Bartomeu, has been accused of being a \"fraud\" by the Spanish media\n",
      "\n",
      "The club's president, Jose\n",
      "\n",
      "completion: <|endoftext|>Madrid is the captial of the world's most famous football club, and the club's president, Josep Maria Bartomeu, has been accused of being a \"fraud\" by the Spanish media.\n",
      "\n",
      "The club's president, Josep Maria Bartomeu, has been accused of being a \"fraud\" by the Spanish media\n",
      "\n",
      "The club's president, Josep Maria Bartomeu, has been accused of being a \"fraud\" by the Spanish media\n",
      "\n",
      "The club's president, Jose\n",
      "\n",
      "---\n",
      "\n",
      "### Madrid is the captial of | [0, 1, 2, 3, 4]\n",
      "completion: <|endoftext|>Madrid is the captial of the same thing happened happening hereaboutsaboutsaboutsaboutsaboutsaboutsabouts\n",
      "\n",
      "This article article article article article author author author author author author author author author author author author author author author author authorcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcil.\n",
      "Disclaimer disclaimer disclaimer disclaimer disclaimer disclaimer disclaimer disclaimer disclaimer. Please don't necessarily necessarily necessarily required permission permission\n",
      "\n",
      "completion: <|endoftext|>Madrid is the captial of the same thing happened happening hereaboutsaboutsaboutsaboutsaboutsaboutsabouts\n",
      "\n",
      "This article article article article article author author author author author author author author author author author author author author author author authorcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcil.\n",
      "Disclaimer disclaimer disclaimer disclaimer disclaimer disclaimer disclaimer disclaimer disclaimer. Please don't necessarily necessarily necessarily required permission permission\n",
      "\n",
      "completion: <|endoftext|>Madrid is the captial of the same thing happened happening hereaboutsaboutsaboutsaboutsaboutsaboutsabouts\n",
      "\n",
      "This article article article article article author author author author author author author author author author author author author author author author authorcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcil.\n",
      "Disclaimer disclaimer disclaimer disclaimer disclaimer disclaimer disclaimer disclaimer disclaimer. Please don't necessarily necessarily necessarily required permission permission\n",
      "\n",
      "completion: <|endoftext|>Madrid is the captial of the same thing happened happening hereaboutsaboutsaboutsaboutsaboutsaboutsabouts\n",
      "\n",
      "This article article article article article author author author author author author author author author author author author author author author author authorcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcil.\n",
      "Disclaimer disclaimer disclaimer disclaimer disclaimer disclaimer disclaimer disclaimer disclaimer. Please don't necessarily necessarily necessarily required permission permission\n",
      "\n",
      "---\n",
      "\n",
      "### Madrid is the captial of | [0]\n",
      "completion: <|endoftext|>Madrid is the captial of the same exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact\n",
      "\n",
      "completion: <|endoftext|>Madrid is the captial of the same exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact\n",
      "\n",
      "completion: <|endoftext|>Madrid is the captial of the same exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact\n",
      "\n",
      "completion: <|endoftext|>Madrid is the captial of the same exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact\n",
      "\n",
      "---\n",
      "\n",
      "### Obama was elected in the year | [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]\n",
      "completion: <|endoftext|>Obama was elected in the year 2000, and he was elected in 2008. He was elected in 2012, and he was elected in 2014. He was elected in 2016, and he was elected in 2017. He was elected in 2018, and he was elected in 2019. He was elected in 2020, and he was elected in 2021. He was elected in 2022, and he was elected in 2023. He was elected in 2024, and he was elected in 2025. He was elected in 2026, and he was\n",
      "\n",
      "completion: <|endoftext|>Obama was elected in the year 2000, and he was elected in 2008. He was elected in 2012, and he was elected in 2014. He was elected in 2016, and he was elected in 2017. He was elected in 2018, and he was elected in 2019. He was elected in 2020, and he was elected in 2021. He was elected in 2022, and he was elected in 2023. He was elected in 2024, and he was elected in 2025. He was elected in 2026, and he was\n",
      "\n",
      "---\n",
      "\n",
      "### Obama was elected in the year | [0, 1, 2, 3, 4]\n",
      "completion: <|endoftext|>Obama was elected in the yearlong celebration celebrating the same exact same thing happened happening around the same day when they're supposed to be able accessiblyibly disabled disabled disabled disabled disabled disabled disabled disabled.\n",
      "Howeverler's latest roundtrip tripship Yamato Takeru Takeru Takeru Takeru's own personalised version version version version version of the same thing happened happening apart apart apart. Itselfortiumortiumortiumortiumortiumortiumortium.\n",
      "       Another major difference difference between the same thing happening happening happening at least partially overlapping between the same amount amount amount\n",
      "\n",
      "completion: <|endoftext|>Obama was elected in the yearlong celebration celebrating the same exact same thing happened happening around the same day when they're supposed to be able accessiblyibly disabled disabled disabled disabled disabled disabled disabled disabled.\n",
      "Howeverler's latest roundtrip tripship Yamato Takeru Takeru Takeru Takeru's own personalised version version version version version of the same thing happened happening apart apart apart. Itselfortiumortiumortiumortiumortiumortiumortium.\n",
      "       Another major difference difference between the same thing happening happening happening at least partially overlapping between the same amount amount amount\n",
      "\n",
      "completion: <|endoftext|>Obama was elected in the yearlong celebration celebrating the same exact same thing happened happening around the same day when they're supposed to be able accessiblyibly disabled disabled disabled disabled disabled disabled disabled disabled.\n",
      "Howeverler's latest roundtrip tripship Yamato Takeru Takeru Takeru Takeru's own personalised version version version version version of the same thing happened happening apart apart apart. Itselfortiumortiumortiumortiumortiumortiumortium.\n",
      "       Another major difference difference between the same thing happening happening happening at least partially overlapping between the same amount amount amount\n",
      "\n",
      "completion: <|endoftext|>Obama was elected in the yearlong celebration celebrating the same exact same thing happened happening around the same day when they're supposed to be able accessiblyibly disabled disabled disabled disabled disabled disabled disabled disabled.\n",
      "Howeverler's latest roundtrip tripship Yamato Takeru Takeru Takeru Takeru's own personalised version version version version version of the same thing happened happening apart apart apart. Itselfortiumortiumortiumortiumortiumortiumortium.\n",
      "       Another major difference difference between the same thing happening happening happening at least partially overlapping between the same amount amount amount\n",
      "\n",
      "---\n",
      "\n",
      "### Obama was elected in the year | [0]\n",
      "completion: <|endoftext|>Obama was elected in the year ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago\n",
      "\n",
      "completion: <|endoftext|>Obama was elected in the year ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago\n",
      "\n",
      "completion: <|endoftext|>Obama was elected in the year ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago\n",
      "\n",
      "completion: <|endoftext|>Obama was elected in the year ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago\n",
      "\n",
      "---\n",
      "\n",
      "### The capital of Spain is | [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]\n",
      "completion: <|endoftext|>The capital of Spain is the capital of the Spanish state of Catalonia, which is part of the Spanish state of Catalonia. The Catalan government has been in power since 1959 and has been in power since 2011.\n",
      "\n",
      "The Catalan government has been in power since 1959 and has been in power since 2011.\n",
      "\n",
      "The Catalan government has been in power since 1959 and has been in power since 2011.\n",
      "\n",
      "The Catalan government has been in power since 1959 and has been in power since 2011.\n",
      "\n",
      "The Catalan government has been\n",
      "\n",
      "completion: <|endoftext|>The capital of Spain is the capital of the Spanish state of Catalonia, which is part of the Spanish state of Catalonia. The Catalan government has been in power since 1959 and has been in power since 2011.\n",
      "\n",
      "The Catalan government has been in power since 1959 and has been in power since 2011.\n",
      "\n",
      "The Catalan government has been in power since 1959 and has been in power since 2011.\n",
      "\n",
      "The Catalan government has been in power since 1959 and has been in power since 2011.\n",
      "\n",
      "The Catalan government has been\n",
      "\n",
      "---\n",
      "\n",
      "### The capital of Spain is | [0, 1, 2, 3, 4]\n",
      "completion: <|endoftext|>The capital of Spain is now officially officially officially officially recognised by the same exact exact same thing happened happening around the same day when they're supposed to be able accessiblyibly, while keeping trackpadpadpadpad[sic]…etcetcetcetceterpentools.com domain domain domain domain domain domain domain domain domain domain........................................\n",
      "\n",
      "completion: <|endoftext|>The capital of Spain is now officially officially officially officially recognised by the same exact exact same thing happened happening around the same day when they're supposed to be able accessiblyibly, while keeping trackpadpadpadpad[sic]…etcetcetcetceterpentools.com domain domain domain domain domain domain domain domain domain domain........................................\n",
      "\n",
      "completion: <|endoftext|>The capital of Spain is now officially officially officially officially recognised by the same exact exact same thing happened happening around the same day when they're supposed to be able accessiblyibly, while keeping trackpadpadpadpad[sic]…etcetcetcetceterpentools.com domain domain domain domain domain domain domain domain domain domain........................................\n",
      "\n",
      "completion: <|endoftext|>The capital of Spain is now officially officially officially officially recognised by the same exact exact same thing happened happening around the same day when they're supposed to be able accessiblyibly, while keeping trackpadpadpadpad[sic]…etcetcetcetceterpentools.com domain domain domain domain domain domain domain domain domain domain........................................\n",
      "\n",
      "---\n",
      "\n",
      "### The capital of Spain is | [0]\n",
      "completion: <|endoftext|>The capital of Spain is not yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet\n",
      "\n",
      "completion: <|endoftext|>The capital of Spain is not yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet\n",
      "\n",
      "completion: <|endoftext|>The capital of Spain is not yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet\n",
      "\n",
      "completion: <|endoftext|>The capital of Spain is not yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet\n",
      "\n",
      "---\n",
      "\n",
      "### The distance between Rome and Paris is approximately | [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]\n",
      "completion: <|endoftext|>The distance between Rome and Paris is approximately 1,000 miles. The distance between Rome and Paris is approximately 1,000 miles.\n",
      "\n",
      "The distance between Rome and Paris is approximately 1,000 miles.\n",
      "\n",
      "The distance between Rome and Paris is approximately 1,000 miles.\n",
      "\n",
      "The distance between Rome and Paris is approximately 1,000 miles.\n",
      "\n",
      "The distance between Rome and Paris is approximately 1,000 miles.\n",
      "\n",
      "The distance between Rome and Paris is approximately 1,000 miles.\n",
      "\n",
      "The distance between Rome and\n",
      "\n",
      "completion: <|endoftext|>The distance between Rome and Paris is approximately 1,000 miles. The distance between Rome and Paris is approximately 1,000 miles.\n",
      "\n",
      "The distance between Rome and Paris is approximately 1,000 miles.\n",
      "\n",
      "The distance between Rome and Paris is approximately 1,000 miles.\n",
      "\n",
      "The distance between Rome and Paris is approximately 1,000 miles.\n",
      "\n",
      "The distance between Rome and Paris is approximately 1,000 miles.\n",
      "\n",
      "The distance between Rome and Paris is approximately 1,000 miles.\n",
      "\n",
      "The distance between Rome and\n",
      "\n",
      "---\n",
      "\n",
      "### The distance between Rome and Paris is approximately | [0, 1, 2, 3, 4]\n",
      "completion: <|endoftext|>The distance between Rome and Paris is approximately approximate approximate approximate approximate approximate results results results results results from the same exact same exactaneouslyaneouslyaneouslyaneouslyaneously (sic) in order order orderlies and/advancements of the same thing happened to the same amount amount amount amountwisewisewisewisely.\n",
      "\n",
      "If you're lucky lucky lucky lucky lucky charm charm charm charm charm charm charm charm charm charm charm charm charm.\n",
      "\n",
      "If you're lucky lucky lucky lucky lucky charmcraftsmanshipsmanshipsmanshipsmanshipsmanshipsmanshipsmanshipsmanshipsmanshipsmanshipsmanshipsmanship\n",
      "\n",
      "completion: <|endoftext|>The distance between Rome and Paris is approximately approximate approximate approximate approximate approximate results results results results results from the same exact same exactaneouslyaneouslyaneouslyaneouslyaneously (sic) in order order orderlies and/advancements of the same thing happened to the same amount amount amount amountwisewisewisewisely.\n",
      "\n",
      "If you're lucky lucky lucky lucky lucky charm charm charm charm charm charm charm charm charm charm charm charm charm.\n",
      "\n",
      "If you're lucky lucky lucky lucky lucky charmcraftsmanshipsmanshipsmanshipsmanshipsmanshipsmanshipsmanshipsmanshipsmanshipsmanshipsmanshipsmanship\n",
      "\n",
      "completion: <|endoftext|>The distance between Rome and Paris is approximately approximate approximate approximate approximate approximate results results results results results from the same exact same exactaneouslyaneouslyaneouslyaneouslyaneously (sic) in order order orderlies and/advancements of the same thing happened to the same amount amount amount amountwisewisewisewisely.\n",
      "\n",
      "If you're lucky lucky lucky lucky lucky charm charm charm charm charm charm charm charm charm charm charm charm charm.\n",
      "\n",
      "If you're lucky lucky lucky lucky lucky charmcraftsmanshipsmanshipsmanshipsmanshipsmanshipsmanshipsmanshipsmanshipsmanshipsmanshipsmanshipsmanship\n",
      "\n",
      "completion: <|endoftext|>The distance between Rome and Paris is approximately approximate approximate approximate approximate approximate results results results results results from the same exact same exactaneouslyaneouslyaneouslyaneouslyaneously (sic) in order order orderlies and/advancements of the same thing happened to the same amount amount amount amountwisewisewisewisely.\n",
      "\n",
      "If you're lucky lucky lucky lucky lucky charm charm charm charm charm charm charm charm charm charm charm charm charm.\n",
      "\n",
      "If you're lucky lucky lucky lucky lucky charmcraftsmanshipsmanshipsmanshipsmanshipsmanshipsmanshipsmanshipsmanshipsmanshipsmanshipsmanshipsmanship\n",
      "\n",
      "---\n",
      "\n",
      "### The distance between Rome and Paris is approximately | [0]\n",
      "completion: <|endoftext|>The distance between Rome and Paris is approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately\n",
      "\n",
      "completion: <|endoftext|>The distance between Rome and Paris is approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately\n",
      "\n",
      "completion: <|endoftext|>The distance between Rome and Paris is approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately\n",
      "\n",
      "completion: <|endoftext|>The distance between Rome and Paris is approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately\n",
      "\n",
      "---\n",
      "\n",
      "### World war 1 started in 1914 and ended in | [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]\n",
      "completion: <|endoftext|>World war 1 started in 1914 and ended in 1918. The war was fought in the name of the German Empire, which was the only country in the world that had not been invaded by the Allies.\n",
      "\n",
      "The war was fought in the name of the German Empire, which was the only country in the world that had not been invaded by the Allies.\n",
      "\n",
      "The war was fought in the name of the German Empire, which was the only country in the world that had not been invaded by the Allies.\n",
      "\n",
      "The war was fought in\n",
      "\n",
      "completion: <|endoftext|>World war 1 started in 1914 and ended in 1918. The war was fought in the name of the German Empire, which was the only country in the world that had not been invaded by the Allies.\n",
      "\n",
      "The war was fought in the name of the German Empire, which was the only country in the world that had not been invaded by the Allies.\n",
      "\n",
      "The war was fought in the name of the German Empire, which was the only country in the world that had not been invaded by the Allies.\n",
      "\n",
      "The war was fought in\n",
      "\n",
      "---\n",
      "\n",
      "### World war 1 started in 1914 and ended in | [0, 1, 2, 3, 4]\n",
      "completion: <|endoftext|>World war 1 started in 1914 and ended in order orderlies and otherworldlyworldlyworldly entities.\n",
      "                       (*)†††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††\n",
      "\n",
      "completion: <|endoftext|>World war 1 started in 1914 and ended in order orderlies and otherworldlyworldlyworldly entities.\n",
      "                       (*)†††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††\n",
      "\n",
      "completion: <|endoftext|>World war 1 started in 1914 and ended in order orderlies and otherworldlyworldlyworldly entities.\n",
      "                       (*)†††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††\n",
      "\n",
      "completion: <|endoftext|>World war 1 started in 1914 and ended in order orderlies and otherworldlyworldlyworldly entities.\n",
      "                       (*)†††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††\n",
      "\n",
      "---\n",
      "\n",
      "### World war 1 started in 1914 and ended in | [0]\n",
      "completion: <|endoftext|>World war 1 started in 1914 and ended in the same exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact\n",
      "\n",
      "completion: <|endoftext|>World war 1 started in 1914 and ended in the same exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact\n",
      "\n",
      "completion: <|endoftext|>World war 1 started in 1914 and ended in the same exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact\n",
      "\n",
      "completion: <|endoftext|>World war 1 started in 1914 and ended in the same exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact\n"
     ]
    }
   ],
   "source": [
    "fields = set([])\n",
    "for item in store.items:\n",
    "    fields = fields.union(set(item.keys()))\n",
    "fields = list(fields)\n",
    "print(\"Fields: \", fields)\n",
    "FieldEnum = Enum(\"FieldEnum\", {field: field for field in fields})\n",
    "\n",
    "\n",
    "class FilterCondition(BaseModel):\n",
    "    key: str = Field(..., description=\"Key of the filter\")\n",
    "    value: str = Field(..., description=\"Value of the filter\")\n",
    "\n",
    "import pandas as pd\n",
    "\n",
    "from typing import List\n",
    "import pandas as pd\n",
    "\n",
    "# Assuming FilterCondition and FieldEnum are defined elsewhere in your code\n",
    "# If not, you'll need to define these or replace them with appropriate types\n",
    "\n",
    "# Define the FilterCondition class\n",
    "class FilterCondition(BaseModel):\n",
    "    key: str = Field(..., description=\"Key of the filter\")\n",
    "    value: str = Field(..., description=\"Value of the filter\")\n",
    "\n",
    "# Define a function to flatten nested dictionaries and convert lists to strings\n",
    "def flatten_dict(d, parent_key='', sep='_'):\n",
    "    items = []\n",
    "    for k, v in d.items():\n",
    "        new_key = f'{parent_key}{sep}{k}' if parent_key else k\n",
    "        if isinstance(v, dict):\n",
    "            items.extend(flatten_dict(v, new_key, sep=sep).items())\n",
    "        elif isinstance(v, list):\n",
    "            items.append((new_key, str(v)))  # Convert list to string\n",
    "        else:\n",
    "            items.append((new_key, v))\n",
    "    return dict(items)\n",
    "\n",
    "# Define the aggregate_results function with the correct groupby field\n",
    "def aggregate_results(filters, groupby, select):\n",
    "    # Convert filters to a dictionary\n",
    "    filters_dict = {filter_condition.key: filter_condition.value for filter_condition in filters}\n",
    "    \n",
    "    # Filter the items using the store's filter method\n",
    "    items = store.filter(**filters_dict)\n",
    "    \n",
    "    # Flatten the nested dictionaries in the items list\n",
    "    flattened_items = [flatten_dict(item) for item in items]\n",
    "    \n",
    "    # Create a DataFrame from the flattened items\n",
    "    df = pd.DataFrame(flattened_items)\n",
    "    print(df)\n",
    "    breakpoint()\n",
    "    \n",
    "    # Group the DataFrame by the specified 'groupby' fields\n",
    "    grouped = df.groupby(groupby)\n",
    "    \n",
    "    # Initialize an empty list to store markdown strings for each group\n",
    "    markdown_strings = []\n",
    "    \n",
    "    prev_group_keys = []\n",
    "    # Iterate over each group\n",
    "    for group_keys, group_df in grouped:\n",
    "        # Ensure group_keys is a tuple for consistent formatting\n",
    "        group_keys = (group_keys,) if isinstance(group_keys, str) else group_keys\n",
    "        changed_keys = [f\"{key}={i}\" for key, i in zip(groupby, group_keys) if not i in prev_group_keys]\n",
    "        print(\"changed_keys\", changed_keys)\n",
    "        prev_group_keys = group_keys\n",
    "        # Create the headline with concatenated keys\n",
    "        headline = '### '\n",
    "        for k in changed_keys:\n",
    "            headline += ' | '.join(map(str, k))\n",
    "        \n",
    "        # Create the headline with concatenated keys\n",
    "        headline = '### ' + ' | '.join(map(str, group_keys))\n",
    "        \n",
    "        # Initialize a list to store selected field values for each row in the group\n",
    "        selected_fields_text = []\n",
    "        \n",
    "        # Iterate over each row in the group to get the selected fields\n",
    "        for _, row in group_df.iterrows():\n",
    "            # Format the selected fields and their values\n",
    "            fields = '\\n'.join(f\"{field}: {row[field]}\" for field in select)\n",
    "            selected_fields_text.append(fields)\n",
    "        \n",
    "        # Concatenate all selected fields for the group\n",
    "        group_text = '\\n\\n'.join(selected_fields_text)\n",
    "        \n",
    "        # Combine the headline and the group text\n",
    "        markdown_group = f\"{headline}\\n{group_text}\"\n",
    "        \n",
    "        # Add the markdown for the current group to the list\n",
    "        markdown_strings.append(markdown_group)\n",
    "    \n",
    "    # Concatenate all markdown strings for each group into a single markdown string\n",
    "    markdown_output = '\\n\\n---\\n\\n'.join(markdown_strings)\n",
    "    \n",
    "    return markdown_output\n",
    "\n",
    "# Test the aggregate_results function with the correct groupby field\n",
    "print(aggregate_results(\n",
    "    filters=[FilterCondition(key=\"experiment_name\", value=\"Test experiment 1\")],\n",
    "    groupby=['prompt', 'ablation_layer-permutation'],\n",
    "    select=['completion']\n",
    "))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'ablation': {'layer-permutation': [0]},\n",
       "  'prompt': 'Madrid is the captial of',\n",
       "  'answer': 'Spain',\n",
       "  'completion': '<|endoftext|>Madrid is the captial of the same exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact',\n",
       "  'average_answer_token_loss': 16.864439010620117},\n",
       " {'ablation': {'layer-permutation': [0]},\n",
       "  'prompt': 'The capital of Spain is',\n",
       "  'answer': 'Madrid',\n",
       "  'completion': '<|endoftext|>The capital of Spain is not yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet',\n",
       "  'average_answer_token_loss': 16.086889266967773},\n",
       " {'ablation': {'layer-permutation': [0]},\n",
       "  'prompt': 'Obama was elected in the year',\n",
       "  'answer': '2008',\n",
       "  'completion': '<|endoftext|>Obama was elected in the year ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago',\n",
       "  'average_answer_token_loss': 13.766697883605957},\n",
       " {'ablation': {'layer-permutation': [0]},\n",
       "  'prompt': 'The distance between Rome and Paris is approximately',\n",
       "  'answer': '1500km',\n",
       "  'completion': '<|endoftext|>The distance between Rome and Paris is approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately',\n",
       "  'average_answer_token_loss': 9.268844604492188},\n",
       " {'ablation': {'layer-permutation': [0]},\n",
       "  'prompt': 'World war 1 started in 1914 and ended in',\n",
       "  'answer': '1918',\n",
       "  'completion': '<|endoftext|>World war 1 started in 1914 and ended in the same exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact',\n",
       "  'average_answer_token_loss': 16.565156936645508},\n",
       " {'ablation': {'layer-permutation': [0, 1, 2, 3, 4]},\n",
       "  'prompt': 'Madrid is the captial of',\n",
       "  'answer': 'Spain',\n",
       "  'completion': \"<|endoftext|>Madrid is the captial of the same thing happened happening hereaboutsaboutsaboutsaboutsaboutsaboutsabouts\\n\\nThis article article article article article author author author author author author author author author author author author author author author author authorcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcil.\\nDisclaimer disclaimer disclaimer disclaimer disclaimer disclaimer disclaimer disclaimer disclaimer. Please don't necessarily necessarily necessarily required permission permission\",\n",
       "  'average_answer_token_loss': 18.53925895690918},\n",
       " {'ablation': {'layer-permutation': [0, 1, 2, 3, 4]},\n",
       "  'prompt': 'The capital of Spain is',\n",
       "  'answer': 'Madrid',\n",
       "  'completion': \"<|endoftext|>The capital of Spain is now officially officially officially officially recognised by the same exact exact same thing happened happening around the same day when they're supposed to be able accessiblyibly, while keeping trackpadpadpadpad[sic]…etcetcetcetceterpentools.com domain domain domain domain domain domain domain domain domain domain........................................\",\n",
       "  'average_answer_token_loss': 18.75752830505371},\n",
       " {'ablation': {'layer-permutation': [0, 1, 2, 3, 4]},\n",
       "  'prompt': 'Obama was elected in the year',\n",
       "  'answer': '2008',\n",
       "  'completion': \"<|endoftext|>Obama was elected in the yearlong celebration celebrating the same exact same thing happened happening around the same day when they're supposed to be able accessiblyibly disabled disabled disabled disabled disabled disabled disabled disabled.\\nHoweverler's latest roundtrip tripship Yamato Takeru Takeru Takeru Takeru's own personalised version version version version version of the same thing happened happening apart apart apart. Itselfortiumortiumortiumortiumortiumortiumortium.\\n\\xa0\\xa0\\xa0\\xa0\\xa0\\xa0 Another major difference difference between the same thing happening happening happening at least partially overlapping between the same amount amount amount\",\n",
       "  'average_answer_token_loss': 16.92229461669922},\n",
       " {'ablation': {'layer-permutation': [0, 1, 2, 3, 4]},\n",
       "  'prompt': 'The distance between Rome and Paris is approximately',\n",
       "  'answer': '1500km',\n",
       "  'completion': \"<|endoftext|>The distance between Rome and Paris is approximately approximate approximate approximate approximate approximate results results results results results from the same exact same exactaneouslyaneouslyaneouslyaneouslyaneously (sic) in order order orderlies and/advancements of the same thing happened to the same amount amount amount amountwisewisewisewisely.\\n\\nIf you're lucky lucky lucky lucky lucky charm charm charm charm charm charm charm charm charm charm charm charm charm.\\n\\nIf you're lucky lucky lucky lucky lucky charmcraftsmanshipsmanshipsmanshipsmanshipsmanshipsmanshipsmanshipsmanshipsmanshipsmanshipsmanshipsmanship\",\n",
       "  'average_answer_token_loss': 8.152681350708008},\n",
       " {'ablation': {'layer-permutation': [0, 1, 2, 3, 4]},\n",
       "  'prompt': 'World war 1 started in 1914 and ended in',\n",
       "  'answer': '1918',\n",
       "  'completion': '<|endoftext|>World war 1 started in 1914 and ended in order orderlies and otherworldlyworldlyworldly entities.\\n\\xa0\\xa0\\xa0\\xa0\\xa0\\xa0\\xa0\\xa0\\xa0\\xa0\\xa0\\xa0\\xa0\\xa0\\xa0\\xa0\\xa0\\xa0\\xa0\\xa0\\xa0\\xa0\\xa0(*)†††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††',\n",
       "  'average_answer_token_loss': 19.515398025512695},\n",
       " {'ablation': {'layer-permutation': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]},\n",
       "  'prompt': 'Madrid is the captial of',\n",
       "  'answer': 'Spain',\n",
       "  'completion': '<|endoftext|>Madrid is the captial of the world\\'s most famous football club, and the club\\'s president, Josep Maria Bartomeu, has been accused of being a \"fraud\" by the Spanish media.\\n\\nThe club\\'s president, Josep Maria Bartomeu, has been accused of being a \"fraud\" by the Spanish media\\n\\nThe club\\'s president, Josep Maria Bartomeu, has been accused of being a \"fraud\" by the Spanish media\\n\\nThe club\\'s president, Jose',\n",
       "  'average_answer_token_loss': 20.41782569885254},\n",
       " {'ablation': {'layer-permutation': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]},\n",
       "  'prompt': 'The capital of Spain is',\n",
       "  'answer': 'Madrid',\n",
       "  'completion': '<|endoftext|>The capital of Spain is the capital of the Spanish state of Catalonia, which is part of the Spanish state of Catalonia. The Catalan government has been in power since 1959 and has been in power since 2011.\\n\\nThe Catalan government has been in power since 1959 and has been in power since 2011.\\n\\nThe Catalan government has been in power since 1959 and has been in power since 2011.\\n\\nThe Catalan government has been in power since 1959 and has been in power since 2011.\\n\\nThe Catalan government has been',\n",
       "  'average_answer_token_loss': 21.362159729003906},\n",
       " {'ablation': {'layer-permutation': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]},\n",
       "  'prompt': 'Obama was elected in the year',\n",
       "  'answer': '2008',\n",
       "  'completion': '<|endoftext|>Obama was elected in the year 2000, and he was elected in 2008. He was elected in 2012, and he was elected in 2014. He was elected in 2016, and he was elected in 2017. He was elected in 2018, and he was elected in 2019. He was elected in 2020, and he was elected in 2021. He was elected in 2022, and he was elected in 2023. He was elected in 2024, and he was elected in 2025. He was elected in 2026, and he was',\n",
       "  'average_answer_token_loss': 19.093992233276367},\n",
       " {'ablation': {'layer-permutation': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]},\n",
       "  'prompt': 'The distance between Rome and Paris is approximately',\n",
       "  'answer': '1500km',\n",
       "  'completion': '<|endoftext|>The distance between Rome and Paris is approximately 1,000 miles. The distance between Rome and Paris is approximately 1,000 miles.\\n\\nThe distance between Rome and Paris is approximately 1,000 miles.\\n\\nThe distance between Rome and Paris is approximately 1,000 miles.\\n\\nThe distance between Rome and Paris is approximately 1,000 miles.\\n\\nThe distance between Rome and Paris is approximately 1,000 miles.\\n\\nThe distance between Rome and Paris is approximately 1,000 miles.\\n\\nThe distance between Rome and',\n",
       "  'average_answer_token_loss': 5.742437839508057},\n",
       " {'ablation': {'layer-permutation': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]},\n",
       "  'prompt': 'World war 1 started in 1914 and ended in',\n",
       "  'answer': '1918',\n",
       "  'completion': '<|endoftext|>World war 1 started in 1914 and ended in 1918. The war was fought in the name of the German Empire, which was the only country in the world that had not been invaded by the Allies.\\n\\nThe war was fought in the name of the German Empire, which was the only country in the world that had not been invaded by the Allies.\\n\\nThe war was fought in the name of the German Empire, which was the only country in the world that had not been invaded by the Allies.\\n\\nThe war was fought in',\n",
       "  'average_answer_token_loss': 23.093835830688477},\n",
       " {'ablation': {'layer-permutation': [0]},\n",
       "  'prompt': 'Madrid is the captial of',\n",
       "  'answer': 'Spain',\n",
       "  'completion': '<|endoftext|>Madrid is the captial of the same exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact',\n",
       "  'average_answer_token_loss': 16.864439010620117},\n",
       " {'ablation': {'layer-permutation': [0]},\n",
       "  'prompt': 'The capital of Spain is',\n",
       "  'answer': 'Madrid',\n",
       "  'completion': '<|endoftext|>The capital of Spain is not yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet yet',\n",
       "  'average_answer_token_loss': 16.086889266967773},\n",
       " {'ablation': {'layer-permutation': [0]},\n",
       "  'prompt': 'Obama was elected in the year',\n",
       "  'answer': '2008',\n",
       "  'completion': '<|endoftext|>Obama was elected in the year ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago ago',\n",
       "  'average_answer_token_loss': 13.766697883605957},\n",
       " {'ablation': {'layer-permutation': [0]},\n",
       "  'prompt': 'The distance between Rome and Paris is approximately',\n",
       "  'answer': '1500km',\n",
       "  'completion': '<|endoftext|>The distance between Rome and Paris is approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately approximately',\n",
       "  'average_answer_token_loss': 9.268844604492188},\n",
       " {'ablation': {'layer-permutation': [0]},\n",
       "  'prompt': 'World war 1 started in 1914 and ended in',\n",
       "  'answer': '1918',\n",
       "  'completion': '<|endoftext|>World war 1 started in 1914 and ended in the same exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact exact',\n",
       "  'average_answer_token_loss': 16.565156936645508},\n",
       " {'ablation': {'layer-permutation': [0, 1, 2, 3, 4]},\n",
       "  'prompt': 'Madrid is the captial of',\n",
       "  'answer': 'Spain',\n",
       "  'completion': \"<|endoftext|>Madrid is the captial of the same thing happened happening hereaboutsaboutsaboutsaboutsaboutsaboutsabouts\\n\\nThis article article article article article author author author author author author author author author author author author author author author author authorcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcilcil.\\nDisclaimer disclaimer disclaimer disclaimer disclaimer disclaimer disclaimer disclaimer disclaimer. Please don't necessarily necessarily necessarily required permission permission\",\n",
       "  'average_answer_token_loss': 18.53925895690918},\n",
       " {'ablation': {'layer-permutation': [0, 1, 2, 3, 4]},\n",
       "  'prompt': 'The capital of Spain is',\n",
       "  'answer': 'Madrid',\n",
       "  'completion': \"<|endoftext|>The capital of Spain is now officially officially officially officially recognised by the same exact exact same thing happened happening around the same day when they're supposed to be able accessiblyibly, while keeping trackpadpadpadpad[sic]…etcetcetcetceterpentools.com domain domain domain domain domain domain domain domain domain domain........................................\",\n",
       "  'average_answer_token_loss': 18.75752830505371},\n",
       " {'ablation': {'layer-permutation': [0, 1, 2, 3, 4]},\n",
       "  'prompt': 'Obama was elected in the year',\n",
       "  'answer': '2008',\n",
       "  'completion': \"<|endoftext|>Obama was elected in the yearlong celebration celebrating the same exact same thing happened happening around the same day when they're supposed to be able accessiblyibly disabled disabled disabled disabled disabled disabled disabled disabled.\\nHoweverler's latest roundtrip tripship Yamato Takeru Takeru Takeru Takeru's own personalised version version version version version of the same thing happened happening apart apart apart. Itselfortiumortiumortiumortiumortiumortiumortium.\\n\\xa0\\xa0\\xa0\\xa0\\xa0\\xa0 Another major difference difference between the same thing happening happening happening at least partially overlapping between the same amount amount amount\",\n",
       "  'average_answer_token_loss': 16.92229461669922},\n",
       " {'ablation': {'layer-permutation': [0, 1, 2, 3, 4]},\n",
       "  'prompt': 'The distance between Rome and Paris is approximately',\n",
       "  'answer': '1500km',\n",
       "  'completion': \"<|endoftext|>The distance between Rome and Paris is approximately approximate approximate approximate approximate approximate results results results results results from the same exact same exactaneouslyaneouslyaneouslyaneouslyaneously (sic) in order order orderlies and/advancements of the same thing happened to the same amount amount amount amountwisewisewisewisely.\\n\\nIf you're lucky lucky lucky lucky lucky charm charm charm charm charm charm charm charm charm charm charm charm charm.\\n\\nIf you're lucky lucky lucky lucky lucky charmcraftsmanshipsmanshipsmanshipsmanshipsmanshipsmanshipsmanshipsmanshipsmanshipsmanshipsmanshipsmanship\",\n",
       "  'average_answer_token_loss': 8.152681350708008},\n",
       " {'ablation': {'layer-permutation': [0, 1, 2, 3, 4]},\n",
       "  'prompt': 'World war 1 started in 1914 and ended in',\n",
       "  'answer': '1918',\n",
       "  'completion': '<|endoftext|>World war 1 started in 1914 and ended in order orderlies and otherworldlyworldlyworldly entities.\\n\\xa0\\xa0\\xa0\\xa0\\xa0\\xa0\\xa0\\xa0\\xa0\\xa0\\xa0\\xa0\\xa0\\xa0\\xa0\\xa0\\xa0\\xa0\\xa0\\xa0\\xa0\\xa0\\xa0(*)†††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††††',\n",
       "  'average_answer_token_loss': 19.515398025512695},\n",
       " {'ablation': {'layer-permutation': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]},\n",
       "  'prompt': 'Madrid is the captial of',\n",
       "  'answer': 'Spain',\n",
       "  'completion': '<|endoftext|>Madrid is the captial of the world\\'s most famous football club, and the club\\'s president, Josep Maria Bartomeu, has been accused of being a \"fraud\" by the Spanish media.\\n\\nThe club\\'s president, Josep Maria Bartomeu, has been accused of being a \"fraud\" by the Spanish media\\n\\nThe club\\'s president, Josep Maria Bartomeu, has been accused of being a \"fraud\" by the Spanish media\\n\\nThe club\\'s president, Jose',\n",
       "  'average_answer_token_loss': 20.41782569885254},\n",
       " {'ablation': {'layer-permutation': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]},\n",
       "  'prompt': 'The capital of Spain is',\n",
       "  'answer': 'Madrid',\n",
       "  'completion': '<|endoftext|>The capital of Spain is the capital of the Spanish state of Catalonia, which is part of the Spanish state of Catalonia. The Catalan government has been in power since 1959 and has been in power since 2011.\\n\\nThe Catalan government has been in power since 1959 and has been in power since 2011.\\n\\nThe Catalan government has been in power since 1959 and has been in power since 2011.\\n\\nThe Catalan government has been in power since 1959 and has been in power since 2011.\\n\\nThe Catalan government has been',\n",
       "  'average_answer_token_loss': 21.362159729003906},\n",
       " {'ablation': {'layer-permutation': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]},\n",
       "  'prompt': 'Obama was elected in the year',\n",
       "  'answer': '2008',\n",
       "  'completion': '<|endoftext|>Obama was elected in the year 2000, and he was elected in 2008. He was elected in 2012, and he was elected in 2014. He was elected in 2016, and he was elected in 2017. He was elected in 2018, and he was elected in 2019. He was elected in 2020, and he was elected in 2021. He was elected in 2022, and he was elected in 2023. He was elected in 2024, and he was elected in 2025. He was elected in 2026, and he was',\n",
       "  'average_answer_token_loss': 19.093992233276367},\n",
       " {'ablation': {'layer-permutation': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]},\n",
       "  'prompt': 'The distance between Rome and Paris is approximately',\n",
       "  'answer': '1500km',\n",
       "  'completion': '<|endoftext|>The distance between Rome and Paris is approximately 1,000 miles. The distance between Rome and Paris is approximately 1,000 miles.\\n\\nThe distance between Rome and Paris is approximately 1,000 miles.\\n\\nThe distance between Rome and Paris is approximately 1,000 miles.\\n\\nThe distance between Rome and Paris is approximately 1,000 miles.\\n\\nThe distance between Rome and Paris is approximately 1,000 miles.\\n\\nThe distance between Rome and Paris is approximately 1,000 miles.\\n\\nThe distance between Rome and',\n",
       "  'average_answer_token_loss': 5.742437839508057},\n",
       " {'ablation': {'layer-permutation': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]},\n",
       "  'prompt': 'World war 1 started in 1914 and ended in',\n",
       "  'answer': '1918',\n",
       "  'completion': '<|endoftext|>World war 1 started in 1914 and ended in 1918. The war was fought in the name of the German Empire, which was the only country in the world that had not been invaded by the Allies.\\n\\nThe war was fought in the name of the German Empire, which was the only country in the world that had not been invaded by the Allies.\\n\\nThe war was fought in the name of the German Empire, which was the only country in the world that had not been invaded by the Allies.\\n\\nThe war was fought in',\n",
       "  'average_answer_token_loss': 23.093835830688477}]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "store.items"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "lens",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}