Merge pull request #19 from Griboedow/copilot/consider-claude-and-openai

Add optional LLM polishing step after Pandoc conversion (Claude + OpenAI)

Author: Griboedow

README.md

@@ -4,6 +4,7 @@ MediaWiki extension for **importing** documents/webpages into wiki pages and **e
 
 - **Import**: convert DOCX, ODT, PDF, DOC, or a webpage URL into a wiki page (with images)
 - **Export**: download wiki pages as DOCX, ODT, EPUB, PDF, HTML, RTF, or TXT
+- **AI cleanup**: optional LLM-powered post-conversion wikitext polish (OpenAI or Claude)
 
 MediaWiki page: https://www.mediawiki.org/wiki/Extension:PandocUltimateConverter
 
@@ -47,9 +48,40 @@ What happens during conversion:
 - Images are extracted and uploaded to the wiki automatically (duplicates are skipped)
 - The uploaded source file is removed after conversion
 - Temporary files are cleaned up
-
 A legacy (non-Codex) form is available at `Special:PandocUltimateConverter?codex=0`.
 
+## AI Cleanup (LLM Polish)
+
+The extension can optionally run an LLM (OpenAI or Claude) to clean up wikitext after conversion — fixing formatting issues, removing artefacts, and improving readability.
+
+### Setup
+
+Add to `LocalSettings.php`:
+```php
+$wgPandocUltimateConverter_LlmProvider = 'openai';   // or 'claude'
+$wgPandocUltimateConverter_LlmApiKey   = 'sk-...';
+// Optional: override the default model
+// $wgPandocUltimateConverter_LlmModel = 'gpt-5.4-nano';   // OpenAI default; or 'claude-3-5-haiku-20241022' for Claude
+```
+
+### Usage
+
+There are two ways to use AI cleanup:
+
+1. **Batch mode** — check the "Polish with AI" checkbox before clicking **Convert all**. Each item is converted first, then automatically queued for AI cleanup. The conversion queue and the AI cleanup queue run in parallel.
+2. **Per-item** — click the ✨ button on any already-converted item to run AI cleanup on demand.
+
+If AI cleanup fails, a per-item error is shown with a **Retry** button.
+
+### LLM Configuration
+
+| Parameter | Default | Description |
+|-----------|---------|-------------|
+| `PandocUltimateConverter_LlmProvider` | `null` | `"openai"` or `"claude"`. Leave null to disable. |
+| `PandocUltimateConverter_LlmApiKey` | `null` | API key for the configured provider. |
+| `PandocUltimateConverter_LlmModel` | `null` | Model override. Defaults to `gpt-5.4-nano` (OpenAI) or `claude-3-5-haiku-20241022` (Claude). |
+| `PandocUltimateConverter_LlmPrompt` | `null` | Custom system prompt for the cleanup step. |
+
 ## Export (Special:PandocExport)
 
 Export one or more wiki pages to an external document format.
@@ -105,6 +137,10 @@ All parameters are set in `LocalSettings.php` with the `$wg` prefix.
 | `PandocUltimateConverter_FiltersToUse` | `[]` | Custom [Pandoc Lua filters](https://pandoc.org/filters.html) to apply. Must be in the `filters/` folder. |
 | `PandocUltimateConverter_UseColorProcessors` | `false` | Preserve text/background colors from DOCX/ODT files. |
 | `PandocUltimateConverter_ShowExportInPageTools` | `true` | Show "Export" in the page Actions menu. |
+| `PandocUltimateConverter_LlmProvider` | `null` | LLM provider: `"openai"` or `"claude"`. |
+| `PandocUltimateConverter_LlmApiKey` | `null` | API key for the LLM provider. |
+| `PandocUltimateConverter_LlmModel` | `null` | Model name override. |
+| `PandocUltimateConverter_LlmPrompt` | `null` | Custom system prompt for AI cleanup. |
 
 ### Built-in Lua filters
 
@@ -176,26 +212,26 @@ $wgPandocUltimateConverter_LibreOfficeExecutablePath = 'C:\Program Files\LibreOf
 
 ## Action API
 
-The extension exposes `action=pandocconvert` for programmatic conversions.
+The extension exposes three API modules. Write operations (`pandocconvert`, `pandocllmpolish`) require a CSRF token and POST.
 
-Requires a CSRF token and POST. Obtain a token:
+Obtain a CSRF token first:
 ```
 GET /api.php?action=query&meta=tokens&format=json
 ```
 
-**Convert a URL:**
-```
-POST /api.php
-action=pandocconvert&url=https://example.com&pagename=My Article&forceoverwrite=1&token=<csrf>&format=json
-```
+### action=pandocconvert
+
+Converts a file or URL to a wiki page. Requires a CSRF token and POST.
 
-**Convert an uploaded file:**
 ```
 POST /api.php
-action=pandocconvert&filename=Document.docx&pagename=My Article&forceoverwrite=1&token=<csrf>&format=json
+action=pandocconvert&pagename=My Article&url=https://example.com&forceoverwrite=1&token=<csrf>&format=json
 ```
 
-### API parameters
+**Response:**
+```json
+{ "pandocconvert": { "result": "success", "pagename": "My Article" } }
+```
 
 | Parameter | Required | Description |
 |-----------|----------|-------------|
@@ -205,15 +241,62 @@ action=pandocconvert&filename=Document.docx&pagename=My Article&forceoverwrite=1
 | `forceoverwrite` | no | `1` to overwrite existing page (default: `0`) |
 | `token` | yes | CSRF token |
 
+### action=pandocllmpolish
+
+Runs LLM AI cleanup on an existing wiki page's wikitext. Requires a CSRF token and POST. The LLM provider must be [configured](#llm-configuration).
+
+```
+POST /api.php
+action=pandocllmpolish&pagename=My Article&token=<csrf>&format=json
+```
+
+**Response:**
+```json
+{ "pandocllmpolish": { "result": "success", "pagename": "My Article" } }
+```
+
+| Parameter | Required | Description |
+|-----------|----------|-------------|
+| `pagename` | yes | Title of existing wiki page to polish |
+| `token` | yes | CSRF token |
+
+### action=pandocurltitle
+
+Fetches remote URLs and extracts their HTML `<title>` tags. Used internally by the Codex UI to suggest page names for URL imports. GET request, no token required.
+
+```
+GET /api.php?action=pandocurltitle&urls=https://example.com&format=json
+```
+
+**Response:**
+```json
+{ "pandocurltitle": { "results": [ { "url": "https://example.com", "title": "Example Domain" } ] } }
+```
+
+Accepts multiple URLs (pipe-separated). Only `http`/`https` URLs are accepted.
+
+| Parameter | Required | Description |
+|-----------|----------|-------------|
+| `urls` | yes | One or more URLs (pipe-separated) to fetch titles from |
+
 ### API error codes
 
+**pandocconvert:**
+
 | Code | Meaning |
 |------|---------|
 | `nosource` | Neither `filename` nor `url` supplied |
 | `multiplesource` | Both `filename` and `url` supplied |
 | `invalidurlscheme` | URL is not `http`/`https` |
 | `pageexists` | Page exists and `forceoverwrite` not set |
-| `conversionfailed` | Pandoc conversion failed |
+
+**pandocllmpolish:**
+
+| Code | Meaning |
+|------|---------|
+| `pagenotfound` | The specified page does not exist |
+| `notconfigured` | LLM provider is not configured on this wiki |
+| `notwikitext` | The page content is not wikitext |
 
 ## Debugging
 

extension.json

@@ -1,6 +1,6 @@
 {
   "name": "PandocUltimateConverter",
-  "version": "0.5.0",
+  "version": "0.6.0",
   "author": [
     "[https://www.mediawiki.org/wiki/User:Urfiner Urfiner] (Nikolai Kochkin)"
   ],
@@ -20,6 +20,7 @@
   },
   "APIModules": {
     "pandocconvert": "MediaWiki\\Extension\\PandocUltimateConverter\\Api\\ApiPandocConvert",
+    "pandocllmpolish": "MediaWiki\\Extension\\PandocUltimateConverter\\Api\\ApiPandocLlmPolish",
     "pandocurltitle": "MediaWiki\\Extension\\PandocUltimateConverter\\Api\\ApiPandocUrlTitle"
   },
   "ExtensionMessagesFiles": {
@@ -95,7 +96,8 @@
         "pandocultimateconverter-codex-overwrite-toggle",
         "pandocultimateconverter-codex-status-uploading",
         "pandocultimateconverter-codex-status-converting",
-        "pandocultimateconverter-codex-status-done",
+        "pandocultimateconverter-codex-status-done-converted",
+        "pandocultimateconverter-codex-status-done-polished",
         "pandocultimateconverter-codex-status-error",
         "pandocultimateconverter-codex-retry",
         "pandocultimateconverter-codex-column-source",
@@ -109,7 +111,12 @@
         "pandocultimateconverter-codex-convert-one",
         "pandocultimateconverter-codex-switch-classic",
         "pandocultimateconverter-codex-stop",
-        "pandocultimateconverter-codex-stop-requested"
+        "pandocultimateconverter-codex-stop-requested",
+        "pandocultimateconverter-codex-llm-polish-toggle",
+        "pandocultimateconverter-codex-llm-polish-btn",
+        "pandocultimateconverter-codex-status-polishing",
+        "pandocultimateconverter-codex-status-polish-error",
+        "pandocultimateconverter-conversion-complete-comment"
       ]
     },
     "ext.PandocUltimateConverter.export": {
@@ -211,6 +218,30 @@
       "path": false,
       "description": "When true, show an 'Export' action in the page tools menu (Actions tab) on content pages.",
       "public": true
+    },
+    "PandocUltimateConverter_LlmProvider": {
+      "value": null,
+      "path": false,
+      "description": "LLM provider to use for optional wikitext cleanup. Supported values: 'openai' or 'claude'. Leave null to disable LLM polishing.",
+      "public": false
+    },
+    "PandocUltimateConverter_LlmApiKey": {
+      "value": null,
+      "path": false,
+      "description": "API key for the configured LLM provider. Required when LlmProvider is set.",
+      "public": false
+    },
+    "PandocUltimateConverter_LlmModel": {
+      "value": null,
+      "path": false,
+      "description": "Model name to use for LLM polishing. Defaults to 'gpt-5.4-nano' for OpenAI and 'claude-3-5-haiku-20241022' for Claude.",
+      "public": false
+    },
+    "PandocUltimateConverter_LlmPrompt": {
+      "value": null,
+      "path": false,
+      "description": "Custom instruction prompt for the LLM cleanup step. Uses a sensible default if not set.",
+      "public": false
     }
   },
   "ConfigRegistry": {

i18n/ar.json

@@ -39,6 +39,8 @@
 	"apihelp-pandocconvert-param-forceoverwrite": "إذا تم التعيين، فسيتم استبدال الصفحة المستهدفة إن وُجدت.",
 	"apihelp-pandocconvert-example-file": "تحويل ملف مرفوع مسبقاً باسم Document.docx إلى الصفحة MyArticle.",
 	"apihelp-pandocconvert-example-url": "تحويل المحتوى من https://example.com إلى الصفحة MyArticle.",
+	"apihelp-pandocllmpolish-summary": "تشغيل تنظيف الذكاء الاصطناعي (LLM) على نص الويكي لصفحة موجودة.",
+	"apihelp-pandocllmpolish-param-pagename": "عنوان صفحة الويكي الموجودة المراد تحسينها.",
 	"pandocultimateconverter-codex-description": "أضف ملفات أو روابط URL لتحويلها إلى صفحات ويكي. يمكنك معالجة عناصر متعددة دفعةً واحدة.",
 	"pandocultimateconverter-codex-switch-classic": "التبديل إلى النموذج الكلاسيكي",
 	"pandocultimateconverter-codex-tab-files": "الملفات",
@@ -54,9 +56,20 @@
 	"pandocultimateconverter-codex-convert-all": "تحويل الكل",
 	"pandocultimateconverter-codex-clear-all": "مسح الكل",
 	"pandocultimateconverter-codex-overwrite-toggle": "السماح بالكتابة فوق الصفحات الموجودة",
+	"pandocultimateconverter-codex-llm-polish-toggle": "تحسين بالذكاء الاصطناعي (تنظيف LLM)",
+	"pandocultimateconverter-codex-llm-polish-btn": "تنظيف الذكاء الاصطناعي",
+	"pandocultimateconverter-codex-status-polishing": "جارٍ التنظيف بالذكاء الاصطناعي…",
+	"pandocultimateconverter-codex-status-polish-error": "فشل تنظيف الذكاء الاصطناعي: $1",
+	"pandocultimateconverter-llmpolish-comment": "تنظيف الذكاء الاصطناعي (LLM)",
+	"apierror-pandocllmpolish-pagenotfound": "الصفحة غير موجودة: $1",
+	"apierror-pandocllmpolish-notconfigured": "لم يتم تهيئة تنظيف LLM على هذه الويكي.",
+	"apierror-pandocllmpolish-notwikitext": "محتوى الصفحة ليس نص ويكي.",
+	"apierror-pandocllmpolish-failed": "فشل تنظيف LLM: $1",
 	"pandocultimateconverter-codex-status-uploading": "جارٍ الرفع…",
 	"pandocultimateconverter-codex-status-converting": "جارٍ التحويل…",
 	"pandocultimateconverter-codex-status-done": "تمّ",
+	"pandocultimateconverter-codex-status-done-converted": "اكتمل التحويل",
+	"pandocultimateconverter-codex-status-done-polished": "اكتمل تنظيف الذكاء الاصطناعي",
 	"pandocultimateconverter-codex-status-error": "خطأ: $1",
 	"pandocultimateconverter-codex-retry": "إعادة المحاولة",
 	"pandocultimateconverter-codex-column-source": "المصدر",

i18n/cs.json

@@ -39,6 +39,8 @@
 	"apihelp-pandocconvert-param-forceoverwrite": "Pokud je nastaveno, přepíše cílovou stránku, pokud již existuje.",
 	"apihelp-pandocconvert-example-file": "Převést již nahraný soubor Document.docx na stránku MyArticle.",
 	"apihelp-pandocconvert-example-url": "Převést obsah z https://example.com na stránku MyArticle.",
+	"apihelp-pandocllmpolish-summary": "Spustit vyčistění wikitextu existující stránky pomocí AI (LLM).",
+	"apihelp-pandocllmpolish-param-pagename": "Název existující wiki stránky k vylepšení.",
 	"pandocultimateconverter-codex-description": "Přidejte soubory nebo URL adresy k převodu na wiki stránky. Můžete zpracovat více položek najednou.",
 	"pandocultimateconverter-codex-switch-classic": "Přepnout na klasický formulář",
 	"pandocultimateconverter-codex-tab-files": "Soubory",
@@ -54,9 +56,20 @@
 	"pandocultimateconverter-codex-convert-all": "Převést vše",
 	"pandocultimateconverter-codex-clear-all": "Vymazat vše",
 	"pandocultimateconverter-codex-overwrite-toggle": "Povolit přepsání existujících stránek",
+	"pandocultimateconverter-codex-llm-polish-toggle": "Vylepšit pomocí AI (čištění LLM)",
+	"pandocultimateconverter-codex-llm-polish-btn": "Čištění AI",
+	"pandocultimateconverter-codex-status-polishing": "Čistění AI…",
+	"pandocultimateconverter-codex-status-polish-error": "Čištění AI selhalo: $1",
+	"pandocultimateconverter-llmpolish-comment": "Čištění AI (LLM)",
+	"apierror-pandocllmpolish-pagenotfound": "Stránka nenalezena: $1",
+	"apierror-pandocllmpolish-notconfigured": "Čištění LLM není nakonfigurováno na této wiki.",
+	"apierror-pandocllmpolish-notwikitext": "Obsah stránky není wikitext.",
+	"apierror-pandocllmpolish-failed": "Čištění LLM selhalo: $1",
 	"pandocultimateconverter-codex-status-uploading": "Nahrávání…",
 	"pandocultimateconverter-codex-status-converting": "Převádění…",
 	"pandocultimateconverter-codex-status-done": "Hotovo",
+	"pandocultimateconverter-codex-status-done-converted": "Převod dokončen",
+	"pandocultimateconverter-codex-status-done-polished": "Čištění AI dokončeno",
 	"pandocultimateconverter-codex-status-error": "Chyba: $1",
 	"pandocultimateconverter-codex-retry": "Zkusit znovu",
 	"pandocultimateconverter-codex-column-source": "Zdroj",

i18n/de.json

@@ -39,6 +39,8 @@
 	"apihelp-pandocconvert-param-forceoverwrite": "Wenn gesetzt, wird die Zielseite überschrieben, falls sie bereits existiert.",
 	"apihelp-pandocconvert-example-file": "Eine bereits hochgeladene Datei namens Document.docx in die Seite MyArticle konvertieren.",
 	"apihelp-pandocconvert-example-url": "Den Inhalt von https://example.com in die Seite MyArticle konvertieren.",
+	"apihelp-pandocllmpolish-summary": "KI-Bereinigung des Wikitexts einer bestehenden Wiki-Seite durchführen.",
+	"apihelp-pandocllmpolish-param-pagename": "Titel der bestehenden Wiki-Seite, die bereinigt werden soll.",
 	"pandocultimateconverter-codex-description": "Dateien oder URLs hinzufügen, um sie in Wiki-Seiten zu konvertieren. Sie können mehrere Elemente gleichzeitig verarbeiten.",
 	"pandocultimateconverter-codex-switch-classic": "Zum klassischen Formular wechseln",
 	"pandocultimateconverter-codex-tab-files": "Dateien",
@@ -54,9 +56,20 @@
 	"pandocultimateconverter-codex-convert-all": "Alle konvertieren",
 	"pandocultimateconverter-codex-clear-all": "Alle entfernen",
 	"pandocultimateconverter-codex-overwrite-toggle": "Überschreiben bestehender Seiten erlauben",
+	"pandocultimateconverter-codex-llm-polish-toggle": "Mit KI überarbeiten (LLM-Bereinigung)",
+	"pandocultimateconverter-codex-llm-polish-btn": "KI-Bereinigung",
+	"pandocultimateconverter-codex-status-polishing": "KI-Bereinigung…",
+	"pandocultimateconverter-codex-status-polish-error": "KI-Bereinigung fehlgeschlagen: $1",
+	"pandocultimateconverter-llmpolish-comment": "KI-Bereinigung (LLM)",
+	"apierror-pandocllmpolish-pagenotfound": "Seite nicht gefunden: $1",
+	"apierror-pandocllmpolish-notconfigured": "Die KI-Bereinigung ist für dieses Wiki nicht konfiguriert.",
+	"apierror-pandocllmpolish-notwikitext": "Der Seiteninhalt ist kein Wikitext.",
+	"apierror-pandocllmpolish-failed": "KI-Bereinigung fehlgeschlagen: $1",
 	"pandocultimateconverter-codex-status-uploading": "Wird hochgeladen…",
 	"pandocultimateconverter-codex-status-converting": "Wird konvertiert…",
 	"pandocultimateconverter-codex-status-done": "Fertig",
+	"pandocultimateconverter-codex-status-done-converted": "Konvertierung abgeschlossen",
+	"pandocultimateconverter-codex-status-done-polished": "KI-Bereinigung abgeschlossen",
 	"pandocultimateconverter-codex-status-error": "Fehler: $1",
 	"pandocultimateconverter-codex-retry": "Erneut versuchen",
 	"pandocultimateconverter-codex-column-source": "Quelle",

i18n/en.json

@@ -45,6 +45,9 @@
     "apihelp-pandocconvert-example-file": "Convert an already-uploaded file named Document.docx to the page MyArticle.",
     "apihelp-pandocconvert-example-url": "Convert the content at https://example.com to the page MyArticle.",
 
+    "apihelp-pandocllmpolish-summary": "Run LLM AI cleanup on the wikitext of an existing wiki page.",
+    "apihelp-pandocllmpolish-param-pagename": "Title of the existing wiki page to polish.",
+
     "pandocultimateconverter-codex-description": "Add files or URLs to convert into wiki pages. You can process multiple items at once.",
 	"pandocultimateconverter-codex-switch-classic": "Switch to classic form",
     "pandocultimateconverter-codex-tab-files": "Files",
@@ -60,9 +63,21 @@
     "pandocultimateconverter-codex-convert-all": "Convert all",
     "pandocultimateconverter-codex-clear-all": "Clear all",
     "pandocultimateconverter-codex-overwrite-toggle": "Allow overwriting existing pages",
+    "pandocultimateconverter-codex-llm-polish-toggle": "Polish with AI (LLM cleanup)",
+    "pandocultimateconverter-codex-llm-polish-btn": "AI cleanup",
+    "pandocultimateconverter-codex-status-polishing": "AI cleanup…",
+    "pandocultimateconverter-codex-status-polish-error": "AI cleanup failed: $1",
+    "pandocultimateconverter-llmpolish-comment": "LLM AI cleanup",
+    "apierror-pandocllmpolish-pagenotfound": "Page not found: $1",
+    "apierror-pandocllmpolish-notconfigured": "LLM polish is not configured on this wiki.",
+    "apierror-pandocllmpolish-notwikitext": "The page content is not wikitext.",
+    "apierror-pandocllmpolish-failed": "LLM polish failed: $1",
+
     "pandocultimateconverter-codex-status-uploading": "Uploading…",
     "pandocultimateconverter-codex-status-converting": "Converting…",
     "pandocultimateconverter-codex-status-done": "Done",
+    "pandocultimateconverter-codex-status-done-converted": "Conversion done",
+    "pandocultimateconverter-codex-status-done-polished": "AI cleanup done",
     "pandocultimateconverter-codex-status-error": "Error: $1",
     "pandocultimateconverter-codex-retry": "Retry",
     "pandocultimateconverter-codex-column-source": "Source",
@@ -71,7 +86,7 @@
     "pandocultimateconverter-codex-column-actions": "Actions",
     "pandocultimateconverter-codex-navigate-warning": "Conversion is in progress. Are you sure you want to leave?",
     "pandocultimateconverter-codex-invalid-url": "Invalid URL (only http and https are supported)",
-    "pandocultimateconverter-codex-convert-one": "Convert this item",
+    "pandocultimateconverter-codex-convert-one": "Convert",
 	"pandocultimateconverter-codex-remove-item": "Remove",
 	"pandocultimateconverter-codex-stop": "Stop",
 	"pandocultimateconverter-codex-stop-requested": "Stopping…",