[Share] Convert Annotated Text to Markdown

[TideDra]

Description

This script converts annotated text (highlight, underlined) which may contain math expressions to markdown via Multimodal LLM. The script needs to run external python file to process image and query the MLLM.

Before:

After:

This script is similar to #220 , but is more powerful. #220 can only convert image annotation in a note editor, while this script directly converts text annotation and replace it. Moreover, MLLM is much more accurate, flexible and cheap than traditional Math OCR tools, which can also recognize a mixture of plain text and math expressions.

Event

Creat Annotation

Operation

Script

Data

/**
 * Convert Annotated Text to Markdown
 * @author Zotero Community
 * @usage 
 * @link https://github.com/windingwind/zotero-actions-tags/discussions/
 * @see https://github.com/windingwind/zotero-actions-tags/discussions/
 */
if (!item) return;
if (item.annotationType=='image') return
if (item.annotationAuthorName == "FormulaExtractor") return;

const annotationPosition = JSON.parse(item.annotationPosition)
const pageIndex = annotationPosition.pageIndex
const rects = annotationPosition.rects

const min_x = Math.min(...(rects.map(rect=>{return rect[0]})))
const min_y = Math.min(...(rects.map(rect=>{return rect[1]})))
const max_x = Math.max(...(rects.map(rect=>{return rect[2]})))
const max_y = Math.max(...(rects.map(rect=>{return rect[3]})))
const rect = [min_x,min_y,max_x,max_y]
const Zotero_Tabs = require("Zotero_Tabs")
await generateImageAnnotation(Zotero, Zotero_Tabs, pageIndex, rect)
const img_path = await Zotero.Annotations.getCacheImagePath(temp_item)
const python_path = "/opt/homebrew/Caskroom/miniconda/base/bin/python"
const script_path = "/Users/geary.z/Zotero/IMG2MD.py"
const sleep = (delay) => new Promise((resolve) => setTimeout(resolve, delay))
while(!await Zotero.Annotations.hasCacheImage(temp_item)){
  await sleep(100)
}
let response = await Zotero.Utilities.Internal.subprocess(python_path,[script_path,img_path,JSON.stringify(rects)])

try {response = JSON.parse(response)}
catch (error) {}

Zotero.Annotations.removeCacheImage(temp_item)
temp_item.eraseTx()

const success = response.hasOwnProperty("choices")

if (success) {
  const result = response.choices[0].message.content
  item.annotationText = result.replace(/\$\$(.*?)\$\$/g, '$$\n$1\n$$');;
  return `Convert Successfully`;
} else {
  return `Error! Get response: ${JSON.stringify(response)}`;
}


function _generateObjectKey() {
  let len = 8;
  let allowedKeyChars = '23456789ABCDEFGHIJKLMNPQRSTUVWXYZ';

  var randomstring = '';
  for (var i = 0; i < len; i++) {
    var rnum = Math.floor(Math.random() * allowedKeyChars.length);
    randomstring += allowedKeyChars.substring(rnum, rnum + 1);
  }
  return randomstring;
}

async function generateImageAnnotation(Zotero, Zotero_Tabs, pageIndex, rect) {
  const reader = Zotero.Reader.getByTabID(Zotero_Tabs._tabs[Zotero_Tabs.selectedIndex].id)
  const attachment = reader._item
  const annotation = {
    type: 'image',
    color: "#f0bbcd",
    pageLabel: pageIndex + 1,
    position: {
      pageIndex: pageIndex,
      rects: [rect]
    }
  };
  annotation.sortIndex = "00000|000000|00000"
  annotation.key = _generateObjectKey();
  // reader._internalReader._annotationManager.addAnnotation(annotation)
  
  annotation.pageLabel = annotation.pageLabel || '';
  annotation.text = annotation.text || '';
  annotation.comment = "";
  annotation.tags = [];
  // Automatically set properties
  annotation.id = _generateObjectKey();
  annotation.dateCreated = (new Date()).toISOString();
  annotation.dateModified = annotation.dateCreated;
  annotation.authorName = "FormulaExtractor";
  annotation.isAuthorNameAuthoritative = false;
  
  temp_item = await Zotero.Annotations.saveFromJSON(attachment, annotation);
  
  reader._internalReader._primaryView._render();
  return temp_item
}

Anything else

The python file IMG2MD.py is:

from PIL import Image
import sys
import json
import requests
import os
import base64
import requests

#  base 64 编码格式
def encode_image(image_path):
  with open(image_path, "rb") as image_file:
    return base64.b64encode(image_file.read()).decode('utf-8')

def get_response(image_path):
    base64_image = encode_image(image_path)
    endpoint = "https://dashscope.aliyuncs.com/compatible-mode/v1/chat/completions"
    api_key = "YourAPIKEY"
    model = "qwen-vl-max"
    headers = {
       "Content-Type": "application/json",
       "Authorization": f"Bearer {api_key}"
       }
    payload = {
        "model": model,
        "messages": [
            {
            "role": "system",
            "content": [
                {
                "type": "text",
                "text": "You are an expert in converting images to markdown. Please help the user to convert the contents in given image to markdown. Wrap inline math expressions with a pair of single dollar symbol like '$...$', and wrap standalone math expressions between lines with a pair of double dollar symbols like '$$\\n...\\n$$', just like the common practice in LaTeX."
                }
            ]
            },
            {
            "role": "user",
            "content": [
                {
                "type": "image_url",
                "image_url": {
                    "url": f"data:image/jpeg;base64,{base64_image}"
                }
                },
                {
                "type": "text",
                "text": "Please convert the contents in this image to markdown."
                }
            ]
            }
        ]
        }
    response = requests.post(endpoint, headers=headers, json=payload)
    return response.text

if __name__ == "__main__":
    img_path, rects = sys.argv[1], sys.argv[2]
    rects = eval(rects)
    min_x = 1e9
    min_y = 1e9
    max_x = -1e9
    max_y = -1e9
    for rect in rects:
        min_x = min(min_x, rect[0])
        min_y = min(min_y, rect[1])
        max_x = max(max_x, rect[2])
        max_y = max(max_y, rect[3])
    #normalize rects to [0,1]
    for rect in rects:
        rect[0] = (rect[0] - min_x) / (max_x - min_x)
        rect[1] = (rect[1] - min_y) / (max_y - min_y) 
        rect[2] = (rect[2] - min_x) / (max_x - min_x) 
        rect[3] = (rect[3] - min_y) / (max_y - min_y) - 0.007
    
    img = Image.open(img_path).transpose(Image.FLIP_TOP_BOTTOM) #coordinate system is different
    blank_board = Image.new('RGB', img.size, (255, 255, 255))
    for rect in rects:
        target_rect = [rect[0] * img.size[0], rect[1] * img.size[1], rect[2] * img.size[0], rect[3] * img.size[1]]
        target_rect = [int(v) for v in target_rect]
        target_pixel = img.crop(target_rect)
        blank_board.paste(target_pixel, target_rect)
    blank_board = blank_board.transpose(Image.FLIP_TOP_BOTTOM)
    blank_board.save(img_path)

    response = get_response(img_path)
    print(response)

Please replace the endpoint, api_key and model to your owns that supports OpenAI request format. But note that the prompt is designed for Qwen-VL-max.

And make sure the required python packages are installed.

Each request takes 5~10 seconds, depending on the length of text.

[TideDra]

The best practice is to use this script together with the note template windingwind/zotero-better-notes#1139 , which generates note from annotations and render the markdown annotations correctly.

[wzr-ms]

我来评论，这个action就是配置老是有问题，一直无法正常运行脚本

[TideDra]

可能是环境问题。我是zotero7.0.6-beta.2+b4239fd4c，macOS。我手头没有windows设备，不太好复现你的问题。你可以用zotero的javascript调试器测试一下Zotero.Utilities.Internal.subprocess能不能用。

[wzr-ms]

我采用作者大大改的笔记，还挺好用的，就是这个问题有点费脑，逼得我只能用最初的OCR。混合识别也能实现就是准确率有待提升