vmware_learn_qupath.yaml

# Formerly called: anthropic_filemap.yaml
# This template is heavily inspired by anthropic's computer use demo, but you can use
# it with any LM.
agent:
  run_mode: swe_gui
  max_trajectory_length: 100
  use_workflow_memory: true
  update_workflow_memory: true
  templates:
    system_template: |-
      You are a helpful assistant that can interact with a computer to solve tasks. You are good at understanding pathology figures, using softwares and writing code like a biologist.
    instance_template: |-
      <working_dir>
      {{working_dir}}
      </working_dir>
      Consider the following problem statement:

      <pr_description>
      {{problem_statement}}
      </pr_description>

      Your goal is to learn how to use QuPath effectively to analyze pathology images, including tasks such as image annotation, segmentation, and measurement, etc.

      Follow these steps to learn from a tutorial:
      1. Choose a tutorial that you have not mastered its workflow yet.
      2. Carefully read the tutorial and understand the steps involved. Select an image in the working directory to practice the steps.
      3. Try to follow the steps in the tutorial using QuPath using GUI interface. If you encounter any issues, troubleshoot them and try to resolve them.
      4. Reach key progress milestones as evidence of your learning. 
      5. After completing the tutorial, summarize what you have learned and how you can apply it to future tasks, including any challenges you faced and how you overcame them.
      6. Once you have mastered any one of the tutorials, exit the agent with action "FINISH".
      7. Update your learned skills in the workflow file `learned_workflow.txt`. Remember to include tricky action code snippets in the workflow for future reference. You must update workflows after each subagent session.

      Your thinking should be thorough and so it's fine if it's very long. Whenever you view a image or a pdf, you must analyze it and write down your key observations in the subsequent thinking section, before proceeding to the next step.
    
    next_step_template: |-
      OBSERVATION:
      {{observation}}
    next_step_no_output_template: |-
      Your command ran successfully and did not produce any output.
  tools:
    type: vmware
    execution_timeout: 600
    env_variables:
      PAGER: cat
      MANPAGER: cat
      LESS: -R
      PIP_PROGRESS_BAR: 'off'
      TQDM_DISABLE: '1'
      GIT_PAGER: cat
      LOCAL_DEV_PATH: "/home/user/Documents/WorkingDir"
    bundles:
      - path: tools/registry
      - path: tools/edit_anthropic
      # # - path: tools/image_tools
      # - path: tools/cellprofiler
      # - path: tools/napari_viewer_tools
      - path: tools/pdf_tools
      # - path: tools/module_search
    # Enable subagents for GUI software automation
    enable_subagents: true
    subagent_bundles: 
      - path: subagents/qupath
    registry_variables:
      USE_FILEMAP: 'true'
      SUBMIT_REVIEW_MESSAGES:
        - |
          Thank you for your work on this issue. Please carefully follow the steps below to help review your changes.

          1. If you made any changes to your code after running the reproduction script, please run the reproduction script again.
            If the reproduction script is failing, please revisit your changes and make sure they are correct.
            If you have already removed your reproduction script, please ignore this step.
          2. Remove your reproduction script (if you haven't done so already).
          3. If you have modified any TEST files, please revert them to the state they had before you started fixing the issue.
            You can do this with `git checkout -- /path/to/test/file.py`. Use below <diff> to find the files you need to revert.
          4. Run the submit command again to confirm.

          Here is a list of all of your changes:

          <diff>
          {{diff}}
          </diff>
    enable_bash_tool: true
    parse_function:
      type: function_calling
  history_processors:
    - type: keep_important
      last_n_messages: 50
      last_n_tagged_messages: 3

  # Desktop agent configuration for GUI automation
  desktop_agent:
    platform: "ubuntu"
    model: "claude-sonnet-4-5-20250929"
    max_tokens: 1500
    top_p: 0.9
    temperature: 0.5
    action_space: "computer_13"  # "computer_13" or "pyautogui"
    observation_type: "screenshot"
    max_trajectory_length: 5
    a11y_tree_max_tokens: 10000

  reflection_template: |
    PREVIOUS ATTEMPT ANALYSIS - QuPath Learning

    Based on the given trajectory, provide a structured reflection on:

    1. Workflow Learned: Summarize what QuPath workflow or techniques were learned
    2. QUALITY ASSESSMENT: Give a detailed assessment of the quality of the learning attempt, including what was done well and what could be improved
    3. NEXT STEPS: Specific improvements and things to learn for the next attempt

    Provide a concise technical summary and reflection focusing on issues and improvements:

  summary_template: |
    FINAL ATTEMPT SUMMARY - QuPath Learning

    Based on the final trajectory, provide a structured summary in this format: 

    Tutorials completed and their purposes:
    - Tutorial 1: Purpose
    - Tutorial 2: Purpose
    - ...

    Summary of things learned:
    - Key point 1
    - Key point 2
    - ...
  
  workflow_template: |
    WORKFLOW EXTRACTION - QuPath using

    Given the task as well as agent trajectory, your task is to extract the common workflows to solve this type of task.
    Try to generate as many workflows that can cover all the tasks and the subtasks covered in the input.

    ## Here are some examples of workflows as reference:

    Workflow - Image Navigating and Visualization
    [Thoughts]:Effective navigation is crucial for large whole slide images. Different zoom levels reveal different information (annotations vs detections). Overlay controls are essential for validating analysis results

    Steps:
    1. Use Move tool (M key) for panning around large images
    2. Zoom in/out with scroll wheel or double-click magnification value for precise levels
    3. Click overview thumbnail (top right) to jump quickly to different regions
    4. Toggle overlay displays using toolbar buttons (annotations, detections, names, measurements)
    5. Adjust brightness/contrast in right panel for optimal visualization
    6. Use scalebar to verify correct image scaling and measurements

    Workflow - ROI Creation and Annotation
    [Thoughts]:Creating accurate ROIs is fundamental for analysis. QuPath offers multiple tools for different shapes and precision. Annotations help label regions for training or reference.
    
    Steps:
    1. Select appropriate annotation tool (Rectangle, Ellipse, Polygon, Brush, Wand)
    2. Create annotations around regions of interest (tissue areas, specific structures)
    3. Use Brush tool in additive/subtractive mode (Alt+drag) to refine regions
    4. Lock annotations (right-click) to prevent accidental modification
    5. Classify annotations if needed for different tissue types or regions
    6. Resolve hierarchy (Objects → Annotations → Resolve hierarchy) to establish relationships

    Based on the above examples, please extract more workflows that can cover all the tasks and subtasks in the input trajectory.

  evaluation:
    type: cellprofiler
    ground_truth_dir: "/Users/machang/Documents/research-work/CellMMAgent/metrics/ground_truth"
    compare_dir: "/Users/machang/Documents/research-work/CellMMAgent/CellDev/measurements"
    threshold: 0.5

  retry_loop:
    type: score
    max_attempts: 1
    accept_score: 0.8
    cost_limit: 20.0
    max_trajectory_length: 100
    model:
      name: claude-sonnet-4-20250514
      per_instance_cost_limit: 50.0
      temperature: 0.0
      max_output_tokens: 500
    reviewer_config:
      type: reviewer
      system_template: "You are a helpful assistant."
      instance_template: |
        This is a tutorial learning task, therefore the score is not important and act as a placeholder.
        The goal is to provide a score between 0.0 and 1.0 based on the unit test results in format <score>{your score here}</score>.

        Unit test results:
        {{test}}

        Generate Review:
      traj_formatter:
        filter: []
        output_filter: []
        item_template: "Model: {{response}}\n\nObservation: {{observation}}"
        only_show_last_n_output: 0