OS AI Agent

This repo contains OmniSight, a small desktop overlay that lets you ask Gemini questions about whatever is currently on your screen.

The app is built for practical desktop work. You can freeze the full screen, snip a smaller region, compare the current capture with the previous one, and keep a live screen snapshot ready in the background so the assistant has fresh context.

Screenshot

What It Does

• Captures the full desktop across multiple displays
• Lets you snip a smaller area when you only want to focus on one part of the screen
• Streams Gemini responses inside the app instead of waiting for a full block of text
• Keeps a previous capture so you can ask questions like "what changed?"
• Supports a global hotkey for fast screen freezing
• Stores user settings locally so the selected model, hotkey, and live-monitor interval persist

Stack

• Python
• PyQt6 + Qt WebEngine for the desktop UI
• mss and Pillow for screen capture and image handling
• Google Gemini via google-generativeai
• qasync to bridge Qt and asyncio
• PyInstaller for packaging

Tested with Python 3.13.

Current Scope

This project was built and tested as a desktop app first, with the smoothest experience on macOS.

macOS users should expect to grant:

• Screen Recording permission for capture
• Accessibility permission for the global hotkey

There is code for Windows and Linux paths, but the overlay behavior and permission handling were clearly written with macOS as the main target.

Getting Started

1. Create a virtual environment

python3 -m venv .venv
.venv/bin/pip install -r requirements.txt

2. Add your Gemini API key

cp .env.example .env

Then edit .env and set:

GEMINI_API_KEY=your_api_key_here

You can also override the default model there if you want.

3. Run the app

.venv/bin/python main.py

Packaging

The project includes a PyInstaller spec:

.venv/bin/pyinstaller --clean omnisight.spec

The build output goes to dist/.

The spec intentionally does not bundle your real .env file. If you run a packaged build, keep your runtime .env available where the app can read it at startup. On macOS, the simplest place is inside the app bundle's Contents/Resources directory.

How It Works

When you send a prompt, OmniSight captures the current desktop, optionally includes the previous capture for comparison, and uploads the relevant screenshots to Gemini. The app also creates per-display detail crops so smaller code or UI text is easier for the model to inspect.

Conversation history is kept in memory for the active session. App settings and logs are written to a user-writable app data directory:

• macOS: ~/Library/Application Support/OmniSight
• Windows: %APPDATA%\OmniSight
• Linux: ${XDG_CONFIG_HOME:-~/.config}/omnisight

Privacy Notes

This app sends screenshots to Google's Gemini API when you ask a question. That is the core behavior of the tool, so it is worth stating plainly: if sensitive material is on screen, it can be included in what gets sent upstream.

The app shows a privacy warning on first launch, but you should still treat it like any other cloud-backed screen analysis tool and use it carefully.

Project Layout

• main.py wires together the UI, capture flow, config handling, and app lifecycle
• ui.py contains the PyQt overlay and snipping interface
• capture.py handles multi-monitor capture and permission checks
• ai_client.py manages Gemini uploads and streaming responses
• omnisight.spec defines the packaged desktop build

Notes

• The UI name is OmniSight throughout the code and application window.
• The repo name stayed broader, but the desktop app itself is OmniSight.