data-ai MCP Server
Autonomous GUI agent — give it a task, it operates the desktop. Visual memory, one-shot UI learning. | 自主GUI代理——给它一个任务，它操作桌面。视觉记忆，一次学习即可操作。
Discovered via agent-topic:ai-agent and last synced 3mo ago.
Install instructions not detected yet
Check the source repository for the latest setup steps.
OpenClaw
OpenAI API
Claude Code CLI
Finds
Provider
Anthropic API
Full results: [benchmarks/osworld/multi_apps.md](benchmarks/osworld/multi_apps.md) ## Quick Start ### Step 1: Install GUI Agent Harness > **This harness is an OpenProgram program — it runs *inside* OpenProgram, > not on its own.** Install OpenProgram first, then add this harness to it. > (Installing this repo by itself will import-error on `openprogram`.) ```bash # 1. Install the host (one step, all platforms) pip install openprogram # 2. Add this harness — clones it into OpenProgram's functions/agentics/ # and installs the harness's own heavy deps (ultralytics→torch, OpenCV, # Pillow, pynput …). Restart OpenProgram; it's auto-detected and # `gui_agent` becomes available. openprogram programs install gui ``` > **Platform note:** the **core action layer** — screen capture, mouse/keyboard > input, window focus, and clipboard — runs on **macOS, Windows, and Linux** > (screenshots via Pillow `ImageGrab`, input via `pynput`, window control via > the Win32 API on Windows / `wmctrl`+`xdotool` on Linux). The **advanced visual > perception** (Apple Accessibility window introspection and Apple Vision OCR) is > macOS-tuned; on Windows/Linux the agent falls back to YOLO UI detection + > EasyOCR. HiDPI display scaling is handled automatically — the process is made > per-monitor DPI-aware on Windows and screenshot↔click coordinates are unified, > so clicks land correctly on scaled (125% / 150% / …) displays. <details> <summary><b>Manual install / local development</b></summary> `programs install gui` just clones into OpenProgram's agentics folder and pip-installs the clone (resolving this repo's own `pyproject.toml` deps). By hand, or to develop in place: ```bash # find OpenProgram's agentics folder AGENTICS=$(python -c "import openprogram,os;print(os.path.join(os.path.dirname(openprogram.__file__),'functions','agentics'))") # clone in (a real directory — no symlink; works on Windows too) git clone https://github.com/Fzkuji/GUI-Agent-Harness "$AGENTICS/GUI-Agent-Harness" # install this harness's deps (cv2 / ultralytics / … from its pyproject) pip install "$AGENTICS/GUI-Agent-Harness" ``` Note: `openprogram` is the **host**, not a dependency of this repo — it's always present at runtime (the harness loads from inside it), so this repo's `pyproject.toml` does not pin it. Installing this repo standalone (without OpenProgram) will `pip install` fine but fail at import with `ModuleNotFoundError: openprogram`. </details> ### Step 2: Set up an LLM provider GUI Agent Harness needs an LLM to make decisions. Install at least one provider: **Option A: Claude Code CLI (recommended)** ```bash npm install -g @anthropic-ai/claude-code claude login ``` Uses your Claude subscription — no per-token cost. The agent runs as `claude -p` under the hood. **Option B: Anthropic API** ```bash export ANTHROPIC_API_KEY=sk-ant-... ``` Pay-per-token. Set the key in your shell profile for persistence. **Option C: OpenAI API** ```bash export OPENAI_API_KEY=sk-... ``` The system auto-detects the best available provider. You can also force one with `--provider`. ### Step 3: Platform setup **macOS:** - Grant accessibility permissions: System Settings → Privacy & Security → Accessibility → add your Terminal app - Apple Vision OCR works automatically (no extra install) **Linux:** - Install EasyOCR for text detection: `pip install easyocr` - Window focus / bounds use `wmctrl` / `xdotool` — install via your package manager. **Windows:** - Install EasyOCR for text detection: `pip install easyocr` - Window focus / list / bounds use the built-in Win32 API (no extra install). - Visual perception is macOS-tuned (Windows uses YOLO + EasyOCR). HiDPI display scaling is auto-detected, so clicks land correctly on scaled displays. ### Step 4: Run ```bash # Local desktop gui-agent --work-dir /private/tmp/gui-agent-firefox --app firefox "Open Firefox and go to google.com" # Remote VM (e.g., OSWorld) gui-agent --work-dir /private/tmp/gui-agent-vm --vm http://VM_IP:5000 "Install the Orchis GNOME theme" # Specify provider and model gui-agent --work-dir /private/tmp/gui-agent-wechat --provider claude-code --model opus --app wechat "Send hello in WeChat" ``` ### Use as LLM skill GUI Agent Harness is designed to be called by an LLM as a tool. After `pip install`, register the project as a skill so your LLM can discover and use it. LLM skill systems typically scan a skills directory for subdirectories containing a `SKILL.md` file. To register GUI Agent Harness, copy or symlink the project into your LLM's skills directory: ```bash # Example: copy into OpenClaw's skills directory cp -r GUI-Agent-Harness ~/.openclaw/skills/gui-agent # Or symlink (recommended — stays in sync with git) ln -s /path/to/GUI-Agent-Harness ~/.openclaw/skills/gui-agent ``` **Claude Code** auto-discovers `SKILL.md` from the current working directory or configured skill paths: ```bash # Option 1: work from the project directory (auto-discovered) cd /path/to/GUI-Agent-Harness # Option 2: add to Claude Code's skill search paths claude config set skillPaths '["<path-to-GUI-Agent-Harness>"]' ``` Once registered, the LLM reads `SKILL.md` and knows when and how to call `gui-agent` — no further configuration needed. ## CLI Options ``` gui-agent [OPTIONS] TASK Arguments: TASK Natural language task description Options: --work-dir PATH Required. Runtime working directory for file writes/commands --vm URL Remote VM HTTP API (e.g., http://172.16.82.132:5000) --provider NAME LLM provider: auto, claude-code, openclaw, anthropic, openai --model NAME Override model name (e.g., opus, sonnet, gpt-4o) --max-steps N Max actions before stopping (default: 15) --app NAME App name for component memory (default: desktop) --no-general Disable command-line fallback; use GUI actions only ``` Detailed Chinese usage notes, including local app, VM, OSWorld, and Python-call examples, are in [docs/USAGE_CN.md](docs/USAGE_CN.md). ## Architecture ``` gui-agent --work-dir /path/to/work-dir "task description" │ ▼ gui_agent() ← @agentic_function, drives the loop │ ├── for step in 1..max_steps: │ │ │ ▼ │ gui_step() ← @agentic_function, orchestration │ │ │ ├── 1. Observe (Python) — screenshot + detect + match + state ID │ ├── 2. Verify (LLM) — check previous action's result │ ├── 3. Plan (LLM) — decide next action │ └── 4. Dispatch (Python) — execute: click/type/scroll/general │ │ │ ▼ │ build_step_feedback() ← structured result → next iteration │ └── return result summary ``` **Observe** — Pure Python. Takes a screenshot, runs GPA-GUI-Detector + OCR, matches against stored component templates, identifies the current UI state. **Verify** — LLM call. Examines the screenshot after the previous action. Reports whether the action succeeded. Does not decide task completion. **Plan** — LLM call. Sees the screenshot, detected components, verification result, and known state transitions. Chooses one action (click, type, scroll, general, done). **Dispatch** — Pure Python. Executes the planned action. For clicks, uses template matching to find precise coordinates. For `general`, delegates to the LLM with full tool access (Bash, file I/O, etc.). ## Visual Memory When a UI element is first detected, it gets a **dual representation**: a cropped visual template (for fast matching) and a VLM-assigned label (for reasoning). Stored per-app, reused across all future sessions. ``` memory/ ├── linux/ # Platform-specific memory │ └── apps/ │ ├── desktop/ # General desktop components │ ├── chromium/ # Browser UI │ │ └── sites/ # Per-website memory │ ├── gimp/ │ └── libreoffice-calc/ │ ├── components.json # Component registry │ ├── states.json # UI states (component sets) │ ├── transitions.json # State graph edges │ └── components/ # Template images ``` **Activity-based forgetting** — Components track consecutive misses. After 15 misses, auto-removed. Keeps memory aligned with the app's current UI. **State matching** — States are sets of visible components, matched by Jaccard similarity (>0.7 = same state, >0.85 = auto-merge). ## Detection Stack
Value
Samples
Tasks
46
91
Windows
none