web MCP Server
Autonomous AI Agent SDK for React Native & Expo — AI reads your live UI, acts via natural language, real-time voice agent (Gemini Live), and AI-powered testing via MCP (Model Context Protocol). One component. Zero config.
Discovered via github-topic:model-context-protocol and last synced 3mo ago.
Install instructions not detected yet
Check the source repository for the latest setup steps.
Mechanism
Description
`Record<string, string>`
`number`
Default
`string`
User query
--- ## 💬 Human Support Mode Transform the AI agent into a production-grade support system. The AI resolves issues directly inside your app UI — no backend API integrations required. When it can't help, it escalates to a live human agent. ```tsx import { buildSupportPrompt, createEscalateTool } from '@mobileai/react-native'; <AIAgent analyticsKey="mobileai_pub_xxx" // required for MobileAI escalation instructions={{ system: buildSupportPrompt({ enabled: true, greeting: { message: "Hi! 👋 How can I help you today?", agentName: "Support", }, quickReplies: [ { label: "Track my order", icon: "📦" }, { label: "Cancel order", icon: "❌" }, { label: "Talk to a human", icon: "👤" }, ], escalation: { provider: 'mobileai' }, csat: { enabled: true }, }), }} customTools={{ escalate: createEscalateTool({ provider: 'mobileai' }) }} userContext={{ userId: user.id, name: user.name, email: user.email, plan: 'pro', }} > <App /> </AIAgent> ``` ### What Happens on Escalation 1. AI creates a ticket in the **MobileAI Dashboard** inbox 2. User receives a real-time live chat thread (WebSocket) 3. Support agent replies — user sees messages instantly 4. Ticket is closed when resolved — a CSAT survey appears ### Escalation Providers
`string`
`boolean`
`ScreenMap`
`(content: string) => string`
`{ userId?, name?, email?, plan?, custom? }`
`(question) => Promise<string>`
Duration, event count
User reaction to AI controlling UI
`3100`
`string`
`number`
`Record<string, (payload: Record<string, unknown>) => void>`
`ProactiveHelpConfig`
Before task execution starts
`boolean`
`boolean`
What it does
2 (done, query_knowledge)
`string`
`boolean`
`KnowledgeEntry[] \
`React.RefObject<any>[]`
`string`
After each step (with full history)
Screen name, previous screen
`boolean`
Price consistency (list → detail)
`3101`
`NavigationContainerRef`
`boolean`
`{ system?, getScreenInstructions? }`
After task completes (success or failure)
Who
When
`boolean`
--- ## 📦 Installation Install the public React Native SDK: ```bash npm install @mobileai/react-native ``` Requires React Native `>=0.83.0 <0.84.0` and works with **Expo managed workflow** through a development build or prebuild. The base package includes native modules for screenshot capture and the elevated overlay, so Expo Go is not supported after installing it. ```bash npx expo prebuild npx expo run:ios npx expo run:android ``` For React Native CLI apps, run the normal native install/build step after installing the package, such as `cd ios && pod install`. ### Screenshot Capture <details> <summary><b>📸 Screenshots</b> — for image/video content understanding</summary> `react-native-view-shot` is a required native dependency for screenshot capture and is included with `@mobileai/react-native`, so you do **not** need to add it separately. Rebuild the native app after install so it can be autolinked. </details> <details> <summary><b>🎙️ Speech-to-Text in Text Mode</b> — dictate messages instead of typing</summary> ```bash npx expo install expo-speech-recognition ``` Automatically detected. No extra config needed — a mic icon appears in the text chat bar, letting users speak their message instead of typing. This is separate from voice mode. </details> <details> <summary><b>🎤 Voice Mode</b> — real-time bidirectional voice agent</summary> ```bash npm install react-native-audio-api ``` **Expo Managed** — add to `app.json`: ```json { "expo": { "android": { "permissions": ["RECORD_AUDIO", "MODIFY_AUDIO_SETTINGS"] }, "ios": { "infoPlist": { "NSMicrophoneUsageDescription": "Required for voice chat with AI assistant" } } } } ``` Then rebuild: `npx expo prebuild && npx expo run:android` (or `run:ios`) **Expo Bare / React Native CLI** — add `RECORD_AUDIO` + `MODIFY_AUDIO_SETTINGS` to `AndroidManifest.xml` and `NSMicrophoneUsageDescription` to `Info.plist`, then rebuild. > Hardware echo cancellation (AEC) is automatically enabled — no extra setup. </details> <details> <summary><b>💬 Human Support & Ticket Persistence</b> — persist tickets and discovery tooltip state across sessions</summary> ```bash npx expo install @react-native-async-storage/async-storage ``` **Optional** but recommended when using: - **Human escalation support** — tickets survive app restarts - **Discovery tooltip** — remembers if the user has already seen it Without it, both features gracefully degrade: tickets are only visible during the current session, and the tooltip shows every launch instead of once. </details> --- ## 🚀 Quick Start ### 1. Enable Screen Mapping (optional, recommended) Add one line to your `metro.config.js` — the AI gets a map of every screen in your app, auto-generated on each dev start: ```js // metro.config.js require('@mobileai/react-native/generate-map').autoGenerate(__dirname); ``` Or generate it manually anytime: ```bash npx @mobileai/react-native generate-map ``` > Without this, the AI can only see the currently mounted screen — it has no idea what other screens exist or how to reach them. Example: *"Write a review for the Laptop Stand"* — the AI sees the Home screen but doesn't know a `WriteReview` screen exists 3 levels deep. With a map, it sees every screen in your app and knows exactly how to get there: `Home → Products → Detail → Reviews → WriteReview`. ### 2. Wrap Your App If you use a MobileAI publishable key, the SDK now defaults to the hosted MobileAI text and voice proxies automatically. You only need to pass `proxyUrl` and `voiceProxyUrl` when you want to override them with your own backend. #### React Navigation ```tsx import { AIAgent } from '@mobileai/react-native'; import { NavigationContainer, useNavigationContainerRef } from '@react-navigation/native'; import screenMap from './ai-screen-map.json'; // auto-generated by step 1 export default function App() { const navRef = useNavigationContainerRef(); return ( <AIAgent // Your MobileAI Dashboard ID // This now auto-configures the hosted MobileAI text + voice proxies too. analyticsKey="mobileai_pub_xxxxxxxx" navRef={navRef} screenMap={screenMap} // optional but recommended > <NavigationContainer ref={navRef}> {/* Your existing screens — zero changes needed */} </NavigationContainer> </AIAgent> ); } ``` #### Expo Router In your root layout (`app/_layout.tsx`): ```tsx import { AIAgent } from '@mobileai/react-native'; import { Slot, useNavigationContainerRef } from 'expo-router'; import screenMap from './ai-screen-map.json'; // auto-generated by step 1 export default function RootLayout() { const navRef = useNavigationContainerRef(); return ( <AIAgent // Hosted MobileAI proxies are inferred automatically from analyticsKey analyticsKey="mobileai_pub_xxxxxxxx" navRef={navRef} screenMap={screenMap} > <Slot /> </AIAgent> ); } ``` ### Choose Your Provider The examples above use **Gemini** (default). To use **OpenAI** for text mode, add the `provider` prop. Voice mode is not supported with OpenAI. ```tsx <AIAgent provider="openai" apiKey="YOUR_OPENAI_API_KEY" // model="gpt-4.1-mini" ← default, or use any OpenAI model navRef={navRef} > {/* Same app, different brain */} </AIAgent> ``` A floating chat bar appears automatically. Ask the AI to navigate, tap buttons, fill forms, answer questions. ### Hosted MobileAI Defaults For the standard MobileAI Cloud setup, this is enough: ```tsx <AIAgent analyticsKey="mobileai_pub_xxxxxxxx" navRef={navRef} /> ``` Only pass explicit proxy props when: - you want to use your own backend proxy - you want a dedicated voice proxy - you are self-hosting the MobileAI backend ### Knowledge-Only Mode — AI Assistant Without UI Automation Set `enableUIControl={false}` for a lightweight FAQ / support assistant. Single LLM call, ~70% fewer tokens: ```tsx <AIAgent enableUIControl={false} knowledgeBase={KNOWLEDGE} /> ```
Type
`ReactNode`
`boolean`
`Record<string, ToolDefinition \
`React.RefObject<any>[]`
`string`
Before each agent step
Button label, screen, coordinates, `actor: 'user'`
`boolean`
`Record<string, string>`
`number`
`string`
`{ chat?: Partial<RichUITheme>, zone?: Partial<RichUITheme>, support?: Partial<RichUITheme> }`
`(result) => void`
`ChatBarTheme`
Success, steps, cost
`string`
`boolean`
`string`
`number`
`{ push, replace, back }`
`Partial<RichUITheme>`
`'fcm' \
`string`
Tool name, args, result
`boolean`
`'gemini' \
`'copilot' \
`boolean`
`number`
`BlockDefinition[] \
`string`
`Record<string, string>`
`(usage) => void`
Device, OS, SDK version
Effect
`boolean`
`React.ComponentType<any>[]`
Needed for
Version
Better navigation support in navigation-heavy apps
`>=18.0.0`
Voice input (microphone capture)
Voice input on Expo
Image attachments in chat
What it does
Read-only. The AI reads the screen and guides the user in plain language. Cannot tap, type, scroll, or navigate.
**Default.** Performs approved actions. Asks once before starting a flow, executes steps silently, then asks before irreversible commits.
Native build (not Expo Go)
Native build (not Expo Go)
Development build or prebuild
Full autonomy. All actions execute without confirmation. Use only for trusted, low-risk automation flows.
Use when running OpenCLI commands to interact with websites (Bilibili, Twitter, Reddit, Xiaohongshu, etc.), desktop apps (Cursor, Notion), or public APIs (HackerNews, arXiv). Covers installation, command reference, and output formats for 87+ adapters.
AI-optimized web search via Tavily API. Returns concise, relevant results for AI agents.
MUST use for ANY PDF or image format conversion task — converting PDF and images (JPG/JPEG/PNG/BMP/TIFF/TIF/WEBP/JPEG2000) to 10 formats (Word, Excel, PPT, H...
Unified multi-provider web search and URL extraction skill with intelligent auto-routing across Serper, Brave, Tavily, Querit, Linkup, Exa, Firecrawl, Perple...
Learn how to use the Web Monitor Claude skill. Complete guide with installation instructions and examples.
Learn how to use the Event-Watcher Claude skill. Complete guide with installation instructions and examples.
Learn how to use the Agent Onboarding Claude skill. Complete guide with installation instructions and examples.