Skip to main content

Human Proxy — Act As You

V2 only — invite-only edition. This is part of AI Partner V2 and is not in the open-source V1 you self-host from the Quick Start. V2 is available now, by invite. See V1 vs V2.

Overview​

AI Partner can act as you across five communication channels:

ChannelInboundOutboundYour identity
EmailRead, classify, draft replySend as you (Gmail or SMTP)your@email.com
Slack DMRead your DMs (user OAuth)Reply in your workspaceYour Slack identity
Telegram DMRead your personal DMs (userbot)Reply from your real accountYour Telegram account
Phone (Twilio)Take voicemails, transcribePlace outbound calls (gated)Your Twilio number
MeetingJoin, listen, respondSpeak with your voice cloneYour voice

Every outbound action routes through AuthorityPolicy before executing. Nothing is sent without either automatic authorization or your explicit approval.

Telegram serves two roles:

  1. Approval channel — you receive approval requests and tap buttons to approve/edit/skip
  2. Notification channel — heartbeat briefings, goal completions, file deliveries
  1. 1
    Create a Telegram bot
    1. Open Telegram and search for @BotFather
    2. Send /newbot
    3. Choose a name (e.g., "AI Partner") and username (e.g., my_ai_partner_bot)
    4. Copy the bot token: 7123456789:AAF-your-token-here
  2. 2
    Get your Telegram chat ID
    1. Send any message to your new bot
    2. Open https://api.telegram.org/bot<YOUR_TOKEN>/getUpdates in a browser
    3. Find "chat":{"id":123456789} — that number is your chat ID
  3. 3
    Enter credentials in AI Partner

    Go to Settings → Messaging → Telegram:

    • Bot token: 7123456789:AAF-your-token-here
    • Your chat ID: 123456789
    • Click Test Connection — you should receive a test message in Telegram
  4. 4
    Configure your authority policy

    Edit workspace/AUTHORITY.md to add draft_and_ask rules for the actions you want approval on:

    ## Draft + ask
    - action: reply_email + counterparty: client
    - action: send_slack_dm + counterparty: colleague

    Now any time the agent wants to reply to a client email, it'll ask via Telegram first.

Setting up email proxy (Gmail)​

  1. 1
    Generate a Gmail App Password
    1. Go to myaccount.google.com/security
    2. Enable 2-Step Verification (required for app passwords)
    3. Under 2-Step Verification → scroll down → App passwords
    4. Select "Mail" and "Other (custom name)" → enter "AI Partner"
    5. Copy the generated 16-character password
  2. 2
    Add to .env
    GMAIL_USER=your@gmail.com
    GMAIL_APP_PASSWORD=xxxx-xxxx-xxxx-xxxx

    Then restart: docker compose restart app

  3. 3
    Set up email trigger (optional)

    To have AI Partner monitor your inbox automatically (not just on-demand):

    Go to Settings → Integrations → Email Trigger:

    • Enable auto-monitoring
    • Set check interval (default: every 5 minutes)
    • Set intent filter (which emails warrant a reply vs. FYI)
  4. 4
    Test it

    Ask in chat:

    Check my inbox and summarize the last 5 unread emails.

    If configured correctly, you'll see a list of emails in the chat within 10–15 seconds.

Setting up Slack proxy​

This uses user OAuth — meaning the agent reads and sends as your actual Slack identity, not a bot.

Slack user OAuth gives the agent access to your personal Slack account, including DMs. Make sure your AUTHORITY.md has appropriate rules before enabling this.

  1. 1
    Get your Slack user token
    1. Go to api.slack.com/apps → Create New App
    2. OAuth & Permissions → User Token Scopes → add: chat:write, im:read, im:history, channels:read
    3. Install to workspace → copy the User OAuth Token (starts with xoxp-)
  2. 2
    Add to .env
    SLACK_USER_TOKEN=xoxp-your-token-here
  3. 3
    Test it
    List my unread Slack DMs from the last 24 hours.

Setting up Telegram DM proxy (userbot)​

This is different from the bot setup above. The userbot reads your personal Telegram DMs — messages sent to your real Telegram account, not to your bot.

The Telegram userbot logs into your actual Telegram account using your phone number and API credentials. Telegram's Terms of Service allow this for personal use. Do not use this to spam or mass-message.

  1. 1
    Get Telegram API credentials
    1. Go to my.telegram.org/apps
    2. Log in with your phone number
    3. Create an application → copy API ID and API Hash
  2. 2
    Add to .env
    TELEGRAM_API_ID=12345678
    TELEGRAM_API_HASH=abcdef1234567890abcdef1234567890
    TELEGRAM_PHONE=+919876543210
  3. 3
    Complete the auth flow

    On first start, AI Partner will prompt you for the confirmation code Telegram sends to your phone. Enter it in the Setup Wizard or via the CLI.

Setting up phone (Twilio)​

AI Partner can take voicemails and place outbound calls on your behalf.

  1. 1
    Get a Twilio number
    1. Sign up at twilio.com
    2. Buy a phone number
    3. Copy your Account SID and Auth Token from the dashboard
  2. 2
    Add to .env
    TWILIO_ACCOUNT_SID=ACxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    TWILIO_AUTH_TOKEN=your-auth-token
    TWILIO_PHONE_NUMBER=+14155551234
  3. 3
    Configure your voicemail webhook

    In your Twilio console, set the incoming call webhook to:

    https://your-domain.com/api/phone/incoming

    When someone calls your Twilio number, AI Partner greets them, records their message, transcribes it with Whisper, and notifies you via Telegram.

  4. 4
    Place outbound calls (gated)

    By default, outbound calls are blocked in AUTHORITY.md. To enable them:

    ## Draft + ask
    - action: place_phone_call + counterparty: vendor

    Now you can say:

    Call Mike at the plumber's office and ask if they can come Thursday morning.

    The agent drafts the call script, sends you a Telegram approval, and places the call after you approve.

The CounterpartyStore — everyone you interact with​

Every person AI Partner interacts with on your behalf gets a counterparty record:

Name: Sarah Johnson
Company: Acme Inc.
Relationship class: client
Email: sarah@acme.com
Slack: @sarah_acme
Phone: +14155559876

Tone notes: prefers bullet points, quick to respond, formal in email
Last contact: 2026-05-10 via email
Open commitments:
  - "Send revised project timeline by May 15" (from email 2026-05-10)

This record is updated automatically every time the agent interacts with Sarah. The next time she emails, the agent knows her tone, your relationship, and any open commitments — without you telling it again.

View all contacts: sidebar → Memory → Counterparties

Testing the full flow​

Here's a complete test you can run immediately after setup:

1. Ask AI Partner: "Draft a reply to the most recent email from any colleague.
   Use my voice, keep it professional. Send me a Telegram preview."

2. Check Telegram — you should see the draft with Approve/Edit/Skip buttons.

3. Tap "Edit" and reply with a correction.

4. Confirm AI Partner sent the corrected version.

If this works, your human proxy is fully operational.

Identity persistence + credentials vault (Phase 5)​

Before Phase 5, every bye partner → wake up partner cycle wiped your login state. The fix is two volumes:

VolumeMount pointSurvivesWhat lives there
aipartner-home-${userId}/home/partnercontainer restartChromium profile, gh auth, SSH keys, app configs, pip packages, npm caches
aipartner-workspace/workspacecontainer restartgenerated files, learned skills, goal output directories

On every wake up partner, the entrypoint:

  1. Mounts the per-user volume
  2. Creates .config/chromium-profile, .config/gh, .ssh, .secrets, .mozilla with correct permissions
  3. Decrypts every .secrets/*.age file into an env var (uppercased name) — so the agent reaches in with $GH_TOKEN etc. without prompting you again

Chromium now uses --user-data-dir=/home/partner/.config/chromium-profile — so cookies, saved logins, and storage state survive between sessions.

Secrets vault — never paste tokens into chat again​

When the user says set my gh token to ghp_xxxx, the agent calls credentials_set (one of the new Phase 5 MCP tools):

credentials_set(name="gh_token", value="ghp_xxxx", scope="github")
  → age encrypts the value → /home/partner/.secrets/gh_token.age
  → audit log: "user set gh_token@github"
  → returns: { name, scope, env_var: "GH_TOKEN", set_at }

On the next wake-up the boot-time loader sees gh_token.age, decrypts it, and exports GH_TOKEN so gh picks it up via env automatically. The agent can also read at runtime via credentials_get.

MCP tools exposed to the agent (gated to partner mode — they refuse to run when partner is asleep):

ToolWhat it does
credentials_setEncrypt + store a value
credentials_getDecrypt + return one value (treat as sensitive — don't echo to chat)
credentials_listList names + scope (NEVER values)
credentials_unsetPermanently remove

The age identity (private key) lives at /home/partner/.secrets/identity.age-key and is generated on first set. Threat model: anyone with root inside the T3 container can decrypt your secrets. The encryption protects against volume theft / snapshot leaks, not against attackers inside your own container. For higher assurance (TPM-backed key, hardware-token unwrap), file an issue.

Taking control mid-task (HITL takeover) — now works for T3 partner​

When the agent hits a CAPTCHA or login wall during a partner goal, it calls t3_request_handoff. The Inspector panel shows a "Take Control" button. Before Phase 5, clicking it forwarded events to the wrong browser (headless BrowserServer instead of the live T3 Chromium you were watching). After Phase 5, the handler is T3-aware:

User clicks/types/scrolls in Inspector
  ↓
GatewayService._takeoverTarget(userId)
  ├─ T3 partner pinned for this user? → use T3 Chromium's Playwright page
  └─ no                                → fall back to BrowserServer (T1/T2 path)
  ↓
page.mouse.click(x, y) / page.keyboard.type(text) / page.keyboard.press(key) / page.mouse.wheel(0, dy)
  ↓
T3 frame stream already running — Inspector picks up the next frame

So during a partner goal:

  1. Agent navigates to a login page → t3_request_handoff fires
  2. You see the page live in Inspector
  3. Click "Take Control"
  4. Click directly on the live screenshot — the click lands in T3 Chromium, not in some shadow browser
  5. Type your password — keystrokes go to T3
  6. Click "Release Control" — agent resumes, sees the now-authenticated page state

Combined with the persistent volume from Phase 5, you only need to log in once per site per user. The next partner session has your cookies already.