跳轉到

電腦操作

電腦操作工具可讓你建構瀏覽器、行動裝置與桌面控制代理,與介面互動並自動執行任務。模型能透過螢幕截圖「看見」電腦畫面,並產生滑鼠點選、鍵盤輸入等具體的 UI 動作來「採取行動」。與函式呼叫類似,你需要自行實作用戶端執行環境,以接收並執行電腦操作動作。

支援的模型清單請參閱模型版本。Gemini 4 與 Gemini 3.x 模型支援多項進階功能:

  • 多環境支援:為瀏覽器、行動裝置與桌面環境建構代理。
  • 附帶意圖的精簡動作:每個動作都包含 intent 欄位,說明模型在每個步驟背後的推理。
  • 可設定的安全政策:透過內建政策類別與覆寫,微調安全行為。
  • 提示詞注入偵測:可選擇啟用螢幕截圖掃描,偵測隱藏的對抗性指令。

運用電腦操作,你可以建構能執行以下工作的代理:

  • 在網站上自動處理重複的資料輸入或表單填寫。
  • 對網頁應用程式與使用者流程進行自動化測試
  • 在多個網站間進行研究(例如從電子商務網站蒐集產品資訊、價格與評論,作為購買決策的參考)

以下是一個最小範例,示範如何初始化用戶端,並在瀏覽器環境中啟用 computer_use 工具,將提示詞傳送給模型:

from google import genai

client = genai.Client()

interaction = client.interactions.create(
    model="gemini-4-hydrogen",
    input="Search for 'Gemini API' on Google.",
    tools=[{"type": "computer_use", "environment": "browser"}]
)

print(interaction)
import { GoogleGenAI } from '@google/genai';

const ai = new GoogleGenAI();

const interaction = await ai.interactions.create({
  model: 'gemini-4-hydrogen',
  input: "Search for 'Gemini API' on Google.",
  tools: [{ type: "computer_use", environment: "browser" }]
});

console.log(interaction);
import com.google.genai.Client;
import com.google.genai.gaos.models.interactions.ComputerUse;
import com.google.genai.gaos.models.interactions.CreateModelInteraction;
import com.google.genai.gaos.models.interactions.EnvironmentEnum;
import com.google.genai.gaos.models.interactions.Interaction;
import com.google.genai.gaos.models.interactions.InteractionsInput;
import com.google.genai.gaos.models.operations.CreateInteractionRequestBody;
import java.util.Arrays;

Client client = new Client();

CreateModelInteraction params =
    CreateModelInteraction.builder()
        .model("gemini-4-hydrogen")
        .input(InteractionsInput.of("Search for 'Gemini API' on Google."))
        .tools(
            Arrays.asList(
                ComputerUse.builder().environment(EnvironmentEnum.BROWSER).build()))
        .build();

Interaction interaction =
    client.interactions.create(CreateInteractionRequestBody.of(params)).interaction().get();

System.out.println(interaction);
package main

import (
    "context"
    "fmt"
    "log"

    "google.golang.org/genai"
    "google.golang.org/genai/interactions/models/interactions"
    "google.golang.org/genai/interactions/models/operations"
)

func main() {
    ctx := context.Background()
    client, err := genai.NewClient(ctx, nil)
    if err != nil {
        log.Fatal(err)
    }

    res, err := client.Interactions.Create(ctx, operations.CreateInteractionRequest{
        Body: operations.NewCreateInteractionRequestBody(interactions.CreateModelInteraction{
            Model: interactions.Model("gemini-4-hydrogen"),
            Input: interactions.NewInteractionsInput("Search for 'Gemini API' on Google."),
            Tools: []interactions.Tool{
                interactions.NewTool(interactions.ComputerUse{
                    Environment: interactions.EnvironmentEnumBrowser.ToPointer(),
                }),
            },
        }),
    })
    if err != nil {
        log.Fatal(err)
    }

    fmt.Println(res.Interaction)
}

電腦操作可能包含錯誤與安全漏洞。我們建議,重要任務務必密切監督,並避免將電腦操作功能用於涉及重大決策、敏感資料,或嚴重錯誤無法挽回的動作。我們鼓勵你詳閱安全最佳做法、禁止使用政策與 Gemini API 附加服務條款。

電腦操作的運作方式

若要用電腦操作模型建構代理,你需要在應用程式與 API 之間建立一個連續迴圈。以下是你的程式碼在每個步驟要做的事:

  1. 將要求傳送給模型
  2. 你的應用程式會傳送 API 要求,其中包含電腦操作工具、你的設定(例如目標環境)、使用者的提示詞,以及目前畫面的螢幕截圖。

  3. 接收模型回應

  4. 模型會分析畫面與提示詞,回傳一個回應,其中包含建議的 function_call,代表一個 UI 動作(例如點選、捲動或按鍵)。
  5. 對於 Gemini 4 與 Gemini 3.x 模型,回應還包含推理 intent,說明模型選擇該動作的原因。
  6. 回應也可能包含來自內部安全系統的 safety_decision,將動作分類為一般/允許、require_confirmation(需要使用者核准)或封鎖。

  7. 執行收到的動作

  8. 若動作獲允許(或使用者確認),你的用戶端程式碼會解析 function_call,將正規化座標換算為符合你的檢視區的座標,並使用自動化工具(例如 Playwright)在目標環境中執行該動作。若動作被封鎖,你的用戶端應停止執行或處理該中斷。

  9. 擷取新的環境狀態

  10. 動作執行完畢後,你的應用程式會擷取新的螢幕截圖,並透過 function_result 傳回給模型,以要求下一步。

接著,這個流程會從步驟 2 重複進行,持續向模型要求下一個動作,直到任務完成或終止。

如何實作電腦操作

使用電腦操作工具進行建構之前,你需要先備妥:

  • 安全的執行環境:在沙箱化的 VM 或容器中執行代理,將其與主機系統隔離,並限制其潛在影響。參考實作包含可立即使用的 Docker 沙箱,可作為起點。
  • 用戶端動作處理常式:實作用戶端邏輯,以執行座標、輸入文字並擷取螢幕截圖。

以下範例使用網頁瀏覽器作為執行環境,並以 Playwright 作為用戶端處理常式。

0. 設定 Playwright

首先,安裝所需的套件:

pip install google-genai playwright
playwright install chromium

接著,初始化一個 Playwright 瀏覽器執行個體以供執行:

from playwright.sync_api import sync_playwright

# 1. Configure screen dimensions for the target environment
SCREEN_WIDTH = 1440
SCREEN_HEIGHT = 900

# 2. Start the Playwright browser
# In production, utilize a sandboxed environment.
playwright = sync_playwright().start()
# Set headless=False to see the actions performed on your screen
browser = playwright.chromium.launch(headless=False)

# 3. Create a context and page with the specified dimensions
context = browser.new_context(
    viewport={"width": SCREEN_WIDTH, "height": SCREEN_HEIGHT}
)
page = context.new_page()

# 4. Navigate to an initial page to start the task
page.goto("https://www.google.com")

# The 'page', 'SCREEN_WIDTH', and 'SCREEN_HEIGHT' variables
# will be used in the steps below.

1. 將要求傳送給模型

初始化用戶端程式庫並設定電腦操作工具。請注意,發出要求時無需指定顯示尺寸;模型會依螢幕的高度與寬度預測像素座標。

使用 google-genai Python SDK(版本 2.7.0 以上),設定以瀏覽器環境為目標的要求:

from google import genai

client = genai.Client()

interaction = client.interactions.create(
    model='gemini-4-hydrogen',
    input="Find a flight from SF to Hawaii on Jun 30th, coming back on Jul 6th",
    tools=[
        {
            "type": "computer_use",
            "environment": "browser",
            "enable_prompt_injection_detection": True
        }
    ]
)

print(interaction)

使用 @google/genai Node.js SDK,設定以瀏覽器環境為目標的要求:

import { GoogleGenAI } from '@google/genai';

const ai = new GoogleGenAI();

const interaction = await ai.interactions.create({
  model: 'gemini-4-hydrogen',
  input: "Find a flight from SF to Hawaii on Jun 30th, coming back on Jul 6th",
  tools: [
    {
      type: "computer_use",
      environment: "browser",
      enable_prompt_injection_detection: true
    }
  ]
});

console.log(interaction);
import com.google.genai.Client;
import com.google.genai.gaos.models.interactions.ComputerUse;
import com.google.genai.gaos.models.interactions.CreateModelInteraction;
import com.google.genai.gaos.models.interactions.EnvironmentEnum;
import com.google.genai.gaos.models.interactions.Interaction;
import com.google.genai.gaos.models.interactions.InteractionsInput;
import com.google.genai.gaos.models.operations.CreateInteractionRequestBody;
import java.util.Arrays;

Client client = new Client();

CreateModelInteraction params =
    CreateModelInteraction.builder()
        .model("gemini-4-hydrogen")
        .input(
            InteractionsInput.of(
                "Find a flight from SF to Hawaii on Jun 30th, coming back on Jul 6th"))
        .tools(
            Arrays.asList(
                ComputerUse.builder()
                    .environment(EnvironmentEnum.BROWSER)
                    .enablePromptInjectionDetection(true)
                    .build()))
        .build();

Interaction interaction =
    client.interactions.create(CreateInteractionRequestBody.of(params)).interaction().get();

System.out.println(interaction);
package main

import (
    "context"
    "fmt"
    "log"

    "google.golang.org/genai"
    "google.golang.org/genai/interactions/models/interactions"
    "google.golang.org/genai/interactions/models/operations"
)

func main() {
    ctx := context.Background()
    client, err := genai.NewClient(ctx, nil)
    if err != nil {
        log.Fatal(err)
    }

    res, err := client.Interactions.Create(ctx, operations.CreateInteractionRequest{
        Body: operations.NewCreateInteractionRequestBody(interactions.CreateModelInteraction{
            Model: interactions.Model("gemini-4-hydrogen"),
            Input: interactions.NewInteractionsInput("Find a flight from SF to Hawaii on Jun 30th, coming back on Jul 6th"),
            Tools: []interactions.Tool{
                interactions.NewTool(interactions.ComputerUse{
                    Environment:                    interactions.EnvironmentEnumBrowser.ToPointer(),
                    EnablePromptInjectionDetection: genai.Ptr(true),
                }),
            },
        }),
    })
    if err != nil {
        log.Fatal(err)
    }

    fmt.Println(res.Interaction)
}

使用 curl 傳送要求:

curl -X POST \
  "https://generativelanguage.googleapis.com/v1/interactions" \
  -H "x-goog-api-key: $GEMINI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gemini-4-hydrogen",
    "input": "Find me a flight from SF to Hawaii on Jun 30th, coming back on Jul 6th. Start by navigating directly to flights.google.com",
    "tools": [
      {
        "type": "computer_use",
        "environment": "browser",
        "enable_prompt_injection_detection": true
      }
    ]
  }'

2. 接收模型回應

模型回應會建議一個函式呼叫,其中包含座標,以及專為該動作說明的推理意圖:

{
  "steps": [
    {
      "type": "function_call",
      "name": "click",
      "arguments": {
        "x": 450,
        "y": 120,
        "intent": "Click the search box to type the destination."
      }
    }
  ]
}

3. 執行收到的動作

你的應用程式必須解析回應中的座標,將其從正規化的 1000x1000 座標換算,並執行該動作:

from typing import Any, List, Tuple
import time

def denormalize_x(x: int, screen_width: int) -> int:
    """Convert normalized x coordinate (0-1000) to actual pixel coordinate."""
    return int(x / 1000 * screen_width)

def denormalize_y(y: int, screen_height: int) -> int:
    """Convert normalized y coordinate (0-1000) to actual pixel coordinate."""
    return int(y / 1000 * screen_height)

def execute_function_calls(interaction, page, screen_width, screen_height):
    results = []
    function_calls = [
        step for step in interaction.steps if step.type == "function_call"
    ]

    for function_call in function_calls:
        action_result = {}
        fname = function_call.name
        args = function_call.arguments
        print(f"  -> Executing: {fname} (Intent: {args.get('intent', 'N/A')})")

        try:
            if fname == "open_app":
                pass # Handled / already open
            elif fname in ("click", "double_click", "triple_click", "middle_click", "right_click", "move", "long_press"):
                actual_x = denormalize_x(args["x"], screen_width)
                actual_y = denormalize_y(args["y"], screen_height)

                if fname == "click":
                    page.mouse.click(actual_x, actual_y)
                elif fname == "double_click":
                    page.mouse.dblclick(actual_x, actual_y)
                elif fname == "right_click":
                    page.mouse.click(actual_x, actual_y, button="right")
                elif fname == "middle_click":
                    page.mouse.click(actual_x, actual_y, button="middle")
                elif fname == "move":
                    page.mouse.move(actual_x, actual_y)
            elif fname == "type":
                actual_x = denormalize_x(args["x"], screen_width) if "x" in args else None
                actual_y = denormalize_y(args["y"], screen_height) if "y" in args else None
                text = args["text"]
                press_enter = args.get("press_enter", False)

                if actual_x is not None and actual_y is not None:
                    page.mouse.click(actual_x, actual_y)
                # Clear field first
                page.keyboard.press("Meta+A")
                page.keyboard.press("Backspace")
                page.keyboard.type(text)
                if press_enter:
                    page.keyboard.press("Enter")
            elif fname == "navigate":
                page.goto(args["url"])
            elif fname == "go_back":
                page.go_back()
            elif fname == "go_forward":
                page.go_forward()
            elif fname == "wait":
                time.sleep(args.get("seconds", 1))
            else:
                print(f"Warning: Custom or unhandled function {fname}")

            page.wait_for_load_state(timeout=5000)
            time.sleep(1)

        except Exception as e:
            print(f"Error executing {fname}: {e}")
            action_result = {"error": str(e)}

        results.append((fname, function_call.id, action_result))

    return results
function denormalizeX(x, screenWidth) {
    // Convert normalized x coordinate (0-1000) to actual pixel coordinate.
    return Math.floor((x / 1000) * screenWidth);
}

function denormalizeY(y, screenHeight) {
    // Convert normalized y coordinate (0-1000) to actual pixel coordinate.
    return Math.floor((y / 1000) * screenHeight);
}

async function executeFunctionCalls(interaction, page, screenWidth, screenHeight) {
    const results = [];
    const functionCalls = interaction.steps.filter(step => step.type === "function_call");

    for (const functionCall of functionCalls) {
        const actionResult = {};
        const fname = functionCall.name;
        const args = functionCall.arguments;
        console.log(`  -> Executing: ${fname} (Intent: ${args.intent || 'N/A'})`);

        try {
            if (fname === "open_app") {
                // Handled / already open
            } else if (["click", "double_click", "triple_click", "middle_click", "right_click", "move", "long_press"].includes(fname)) {
                const actualX = denormalizeX(args.x, screenWidth);
                const actualY = denormalizeY(args.y, screenHeight);

                if (fname === "click") {
                    await page.mouse.click(actualX, actualY);
                } else if (fname === "double_click") {
                    await page.mouse.dblclick(actualX, actualY);
                } else if (fname === "right_click") {
                    await page.mouse.click(actualX, actualY, { button: "right" });
                } else if (fname === "middle_click") {
                    await page.mouse.click(actualX, actualY, { button: "middle" });
                } else if (fname === "move") {
                    await page.mouse.move(actualX, actualY);
                }
            } else if (fname === "type") {
                const actualX = args.x !== undefined ? denormalizeX(args.x, screenWidth) : null;
                const actualY = args.y !== undefined ? denormalizeY(args.y, screenHeight) : null;
                const text = args.text;
                const pressEnter = args.press_enter || false;

                if (actualX !== null && actualY !== null) {
                    await page.mouse.click(actualX, actualY);
                }
                // Clear field first
                await page.keyboard.press("Meta+A");
                await page.keyboard.press("Backspace");
                await page.keyboard.type(text);
                if (pressEnter) {
                    await page.keyboard.press("Enter");
                }
            } else if (fname === "navigate") {
                await page.goto(args.url);
            } else if (fname === "go_back") {
                await page.goBack();
            } else if (fname === "go_forward") {
                await page.goForward();
            } else if (fname === "wait") {
                await new Promise(resolve => setTimeout(resolve, (args.seconds || 1) * 1000));
            } else {
                console.log(`Warning: Custom or unhandled function ${fname}`);
            }

            await page.waitForLoadState('load', { timeout: 5000 }).catch(() => {});
            await new Promise(resolve => setTimeout(resolve, 1000));
        } catch (e) {
            console.log(`Error executing ${fname}: ${e}`);
            actionResult.error = e.message;
        }

        results.push([fname, functionCall.id, actionResult]);
    }

    return results;
}
import com.google.genai.gaos.models.interactions.FunctionCallStep;
import com.google.genai.gaos.models.interactions.Interaction;
import com.google.genai.gaos.models.interactions.Step;
import java.util.ArrayList;
import java.util.Collections;
import java.util.HashMap;
import java.util.List;
import java.util.Map;

class ActionExecutor {
  int denormalizeX(int x, int screenWidth) {
    return (int) (x / 1000.0 * screenWidth);
  }

  int denormalizeY(int y, int screenHeight) {
    return (int) (y / 1000.0 * screenHeight);
  }

  List<Map<String, Object>> executeFunctionCalls(
      Interaction interaction, int screenWidth, int screenHeight) {
    List<Map<String, Object>> results = new ArrayList<>();

    for (Step step : interaction.steps().orElse(Collections.emptyList())) {
      if (step instanceof FunctionCallStep) {
        FunctionCallStep functionCall = (FunctionCallStep) step;
        String fname = functionCall.name().orElse("");
        Map<String, Object> args = functionCall.arguments().orElse(Collections.emptyMap());
        Map<String, Object> actionResult = new HashMap<>();

        System.out.println(
            "  -> Executing: " + fname + " (Intent: " + args.getOrDefault("intent", "N/A") + ")");

        try {
          if (fname.equals("click")) {
            int actualX = denormalizeX(((Number) args.get("x")).intValue(), screenWidth);
            int actualY = denormalizeY(((Number) args.get("y")).intValue(), screenHeight);
            // Perform mouse click at (actualX, actualY) using your browser automation library
          } else if (fname.equals("type")) {
            String text = (String) args.get("text");
            // Type text into active element using your browser automation library
          } else if (fname.equals("navigate")) {
            String url = (String) args.get("url");
            // Navigate browser to url
          }
        } catch (Exception e) {
          actionResult.put("error", e.getMessage());
        }

        Map<String, Object> entry = new HashMap<>();
        entry.put("name", fname);
        entry.put("callId", functionCall.id().orElse(""));
        entry.put("result", actionResult);
        results.add(entry);
      }
    }
    return results;
  }
}
package main

import (
    "fmt"

    "google.golang.org/genai/interactions/models/interactions"
)

func denormalizeX(x, screenWidth int) int {
    return int(float64(x) / 1000.0 * float64(screenWidth))
}

func denormalizeY(y, screenHeight int) int {
    return int(float64(y) / 1000.0 * float64(screenHeight))
}

func executeFunctionCalls(interaction *interactions.Interaction, screenWidth, screenHeight int) []map[string]any {
    var results []map[string]any

    for _, step := range interaction.Steps {
        if functionCall := step.FunctionCallStep; functionCall != nil {
            fname := functionCall.Name
            args := functionCall.Arguments
            actionResult := map[string]any{}

            intent := args["intent"]
            if intent == nil {
                intent = "N/A"
            }
            fmt.Printf("  -> Executing: %s (Intent: %v)\n", fname, intent)

            switch fname {
            case "click":
                xVal, _ := args["x"].(float64)
                yVal, _ := args["y"].(float64)
                actualX := denormalizeX(int(xVal), screenWidth)
                actualY := denormalizeY(int(yVal), screenHeight)
                _ = actualX
                _ = actualY
                // Perform mouse click at (actualX, actualY) using your browser automation library
            case "type":
                text, _ := args["text"].(string)
                _ = text
                // Type text into active element using your browser automation library
            case "navigate":
                url, _ := args["url"].(string)
                _ = url
                // Navigate browser to url
            }

            results = append(results, map[string]any{
                "name":   fname,
                "callId": functionCall.ID,
                "result": actionResult,
            })
        }
    }
    return results
}

func main() {
    // Example helper usage with an Interaction response
}

4. 擷取新的環境狀態

執行動作後,將函式執行的結果傳回給模型,讓模型運用這些資訊產生下一個動作。若執行了多個動作(平行呼叫),你必須在下一個使用者回合中,為每個動作各傳送一個 function_result。

import json
import base64

def get_function_responses(page, results):
    screenshot_bytes = page.screenshot(type="png")
    current_url = page.url
    function_responses = []
    for name, call_id, result in results:
        function_responses.append({
            "type": "function_result",
            "name": name,
            "call_id": call_id,
            "result": [
                {
                    "type": "text",
                    "text": json.dumps({"url": current_url, **result})
                },
                {
                    "type": "image",
                    "data": base64.b64encode(screenshot_bytes).decode("utf-8"),
                    "mime_type": "image/png"
                }
            ]
        })
    return function_responses
async function getFunctionResponses(page, results) {
    const screenshotBuffer = await page.screenshot({ type: 'png' });
    const screenshotBase64 = screenshotBuffer.toString('base64');
    const currentUrl = page.url();
    const functionResponses = [];

    for (const [name, callId, result] of results) {
        functionResponses.push({
            type: "function_result",
            name: name,
            call_id: callId,
            result: [
                {
                    type: "text",
                    text: JSON.stringify({ url: currentUrl, ...result })
                },
                {
                    type: "image",
                    data: screenshotBase64,
                    mime_type: "image/png"
                }
            ]
        });
    }
    return functionResponses;
}

定義好如何擷取環境狀態並加以格式化後,你就能將這些步驟組合成一個連續的執行迴圈。

import com.google.genai.gaos.models.interactions.FunctionResultStep;
import com.google.genai.gaos.models.interactions.FunctionResultStepResultUnion;
import com.google.genai.gaos.models.interactions.ImageContent;
import com.google.genai.gaos.models.interactions.ImageContentMimeType;
import com.google.genai.gaos.models.interactions.Step;
import com.google.genai.gaos.models.interactions.TextContent;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Base64;
import java.util.List;
import java.util.Map;

class StateCapturer {
  List<Step> getFunctionResponses(
      byte[] screenshotBytes, String currentUrl, List<Map<String, Object>> results) {
    List<Step> functionResponses = new ArrayList<>();
    String base64Screenshot = Base64.getEncoder().encodeToString(screenshotBytes);

    for (Map<String, Object> entry : results) {
      String name = (String) entry.get("name");
      String callId = (String) entry.get("callId");
      String jsonResult = String.format("{\"url\": \"%s\"}", currentUrl);

      FunctionResultStep responseStep =
          FunctionResultStep.builder()
              .name(name)
              .callId(callId)
              .result(
                  FunctionResultStepResultUnion.of(
                      Arrays.asList(
                          TextContent.builder().text(jsonResult).build(),
                          ImageContent.builder()
                              .data(base64Screenshot)
                              .mimeType(ImageContentMimeType.IMAGE_PNG)
                              .build())))
              .build();
      functionResponses.add(responseStep);
    }
    return functionResponses;
  }
}
package main

import (
    "encoding/base64"
    "fmt"

    "google.golang.org/genai"
    "google.golang.org/genai/interactions/models/interactions"
)

func getFunctionResponses(screenshotBytes []byte, currentURL string, results []map[string]any) []interactions.Step {
    var functionResponses []interactions.Step
    base64Screenshot := base64.StdEncoding.EncodeToString(screenshotBytes)

    for _, entry := range results {
        name, _ := entry["name"].(string)
        callID, _ := entry["callId"].(string)
        jsonResult := fmt.Sprintf(`{"url": "%s"}`, currentURL)

        responseStep := interactions.NewStep(interactions.FunctionResultStep{
            Name:   genai.Ptr(name),
            CallID: callID,
            Result: interactions.NewFunctionResultStepResultUnion([]interactions.FunctionResultSubcontent{
                interactions.NewFunctionResultSubcontent(interactions.TextContent{
                    Text: jsonResult,
                }),
                interactions.NewFunctionResultSubcontent(interactions.ImageContent{
                    Data:     genai.Ptr(base64Screenshot),
                    MimeType: interactions.ImageContentMimeType("image/png").ToPointer(),
                }),
            }),
        })
        functionResponses = append(functionResponses, responseStep)
    }
    return functionResponses
}

func main() {
    // Example helper usage to build FunctionResultStep responses
}

建構代理迴圈

若要支援多步驟互動,請將如何實作電腦操作一節的四個步驟組合成單一迴圈。這個迴圈會持續向模型要求動作,並將結果回傳給模型,直到任務完成。

請記得正確管理對話記錄:在每個步驟中,將模型回應與你的函式回應都附加到記錄中。

import time
from typing import Any, List, Tuple
from playwright.sync_api import sync_playwright

from google import genai

client = genai.Client()

# Constants for screen dimensions
SCREEN_WIDTH = 1440
SCREEN_HEIGHT = 900

# Setup Playwright
print("Initializing browser...")
playwright = sync_playwright().start()
browser = playwright.chromium.launch(headless=False)
context = browser.new_context(viewport={"width": SCREEN_WIDTH, "height": SCREEN_HEIGHT})
page = context.new_page()

# Define helper functions. Copy/paste from steps 3 and 4
# def denormalize_x(...)
# def denormalize_y(...)
# def execute_function_calls(...)
# def get_function_responses(...)

try:
    # Go to initial page
    page.goto("https://aistudio.google.com/docs")

    # Take initial screenshot
    initial_screenshot = page.screenshot(type="png")
    USER_PROMPT = "Go to aistudio.google.com/docs and search for pricing."
    print(f"Goal: {USER_PROMPT}")

    # First interaction
    interaction = client.interactions.create(
        model='gemini-4-hydrogen',
        input=[
            {"type": "text", "text": USER_PROMPT},
            {"type": "image", "data": base64.b64encode(initial_screenshot).decode("utf-8"), "mime_type": "image/png"}
        ],
        tools=[{
            "type": "computer_use",
            "environment": "browser",
            "enable_prompt_injection_detection": True
        }]
    )

    # Agent Loop
    turn_limit = 5
    for i in range(turn_limit):
        print(f"\n--- Turn {i+1} ---")

        has_function_calls = any(
            step.type == "function_call"
            for step in interaction.steps
        )
        if not has_function_calls:
            text_response = " ".join([
                content_block.text for step in interaction.steps if step.type == "model_output"
                for content_block in step.content if content_block.type == "text"
            ])
            print("Agent finished:", text_response)
            break

        print("Executing actions...")
        results = execute_function_calls(interaction, page, SCREEN_WIDTH, SCREEN_HEIGHT)

        print("Capturing state...")
        function_responses = get_function_responses(page, results)

        # Continue conversation with function responses
        interaction = client.interactions.create(
            model='gemini-4-hydrogen',
            previous_interaction_id=interaction.id,
            input=function_responses,
            tools=[{
                "type": "computer_use",
                "environment": "browser",
                "enable_prompt_injection_detection": True
            }]
        )

finally:
    # Cleanup
    print("\nClosing browser...")
    browser.close()
    playwright.stop()
import { chromium } from 'playwright';
import { GoogleGenAI } from '@google/genai';

const ai = new GoogleGenAI();

// Constants for screen dimensions
const SCREEN_WIDTH = 1440;
const SCREEN_HEIGHT = 900;

console.log("Initializing browser...");
const browser = await chromium.launch({ headless: false });
const context = await browser.newContext({
    viewport: { width: SCREEN_WIDTH, height: SCREEN_HEIGHT }
});
const page = await context.newPage();

// Define helper functions. Copy/paste from steps 3 and 4:
// function denormalizeX(...)
// function denormalizeY(...)
// async function executeFunctionCalls(...)
// async function getFunctionResponses(...)

try {
    // Go to initial page
    await page.goto("https://aistudio.google.com/docs");

    // Take initial screenshot
    const initialScreenshotBuffer = await page.screenshot({ type: 'png' });
    const initialScreenshotBase64 = initialScreenshotBuffer.toString('base64');
    const USER_PROMPT = "Go to aistudio.google.com/docs and search for pricing.";
    console.log(`Goal: ${USER_PROMPT}`);

    // First interaction
    let interaction = await ai.interactions.create({
        model: 'gemini-4-hydrogen',
        input: [
            { type: 'text', text: USER_PROMPT },
            { type: 'image', data: initialScreenshotBase64, mime_type: 'image/png' }
        ],
        tools: [{
            type: 'computer_use',
            environment: 'browser',
            enable_prompt_injection_detection: true
        }]
    });

    // Agent Loop
    const turnLimit = 5;
    for (let i = 0; i < turnLimit; i++) {
        console.log(`\n--- Turn ${i + 1} ---`);

        const hasFunctionCalls = (interaction.steps || []).some(step => step.type === "function_call");
        if (!hasFunctionCalls) {
            const textResponses = [];
            for (const step of interaction.steps || []) {
                if (step.type === "model_output") {
                    for (const contentBlock of step.content || []) {
                        if (contentBlock.type === "text") {
                            textResponses.push(contentBlock.text);
                        }
                    }
                }
            }
            console.log("Agent finished:", textResponses.join(" "));
            break;
        }

        console.log("Executing actions...");
        const results = await executeFunctionCalls(interaction, page, SCREEN_WIDTH, SCREEN_HEIGHT);

        console.log("Capturing state...");
        const functionResponses = await getFunctionResponses(page, results);

        // Continue conversation with function responses
        interaction = await ai.interactions.create({
            model: 'gemini-4-hydrogen',
            previous_interaction_id: interaction.id,
            input: functionResponses,
            tools: [{
                type: 'computer_use',
                environment: 'browser',
                enable_prompt_injection_detection: true
            }]
        });
    }
} finally {
    // Cleanup
    console.log("\nClosing browser...");
    await browser.close();
}
import com.google.genai.Client;
import com.google.genai.gaos.models.interactions.ComputerUse;
import com.google.genai.gaos.models.interactions.Content;
import com.google.genai.gaos.models.interactions.CreateModelInteraction;
import com.google.genai.gaos.models.interactions.EnvironmentEnum;
import com.google.genai.gaos.models.interactions.FunctionCallStep;
import com.google.genai.gaos.models.interactions.ImageContent;
import com.google.genai.gaos.models.interactions.ImageContentMimeType;
import com.google.genai.gaos.models.interactions.Interaction;
import com.google.genai.gaos.models.interactions.InteractionsInput;
import com.google.genai.gaos.models.interactions.ModelOutputStep;
import com.google.genai.gaos.models.interactions.Step;
import com.google.genai.gaos.models.interactions.TextContent;
import com.google.genai.gaos.models.operations.CreateInteractionRequestBody;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Base64;
import java.util.Collections;
import java.util.List;

Client client = new Client();

// Constants for screen dimensions
int screenWidth = 1440;
int screenHeight = 900;

// Capture initial screenshot from browser driver (e.g. Playwright)
byte[] initialScreenshot = new byte[0];
String base64Screenshot = Base64.getEncoder().encodeToString(initialScreenshot);
String userPrompt = "Go to aistudio.google.com/docs and search for pricing.";
System.out.println("Goal: " + userPrompt);

ComputerUse computerUseTool =
    ComputerUse.builder()
        .environment(EnvironmentEnum.BROWSER)
        .enablePromptInjectionDetection(true)
        .build();

CreateModelInteraction initialParams =
    CreateModelInteraction.builder()
        .model("gemini-4-hydrogen")
        .input(
            InteractionsInput.ofContent(
                Arrays.asList(
                    TextContent.builder().text(userPrompt).build(),
                    ImageContent.builder()
                        .data(base64Screenshot)
                        .mimeType(ImageContentMimeType.IMAGE_PNG)
                        .build())))
        .tools(Arrays.asList(computerUseTool))
        .build();

Interaction interaction =
    client.interactions.create(CreateInteractionRequestBody.of(initialParams)).interaction().get();

int turnLimit = 5;
for (int i = 0; i < turnLimit; i++) {
  System.out.println("\n--- Turn " + (i + 1) + " ---");

  boolean hasFunctionCalls =
      interaction.steps().orElse(Collections.emptyList()).stream()
          .anyMatch(step -> step instanceof FunctionCallStep);

  if (!hasFunctionCalls) {
    StringBuilder textResponse = new StringBuilder();
    for (Step step : interaction.steps().orElse(Collections.emptyList())) {
      if (step instanceof ModelOutputStep) {
        for (Content contentBlock :
            ((ModelOutputStep) step).content().orElse(Collections.emptyList())) {
          if (contentBlock instanceof TextContent) {
            textResponse.append(((TextContent) contentBlock).text().orElse("")).append(" ");
          }
        }
      }
    }
    System.out.println("Agent finished: " + textResponse.toString().trim());
    break;
  }

  System.out.println("Executing actions and capturing state...");
  // Execute function calls against browser driver and capture List<Step> functionResponses
  List<Step> functionResponses = new ArrayList<>();

  CreateModelInteraction nextParams =
      CreateModelInteraction.builder()
          .model("gemini-4-hydrogen")
          .previousInteractionId(interaction.id().get())
          .input(InteractionsInput.ofStep(functionResponses))
          .tools(Arrays.asList(computerUseTool))
          .build();

  interaction =
      client.interactions.create(CreateInteractionRequestBody.of(nextParams)).interaction().get();
}
package main

import (
    "context"
    "encoding/base64"
    "fmt"
    "log"
    "strings"

    "google.golang.org/genai"
    "google.golang.org/genai/interactions/models/interactions"
    "google.golang.org/genai/interactions/models/operations"
)

func main() {
    ctx := context.Background()
    client, err := genai.NewClient(ctx, nil)
    if err != nil {
        log.Fatal(err)
    }

    // Constants for screen dimensions
    screenWidth := 1440
    screenHeight := 900
    _ = screenWidth
    _ = screenHeight

    // Capture initial screenshot from browser driver (e.g. Playwright)
    initialScreenshot := []byte{}
    base64Screenshot := base64.StdEncoding.EncodeToString(initialScreenshot)
    userPrompt := "Go to aistudio.google.com/docs and search for pricing."
    fmt.Println("Goal:", userPrompt)

    computerUseTool := interactions.NewTool(interactions.ComputerUse{
        Environment:                    interactions.EnvironmentEnumBrowser.ToPointer(),
        EnablePromptInjectionDetection: genai.Ptr(true),
    })

    res, err := client.Interactions.Create(ctx, operations.CreateInteractionRequest{
        Body: operations.NewCreateInteractionRequestBody(interactions.CreateModelInteraction{
            Model: interactions.Model("gemini-4-hydrogen"),
            Input: interactions.NewInteractionsInput([]interactions.Content{
                interactions.NewContent(interactions.TextContent{Text: userPrompt}),
                interactions.NewContent(interactions.ImageContent{
                    Data:     genai.Ptr(base64Screenshot),
                    MimeType: interactions.ImageContentMimeType("image/png").ToPointer(),
                }),
            }),
            Tools: []interactions.Tool{computerUseTool},
        }),
    })
    if err != nil {
        log.Fatal(err)
    }
    interaction := res.Interaction

    turnLimit := 5
    for i := 0; i < turnLimit; i++ {
        fmt.Printf("\n--- Turn %d ---\n", i+1)

        hasFunctionCalls := false
        for _, step := range interaction.Steps {
            if step.FunctionCallStep != nil {
                hasFunctionCalls = true
                break
            }
        }

        if !hasFunctionCalls {
            var parts []string
            for _, step := range interaction.Steps {
                if outStep := step.ModelOutputStep; outStep != nil {
                    for _, contentBlock := range outStep.Content {
                        if textContent := contentBlock.TextContent; textContent != nil {
                            parts = append(parts, textContent.GetText())
                        }
                    }
                }
            }
            fmt.Println("Agent finished:", strings.TrimSpace(strings.Join(parts, " ")))
            break
        }

        fmt.Println("Executing actions and capturing state...")
        // Execute function calls against browser driver and capture []interactions.Step functionResponses
        var functionResponses []interactions.Step

        nextRes, err := client.Interactions.Create(ctx, operations.CreateInteractionRequest{
            Body: operations.NewCreateInteractionRequestBody(interactions.CreateModelInteraction{
                Model:                 interactions.Model("gemini-4-hydrogen"),
                PreviousInteractionID: interaction.ID,
                Input:                 interactions.NewInteractionsInput(functionResponses),
                Tools:                 []interactions.Tool{computerUseTool},
            }),
        })
        if err != nil {
            log.Fatal(err)
        }
        interaction = nextRes.Interaction
    }
}

支援的環境

Gemini 4 與 Gemini 3.x 模型支援在 computer_use 設定中指定的三種環境:

瀏覽器環境(ENVIRONMENT_BROWSER)

瀏覽器工具可用的動作:

指令名稱 說明 參數(函式呼叫中)
click 在指定座標按下滑鼠左鍵。 y: int (0-999)x: int (0-999)intent: str
double_click 在指定座標連按兩下。 y: int (0-999)x: int (0-999)intent: str
triple_click 在指定座標連按三下。 y: int (0-999)x: int (0-999)intent: str
middle_click 在指定座標按下中鍵。 y: int (0-999)x: int (0-999)intent: str
right_click 在指定座標按下滑鼠右鍵。 y: int (0-999)x: int (0-999)intent: str
mouse_down 在指定座標按住滑鼠按鈕不放。 y: int (0-999)x: int (0-999)intent: str
mouse_up 在指定座標放開滑鼠按鈕。 y: int (0-999)x: int (0-999)intent: str
move 將游標移到指定位置。 y: int (0-999)x: int (0-999)intent: str
type 輸入文字。 text: strpress_enter: bool (Optional, default false)intent: str
drag_and_drop 將項目從起始座標拖曳到結束座標。 start_y: int (0-999)start_x: int (0-999)end_y: int (0-999)end_x: int (0-999)intent: str
wait 暫停執行指定的秒數。 seconds: int (Optional, default 1)intent: str
press_key 按下指定按鍵後放開。 key: strintent: str
key_down 按住指定按鍵不放。 key: strintent: str
key_up 放開指定按鍵。 key: strintent: str
hotkey 按下指定的按鍵組合。 keys: List[str]``intent: str
take_screenshot 回傳目前畫面的螢幕截圖。 intent: str
scroll 在指定座標以像素距離向上、下、左或右捲動。 y: int (0-999)x: int (0-999)direction: str ("up", "down", "left", "right")magnitude_in_pixels: int (0-999, Optional, default 300)intent: str
go_back 回到瀏覽器歷史紀錄中的上一個網頁。 intent: str
navigate 直接前往指定的 URL。 url: strintent: str
go_forward 前往瀏覽器歷史紀錄中的下一個網頁。 intent: str

行動環境(ENVIRONMENT_MOBILE)

針對 Android 最佳化的環境動作:

指令名稱 說明 參數(函式呼叫中)
open_app 依名稱開啟應用程式。 app_name: strintent: str
click 在指定座標按下滑鼠左鍵。 y: int (0-999)x: int (0-999)intent: str
list_apps 列出裝置上可用的應用程式,並回傳其名稱與套件名稱。 intent: str
wait 暫停執行指定的秒數。 seconds: int (Optional, default 1)intent: str
go_back 回到上一個畫面或網頁。 intent: str
type 輸入文字。 text: strpress_enter: bool (Optional, default false)intent: str
drag_and_drop 將項目從起始座標拖曳到結束座標。 start_y: int (0-999)start_x: int (0-999)end_y: int (0-999)end_x: int (0-999)intent: str
long_press 在畫面上的指定座標長按。 y: int (0-999)x: int (0-999)seconds: int (Optional, default 2)intent: str
press_key 按下指定按鍵後放開。 key: strintent: str
take_screenshot 回傳目前畫面的螢幕截圖。 intent: str

桌面環境(ENVIRONMENT_DESKTOP)

桌面環境的作業系統層級游標指令:

指令名稱 說明 參數(函式呼叫中)
click 在指定座標按下滑鼠左鍵。 y: int (0-999)x: int (0-999)intent: str
double_click 在指定座標連按兩下。 y: int (0-999)x: int (0-999)intent: str
triple_click 在指定座標連按三下。 y: int (0-999)x: int (0-999)intent: str
middle_click 在指定座標按下中鍵。 y: int (0-999)x: int (0-999)intent: str
right_click 在指定座標按下滑鼠右鍵。 y: int (0-999)x: int (0-999)intent: str
mouse_down 在指定座標按住滑鼠按鈕不放。 y: int (0-999)x: int (0-999)intent: str
mouse_up 在指定座標放開滑鼠按鈕。 y: int (0-999)x: int (0-999)intent: str
move 將游標移到指定位置。 y: int (0-999)x: int (0-999)intent: str
type 輸入文字。 text: strpress_enter: bool (Optional, default false)intent: str
drag_and_drop 將項目從起始座標拖曳到結束座標。 start_y: int (0-999)start_x: int (0-999)end_y: int (0-999)end_x: int (0-999)intent: str
wait 暫停執行指定的秒數。 seconds: int (Optional, default 1)intent: str
press_key 按下指定按鍵後放開。 key: strintent: str
key_down 按住指定按鍵不放。 key: strintent: str
key_up 放開指定按鍵。 key: strintent: str
hotkey 按下指定的按鍵組合。 keys: List[str]``intent: str
take_screenshot 回傳目前畫面的螢幕截圖。 intent: str
scroll 在指定座標以像素距離向上、下、左或右捲動。 y: int (0-999)x: int (0-999)direction: str ("up", "down", "left", "right")magnitude_in_pixels: int (0-999, Optional, default 300)intent: str

自訂的使用者定義函式

你可以加入自訂的使用者定義函式,擴充模型的功能。例如,在人類參與(HITL)的情境中,你可以排除預設的預定義動作,並註冊自訂動作。

排除標準的預定義瀏覽器動作(例如 click),並註冊自訂的 yield_to_user 工具:

from google import genai

client = genai.Client()

yield_to_user_tool = {
    "type": "function",
    "name": "yield_to_user",
    "description": "Yields control back to the user for assistance or verification when an automated action is unsafe or ambiguous.",
    "parameters": {
        "type": "object",
        "properties": {
            "reason": {
                "type": "string",
                "description": "The reason why the agent is yielding control to the human."
            }
        },
        "required": ["reason"]
    }
}

interaction = client.interactions.create(
    model="gemini-4-hydrogen",
    input="Click the submit button. If you need a second factor authentication code, ask me.",
    tools=[
        {
            "type": "computer_use",
            "environment": "mobile",
            "excluded_predefined_functions": ["click"]
        },
        yield_to_user_tool
    ]
)

排除標準的預定義瀏覽器動作(例如 click),並註冊自訂的 yield_to_user 工具:

import { GoogleGenAI } from '@google/genai';

const ai = new GoogleGenAI();

const interaction = await ai.interactions.create({
    model: "gemini-4-hydrogen",
    input: "Click the submit button. If you need a second factor authentication code, ask me.",
    tools: [
        {
            type: "computer_use",
            environment: "mobile",
            excluded_predefined_functions: ["click"]
        },
        {
            type: "function",
            name: "yield_to_user",
            description: "Yields control back to the user for assistance or verification when an automated action is unsafe or ambiguous.",
            parameters: {
                type: "object",
                properties: {
                    reason: {
                        type: "string",
                        description: "The reason why the agent is yielding control to the human."
                    }
                },
                required: ["reason"]
            }
        }
    ]
});
import com.google.genai.Client;
import com.google.genai.gaos.models.interactions.ComputerUse;
import com.google.genai.gaos.models.interactions.CreateModelInteraction;
import com.google.genai.gaos.models.interactions.EnvironmentEnum;
import com.google.genai.gaos.models.interactions.Function;
import com.google.genai.gaos.models.interactions.Interaction;
import com.google.genai.gaos.models.interactions.InteractionsInput;
import com.google.genai.gaos.models.operations.CreateInteractionRequestBody;
import java.util.Arrays;
import java.util.Collections;
import java.util.HashMap;
import java.util.Map;

Client client = new Client();

Map<String, Object> reasonProp = new HashMap<>();
reasonProp.put("type", "string");
reasonProp.put("description", "The reason why the agent is yielding control to the human.");

Map<String, Object> properties = new HashMap<>();
properties.put("reason", reasonProp);

Map<String, Object> parameters = new HashMap<>();
parameters.put("type", "object");
parameters.put("properties", properties);
parameters.put("required", Collections.singletonList("reason"));

Function yieldToUserTool =
    Function.builder()
        .name("yield_to_user")
        .description(
            "Yields control back to the user for assistance or verification when an automated action is unsafe or ambiguous.")
        .parameters(parameters)
        .build();

CreateModelInteraction params =
    CreateModelInteraction.builder()
        .model("gemini-4-hydrogen")
        .input(
            InteractionsInput.of(
                "Click the submit button. If you need a second factor authentication code, ask me."))
        .tools(
            Arrays.asList(
                ComputerUse.builder()
                    .environment(EnvironmentEnum.MOBILE)
                    .excludedPredefinedFunctions(Arrays.asList("click"))
                    .build(),
                yieldToUserTool))
        .build();

Interaction interaction =
    client.interactions.create(CreateInteractionRequestBody.of(params)).interaction().get();
package main

import (
    "context"
    "log"

    "google.golang.org/genai"
    "google.golang.org/genai/interactions/models/interactions"
    "google.golang.org/genai/interactions/models/operations"
)

func main() {
    ctx := context.Background()
    client, err := genai.NewClient(ctx, nil)
    if err != nil {
        log.Fatal(err)
    }

    yieldToUserTool := interactions.NewTool(interactions.Function{
        Name:        genai.Ptr("yield_to_user"),
        Description: genai.Ptr("Yields control back to the user for assistance or verification when an automated action is unsafe or ambiguous."),
        Parameters: map[string]any{
            "type": "object",
            "properties": map[string]any{
                "reason": map[string]any{
                    "type":        "string",
                    "description": "The reason why the agent is yielding control to the human.",
                },
            },
            "required": []string{"reason"},
        },
    })

    _, err = client.Interactions.Create(ctx, operations.CreateInteractionRequest{
        Body: operations.NewCreateInteractionRequestBody(interactions.CreateModelInteraction{
            Model: interactions.Model("gemini-4-hydrogen"),
            Input: interactions.NewInteractionsInput("Click the submit button. If you need a second factor authentication code, ask me."),
            Tools: []interactions.Tool{
                interactions.NewTool(interactions.ComputerUse{
                    Environment:                 interactions.EnvironmentEnumMobile.ToPointer(),
                    ExcludedPredefinedFunctions: []string{"click"},
                }),
                yieldToUserTool,
            },
        }),
    })
    if err != nil {
        log.Fatal(err)
    }
}

管理思考層級

對於電腦操作代理,你可以設定不同的思考層級,在動作品質與執行速度之間取得平衡。較低的思考層級通常就能為標準自動化任務帶來良好的平衡。

安全與資安

設定安全政策

Gemini 4 與 Gemini 3.x 模型內建安全服務類別,可協助判斷是否需要使用者確認。

安全政策類別 說明
FINANCIAL_TRANSACTIONS 封鎖涉及付款、零售結帳或受管制商品的動作,或觸發確認流程。
SENSITIVE_DATA_MODIFICATION 保護健康、財務或政府紀錄,避免未經授權的修改。
COMMUNICATION_TOOL 限制代理自主傳送電子郵件、聊天訊息或草稿。
ACCOUNT_CREATION 限制代理在網站上自主註冊新帳號。
DATA_MODIFICATION 管控整體檔案系統修改、資料共用與儲存空間刪除。
USER_CONSENT_MANAGEMENT Cookie 同意橫幅與隱私提示需由使用者接手處理。
LEGAL_TERMS_AND_AGREEMENTS 防止模型自主接受服務條款或具法律約束力的合約。

安全覆寫

你可以傳入覆寫設定,覆寫特定政策:

安全覆寫代表你的偏好,但在某些情況下,模型仍可能回傳 safety_decision,其中帶有 require_confirmation。無論是否設定覆寫,你的應用程式都應一律實作安全決策的處理邏輯。

from google import genai

client = genai.Client()

interaction = client.interactions.create(
    model="gemini-4-hydrogen",
    input="Clean up the local folder by archiving old logs.",
    tools=[
        {
            "type": "computer_use",
            "environment": "desktop",
            "disabled_safety_policies": [
                "data_modification"
            ]
        }
    ]
)
import { GoogleGenAI } from '@google/genai';

const ai = new GoogleGenAI();

const interaction = await ai.interactions.create({
    model: "gemini-4-hydrogen",
    input: "Clean up the local folder by archiving old logs.",
    tools: [
        {
            type: "computer_use",
            environment: "desktop",
            disabled_safety_policies: [
                "data_modification"
            ]
        }
    ]
});
import com.google.genai.Client;
import com.google.genai.gaos.models.interactions.ComputerUse;
import com.google.genai.gaos.models.interactions.CreateModelInteraction;
import com.google.genai.gaos.models.interactions.DisabledSafetyPolicy;
import com.google.genai.gaos.models.interactions.EnvironmentEnum;
import com.google.genai.gaos.models.interactions.Interaction;
import com.google.genai.gaos.models.interactions.InteractionsInput;
import com.google.genai.gaos.models.operations.CreateInteractionRequestBody;
import java.util.Arrays;

Client client = new Client();

CreateModelInteraction params =
    CreateModelInteraction.builder()
        .model("gemini-4-hydrogen")
        .input(InteractionsInput.of("Clean up the local folder by archiving old logs."))
        .tools(
            Arrays.asList(
                ComputerUse.builder()
                    .environment(EnvironmentEnum.DESKTOP)
                    .disabledSafetyPolicies(
                        Arrays.asList(DisabledSafetyPolicy.DATA_MODIFICATION))
                    .build()))
        .build();

Interaction interaction =
    client.interactions.create(CreateInteractionRequestBody.of(params)).interaction().get();
package main

import (
    "context"
    "log"

    "google.golang.org/genai"
    "google.golang.org/genai/interactions/models/interactions"
    "google.golang.org/genai/interactions/models/operations"
)

func main() {
    ctx := context.Background()
    client, err := genai.NewClient(ctx, nil)
    if err != nil {
        log.Fatal(err)
    }

    _, err = client.Interactions.Create(ctx, operations.CreateInteractionRequest{
        Body: operations.NewCreateInteractionRequestBody(interactions.CreateModelInteraction{
            Model: interactions.Model("gemini-4-hydrogen"),
            Input: interactions.NewInteractionsInput("Clean up the local folder by archiving old logs."),
            Tools: []interactions.Tool{
                interactions.NewTool(interactions.ComputerUse{
                    Environment: interactions.EnvironmentEnumDesktop.ToPointer(),
                    DisabledSafetyPolicies: []interactions.DisabledSafetyPolicy{
                        interactions.DisabledSafetyPolicyDataModification,
                    },
                }),
            },
        }),
    })
    if err != nil {
        log.Fatal(err)
    }
}

提示詞注入偵測

Gemini 3.5 Flash-Lite 以上版本的電腦操作支援進階安全機制,可偵測提示詞注入攻擊。啟用後,這項功能會檢查隨附的螢幕截圖是否含有隱藏的對抗性指令(例如「Ignore previous commands」),偵測到時即封鎖執行。

提示詞注入偵測為選擇性啟用的功能,預設值為 false。

以下範例示範如何在電腦操作工具設定中啟用提示詞注入偵測:

from google import genai

client = genai.Client()

interaction = client.interactions.create(
    model="gemini-4-hydrogen",
    input="Search for flight deals and summarize top results.",
    tools=[
        {
            "type": "computer_use",
            "environment": "desktop",
            "enable_prompt_injection_detection": True,
        }
    ],
)
import { GoogleGenAI } from '@google/genai';

const ai = new GoogleGenAI();

const interaction = await ai.interactions.create({
    model: "gemini-4-hydrogen",
    input: "Search for flight deals and summarize top results.",
    tools: [
        {
            type: "computer_use",
            environment: "desktop",
            enable_prompt_injection_detection: true,
        }
    ]
});
import com.google.genai.Client;
import com.google.genai.gaos.models.interactions.ComputerUse;
import com.google.genai.gaos.models.interactions.CreateModelInteraction;
import com.google.genai.gaos.models.interactions.EnvironmentEnum;
import com.google.genai.gaos.models.interactions.Interaction;
import com.google.genai.gaos.models.interactions.InteractionsInput;
import com.google.genai.gaos.models.operations.CreateInteractionRequestBody;
import java.util.Arrays;

Client client = new Client();

CreateModelInteraction params =
    CreateModelInteraction.builder()
        .model("gemini-4-hydrogen")
        .input(InteractionsInput.of("Search for flight deals and summarize top results."))
        .tools(
            Arrays.asList(
                ComputerUse.builder()
                    .environment(EnvironmentEnum.DESKTOP)
                    .enablePromptInjectionDetection(true)
                    .build()))
        .build();

Interaction interaction =
    client.interactions.create(CreateInteractionRequestBody.of(params)).interaction().get();
package main

import (
    "context"
    "log"

    "google.golang.org/genai"
    "google.golang.org/genai/interactions/models/interactions"
    "google.golang.org/genai/interactions/models/operations"
)

func main() {
    ctx := context.Background()
    client, err := genai.NewClient(ctx, nil)
    if err != nil {
        log.Fatal(err)
    }

    _, err = client.Interactions.Create(ctx, operations.CreateInteractionRequest{
        Body: operations.NewCreateInteractionRequestBody(interactions.CreateModelInteraction{
            Model: interactions.Model("gemini-4-hydrogen"),
            Input: interactions.NewInteractionsInput("Search for flight deals and summarize top results."),
            Tools: []interactions.Tool{
                interactions.NewTool(interactions.ComputerUse{
                    Environment:                    interactions.EnvironmentEnumDesktop.ToPointer(),
                    EnablePromptInjectionDetection: genai.Ptr(true),
                }),
            },
        }),
    })
    if err != nil {
        log.Fatal(err)
    }
}
curl "https://generativelanguage.googleapis.com/v1/interactions?key=${GEMINI_API_KEY}" \
-H 'Content-Type: application/json' \
-d '{
  "model": "gemini-4-hydrogen",
  "input": "Search for flight deals and summarize top results.",
  "tools": [
    {
      "type": "computer_use",
      "environment": "desktop",
      "enable_prompt_injection_detection": true
    }
  ]
}'

確認安全決策

回應的函式呼叫參數中可能包含 safety_decision 參數:

{
  "steps": [
    {
      "type": "function_call",
      "name": "click",
      "arguments": {
        "x": 60,
        "y": 100,
        "safety_decision": {
          "explanation": "Must check check-box",
          "decision": "require_confirmation"
        }
      }
    }
  ]
}

若 safety_decision 為 require_confirmation,請提示最終使用者。若使用者確認,請將 safety_acknowledgement 設定於 function_result 中。

def get_safety_confirmation(safety_decision):
    # Prompt user for confirmation
    print(f"Safety confirmation required: {safety_decision.get('explanation', '')}")
    return "CONTINUE" # Or TERMINATE

# Inside execute_function_calls, check for safety_decision:
if 'safety_decision' in function_call.arguments:
    decision = get_safety_confirmation(function_call.arguments['safety_decision'])
    if decision == "TERMINATE":
        break
    # Include safety_acknowledgement inside the action result
    action_result["safety_acknowledgement"] = True

安全最佳做法

電腦操作帶來獨特的安全與營運風險,因為代表使用者行事的模型可能在畫面上遇到不受信任的內容,或在執行動作時出錯。請落實以下最佳做法,保護使用者資料與系統:

  1. 人類參與(HITL):
  2. 強制使用者確認:當安全回應顯示 require_confirmation 時,提示使用者核准。
  3. 提供自訂安全指令:實作自訂系統指令,定義並落實你自己的安全界限。例如:
            from google import genai

            client = genai.Client()

            system_instruction = """
            ## **RULE 1: Seek User Confirmation (USER_CONFIRMATION)**

            This is your first and most important check. If the next required action falls
            into any of the following categories, you MUST stop immediately, and seek the
            user's explicit permission.

            **Procedure for Seeking Confirmation:**
            * **For Consequential Actions:** Perform all preparatory steps (e.g., navigating,
              filling out forms, typing a message). You will ask for confirmation **AFTER**
              all necessary information is entered on the screen, but **BEFORE** you perform
              the final, irreversible action (e.g., before clicking "Send", "Submit",
              "Confirm Purchase", "Share").
            * **For Prohibited Actions:** If the action is strictly forbidden (e.g., accepting
              legal terms, solving a CAPTCHA), you must first inform the user about the
              required action and ask for their confirmation to proceed.

            **USER_CONFIRMATION Categories:**

            *   **Consent and Agreements:** You are FORBIDDEN from accepting, selecting, or
                agreeing to any of the following on the user's behalf. You must ask the
                user to confirm before performing these actions.
                *   Terms of Service
                *   Privacy Policies
                *   Cookie consent banners
                *   End User License Agreements (EULAs)
                *   Any other legally significant contracts or agreements.
            *   **Robot Detection:** You MUST NEVER attempt to solve or bypass the
                following. You must ask the user to confirm before performing these actions.
                *   CAPTCHAs (of any kind)
                *   Any other anti-robot or human-verification mechanisms, even if you are
                    capable.
            *   **Financial Transactions:**
                *   Completing any purchase.
                *   Managing or moving money (e.g., transfers, payments).
                *   Purchasing regulated goods or participating in gambling.
            *   **Sending Communications:**
                *   Sending emails.
                *   Sending messages on any platform (e.g., social media, chat apps).
                *   Posting content on social media or forums.
            *   **Accessing or Modifying Sensitive Information:**
                *   Health, financial, or government records (e.g., medical history, tax
                    forms, passport status).
                *   Revealing or modifying sensitive personal identifiers (e.g., SSN, bank
                    account number, credit card number).
            *   **User Data Management:**
                *   Accessing, downloading, or saving files from the web.
                *   Sharing or sending files/data to any third party.
                *   Transferring user data between systems.
            *   **Browser Data Usage:**
                *   Accessing or managing Chrome browsing history, bookmarks, autofill data,
                    or saved passwords.
            *   **Security and Identity:**
                *   Logging into any user account.
                *   Any action that involves misrepresentation or impersonation (e.g.,
                    creating a fan account, posting as someone else).
            *   **Insurmountable Obstacles:** If you are technically unable to interact with
                a user interface element or are stuck in a loop you cannot resolve, ask the
                user to take over.
            ---

            ## **RULE 2: Default Behavior (ACTUATE)**

            If an action does **NOT** fall under the conditions for `USER_CONFIRMATION`,
            your default behavior is to **Actuate**.

            **Actuation Means:**  You MUST proactively perform all necessary steps to move
            the user's request forward. Continue to actuate until you either complete the
            non-consequential task or encounter a condition defined in Rule 1.

            *   **Example 1:** If asked to send money, you will navigate to the payment
                portal, enter the recipient's details, and enter the amount. You will then
                **STOP** as per Rule 1 and ask for confirmation before clicking the final
                "Send" button.
            *   **Example 2:** If asked to post a message, you will navigate to the site,
                open the post composition window, and write the full message. You will then
                **STOP** as per Rule 1 and ask for confirmation before clicking the final
                "Post" button.

                After the user has confirmed, remember to get the user's latest screen
                before continuing to perform actions.

            # Final Response Guidelines:
            Write final response to the user in the following cases:
            - User confirmation
            - When the task is complete or you have enough information to respond to the user
            """

            interaction = client.interactions.create(
                model="gemini-4-hydrogen",
                system_instruction=system_instruction,
                input="Prepare a draft but do not send.",
                tools=[{
                    "type": "computer_use",
                    "environment": "browser"
                }]
            )
            import { GoogleGenAI } from '@google/genai';

            const ai = new GoogleGenAI();

            const systemInstruction = `
            ## **RULE 1: Seek User Confirmation (USER_CONFIRMATION)**

            This is your first and most important check. If the next required action falls
            into any of the following categories, you MUST stop immediately, and seek the
            user's explicit permission.

            **Procedure for Seeking Confirmation:**
            * **For Consequential Actions:** Perform all preparatory steps (e.g., navigating,
              filling out forms, typing a message). You will ask for confirmation **AFTER**
              all necessary information is entered on the screen, but **BEFORE** you perform
              the final, irreversible action (e.g., before clicking "Send", "Submit",
              "Confirm Purchase", "Share").
            * **For Prohibited Actions:** If the action is strictly forbidden (e.g., accepting
              legal terms, solving a CAPTCHA), you must first inform the user about the
              required action and ask for their confirmation to proceed.

            **USER_CONFIRMATION Categories:**

            *   **Consent and Agreements:** You are FORBIDDEN from accepting, selecting, or
                agreeing to any of the following on the user's behalf. You must ask the
                user to confirm before performing these actions.
                *   Terms of Service
                *   Privacy Policies
                *   Cookie consent banners
                *   End User License Agreements (EULAs)
                *   Any other legally significant contracts or agreements.
            *   **Robot Detection:** You MUST NEVER attempt to solve or bypass the
                following. You must ask the user to confirm before performing these actions.
                *   CAPTCHAs (of any kind)
                *   Any other anti-robot or human-verification mechanisms, even if you are
                    capable.
            *   **Financial Transactions:**
                *   Completing any purchase.
                *   Managing or moving money (e.g., transfers, payments).
                *   Purchasing regulated goods or participating in gambling.
            *   **Sending Communications:**
                *   Sending emails.
                *   Sending messages on any platform (e.g., social media, chat apps).
                *   Posting content on social media or forums.
            *   **Accessing or Modifying Sensitive Information:**
                *   Health, financial, or government records (e.g., medical history, tax
                    forms, passport status).
                *   Revealing or modifying sensitive personal identifiers (e.g., SSN, bank
                    account number, credit card number).
            *   **User Data Management:**
                *   Accessing, downloading, or saving files from the web.
                *   Sharing or sending files/data to any third party.
                *   Transferring user data between systems.
            *   **Browser Data Usage:**
                *   Accessing or managing Chrome browsing history, bookmarks, autofill data,
                    or saved passwords.
            *   **Security and Identity:**
                *   Logging into any user account.
                *   Any action that involves misrepresentation or impersonation (e.g.,
                    creating a fan account, posting as someone else).
            *   **Insurmountable Obstacles:** If you are technically unable to interact with
                a user interface element or are stuck in a loop you cannot resolve, ask the
                user to take over.
            ---

            ## **RULE 2: Default Behavior (ACTUATE)**

            If an action does **NOT** fall under the conditions for \`USER_CONFIRMATION\`,
            your default behavior is to **Actuate**.

            **Actuation Means:**  You MUST proactively perform all necessary steps to move
            the user's request forward. Continue to actuate until you either complete the
            non-consequential task or encounter a condition defined in Rule 1.

            *   **Example 1:** If asked to send money, you will navigate to the payment
                portal, enter the recipient's details, and enter the amount. You will then
                **STOP** as per Rule 1 and ask for confirmation before clicking the final
                "Send" button.
            *   **Example 2:** If asked to post a message, you will navigate to the site,
                open the post composition window, and write the full message. You will then
                **STOP** as per Rule 1 and ask for confirmation before clicking the final
                "Post" button.

                After the user has confirmed, remember to get the user's latest screen
                before continuing to perform actions.

            # Final Response Guidelines:
            Write final response to the user in the following cases:
            - User confirmation
            - When the task is complete or you have enough information to respond to the user
            `;

            const interaction = await ai.interactions.create({
                model: "gemini-4-hydrogen",
                system_instruction: systemInstruction,
                input: "Prepare a draft but do not send.",
                tools: [{
                    type: "computer_use",
                    environment: "browser"
                }]
            });
import com.google.genai.Client;
import com.google.genai.gaos.models.interactions.ComputerUse;
import com.google.genai.gaos.models.interactions.CreateModelInteraction;
import com.google.genai.gaos.models.interactions.EnvironmentEnum;
import com.google.genai.gaos.models.interactions.Interaction;
import com.google.genai.gaos.models.interactions.InteractionsInput;
import com.google.genai.gaos.models.operations.CreateInteractionRequestBody;
import java.util.Arrays;

Client client = new Client();

String systemInstruction =
    "## **RULE 1: Seek User Confirmation (USER_CONFIRMATION)**\n\n"
        + "This is your first and most important check. If the next required action falls "
        + "into any of the following categories, you MUST stop immediately, and seek the "
        + "user's explicit permission.\n\n"
        + "## **RULE 2: Default Behavior (ACTUATE)**\n\n"
        + "If an action does **NOT** fall under the conditions for `USER_CONFIRMATION`, "
        + "your default behavior is to **Actuate**.";

CreateModelInteraction params =
    CreateModelInteraction.builder()
        .model("gemini-4-hydrogen")
        .systemInstruction(systemInstruction)
        .input(InteractionsInput.of("Prepare a draft but do not send."))
        .tools(
            Arrays.asList(
                ComputerUse.builder().environment(EnvironmentEnum.BROWSER).build()))
        .build();

Interaction interaction =
    client.interactions.create(CreateInteractionRequestBody.of(params)).interaction().get();
package main

import (
    "context"
    "log"

    "google.golang.org/genai"
    "google.golang.org/genai/interactions/models/interactions"
    "google.golang.org/genai/interactions/models/operations"
)

func main() {
    ctx := context.Background()
    client, err := genai.NewClient(ctx, nil)
    if err != nil {
        log.Fatal(err)
    }

    systemInstruction := "## **RULE 1: Seek User Confirmation (USER_CONFIRMATION)**\n\n" +
        "This is your first and most important check. If the next required action falls " +
        "into any of the following categories, you MUST stop immediately, and seek the " +
        "user's explicit permission.\n\n" +
        "## **RULE 2: Default Behavior (ACTUATE)**\n\n" +
        "If an action does **NOT** fall under the conditions for `USER_CONFIRMATION`, " +
        "your default behavior is to **Actuate**."

    _, err = client.Interactions.Create(ctx, operations.CreateInteractionRequest{
        Body: operations.NewCreateInteractionRequestBody(interactions.CreateModelInteraction{
            Model:             interactions.Model("gemini-4-hydrogen"),
            SystemInstruction: genai.Ptr(systemInstruction),
            Input:             interactions.NewInteractionsInput("Prepare a draft but do not send."),
            Tools: []interactions.Tool{
                interactions.NewTool(interactions.ComputerUse{
                    Environment: interactions.EnvironmentEnumBrowser.ToPointer(),
                }),
            },
        }),
    })
    if err != nil {
        log.Fatal(err)
    }
}
  1. 安全的執行環境:在安全、沙箱化的環境中執行代理,以限制其潛在影響。可以是沙箱化的虛擬機器(VM)、容器(例如 Docker),或權限受限的專用瀏覽器設定檔。沙箱設定指引(使用 Docker)請參閱 GitHub 參考實作。
  2. 輸入清理:清理提示詞中所有使用者產生的文字,降低非預期指令或提示詞注入的風險。這是一層有用的安全防護,但不能取代安全的執行環境。
  3. 內容防護:使用防護機制與內容安全 API,評估使用者輸入、工具輸入與輸出,以及代理的回應是否適當,並偵測提示詞注入與越獄行為。
  4. 允許清單與封鎖清單:實作篩選機制,控制模型可以前往的位置與可以執行的動作。先從禁止網站的封鎖清單著手是個好起點,而限制更嚴格的允許清單則更加安全。
  5. 可觀測性與記錄:保留詳細記錄,以利除錯、稽核與事件應變。你的用戶端應記錄提示詞、螢幕截圖、模型建議的動作(function_call)、安全回應,以及用戶端最終執行的所有動作。
  6. 環境管理:確保 GUI 環境保持一致。意料之外的彈出視窗、通知或版面變化都可能讓模型混淆。盡可能讓每個新任務都從已知的乾淨狀態開始。

模型版本

你可以搭配以下模型使用電腦操作:

  • Gemini 4 Carbon(gemini-4-carbon):我們能力最強的推理模型,支援電腦操作,適用於複雜推理與代理式 UI 工作流程。
  • Gemini 4 Argon(gemini-4-argon):高效的 Gemini 4 前沿推理模型,支援電腦操作;相較於 Gemini 4 Carbon,回應更快、每回合的權杖消耗更低。
  • Gemini 4 Hydrogen(gemini-4-hydrogen):兼具尖端智慧、速度與效率,適用於日常程式設計、電腦操作與代理式 UI 工作流程。
  • Gemini 3.8 Flash(gemini-3.8-flash):上一代 Flash 模型,支援電腦操作,具備 UI 互動與可靠的工具呼叫能力。
  • Gemini 3.5 Flash-Lite(gemini-3.5-flash-lite):低延遲、高成本效益的模型,支援電腦操作。
  • Gemini 3 Flash Preview(gemini-3-flash-preview):支援電腦操作的預覽模型。

後續步驟