API Documentation

Get Coordinate From Description (Recommended)

Analyzes an image and returns the coordinate in a single, synchronous request. This is the recommended approach for most applications.

Parameters

• image OR imageBase64 - The screenshot to analyze (see details below)

  Option 1: File Upload

  Upload the image as a file using multipart/form-data:

  // Using form data
  const formData = new FormData();
  formData.append('image', imageFile);  // imageFile is your File or Blob object
  formData.append('taskDescription', 'Find the login button');

  Option 2: Base64 String

  Send the image as a base64-encoded string in the request body:

  // Using JSON with base64
  const requestBody = {
    imageBase64: 'data:image/png;base64,iVBORw0KGgoAAAANSUhEU...', // Your base64 image
    taskDescription: 'Find the login button'
  };

  The base64 string can be provided in two formats:

  • Data URL format: data:image/png;base64,iVBORw0KGgoA... (includes MIME type)
  • Raw base64: iVBORw0KGgoA... (without MIME type prefix, assumed to be PNG)
• taskDescription - Description of what to find/click in the image
• mechanism (optional) - The model to use. Options:
  • "screengrasp2" (default) - Reasoning Click Prediction Model using an ensemble approach. You can also use "screengrasp2-low", "screengrasp2-medium", or "screengrasp2-high" as a shortcut to specify reasoning effort (see parameters below).
  • "llabs" - CUA-NAV by LLABS model - Great for most tasks
  • "anthropic-computer-use" - Advanced computer interaction model
  • "openai-computer-use" - OpenAI's Computer Use model
  • "qwen25-vl-72b" - Qwen 25 VL 72B model
• parameters (optional) - Additional configuration parameters:
  • reasoningEffort - Only in effect for ScreenGrasp2. Controls the balance between speed and accuracy:
    • "low" (default) - Fastest result, minimal token consumption (typically 50 tokens), often faster than OpenAI or Anthropic Computer Use
    • "medium" - Higher reliability with reasonable speed, takes more time to confirm correct results, may consume up to 200 tokens (most often still only 50 tokens)
    • "high" - Highest reliability, uses advanced image preprocessing to analyze separate image sections in greater detail, consumes 200-600 tokens, slightly slower but more thorough. Especially recommended for high resolution screenshots.
    You can also use the "-low", "-medium", or "-high" suffixes with the "screengrasp2" mechanism identifier instead of using this parameter to specify reasoning effort.

  Example

  // Including parameters in your request
  const requestBody = {
    imageBase64: 'data:image/png;base64,iVBORw0KGgoAAAANSUhEU...', 
    taskDescription: 'Find the login button',
    mechanism: 'screengrasp2',
    parameters: {
      reasoningEffort: 'high'  // For highest accuracy
    }
  };

Response

• On success:
  • x, y - Coordinates of the predicted click position
  • confidence - Confidence score (0-1)
  • analysisId - ID of the analysis
  • mechanism - The model used for the prediction
  • explanation (sometimes included) - The reasoning behind the prediction
• On error: error message and status code