AgentQL Tools

The agentql.tools module provides utility methods to help with data extraction and web automation.

Paginate

The following example demonstrates how to use the paginate method to collect data from multiple pages:

agentql_pagination_example.py
python
import agentql
from agentql.tools.sync_api import paginate 
from playwright.sync_api import sync_playwright

with sync_playwright() as p, p.chromium.launch(headless=False) as browser:
    page = agentql.wrap(browser.new_page())
    page.goto("https://news.ycombinator.com/")

    # Define the query to extract the titles of the posts
    QUERY = """
    {
        posts[] {
            title
        }
    }
    """

    # Collect data from the first 3 pages using the query
    paginated_data = paginate(page, QUERY, 3)
    print(paginated_data)

Methods

Paginate

Collects data from multiple pages using an AgentQL query. Internally, the function first attempts to find the operable element to navigate to the next page and click it, then uses the provided query to extract the data from the page. The function then repeats this process for the specified number of pages.

note

The paginate function returns data collected from each page into a single, aggregated list. If you wish to step through each page's data, use the navigate_to_next_page method instead.

Usage

paginate_example.py
python
paginated_data = paginate(page, QUERY, 3)

Arguments

  • page AgentQL Page

    The AgentQL Page object.

  • query str

    An AgentQL query in String format.

  • number_of_pages int

    Number of pages to paginate over.

  • timeout int (optional):

    Timeout value in seconds for the connection with backend API service for querying the pagination element.

  • wait_for_network_idle bool (optional)

    Whether to wait for network reaching full idle state before querying the page for pagination element. If set to False, this method will only check for whether page has emitted load event. Default is True.

  • include_hidden bool (optional)

    Whether to include hidden elements on the page when querying for pagination element. Defaults to False.

  • mode ResponseMode (optional):

    The mode of the query for retrieving the pagination element. It can be either standard or fast. Defaults to fast mode.

  • force_click bool (optional):

    Whether to force click on the pagination element. Defaults to False.

Returns

  • List[dict]

    List of dictionaries containing the data from each page.

Query document

The following example demonstrates how to use the query_document method to query a document (.pdf, .png, .jpg, .jpeg) using AgentQL.

note

The query_document function consumes 1 api call per image (JPG, JPEG, JPG), and 1 api call for each page within a PDF.

query_document_example.py
python
from agentql.tools.sync_api import query_document

QUERY = """
{
    name
}
"""

file_path = "path/to/file.pdf"

async def main():
    response = query_document(
        file_path,
        query=QUERY,
    )
    print(f"name: {response['name']}")

main()

Methods

query_document

Queries a document (.pdf, .png, .jpg, .jpeg) using AgentQL.

Usage

query_document_example.py
python
response = query_document(
        file_path,
        query=QUERY,
    )

Arguments

  • file_path str

    The path to the document to query.

  • query str

    The query to execute on the document. Either query or prompt must be provided but not both.

  • prompt str

    The prompt to execute on the document. Either query or prompt must be provided but not both.

  • timeout int (optional):

    Timeout value in seconds for the connection with backend API service.

  • mode ResponseMode (optional): The mode of the query. It can be either standard or fast. Defaults to fast mode.

Returns

  • dict

    Data that matches the query.

Types

ResponseMode

The ResponseMode type specifies the mode of querying for query_elements(), query_data(), and get_by_prompt() methods. It's expecting the following two values:

  • standard

Executes the query in Standard Mode. Use this mode when your queries are complex or extensive data retrieval is necessary.

  • fast

Executes the query more quickly, potentially at the cost of response accuracy. This mode is useful in situations where speed is prioritized, and the query is straightforward.

Use remote browser

The following example demonstrates how to use the create_browser_session method to create a remote browser session:

remote_browser_example.py
python
import agentql
from agentql.tools.sync_api import create_browser_session, UserAgentPreset 
from playwright.sync_api import sync_playwright

session = create_browser_session(ua_preset=UserAgentPreset.WINDOWS)

with sync_playwright() as p:
    browser = p.chromium.connect_over_cdp(session.cdp_url)
    page = agentql.wrap(browser.new_page())

    # Use AgentQL as normal
    page.goto("https://example.com")

    # Get streaming URL to view the page
    print(f"View at: {session.get_page_streaming_url(0)}")

Methods

create_browser_session

Creates a new remote browser session that provides a Chrome DevTools Protocol (CDP) URL for connecting to a remote browser instance and streaming URLs for real-time page viewing.

note

Remote browser sessions allow you to run browser automation on remote infrastructure while maintaining the familiar AgentQL API. See the Remote Browser Guide for complete usage examples.

Usage

create_browser_session_example.py
python
session = create_browser_session(ua_preset=UserAgentPreset.MACOS)

Arguments

  • ua_preset UserAgentPreset (optional)

    User agent preset to simulate different operating systems. Can be UserAgentPreset.WINDOWS, UserAgentPreset.MACOS, or UserAgentPreset.LINUX.

  • profile BrowserProfile (optional)

    Browser profile preset for anti-detection configuration. Can be BrowserProfile.LIGHT or BrowserProfile.STEALTH.

  • inactivity_timeout_seconds int (optional)

    Inactivity timeout in seconds for the remote browser session. If not specified, uses the default timeout.

Returns

  • BrowserSession

    Object containing CDP URL and methods for streaming page access.

Raises

  • APIKeyError: If API key is not set or invalid
  • Exception: If session creation fails

Types

BrowserSession

Represents an allocated remote browser session with CDP access and streaming capabilities.

Properties

  • cdp_url str

    Chrome DevTools Protocol URL for connecting to the remote browser.

Methods

  • get_page_streaming_url(page_num) -> str

    Returns a streaming URL for viewing a specific page in real-time.

    Arguments:

    • page_num int: Page number (0-indexed)

UserAgentPreset

Enum for user agent presets to simulate different operating systems:

  • UserAgentPreset.WINDOWS: Windows user agent and browser characteristics
  • UserAgentPreset.MACOS: macOS user agent and browser characteristics
  • UserAgentPreset.LINUX: Linux user agent and browser characteristics

BrowserProfile

Enum for browser profile presets:

  • BrowserProfile.LIGHT: Basic browser profile with standard configurations
  • BrowserProfile.STEALTH: Enhanced profile with advanced stealth measures for bot detection avoidance