AgentQL Tools
The agentql.tools
module provides utility methods to help with data extraction and web automation.
- Paginate Collect data from multiple pages
- Query Document Query documents (.pdf, .png, .jpg, .jpeg) using AgentQL.
- Use remote browser Create a remote browser session for browser automation.
Paginate
The following example demonstrates how to use the paginate
method to collect data from multiple pages:
import agentql
from agentql.tools.sync_api import paginate
from playwright.sync_api import sync_playwright
with sync_playwright() as p, p.chromium.launch(headless=False) as browser:
page = agentql.wrap(browser.new_page())
page.goto("https://news.ycombinator.com/")
# Define the query to extract the titles of the posts
QUERY = """
{
posts[] {
title
}
}
"""
# Collect data from the first 3 pages using the query
paginated_data = paginate(page, QUERY, 3)
print(paginated_data)
Methods
Paginate
Collects data from multiple pages using an AgentQL query. Internally, the function first attempts to find the operable element to navigate to the next page and click it, then uses the provided query to extract the data from the page. The function then repeats this process for the specified number of pages.
The paginate
function returns data collected from each page into a single, aggregated list. If you wish to step through each page's data, use the navigate_to_next_page
method instead.
Usage
paginated_data = paginate(page, QUERY, 3)
Arguments
-
page
AgentQL PageThe AgentQL Page object.
-
query
strAn AgentQL query in String format.
-
number_of_pages
intNumber of pages to paginate over.
-
timeout
int (optional):Timeout value in seconds for the connection with backend API service for querying the pagination element.
-
wait_for_network_idle
bool (optional)Whether to wait for network reaching full idle state before querying the page for pagination element. If set to
False
, this method will only check for whether page has emittedload
event. Default isTrue
. -
include_hidden
bool (optional)Whether to include hidden elements on the page when querying for pagination element. Defaults to
False
. -
mode
ResponseMode (optional):The mode of the query for retrieving the pagination element. It can be either
standard
orfast
. Defaults tofast
mode. -
force_click
bool (optional):Whether to
force
click on the pagination element. Defaults toFalse
.
Returns
-
List of dictionaries containing the data from each page.
Query document
The following example demonstrates how to use the query_document
method to query a document (.pdf, .png, .jpg, .jpeg) using AgentQL.
The query_document
function consumes 1 api call per image (JPG, JPEG, JPG), and 1 api call for each page within a PDF.
from agentql.tools.sync_api import query_document
QUERY = """
{
name
}
"""
file_path = "path/to/file.pdf"
async def main():
response = query_document(
file_path,
query=QUERY,
)
print(f"name: {response['name']}")
main()
Methods
query_document
Queries a document (.pdf, .png, .jpg, .jpeg) using AgentQL.
Usage
response = query_document(
file_path,
query=QUERY,
)
Arguments
-
file_path
strThe path to the document to query.
-
query
strThe query to execute on the document. Either query or prompt must be provided but not both.
-
prompt
strThe prompt to execute on the document. Either query or prompt must be provided but not both.
-
timeout
int (optional):Timeout value in seconds for the connection with backend API service.
-
mode
ResponseMode (optional): The mode of the query. It can be eitherstandard
orfast
. Defaults tofast
mode.
Returns
-
Data that matches the query.
Types
ResponseMode
The ResponseMode
type specifies the mode of querying for query_elements()
, query_data()
, and get_by_prompt()
methods. It's expecting the following two values:
standard
Executes the query in Standard Mode. Use this mode when your queries are complex or extensive data retrieval is necessary.
fast
Executes the query more quickly, potentially at the cost of response accuracy. This mode is useful in situations where speed is prioritized, and the query is straightforward.
Use remote browser
The following example demonstrates how to use the create_browser_session
method to create a remote browser session:
import agentql
from agentql.tools.sync_api import create_browser_session, UserAgentPreset
from playwright.sync_api import sync_playwright
session = create_browser_session(ua_preset=UserAgentPreset.WINDOWS)
with sync_playwright() as p:
browser = p.chromium.connect_over_cdp(session.cdp_url)
page = agentql.wrap(browser.new_page())
# Use AgentQL as normal
page.goto("https://example.com")
# Get streaming URL to view the page
print(f"View at: {session.get_page_streaming_url(0)}")
Methods
create_browser_session
Creates a new remote browser session that provides a Chrome DevTools Protocol (CDP) URL for connecting to a remote browser instance and streaming URLs for real-time page viewing.
Remote browser sessions allow you to run browser automation on remote infrastructure while maintaining the familiar AgentQL API. See the Remote Browser Guide for complete usage examples.
Usage
session = create_browser_session(ua_preset=UserAgentPreset.MACOS)
Arguments
-
ua_preset
UserAgentPreset (optional)User agent preset to simulate different operating systems. Can be
UserAgentPreset.WINDOWS
,UserAgentPreset.MACOS
, orUserAgentPreset.LINUX
.
Returns
-
Object containing CDP URL and methods for streaming page access.
Raises
APIKeyError
: If API key is not set or invalidException
: If session creation fails
Types
BrowserSession
Represents an allocated remote browser session with CDP access and streaming capabilities.
Properties
-
cdp_url
strChrome DevTools Protocol URL for connecting to the remote browser.
Methods
-
get_page_streaming_url(page_num)
-> strReturns a streaming URL for viewing a specific page in real-time.
Arguments:
page_num
int: Page number (0-indexed)
UserAgentPreset
Enum for user agent presets to simulate different operating systems:
UserAgentPreset.WINDOWS
: Windows user agent and browser characteristicsUserAgentPreset.MACOS
: macOS user agent and browser characteristicsUserAgentPreset.LINUX
: Linux user agent and browser characteristics