How to open a headless browser with AgentQL

Headless browsers are powerful tools for web automation, testing, and scraping. They let you run a browser without the need to spin up a visual interface. This can allow scripts to execute faster or in the background. AgentQL's SDKs leverage headless browsers to interact with web pages and execute queries without the need for a visible user interface.

Overview

This guide shows you how and when to use a headless browser, and how to execute queries inside one with AgentQL.

Why use a headless browser?

Headless browsers offer several advantages:

  • Speed: They're faster than full browsers as they don't render visuals.
  • Resource efficiency: They use less memory and CPU.
  • Automation: Perfect for running tests or scripts without manual intervention.
  • Server-side operation: Can run on machines without a GUI.

AgentQL and Playwright

AgentQL's SDK uses Playwright, a powerful browser automation library, to handle headless browsing. Playwright supports multiple browser engines and provides a rich API for web automation.

Running a query in a headless browser

Here's a basic example of how to use AgentQL with a headless browser:

import agentql
from playwright.sync_api import sync_playwright

# Initialise the browser
with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = agentql.wrap(browser.new_page())

    page.goto("https://scrapeme.live/shop/")

    SEARCH_QUERY = """
    {
        products[] {
            product_name
            product_price
        }
    }
    """
    response = page.query_data(SEARCH_QUERY)

    print("RESPONSE:", response)

    # Close the browser
    browser.close()

This script does the following:

  1. Initializes a headless browser
  2. Navigates to a webpage
  3. Creates an AgentQL query
  4. Executes the query and retrieves the result
  5. Closes the browser

Related content