A query language for the web

AgentQL is an AI-powered query language and parser that use natural language selectors for web scraping and automation. It offers resilient, self-healing, cross-site compatible queries, and structured data output, so you can write your script once and execute anywhere.

Fetching data for web scraping

You can use this example query can use to locate products, their names, descriptions, and prices on any eCommerce page:

{
    products[] {
      name
      description
      price(integer)
    }
}

You could think up additional fields like "color" or "size"—whatever you add, AgentQL will find. Pass this query to query_data to return a JSON object containing the data you requested:

example_script.py
python
page.query_data("""
    {
        products[] {
          name
          description
          price(integer)
        }
    }
    """
)

Try it out with any shopfront in our playground.

Learn more about how to use AgentQL for data extraction in our scraping docs.

Fetching elements for workflow automation

You can use this example to locate a search box and its button on a web page.

{
    search_input_field
    search_button
}

Using our SDK, you can pass an AgentQL query to query_elements to return Playwright locators and perform actions on them through the browser:

example_script.py
python
page.query_elements(
    {
        search_input_field
        search_button
    }
)

Or return a single element using a natural language prompt describing the element you're looking for with AgentQL's get_by_prompt method:

example_script.py
python
page.get_by_prompt("the search input field")

Learn more about how to use AgentQL for workflow automation in our automation docs.

Advantages over traditional parsers

With AgentQL, developers can write concise, reusable queries that work across multiple sites with similar data structures. Its AI-powered selectors adapt to UI changes, significantly reducing maintenance overhead. For example, a single AgentQL query could extract listing data from both Redfin and Zillow, and continue to function even after site updates, A/B tests, and even redesigns. AgentQL pinpoints the data necessary, removing the need to build custom LLM pipelines to sift through HTML soup. This approach allows developers to focus on data utilization rather than constantly updating scraping logic, leading to more robust and efficient data pipelines.

  • Works on any page—public or private, any site, any URL, even behind authentication
  • Self-healing—in the face of dynamic content and changing page structures, AgentQL still returns the same results
  • Reusable code—the same query works for scraping across multiple similar pages
  • Structured format you define—shape the JSON response with your query

AgentQL lets you spend less time writing lines of code and updating broken selectors and more time extracting data and building automations.

Features

Semantic Selectors

AgentQL uses AI to build a semantic understanding of the context surrounding the web elements on a page. Elements are found based on their meaning and context, not just their position in the DOM, making them more:

  • Stable even when website layouts change over time.
  • Reusable across sites, standardizing outputs.
  • Intuitive for both developers and non-technical users to write

Natural Language Queries

With AgentQL, you describe what you're looking for in plain English and can even pass entire prompts to explain in great detail what you’re looking for on a page. This makes queries more readable across teams and time and more maintainable.

Controlled Output

Your query defines the structure of your output data, eliminating post-processing steps.

Deterministic Results

AgentQL not only allows you to define an exact response structure, but also provides consistent and reliable results. You will get the same output for the same input, every time. This lets you confidently automate processes and tests.

Primary Use Cases

AgentQL is useful for data scraping and extraction, web automation, and testing.

Data Scraping

AgentQL can extract structured data from websites:

  • Gather pricing details from multiple storefronts
  • Collect social media metrics
  • Aggregate news and articles from multiple sources

Web Automation

Streamline repetitive tasks and complex workflows:

  • Automate form submissions
  • Interact with web sites programmatically
  • Create powerful web-based bots and agents

End-to-End Testing

Build more robust and maintainable test suites:

  • Write tests that are resistant to UI changes
  • Reduce test flakiness and maintenance overhead

Conclusion

AgentQL can change how you interact with web content. If you’d like to dive in, check these out: