Frequently Asked Questions

Q. How do I debug my script?

Answer In Python SDK, the debug mode is enabled by setting the logging level to DEBUG in your script:

import logging

logging.basicConfig(level=logging.DEBUG)

In JavaScript SDK, the debug mode is enabled by adding the DEBUG=True flag when running the script:

terminal
DEBUG=True node example_script.js

Turning on the debug mode will output relevant debug information to the console.

Q. How do I specify the format of data returned by AgentQL?

Answer Since the release of 0.4.7 version, AgentQL supports providing context to query in the following way:

{
    products[] {
        price(Format the output as a numerical value only, no dollar sign)
        name(Format the output with the following prefix: amazon_)
    }
}

As you can see, by providing formatting instructions as a context into the query, you could instruct AgentQL to return the expected data format.

Q. While writing AgentQL Script, how do I ensure the web page is loaded entirely?

Answer It's currently not possible to programmatically define when a page has finished loading—a script may bring new information on to the page, as in the case of infinite scrolling or lazy loading images.

You can use Playwright API to scroll the page up and down to load the more content.

example_script.js
js
const { wrap } = require("agentql");
const { chromium } = require("playwright");

const QUERY = `
  {
    comments[]
  }
`;

async function main() {
    const browser = await chromium.launch({headless: false});
    const page = await wrap(await browser.newPage());
    await page.goto("https://www.youtube.com/watch?v=1ZvYrAaJOH4");

    for (let i = 0; i < 3; i++) {
        // Scroll down 3 times, each by 1000 pixels
        await page.mouse.wheel(delta_x=0, delta_y=1000);
        await page.waitForPageReadyState();  // Give it the opportunity to load more content
    }

    await page.mouse.wheel(delta_x=0, delta_y=-3000);  // Scroll up 3000 pixels, back to top

    const response = await page.queryData(QUERY);
    console.log(`Video comments: \n${JSON.stringify(response, null, 2)}`);

    await browser.close();
}

main();

Q. How do I make sure that my script waits enough for all the element to appear on the page?

Answer

AgentQL SDK provides the waitForPageReadyState() API to determine page readiness. This method waits for the page to be in a ready state and all the necessary network requests to be completed. However, there are some cases where the page might not be ready even after the method returns. This could be due to the dynamic nature of the page. In such cases, you can use the page.waitForTimeout function to add a additional delay to your script.

example_script.js
js
const { wrap } = require("agentql");
const { chromium } = require("playwright");

const QUERY = `
{
    related_videos[] {
        video_name
        video_rating
    }
}
`;

async function main() {
    const browser = await chromium.launch({headless: false});
    const page = await wrap(await browser.newPage());
    await page.goto("https://www.youtube.com/watch?v=1ZvYrAaJOH4");

    await page.waitForPageReadyState();  // Wait for the page to fully load

    await page.waitForTimeout(5000);  // Give additional time if needed

    const response = await page.queryData(QUERY);
    console.log(`Related videos: \n${JSON.stringify(response, null, 2)}`);

    await browser.close();
}

main();