Add website sources
Crawl a website, select discovered pages, add a single URL, and diagnose pages that cannot be indexed.
Crawl a website
Open Website sources
Go to Training Data → Website and enter the full public origin, including https://.
Discover and filter pages
Start the crawl, review discovered links, and filter the list. Select only pages that contain reliable customer-facing information.
Add selected pages
Submit the selection. Each URL is processed separately and receives its own status.
Verify the result
Wait for Indexed, preview extracted text, and ask a question that requires the page.
Use the individual URL option for a single page that is not linked from the main site.
On the Free plan, each website crawl trains the first 100 pages Fetchply discovers. If your site has more pages, an upgrade popup appears across the agent dashboard whether the pages were found during first setup or added later. The agent Overview also keeps a reminder visible so you can return to the plan options later. This crawl limit does not apply to Shopify-connected agents.
Manage crawled pages
Below the crawl form, the Websites table shows one row per crawled site with its page counts; removing a website deletes all of its pages from the agent's knowledge. The Crawled Pages table lists every page with its status. Search by title or address, filter by status (for example only failed pages), retry a failed page, open the original page, or delete a page you no longer want the agent to use.
Pages with very little readable text can fail. Fetchply rejects extracted pages with fewer than about ten words.
Login walls, robots rules, bot protection, client-only rendering, redirects, and network errors can prevent extraction. Use a file or text source when the public page cannot be read safely.
Retry failed pages one at a time
An agent trains one job at a time, so Retry page works on a single page per run. While a retry is in progress the other retry buttons stay dimmed until it finishes, then you can start the next one.
To refresh many pages in one job, use Start Re-Train on the agent's Re-Train Content page. That reads the pages your site links to, so it recovers failed pages the site still links to. A page nobody links to, such as one you added on its own, does not come back that way; add it again as an individual URL instead.
Troubleshooting
"A training job is already in progress for this agent" Something is already training this agent: an earlier retry, a re-train, a scheduled refresh, a file that is still being processed, or another team member's action. Wait for it to finish, watch the page status change, then retry the page again.
The retry button is dimmed Another page on the same agent is retrying right now. It becomes available again as soon as that page finishes.
Nothing happens after clicking retry several times quickly Only the first click starts a job; the rest are ignored on purpose so the agent is not asked to train twice at the same time. Give the first retry time to finish before starting another.
The agent says it is training but nothing is progressing Open the agent, choose More agent actions, and select Stop training. Once it stops, retry the page again.
"Failed to queue recrawl job" The retry could not be started. Refresh the page and try once more. If it keeps happening, remove the page and add it again as an individual URL.
The page fails again after a retry The page itself cannot be read. Check whether it needs a login, blocks automated visitors, or renders its text only in the browser, and add the content as a file or text source instead.
A retry finishes but the answer does not change The agent may have stored an older copy of the page. Delete the page from Crawled Pages, then add it again as an individual URL so the newest text is indexed.