Efficiently build your agents knowledge base with the TIXAE AI Crawler
Set Source URLs
new job
and Enter the main URL(s) you want the crawler to begin with (e.g., https://www.tixaeagents.ai).Configure Crawl Settings
Regular crawl
Deep crawl
Refresh Rate
/collections/protein-powder
more frequently than the main page, as product information changes more often.Specify Page Limit
/sitemap.xml
at the end of a valid URL. Ex. https://www.tixaeagents.ai/sitemap.xml or use thisDefine URL Patterns
Example: Match URLs
Example: Unmatch Patterns
Unique document ID
URL
Title
Description
Character count
Select Pages
Export
Import
URL
Title
Description
Detected URLs
Content
Completed
.