AI agent context: a documentation index for this site is available at /llms.txt. A markdown version of this page is also available by appending .md to this page URL.
Web scraping
Web scraping is a Sitemap and Web source configuration option that lets administrators exclude irrelevant sections, create metadata, and generate sub-items when indexing web pages.