The Maximum Pages to Crawl Onsite option is useful for subscribers who have implemented a mini-portal for creating search functionality around multiple domains / web sites.

Based on your account's page limitations, there may be times when you wish to limit the total number of pages that FusionBot includes in your index from your main site (the actual site / URL used when subscribing to FusionBot), so that the FusionBot spider will stop crawling your main site and begin crawling the URLs entered as part of your mini-portal.

For example, if you have subscribed to the FusionBot Bronze Package, your default page limitation is 500 pages. If your default site subscribed to FusionBot has a large amount of content that would potentially use your entire page allowance, or a large share of this allowance, and you wish to include additional sites in your search results via the mini-portal feature, you can specify via the "Maximum Pages to Crawl Onsite" field, located by clicking the "Spider" tab, 'index options' option while logged into your account, the maximum number of pages that FusionBot should crawl on your main site before stopping and moving on to your mini-portal entries.

Therefore, if you have 200 pages of content that you would like to include from off-site URLs entered in your mini-portal, you can restrict your main site to the first 300 pages that FusionBot finds on your site, by entering 300 into the "Maximum Pages to Crawl Onsite" field. Doing so will allow for the remaining 200 pages available for your package level to be used for your mini-portal entries.

If you would also like to restrict the number of pages crawled for each entry in your mini-portal, please reference the "PAGES=" variable when entering your mini-portal sites at the following FAQ.