Learn how to add a location for content crawling, schedule the crawler for each location and manually crawl locations.

Adding Locations to be Crawled:

Navigate to Administration > Crawler Setup and go to the Content Locations tab. Click on the Add Location button, then add the URL you want to index. The crawler will crawl and index all subfolders by default. In the “Excluded” section you can add any sites and subfolders you wish to exclude. You can also add any pages within the excluded sites that you wish to include as an exception.

Note: The format of excludes and includes should just be the subfolder from the “/” in the URL as opposed to the full URL of the subfolder. For example, for to exclude GrokSpark Blog we would just add “/blog” to the excludes list on our Web page location.

Note: To define a Content Type as part of a set of subfolders, just exclude those from one Location and include them in another. For example, you can exclude /blog from your Web page Content Type and then add them as a Location for a Blog post Content Type.

Scheduling:

When adding a location you can select when you want the location to be crawled or choose to do the crawling and indexing manually.

Location Actions:

Once you have added at least one location, there are a few actions you can take from the Content Locations tab.

In the Actions column next to each location you have added you can take the following actions: You can edit a site, delete a site or you can also click the “Re-crawl Site” icon to recrawl and index a location.

Content Types:

The Content Types tab allows you to add Content Types and shows you the number of indexed content listings for each type.

To add a Content Type, just click the Add Content Type button, type in the name of your new Content Type and click Add Content Type.