Add or Edit a Sitemap Source

Members of the Administrators and Content Managers built-in groups can use a Sitemap source to make the content of listed web pages from a sitemap file or a Sitemaps index file searchable.

A sitemap file is a component that can be added to a website and is required when using a Sitemap source. The file contains a list of the website’s URLs along with their respective metadata which include the LMD (last-modified-date). This enables the Sitemap source to perform a refresh rather than a rescan, as is the case with a Web source. For this reason, although a Sitemap source requires the extra step of adding a sitemap file, it offers an increased performance compared to a Web source.

For secured websites (non-public accessible Sitemap), the source supports several authentication modes.

Source Key Characteristics

Requirements

Supported Sitemap File Formats

The source can include web pages from the following sitemap file formats:

• XML (sitemap and index)

• Text

• Syndication Feeds (Atom 1.0 and RSS 2.0)

• HTML

Supported Authentication Schemes

The source can authenticate with the following authentication schemes:

• Basic

• Digest

• NTLM

• Negotiate/Kerberos

• Form based

You can enter the authentication parameters in the “Authentication” Section.

Add or Edit a Sitemap Source

When adding a source, select the Sitemap option.

To edit a source, on the Sources page, click the desired source, and then, in the Action bar, click Edit.

The completion steps are especially important when creating or editing a source of this type.

“Configuration” Tab

In the Add/Edit a Sitemap Source panel, the Configuration tab is selected by default. It contains your source’s general and authentication information, as well as other parameters.

General Information

Source Name

Enter a name for your source.

URLs

Enter the URL(s) to a sitemap index file or sitemap files in either the http:// or https:// form.

Avoid including more than one sitemap in a given source. Instead, create one source for each sitemap.

User Agent

Enter the user agent string you want the Sitemap source to send with HTTP requests to identify itself when downloading pages.

The default value used is: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko) (compatible; Coveobot/2.0;+https://www.coveo.com/bot.html).

Paired Crawling Module

If your source is a Crawling Module source and if you have more than one Crawling Module linked to this organization, select the one with which you want to pair your source. If you change the Crawling Module instance paired with your source, a successful rebuild is required for your change to apply.

Character Optical Recognition (OCR)

Check this box if you want Coveo Cloud to extract text from image files or PDF files containing images. OCR-extracted text is processed as item data, meaning that it’s fully searchable and will appear in the item Quick View. See Enable Optical Character Recognition for details on this feature.

Index

When adding a source, if you have more than one logical (non-Elasticsearch) index in your organization, select the index in which the retrieved content will be stored (see Leverage Many Coveo Indexes). If your organization only has one index, this drop-down menu isn’t visible and you have no decision to make.

“Authentication” Section

When the Sitemap you want to make searchable uses one of the supported authentication types to secure access to its content, expand the Authentication section to configure the source credentials allowing your Coveo organization to gain access to the secured content. You can refer to the Source Credentials Leading Practices for additional information.

The Sitemap source supports the following authentication types. Click the desired authentication method for details on the parameters to configure.

• Basic authentication

  Select this option when the desired website uses the normal NTLM identity. See Basic access authentication for details on how this option works.

• Manual form authentication

  Select this option when the desired website presents users with a form to fill to log in. You must specify the form input names and values.

• Automatic form authentication

  Select this option when the desired website presents users with a form to fill to log in. This is a simplified version of the Manual form authentication. A web automation framework supporting JavaScript rendering executes the authentication process by autodetecting the form inputs.

Basic Authentication

When selecting Basic authentication, enter the credentials of an account on the website you’re making searchable. See Source Credentials Leading Practices.

Manual Form Authentication

When selecting Manual form authentication:

1. In the Form URL box, enter the website login page URL.

2. (Optional) When there’s more than one form on the login page, enter the Form name.

3. Click the Action method drop-down menu, and then select the HTTP verb used to submit the authentication request, which is whether POST or GET.

4. Click the Content type drop-down menu, and then select the content-type headers for HTTP POST requests used by the form authentication process.

5. (Optional) When the authentication request is sent to another URL than the specified form URL, enter the Action URL. Otherwise, leave empty.

6. In the Username input name and the Password input name inputs, inspect the form HTML code for both parameters, locate the corresponding <input name='abc' type='text' /> element, and then enter the name attribute value.

   Based on the HTML code below:

   <input name="login" type="email" />

   <input name="pwd" type="password" />

   login is the username input name and pwd is the password input name.

7. In the Username input value and the Password input value inputs, enter the username and password parameter values respectively.

8. When your form has other parameters than username and password, we recommend that you select the Autodetect hidden inputs check box. Otherwise, you must enter the Input name and Input value for each parameter.

   Under Other inputs, input values are displayed in clear text. You must therefore ensure to enter your sensitive information, i.e., username and password, above the Other inputs, in the Username input name, Password input name, Username input value, and Password input value boxes (see steps 6 and 7).

   

9. Under Confirmation method, select the method that will determine if the authentication request failed.

   Depending on the selected confirmation method, enter the appropriate value:

   • When selecting Redirection to, enter the URL where the Coveo crawler is redirected when the login fails.

     https://mycompany.com/login/failed.html

   • When selecting Missing cookie, in the Value input, enter the name of the cookie that’s set when an authentication is successful.

     ASP.NET_SessionId

   • If you select Missing URL, in the Value input, enter a regex. Coveo will consider that an authentication request has failed if its crawler is not redirected to an URL matching the specified pattern.

   • When selecting Missing text, in the Value input, enter a string to show to authenticated users.

     • Hello, jsmith@mycompany.com!

     • Log out

   • When selecting Text presence, in the Value input, enter a string to show when a login fails.

     • An error has occurred.

     • Your username or password is invalid.

   • When selecting URL presence, in the Value input, enter a regex. Coveo will consider that an authentication request has failed if its crawler is redirected to an URL matching the specified pattern.

Automatic Form Authentication

When selecting Automatic form authentication:

1. In the Form URL box, enter the website login page URL.

2. In the Username and Password boxes, enter the credentials to log in. See Source Credentials Leading Practices.

3. Under Confirmation method, select the method that will determine if the authentication request failed.

   Depending on the selected confirmation method, enter the appropriate value:

   • When selecting Redirection to, enter the URL where the Coveo crawler is redirected when the login fails.

     https://mycompany.com/login/failed.html

   • When selecting Missing cookie, in the Value input, enter the name of the cookie that’s set when an authentication is successful.

     ASP.NET_SessionId

   • If you select Missing URL, in the Value input, enter a regex. Coveo will consider that an authentication request has failed if its crawler is not redirected to an URL matching the specified pattern.

   • When selecting Missing text, in the Value input, enter a string to show to authenticated users.

     • Hello, jsmith@mycompany.com!

     • Log out

   • When selecting Text presence, in the Value input, enter a string to show when a login fails.

     • An error has occurred.

     • Your username or password is invalid.

   • When selecting URL presence, in the Value input, enter a regex. Coveo will consider that an authentication request has failed if its crawler is redirected to an URL matching the specified pattern.

   In addition, if you want Coveo’s first request to be for authentication, regardless of whether authentication is actually required, check the Force login box.

4. If the website relies heavily on asynchronous requests, under Loading delay, enter the maximum delay in milliseconds to wait for the JavaScript to execute on each web page.

5. (Optional) If your automatic form authentication configuration doesn’t work, or if the web page requires specific actions during the login process, you might have to enter a Custom login sequence configuration. Contact the Coveo Support team for help.

“Content to Include” Section

When the website pages indexed by your Sitemap source are rendered dynamically with JavaScript, expand the Content to Include section, and then select the JavaScript-rendered check box. By default, the Sitemap source doesn’t execute the JavaScript code in crawled website pages.

“Crawling Settings” Section

When a target website sometimes responds slower:

1. In the Request timeout box, use the + and - buttons to select the web request timeout value in seconds. The default is 100 seconds. When the value is 0, there’s no timeout. By increasing the timeout value, you increase the delay tolerance and avoid timeout errors.

2. Under Request interval delay, enter the number of milliseconds there should be between each request sent to retrieve your Sitemap content. The default value is 0, which means there’s no speed limitation. To decrease the crawling speed, enter a higher number. The maximum possible delay is 5000 milliseconds.

“Web Scraping” Section

When you want to exclude page sections (such as headers and footers) or extract information from the pages to create metadata, expand the Web Scraping section to use this powerful feature.

"sensitive": false,
"value": "[\n  {\n      \"for\": {\n        \"urls\": [\".*\"]\n      },\n      \"exclude\": [\n        { \"path\": \"#ohHeader\" },\n        { \"path\": \"#MainSection > div.col-md-3\" },\n        { \"path\": \"#answerLink\" }\n      ],\n      \"metadata\": {\n        \"topicTitle\": { \"path\": \"div.topic  h1::text\" },\n        \"topicLastUpdate\": { \"path\": \"#LastUpdate::text\" }\n      }\n  }\n]"
}

“Content Security” Tab

Select who will be able to access the source items through a Coveo-powered search interface. For details on this parameter, see Content Security.

“Access” Tab

In the Access tab, determine whether each group and API key can view or edit the source configuration (see Resource Access):

1. In the Access Level column, select View or Edit for each available group.

2. On the left-hand side of the tab, if available, click Groups or API Keys to switch lists.

Completion

1. Finish adding or editing your source:

   • When you want to save your source configuration changes without starting a build/rebuild, such as when you know you want to do other changes soon, click Add Source/Save.

     To add the source content or to make your changes effective, on the Sources page, you must click Start initial build or Start required rebuild in the source Status column.

     OR

   • When you’re done editing the source and want to make changes effective, click Add and Build Source/Save and Rebuild Source.

     Back on the Sources page, you can review the progress of your source addition or modification.

   Once the source is built or rebuilt, you can review its content in the Content Browser.

2. Optionally, consider editing or adding mappings.

   You can only manage mapping rules once you build the source (see Refresh, Rescan, or Rebuild Sources).

What’s Next?

• Adapt the source update schedule to your needs.

• If you experienced issues while building the source, consider using Sitemap connector hidden parameters.

• If you’re using the Crawling Module to retrieve your content, consider subscribing to deactivation notifications to receive an alert when a Crawling Module component becomes obsolete and stops the content crawling process.