Troubleshooting Web source issues

This is for:

Developer System Administrator

This article provides help troubleshooting common issues when indexing content with the Web source.

Identify the issue you’re facing using the case context and symptoms provided. Then, apply the recommended resolution steps and rebuild your source.

Important

Troubleshooting symptoms are provided as a guide. Actual symptoms may vary. For example, Coveo may or may not return an error mentioned among the issue symptoms.

Review the Activity Browser (platform-ca | platform-eu | platform-au) page for a fuller picture of an abnormal indexing activity. You can also download the source update logs for a chronological account of what happened during the indexing process.

Issues are divided into categories. Click a category description below to reach the related section.

Missing pages

Blocklisting

Context and symptoms:

Click for likely cause and resolution

Cause: Your network may be blocking inbound requests from Coveo.

Resolution: Allow inbound requests from Coveo or consider installing the Coveo On-Premises Crawling Module on your infrastructure to push documents to Coveo instead.
RespectUrlCasing setting issue

Context and symptoms:

  • Missing pages have URLs that contain uppercase characters. All indexed pages have URLs that only contain lowercase characters.

  • Other symptoms may vary. For example, the Activity Browser (platform-ca | platform-eu | platform-au) page may display an authentication issue (e.g., a WEB_FORBIDDEN_ERROR error code).

  • The RespectUrlCasing JSON parameter is set to false.

Click for likely cause and resolution

Cause: With RespectUrlCasing set to false, the Web source crawler lowercases a URL it discovers, and then requests the lowercased URL from the server. If the web server is case sensitive, it doesn’t recognize the requested URL and doesn’t serve the request.

Resolution: Access the Edit a source JSON configuration panel and set RespectUrlCasing to true.
Crawling rules issue

Symptom: The Content Browser (platform-ca | platform-eu | platform-au) doesn’t show all the web pages you wanted to index.

Click for likely cause and resolution

Cause: Your current Crawling rules exclusions and inclusions are filtering out the pages you wanted to index.

Resolution: Open your source and review your exclusions and inclusions.

To be indexed, a page:

  • Must not match any exclusion rule, AND

  • It must match at least one inclusion rule (e.g., by selecting the Include all pages that were not excluded option).
Starting URL exclusion

Context and symptoms:

Click for likely cause and resolution

Cause: Your current Crawling rules exclusions and inclusions are filtering out that starting URL. Consequently, the crawler can’t index the pages that are reachable via that starting URL.

Resolution: Open your source. Make adjustments to your exclusions and inclusions to ensure the starting URL and all pages accessible through it aren’t filtered out.

To be indexed, a page:

  • must not match any exclusion rule, AND

  • it must match at least one inclusion rule (e.g., by selecting the Include all pages that were not excluded option).
301 Moved Permanently redirect

Context and symptoms:

  • The Activity Browser (platform-ca | platform-eu | platform-au) page may show a NO_DOCUMENT_INDEXED error code.

  • Your Starting URL doesn’t include a www segment (e.g., https://abc.com).

  • When trying to access that starting URL manually in a browser, you’re automatically redirected to a page that includes the www segment (e.g., https://www.abc.com).

Click for likely cause and resolution

Cause: By default, the Web source only indexes pages that are internal to the website. The Web source is considering the page it’s redirected to (e.g., https://www.abc.com) as external to the website (e.g., https://abc.com). This internal/external validation is unrelated to exclusion and inclusion rules.

Resolution: If you’re only getting started with a new Web source, you might simply want to delete the source, start fresh with a new one, and include the www segment in the Starting URL. Otherwise, you can proceed as follows:

  1. Access the Edit a source JSON configuration panel.

  1. In the JSON configuration field, under the parameters property, add the following code block:

    "IndexSubdomains": {
  "sensitive": false,
  "value": "true"
},

    The JSON configuration field should now look as follows:

    Adding the IndexSubdomains parameter | Coveo

  1. Click Save to exit the Edit a source JSON configuration panel.

  2. If necessary, make adjustments to your exclusions and inclusions to ensure the redirection URLs (e.g., https://www.abc.com/something) aren’t filtered out. You might also need to add exclusion or inclusion rules to filter out unwanted subdomain pages.

  3. Rebuild your source.
Orphan pages

Context and symptoms:

Click for likely cause and resolution

Cause: The missing pages may be orphan pages.

Resolution:

  • Add links to these pages in your website so that the Web source crawler and other search engines may reach them, OR

  • Open your source. Add Starting URLs for your orphan pages. If necessary, make adjustments to your exclusions and inclusions to ensure the added Starting URLs aren’t filtered out.
Missing or invalid basic authentication configuration

Context and symptoms:

  • A page isn’t indexed.

  • The Activity Browser (platform-ca | platform-eu | platform-au) page may show a WEB_AUTHENTICATION_ERROR error code.

  • When trying to access that page manually in a browser, you’re prompted for credentials in a pop-up window.

Click for likely cause and resolution

Cause: Accessing the page content requires basic authentication.

Resolution:

  • Request the authentication credentials from the web server administrator. Then, open your source and configure Basic authentication on the source.

  • If you’re using a password manager (e.g., LastPass), it may replace the previously recorded username and password with different ones as you edit the source. We recommend checking your password manager options and ensuring that it respects the autocomplete="off" attribute.
Missing or invalid form authentication configuration

Context and symptoms:

Click for likely cause and resolution

Cause: Accessing the page requires form authentication.

Resolution:

  • Request the authentication credentials from the web server administrator. Then, open your source and configure Form authentication on the source.

  • If you’re using a password manager (e.g., LastPass), it may replace the previously recorded username and password with different ones as you edit the source. We recommend checking your password manager options and ensuring that it respects the autocomplete="off" attribute.
Authentication status validation issue

Context and symptoms:

  • A page isn’t indexed.

  • Accessing that page requires form authentication.

  • When trying to access that page manually in a browser, the form authentication Login address page is displayed. Typing in the credentials and submitting the login page brings up the page to be indexed.

Click for likely cause and resolution

Cause: The authentication Validation method might not be configured properly.

Resolution: Open your source. Make sure the Validation method you have selected and the associated value are adequate.
Redirection to login page issue

Context and symptoms:

  • A page isn’t indexed.

  • Accessing that page requires form authentication and your source configuration Validation method is Redirection to URL.

  • When trying to access that page manually in a browser, the form authentication Login address page isn’t displayed.

Click for likely cause and resolution

Cause: The Redirection to URL Validation method doesn’t work in your use case. Consequently, the Web source crawler doesn’t know it must authenticate before accessing the page to index.

Resolution:

  • Open your source and choose another Validation method. Select a method based on the way the web server responds when you manually try to access the page to index (when unauthenticated).

  • If no reliable validation method can be found, try enabling the form authentication Force authentication option.
Content freshness issue

Symptom: Pages recently added to the website are still not appearing in the Content Browser (platform-ca | platform-eu | platform-au).

Click for likely cause and resolution

Cause and Resolution: See Indexed content isn’t up to date.

Extra or unwanted pages

Query parameters

Context and symptoms:

Example:

Two items in the Content Browser with identical URIs except for query string parameter values
Click for likely cause and resolution

Cause: You’re currently not specifying that the query string parameter should be ignored.

Resolution: Open your source. In the Advanced settings tab, add the parameter to the Query parameters to ignore list.

Example:

Adding a parameter to the Query parameters to ignore list
Multiple URL variants

Context and symptoms:

Click for likely cause and resolution

Cause: The Web source crawler discovers multiple variants of the same page, each with different URL casings.

Resolution: If the web server is case insensitive, access the Edit a source JSON configuration panel and set RespectUrlCasing to false.

⚠️ Don’t set RespectUrlCasing to false if the web server is case sensitive. If you do, pages with uppercase characters in their URL won’t be indexed.
Missing filtering

Symptom: The Content Browser (platform-ca | platform-eu | platform-au) shows web pages you don’t want to index.

Click for likely cause and resolution

Cause: Your current Crawling rules exclusions and inclusions don’t filter out the unwanted pages.

Resolution: Open your source and configure exclusions and inclusions to filter out the unwanted pages.

To be indexed, a page:

  • must not match any exclusion rule, AND

  • it must match at least one inclusion rule (e.g., by selecting the Include all pages that were not excluded option).
Content freshness issue

Symptom: Pages recently deleted from the website are still appearing in the Content Browser (platform-ca | platform-eu | platform-au).

Click for likely cause and resolution

Cause and Resolution: See Indexed content isn’t up to date.

Unexpected or missing content inside pages

Indexing by reference

Context and symptoms:

Indexing by reference
Click for likely cause and resolution

Cause: You may be indexing by reference. When indexing by reference, the body of the web page (used for the Quick view) isn’t retrieved and no excerpt (used for the item description) is generated.

Resolution:

Access the Edit a source JSON configuration panel. If HTML documents are currently indexed by Reference, change that value to Retrieve.

Indexing by retrieve
Web scraping issue

Context and symptoms:

Click for likely cause and resolution

Cause: A web scraping configuration may be removing the missing sections.

Resolution: Open your source and review your web scraping configurations:

  • Check that web scraping configurations appear in an order that makes sense. The first matching configuration is applied to the page, and the following are ignored.

  • In the Basic configuration settings of the applied web scraping configuration, try changing or adding a rule to exclude your page.

  • In the Elements to exclude tab of the applied web scraping configuration, try making your selector more restrictive to avoid removing sections from your page.
Missing dynamic content

Context and symptoms:

  • When accessing the quick view of an item, sections of the actual web page are missing.

  • Your web page contains dynamically rendered content (e.g., responses to JavaScript API calls).

Click for likely cause and resolution

Cause: The source may be crawling your page before all its dynamic content is rendered.

Resolution: Open your source. In the Advanced settings tab, make sure Execute JavaScript on pages is enabled. Increase the Time the crawler waits before considering a page as rendered value, if necessary.
Login page content instead of proper page content

Context and symptoms:

  • When accessing the quick view of an item, you notice that a login page content appears instead of the content of the page specified by the URI. This symptom will likely repeat itself over many items.

  • When trying to access the page to index manually in a browser, you’re redirected to that login page.

Click for likely cause and resolution

Cause: The page to index is protected and form authentication isn’t properly set up.

Resolution:

  1. Request the login page authentication credentials from the web server administrator.

  2. Open your source.

  3. Configure form authentication. Use the provided username and password and set the Login address to the login page URL.

  4. Set the Validation method to Redirection to URL and the Value to the login page URL.

  5. Rebuild your source.

  6. Validate that the item now contains the proper content.
Indexing pipeline extension

Context and symptoms:

Click for likely cause and resolution

Cause: An indexing pipeline extension (IPE) may be removing the missing sections.

Resolution: Review the logs for the items affected by the extensions. Make necessary adjustments to the extension script or conditions.

Unexpected item field values

Inexistent field

Context and symptoms:

Click for likely cause and resolution

Cause: The field doesn’t exist. You need to create the field and the field mapping.

Resolution:

  1. On the Fields (platform-ca | platform-eu | platform-au) page, at the top right, click Add field.

  2. Follow instructions in the Add or edit a field article to configure your field.

  3. On the Sources (platform-ca | platform-eu | platform-au) page, click your source, and then click More > View metadata.

  4. Choose the metadata you want to use to populate the field.

  5. On the Sources (platform-ca | platform-eu | platform-au) page, click your source, and then click More > Manage mappings in the Action bar.

  6. Follow instructions in the Add or edit a mapping rule section to configure your mapping.
Field mapping issue

Context and symptoms:

Click for likely cause and resolution

Cause: There may be a field mapping issue.

Resolution:

  1. On the Sources (platform-ca | platform-eu | platform-au) page, click your source, and then click More > View metadata.

  2. Make sure the metadata that should be used to populate your field appears. If the metadata is being used to populate a field, it will be shown as Indexed. If you see two entries under the same metadata name, take note of the indexed and not indexed metadata Origin values for the final step in this procedure.

  3. On the Sources (platform-ca | platform-eu | platform-au) page, click your source, and then click More > Manage mappings in the Action bar.

  4. Make sure the mapping rule for the field references the right metadata name.

  5. Add or edit the Origin value in the field mapping rule (e.g., %[description:crawler]).
Metadata extraction issue

Context and symptoms:

Click for likely cause and resolution

Cause: There may be a metadata extraction issue specifically for that item.

Resolution: Search for reasons why the metadata extraction process wouldn’t be working on your specific item. For example, if you’re using a web scraping configuration, open your source, and then validate the following:

  • Your item matches the Basic configuration rules you set for that web scraping configuration. Also remember that only the first matching web scraping configuration is applied to the page.

  • Your CSS or XPath selector works for that specific item.
Title field value selection

Symptom: The item title field value isn’t ideal.

Click for likely cause and resolution

Cause: Coveo has a title field selection process to ensure all indexed items have titles. This process may not return ideal titles in your use case.

Resolution: Coveo automatically extracts several pieces of metadata that you can use as item titles. See Item title selection mapping rule options to control the value selection process. Edit the title field mappings on your source.
Metadata origin selection

Context and symptoms:

  • The indexed item has a value for the given field, but that value isn’t the expected one.

  • On the Sources (platform-ca | platform-eu | platform-au) page, when you click the source and then click More > View metadata, you see two entries under the same metadata name.

Example:

metadata value conflict
Click for likely cause and resolution

Cause: There’s a metadata origin selection issue.

For example, you’ve configured a web scraping configuration to extract a description metadata. The Web source may also be automatically extracting description metadata from the page <meta> tags.

When values for the same metadata name are extracted in the crawling stage and in the processing (or converter) stage of the Coveo indexing pipeline, the latter value is used by default to populate the mapped field.

Example:

item field value

Resolution:

  • Use a unique metadata name and create a dedicated field for the custom metadata you’re extracting, OR

  • Access the Edit the mappings of a source panel. Specify the origin value in the field mapping rule (e.g., %[description:crawler]) to populate the field with the custom metadata you’re extracting.
Overwritten crawler metadata

Context and symptoms:

  • The indexed item has a value for the given field.

  • On the Sources (platform-ca | platform-eu | platform-au) page, when you click the source and then click More > View metadata, you see an entry under the metadata name you chose under the Crawler origin.

  • You specified the origin value in the field mapping rule (e.g., %[description:crawler]), but the field value isn’t the expected one.

Click for likely cause and resolution

Cause: There’s a metadata conflict.

You can have two configurations extracting values for the same metadata name at the crawling stage. When this happens, one value overwrites the other and you only see one Crawler origin entry for that metadata name on the View Metadata subpage.

Resolution: Change the metadata name in your configuration to make it unique and adjust your field mapping rule accordingly.

Indexing is slow

Source scope

Symptom: Indexing the source pages is taking a long time.

Click for likely cause and resolution

Cause: The Web source may be crawling and indexing a very high number of pages, and maybe even unwanted pages. This may be due to a number of reasons (e.g., high number of starting URLs, too broad crawling rule exclusions and inclusions).

Resolution:

  • See Extra or unwanted pages.

  • Consider breaking up the Web source into multiple sources. This helps performance-wise and it simplifies source configurations and troubleshooting, OR

  • Consider using one or multiple Sitemap sources instead.
Crawl delay

Symptom: Indexing the source pages is taking a long time.

Click for likely cause and resolution

Cause: The Time the crawler waits between requests to your server may be unnecessarily high.

Resolution: Provide proof of website ownership. Then, open your source and, in the Advanced settings tab, reduce the Time the crawler waits between requests to your server value.
ExpandBeforeFiltering setting

Symptom: Indexing the source pages is taking a long time.

Click for likely cause and resolution

Cause: The source may be configured with ExpandBeforeFiltering set to true.

Resolution: Consider editing the source JSON configuration and setting ExpandBeforeFiltering to false.

Indexed content isn’t up to date

Source rescan schedule

Symptom: Recent changes to website pages aren’t reflected in the Content Browser (platform-ca | platform-eu | platform-au).

Click for likely cause and resolution

Cause:

  • Your source rescan schedule may be disabled, OR

  • The time interval between consecutive rescans might be too long.

Resolution: Make sure the rescan schedule is enabled and that its recurrence settings are adequate.
Number of items limit reached

Context and symptoms:

Click for likely cause and resolution

Cause: Indexing is blocked because you have reached the 200% license item usage threshold.

Resolution:

  • If possible, delete unused sources to bring the item count below the 200% threshold. Then, see the July 20, 2023 Coveo Relevance Cloud update for suggestions on how to reduce your item count even more.

  • To reassess your needs and discuss your options, contact Coveo account executive.