---
title: Troubleshooting Sitemap source issues
slug: n1ab0354
canonical_url: https://docs.coveo.com/en/n1ab0354/
collection: index-content
source_format: adoc
---
# Troubleshooting Sitemap source issues
This article provides troubleshooting best practices and lists common issues when [indexing](https://docs.coveo.com/en/204/) content with the Sitemap source.

## Important: Troubleshooting fundamentals
Though the information provided in the [Common issues](#common-issues) section will often help you identify and resolve a problem, keep the following in mind:

* A given set of symptoms can be caused by different underlying issues.

* When you expand a content update activity in the **Activity** panel or [**Activity Browser**](https://platform.cloud.coveo.com/admin/#/orgid/activity/browser/) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/activity/browser/) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/activity/browser/) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/activity/browser/)) page, the error code and messages displayed may only be general indicators of the problem.

* Coveo only halts an indexing operation and displays an error when specific conditions are met.

Consequently, finding the root cause of an issue may require more granular information, which only update logs can deliver.






To download an update log

. On the [**Sources**](https://platform.cloud.coveo.com/admin/#/orgid/content/sources/) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/content/sources/) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/content/sources/) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/content/sources/)) page, click the desired resource, and then click **Activity** in the Action bar.

. In the **Activity** panel that opens, click the desired activity, and then click **Download logs** in the Action bar.
The downloaded file is named after the unique operation ID representing the selected activity.


To locate issue root causes in logs

. Open the log file in a text file viewer.

. Look for `WARN`,`ERROR`, and `FATAL` messages.

![Indexing activity logs with a warning message | Coveo](https://docs.coveo.com/en/assets/images/index-content/warn-message-in-logs.png)

> **Tip**
>
> Use a log file viewer that supports highlighting by log level to make these messages more noticeable.

. If necessary, review `NOTICE` and `INFO` messages.
They sometimes reveal a configuration that you overlooked and that may be causing the issue.

![Indexing activity logs with an info message | Coveo](https://docs.coveo.com/en/assets/images/index-content/info-message-in-logs.png)


## Common issues

Issues are divided into categories.
Click a category description below to reach the related section.

* [Missing items](#missing-items)

* [Extra or unwanted items](#extra-or-unwanted-items)

* [Unexpected or missing content inside items](#unexpected-or-missing-content-inside-items)

* [Unexpected item field values](#unexpected-item-field-values)

* [Indexing is slow](#indexing-is-slow)

* [Indexed content isn't up to date](#indexed-content-is-not-up-to-date)

### Missing items

#### User agent blocklisting

> **Context and symptoms**
>
> * No items or a limited number of items are indexed.
> 
> * Other symptoms may vary.
> For example:
> 
>  ** The [**Activity Browser**](https://platform.cloud.coveo.com/admin/#/orgid/activity/browser/) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/activity/browser/) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/activity/browser/) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/activity/browser/)) page displays a `SITEMAP_FORBIDDEN_ERROR` code.
> The error message states `Forbidden access.
> The connector doesn't have required permissions or it is blocked by a security feature.`
> 
>  **  The activity [download logs](#important-troubleshooting-fundamentals) may show a NOTICE message in the following format:
>  
> ```text
CDN/Caching Detection: Provider = '<PROVIDER_NAME>', Host = '<HOST_URL>', Detected Header: '<HEADER_NAME>' = '<HEADER_VALUE>', Reference header: <REFERENCE_HEADER_INFO>. Please consult https://docs.coveo.com/en/1831/index-content/ip-addresses-to-allowlist to allowlist the IP addresses by the CDN/Caching provider if needed.
```
>  
> This message indicates that the source has detected that pages at `<HOST_URL>` are served by the `<PROVIDER_NAME>` CDN/Caching provider, which may interfere with requests from the Coveo crawler.
> 
> .Likely cause and resolution
> <details><summary>Details</summary>
> 
> **Cause**
> 
> Your web server or CDN/Caching provider may be blocking requests from the Coveo crawler based on its user agent.
> 
> This can happen, for example, if you're using the `mod_rewrite` module on an Apache server or the `URL Rewrite` module on an IIS server.
> The source automatically detects the following CDN and caching providers, which can also interfere with requests from the Coveo crawler: Akamai, Amazon CloudFront, Cloudflare, Fastly, Incapsula (Imperva), and Varnish.
> 
> **Resolution**
> 
> . Chrome and other web browsers allow you to emulate web requests by [overriding the default user agent string](https://developer.chrome.com/docs/devtools/device-mode/override-user-agent).
> Use this feature to test whether your web server or CDN/Caching provider restricts, filters, or otherwise handles requests differently based on the Coveo crawler user agent string (that is, `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko) (compatible; Coveobot/2.0;+http://www.coveo.com/bot.html)`).
> 
> . If applicable, update your web server, CDN/Caching provider, or Sitemap source configuration to allow or properly handle requests.
> 
> .. (Recommended) Update your web server or CDN/Caching provider configuration so that requests from the Coveo crawler user agent are allowed and processed as expected.
> 
> .. On the **Advanced settings** subtab of your source configuration, set the [user agent string](https://docs.coveo.com/en/1967#user-agent-string) to a value that your web server accepts and processes without restriction.
> 
> . [Rebuild](https://docs.coveo.com/en/3390#refresh-rescan-or-rebuild-sources) your source.
> 
> </details>


#### IP blocklisting

> **Context and symptoms**
>
> * No items or a limited number of items are indexed.
> 
> * Other symptoms may vary.
> For example:
> 
>  **  The activity [download logs](#important-troubleshooting-fundamentals) may show a NOTICE message in the following format:
>  
> ```text
CDN/Caching Detection: Provider = '<PROVIDER_NAME>', Host = '<HOST_URL>', Detected Header: '<HEADER_NAME>' = '<HEADER_VALUE>', Reference header: <REFERENCE_HEADER_INFO>. Please consult https://docs.coveo.com/en/1831/index-content/ip-addresses-to-allowlist to allowlist the IP addresses by the CDN/Caching provider if needed.
```
>  
> This message indicates that the source has detected that pages at `<HOST_URL>` are served by the `<PROVIDER_NAME>` CDN/Caching provider, which commonly interferes with requests from the Coveo crawler.
> 
>  ** An HTTP request may simply time out with no error.
> 
> .Likely cause and resolution
> <details><summary>Details</summary>
> 
> **Cause**
> 
> Your infrastructure may be restricting, filtering, or otherwise preventing inbound requests from [Coveo Platform](https://docs.coveo.com/en/186/) IP addresses.
> The source automatically detects the following CDN and caching providers, which can interfere with requests from the Coveo crawler: Akamai, Amazon CloudFront, Cloudflare, Fastly, Incapsula (Imperva), and Varnish.
> 
> **Resolution**
> 
> Configure your infrastructure to allow and properly handle inbound requests from the [Coveo Platform](https://docs.coveo.com/en/186/).
> If the [download logs](#important-troubleshooting-fundamentals) are showing a CDN/Caching detection message, you may need to [allow requests from Coveo Platform IP addresses](https://docs.coveo.com/en/1831/) at the CDN/Caching provider level.
> 
> Alternatively, consider installing the [Coveo Crawling Module](https://docs.coveo.com/en/3260/) on your infrastructure to push [items](https://docs.coveo.com/en/210/) to the [Coveo Platform](https://docs.coveo.com/en/186/) instead.
> 
> </details>


#### Sitemap or sitemap index file can't be reached

> **Context and symptoms**
>
> * No items, or a limited number of items referenced in the sitemaps are indexed.
> 
> * The [**Activity Browser**](https://platform.cloud.coveo.com/admin/#/orgid/activity/browser/) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/activity/browser/) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/activity/browser/) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/activity/browser/)) page may display a `SITEMAP_UNABLE_TO_FETCH_CONTENT_ERROR` error code.
> 
> * The [**Activity Browser**](https://platform.cloud.coveo.com/admin/#/orgid/activity/browser/) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/activity/browser/) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/activity/browser/) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/activity/browser/)) page may show a low or sudden drop in the **Items per hour** value during indexing.
> 
> * The [**Activity Browser**](https://platform.cloud.coveo.com/admin/#/orgid/activity/browser/) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/activity/browser/) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/activity/browser/) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/activity/browser/)) page may display a `No such host is known` error code.
> 
> .Likely cause and resolution
> <details><summary>Details</summary>
> 
> **Cause**
> 
> The following are potential causes:
> 
> . The Coveo crawler can't access the sitemap or sitemap index file due to server throttling.
> 
> . A sitemap or sitemap index file URL is incorrect.
> 
> . Your network may be blocking inbound requests from the [Coveo Platform](https://docs.coveo.com/en/186/).
> 
> **Resolution**
> 
> Depending on the cause, apply the corresponding resolution below:
> 
> . On the [**Sources**](https://platform.cloud.coveo.com/admin/#/orgid/content/sources/) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/content/sources/) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/content/sources/) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/content/sources/)) page, open your source and try increasing the [**Time the crawler waits between requests to your server**](https://docs.coveo.com/en/1967#time-the-crawler-waits-between-requests-to-your-server) value (for example, `1000` milliseconds).
> Also check your [refresh and rescan schedules](https://docs.coveo.com/en/1933#schedule-a-source-update).
> Overlapping refresh and rescan schedules can also cause an overload on your server.
> 
> . On the [**Sources**](https://platform.cloud.coveo.com/admin/#/orgid/content/sources/) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/content/sources/) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/content/sources/) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/content/sources/)) page, open your source and validate its [Sitemap URLs](https://docs.coveo.com/en/1967#sitemap-urls).
> For example, make sure your Sitemap URLs use the correct protocol (HTTP or HTTPS).
> When you're done and you've saved your changes, [rebuild](https://docs.coveo.com/en/3390#refresh-rescan-or-rebuild-sources) your source.
> 
> . [Allow inbound requests](https://docs.coveo.com/en/1831/) from the [Coveo Platform](https://docs.coveo.com/en/186/) or consider installing the [Coveo Crawling Module](https://docs.coveo.com/en/3260/) on your infrastructure to push [items](https://docs.coveo.com/en/210/) to Coveo instead.
> 
> </details>


#### Sitemap URL exclusion

> **Context and symptoms**
>
> * The [**Content Browser**](https://platform.cloud.coveo.com/admin/#/orgid/content/browser/) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/content/browser/) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/content/browser/) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/content/browser/)) doesn't show all the items you wanted to index.
> 
> * All items listed in one of the source [**Sitemap URLs**](https://docs.coveo.com/en/1967#sitemap-urls) aren't indexed.
> 
> * The [**Activity Browser**](https://platform.cloud.coveo.com/admin/#/orgid/activity/browser/) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/activity/browser/) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/activity/browser/) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/activity/browser/)) page may display a `NO_DOCUMENT_INDEXED` error code.
> 
> .Likely cause and resolution
> <details><summary>Details</summary>
> 
> **Cause**
> 
> The Sitemap source [exclusion and inclusion](https://docs.coveo.com/en/1967#exclusions-and-inclusions) rules may be filtering out that sitemap URL.
> Consequently, items listed in that sitemap file aren't indexed.
> 
> **Resolution**
> 
> On the [**Sources**](https://platform.cloud.coveo.com/admin/#/orgid/content/sources/) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/content/sources/) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/content/sources/) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/content/sources/)) page, open your source and review its [exclusion and inclusion](https://docs.coveo.com/en/1967#exclusions-and-inclusions) rules.
> Ensure your sitemap file URL:
> 
> * Doesn't match any exclusion rule
> 
> * Matches at least one inclusion rule.
> 
> </details>


#### NO_DOCUMENT_INDEXED errors

> **Context and symptoms**
>
> * No items referenced in the sitemap file are indexed.
> 
> * The [**Activity Browser**](https://platform.cloud.coveo.com/admin/#/orgid/activity/browser/) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/activity/browser/) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/activity/browser/) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/activity/browser/)) page displays a `NO_DOCUMENT_INDEXED` error code.
> 
> .Likely cause and resolution
> <details><summary>Details</summary>
> 
> **Cause**
> 
> The `NO_DOCUMENT_INDEXED` is a generic error code used when an indexing operation fails.
> It tells you that the source didn't index any items but tells you little about the reason why.
> The indexing operation may have failed for a variety of reasons, some of which have been covered previously.
> 
> For example, it's possible no items were indexed because:
> 
> . The sitemap file was actually empty at indexing time.
> 
> . The sitemap is invalid.
> 
> **Resolution**
> 
> More information is needed to diagnose the issue.
> Review the source activity logs as follows:
> 
> . On the [**Sources**](https://platform.cloud.coveo.com/admin/#/orgid/content/sources/) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/content/sources/) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/content/sources/) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/content/sources/)) page, click your source, and then click **Activity** in the Action bar.
> 
> . In the **Activity** panel that opens, click the desired activity (having a `Failed` result), and then click **Download Logs** in the Action bar.
> 
> . Open the downloaded log file in your preferred text editor.
> 
> . Starting from the top of the logs, and working your way down, look for the first WARN or FATAL error message.
> For example, the WARN message in the logs below indicates that Coveo wasn't able to parse the sitemap file because it's invalid (that is, the `<loc>` element is missing).
>  
> ![Invalid sitemap file | Coveo](:https://docs.coveo.com/en/assets/images/index-content/troubleshooting-invalid-sitemap-logs.png)
>  
> {empty}
> 
> . Once you've identified the issue, take the necessary steps to resolve it, and then [rebuild](https://docs.coveo.com/en/3390#refresh-rescan-or-rebuild-sources) your source.
> 
> </details>


#### Missing or invalid basic authentication configuration

> **Context and symptoms**
>
> * A page listed in the sitemap file isn't indexed.
> 
> * The [**Activity Browser**](https://platform.cloud.coveo.com/admin/#/orgid/activity/browser/) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/activity/browser/) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/activity/browser/) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/activity/browser/)) downloadable logs may show the following message: `Authentication failed. The provided credentials may be invalid or expired.`
> 
> * When trying to access that page in a browser, you're prompted for credentials in a pop-up window.
> 
> .Likely cause and resolution
> <details><summary>Details</summary>
> 
> **Cause**
> 
> Accessing the page content requires basic authentication.
> 
> **Resolution**
> 
> * Request authentication credentials from the web server administrator.
> Then, on the [**Sources**](https://platform.cloud.coveo.com/admin/#/orgid/content/sources/) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/content/sources/) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/content/sources/) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/content/sources/)) page, open your source and set up [basic authentication](https://docs.coveo.com/en/1967#basic-authentication).
> 
> * If you're using a password manager (for example, LastPass), it may replace the previously recorded username and password with different ones as you edit the source.
> Check your password manager options and ensure that it respects the `autocomplete="off"` attribute.
> 
> </details>


#### Missing or invalid form authentication configuration

> **Context and symptoms**
>
> * A page listed in the sitemap file isn't indexed.
> 
> * The [**Activity Browser**](https://platform.cloud.coveo.com/admin/#/orgid/activity/browser/) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/activity/browser/) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/activity/browser/) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/activity/browser/)) downloadable logs may show the following exception: `The form authentication request for "<URL>" was submitted to the form but it failed to authenticate.`
> 
> * When trying to access that page in a browser, a login page is displayed instead.
> 
> .Likely cause and resolution
> <details><summary>Details</summary>
> 
> **Cause**
> 
> Accessing the page content requires form authentication.
> 
> **Resolution**
> 
> * Request authentication credentials from the web server administrator.
> Then, on the [**Sources**](https://platform.cloud.coveo.com/admin/#/orgid/content/sources/) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/content/sources/) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/content/sources/) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/content/sources/)) page, open your source and set up [form authentication](https://docs.coveo.com/en/1967#form-authentication).
> 
> * If you're using a password manager (for example, LastPass), it may replace the previously recorded username and password with different ones as you edit the source.
> Check your password manager options and ensure that it respects the `autocomplete="off"` attribute.
> 
> </details>


#### Authentication validation method issue

> **Context and symptoms**
>
> * A page listed in the sitemap file isn't indexed.
> 
> * Accessing the page content requires form authentication.
> 
> * Form authentication _is_ configured on the source.
> 
> * When trying to access that page in a browser, the form authentication **Login page address** page is displayed.
> Typing in the credentials and submitting the login page brings up the page to be indexed.
> 
> .Likely cause and resolution
> <details><summary>Details</summary>
> 
> **Cause**
> 
> The authentication validation method may not be configured properly.
> 
> **Resolution**
> 
> On the [**Sources**](https://platform.cloud.coveo.com/admin/#/orgid/content/sources/) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/content/sources/) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/content/sources/) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/content/sources/)) page, open your source and ensure its [**Validation method**](https://docs.coveo.com/en/1967#validation-method) and the associated **Value** are adequate.
> 
> </details>


#### Redirection to login page issue

> **Context and symptoms**
>
> * A page listed in the sitemap file isn't indexed.
> 
> * Accessing the page content requires form authentication and your source configuration **Validation method** is `Redirection to URL`.
> 
> * When trying to access that page manually in a browser, the form authentication **Login page address** page _isn't_ displayed.
> 
> .Likely cause and resolution
> <details><summary>Details</summary>
> 
> **Cause**
> 
> The `Redirection to URL` validation method doesn't work in your use case.
> Consequently, the Sitemap source crawler doesn't know it must authenticate before accessing the page to index.
> 
> **Resolution**
> 
> * On the [**Sources**](https://platform.cloud.coveo.com/admin/#/orgid/content/sources/) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/content/sources/) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/content/sources/) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/content/sources/)) page, open your source and select another validation method.
> Select a method based on the way the web server responds when you manually try to access the page to index (when unauthenticated).
> 
> * If no reliable validation method can be found, try enabling the form authentication **Force authentication** option.
> 
> </details>


#### Content freshness issue

> **Context and symptoms**
>
> Items recently added to the site are still not appearing in the [**Content Browser**](https://platform.cloud.coveo.com/admin/#/orgid/content/browser/) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/content/browser/) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/content/browser/) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/content/browser/)).
> 
> .Likely cause and resolution
> <details><summary>Details</summary>
> 
> **Cause and Resolution**
> 
> See [Indexed content is not up to date](#indexed-content-is-not-up-to-date).
> 
> </details>


### Extra or unwanted items

#### Missing filtering

> **Context and symptoms**
>
> All URLs listed in the sitemap file are indexed.
> 
> .Likely cause and resolution
> <details><summary>Details</summary>
> 
> **Cause**
> 
> By default, the Sitemap source contains no [exclusions](https://docs.coveo.com/en/1967#exclusions-and-inclusions) and the [inclusions](https://docs.coveo.com/en/1967#exclusions-and-inclusions) are set to **Include all non-excluded pages**.
> In other words, you're not filtering out any URLs listed in the sitemap file.
> 
> **Resolution**
> 
> * Remove any unwanted URLs from your sitemap file, OR
> 
> * On the [**Sources**](https://platform.cloud.coveo.com/admin/#/orgid/content/sources/) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/content/sources/) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/content/sources/) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/content/sources/)) page, open your source and define [inclusion and exclusion](https://docs.coveo.com/en/1967#exclusions-and-inclusions) rules to filter out unwanted URLs.
>  
> To be indexed, a page:
> 
> ** must not match any exclusion rule, AND
> 
> ** it must match at least one inclusion rule (for example, by selecting the **Include all non-excluded pages** option).
> 
> ⚠️ Make sure you don't exclude your **Sitemap URLs**.
> 
> </details>


#### Duplicate items due to redirects

> **Context and symptoms**
>
> * When reviewing the [**Content Browser**](https://platform.cloud.coveo.com/admin/#/orgid/content/browser/) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/content/browser/) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/content/browser/) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/content/browser/)), you notice duplicate items.
> Only the `Clickable URI` values of these indexed items differ.
> 
> * Each `Clickable URI` reflects a URL listed in your sitemap file.
> 
> .Likely cause and resolution
> <details><summary>Details</summary>
> 
> **Cause**
> 
> Your sitemap file contains several URLs that redirect to the same page.
> For each URL listed in your sitemap file, the Sitemap source crawler creates an index item with the `Clickable URI` reflecting the URL that the crawler tried reaching (that is, the URL listed in the sitemap file).
> 
> **Resolution**
> 
> There are two ways to address this issue:
> 
> * (Recommended) Remove all URLs that are redirected from your sitemap file, only keeping the final destination URLs.
> 
> * Add exclusion rules to your source configuration to filter out the URLs that are redirected.
> 
> .. On the [**Sources**](https://platform.cloud.coveo.com/admin/#/orgid/content/sources/) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/content/sources/) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/content/sources/) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/content/sources/)) page, open your source.
> 
> .. In the [Exclusions and inclusions](https://docs.coveo.com/en/1967#exclusions-and-inclusions) section, add an `is` exclusion rule for each URL that's redirected.
>  
> ![Excluding URLs one by one | Coveo](:https://docs.coveo.com/en/assets/images/index-content/exclude-urls-one-by-one.png)
> 
> After performing the changes, the next scheduled [rescan](https://docs.coveo.com/en/1933#schedule-a-source-update) will update your source automatically.
> Alternatively, you can [rebuild](https://docs.coveo.com/en/3390#refresh-rescan-or-rebuild-sources) your source to apply the changes immediately.
> 
> </details>


#### Content freshness issue

> **Context and symptoms**
>
> Items recently deleted from the site are still appearing in the [**Content Browser**](https://platform.cloud.coveo.com/admin/#/orgid/content/browser/) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/content/browser/) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/content/browser/) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/content/browser/)).
> 
> .Likely cause and resolution
> <details><summary>Details</summary>
> 
> **Cause and Resolution**
> 
> See [Indexed content is not up to date](#indexed-content-is-not-up-to-date).
> 
> </details>


#### Ignored robots.txt directives or noindex meta tags

> **Context and symptoms**
>
> Pages that contain a `<meta name="robots" content="noindex" />` tag or that are disallowed in the `robots.txt` file are still indexed.
> 
> .Likely cause and resolution
> <details><summary>Details</summary>
> 
> **Cause**
> 
> The Sitemap source ignores `robots.txt` file directives and `noindex` meta tags.
> 
> **Resolution**
> 
> Filter out the pages using a supported method, for example:
> 
> * [Target a sitemap file](https://docs.coveo.com/en/1967#sitemap-urls) that only lists the pages you want to index.
> 
> * Use [inclusion and exclusion rules](https://docs.coveo.com/en/1967#exclusions-and-inclusions) in your source configuration.
> 
> * If `<meta name="robots" content="noindex" />` tags are being used in your pages, create a [post-conversion extension](https://docs.coveo.com/en/1556/) with a script similar to the following to reject those items:
>  
> ```python
def get_safe_meta_data(meta_data_name):
    safe_meta = ''
    meta_data_value = document.get_meta_data_value(meta_data_name)
    if meta_data_value:
        safe_meta = meta_data_value[-1]
    return safe_meta

robots = get_safe_meta_data('robots').lower()
directives = [directive.strip() for directive in robots.split(',') if directive.strip()]

if 'noindex' in directives or 'none' in directives:
    log('REJECT: robots meta noindex/none')
    document.reject()
```
>  
> [IMPORTANT]
> 
> </details>
> * You must use a post-conversion extension because the script requires the `robots` metadata values, which the [converter](https://docs.coveo.com/en/1893#processing) sets during processing.
> 
> * Using an extension significantly impacts indexing performance.
#### =====


### Unexpected or missing content inside items

#### Indexing by reference

> **Context and symptoms**
>
> * When viewing source items in the [**Content Browser**](https://platform.cloud.coveo.com/admin/#/orgid/content/browser/) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/content/browser/) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/content/browser/) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/content/browser/)), item **Description** areas are empty.
> 
> * If you then click a specific item, and then click **Properties**, the **Quick view** tab isn't displayed.
> 
> ![Indexing by reference results details panel | Coveo](:https://docs.coveo.com/en/assets/images/index-content/indexing-by-reference-results.png)
> 
> .Likely cause and resolution
> <details><summary>Details</summary>
> 
> **Cause**
> 
> You may be indexing by reference.
> When indexing by reference, the body of the web page (used for the Quick view) isn't retrieved and no excerpt (used for the item description) is generated.
> 
> **Resolution**
> 
> On the [**Sources**](https://platform.cloud.coveo.com/admin/#/orgid/content/sources/) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/content/sources/) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/content/sources/) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/content/sources/)) page, click the source, and then click **Edit configuration with JSON** in the **More** menu.
> If HTML documents are currently indexed by `Reference`, change that value to `Retrieve`.
> 
> ![JSON configuration indexing HTML by retrieve example | Coveo](:https://docs.coveo.com/en/assets/images/index-content/indexing-html-by-retrieve.png)
> 
> </details>


#### Broken images in the Quick view

> **Context and symptoms**
>
> When [accessing the Quick view of an item](https://docs.coveo.com/en/1712#access-the-quick-view-tab), images are broken.
> 
> ![Broken image in the Quick view](:https://docs.coveo.com/en/assets/images/index-content/quick-view-images-broken.png)
> 
> .Likely cause and resolution
> <details><summary>Details</summary>
> 
> **Cause**
> 
> The connector retrieves web page HTML as is and doesn't retrieve the images referenced in the HTML.
> The Content Browser [Quick view](https://docs.coveo.com/en/1712#quick-view-tab) displays this HTML without any alteration.
> This means it doesn't replace relative paths, such as `<img src="/sites/.../myimage.jpg">`, with the corresponding absolute paths, such as `<img src="https://.../myimage.jpg">`.
> As a result, when web pages contain images that are referenced using relative paths, the images can't be displayed in the Content Browser Quick view.
> 
> Images that require authentication to be viewed also appear broken when browsing the web page item Quick view in the Content Browser.
> 
> **Resolution**
> 
> None.
> This is a known limitation of the Content Browser Quick view.
> 
> The Quick view is intended to provide a preview of the item content, not a full rendering of the web page.
> To view the full web page, users can open the original document by clicking the item `clickable URI` link in the search results.
> 
> </details>


#### YouTube player not available in the Quickview component

> **Context and symptoms**
>
> In the Quickview component of a Coveo JavaScript Search Framework search result, the YouTube player isn't available.
> You notice the following symptoms:
> 
> * The YouTube video [iframe](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/iframe) shows the following error message:
>  
> ```
Try watching this video on www.youtube.com, or enable JavaScript if it is disabled in your browser.
```
> 
> * In the browser console, the following message appears:
>  
> ```
Blocked script execution in '<INSERT SEARCH PAGE URL>' because the document's frame is sandboxed and the 'allow-scripts' permission is not set.
```
> 
> .Likely cause and resolution
> <details><summary>Details</summary>
> 
> **Cause**
> 
> For security reasons, the only way to view a YouTube video in the YouTube player within a Coveo JavaScript Search Framework result template is by:
> 
> * Indexing items with the YouTube source.
> 
> * Using the [CoveoYouTubeThumbnail component](https://coveo.github.io/search-ui/components/youtubethumbnail.html).
> (Note: The [**Content Browser**](https://platform.cloud.coveo.com/admin/#/orgid/content/browser/) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/content/browser/) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/content/browser/) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/content/browser/)) Quick view doesn't meet this requirement.)
> 
> **Resolution**
> 
> . Index YouTube videos with the [YouTube source](https://docs.coveo.com/en/1637/).
> 
> . In your Coveo JavaScript Search Framework search interface, use the [CoveoYouTubeThumbnail component](https://coveo.github.io/search-ui/components/youtubethumbnail.html) to show a relevant image of the result video content.
> Clicking the thumbnail starts the video.
> 
> The following is a sample implementation:
> 
> ```html
<div class="CoveoResultList" data-layout="list" data-wait-animation="fade" data-auto-select-fields-to-include="true">
  <script id="YouTubeVideo" class="result-template" type="text/html" data-layout="list" data-field-filetype="YouTubeVideo">
    <div class="coveo-result-frame">
      <div class="coveo-result-row">
        <div class="coveo-result-cell" style="width:220px; padding-top:7px">
          <span class="CoveoYouTubeThumbnail"></span>
        </div>
        <div class="coveo-result-cell">
          <div class="coveo-result-row">
            <div class="coveo-result-cell" style="font-size:16px" role="heading" aria-level="2">
              <a class="CoveoResultLink"></a>
            </div>
            <div class="coveo-result-cell" style="text-align:right; width:120px;font-size:12px">
              <span class="CoveoFieldValue" data-field="@date" data-helper="dateTime"></span>
            </div>
          </div>
          <div class="coveo-result-row" style="margin-top:10px;">
            <div class="coveo-result-cell">
              <span class="CoveoExcerpt"></span>
            </div>
          </div>
          <div class="coveo-result-row" style="margin-top:10px;">
            <div class="coveo-result-cell">
              <span class="CoveoFieldValue" data-field="@author" data-text-caption="Author" style="margin-right:30px;"></span>
              <span class="CoveoFieldValue" data-field="@ytvideoduration" data-helper="timeSpan" data-helper-options-is-milliseconds="false" data-text-caption="Length" style="margin-right:30px;"></span>
              <span class="CoveoFieldValue" data-field="@ytviewcount" data-helper="number" data-helper-options-format="n0" data-text-caption="Views" style="margin-right:30px;"></span>
              <span class="CoveoFieldValue" data-field="@language" data-text-caption="Language" style="margin-right:30px;"></span>
            </div>
          </div>
          <div class="coveo-result-row">
            <div class="coveo-result-cell">
              <div class="CoveoMissingTerms"></div>
            </div>
          </div>
        </div>
      </div>
    </div>
  </script>
  <script id="YouTubePlaylist" class="result-template" type="text/html" data-layout="list" data-field-filetype="YouTubePlaylist">
    <div class="coveo-result-frame">
      <div class="coveo-result-cell" style="vertical-align:top;text-align:center;width:32px;">
        <span class="CoveoIcon" data-small="true" data-with-label="false"></span>
      </div>
      <div class="coveo-result-cell" style="vertical-align: top;padding-left: 16px;">
        <div class="coveo-result-row" style="margin-top:0;">
          <div class="coveo-result-cell coveo-no-wrap" style="vertical-align:top;font-size:16px;" role="heading" aria-level="2">
            <a class="CoveoResultLink"></a>
          </div>
          <div class="coveo-result-cell" style="width:120px;text-align:right;font-size:12px">
            <div class="coveo-result-row">
              <span class="CoveoFieldValue" data-field="@date" data-helper="date"></span>
            </div>
          </div>
        </div>
        <div class="coveo-result-row" style="margin-top:10px;">
          <div class="coveo-result-cell">
            <span class="CoveoFieldValue" data-field="@filetype" data-text-caption="Type" style="margin-right:30px;"></span>
            <span class="CoveoFieldValue" data-field="@author" data-text-caption="Author" style="margin-right:30px;"></span>
            <span class="CoveoFieldValue" data-field="@ytitemcount" data-text-caption="NumberOfVideos" style="margin-right:30px;"></span>
          </div>
        </div>
        <div class="coveo-result-row" style="margin-top:12px;">
          <div class="coveo-result-cell" style="padding-top:5px; padding-bottom:5px; font-size:13px;">
            <span class="CoveoResultFolding" data-result-template-id="YouTubeVideo"></span>
          </div>
        </div>
        <div class="coveo-result-row">
          <div class="coveo-result-cell">
            <div class="CoveoMissingTerms"></div>
          </div>
        </div>
      </div>
    </div>
  </script>
</div>
```
> 
> </details>


#### Copy protection on PDF

> **Context and symptoms**
>
> When viewing a PDF item in the [**Content Browser**](https://platform.cloud.coveo.com/admin/#/orgid/content/browser/) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/content/browser/) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/content/browser/) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/content/browser/)), you notice the following:
> 
> * There's no description.
> 
> * The Quick view shows the following:
>  
> ![Copy protected PDF | Coveo](:https://docs.coveo.com/en/assets/images/index-content/copy-protected-pdf.png)
> 
> .Likely cause and resolution
> <details><summary>Details</summary>
> 
> **Cause**
> 
> The PDF is password-protected.
> 
> ![Document security on document in file system | Coveo](:https://docs.coveo.com/en/assets/images/index-content/document-security-on-pwd-protected-pdf.png)
> 
> Therefore, the source can't retrieve the document binary content it needs to generate the description and the Quick view.
> 
> **Resolution**
> 
> . If acceptable, remove the password protection on the PDF in the file system.
> 
> . [Rebuild](https://docs.coveo.com/en/3390#refresh-rescan-or-rebuild-sources) your source.
> 
> </details>


#### Web scraping issue

> **Context and symptoms**
>
> * When [accessing the Quick view of an item](https://docs.coveo.com/en/1712#access-the-quick-view-tab), sections of the actual web page are missing.
> 
> * You have one or multiple web scraping configurations configured on your source.
> 
> .Likely cause and resolution
> <details><summary>Details</summary>
> 
> **Cause**
> 
> A web scraping configuration may be removing the missing sections.
> 
> **Resolution**
> 
> On the [**Sources**](https://platform.cloud.coveo.com/admin/#/orgid/content/sources/) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/content/sources/) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/content/sources/) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/content/sources/)) page, open your source and review its [web scraping configurations](https://docs.coveo.com/en/1967#web-scraping-subtab).
> Focus on your [**Pages to target**](https://docs.coveo.com/en/mahe0350#configuration-info) configurations and whether you can set more restrictive [**Elements to exclude**](https://docs.coveo.com/en/mahe0350#elements-to-exclude) selectors.
> 
> </details>


#### Missing dynamic content

> **Context and symptoms**
>
> * When [accessing the Quick view of an item](https://docs.coveo.com/en/1712#access-the-quick-view-tab), sections of the actual web page are missing.
> 
> * Your web page contains dynamically rendered content (for example, responses to JavaScript API calls).
> 
> .Likely cause and resolution
> <details><summary>Details</summary>
> 
> **Cause**
> 
> The source may be crawling your page before all its dynamic content is rendered.
> 
> **Resolution**
> 
> On the [**Sources**](https://platform.cloud.coveo.com/admin/#/orgid/content/sources/) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/content/sources/) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/content/sources/) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/content/sources/)) page, open your source.
> In the [**Advanced settings**](https://docs.coveo.com/en/1967#advanced-settings-subtab) subtab, make sure **Execute JavaScript on pages** is enabled.
> Increase the **Add time for the crawler to wait before considering a page as fully rendered** value, if need be.
> 
> </details>


#### HTML pages indexed as txt items

> **Context and symptoms**
>
> When accessing the [**Content Browser**](https://platform.cloud.coveo.com/admin/#/orgid/content/browser/) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/content/browser/) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/content/browser/) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/content/browser/)), pages are appearing under the `txt` file type, instead of `html`.
> 
> .Likely cause and resolution
> <details><summary>Details</summary>
> 
> **Cause**
> 
> The web page, at the moment it's crawled, isn't valid HTML.
> If the page includes dynamic content, it might not be fully rendered when the crawler processes it.
> 
> **Resolution**
> 
> . If the page includes dynamic content, make sure it's fully rendered when the crawler processes it.
> 
> .. On the [**Sources**](https://platform.cloud.coveo.com/admin/#/orgid/content/sources/) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/content/sources/) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/content/sources/) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/content/sources/)) page, open your source configuration.
> 
> .. In the [**Advanced settings**](https://docs.coveo.com/en/1967#advanced-settings-subtab) subtab, make sure [**Execute JavaScript on pages**](https://docs.coveo.com/en/1967#execute-javascript-on-pages) is enabled.
> 
> .. Set or increase the **Time the crawler waits before considering a page as fully rendered** value (for example, 300 milliseconds).
> 
> .. Save and rebuild your source.
> 
> . Fix the HTML of web pages still indexed as `txt`.
> 
> .. Use an [HTML markup validator](https://validator.w3.org/) to identify the most significant issues with the page.
> 
> .. Fix these markup issues.
> 
> .. [Rebuild](https://docs.coveo.com/en/3390#refresh-rescan-or-rebuild-sources) your source.
> 
> </details>


#### Login page content instead of proper page content

> **Context and symptoms**
>
> * When [accessing the Quick view of an item](https://docs.coveo.com/en/1712#access-the-quick-view-tab), you notice that a login page content appears instead of the content of the page specified by the URI.
> This symptom will likely repeat itself over many items.
> 
> * When trying to access the page to index manually in a browser, you're redirected to that login page.
> 
> .Likely cause and resolution
> <details><summary>Details</summary>
> 
> **Cause**
> 
> The page to index is protected and form authentication isn't properly set up.
> 
> **Resolution**
> 
> . Request the login page authentication credentials from the web server administrator.
> 
> . On the [**Sources**](https://platform.cloud.coveo.com/admin/#/orgid/content/sources/) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/content/sources/) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/content/sources/) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/content/sources/)) page, open your source and set up [form authentication](https://docs.coveo.com/en/1967#form-authentication) with the **Login page address** and the provided username and password.
> 
> . Set the **Validation method** to `Redirection to URL` and the **Value** to the **Login page address** value.
> 
> . [Rebuild](https://docs.coveo.com/en/3390#refresh-rescan-or-rebuild-sources) your source.
> 
> . Validate that the item now contains the [proper content](https://docs.coveo.com/en/1712#access-the-quick-view-tab).
> 
> </details>


#### Indexing pipeline extension

> **Context and symptoms**
>
> * When [accessing the Quick view of an item](https://docs.coveo.com/en/1712#access-the-quick-view-tab), sections of the actual web page are missing.
> 
> * The [**Extensions**](https://platform.cloud.coveo.com/admin/#/orgid/content/extensions/) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/content/extensions/) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/content/extensions/) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/content/extensions/)) page shows you have one or several [indexing pipeline extensions (IPEs)](https://docs.coveo.com/en/206/) in your [Coveo organization](https://docs.coveo.com/en/185/).
> 
> .Likely cause and resolution
> <details><summary>Details</summary>
> 
> **Cause**
> 
> An [indexing pipeline extension (IPE)](https://docs.coveo.com/en/206/) may be removing the missing sections.
> 
> **Resolution**
> 
> [Review the logs](https://docs.coveo.com/en/1645#inspect-impacted-item-logs) for the items affected by the extensions.
> Make necessary adjustments to the extension script or [conditions](https://docs.coveo.com/en/64/).
> 
> </details>


### Unexpected item field values

#### Inexistent field

> **Context and symptoms**
>
> * When inspecting an item in the [**Content Browser**](https://platform.cloud.coveo.com/admin/#/orgid/content/browser/) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/content/browser/) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/content/browser/) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/content/browser/)), the expected field name doesn't appear.
> 
> * On the [**Fields**](https://platform.cloud.coveo.com/admin/#/orgid/content/fields/) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/content/fields/) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/content/fields/) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/content/fields/)) page, the expected field doesn't appear.
> 
> .Likely cause and resolution
> <details><summary>Details</summary>
> 
> **Cause**
> 
> The field doesn't exist.
> You need to create the field and the field [mapping](https://docs.coveo.com/en/217/).
> 
> **Resolution**
> 
> . On the [**Fields**](https://platform.cloud.coveo.com/admin/#/orgid/content/fields/) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/content/fields/) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/content/fields/) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/content/fields/)) page, at the upper right, click **Add field**.
> 
> . Follow instructions in the [Add a field](https://docs.coveo.com/en/1833#add-a-field) article to configure your field.
> 
> . On the [**Sources**](https://platform.cloud.coveo.com/admin/#/orgid/content/sources/) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/content/sources/) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/content/sources/) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/content/sources/)) page, click your source, and then click **More** > **View and map metadata**.
> 
> . Choose the [metadata](https://docs.coveo.com/en/218/) you want to use to populate the field.
> 
> . On the [**Sources**](https://platform.cloud.coveo.com/admin/#/orgid/content/sources/) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/content/sources/) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/content/sources/) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/content/sources/)) page, click your source, and then click **Mappings** in the Action bar.
> 
> . Follow instructions in the [Manage mappings](https://docs.coveo.com/en/1640#manage-mappings) section to configure your mapping.
> 
> </details>


#### Field mapping issue

> **Context and symptoms**
>
> * When inspecting items that should have values for the field in the [**Content Browser**](https://platform.cloud.coveo.com/admin/#/orgid/content/browser/) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/content/browser/) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/content/browser/) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/content/browser/)), the expected field name doesn't appear in any item.
> 
> * On the [**Fields**](https://platform.cloud.coveo.com/admin/#/orgid/content/fields/) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/content/fields/) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/content/fields/) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/content/fields/)) page, the expected field appears.
> 
> .Likely cause and resolution
> <details><summary>Details</summary>
> 
> **Cause**
> 
> There may be a field [mapping](https://docs.coveo.com/en/217/) issue.
> 
> **Resolution**
> 
> . On the [**Sources**](https://platform.cloud.coveo.com/admin/#/orgid/content/sources/) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/content/sources/) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/content/sources/) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/content/sources/)) page, click your source, and then click **More** > **View and map metadata**.
> 
> . Make sure the metadata that should be used to populate your field appears.
> If the metadata _is_ being used to populate a field, it will be shown as `Indexed`.
> If you see two entries under the same metadata name, take note of the indexed and not indexed metadata `Origin` values for the final step in this procedure.
> 
> . On the [**Sources**](https://platform.cloud.coveo.com/admin/#/orgid/content/sources/) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/content/sources/) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/content/sources/) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/content/sources/)) page, click your source, and then click **Mappings** in the Action bar.
> 
> . Make sure the mapping rule for the field references the right metadata name.
> 
> . Add or edit the `Origin` value in the [field mapping rule](https://docs.coveo.com/en/1839/) (for example, `%[description:crawler]`).
> 
> </details>


#### Metadata extraction issue

> **Context and symptoms**
>
> * When inspecting an item in the [**Content Browser**](https://platform.cloud.coveo.com/admin/#/orgid/content/browser/) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/content/browser/) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/content/browser/) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/content/browser/)), the expected field name doesn't appear.
> 
> * When inspecting other items from the source in the [**Content Browser**](https://platform.cloud.coveo.com/admin/#/orgid/content/browser/) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/content/browser/) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/content/browser/) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/content/browser/)), the field appears with values in some or all of them.
> 
> .Likely cause and resolution
> <details><summary>Details</summary>
> 
> **Cause**
> 
> There may be a metadata extraction issue specifically for that item.
> 
> **Resolution**
> 
> Search for reasons why the metadata extraction process wouldn't be working on your specific item.
> For example, if you're using a web scraping configuration, go to the [**Sources**](https://platform.cloud.coveo.com/admin/#/orgid/content/sources/) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/content/sources/) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/content/sources/) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/content/sources/)) page and validate the following in your source configuration:
> 
> * Your item matches the [**Configuration info**](https://docs.coveo.com/en/mahe0350#configuration-info) rules you set for that web scraping configuration.
> Also remember that only the first matching web scraping configuration is applied to the page in [single-match mode](https://docs.coveo.com/en/mc1f3573#single-match-vs-multi-match).
> 
> * Your [CSS or XPath selector](https://docs.coveo.com/en/mahe0350#metadata-to-extract) works for that specific item.
> 
> </details>


#### Title field value selection

> **Context and symptoms**
>
> The item `title` field value isn't ideal.
> 
> .Likely cause and resolution
> <details><summary>Details</summary>
> 
> **Cause**
> 
> Coveo has a `title` field selection process to ensure all indexed items have titles.
> This process may not return ideal titles in your use case.
> 
> **Resolution**
> 
> Coveo automatically extracts several pieces of metadata that you can use as item titles.
> See [Item title selection](https://docs.coveo.com/en/1839#item-title-selection) mapping rule options to control the value selection process.
> [Edit the `title` field mappings](https://docs.coveo.com/en/1640#manage-mappings) on your source.
> 
> </details>


#### Metadata origin selection

> **Context and symptoms**
>
> * The indexed item has a value for the given field, but that value isn't the expected one.
> 
> * On the [**Sources**](https://platform.cloud.coveo.com/admin/#/orgid/content/sources/) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/content/sources/) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/content/sources/) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/content/sources/)) page, when you click the source and then click **More** > **View and map metadata**, you see two entries under the same metadata name.
> 
> Example:
> 
> ![metadata value conflict](:https://docs.coveo.com/en/assets/images/index-content/view-metadata-conflicts.png)
> 
> .Likely cause and resolution
> <details><summary>Details</summary>
> 
> **Cause**
> 
> There's a metadata origin selection issue.
> 
> For example, you've configured a web scraping configuration to extract a `description` metadata.
> The Sitemap source may also be [automatically extracting](https://docs.coveo.com/en/1967#source-key-characteristics) `description` metadata from the page `<meta>` tags.
> 
> When values for the same metadata name are extracted in the [crawling stage](https://docs.coveo.com/en/2684#crawling) and in the [processing (or _converter_) stage](https://docs.coveo.com/en/2684#processing) of the [Coveo indexing pipeline](https://docs.coveo.com/en/184/), the latter value is used by default to populate the mapped field.
> 
> Example:
> 
> ![item field value](:https://docs.coveo.com/en/assets/images/index-content/metadata-conflict-field-value.png)
> 
> **Resolution**
> 
> * Use a unique metadata name and create a dedicated field for the custom metadata you're extracting, OR
> 
> * Specify the `origin` value in the [field mapping rule](https://docs.coveo.com/en/1839/) (for example, `%[description:crawler]`) to populate the field with the custom metadata you're extracting.
> 
> </details>


#### Overwritten crawler metadata

> **Context and symptoms**
>
> * The indexed item has a value for the given field.
> 
> * On the [**Sources**](https://platform.cloud.coveo.com/admin/#/orgid/content/sources/) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/content/sources/) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/content/sources/) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/content/sources/)) page, when you click the source and then click **More** > **View and map metadata**, you see an entry under the metadata name you chose under the `Crawler` origin.
> 
> * You specified the `origin` value in the field mapping rule (for example, `%[description:crawler]`), but the field value isn't the expected one.
> 
> .Likely cause and resolution
> <details><summary>Details</summary>
> 
> **Cause**
> 
> There's a metadata conflict.
> 
> You can have two configurations extracting values for the same metadata name at the crawling stage (for example, one in a web scraping configuration, and another in the [sitemap XML file](https://docs.coveo.com/en/2656/)).
> When this happens, one value overwrites the other and you only see one `Crawler` origin entry for that metadata name on the **View and map metadata** subpage.
> 
> **Resolution**
> 
> Change the metadata name in your configuration to make it unique and adjust your [field mapping rule](https://docs.coveo.com/en/1839/) accordingly.
> 
> </details>


### Indexing is slow

#### Throttling

> **Context and symptoms**
>
> * The [**Activity Browser**](https://platform.cloud.coveo.com/admin/#/orgid/activity/browser/) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/activity/browser/) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/activity/browser/) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/activity/browser/)) page may show a low or sudden drop in the **Items per hour** value during indexing.
> 
> * Some URLs listed in the sitemap file aren't indexed.
> 
> .Likely cause and resolution
> <details><summary>Details</summary>
> 
> **Cause**
> 
> The **Time the crawler waits between requests to your server** value may be too low and the Sitemap crawler doesn't take into account website `robots.txt` [Crawl-delay directives](https://en.wikipedia.org/wiki/Robots_exclusion_standard#Crawl-delay_directive).
> The Sitemap crawler may be getting throttled by the web server.
> 
> **Resolution**
> 
> On the [**Sources**](https://platform.cloud.coveo.com/admin/#/orgid/content/sources/) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/content/sources/) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/content/sources/) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/content/sources/)) page, open your source and increase the [**Time the crawler waits between requests to your server**](https://docs.coveo.com/en/1967#time-the-crawler-waits-between-requests-to-your-server) value (for example, `1000` milliseconds).
> 
> Also ensure your source
> 
> </details>


### Indexed content is not up to date

#### Refresh limitations

> **Context and symptoms**
>
> * Recently deleted and added sitemap file entries aren't reflected in the [**Content Browser**](https://platform.cloud.coveo.com/admin/#/orgid/content/browser/) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/content/browser/) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/content/browser/) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/content/browser/)).
> 
> * The [**Activity Browser**](https://platform.cloud.coveo.com/admin/#/orgid/activity/browser/) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/activity/browser/) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/activity/browser/) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/activity/browser/)) shows schedule-triggered [refresh](https://docs.coveo.com/en/2710/) activities _are_ taking place.
> 
> .Likely cause and resolution
> <details><summary>Details</summary>
> 
> **Cause**
> 
> A source refresh doesn't consider deleted and new sitemap file entries.
> A [rebuild](https://docs.coveo.com/en/2712/) or [rescan](https://docs.coveo.com/en/2711/) is required to reflect these changes in your index.
> 
> **Resolution**
> 
> Make sure the Sitemap source [rescan schedule](https://docs.coveo.com/en/1933#schedule-a-source-update) is enabled.
> The default `daily` recurrence should suffice.
> 
> </details>


#### Caching

> **Context and symptoms**
>
> * Changes to existing items made prior to the last content update operation aren't reflected in the [**Content Browser**](https://platform.cloud.coveo.com/admin/#/orgid/content/browser/) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/content/browser/) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/content/browser/) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/content/browser/)).
> 
> * The [**Activity Browser**](https://platform.cloud.coveo.com/admin/#/orgid/activity/browser/) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/activity/browser/) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/activity/browser/) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/activity/browser/)) shows that schedule-triggered [rescan](https://docs.coveo.com/en/2711/) and [refresh](https://docs.coveo.com/en/2710/) activities _are_ taking place.
> 
> * The activity [download logs](#important-troubleshooting-fundamentals) may show a NOTICE message in the following format:
>  
> ```text
CDN/Caching Detection: Provider = 'Varnish', ... Please consult https://docs.coveo.com/en/1831/index-content/ip-addresses-to-allowlist to allowlist the IP addresses by the CDN/Caching provider if needed.
```
>  
> This message indicates that the source has detected that pages from the host are served by Varnish, which is a caching provider.
> 
> .Likely cause and resolution
> <details><summary>Details</summary>
> 
> **Cause**
> 
> The Sitemap source uses the last-modified values in the sitemap entries (for example, the `lastmod` element) to determine whether a page has changed since the last crawl.
> Content freshness issues will occur if your infrastructure is serving a cached version of the sitemap file whose last-modified (`lastmod`) values don't reflect recent changes.
> 
> Your infrastructure may also be serving a cached and outdated version of the page listed in the sitemap file.
> 
> These caching issues can occur if you use Incremental Static Regeneration (ISR) technology.
> Caching providers such as Varnish also commonly interfere with requests from the Coveo crawler.
> 
> **Resolution**
> 
> Ensure your infrastructure serves up-to-date versions of the sitemap file and the pages it lists.
> For example:
> 
> * Add middleware that detects the Coveo crawler and rewrites the request to the same URL with a fixed cache-busting query parameter (for example, `?__coveo=1`) so the crawler receives a fresh HTML variant of the page.
> 
> * If you have an HTML sitemap page that relies on JavaScript to render, serve that page with `Content-Type: text/html` (or `application/xhtml+xml`) so Coveo executes the JavaScript and builds a fresh representation at each crawl.
> 
> * Configure your caching provider to [allow inbound requests](https://docs.coveo.com/en/1831/) from the [Coveo Platform](https://docs.coveo.com/en/186/).
> 
> </details>


#### Number of items limit reached

> **Context and symptoms**
>
> * Recent updates to web pages and newly added pages aren't reflected in the [**Content Browser**](https://platform.cloud.coveo.com/admin/#/orgid/content/browser/) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/content/browser/) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/content/browser/) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/content/browser/)).
> 
> * The [**Activity Browser**](https://platform.cloud.coveo.com/admin/#/orgid/activity/browser/) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/activity/browser/) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/activity/browser/) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/activity/browser/)) shows schedule-triggered {refresh} and {rescan} activities are failing with a `DOCUMENT_LIMIT_EXCEEDED` error code.
> 
> * On the [**System Performance**](https://platform.cloud.coveo.com/admin/#/orgid/organization/system-performance/) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/organization/system-performance/) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/organization/system-performance/) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/organization/system-performance/)) page, the **Index size** section shows that your index contains more than twice as many items as your license allows.
> 
> .Likely cause and resolution
> <details><summary>Details</summary>
> 
> **Cause**
> 
> Indexing is blocked because you've reached the 200% license item usage threshold.
> 
> **Resolution**
> 
> * If possible, delete unused sources to bring the item count below the 200% threshold.
> Then, see the [July 20, 2023 Coveo Platform update](https://docs.coveo.com/en/o4hf8088#check-your-number-of-indexed-items) for suggestions on how to reduce your item count even more.
> 
> * To reassess your needs and discuss your options, contact your Coveo Customer Success Manager.
> 
> </details>


#### Last modification date refresh support conditions

> **Context and symptoms**
>
> * Recent changes to existing items listed in the sitemap file aren't reflected in the [**Content Browser**](https://platform.cloud.coveo.com/admin/#/orgid/content/browser/) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/content/browser/) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/content/browser/) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/content/browser/)).
> 
> * The [**Activity Browser**](https://platform.cloud.coveo.com/admin/#/orgid/activity/browser/) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/activity/browser/) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/activity/browser/) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/activity/browser/)) shows schedule-triggered refresh activities _are_ taking place.
> 
> .Likely cause and resolution
> <details><summary>Details</summary>
> 
> **Cause**
> 
> Your sitemap file doesn't meet all `Last Modification Date` refresh support conditions specified in the [source key characteristics](https://docs.coveo.com/en/1967#source-key-characteristics) table.
> The schedule is running but changes aren't being indexed.
> 
> **Resolution**
> 
> As a workaround, consider changing the Sitemap source [rescan schedule](https://docs.coveo.com/en/1933#schedule-a-source-update) from a `daily` to an `hourly` recurrence.
> Disable the refresh schedule to save resources.
> 
> </details>


#### SkipOnSitemapError setting

> **Context and symptoms**
>
> * Recent changes to existing items aren't reflected in the [**Content Browser**](https://platform.cloud.coveo.com/admin/#/orgid/content/browser/) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/content/browser/) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/content/browser/) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/content/browser/)).
> 
> * The source configuration targets multiple sitemap files.
> 
> * The [**Activity Browser**](https://platform.cloud.coveo.com/admin/#/orgid/activity/browser/) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/activity/browser/) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/activity/browser/) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/activity/browser/)) shows no error messages on source rebuild, refresh, and rescan activities.
> 
> .Likely cause and resolution
> <details><summary>Details</summary>
> 
> **Cause**
> 
> The source may be configured with `SkipOnSitemapError` set to `true` and encountering exceptions on one sitemap file during content update activities.
> 
> **Resolution**
> 
> * To help expose and isolate such issues, consider breaking up the Sitemap source into multiple sources, with [`SkipOnSitemapError`](https://docs.coveo.com/en/3158#skiponsitemaperror-boolean) set to `false` on each of them, OR
> 
> * Make sure all referenced source sitemap files are accessible.
> Make adjustments to the source [**Sitemap URLs**](https://docs.coveo.com/en/1967#sitemap-urls), if necessary.
> 
> </details>