---
title: Add a Web source
slug: malf0160
canonical_url: https://docs.coveo.com/en/malf0160/
collection: index-content
source_format: adoc
---
# Add a Web source

:figure-caption!:

Members with the [required privileges](#required-privileges) can use a Web [source](https://docs.coveo.com/en/246/) to make the content of a website searchable.

The Web source [crawler](https://docs.coveo.com/en/2121/) behaves similarly to bots of web search engines such as Google.
The source only needs a starting URL and then automatically discovers all the pages of the site following the site navigation and all `<a href>` hyperlinks in the page HTML, including those without visible link text.
Only pages that are discovered are [indexed](https://docs.coveo.com/en/204/), and in the order they're discovered.

## Source key characteristics

The following table presents the main characteristics of a Web source.

[%header,cols="3,3,2,4"]
|===
2+|Features
|Supported
|Additional information

2+|Indexable content
^|Webpages (complete)
|

2+|Compressed HTTP responses
^|[check]
|The source automatically handles compressed web server HTTP responses with the following `Content-Encoding` header values: `gzip`, `deflate`, and `br`.

.3+.^|[Content update operations](https://docs.coveo.com/en/2039/)
|[refresh](https://docs.coveo.com/en/2710/)
^|[x]
|

|[rescan](https://docs.coveo.com/en/2711/)
^|[check]
a|[Takes place every day by default](https://docs.coveo.com/en/1933/)  
  
The source uses changes to metadata such as the ETag and Last-Modified server response headers, as well as content size, to determine if an item has changed since the last update.

|[rebuild](https://docs.coveo.com/en/2712/)
^|[check]
|

.3+.^|[Content security options](https://docs.coveo.com/en/1779/)
|[Same users and groups as in your content system](https://docs.coveo.com/en/1779#same-users-and-groups-as-in-your-content-system)
^|[x]
|

|[Specific users and groups](https://docs.coveo.com/en/1779#specific-users-and-groups)
^|[check]
|

|[Everyone](https://docs.coveo.com/en/1779#everyone)
^|[check]
|

.2+.^|[Authentication methods](#authentication-subtab)
|Basic authentication
^|[check]
|

|Form authentication
^|[check]
|

2+|[Crawling rules](#crawling-rules-subtab)
^|[check]
|A variety of basic and advanced rules may be used to ignore the webpages you don't want to index.

.4+|[Metadata indexing for search](#index-metadata)

|Automatic mapping of [metadata](https://docs.coveo.com/en/218/) to [fields](https://docs.coveo.com/en/200/) that have the same name
2+a|Disabled by default.
To enable, [access the JSON configuration of your source](https://docs.coveo.com/en/1685#access-the-edit-configuration-with-json-panel), and set [`performFieldMappingUsingAllOrigins`](https://docs.coveo.com/en/1640#about-the-performfieldmappingusingallorigins-setting) to `true`.

|Automatically indexed [metadata](https://docs.coveo.com/en/218/)
2+a|Examples of [auto-populated default fields](https://docs.coveo.com/en/1833#field-origin) (no user-defined metadata required):  
  

* `clickableuri`

* `date`

* `filetype`

* `language` (autodetected from document content)

* `title`  
&nbsp;

The [`author`](https://docs.coveo.com/en/1833#field-origin) field will also be auto-populated if the content item contains an `author` metadata value.  
  

After a content update, [inspect your item field values](https://docs.coveo.com/en/2053#inspect-search-results) in the **Content Browser**.

|Extracted but not indexed metadata
2+a|The source automatically extracts the `content` attribute of `<meta>` tags when the tag includes one of the following attributes: `name`, `property`, `itemprop`, or `http-equiv`.
For example, if the HTML of a page contains the following: `<meta name="author" content="jsmith"/>`, the Web source extracts _jsmith_ as the `author` metadata.  
  
After a rebuild, review the [**View and map metadata**](https://docs.coveo.com/en/m9ti0339#view-and-map-metadata-subpage) subpage for the list of indexed metadata, and [index additional metadata](https://docs.coveo.com/en/m9ti0339#index-metadata).

|Custom metadata extraction
2+a|Available using the following source features:  
  
&bull; [Web scraping](#web-scraping-subtab)  
  
&bull; [IndexJsonLdMetadata](https://docs.coveo.com/en/mc1f0219#indexjsonldmetadata-boolean) JSON configuration parameter

2+|[JavaScript content rendering](#execute-javascript-on-pages)
^|[check]
|The crawler can run JavaScript on a webpage to dynamically render content before indexing the page.

2+|[Shadow DOM content retrieval](#execute-javascript-on-pages)
^|[check]
|If you choose to render JavaScript content, you can also specify whether the crawler should traverse and index attached Shadow DOM content.

2+|[Web scraping](#web-scraping-subtab)
^|[check]
|Exclude irrelevant sections in pages, extract custom [metadata](https://docs.coveo.com/en/218/), and generate sub-items.

2+|[Optical Character Recognition (OCR)](#content-and-images)
^|[check]
|Available at an extra charge.
Contact [Coveo Sales](https://www.coveo.com/en/contact) to add this feature to your [Coveo organization](https://docs.coveo.com/en/185/) license.

2+|[Robots.txt crawl-delay and page restrictions](#directives-overrides)
^|[check]
|Some lesser-known `robots.txt` directives such as `visit-time` and `request-rate` aren't supported.
|===

## Limitations

* Only pages reachable through website page _hyperlinks_ are indexed.
For example, the Web source crawler doesn't follow options in a `<select>` tag.

* Refresh isn't available.
A daily rescan is defined, but not enabled by default.
You can [enable this daily rescan](https://docs.coveo.com/en/1933#schedule-a-source-update) on a per-source basis.

* Multi-factor authentication (MFA) and CAPTCHA aren't supported.

* The Web source crawler can handle up to 200 cookies for the same domain, and a total of 3000 cookies.
If the crawled sites add cookies beyond these limits, the crawler will drop older cookies, which can cause issues (for example, if a dropped cookie is required for authentication).

* Indexing page permissions isn't supported.

* Although the [`MaxPageSizeInBytes`](https://docs.coveo.com/en/mc1f0219#maxpagesizeinbytes-integer) is set to `0` (unlimited size) by default in the [source JSON configuration](https://docs.coveo.com/en/1685/), the [Coveo indexing pipeline](https://docs.coveo.com/en/184/) can handle web pages up to 512 MB only.
Larger pages are [indexed by reference](https://docs.coveo.com/en/l3qg9275#file-type-configurations) (that is, their content is ignored by the Coveo [crawler](https://docs.coveo.com/en/2121/), and only their metadata and path are searchable).
Therefore, no search result [Quick view](https://docs.coveo.com/en/2760#search-result-quick-view) is available for these larger [items](https://docs.coveo.com/en/210/).

* JavaScript usage and limitations:

** The [**Execute JavaScript on pages**](#execute-javascript-on-pages) and **Add time for the crawler to wait before considering a page as fully rendered** settings only pertain to webpage content retrieval for indexing.
When authenticating, the Web crawler applies the [Loading delay](#loading-delay) or the [custom login sequence](#custom-login-sequence) [wait delay](https://docs.coveo.com/en/3289#wait-delay) values.

** Content in pop-up windows and elements that require interaction aren't indexed.

** When the [**Execute JavaScript on pages**](#execute-javascript-on-pages) option is enabled, the source doesn't support the [`UseProxy`](https://docs.coveo.com/en/mc1f0219#useproxy-boolean) parameter.

* The [`UseProxy`](https://docs.coveo.com/en/mc1f0219#useproxy-boolean) parameter can't be used in combination with [**Form authentication**](https://docs.coveo.com/en/malf0160#form-authentication).

## Leading practices

* [Favor using a Sitemap source](https://docs.coveo.com/en/2680#sitemap-or-web-source) when the site features a sitemap file.

* When a [connector](https://docs.coveo.com/en/2734/) exists for the technology that powers the website, instead create a source based on [that connector](https://docs.coveo.com/en/1702#choosing-the-right-connector), which typically indexes content, metadata, and permissions more effectively.

**Example**

You want to make an Atlassian Confluence-powered site content searchable.
Create a **Confluence** source, not a **Web** source.

* It's best to create or edit your source in your sandbox organization first.
Once you have confirmed that it indexes the desired content, you can copy your source configuration to your production organization, either [with a snapshot](https://docs.coveo.com/en/3239/) or manually.
See [About non-production organizations](https://docs.coveo.com/en/2959/) for more information and best practices regarding sandbox organizations.

* Always try authenticating without a [custom login sequence](https://docs.coveo.com/en/3289/) first.
You should only start working on a custom login sequence when you're sure your form authentication details (that is, login address, user credentials, validation method) are accurate and that the standard form authentication process doesn't work.

* Ensure that you have the right to crawl the public content in the event where you aren't the owner of the website.
Crawling sites that you don't own nor have the right to crawl could create reachability issues.

Some sites use infrastructure components such as CDN/Caching providers (for example, Akamai, Cloudflare, and Varnish) that can affect Coveo's ability to retrieve content.
If you're unfamiliar with these mechanisms, learn about them before you configure your source.
For example, a CDN/Caching provider can detect the Coveo crawler and block it from further crawling.

* The number of [items](https://docs.coveo.com/en/210/) that a source processes per hour (crawling speed) depends on various factors, such as network bandwidth and source configuration.
See [About crawling speed](https://docs.coveo.com/en/2078/) for information on what can impact crawling speed, as well as possible solutions.

* Leverage the [**Time the crawler waits between requests to your server**](#time-the-crawler-waits-between-requests-to-your-server) parameter to increase the crawling speed for the sites you own.
Contact the [Coveo Support](https://connect.coveo.com/s/case/Case/Default) team for help if needed.

* [Schedule rescan operations](https://docs.coveo.com/en/1933/) following the rate at which your source content changes.

* To index only one or a few specific pages of a site such as for a test, enter the pages to index as [**Starting URLs**](#starting-urls).
Then, set the [**Number of page levels to crawl from a starting URL**](#number-of-page-levels-to-crawl-from-a-starting-url) parameter value to `0`, instructing the crawler to only index the specified pages, and none of their linked pages.

* Though it's possible to index multiple domains by configuring the source outside the main user interface, doing so is a bad practice.
Always create one source per domain.
This helps:

** Prevent the crawler from using your source authentication credentials on an external site.

** Reduce the number and complexity of crawling and scraping rules.

** Optimize source configurations for each site.

** Avoid having a rebuild/rescan issue on one site cause the deletion of indexed items associated with the other sites.

* Don't enable [`ExpandBeforeFiltering`](https://docs.coveo.com/en/mc1f0219#expandbeforefiltering-boolean) unless it's necessary.
Setting the `ExpandBeforeFiltering` parameter to `true` can significantly reduce the crawling speed since the crawler retrieves many pages that can be rejected in the end.

* Group your source and the other implementation [resources](https://docs.coveo.com/en/2820/) together in a [project](https://docs.coveo.com/en/n7ed6189/).
See [Manage projects](https://docs.coveo.com/en/n7ef0517/).

## Prerequisites

Ensure that your license allows you to create this source and that your organization hasn't reached its source limit.
Check the [Coveo Administration Console](https://docs.coveo.com/en/183/) **Settings** page, under **License** > [**Details**](https://platform.cloud.coveo.com/admin/#/orgid/settings/license/details) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/settings/license/details) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/settings/license/details) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/settings/license/details)) and [**Limits**](https://platform.cloud.coveo.com/admin/#/orgid/settings/license/limits) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/settings/license/limits) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/settings/license/limits) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/settings/license/limits)), for your license details and current usage.

## Add a Web source

Follow these steps to add a Web source using the desired [content retrieval method](https://docs.coveo.com/en/1612/):

. Make sure that your organization meets the [prerequisites](#prerequisites) before adding this source.

. On the [**Sources**](https://platform.cloud.coveo.com/admin/#/orgid/content/sources/) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/content/sources/) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/content/sources/) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/content/sources/)) page, click **Add source**.

. In the **Add a source of content** panel, click the **Cloud** (icon:cloud-icon[alt=cloud-icon,width=16]) or [**Crawling Module**](https://docs.coveo.com/en/3260/) ([crawlingmodule]) tile, depending on your [content retrieval context](https://docs.coveo.com/en/1612/).
With the latter, you must [install the Crawling Module](https://docs.coveo.com/en/3263/) to make your source operational.

![Two content retrieval methods for the Web source | Coveo](https://docs.coveo.com/en/assets/images/index-content/source-retrieval-method-tiles.png)

. In the **Add a new Web source** / **Add a new Crawling Module Web source** panel, fill in the following fields.

**Name**: Use a short and descriptive name, using only letters, numbers, hyphens (-), and underscores (_).
The source name can't be modified once it's saved.

**Starting URL**: The URL of a website page from which the crawler starts discovering and following links found in pages, including:

** The protocol (for example, `http`, `https`)

** The [subdomain](https://en.wikipedia.org/wiki/Subdomain), if applicable (for example, the `www` subdomain)

**Examples of valid starting URLs**

* `\https://www.coveo.com`

* `\https://docs.coveo.com/en`

With the cloud Web source, as soon as you've typed the website domain, the source looks for sitemap files in standard website locations.
If sitemap files are found, they're displayed and you're prompted to switch to the Coveo [Sitemap source](https://docs.coveo.com/en/1967/).
Switching to the Sitemap source is [recommended](https://docs.coveo.com/en/2680#sitemap-or-web-source).

. Continue configurations as a Sitemap source or Web source.

** If available, click **Switch to a Sitemap source** and continue [configuring your Sitemap source](https://docs.coveo.com/en/1967#addstep3) with the autodetected sitemap URLs.

OR

** Continue configuring your Web source.

... Fill in the following fields:

**Crawling Module**: If you're creating a Crawling Module Web source, select the installed Crawling Module instance.

**Project**: The [projects](https://docs.coveo.com/en/n7ef0517/) you want to associate your source with.

> **Note**
>
> After source creation, you can update your Coveo project selection under the [**Identification**](#identification-subtab) subtab.

... Click **Next**.

... Select who has [permission to access the content](https://docs.coveo.com/en/1779/) through the search interface and click **Add source**.

> **Note**
>
> This information is editable later in the [**Content security**](#content-security-tab) tab.

... Continue configuring your source.

### "Configuration" tab

The **Configuration** tab lets you manage the crawling rules, web scraping configurations, advanced settings, and authentication methods of your source. These configuration groups are presented in subtabs.

#### "Crawling rules" subtab

The **Crawling rules** subtab lets you define the specific pages to index.

##### Starting URLs

The Starting URL you entered when creating the Web source is automatically added to the **Starting URLs** list.
Add other starting URLs in the same domain to ensure that orphan pages and isolated sections of your website are crawled and indexed.

##### Exclusions and inclusions

Add exclusion and inclusion rules to crawl only specific items based on their URL.

![Exclusions and inclusions user interface screenshot | Coveo](https://docs.coveo.com/en/assets/images/index-content/exclusions-and-inclusions.png)

The following diagram illustrates how the Web [crawler](https://docs.coveo.com/en/2121/) applies the exclusion and inclusion rules.
This flow applies to all pages, including the starting URLs.
You must therefore pay attention to not filter out your starting URLs.

> **Note**
>
> The diagram shows the crawling process with the default source parameter settings.
> Certain parameters (for example, [ExpandBeforeFiltering](https://docs.coveo.com/en/mc1f0219#expandbeforefiltering-boolean)) can fundamentally change this behavior.

![Crawling workflow diagram | Coveo](https://docs.coveo.com/en/assets/images/index-content/crawl-rules-flow.png)

> **About the "Include all non-excluded pages" option**
>
> [.float-group]
> --
> ![Crawling flow with the all-inclusive inclusion rule | Coveo](:https://docs.coveo.com/en/assets/images/index-content/crawl-rules-flow-all-include.png)
> 
> The **Include all non-excluded pages** option automatically adds an "include all" inclusion rule in the background.
> This ensures that all starting URLs meet the `Does URL match at least one inclusion rule?` condition and that all non-excluded pages get crawled.
> 
> --

You can use any of the six types of rules:

* **is** and a URL that includes the protocol.
For example, `+https://myfood.com/+`.

* **contains** and a string found in the URL.
For example, `recipes`.

* **begins with** and a string found at the beginning of the URL and which includes the protocol.
For example, `+https://myfood+`.

* **ends with** and a string found at the end of the URL.
For example, `.pdf`.

* **matches wilcard rule** and a wildcard expression that matches the whole URL.
For example, `+https://myfood.com/recipes*+`.

* **matches regex rule** and a regex rule that matches the whole URL.
For example, `^.**(company-(dev|staging)).**html.?$`.

> **Tip**
>
> When using regex rules, make sure they match the desired URLs with a testing tool such as [Regex101](https://regex101.com/).

#### "Web scraping" subtab

The **Web scraping** subtab lists and lets you manage [web scraping](https://docs.coveo.com/en/2767/) configurations for your source.

When the crawler is about to index a page, it checks whether it must apply web scraping configurations that have been defined.
The crawler considers the [**Pages to target**](https://docs.coveo.com/en/mc1f3573#configuration-info) rules of each of your web scraping configurations, starting with the configuration at the top of your list.
The crawler will either apply [the first matching configuration or all matching configurations](#single-match-vs-multi-match).

Indexing irrelevant page sections and not extracting custom metadata reduces the quality of search results.
With this in mind, all new Web sources are created with a default web scraping configuration that excludes typical repetitive elements found in web pages that shouldn't be indexed.

![Default web scraping configuration | Coveo](https://docs.coveo.com/en/assets/images/index-content/default-web-scraping.png)

Existing Web sources without a web scraping configuration prompt you to add the default configuration when you access the **Web scraping** subtab.

![Apply the default web scraping configuration | Coveo](https://docs.coveo.com/en/assets/images/index-content/apply-default-web-scraping.png)

> **Important**
>
> When no web scraping configuration is defined:
> 
> * All [crawling rules included pages](#crawling-rules-subtab) are indexed in their entirety (that is, no sections are excluded).
> 
> * No custom metadata is extracted.
> 
> * No sub-items are created.

The Web source features two web scraping configuration management modes: [UI-assisted mode](#ui-assisted-mode) and [Edit with JSON mode](#edit-with-json-mode).

##### UI-assisted mode

You can add (+), edit ([edit]), and delete ([delete]) _one_ web scraping configuration at a time with a user interface that makes many technical aspects transparent.
UI-assisted mode is easier to use and more mistake-proof than Edit with JSON mode.

Use this mode except for sub-item related configurations (which are only supported in Edit with JSON mode).

> **Note**
>
> The Web scraping tab displays a message when the aggregated web scraping configuration contains a sub-item related configuration.
> 
> ![Message shown on the Web scraping tab when sub-items are configured | Coveo](:https://docs.coveo.com/en/assets/images/index-content/sub-items-detected.png)

When you add or edit a web scraping configuration using UI-assisted mode, the **Add/Edit a web scraping configuration** panel is displayed.
See [Configurations in UI-assisted mode](https://docs.coveo.com/en/mc1f3573#configurations-in-ui-assisted-mode) for more details.

##### Edit with JSON mode

The **Edit with JSON** button gives access to the _aggregated_ web scraping JSON configuration of the source.
Adding, editing, and deleting configurations directly in the JSON requires more technical skills than using UI-assisted mode.

Use this mode to perform sub-item related configurations and when you want to test your aggregated web scraping configuration with the Coveo Labs [Web Scraper Helper](https://docs.coveo.com/en/mc1f3573#3-use-the-right-tools).

> **Note**
>
> The Web scraping tab displays a message when the aggregated web scraping configuration contains a sub-item related configuration.
> 
> ![Message shown on the Web scraping tab when sub-items are configured | Coveo](:https://docs.coveo.com/en/assets/images/index-content/sub-items-detected.png)

When you add or edit a web scraping configuration in Edit with JSON mode, the **Edit a web scraping JSON configuration** panel is displayed.
See [Configurations in Edit with JSON mode](https://docs.coveo.com/en/mc1f3573#configurations-in-edit-with-json-mode) for more details.

##### Single-match vs multi-match

The Web source can apply web scraping configurations in two ways: single-match or multi-match.

In single-match mode, the crawler applies only the first matching web scraping configuration.
In multi-match mode, the crawler applies all matching web scraping configurations.

The animation below demonstrates the application of three web scraping configurations on a culinary website featuring news articles and recipe pages, in single-match mode (left) and multi-match mode (right).

![Animation showing the single-match and multi-match behaviors | Coveo](https://docs.coveo.com/en/assets/images/index-content/single-vs-multi-match-animation.gif)

Web sources created before mid-December 2023 were created in single-match mode.
All new Web sources are created in multi-match mode.

Coveo converted existing single-match sources containing zero or one web scraping configuration to multi-match mode.
We recommend you convert any remaining single-match Web source to multi-match mode.
If a Web source is currently in single-match mode, the **Web scraping** subtab displays a banner prompting you to convert to multi-match mode.

![Multi-match conversion banner | Coveo](https://docs.coveo.com/en/assets/images/index-content/single-match-switch-message.png)

To convert a source to multi-match mode

. In the **Web scraping** subtab, click **Switch to multi-match mode**.

. Confirm you want to convert the source to multi-match mode.
A green **You're currently in multi-match mode** banner will then appear.

. Click **Save**.

Once your source is fully converted, the **Web scraping** subtab no longer shows the green banner and the subtab description reflects the multi-match mode behavior.

![Web scraping configuration options and description for targeting and processing pages | Coveo](https://docs.coveo.com/en/assets/images/index-content/web-scraping-configuration-description.png)

#### "Advanced settings" subtab

The **Advanced settings** subtab lets you customize the Coveo crawler behavior.
All advanced settings have default values which are adequate in most use cases.

##### Execute JavaScript on pages

Only enable this option when website content you want to consider for indexing is dynamically rendered by JavaScript, as it can significantly increase the time needed to crawl pages.

If you enable **Execute JavaScript on pages**, you'll have the following options:

* **Add time for the crawler to wait before considering a page as fully rendered**:
The default value of this setting is `0`, which means that the crawler doesn't wait after the page is loaded to retrieve its content.
If the JavaScript takes longer to execute than normal or makes asynchronous calls, consider increasing this value to ensure that pages with longer rendering times are indexed with their dynamically rendered content.

* **Enable Shadow DOM content retrieval**:
When you enable this option, the crawler builds a flattened DOM tree by combining the light DOM and the Shadow DOM.
It then processes the resulting structure as it would any other web page.

> **Note**
>
> The crawler adds a custom attribute to the shadow root elements in the flattened DOM, allowing these elements to be targeted using a [special web scraping CSS selector](https://docs.coveo.com/en/mc1f3573#css-selectors).

> **Important**
>
> Building the composed DOM can significantly slow down indexing.
> Enable this option only if the Shadow DOM contains valuable content you need to index.

##### User Agent string

The [user agent](https://en.wikipedia.org/wiki/User_agent) string that the Web source crawler uses to identify itself when requesting pages from your web server.

The default value is `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko) (compatible; Coveobot/2.0;+http://www.coveo.com/bot.html)`.

##### Query parameters to ignore

Add query string parameters that the source should ignore when determining whether a URL corresponds to a distinct item.

By default, the source considers the whole URL to determine whether the page is a distinct item.
The URLs of the website you index can contain one or more query string parameters after the host name and the path.
Some query string parameters may change the content of the page significantly, and therefore legitimately contribute to a distinct page.
Other query string parameters may not affect the content of the page, or very little.
In the latter case, you want to ignore the query string parameter to avoid creating search result duplicates.

**Example**

The URL of a website page for which you get search result duplicates looks as follows:

`+http://www.mysite.com/v1/getitdone.html?lang=en&param1=abc&param2=123+`

The values of `param1` and `param2` can change without affecting the page content while the `lang` value changes the language in which the page appears.
You want to ignore the `param1` and `param2` query string parameters to eliminate search result duplicates, but not `lang`.
In this example, you would therefore add the `param1` and `param2` parameters.

> **Note**
>
> Wildcards or ReGex aren't supported in query string parameter names.
> For instance, in the example above, you can't cover both the `param1` and `param2` query string parameters using `param*`.

##### Directives overrides

* Check the **robots.txt** box if you want the Coveo crawler to ignore directives specified in the website's [`robots.txt`](https://docs.coveo.com/en/mc2a1538#the-robots-txt-override-setting) file.

* Check the **noindex** box if you want the Coveo crawler to index pages that have a [`noindex` directive](https://docs.coveo.com/en/mc2a1538#the-noindex-and-nofollow-links-override-settings) in their `meta` tag or in their `X-Robots-Tag` HTTP response header.

* Check the **nofollow links** box if you want the Coveo crawler to follow links in pages that have a [`nofollow` directive](https://docs.coveo.com/en/mc2a1538#the-noindex-and-nofollow-links-override-settings) in their `meta` tag or in their `X-Robots-Tag` HTTP response header.

* Check the **nofollow anchors** box if you want the Coveo crawler to follow links that have a [`rel="nofollow"` attribute](https://docs.coveo.com/en/mc2a1538#the-nofollow-anchors-override-setting).

##### Number of page levels to crawl from a starting URL

Indicate the number of page link levels (or clicks) the crawler can travel from any starting URL.
A starting URL is level 0.
All pages accessible from a starting URL are considered level 1.

##### Time the crawler waits between requests to your server

Indicate the number of milliseconds between consecutive HTTP requests to the website server.
The default value is 1000 milliseconds, which represents a crawling rate of one page per second.

One page per second is the highest rate at which Coveo can crawl a public website for a cloud Web source without [proof of ownership of the website](#proof-of-website-ownership).
You can enter a number below 1000.
However, the Coveo crawler will only apply a crawling delay below 1000 milliseconds if it can verify that you're the owner of the site.

If you're retrieving content of an internal website using the Crawling Module Web source, the crawling delay you specify applies automatically.
You don't need to prove site ownership as Coveo detects that the crawled site has a [private IP address](https://en.wikipedia.org/wiki/Private_network#Private_IPv4_addresses).

#### "Authentication" subtab

The **Authentication** settings, used by the source crawler, emulate the behavior of a user authenticating to access restricted website content.
If authentication is required, select the authentication type your website uses, whether [**Basic authentication**](#basic-authentication) or [**Form authentication**](#form-authentication).
Then, provide the corresponding login details.

> **Warning**
>
> Whether you use Basic or Form authentication, limit your source [crawling scope](#crawling-rules-subtab) to one domain that you own.
> This reduces the risk of exposing your authentication credentials.

> **Note**
>
> Manual form authentication is now only available on legacy sources.
> We recommend you [migrate existing Manual form authentication sources](#migrate-from-manual-form-authentication) to Form authentication.

##### Basic authentication

When selecting **Basic authentication**, enter the credentials of an account on the website you're making searchable.
See [Source credentials leading practices](https://docs.coveo.com/en/1920/).

> **Important**
>
> When **Execute JavaScript on pages** is enabled on the source, basic authentication significantly impacts indexing performance.

When the Coveo crawler follows links requiring basic authentication while indexing your website, it only uses the basic authentication credentials you entered if the link URL matches the scheme and domain of the [**Starting URLs**](#starting-urls).
If this condition isn't met, the Coveo crawler doesn't try to authenticate.

For example, if your starting URL is `+https://www.example.com+`, the Coveo crawler doesn't even try to authenticate if the link URL uses:

* A different scheme (that is, it uses HTTP instead of HTTPS).

* A different domain (for example, `+https://www.mysite.com+`), unless [`UseHiddenBasicAuthentication`](https://docs.coveo.com/en/mc1f0219#usehiddenbasicauthentication-boolean) is enabled.

##### Form authentication

You can choose between two form authentication workflows:

**Force authentication disabled (recommended)**
<details><summary>Details</summary>

With [Force authentication](#force-authentication) disabled, the workflow typically goes as follows:

. Coveo's crawler requests a protected page.

. The web server redirects the crawler to the [**Login page address**](#login-page-address).

. Using the configured [**Validation method**](#validation-method), the crawler determines it's not authenticated.
This automatically triggers the next step.

. The crawler performs a standard login sequence using the provided **Login details**, or the [**Custom login sequence**](#custom-login-sequence) if one is configured.

. After successful authentication, the web server responds by redirecting back to the requested protected page and returning cookies.

. The crawler follows the server redirect to get the protected page and indexes that page.

. The crawler requests the other pages using the cookies.

This is the default and recommended workflow as it emulates human behavior the best and ensures crawler re-authentication, when needed.

</details>

**Force authentication enabled**
<details><summary>Details</summary>

With [Force authentication](#force-authentication) enabled, the workflow typically goes as follows:

. The crawler performs a standard login sequence using the provided **Login details**, or the [**Custom login sequence**](#custom-login-sequence) if one is configured.

. After successful authentication, the web server responds with cookies that the crawler will use to request other pages.

. The crawler requests the first URL from the web server using the cookies and indexes that page.

. The crawler requests other pages using the cookies.

If the crawler loses authentication at some point (for example, if a cookie expires), it has no way of knowing it must re-authenticate unless you have a proper authentication status [validation method](#validation-method).
As a result, you may notice at some point that your source has indexed some, but _not all_, protected pages.

Only use [**Force authentication**](#force-authentication) when no reliable authentication status [validation method](#validation-method) can be configured.

</details>

> **Note**
>
> The crawler can interact with Shadow DOM elements in your login pages.
> If this is required, make sure the form authentication [loading delay](#loading-delay) allows the Shadow DOM time to load before the crawler begins to interact with the page.

###### Username and password

Enter the credentials required to access the secured content.
See [Source credentials leading practices](https://docs.coveo.com/en/1920/).

###### Login page address

Enter the URL of the website login page where the username and password are to be used.

###### Loading delay

Enter the maximum time the crawler should allow for JavaScript to execute and go through the login sequence before timing out.

###### Validation method

The crawler uses the validation method after requesting a page from the web server to know if it's authenticated or not.
When the validation method reveals that the crawler isn't authenticated, the crawler immediately tries to re-authenticate.

To configure the validation method

. In the dropdown menu, select your preferred authentication status validation method.

. In the **Value(s)** field, specify the corresponding URL, regex or text.

** For **Cookie not found** (recommended):

Enter the name of the cookie returned by the server after _successful_ authentication.
If this cookie isn't found, the crawler will immediately authenticate (or re-authenticate).

**Example**

`ASP.NET_SessionId`

** For **Redirection to URL** (recommended):

Enter the URL where users trying to access protected content on the website are redirected to when they're _not_ authenticated.
If the crawler is redirected to this URL, it will immediately authenticate (or re-authenticate).

**Example**

`+https://mycompany.com/login/failed.html+`

** For **Text not found in page** footnote:not-recommended[Less reliable than the recommended validation methods. Can result in false positives, making form authentication issues harder to troubleshoot.]:

Enter the text that appears on the page after _successful_ authentication.
If this text isn't found on the page, the crawler will immediately authenticate (or re-authenticate).

**Example**

When a user successfully logs in, the page shows a "Hello, <USERNAME>!" greeting text.
If the login [username you specified](#username-and-password) was `+jsmith@mycompany.com+`,
the text to enter would be:

`Hello, \jsmith@mycompany.com!`

**Example**

`Log out`

** For **Text found in page** footnote:not-recommended[]:

Enter the text that appears on the page when a user _isn't_ authenticated.
If this text is found on the page, the crawler will immediately authenticate (or re-authenticate).

**Examples**

* `An error has occurred.`

* `Your username or password is invalid.`

** For **URL matches regex** footnote:not-recommended[]:

Enter a regex rule that matches the URL where users trying to access protected content are redirected to when they're _not_ authenticated.
If the crawler is redirected to a URL that matches this regex, it will immediately authenticate (or re-authenticate).

**Example**

`.+Account\/Login.*`

** For **URL doesn't match regex** footnote:not-recommended[]:

Enter a regex rule that matches the URL where users trying to access protected content are redirected to after _successful_ authentication.
If the crawler isn't redirected to a URL that matches this regex, it will immediately authenticate (or re-authenticate).

###### Force authentication

Select this option if you want Coveo's first request to be for authentication, regardless of whether it is actually required.

> **Important**
>
> You should only force authentication if you have no reliable authentication status [validation method](#validation-method).

###### Custom login sequence

The default login sequence for Web and Sitemap sources supports various third-party login pages, such as OneLogin, Google, Salesforce, and Microsoft.

The default login sequence also tries to detect and log in to first-party login forms.
The login process uses the first `<form>` element that meets all requirements.

**Form requirements for the default login sequence**
<details><summary>Details</summary>

The form must contain the following elements:

**User identity field**: A visible `<input>` element whose `id` or `name` attribute value is either `user`, `email`, `login`, `id`, or `name` (case-insensitive).

**Password field**: A visible `<input>` element whose `type` attribute value is `password` (case-insensitive).

**Submit element**: A visible form submit element.
The source looks for the following element types, in this order:

* An `<input>` element whose `type` attribute value is `submit` or `button`.

* A `<button>` element whose `type` attribute value is `submit`.

* An `<a>` element whose `id` attribute value contains the word "submit" (case-insensitive).

The default login sequence uses the first element of each type in the form.

</details>

If the web page doesn't meet the requirements for the default login sequence, or if your form requires specific actions during the login process, you must configure a [custom login sequence](https://docs.coveo.com/en/3289/).

If the web page doesn't meet the requirements for the default login sequence, or if your form requires specific actions during the login process, you must configure a custom login sequence.
[Web](https://docs.coveo.com/en/malf0160/) and [Sitemap](https://docs.coveo.com/en/1967/) sources provide an interface that lets administrators build custom form [authentication](https://docs.coveo.com/en/2120/) workflows.

> **Important**
>
> Ensure that the default source login sequence fails before you configure a custom login sequence.

###### Post-login sequence

The Web source also supports post-login sequences to handle actions that need to be performed after logging in to a website.
Post-login sequences are configured using the `postLoginSequence` section of the `FormAuthenticationConfiguration` parameter in the source JSON configuration.

**Example**
<details><summary>Details</summary>

After logging into a Salesforce site that you want to crawl using a Web source, you encounter the following Salesforce platform infrastructure popup:

![Salesforce scheduled maintenance popup](https://docs.coveo.com/en/assets/images/index-content/scheduled-maintenance-popup.png)

You inspect the button using your browser developer tools and see the following HTML markup:

```html
<button
  type="button"
  class="maintenance-confirm-button"
  onclick="window.history.back();">
  Got it
</button>
```

To emulate a user clicking _Got it_, you configure your source as follows:

. On the [**Sources**](https://platform.cloud.coveo.com/admin/#/orgid/content/sources/) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/content/sources/) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/content/sources/) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/content/sources/)) page, click your source, and then click **More** > **Edit source with JSON** in the Action bar.

. In the **Edit configuration with JSON** panel, use the search tool ([search2]) to locate the `FormAuthenticationConfiguration` parameter.
Its `value` object currently looks as follows:

```txt
"value": "{\"authenticationFailed\":{\"method\":\"CookieNotSet\",.....\"customLoginSequence\":{}}"
```

. Remove the `}"` at the end of the `value` object so that you get:

```txt
"value": "{\"authenticationFailed\":{\"method\":\"CookieNotSet\",.....\"customLoginSequence\":{}
```

. Append the following JSON snippet immediately after `\"customLoginSequence\":{}` to configure the post-login action:

```txt
,\"postLoginSequence\":{\"name\":\"Handle maintenance page\",\"url\":\"*\",\"urlContainsValue\":\"maintenanceandavailable.jsp\",\"steps\":[{\"name\":\"Dismiss maintenance modal\",\"waitDelayInMilliseconds\":500,\"actions\":[{\"type\":\"click\",\"elementIdentifier\":{\"identifier\":\"maintenance-confirm-button\",\"type\":\"default\",\"findType\":\"classname\"}}]}]}}"
```

**Example: Final FormAuthenticationConfiguration parameter value**
[%collapsible]

</details>
```json
"FormAuthenticationConfiguration": {
    "sensitive": false,
    "value": "{\"authenticationFailed\":{\"method\":\"CookieNotSet\",\"values\":[\"sid\"]},\"inputs\":[],\"formUrl\":\"https://somedomain.my.salesforce.com/apex/RedirectPage?siteType=lwr&basePath=/sitesDemo\",\"enableJavaScript\":true,\"forceLogin\":false,\"javaScriptLoadingDelayInMilliseconds\":1000,\"customLoginSequence\":{},\"postLoginSequence\":{\"name\":\"Handle maintenance page\",\"url\":\"*\",\"urlContainsValue\":\"maintenanceandavailable.jsp\",\"steps\":[{\"name\":\"Dismiss maintenance modal\",\"waitDelayInMilliseconds\":500,\"actions\":[{\"type\":\"click\",\"elementIdentifier\":{\"identifier\":\"maintenance-confirm-button\",\"type\":\"default\",\"findType\":\"classname\"}}]}]}}"
  },
```
####  
This post-login action configuration can be translated as the following instruction to the source crawler: "When a page whose URL contains `maintenanceandavailable.jsp` is encountered, wait 500 milliseconds, and then click the element with the class name `maintenance-confirm-button`."

For more details on the post-login action parameters, see [Configure an action](https://docs.coveo.com/en/3289#configure-an-action).

. Click **Save**.
##### ==== "Crawling Module" subtab

If your source is a [Crawling Module source](https://docs.coveo.com/en/1612/), and if you have [more than one Crawling Module linked to this organization](https://docs.coveo.com/en/3271#deploying-multiple-crawling-module-instances), select the one with which you want to pair your source.
If you change the Crawling Module instance paired with your source, a successful [rebuild](https://docs.coveo.com/en/3390#refresh-rescan-or-rebuild-sources) is required for your change to apply.

#### "Identification" subtab

The **Identification** subtab contains general information about the source.

## Name

The source name.
It can't be modified once it's saved.

:leveloffset!:

##### Project

Use the **Project** selector to associate your source with one or more Coveo [projects](https://docs.coveo.com/en/n7ef0517/).

### "Items" tab

On the **Items** tab, you can specify how the source handles items based on their file type or content type.

#### File types

File types let you define how the source handles [items](https://docs.coveo.com/en/210/) based on their file extension or content type.
For each file type, you can specify whether to index the item content and [metadata](https://docs.coveo.com/en/218/), only the item metadata, or neither.

You should fine-tune the file type configurations with the objective of indexing only the content that's relevant to your users.

**Example**

Your repository contains `.pdf` files, but you don't want them to appear in search results.
You click **Extensions** and then, for the `.pdf` extension, you change the **Default action** and **Action on error** values to `Ignore item`.

For more details about this feature, see [File type handling](https://docs.coveo.com/en/l3qg9275/).

> **Tip**
>
> With [file type handling](https://docs.coveo.com/en/l3qg9275/), using the `Index metadata` default action on HTML items lets you index basic metadata for those items.
> On the other hand, web scraping is used to index custom metadata from the page content, which must first be retrieved.
> These metadata indexing mechanisms are complementary, and you can use them together within a source.
> 
> If there are some items for which you only need to index basic metadata, make sure you don't have a web scraping rule that matches those items.
> This will prevent unnecessary processing and potential issues with retrieving protected page content.

#### Content and images

If you want Coveo to extract text from image files or PDF files containing images, enable the appropriate option.
The extracted text is processed as item data, meaning that it's fully searchable and will appear in the item [Quick view](https://docs.coveo.com/en/2760#search-result-quick-view).

> **Note**
>
> When OCR is enabled, ensure the source's relevant [file type configurations](https://docs.coveo.com/en/l3qg9275/) index the item content.
> Indexing the item's metadata only or ignoring the item will prevent OCR from being applied.

See [Enable optical character recognition](https://docs.coveo.com/en/2937/) for details on this feature.

### "Content security" tab

Select who will be able to access the source items through a Coveo-powered [search interface](https://docs.coveo.com/en/2741/).
For details on the content security options, see [Content security](https://docs.coveo.com/en/1779/).

### "Access" tab







. On the **Access** tab, specify whether each group (and API key, if applicable) in your [Coveo organization](https://docs.coveo.com/en/185/) can view or edit the current source.

For example, when creating a new source, you could decide that members of Group A can edit its configuration, while Group B can only view it.

For more information, see [Custom access level](https://docs.coveo.com/en/3151#custom-access-level).
On the **Access** tab, specify whether each group (and API key, if applicable) in your [Coveo organization](https://docs.coveo.com/en/185/) can view or edit the current source.

For example, when creating a new source, you could decide that members of Group A can edit its configuration, while Group B can only view it.

For more information, see [Custom access level](https://docs.coveo.com/en/3151#custom-access-level).

### Build the source

. Finish adding or editing your source:

** When you're done editing the source and want to make your changes effective, click **Add and build source**/**Save and rebuild source**.

** When you want to save your source configuration changes without starting a build/rebuild, such as when you know you want to make other changes soon, click **Add source**/**Save**.
On the [**Sources**](https://platform.cloud.coveo.com/admin/#/orgid/content/sources/) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/content/sources/) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/content/sources/) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/content/sources/)) page, click **Launch build** or **Start required rebuild** when you're ready to make your changes effective and index your content.

> **Leading practice**
>
> By default, a Jira Software source indexes the entire Jira Software instance content.
> To index only certain projects, click **Save**, and then specify the desired address patterns in your [source JSON configuration](https://docs.coveo.com/en/1685/) before launching the initial build.
> See [Add source filters](https://docs.coveo.com/en/2006#add-source-filters) for further information.

. On the [**Sources**](https://platform.cloud.coveo.com/admin/#/orgid/content/sources/) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/content/sources/) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/content/sources/) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/content/sources/)) page, follow the progress of your source addition or modification.

. Once the source is built or rebuilt, [review its content in the Content Browser](https://docs.coveo.com/en/2053/).

. Optionally, consider [editing or adding mappings](https://docs.coveo.com/en/1640/).


> **Note**
>
> If you selected **Specific URLs** or **User profiles** in the [**Content**](https://docs.coveo.com/en/1739#content) section, some additional items will appear in the Content Browser.
> To retrieve user profiles, Coveo must crawl your SharePoint Online instance, including your host site collection and the documents it contains.
> Items encountered during this process are also retrieved and therefore appear in the Content Browser.

### Index metadata

To use [metadata](https://docs.coveo.com/en/218/) values in [search interface](https://docs.coveo.com/en/2741/) [facets](https://docs.coveo.com/en/198/) or result templates, the metadata must be [mapped](https://docs.coveo.com/en/217/) to [fields](https://docs.coveo.com/en/200/).
Coveo automatically [maps](https://docs.coveo.com/en/217/) only a subset of the metadata it extracts.
You must map any additional metadata to fields manually.

> **Note**
>
> Not clear on the purpose of indexing metadata?
> Watch [this video](https://www.youtube.com/watch?v=BmmmVJ3AWi0).

. On the [**Sources**](https://platform.cloud.coveo.com/admin/#/orgid/content/sources/) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/content/sources/) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/content/sources/) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/content/sources/)) page, click your source, and then click **More** > **View and map metadata** in the Action bar.

. Review the default [metadata](https://docs.coveo.com/en/218/) that your source is extracting from your content.

. Map any currently _not indexed_ metadata that you want to use in facets or result templates to fields.

.. Click the metadata and then, at the top right, click **Add to Index**.

.. In the **Apply a mapping on all item types of a source** panel, select the field you want to map the metadata to, or [add a new field](https://docs.coveo.com/en/1833#add-a-field) if none of the existing fields are appropriate.

> **Note**
>
> For advanced mapping configurations, like applying a mapping to a specific item type, see [Manage mappings](https://docs.coveo.com/en/1640#manage-mappings).

.. Click **Apply mapping**.

. Return to the [**Sources**](https://platform.cloud.coveo.com/admin/#/orgid/content/sources/) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/content/sources/) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/content/sources/) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/content/sources/)) page.

. To reindex your source with your new mappings, click your source, and then click **More** > **Rebuild** in the Action bar.

. Once the source is rebuilt, review your item field values.
They should now include the values of the metadata you selected to index.

.. On the [**Sources**](https://platform.cloud.coveo.com/admin/#/orgid/content/sources/) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/content/sources/) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/content/sources/) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/content/sources/)) page, click your source, and then click **More** > **Open in Content Browser** in the Action bar.

.. Select the card of the item for which you want to inspect properties, and then click **Properties** in the Action bar.

.. In the panel that appears, select the **Fields** tab.

. If needed, extract and map additional metadata.

**More on custom metadata extraction**
<details><summary>Details</summary>

To extract custom metadata, you can use the following methods:

* Configure [web scraping](https://docs.coveo.com/en/2767/) configurations that contain [metadata extraction rules](https://docs.coveo.com/en/mc1f3573#web-scraping-configuration-editing-modes) using CSS or XPath selectors.

* Extract metadata from [JSON-LD `<script>` tags](https://docs.coveo.com/en/mc1f0219#indexjsonldmetadata-boolean).

Then, rebuild and map each extracted custom metadata to a field, as you did for the default metadata.

</details>

### Troubleshooting

After a rebuild, you may notice that your source isn't indexing as expected.
For example, there may be missing or extra items, or the values of some fields may not meet your requirements.

To help you troubleshoot, refer to the [list of common issues and solutions](https://docs.coveo.com/en/n1ab5310/) when using the Web source.

## Required privileges

You can assign privileges to allow access to specific tools in the [Coveo Administration Console](https://docs.coveo.com/en/183/).
The following table indicates the privileges required to view or edit elements of the [**Sources**](https://platform.cloud.coveo.com/admin/#/orgid/content/sources/) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/content/sources/) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/content/sources/) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/content/sources/)) page and associated panels.
See [Manage privileges](https://docs.coveo.com/en/3151/) and [Privilege reference](https://docs.coveo.com/en/1707/) for more information.

> **Note**
>
> The **Edit all** privilege isn't required to create sources.
> When granting privileges for the [Sources](https://docs.coveo.com/en/1707#sources-domain) domain, you can grant a group or API key the **View all** or [**Custom**](https://docs.coveo.com/en/3151#custom-access-level) access level, instead of **Edit all**, and then select the **Can Create** checkbox to allow users to create sources.
> See [Can Create ability dependence](https://docs.coveo.com/en/3151#can-create-ability-dependence) for more information.

## Proof of website ownership

Coveo applies a [Time the crawler waits between requests to your server](#time-the-crawler-waits-between-requests-to-your-server) value below 1000 milliseconds only when you prove ownership of the website you want to index.

To prove ownership of the website you want to index

. Create an empty text file named `coveo-ownership-orgid.txt`, replacing `orgid` with your [Coveo organization ID](https://docs.coveo.com/en/1562#organization-details).

. Upload this file at the root of the website you want to index.

> **Note**
>
> If your site has `robots.txt` directives that include a [`crawl-delay`](https://en.wikipedia.org/wiki/Robots_exclusion_standard#Crawl-delay_directive) parameter with a different value, the slowest crawling speed applies.
> See also the [**robots.txt**](#directives-overrides) option.

## Migrate from manual form authentication

If you're using manual form authentication, you'll see a "Manual form authentication deprecation" warning when viewing the **Authentication** subtab.
You'll want to migrate to form authentication.
To do so, we recommend you create a duplicate of your source and configure form authentication on the duplicate.
When the duplicate is configured and fully tested, you can copy its configuration to the original source.

If you're using a sandbox organization and a snapshot-based phased rollout, the alternative is to copy your original source and related resources configurations to your sandbox using the [resource snapshots](https://docs.coveo.com/en/3239/) feature.
Once your sandbox source authentication configurations updated and fully tested, you can use a snapshot to apply your changes to your production organization source.

Though the following procedure uses the source duplicate method, steps 3 to 8 inclusively are common to both methods.

To migrate from manual form authentication to form authentication

. On the [**Sources**](https://platform.cloud.coveo.com/admin/#/orgid/content/sources/) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/content/sources/) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/content/sources/) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/content/sources/)) page, click your source, and then click **More** > **Duplicate** in the Action bar.

. Name your duplicate.

. Click your duplicate source, and then click **Edit** in the Action bar.

. Select the **Authentication** subtab.

. Select the **Form authentication** radio button.

The following fields will be populated automatically using your existing manual form authentication settings: **Username**, **Password**, **Login page address**, **Validation method** and **Value(s)**, **Force authentication**.

. [Rebuild](https://docs.coveo.com/en/2039#rebuild) your duplicate source.

. Make sure that your duplicate source contains properly indexed content.
Things you should check for:

** Your duplicate source contains the same number of items as the original source.

** For pages that are authentication protected in your website, make sure the [Quick view](https://docs.coveo.com/en/1712#access-the-quick-view-tab) of the corresponding items in your duplicate source shows the content of the actual website page.
If form authentication fails, the item Quick view may display the content of your form authentication login page instead of the actual website page.

. If form authentication is failing, consider making the following adjustments to your duplicate source form authentication configuration:

** Changing the [Validation method](#validation-method) and associated **Value(s)** to a more reliable combination.

** Increasing the [Loading delay](#loading-delay).

** Setting up a [custom login sequence](#custom-login-sequence).


Contact [Coveo Support](https://connect.coveo.com/s/case/Case/Default) if you need help.

. When you're sure the authentication configuration on your duplicate source works, apply the changes to the original source.


.. On the [**Sources**](https://platform.cloud.coveo.com/admin/#/orgid/content/sources/) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/content/sources/) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/content/sources/) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/content/sources/)) page, click your duplicate source, and then click **More** > **Edit configuration with JSON** in the Action bar.

.. Copy the `FormAuthenticationConfiguration` JSON object.
The object looks like the following:

[listing]
....
"FormAuthenticationConfiguration": {
  "sensitive": false,
  "value": "{\"authenticationFailed\":{\"method\":\"RedirectedToUrl\",\"values\":[\"https://something.com/Account/Login\"]},\"inputs\":[],
  \"formUrl\":\"https://something.com/Account/Login\",\"enableJavaScript\":true,\"forceLogin\":false,\"javaScriptLoadingDelayInMilliseconds\":2000,\"customLoginSequence\":{}}"
}
....

.. On the [**Sources**](https://platform.cloud.coveo.com/admin/#/orgid/content/sources/) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/content/sources/) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/content/sources/) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/content/sources/)) page, click your original source, and then click **More** > **Edit configuration with JSON** in the Action bar.

.. Replace the `FormAuthenticationConfiguration` object with the one from your duplicate source.

.. Click **Save**.

## What's next?

. [Schedule source updates](https://docs.coveo.com/en/1933/).

. If you're [using the Crawling Module to retrieve your content](https://docs.coveo.com/en/1612/), consider [subscribing to deactivation notifications](https://docs.coveo.com/en/3271#subscribing-to-crawling-module-notifications) to receive an alert when a Crawling Module component becomes obsolete and stops the content crawling process.