Index Page Content With the FetchPageContentProcessor

The FetchPageContentProcessor is the default and recommended Coveo for Sitecore HTML indexing processor. The processor performs an HTTP request, gets the page content, and sets the value of the Coveo for Sitecore code BinaryData field with this content.

This HTTP request introduces a delay when indexing.

Enabling the Processor

You may want to enable the processor if:

You’re currently not indexing rendered HTML and want to start indexing HTML.
You were using the HtmlContentInBodyWithRequestsProcessor in Coveo for Sitecore 4.1 and upgraded to Coveo for Sitecore 5.

When Currently Not Indexing Rendered HTML

The Coveo Command Center has an Index rendered HTML option. When you select this option, Coveo for Sitecore automatically enables the FetchPageContentProcessor.

Open the Indexing Options section of the Command Center, accessible at http://<INSTANCE_HOSTNAME>/coveo/command-center/index.html#indexing-options/.
Select Index rendered HTML.
Click Apply and Restart.

This procedure produces the following configuration structure in the Coveo.SearchProvider.Custom.config file:

<coveoPostItemProcessingPipeline>
  <processor type="Coveo.SearchProvider.Processors.ExecuteGetBinaryDataPipeline ..." />
</coveoPostItemProcessingPipeline>
<coveoGetBinaryData>
  <processor type="Coveo.SearchProvider.Processors.FetchPageContentProcessor ...">
    ...
  </processor>
</coveoGetBinaryData>

The Coveo.SearchProvider.Custom.config file is located in the <SITECORE_INSTANCE_ROOT>\Website\App_Config\Include\Coveo\ folder in Sitecore 8 instances and in the <SITECORE_INSTANCE_ROOT>\App_Config\Include\Coveo\ folder in Sitecore 9 and 10 instances.

Once the FetchPageContentProcessor processor enabled, you might need to perform configurations (e.g., setting up HTTP request authentication for secured items).

When Currently Using HtmlContentInBodyWithRequestsProcessor

If you upgraded from Coveo for Sitecore 4.1 to Coveo for Sitecore 5, you might still be using the HTMLContentInBodyWithRequestsProcessor. Coveo for Sitecore 5 provides a mechanism to switch from the HTMLContentInBodyWithRequestsProcessor to the FetchPageContentProcessor without having to edit your configuration files.

Open the Indexing Options section of the Command Center, accessible at http://<INSTANCE_HOSTNAME>/coveo/command-center/index.html#indexing-options/.
If the Index rendered HTML option is selected:
1. Select Only index Sitecore item data.
2. Click Apply and Restart.
Select Index rendered HTML.
Click Apply and Restart.

This procedure produces the following configuration structure in the Coveo.SearchProvider.Custom.config file:

<coveoPostItemProcessingPipeline>
  <processor type="Coveo.SearchProvider.Processors.ExecuteGetBinaryDataPipeline ..." />
</coveoPostItemProcessingPipeline>
<coveoGetBinaryData>
  <processor type="Coveo.SearchProvider.Processors.FetchPageContentProcessor ...">
    ...
  </processor>
</coveoGetBinaryData>

Once the FetchPageContentProcessor processor enabled, you might need to perform configurations (e.g., setting up HTTP request authentication for secured items).

Configuring the Processor

The FetchPageContentProcessor contains the <inboundFilter>, <preAuthentication>, and <postProcessing> sections shown below.

<processor type="Coveo.SearchProvider.Processors.FetchPageContentProcessor, Coveo.SearchProviderBase">
  <inboundFilter hint="list:AddInboundFilter">
    <!-- inboundFilter configurations here -->
  </inboundFilter>
  <preAuthentication hint="list:AddPreAuthenticator" />
    <!-- preAuthentication configurations here -->
  </preAuthentication>
  <postProcessing hint="list:AddPostProcessing">
    <!-- postProcessing configurations here -->
  </postProcessing>
</processor>

If you change anything in the configuration, you must rebuild or reindex your items for the new settings to be applied.

Here are more details about the configuration options in each section.

The `<inboundFilter>` Section

This section lets you specify the Sitecore items that you want to provide an HTML representation for. Filtering can reduce the number of requests that log an error in your Sitecore logs.

Available Configurations:

<itemsWithLayout>

This filter specifies that only items that have a layout must be processed, eliminating many unnecessary requests for items that most likely don’t have any HTML content.

We recommend that you keep this filter at all times, but you can remove it in specific scenarios, such as when using Wildcard items, which don’t necessarily have a layout.
Custom Processor

You can implement your own processor using the Coveo.SearchProvider.Processors.FetchPageContent.Filters.IFetchPageContentInboundFilterProcessor interface.

The `<preAuthentication>` Section

This section lets you authenticate the request that will be sent to retrieve the HTML content.

Available Configurations:

FormsRequest

If your page requires authentication to get access to the page, the FormsRequest processor lets you authenticate the request like an end user would.

Here is an overview of what this processor does:
1. Takes the configuration and builds a POST request.
2. Sends the POST request to the login page.
3. Takes the response and stores its cookies.
4. Takes the cookies and assigns them to the HTTP request used to get the binary data.
FormsRequest configurations:
- The credentialsExpireIn attribute is used to keep the cookies for a period of time. In this example, it’s set to 5 minutes.
- The formsAuthConfiguration object contains attributes used to configure the POST request sent to authenticate the user.
  - formsAuthLoginPage: The URL of the login page.
  - formsAuthUserControl: The name attribute value of the input control used by the user to enter their username.
  - formsAuthPasswordControl: The name attribute value of the input control used by the user to enter their password.
  - formsAuthLoginCommand: The name attribute value of the submit control that the user clicks when logging in, followed by the value attribute value of the same submit control.
    
    Spaces must be replaced with the + symbol.
- The username is the Sitecore username used to authenticate the request.
- The password is the password used to authenticate the request.
EXAMPLE
You want to index content from your http://www.secured.com Sitecore website. You can access the authentication page of the website through http://www.secured.com/sitecore/login.

Inspecting this authentication page in your browser, you see the following markup:

The corresponding <preAuthentication> section configuration would look as follows:
<preAuthentication hint="list:AddPreAuthenticator"> <processor type="Coveo.SearchProvider.Processors.FetchPageContent.PreAuthenticators.FormsRequest, Coveo.SearchProviderBase" singleInstance="true"> <credentialsExpireIn>00:05:00</credentialsExpireIn> <formsAuthConfiguration type="Coveo.Framework.Configuration.FormsAuthConfiguration, Coveo.Framework"> <formsAuthLoginPage>http://www.secured.com/sitecore/login</formsAuthLoginPage> <formsAuthUserControl>UserName</formsAuthUserControl> <formsAuthPasswordControl>Password</formsAuthPasswordControl> <formsAuthLoginCommand>LogInBtn=Log+in</formsAuthLoginCommand> </formsAuthConfiguration> <username>sitecore\coveocrawler</username> <password>b</password> </processor> </preAuthentication>
When this processor is enabled, if you’re logging into the Sitecore default login page, you should see the following log during the indexing operation:
37148 14:27:28 INFO AUDIT (sitecore\coveocrawler): Login

AddSingleSignOnHeaders

This processor ensures that the HTTP request has the required headers to follow Single Sign-On redirections.

To configure it, you need to add a list of URLs that are Single Sign-On logins.

EXAMPLE

With the following configuration, if the HTTP request to get the binary data gets redirected to http://myssosite.local/login.aspx, the username mycustomdomain\unicorns and password ra1nb0ws are used to try to login in.

<preAuthentication hint="list:AddPreAuthenticator">
  <processor type="Coveo.SearchProvider.Processors.FetchPageContent.PreAuthenticators.AddSingleSignOnHeaders, Coveo.SearchProviderBase" singleInstance="true">
    <LoginUrls hint="list">
      <myExampleSite>http://myssosite.local/login.aspx</myExampleSite>
    </LoginUrls>
    <Username>mycustomdomain\unicorns</Username>
    <Password>ra1nb0ws</Password>
  </processor>
</preAuthentication>

Custom Processor

You can implement your own authentication processor by using the Coveo.SearchProvider.Processors.FetchPageContent.PreAuthenticators.IFetchPageContentPreAuthenticatorProcessor interface.

This is useful if you have a custom authentication process and need to add headers or set a specific cookie to allow the request to get the content.

The `<postProcessing>` Section

This section lets you process the content of the HTML page before sending it to the index, removing some sections that are useless in the index or that drive relevance down.

To remove HTML content using CSS selectors, you can also use an indexing pipeline extension (IPE) ( see Remove HTML Sections From Indexed Sitecore Items.

Post Processing comes at a performance cost. It requires the HTML byte array to be decoded, modified, and re-encoded.

Available Configurations:

CleanHtml

The CleanHtml processor is used to remove sections that are between two comments.

EXAMPLE

The following configuration removes the content between a BEGIN NOINDEX and a END NOINDEX comment.

<processor type="Coveo.SearchProvider.Processors.FetchPageContent.PostProcessing.CleanHtml, Coveo.SearchProviderBase">
  <startComment>BEGIN NOINDEX</startComment>
  <endComment>END NOINDEX</endComment>
</processor>

For example, given the following markup:

<head>
  <title>My Site</title>
</head>
<body>
  <!-- BEGIN NOINDEX -->
  <header>I don't want to index this header.</header>
  <!-- END NOINDEX -->
  <div>
      Some content.
  </div>
</body>

The result in the BinaryData field is:

<head>
  <title>My Site</title>
</head>
<body>
  <div>
      Some content.
  </div>
</body>

CleanHtmlWithSimpleSelectors

Coveo for Sitecore (February 12, 2021) Coveo for Sitecore (March 26, 2021)

The CleanHtmlWithSimpleSelectors processor had to be deprecated because it was incompatible with content that includes HTML tags with attributes that don’t have a value (e.g., <input type="text" name="lastname" disabled>). When a web page included tags like these, the processor threw an exception such as the following example and failed to remove any content from the page:

ManagedPoolThread #1 20:10:10 ERROR An error occurred while cleaning HTML of item {110D559F-DEA5-42EA-9C1C-8A5DF7E70EF9} at http://sitecore82u7/de-DE
Exception: System.Xml.XmlException
Message: 'class' is an unexpected token. The expected token is '='. Line 9, position 17.
Source: System.Xml
   at System.Xml.XmlTextReaderImpl.Throw(Exception e)
   at System.Xml.XmlTextReaderImpl.ParseAttributes()
   at System.Xml.XmlTextReaderImpl.ParseElement()
   at System.Xml.XmlTextReaderImpl.ParseElementContent()
   at System.Xml.Linq.XContainer.ReadContentFrom(XmlReader r)
   at System.Xml.Linq.XContainer.ReadContentFrom(XmlReader r, LoadOptions o)
   at System.Xml.Linq.XDocument.Load(XmlReader reader, LoadOptions options)
   at System.Xml.Linq.XDocument.Parse(String text, LoadOptions options)
   at Coveo.SearchProvider.Utils.HtmlCleanerWithSimpleSelectors.CleanHtmlContentWithSimpleSelectors(String p_HtmlContent, List`1 p_SimpleSelectors)
   at Coveo.SearchProvider.Processors.FetchPageContent.PostProcessing.CleanHtmlWithSimpleSelectors.Process(FetchPageContentHtmlPostProcessingArgs p_Args)

Custom Processor

You can implement your own HTML processing by using the Coveo.SearchProvider.Processors.FetchPageContent.PostProcessing.IFetchPageContentHtmlPostProcessingProcessor interface.

This is useful if you want to remove content in a different manner than that implementations provided by Coveo for Sitecore.

What's Next for Me?

Index Page Content With the FetchPageContentProcessor

Enabling the Processor

When Currently Not Indexing Rendered HTML

When Currently Using HtmlContentInBodyWithRequestsProcessor

Configuring the Processor

The <inboundFilter> Section

The <preAuthentication> Section

The <postProcessing> Section

The `<inboundFilter>` Section

The `<preAuthentication>` Section

The `<postProcessing>` Section