Add a SharePoint Server source

This is for:

Developer System Administrator

Members with the required privileges can index SharePoint on-premises content and make it searchable.

To retrieve SharePoint Online content, create a SharePoint Online source instead.

Important

To help reduce the risks of SharePoint vulnerability exploitation, always use the latest supported version of SharePoint Server and apply the latest Microsoft security updates.
Tip
Leading practice

The number of items that a source processes per hour (crawling speed) depends on various factors, such as network bandwidth and source configuration. See About crawling speed for information on what can impact crawling speed, as well as possible solutions.

Source key characteristics

The following table presents the main characteristics of a SharePoint Server source.

Features Supported Additional information

SharePoint version

Subscription Edition, 2019, 2016, 2013, and Foundation 2013

Indexable content

Sites, sub-sites, public user profiles[1], personal websites[1], lists, list items, list item attachments, document libraries, document sets, documents, web parts, and microblog posts and replies.

Content update operations

refresh

check

Takes place every six hours by default. A rescan or rebuild is required to take account of deleted user profiles.

rescan

check

Takes place every week by default.

rebuild

check

Content security options

Same users and groups as in your content system

check

On-premises Active Directory permission systems aren’t supported with SharePoint Server sources of the On-Premises type. However, if you use the Crawling Module Active Directory is supported.

Specific users and groups

check

Everyone

check

Metadata indexing for search

Automatic mapping of metadata to fields that have the same name

This setting is disabled by default and not recommended for this source type.

Automatically indexed metadata

Examples of auto-populated default or standard source fields (no user-defined metadata required):
 

• author

• clickableuri

• date

• filename

• filetype

• indexeddate

• language (auto-detected from item content)

• searchablemeta [2]

• title
 

After a content update, inspect your item field values in the Content Browser.

Extracted but not indexed metadata

The SharePoint Server source extracts some of the site, list, list item, and file-level metadata that the SharePoint APIs make available.
 

After a rebuild, review the View and map metadata subpage for the list of indexed metadata, and then index additional metadata.

Custom metadata extraction

Add columns to your lists and libraries. The SharePoint Server source automatically extracts the metadata in these columns during content updates.

Prerequisites

Before you create a SharePoint Server source, you must:

  1. Ensure that your license allows you to create this source and that your organization hasn’t reached its source quota. You can review your quotas on the License & Usage (platform-ca | platform-eu | platform-au) page, along with your usage.

  2. Create a SharePoint account that meets the permission requirements.

SharePoint account permissions

When you want to include SharePoint content, you must create a specific SharePoint account to be used by the source only. Otherwise, you must also change the source Password value each time the account password changes to prevent authentication errors.

  1. Access your SharePoint tenant with an administrator account.

  2. On your SharePoint tenant:

    1. Select or create a user account for the source to use when retrieving your SharePoint content. See the following table to identify the required type of user for your web application enabled authentication.

      SharePoint environment SharePoint web application enabled authentication User type User format

      Classic

      Windows

      Windows account

      domain\username

      or

      username@domain.com

      Claims

      Windows

      Windows account

      Okta

      Okta SSO

      username@domain.com

    2. Grant appropriate SharePoint permissions to the SharePoint account to ensure it has access to the content that you want to make searchable.

      The following table presents the minimal required permissions that the source account must have to perform specific actions.

      Action to perform Minimal required permission

      Content and security indexing, source refresh, and site collection discovery

      Full Read policy for each web application to make searchable.

      Personal site, public user profile, and social tags indexing

      Note

      When including personal sites or public user profiles, the account used as source credentials must not have a personal site on the SharePoint Server being included to prevent failures when attempting to retrieve the list of personal sites.

Add a SharePoint Server source

A SharePoint Server source indexes on-premises (server) content. To retrieve cloud content instead, see Add a SharePoint Online source.

Tip
Leading practice

It’s best to create or edit your source in your sandbox organization first. Once you’ve confirmed that it indexes the desired content, you can copy your source configuration to your production organization, either with a snapshot or manually. Source re-authentication with a separate crawling account may be required in the production organization.

See About non-production organizations for more information and best practices regarding sandbox organizations.

Follow these steps to add a SharePoint Server source using the desired content retrieval method:

  1. Make sure that your organization meets the prerequisites before adding this source.

  2. On the Sources (platform-ca | platform-eu | platform-au) page, click Add source.

  3. In the Add a source of content panel, select the On-Premises (server) or the Crawling Module (crawlingmodule) tab, depending on your content retrieval context. With the latter, you must install the Crawling Module to make your source operational.

  4. Click the SharePoint Server tile.

  5. Configure your source.

"Configuration" tab

In the Add a SharePoint Server Source panel, the Configuration tab is selected by default. It contains the following subtabs.

"Identification" subtab

Name

Enter a name for your source.

Tip
Leading practice

A source name can’t be modified once it’s saved, therefore be sure to use a short and descriptive name, using letters, numbers, hyphens (-), and underscores (_). Avoid spaces and other special characters.
Project

Use the Project selector to associate your source with one or more Coveo projects.

"Authentication" subtab

Select the authentication type to use. The available options are Windows and Okta.

Depending on the selected option, specify the following parameters.

Username and password

The username and password of a dedicated SharePoint administrator account that has access to the content to include, or if using Okta, the username of an Okta administrator account. See Source credentials leading practices.

Okta realm

(Okta only) The SharePoint trusted identity provider realm provided in your Okta application configuration.

Example

urn:okta:sharepoint:exknuavz9hbOItwsS8e7

Okta sign-in URL

(Okta only) The URL to which users should be redirected to authenticate with Okta.

Example

https://dev-782461.oktapreview.com/app/appname/sso/wsfed/passive

"Content to index" subtab

URL

Enter one or more URLs corresponding to the desired site collection, lists, websites, and subsites to make searchable. Each URL must include the protocol and tenant name.

Note

A specific folder in a list isn’t supported.
Examples

  • For a specific web application: https://site:8080/

  • For a specific site collection: https://site:8080/sites/support

  • For a specific website: https://site:8080/sites/support/subsite

  • For a specific list: https://site:8080/sites/support/lists/contacts/allItems.aspx

Scope

Select the option for the content type that you want to include in relation with the source URL you specified. By default, Web application is selected.

Value Content to make searchable

Web application

All site collections of the specified web application.

Site collection

All web sites of the specified site collection.

Web and sub webs

Only the specified web site and its sub webs (also known as subsites).

List

Only the specified list or document library.
Content to include

  • User profiles: Check this box to index public SharePoint user profiles.

    Note

    This option is unavailable if you selected Okta as the authentication type.

  • Personal sites: When the Scope is Web application, check this box to include SharePoint personal sites.

Exclusions and inclusions

Add exclusion and inclusion rules to crawl only specific items based on their URL.

Exclusions and inclusions user interface screenshot | Coveo

The following diagram illustrates how the SharePoint Server crawler applies the exclusion and inclusion rules. This flow applies to all items, including the starting URLs. You must therefore pay attention to not filter out your starting URLs.

Crawling workflow diagram | Coveo
Tip
About the "Include all non-excluded items" option
Crawling flow with the all-inclusive inclusion rule | Coveo

The Include all non-excluded items option automatically adds an "include all" inclusion rule in the background. This ensures that all starting URLs meet the Does URL match at least one inclusion rule? condition and that all non-excluded items get crawled.

The following are common configuration patterns:

  • When you don’t want to exclude content on a URL basis, you don’t add any exclusion rule and you use the default Include all non-excluded items inclusion option.

  • When you want to exclude content on a URL basis, you add exclusion rules for the content to exclude and you use the default Include all non-excluded items inclusion option.

You can use any of the six types of rules:

  • is and a URL that includes the protocol. For example, https://myfood.com/.

  • contains and a string found in the URL. For example, recipes.

  • begins with and a string found at the beginning of the URL and which includes the protocol. For example, https://myfood.

  • ends with and a string found at the end of the URL. For example, .pdf.

  • matches wildcard rule and a wildcard expression that matches the whole URL. For example, https://myfood.com/recipes*.

  • matches regex rule and a regex rule that matches the whole URL. For example, ^.*(company-(dev|staging)).*html.?$.

    Tip

    When using regex rules, make sure they match the desired URLs with a testing tool such as Regex101.
Additional content

  • Reindex all child items on UpdateShallow: Enabling this will reindex all child items after any change, ensuring data integrity but slowing source refresh time.

    Example

    You change your SharePoint site name. In the metadata of the child items, the site name appears under spsitename. If this option isn’t enabled, the children aren’t reindexed and keep an outdated spsitename until the next source rescan or rebuild. However, if this option is enabled, the children are updated along with the parent SharePoint site item.

"Crawling Module" subtab

If your source is a Crawling Module source, and if you have more than one Crawling Module linked to your organization, select the one with which you want to pair your source. If you change the Crawling Module instance paired with your source, a successful rebuild is required for your change to apply.

"Items" tab

On the Items tab, you can specify how the source handles items based on their file type or content type.

File types

File types let you define how the source handles items based on their file extension or content type. For each file type, you can specify whether to index the item content and metadata, only the item metadata, or neither.

You should fine-tune the file type configurations with the objective of indexing only the content that’s relevant to your users.

Example

Your repository contains .pdf files, but you don’t want them to appear in search results. You click Extensions and then, for the .pdf extension, you change the Action by default and Action on error values to Ignore item.

For more details about this feature, see File type handling.

Content and images

If you want Coveo to extract text from image files or PDF files containing images, enable the appropriate option. The extracted text is processed as item data, meaning that it’s fully searchable and will appear in the item Quick view.

Note

When OCR is enabled, ensure the source’s relevant file type configurations index the item content. Indexing the item’s metadata only or ignoring the item will prevent OCR from being applied.

See Enable optical character recognition for details on this feature.

"Content security" tab

Select who will be able to access the source items through a Coveo-powered search interface. For details on the content security options, see Content security.

Important

Coveo for SharePoint Server only partially supports SharePoint permissions. If you select the Same users and groups as in your content system option, keep in mind that Coveo supports SharePoint base system roles, access control lists (ACLs), and domain separation.

On-premises Active Directory permission systems aren’t supported with SharePoint Server sources of the On-Premises type. However, if you use the Crawling Module, Active Directory is supported.

When the Same users and groups as in your content system option is selected and the Permission system is Active Directory checkbox is checked, provide the Active Directory details.

Active Directory details

Username and password

The credentials to access your Active Directory.

Email attributes

By default, Coveo retrieves the email address associated to each security identity from the mail attribute. Optionally, you can specify additional or different attributes to check. Should an attribute contain more than one value, Coveo uses the first one.

Options

Enable Transport Layer Security (TLS)

Select this option to use a TLS protocol to retrieve your security identities. If you do, we strongly recommend selecting StartTLS if you can. Since LDAPS is a much older protocol, you should only select this value if StartTLS is incompatible with your environment.

Expand well-knowns

Select this option if you want the users that are included in your Active Directory well-known security identifiers to be granted access to the indexed content. Supported well-known SIDs are: Everyone, Authenticated Users, Domain Admins, Domain Users, and Anonymous Users.

When enabling this option, you can expect an increase in the duration of the security identity provider refresh operation.

Tip
Leading practice

If your entire content is secured with Everyone or Authenticated users, we recommend selecting the Everyone content security option instead. The result will be the same, that is, all users will be able to access the database content through your search interface, and Coveo’s update operations will be more efficient.

Expand trusted domains

Select this option to have Coveo connect to your root domain to get the security identities of your other domains through the root domain.

If your environment contains more than one domain, you can establish a bidirectional or outbound cross-link relationship between the root domain of your Crawling Module server and your additional domains. When you do so, these domains trust your root domain, and Coveo can get their security identities through this root domain.

When enabling this option, you can expect an increase in the duration of the security identity provider refresh operation. Moreover, if a linked domain is unreachable, Coveo stops the security identity provider refresh operation.

Important

When using the Everyone content security option, see Safely apply content filtering for information on how to ensure that your source content is safely filtered and only accessible by intended users.

"Access" tab

On the Access tab, specify whether each group (and API key, if applicable) in your Coveo organization can view or edit the current source.

For example, when creating a new source, you could decide that members of Group A can edit its configuration, while Group B can only view it.

For more information, see Custom access level.

Build the source

  1. Finish adding or editing your source:

    • When you’re done editing the source and want to make your changes effective, click Add and build source/Save and rebuild source.

    • When you want to save your source configuration changes without starting a build/rebuild, such as when you know you want to make other changes soon, click Add source/Save. On the Sources (platform-ca | platform-eu | platform-au) page, click Launch build or Launch rebuild when you’re ready to make your changes effective and index your content.

  2. On the Sources (platform-ca | platform-eu | platform-au) page, follow the progress of your source addition or modification.

  3. Once the source is built or rebuilt, review its content in the Content Browser.

Index metadata

To use metadata values in search interface facets or result templates, the metadata must be mapped to fields. Coveo automatically maps only a subset of the metadata it extracts. You must map any additional metadata to fields manually.

Note

Not clear on the purpose of indexing metadata? Watch this video.

  1. On the Sources (platform-ca | platform-eu | platform-au) page, click your source, and then click More > View and map metadata in the Action bar.

  2. Review the default metadata that your source is extracting from your content.

  3. Map any currently not indexed metadata that you want to use in facets or result templates to fields.

    Important

    For certain types of content, the source extracts a set of item keys and values, aggregates this information into the coveo_AllMetadata metadata, and indexes this metadata in the searchablemeta field. The keys and values that make up the coveo_AllMetadata metadata are also available as separate metadata. No parsing on your part is required. Just locate the name of the metadata you need on the View and map metadata subpage and proceed, as explained below, to index the metadata in a field.

    1. Click the metadata and then, at the top right, click Add to index.

    2. In the Apply a mapping on all item types of a source panel, select the field you want to map the metadata to, or add a new field if none of the existing fields are appropriate.

      Note

      For advanced mapping configurations, like applying a mapping to a specific item type, see Manage mappings.

    3. Click Apply mapping.

  4. Return to the Sources (platform-ca | platform-eu | platform-au) page.

  5. To reindex your source with your new mappings, click your source, and then click More > Rebuild in the Action bar.

  6. Once the source is rebuilt, review your item field values. They should now include the values of the metadata you selected to index.

    1. On the Sources (platform-ca | platform-eu | platform-au) page, click your source, and then click More > Open in Content Browser in the Action bar.

    2. Select the card of the item for which you want to inspect properties, and then click Properties in the Action bar.

    3. In the panel that appears, select the Fields tab.

Additional adjustments

  1. If your source retrieves your content through the Crawling Module and if access to its content is secured with an Active Directory security identity provider, you must edit the JSON configuration of the security identity provider associated to this source to provide additional information. In the security identity provider JSON configuration, add the following code snippet, in which you replace <HOSTNAME> with either your Active Directory server IP address or domain name to use to connect to your Active Directory.

     "Hostname": {
   "value": "<HOSTNAME>"
 }

  2. Moreover, if you enabled the Enable Transport Layer Security (TLS) option in the Content security tab, ensure your security certificates are public and installed on the Crawling Module server.

Safely apply content filtering

The best way to ensure that your indexed content is seen only by the intended users is to enforce content security by selecting the Same users and groups as in your content system option. Should this option be unavailable, select Specific users and groups instead.

However, if you need to configure your source so that the indexed source content is accessible to Everyone, you should adhere to the following leading practices. These practices ensure that your source content is safely filtered and only accessible by the appropriate users:

Following the above leading practices results in a workflow whereby the user query is authenticated server side via a search token that enforces the search hub from which the query originates. Therefore, the query can’t be modified by users or client-side code. The query then passes through a specific query pipeline based on a search hub condition, and the query results are filtered using the filter rules.

Configure query filters

Filter rules allow you to enter hidden query expressions to be added to all queries going through a given query pipeline. They’re typically used to add a field-based expression to the constant query expression (cq).

Example

You apply the @objectType=="Solution" query filter to the pipeline to which the traffic of your public support portal is directed. As a result, the @objectType=="Solution" query expression is added to any query sent via this support portal.

Therefore, if a user types Speedbit watch wristband in the search box, the items returned are those that match these keywords and whose objectType has the Solution value. Items matching these keywords but having a different objectType value aren’t returned in the user’s search results.

To learn how to configure query pipeline filter rules, see Manage filter rules.

Note

You can also enforce a filter expression directly in the search token.

Use condition-based query pipeline routing

The most recommended and flexible query pipeline routing mechanism is condition-based routing.

When using this routing mechanism, you ensure that search requests are routed to a specific query pipeline according to the search interface from which they originate, and the authentication is done server side.

To accomplish this:

  1. Apply a condition to a query pipeline based on a search hub value, such as Search Hub is Community Search or Search Hub is Agent Panel. This condition ensures that all queries that originate from a specific search hub go through that query pipeline.

  2. Authenticate user queries via a search token that’s generated server side and that contains the search hub parameter that you specified in the query pipeline.

Configure the search token

When using query filters to secure content, the safest way to enforce content security is to authenticate user queries using a search token that’s generated server side. For instance, when using this approach, you can enforce a search hub value in the search token. This makes every authenticated request that originates from a component use the specified search hub, and therefore be routed to the proper query pipeline. Because this configuration is stored server side and encrypted in the search token, it can’t be modified by users or client-side code.

Implementing search token authentication requires you to add server side logic to your web site or application. Therefore, the actual implementation details will vary from one project to another.

The following procedure provides general guidelines:

Note

If you’re using the Coveo In-Product Experience (IPX) feature, see Implement advanced search token authentication.

  1. Authenticate the user.

  2. Call a service exposed through Coveo to request a search token for the authenticated user.

  3. Specify the userIDs for the search token, and enforce a searchHub parameter in the search token.

Note

You can specify other parameters in the search token, such as a query filter.

For more information and examples, see Search token authentication.

Required privileges

You can assign privileges to allow access to specific tools in the Coveo Administration Console. The following table indicates the privileges required to view or edit elements of the Sources (platform-ca | platform-eu | platform-au) page and associated panels. See Manage privileges and Privilege reference for more information.

Actions Service Domain Required access level

View sources, view source update schedules, and subscribe to source notifications

Content

Fields

View

Sources

Organization

Organization

Edit sources, edit source update schedules, and edit source mappings

Organization

Organization

View

Content

Fields

Edit

Sources

View and map metadata

Content

Source metadata

View

Fields

Organization

Organization

Content

Sources

Edit
Notes

  • The Edit all privilege isn’t required to create sources. When granting privileges for the Sources domain, you can grant a group or API key the View all or Custom access level, instead of Edit all, and then select the Can create checkbox to allow users to create sources. See Can create ability dependence for more information.

  • Granting the Edit access level on a domain also implicitly grants View on that domain.

What’s next?

1. Not available in Microsoft SharePoint Foundation.
2. The keys and values indexed in the searchablemeta field are also available as separate indexable metadata. You don’t need to parse them. Proceed as explained in the Indexing metadata section to index the pieces you need.