Add or Edit a SharePoint Online Source

Members of the Administrators and Content Managers built-in groups can include SharePoint Online content and make it searchable. In a Coveo-powered search interface, the source content is accessible to either everyone, the source creator only, or specific users as determined by source permissions (see Content Security). By default, a SharePoint Online source starts a refresh every hour and a rescan every week to retrieve SharePoint Online item changes (addition, modification, or deletion) (see Edit a Source Schedule). A source rescan or rebuild is necessary to capture deleted user profiles.

Following a refresh operation, deleted discussion lists are excluded from your Coveo Cloud SharePoint Online source content, but replies to the original discussion message will only be excluded following the next rescan operation. This is a known issue caused by a limitation of Microsoft SharePoint Online.

Source Features Summary

Features Supported Additional information
SharePoint Online version Latest cloud version  
Searchable content types Sites, sub-sites, user profiles, personal websites, lists, list items, list item attachments, document libraries, document sets, documents, web parts, and microblog posts and replies.
Content update Refresh

Rescan or rebuild is required to retrieve deleted user profiles.

Rescan  
Rebuild  
Content security options Determined by source permissions  
Source creator  
Everyone  

Requirements

SharePoint Online Account With Appropriate Roles and Permissions

When you want to include SharePoint Online content, you must create a specific SharePoint Online account that has access to the content you want to make searchable and that will be only used for the source. If you allow Coveo to retrieve your content through your personal account, you’ll need to also update the source access token each time you change the account password to prevent authentication errors (see Update Access Token).

  1. Access your Azure Portal with an administrator account.

  2. In Azure, create an account with the following roles:

    Role Justification
    Application Administrator

    This role allows the user to consent to give Coveo's Azure Active Directory Application the admin permissions it needs (see Azure Application Permissions).

    If you don't want the crawling account to have that role, you need to consent with a user that has the Global Admin role once before login with the crawling account (see Admin Consent).

    SharePoint Administrator

    This role is needed for the site URLs autodiscovery. This is used when you select All sites (see Content to Include).

    If you don't want the crawling account to have that role, you need to use Specific items (see Content to Include).

  3. Ensure that multi-factor authentication (MFA) is deactivated for this account, as it prevents Coveo Cloud from connecting to your SharePoint Online tenant with the following error: 'https://login.microsoftonline.com/common/oauth2/token' has failed. Http status: 400 - Bad Request.

  4. Access your SharePoint Online tenant with an account that has the SharePoint Administrator role, and then grant appropriate SharePoint Online permissions to the account you created before to ensure it has access to all the content that you want to include.

    The following table presents the minimal required permissions that the account must have to perform the specified action. If you specified sites to crawl and you did not grant the minimal permissions, the crawler will stop. If you selected “All sites”, it will skip sites that the crawling account can’t see.

    Action to perform Minimal required permission
    Content (without security indexing)

    We recommend that you be a site admin for every sites you want to crawl to avoid permission misconfiguration. If you don't want the crawling account to be a site admin, it requires the following permission levels on every sites (see Understanding permission levels in SharePoint):

    • Site permissions:

      • View Pages - View pages in a Web site.

      • Open - Allows users to open a Web site, list, or folder in order to access items inside that container.

    • List Permissions:

      • View Items - View items in lists and documents in document libraries.

      • Open Items - View the source of documents with server-side file handlers.

      • View Versions - View past versions of a list item or document.

    Content (with security indexing)

    Site admin for all SharePoint Online sites that need to be crawled (see Manage site admins in SharePoint Online).

    Personal site and user profile

    Owner of all personal sites (see Adding the Personal Sites Owner Permissions for SharePoint Online).

DNS Records Configuration for Office 365

  1. Log in to Office 365 admin center with an administrator account.

  2. In the navigation bar on the left, select Domains.

  3. In the Manage domains page:

    1. Under Domain Name, select your corporate domain (not company.onmicrosoft.com) check box.

    2. Next to the Action column, under the [domain name], click Domain settings.

  4. On the [domain name] page, in the DNS records section, take note of the DNS records.

  5. Configure these DNS records in your DNS host provider (see Create DNS records for Office 365 when you manage your DNS records).

  6. On the [domain name] page, in the DNS records section, click the Troubleshoot domain link to ensure that the DNS records were correctly configured.

Azure Application Permissions

A SharePoint Online source uses the OAuth 2.0 authorization protocol. To work with Microsoft APIs (CSOM and REST), Coveo Cloud must authenticate via an Azure Active Directory Application. Coveo Cloud obtains “delegated” permissions, i.e., when a user logs in, the Coveo Cloud Platform receives an access token referring to this specific user.

When you create a SharePoint Online source, an Azure application is created in your Azure tenant (see Understand user and admin consent), and you must grant permissions to this application. The access token is then limited to these permissions, which are necessary to successfully crawl SharePoint Online. All following access token permissions needs Admin Consent. As a result, for a user to authenticate through the Coveo Cloud Azure Active Directory application, they must have the Application administrator role, or a user with the Global Admin role must have given consent (see Admin Consent).

The permissions automatically granted to the application are the following:

Required permission Justification
Have full control of all sites (AllSites.FullControl)

Coveo Cloud requires this permission to retrieve permissions of crawled items. Microsoft doesn't offer enough granularity for Coveo to use a permission with fewer privileges.

Some API calls require Coveo to have the AllSites.Read permission to fetch list items, sites and sub-sites, and document content data, but since AllSites.FullControl is required too, AllSites.Read doesn't appear in the list of required permissions.

Read user profiles (User.Read.All)

Coveo Cloud requires this permission mainly to retrieve user profiles and index them as items if you select this option (see User profiles).

Read directory data (Directory.Read.All)

Coveo Cloud requires this permission to fetch:

  • The Directory Role and Directory Role Members (see List Members).

  • All users in Office 365, which is necessary to determine which users are in built-in groups such as Everyone (see List Users and Coveo Cloud V2 Management of Security Identities and Item Permissions).

    The Azure documentation shows that the least privileged permission to retrieve the list of users in a group is actually User.ReadBasic.All, but since Directory.Read.All is already required for other operations, User.ReadBasic.All doesn't appear in the list of required permissions.

Read all groups (Group.Read.All)

Coveo Cloud uses this permission to obtain the ID of a group (represents an Azure Active Directory (Azure AD) group, which can be an Office 365 group, or a security group), and then a list of the group members (see Get Group and List members).

  1. Follow Add or Edit a SharePoint Online Source steps 1 to 4 with a user that has the Global Admin role. This is the only acceptable role to consent (see Common consent scenarios).

  2. Check Consent of behalf of your organization.

  3. Click Accept.

  4. You’ll be redirected to the Add/Edit a SharePoint Online Source panel. Close it and do the steps in Add or Edit a SharePoint Online Source with your crawling account.

Add or Edit a SharePoint Online Source

  1. Ensure that your SharePoint Online instance meets the source requirements (see Requirements).

  2. If not already in the Add/Edit a SharePoint Online Source panel, access the panel:

    • To add a source:

      1. In the main menu, under Content, select Sources > Add source button > SharePoint > SharePoint Online.

      2. In the Sign in to SharePoint Online window that appears, enter your SharePoint Online tenant name, and then click Sign In.

        MyCompany

        You can also enter your full SharePoint Online tenant address.

        https://mycompany.sharepoint.com

      3. Enter the Email and Password of the limited administrator account that you created earlier and that has access to the desired SharePoint Online content, and then click Sign in (see SharePoint Online Account With Appropriate Roles and Permissions).

        Starting March 25, 2019, when you create two SharePoint Online sources retrieving content the same tenant, they share their security providers, which increases the speed of the security identities refresh operation (see Refresh a Security Identity Provider). You must however use the same limited administrator credentials for both sources.

      4. Click Accept to grant the required permissions to the Coveo Cloud application.

      OR

    • To edit a source, in the main menu, under Content, select Sources, and then double-click the desired source.

  3. In the Configuration tab, enter appropriate values for the available parameters:

    • Source name

      A descriptive name for your source under 255 characters (not already in use for another source in this organization).

      SharePoint-Online-Intranet

    • Character optical recognition (OCR)

      Check this box if you want Coveo Cloud to extract text from image files or PDF files containing images (see Enable Optical Character Recognition). OCR-extracted text is processed as item data, meaning that it’s fully searchable and will appear in the item Quick View (see Search Result Quick View).

      Since the OCR feature is available at an extra charge, you must first contact Coveo Sales to add this feature to your organization license. You can then enable it for your source.

    • Index

      When adding a source, if you have more than one logical (non-Elasticsearch) index in your organization, select the index in which the retrieved content will be stored (see Leverage Many Coveo Indexes). If your organization only has one index, this drop-down menu isn’t visible and you have no decision to make.

      • To add a source storing content in an index different than default, you need the View access level on the Logical Index domain (see Privilege Management and Logical Indexes Domain).

      • Once the source is added, you can’t switch to a different index.

  4. In the Content to Include section, select the content to make searchable. Your options are:

    • All sites

      All sites that the crawling account is allowed to access will be searchable (see SharePoint Online Account With Appropriate Roles and Permissions).

    • Specific items

      If you choose to make only certain items searchable, see SharePoint Online Account With Appropriate Roles and Permissions to set the required permissions on the crawling account. In the URL box, enter URLs corresponding to the desired sites, lists, websites, and subwebsites. Each URL must include the protocol and tenant name.

      • For a specific site: https://site:8080/sites/support

      • For a specific website: https://site:8080/sites/support/subsite

      • For a specific list: https://site:8080/sites/support/lists/contacts/allItems.aspx

        A specific folder in a list isn’t supported.

    • None

      If you select this option, see Additional content.

  5. Under Additional content, specify which content you want to make searchable. Your options are:

    • User profiles

      Select to include SharePoint Online user profiles.

      To prevent performance issues, we recommend that you create a separate source for user profiles only.

    • Personal sites

      Select to include SharePoint Online personal sites.

      To prevent performance issues, we recommend that you create a separate source for personal sites only.

    • Folders

      Select to include list folders and document sets.

    • Unapproved items

      Select to include unapproved items from lists where moderation is activated. See Indexing Unapproved Items for details. In lists where moderation is deactivated, Coveo Cloud indexes the latest version of an item, regardless of whether this item is a draft or not. In such a case, this option doesn’t apply.

  6. In the Content Security tab, select who will be able to access the source items through a Coveo-powered search interface. For details on this parameter, see Content Security.

  7. In the Access tab, determine whether each group and API key can view or edit the source configuration (see Understanding Resource Access):

    1. In the Access Level column, select View or Edit for each available group.

    2. On the left-hand side of the tab, if available, click Groups or API Keys to switch lists.

    If you remove the Edit access level from all the groups of which you’re a member, you won’t be able to edit the source again after saving. Only administrators and members of other groups that have Edit access on this resource will be able to do so. To keep your ability to edit this resource, you must grant the Edit access level to at least one of your groups.

  8. Optionally, consider editing or adding mappings (see Adding and Managing Source Mappings).

    You can only manage mapping rules once you build the source (see Refresh, Rescan, or Rebuild Sources).

  9. Complete your source addition or edition:

    • Click Add Source/Save when you want to save your source configuration changes without starting a build/rebuild, such as when you know you want to do other changes soon.

      On the Sources page, you must click Start initial build or Start required rebuild in the source Status column to add the source content or make your changes effective, respectively.

      OR

    • Click Add and Build Source/Save and Rebuild Source when you’re done editing the source and want to make changes effective.

      Back on the Sources page, you can review the progress of your SharePoint Online source addition or modification (see Adding and Managing Sources).

    Once the source is built or rebuilt, you can review its content in the Content Browser (see Inspect Items With the Content Browser).

    If you selected Specific Items and User Profiles in the Content to Include section, some additional items will appear in the Content Browser. To retrieve user profiles, Coveo must dig through your SharePoint Online instance, including your My Site host site collection and the documents it contains. The items it encounters in the process are retrieved as well and therefore appear in the Content Browser.

What’s Next?

Review your source update schedule and optionally change it so that it better fits your needs (see Edit a Source Schedule). By default, your content is refreshed every hour and rescanned every week.

Recommended Articles