Add or edit a source

This is for:

Developer System Administrator

A source is a virtual container which holds all the items related to a specific system (for example, YouTube, Web, SharePoint, Salesforce). The Sources (platform-ca | platform-eu | platform-au) page of the Coveo Administration Console lets you create and manage sources as needed.

When you add a source, Coveo connects to your repository and indexes its content. Depending on its selected content security option, the source content is accessible to one or more end users through a search interface. If the selected option is Same users and groups as in your content system, only the end users who have the appropriate permissions will be able to view the search results.

Sources Page
Tip
Leading practice

The number of items that a source processes per hour (crawling speed) depends on various factors, such as network bandwidth and source configuration. See About crawling speed for information on what can impact crawling speed, as well as possible solutions.

Add a source

Tip
Leading practice

It’s best to create or edit your source in your sandbox organization first. Once you’ve confirmed that it indexes the desired content, you can copy your source configuration to your production organization, either with a snapshot or manually.

See About non-production organizations for more information and best practices regarding sandbox organizations.

  1. On the Sources (platform-ca | platform-eu | platform-au) page, click Add source.

  2. In the Add a source of content panel that appears, select the type of source that you want to create.[1]

  3. See the appropriate article for details on how to create each source.

  4. Review your indexed metadata and index more.

Duplicate a source

If you intend to create two sources with a similar configuration, you should consider creating a source and then duplicating it. The source duplication feature can save you time, especially if the source configuration involves many parameters. Once your first source is duplicated, you can then edit the desired parameters in the source copy.

  1. On the Sources (platform-ca | platform-eu | platform-au) page, click the source you want to duplicate, and then click More > Duplicate in the Action bar.

  1. In the Duplicate a Source panel that appears, enter a Source name for the copy, and then click Duplicate Source.

  2. Back on the Sources (platform-ca | platform-eu | platform-au) page, once your new source has been added, edit it to make the desired changes.

  3. Build your new source.

Edit a source

Tip
Leading practice

It’s best to create or edit your source in your sandbox organization first. Once you’ve confirmed that it indexes the desired content, you can copy your source configuration to your production organization, either with a snapshot or manually.

See About non-production organizations for more information and best practices regarding sandbox organizations.

  1. On the Sources (platform-ca | platform-eu | platform-au) page, click the desired source, and then click Edit in the Action bar. You can also view and edit the source configuration in JSON format. That format lists all of the source parameters in key-value pairs, including the mappings.

  2. In the panel that appears, make the desired changes. See the appropriate article for details on how to edit each source.

  3. When you’re done:

    • If the changes impact source items (for example, you changed the source URL) or you want the changes to be effective as soon as possible, click Save and Rebuild.

    • If your changes don’t impact source items (for example, you updated source credentials) or if you plan to make additional changes in the future, click Save. Then, under the source Status column, you can click Launch rebuild when you’re ready or wait for the next scheduled source rebuild to make the changes effective.

Note

Some parameters such as the source name can’t be edited. If you want to change such a parameter, you must delete your source and add it again. Keep in mind that the process can take a significant amount of time, as Coveo has to reindex your entire content.

Delete a source

Deleting a source permanently erases the source configuration and its indexed content.

  1. On the Sources (platform-ca | platform-eu | platform-au) page, click the source that you want to delete, and then click More > Delete in the Action bar.

  2. Click Delete to confirm.

Notes

  • Deleting items is significantly faster than adding items.

  • The bigger the source, the longer it takes for the index to delete the source content.

Update an access token

Sources such as Salesforce and Zendesk use OAuth, a protocol that grants access to external applications without exposing the user’s real credentials. OAuth has access tokens that need to be updated manually when expired.

Note

The following procedure describes how to update the access token for Salesforce, Zendesk, and Microsoft Dynamics 365 sources. For SharePoint Online sources see Update a SharePoint Online access token. For all other OAuth source types, see the corresponding source configuration article for information on how to authenticate the source.

  1. On the Sources (platform-ca | platform-eu | platform-au) page, click the OAuth source, and then click More > Update access token in the Action bar.

  2. In the Update Access Token confirmation dialog:

    1. If this option is available, check the Also update the token in the source identity security provider configuration box.

      Note

      Before you update the access token used by the source identity security provider, ensure that no source configurations will become invalid.

    2. Click Update.

  3. On the page that appears, enter the email address and password of the account that you created for the source to index your content, and then authorize the update by clicking Authorize, Allow, or Log In, depending on the source system.

Note

You can always revoke this authorization (for example, see Manage OAuth Access for Your Connected Apps for Salesforce accounts and Revoke third-party access to your account information for Google accounts). However, if you do so, Coveo would no longer be able to retrieve the latest content changes and would show outdated search results.

Source management tools

On the Sources page, you can access various tools that let you manage different aspects of a source. Unless otherwise specified, these tools are accessible through the More option in the Action bar.

View the metadata of a source

The View Metadata subpage displays the metadata of your source items in a user-friendly format.

You can use this tool to:

  • Explore the metadata of your source items, such as the title, body, date, and author.

  • Verify that your source items have the correct metadata and mapping rules.

  • Add missing metadata to your index.

See View the metadata of a source for details on this feature.

Refresh, rescan, or rebuild sources

You can manually launch an update operation (refresh, rescan, or rebuild) to ensure that the content displayed in your search interface reflects your actual content. See Refresh, rescan, and rebuild for details.

  1. On the Sources (platform-ca | platform-eu | platform-au) page, click the source that you want to update.

  2. Click More, and then select the desired update operation.

In the Status column, you can monitor the progress of the update operation. You can get more information on source updates from the Activity panel.

Pause a source update

Most sources support pausing an ongoing rescan or rebuild. This can be done on the Sources (platform-ca | platform-eu | platform-au) page by selecting the source, and then Pause Rescan/Rebuild in the More menu. A button then appears in the Action bar to let you resume the update operation.

Changes in the source configuration or source-related resources (for example, mappings, fields) that occur while an update is paused are taken into account when the operation is resumed.

A source update can also be paused when there’s an issue with the source or with Coveo. Once the update is either resumed and completed, or canceled, your source proceeds with any pending updates. The update operation is automatically canceled after 7 days.

Review pending updates

A source update operation is pending when it can’t be executed yet due to another operation already taking place. For example, if you launch a rebuild while a scheduled refresh is ongoing, your rebuild will only take place once the refresh is completed.

You can see pending updates either in the Activity panel or the Activity Browser (platform-ca | platform-eu | platform-au).

Copy the ID of a source

When working with the Coveo Platform API, you may need the ID of a source to conduct certain operations.

On the Sources (platform-ca | platform-eu | platform-au) page, click the desired source, and then, in the More menu, select Copy source ID to clipboard.

Browse the content of a source

You can review the content of a source using the Content Browser.

  1. On the Sources (platform-ca | platform-eu | platform-au) page, click the source that you want to inspect, and then click More > Open in Content Browser in the Action bar.

    On the Content Browser (platform-ca | platform-eu | platform-au) page, the source you selected is automatically selected in the Source facet so that only the corresponding items appear in the search results.

  2. Use the Content Browser features to inspect your source content.

Review the activity of a source

As part of your duties, you may need to review activities related to sources for investigation or troubleshooting purposes. To do so, on the Sources (platform-ca | platform-eu | platform-au) page, select a source, and then click Activity in the Action bar.

See Review resource activity for details on activities and alternative ways to access this information.

Expanding a source update row reveals additional information on the operation, including:

  • Items added: Items indexed for the first time.

  • Items unchanged: Items that were already in the index and that didn’t change since the last source update.

  • Items processed: Items retrieved by the crawler. If they meet the source filter criteria, they’re indexed. If not, they’re filtered out.

  • Items filtered: Items retrieved by the crawler that didn’t meet the source filter criteria and therefore weren’t indexed.

  • Items updated: Items that have already been indexed in the past and that have been updated in your content system since last source update.

Download source update logs

Should you need more information about an ongoing or completed source update operation, you can download a JSON log of the desired activity.

An update log contains nonsensitive information about the update process. This is information related to the authentication process, content expansion, and item filtering, among other things. It also contains any warning or error that hinders part or all the update operation.

To download an update log

  1. On the Sources (platform-ca | platform-eu | platform-au) page, click the desired resource, and then click Activity in the Action bar.

  2. In the Activity panel that opens, click the desired activity, and then click Download Logs in the Action bar. The downloaded JSON file is named after the unique operation ID representing the selected activity.

See Review resource activity for alternative ways to access this information.

Manage source notification subscriptions

You can subscribe to source notifications to be informed when certain activities are triggered by a specific source.

  1. On the Sources (platform-ca | platform-eu | platform-au) page, click the desired source.

  2. In the Action bar, click More, and then select one of the following:

    • Subscribe to notifications

    • Edit subscription to notifications (to access the subscription management panel)

    • Unsubscribe from notifications

Leading practices

  • Use the appropriate source connector and content retrieval method.

  • While you initially build and tune a source configuration, consider:

  • Creating a source doesn’t grant you additional privileges with respect to that source. Only an organization member with the required privileges can edit and delete a given source.

  • While the best practice is to configure well-thought out source update schedules, you can also start a source update (refresh, rescan, or rebuild) manually. This is useful when you want to ensure that the content is up to date following a change in the source.

    Examples

    • Your coworker just imported of many articles into your knowledge base system. You manually start a rescan of your knowledge base source to immediately make these imported articles searchable.

    • You edit a field option and want this option to apply immediately to all items in the source. You manually launch a rebuild operation.

  • Clicking Save when editing a source defers the source rebuild operation, which is necessary to apply changes to the indexed items. If you’re sure that your changes won’t affect the indexed items in any way, such as changes relative to the source credentials, you may click Save to avoid an irrelevant rebuild operation. Because a source may take weeks to rebuild with several million items, you can avoid adding stress to Coveo and system server resources by choosing the Save option whenever possible. However, if you made changes that will affect the indexed items, we recommend that you use the Save and Rebuild option instead.

  • Delete unused sources to free up resources. Also, deleting a source during off-peak hours is faster since the index has less operations to manage simultaneously (for example, fewer queries are performed).

  • Avoid repeatedly creating and deleting a source, particularly one that contains many items, since the deletion process monopolizes a significant amount of resources. You should instead refresh or rebuild it.

  • Schedule source updates.

  • Turn off scheduled source updates for unused or test sources.

  • Should you need to rename your source, you would have to duplicate it since a source name can’t be changed. Depending on your source size, this process may take several hours, as your new source would have to build entirely.

  • When there are many sources listed on the page, in the Action bar, click the Status dropdown menu, and then select Last operation in error to display those whose last update operation failed. To further filter the list of sources in error, you may also choose to search for a specific source.

Reference

The Sources (platform-ca | platform-eu | platform-au) page is a table listing the sources defined in your Coveo organization.

  • The Type column indicates:

  • The Name column indicates the name that was entered when creating the source. A developer may need a source name when configuring the scope of a search interface.

  • The Status column indicates the current state of the source. Depending on the state of the source, the column may also include information regarding the last operation or provide an action to take.

  • The Content column indicates:

    • The number of items indexed in the source.

      Note

      The number of items in the Content column can be different from the number of processed items reported in the Status column.

      Examples:

      • Removed items

        Following a source activity completion, there were 1000 processed items in the Status column, but 10 items were deleted, 2 were added, the rest were updated or didn’t change. In the Content column, there are 992 items.

      • Attachments

        A source includes some items on which there are attachments. An item that has 3 attachments, counts as 1 processed item in the Status column, but counts as 4 items (1 parent item and 3 attachment items) in the Content column.

    • The source size.

Required privileges

The following table indicates the privileges required for members to view or edit elements of the Sources (platform-ca | platform-eu | platform-au) page and associated panels (see Manage privileges and Privilege reference).

Note

The Edit all privilege isn’t required to create sources. When granting privileges for the Sources domain, you can grant a group or API key the View all or Custom access level, instead of Edit all, and then select the Can Create checkbox to allow users to create sources. See Can Create ability dependence for more information.
Actions Service Domain Required access level

View sources, view source update schedules, and subscribe to source notifications

Content

Fields

View

Sources

Organization

Organization

Edit sources, edit source update schedules, and view the View Metadata subpage

Content

Fields

Edit

Sources

Content

Source metadata

View

Organization

Organization

Moreover, the following table lists privileges you can grant in addition to the required ones in the previous tables. These privileges aren’t required to use the basic features of the Sources page, but they’re required to perform specific actions.

Action Service Domain Required access level

When creating or editing a source, select the Crawling Module instance to pair with this source

Content

Crawling Module

View

Manage the Extensions of a source

Content

Extensions

Custom or Edit all

Access the Activity Browser and view all organization activities

Organization

Activities

View

Organization

Download source update logs

Content

Connectivity diagnostic logs

View

Organization

Activities

Organization

Organization
Important

A member with the View access level on the Activities domain can access the Activity Browser. This member can therefore see all activities taking place in the organization, including those from Coveo Administration Console pages that they can’t access.
1. Grayed out sources aren’t allowed by your Coveo license. Contact Coveo Sales to upgrade your license.