Manage your sources

This is for:

Developer System Administrator

A source is a virtual container that holds all the items related to a specific system (for example, YouTube, Web, SharePoint, Salesforce). The Sources (platform-ca | platform-eu | platform-au) page of the Coveo Administration Console lets you create and manage sources as needed.

When you add a source, Coveo connects to your repository and indexes its content. Depending on its selected content security option, the source content is accessible to one or more end users through a search interface. If the selected option is Same users and groups as in your content system, only the end users who have the appropriate permissions will be able to view the search results.

Sources page of the Coveo Administration Console

Leading practices when working with sources

When working with sources, consider the following leading practices:

  • Before you create a source, make sure to use the appropriate connector and content retrieval method.

  • Create your source in your sandbox organization first. Once you’ve confirmed that it indexes the desired content, you can copy your source configuration to your production organization, either with a snapshot or manually. Alternatively, when creating your source, select the Specific users and groups content security option so that only you can see search results from this source.

  • Work iteratively, especially with large content systems. Configure your source to index only one or a few generic items to speed up rebuilds and limit the impact on the indexed system. Once you’re satisfied with the source configuration, you can index all items.

  • Index only the content you need in your search interface for optimal crawling speed.

  • Aim for a good balance between content freshness and resource usage:

    • Select the option that uses the least amount of resources when launching or scheduling source updates. See Refresh, rescan, and rebuild for more information on source updates.

    • Delete any unused source to free up resources.

    • Schedule source updates during off-peak hours, when your Coveo index is less busy with user queries.

Add a source

Tip
Leading practice

It’s best to create or edit your source in your sandbox organization first. Once you’ve confirmed that it indexes the desired content, you can copy your source configuration to your production organization, either with a snapshot or manually.

See About non-production organizations for more information and best practices regarding sandbox organizations.

  1. On the Sources (platform-ca | platform-eu | platform-au) page, click Add source.

  2. In the Add a source of content panel that appears, select the type of source that you want to create or search for your software.

    When you search for your software, Coveo will recommend source types that have proven successful in indexing that specific software, if available. If Coveo recommends more than one option with the same name, see Content retrieval methods for details, if needed.

    More on source recommendations

    When you search for a software, Coveo recommends source types based on established indexing success with that software. Coveo may also recommend a sample configuration from the Coveo connectivity library for a recommended generic source type (for example, REST API, GraphQL API) to help you speed up the configuration process. In such cases, the sample configuration has been tested to work with the software, though it will need some adjustments to meet your specific needs.

    Connector recommendations | Coveo

  3. See the appropriate article for details on how to create each source.

  4. Review your indexed metadata and index more.

Duplicate a source

If you intend to create two sources with a similar configuration, you should consider creating a source and then duplicating it. The source duplication feature can save you time, especially if the source configuration involves many parameters. Once your first source is duplicated, you can then edit the desired parameters in the source copy.

You can also use the duplication feature to rename a source, since source names can’t be edited. However, depending on your source size, this process may take several hours, as your new source will have to build entirely.

  1. On the Sources (platform-ca | platform-eu | platform-au) page, click the source you want to duplicate, and then Duplicate in the More menu.

  2. In the Duplicate a source panel that appears, enter a source name for the copy, and then click Duplicate source.

  3. Back on the Sources (platform-ca | platform-eu | platform-au) page, once your new source has been added, edit it to make the desired changes.

  4. Build your new source.

Edit a source

Tip
Leading practice

It’s best to create or edit your source in your sandbox organization first. Once you’ve confirmed that it indexes the desired content, you can copy your source configuration to your production organization, either with a snapshot or manually.

See About non-production organizations for more information and best practices regarding sandbox organizations.

  1. On the Sources (platform-ca | platform-eu | platform-au) page, click the desired source, and then click Edit in the Action bar. For advanced configurations, you can also edit the source configuration in JSON format. This format lists all of the source parameters in key-value pairs, including the mappings.

  2. In the panel that appears, make the desired changes. See the appropriate article for details on how to edit each source.

    Note

    Some parameters such as the source name can’t be edited. If you want to change such a parameter, you must delete your source and add it again. Alternatively, to change a source’s name, you can duplicate it. Either way, keep in mind that the process can take a significant amount of time, as Coveo has to reindex your entire content.

  3. When you’re done, save your changes by clicking either Save or Save and rebuild source. See About saving options for details on these options.

About saving options

When editing a source, you typically have two options to save your changes: Save and Save and rebuild source.

Because a source with several million items may take weeks to rebuild, you should choose Save whenever possible to avoid unnecessary operations in the Coveo Platform and your content system.

Click Save when:

  • Your changes don’t affect the indexed content. For example, changes relative to the source credentials don’t require a rebuild to take effect.

  • You plan to make additional changes to the source configuration soon.

Then, back on the Sources (platform-ca | platform-eu | platform-au) page, click Launch rebuild when you’re ready to make the changes effective. Alternatively, you can wait for the next scheduled source rebuild to take place.

Click Save and rebuild source when:

  • Your changes will impact the indexed content items. For example, if you change the source URL, different items will be indexed, and you need to rebuild the source to apply this change.

  • You want your changes to be effective as soon as possible.

Delete a source

Deleting a source removes all of its indexed content from your Coveo organization. This operation can’t be undone.

Tip

If possible, delete sources during off-peak hours, when your Coveo index is less busy with user queries.

  1. On the Sources (platform-ca | platform-eu | platform-au) page, click the source that you want to delete, and then Delete in the More menu.

  2. Click Delete source to confirm.

Update an access token

Sources such as Salesforce and Zendesk use OAuth, a protocol that grants access to external applications without exposing the user’s real credentials. OAuth has access tokens that need to be updated manually when expired.

Note

The following procedure describes how to update the access token for Salesforce, Zendesk, and Microsoft Dynamics 365 sources. For SharePoint Online sources see Update a SharePoint Online access token. For all other OAuth source types, see the corresponding source configuration article for information on how to authenticate the source.

  1. On the Sources (platform-ca | platform-eu | platform-au) page, click the OAuth source, and then click More > Update access token in the Action bar.

  2. In the Update Access Token confirmation dialog:

    1. If this option is available, check the Also update the token in the source identity security provider configuration box.

      Note

      Before you update the access token used by the source identity security provider, ensure that no source configurations will become invalid.

    2. Click Update.

  3. On the page that appears, enter the email address and password of the account that you created for the source to index your content, and then authorize the update by clicking Authorize, Allow, or Log In, depending on the source system.

Note

You can always revoke this authorization (for example, see Manage OAuth Access for Your Connected Apps for Salesforce accounts and Revoke third-party access to your account information for Google accounts). However, if you do so, Coveo would no longer be able to retrieve the latest content changes and would show outdated search results.

View the metadata of a source

The View and map metadata subpage displays the metadata of your source items in a user-friendly format.

To access this tool, select a source on the Sources (platform-ca | platform-eu | platform-au) page, and then click More > View and map metadata in the Action bar.

You can use the View and map metadata subpage to:

  • Explore the metadata of your source items, such as the title, body, date, and author.

  • Verify that your source items have the correct metadata and mapping rules.

  • Add missing metadata to your index.

See View the metadata of a source for details on this feature.

Refresh, rescan, or rebuild sources

You can manually launch an update operation (refresh, rescan, or rebuild) to ensure that the content displayed in your search interface reflects your actual content. See Refresh, rescan, and rebuild for details on each operation.

This is useful when you want to ensure that the content is up to date following a change in the source.

Examples

  • Your coworker just imported many articles into your knowledge base system. Instead of waiting for the next schedule update operation, you manually launch a rescan of your knowledge base source. As a result, the content is indexed and available in your search interface sooner.

  • You edit a field option and want this option to apply immediately to all items in the source. You manually launch a rebuild operation.

  1. On the Sources (platform-ca | platform-eu | platform-au) page, click the source that you want to update.

  2. Click the desired update operation in the Action bar or the More menu.

You can follow and review updates in the Activity panel.

Pause a source update

Most sources support pausing an ongoing rescan or rebuild. This can be done on the Sources (platform-ca | platform-eu | platform-au) page by selecting the source, and then Pause Rescan/Rebuild in the More menu.

Changes in the source configuration or source-related resources (for example, mappings, fields) that occur while an update is paused are taken into account when the operation is resumed.

A source update can also be paused when there’s an issue with the source or with Coveo. Once the update is either resumed and completed, or canceled, your source proceeds with any pending updates. The update operation is automatically canceled after 7 days.

Review pending updates

A source update operation is pending when it can’t be executed yet due to another operation already taking place. For example, if you launch a rebuild while a scheduled refresh is ongoing, your rebuild will only take place once the refresh is completed.

You can see pending updates either in the Activity panel or the Activity Browser (platform-ca | platform-eu | platform-au).

Copy a source name or ID

You can copy the name or ID of a source to use it in other parts of the Coveo Platform, including the Coveo Platform API.

To do so, on the Sources (platform-ca | platform-eu | platform-au) page, click copy16px next to the source name.

Copy a source name or ID in the Coveo Administration Console

Browse the content of a source

You can review the content of a source using the Content Browser.

  1. On the Sources (platform-ca | platform-eu | platform-au) page, click the source that you want to inspect, and then click More > Open in Content Browser in the Action bar.

    On the Content Browser (platform-ca | platform-eu | platform-au) page, your source is automatically selected in the Source facet so that only the corresponding items appear in the search results.

  2. Use the Content Browser features to inspect your source content.

Review the activity of a source

As part of your duties, you may need to review activities related to sources for investigation or troubleshooting purposes. To do so, on the Sources (platform-ca | platform-eu | platform-au) page, select a source, and then click Activity in the Action bar.

See Review resource activity for details on activities and alternative ways to access this information.

Expanding a source update row reveals additional information on the operation, including:

  • Items added: Items indexed for the first time.

  • Items unchanged: Items that were already in the index and that didn’t change since the last source update.

  • Items processed: Items retrieved by the crawler. If they meet the source filter criteria, they’re indexed. If not, they’re filtered out.

  • Items filtered: Items retrieved by the crawler that didn’t meet the source filter criteria and therefore weren’t indexed.

  • Items updated: Items that have already been indexed in the past and that have been updated in your content system since last source update.

Download source update logs

Should you need more information about an ongoing or completed source update operation, you can download logs of the desired activity. Log files provide a detailed account of the update process, including any warning or error that hinders part or all the update operation.

Information in update logs is nonsensitive. This is information related to the authentication process, content expansion, and item filtering, among other things.

To download an update log

  1. On the Sources (platform-ca | platform-eu | platform-au) page, click the desired resource, and then click Activity in the Action bar.

  2. In the Activity panel that opens, click the desired activity, and then click Download Logs in the Action bar. The downloaded file is named after the unique operation ID representing the selected activity.

To locate issue root causes in logs

  1. Open the log file in a text file viewer.

  2. Look for WARN,ERROR, and FATAL messages.

    Indexing activity logs with a warning message | Coveo
    Tip

    Use a log file viewer that supports highlighting by log level to make these messages more noticeable.

  3. If necessary, review INFO messages. They sometimes reveal a configuration that you overlooked and that may be causing the issue.

    Indexing activity logs with an info message | Coveo

See Ways to review activity for alternative ways to access this information.

Subscribe to notifications

You can subscribe to source notifications to get an email when certain activities are triggered by a specific source, or when certain activities linked to a source occur.

  1. On the Sources (platform-ca | platform-eu | platform-au) page, click the desired source.

  2. In the Action bar, click More, and then select one of the following:

    • Subscribe to email notifications

    • Edit email notifications

  3. In the panel that appears, enter a name for your subscription.

  4. Select the frequency at which you want to receive emails.

  5. Under Resource types, select the resource types you’re interested in.

  6. Under Operations, select the desired operations that apply to the selected resource types.

  7. If you selected updated operations and want to be notified only when these operations aren’t successful, select Failed under Result.

  8. Click Save.

Required privileges

The following table indicates the privileges required for members to view or edit elements of the Sources (platform-ca | platform-eu | platform-au) page and associated panels (see Manage privileges and Privilege reference).

Note

The Edit all privilege isn’t required to create sources. When granting privileges for the Sources domain, you can grant a group or API key the View all or Custom access level, instead of Edit all, and then select the Can Create checkbox to allow users to create sources. See Can Create ability dependence for more information.
Actions Service Domain Required access level

View sources, view source update schedules, and subscribe to source notifications

Content

Fields

View

Sources

Organization

Organization

Edit sources, edit source update schedules, and edit source mappings

Organization

Organization

View

Content

Fields

Edit

Sources

View and map metadata

Content

Source metadata

View

Fields

Organization

Organization

Content

Sources

Edit

Moreover, the following table lists privileges you can grant in addition to the required ones in the previous tables. These privileges aren’t required to use the basic features of the Sources page, but they’re required to perform specific actions.

Action Service Domain Required access level

When creating or editing a source, select the Crawling Module instance to pair with this source

Content

Crawling Module

View

Manage the Extensions of a source

Content

Extensions

Custom or Edit all

Access the Activity Browser and view all organization activities

Organization

Activities

View

Organization

Download source update logs

Content

Connectivity diagnostic logs

View

Organization

Activities

Organization

Organization
Important

A member with the View access level on the Activities domain can access the Activity Browser. This member can therefore see all activities taking place in the organization, including those from Coveo Administration Console pages that they can’t access.