Adding and Managing Sources

A source is a virtual container which holds all of the items related to a specific system (Twitter, Web, SharePoint, etc.). The Sources page of the Coveo Cloud Administration Console allows you to create and manage sources as needed.

When you add a source, Coveo Cloud connects to your repository and indexes its content. Depending on its selected content security option, the source content is accessible to one or more end users through a search interface. If the selected option is Determined by source permissions, only the end users who have the appropriate permissions will be able to view the search results.

Add a Source

1. On the Sources page, click Add Source.

2. In the Add a Source of Searchable Content panel that appears, select the type of source that you want to create.¹

3. See the appropriate article for details on how to create each source.

   1: Sources labeled Unavailable aren’t allowed by your Coveo Cloud license. Contact Coveo to upgrade your license.

Duplicate a Source

If you intend to create two sources with a similar configuration, you should consider creating a source and then duplicating it. The source duplication feature can save you time, especially if the source configuration involves many parameters. Once your first source is duplicated, you can then edit the desired parameters in the source copy­.

1. On the Sources page, click the source you want to duplicate.

2. In the Action bar, click More, and then select Duplicate.

3. In the Duplicate a Source panel that appears, enter a Source name for the copy, and then click Duplicate Source.

4. Back on the Sources page, once your new source has been added, edit it to make the desired changes.

5. Build your new source.

Edit a Source

1. On the Sources page, click the desired source, and then in the Action bar, click Edit. You can also view and edit the source configuration in JSON format. That format lists all of the source parameters in key-value pairs, including the mappings.

2. In the Edit a <SOURCE_TYPE> Source panel, make the desired changes. See the appropriate article for details on how to edit each source.

3. When you’re done:

   • If the changes impact source items (e.g., you changed the source URL) or you want the changes to be effective as soon as possible, click Save and Rebuild.

   • If your changes don’t impact source items (e.g., you updated source credentials) or if you plan to make additional changes in the future, click Save. Then, under the source Status column, you can click Start required rebuild when you’re ready or wait for the next scheduled source rebuild to make the changes effective.

Delete a Source

Deleting a source permanently erases the source configuration and its indexed content.

1. On the Sources page, click the source that you want to delete.

2. In the Action bar, click More, and then select Delete.

3. Click Delete to confirm.

Update an Access Token

Sources such as Salesforce and Google Drive use OAuth, a protocol that grants access to external applications without exposing the user’s real credentials. OAuth has access tokens that need to be updated manually when expired.

1. On the Sources page, click the OAuth source.

2. In the Action bar, click More, and then select Update access token.

3. In the Update Access Token confirmation dialog:

   1. If this option is available, check the Also update the token in the source identity security provider configuration box.

      Before you update the access token used by the source identity security provider, ensure that no source configurations will become invalid.

   2. Click Update.

4. On the page that appears, enter the email address and password of the account that you created for the source to index your content, and then authorize the update by clicking Authorize, Allow, or Log In, depending on the source system.

   • You can always revoke this authorization (e.g., see Manage OAuth Access for Your Connected Apps for Salesforce accounts and Revoke third-party access to your account information for Google accounts). However, if you do so, Coveo Cloud would no longer be able to retrieve the latest content changes and would show outdated search results.

Source Management Tools

On the Sources page, you can access various tools that enable you to manage different aspects of a source. Unless otherwise specified, these tools are accessible through the More option in the Action bar.

Refresh, Rescan, or Rebuild Sources

When you have the required privileges, you can manually apply different update types to ensure that the source remains up-to-date. The update types include refresh, rescan, and rebuild.

1. On the Sources page, click the source that you want to update.

2. Click More, and then select the update type.

   • Changes on items that have already been processed by an ongoing source update are taken into account by the next scheduled update.

   • Source operations retrieve the latest version of unprocessed items.

     For example, you start a rescan at 12:00 AM with a source of 100 items. At 12:01 AM, when the rescan processes the 50th item, the title of the 81st item changes. The rescan takes the change into account and therefore updates the 81st item title.

In the Status column for the selected source, you can monitor the progress of the operation that you selected. Alternatively, you can get more information on source updates from the Activity panel.

Pause Sources

If the source supports it, you can also pause or cancel an ongoing source update (rescan and rebuild only). This can be done by selecting the source row > More menu > Pause/Cancel <UPDATE_OPERATION>.

Resume a Paused Source Update

When a source update is in the paused state, a button appears in the Action bar to allows the reader to resume the update operation.

1. On the Sources page, click the paused source for which you want to resume the update procedure (refresh, rebuild, or rescan).

2. In the Action bar, click Resume refresh, Resume rebuild, or Resume rescan, depending on the paused update.

Review the Activity of a Source

When you have the required privileges, you can review the activity of a source.

1. On the Sources page, click the source whose activity you want to review.

2. In the Action bar, click Activity.

3. Review the source activity in the Activity panel.

Browse the Content of a Source

You can review the content of a source using the Content Browser.

1. On the Sources page, click the source that you want to inspect.

2. In the Action bar, click More, and select Open in Content Browser.

   On the Content Browser page, the source you selected is automatically selected in the Source facet so that only the corresponding items appear in the search results.

3. Use the Content Browser features to inspect your source content.

Browse the Logs of a Source

You can review the log entries of a source using the Log Browser.

1. On the Sources page, click the source of which you want to inspect the logs.

2. In the Action bar, click More, and then select Open in Log Browser.

   On the Log Browser page, the source you selected is automatically selected in the Source facet so that only the corresponding log entries appear in the search results.

3. Use the Log Browser features to inspect item logs.

Manage Source Notification Subscriptions

You can create source notification subscriptions to receive email notifications when certain activities are triggered by a specific source.

1. On the Sources page, click the source for which you want to manage subscriptions.

2. In the Action bar, click More, and then select one of the following:

   • Subscribe to notifications

   • Edit subscription to notifications (to access the subscription management panel)

   • Unsubscribe from notifications

Leading Practices

• Use the appropriate source connector and content retrieval method.

• While you initially build and tune a source configuration, consider:

  • Selecting the Source creator content security option so that only you can see search results from this source.

  • Indexing only one or a few generic items to speed up rebuilds and limit the impact on your indexed system.

• Creating a source doesn’t grant you additional privileges with respect to that source. Only an organization member with the required privileges can edit and delete a given source.

• While the best practice is to configure well-thought out source update schedules, you can also start a source update (refresh, rescan, or rebuild) manually. This is useful when you want to ensure that the content is up to date following a change in the source

  • Your coworker just imported of many articles into your knowledge base system. You manually start a rescan of your knowledge base source to immediately make these imported articles searchable.

  • You edit a field option and want this option to apply immediately to all items in the source. You manually launch a rebuild operation.

• Clicking Save when editing a source defers the source rebuild operation, which is necessary to apply changes to the indexed items. If you’re sure that your changes won’t affect the indexed items in any way, such as changes relative to the source credentials, you may click Save to avoid an irrelevant rebuild operation. Because a source may take weeks to rebuild with several million items, you can avoid adding stress to Coveo Cloud and system server resources by choosing the Save option whenever possible. However, if you made changes that will affect the indexed items, we recommend that you use the Save and Rebuild option instead.

• Delete unused sources to free up resources. Also, deleting a source during off-peak hours is faster since the index has less operations to manage simultaneously (e.g., fewer queries are performed).

• Avoid repeatedly creating and deleting a source, particularly one that contains many items, since the deletion process monopolizes a significant amount of resources. You should instead refresh or rebuild it.

• Schedule one or more source update types at suitable intervals (see Refresh VS Rescan VS Rebuild and What Should Be the Frequency of Source Refresh Schedules?).

• Turn off source update schedules for unused or test sources.

• Should you need to rename your source, you would have to duplicate it since a source name can’t be changed. Depending on your source size, this process may take several hours, as your new source would have to build entirely.

• When there are numerous sources listed on the page, in the Action bar, click the Status drop-down, and then select In Error in order to display those of which the last update operation failed. To further filter the list of sources in error, you may also choose to search for a specific source.

Reference

The Sources page is essentially a table listing the sources defined in your Coveo organization.

• The Type column indicates:

  • The connector icon and name.

  • The source content retrieval method.

  • The selected Content security option.

• The Name column indicates:

  • The source name that was entered when creating the source. A developer may need a source name when configuring the scope of a search interface.

  • The source ID, which is automatically generated by Coveo Cloud. This ID may be useful when browsing Crawling Module logs.

• The Status column indicates what the source is currently doing. A yellow or red indicator in this colum means that an action is required for a content retrieval operation to take place.

• The Last operation column indicates:

  • When the last update operation took place.

  • How many items were processed or the reason why the operation failed.

  • When the next update operation is due.

• The Content column indicates:

  • The number of items indexed in the source.

    The number of items in the Content column can be different from the number of processed items reported in the Status column.

    Examples:

    • Removed items

      Following a source activity completion, there were 1000 processed items in the Status column, but 10 items were deleted, 2 were added, the rest were updated or didn’t change. In the Content column, there are 992 items.

    • Attachments

      A source includes some items on which there are attachments. An item that has 3 attachments, counts as 1 processed item in the Status column, but counts as 4 items (1 parent item and 3 attachment items) in the Content column.

  • The original size of the source items.

  • The index in which these items are stored, if your organization has more than one (see Leverage Many Coveo Indexes).

Required Privileges

By default, members of the Administrators and Content Managers built-in groups can create and manage sources.

The following table indicates the privileges required to view or edit elements of the Sources page and associated panels (see Privilege Management and Privilege Reference).

Moreover, the following table lists privileges you can grant in addition to the required ones listed above. These privileges aren’t required to use the basic features of the Sources page, but they’re required to perform specific actions.