Adding and Managing Sources

A source is a virtual container holding all items related to a specific system (Twitter®, Web, SharePoint®…). Specific data from items within the source is collected to form an index. Coveo Cloud connects to a specific system, extracts its data content to create a source, indexes that source, and maintains the source up to date. Sources can be made part of the scope of one or more search interfaces to make its content searchable by end users that have appropriate permissions.

Members of the Administrators and Content Managers built-in groups can use the Sources page to initially create your sources, and come back anytime to manage them.

Page Accessibility

Access the Mapping Management Panel

Mappings define the content of Coveo index fields for each source item (see Adding and Managing Source Mappings).

1. On the Sources page, in the source list, select the source whose mappings you want to manage.

2. In the Action bar, click More and select Manage mappings.

Access the Extension Management Panel

In the Edit Source Extensions panel, you can apply indexing pipeline extensions to a source (see Adding and Managing Extensions and Edit Source Extensions).

1. On the Sources page, in the source list, select the source whose extensions you want to manage.

2. In the Action bar, click More, and then select Manage extensions (see Manage Source Extensions).

Add, Edit, Duplicate, and Delete Sources

Add a New Source

1. On the Sources page, click Add Source.

   If the Add Source button is grayed and unresponsive, you do not have all of the required privileges to perform this action.

2. In the panel that appears, select the type of source that you want to create.

   Sources labeled Unavailable are not allowed by your Coveo Cloud license. Contact Coveo to upgrade your license.

3. Refer to the appropriate article for details on how to create each source (see Available Connectors).

   A source name cannot be changed. Should you need to rename your source, you would have to duplicate it. Depending on your source size, this process may take several hours, as your new source would have to build entirely.

Duplicate a Source

If you intend to create two sources with a similar configuration, you should consider creating a source, and then duplicating it. The source duplication feature can save you time, especially if the source configuration involves more than a few parameters. Once your first source is duplicated, you can then edit the desired parameters in the source copy­.

1. On the Sources page, select the source you want to duplicate.

2. In the Action bar, click More, and the select Duplicate.

3. In the Duplicate a Source: [Source Name] panel that appears, enter a Source name for the copy, and then click Duplicate source.

4. Back on the Sources page, once your new source has been added, edit it to make the desired changes (see Edit an Existing Source).

5. Build your new source (see Refresh, Rescan, or Rebuild Sources).

Edit an Existing Source

Once a source is created, you can always come back and edit its configuration.

1. On the Sources page, double-click the source that you want to edit.

   Grayed-out sources are sources for which you only have the View access level (see Understanding the Custom Access Level). You cannot edit these sources, but you can click View in the Action bar to review their configuration.

2. In the Edit a [SourceType] Source panel, make the desired changes, and then depending on the changes you made:

   • When the changes impact source items (e.g., you changed the source URL) or you want the changes to be effective as soon as possible, click Save and Rebuild.

     OR

   • When your changes does not affect source items (e.g., you updated source credentials) or you want to ensure your changes are correct before applying them, click Save.

   • Clicking Save defers the source rebuild operation, which is necessary to apply changes to the indexed items. Therefore, if you made changes that will affect the indexed items, it is recommended to rather use the Save and Rebuild option (see Save and Rebuild). However, if you are sure that your changes will not affect the indexed items in any way, such as changes relative to the source credentials, you may click Save to avoid an irrelevant rebuild operation. Since a source with several millions of items may take weeks to rebuild, choosing the Save option when possible prevents stressing the Coveo Cloud and system server resources.

   • If you change fields or mappings and do not rebuild the source, the changes will apply only to new source items following a source refresh.

3. If you clicked Save, back in the Sources page, under the source Status column, click Start required rebuild when you are ready or wait for the next scheduled source rebuild to make the changes effective.

Edit the Source Configuration in JSON Format

You can view and change the source configuration in JSON format. That format lists all the source parameters in key-value pairs, including the mappings (see Edit a Source JSON Configuration).

Delete an Existing Source

Deleting a source permanently erases the source configuration and its indexed content.

1. On the Sources page, in the source list, select the source that you want to delete.

2. In the Action bar, click More and select Delete.

3. At the Are you sure? prompt, click Delete.

Update Access Token

OAuth sources such as Salesforce and Google Drive use access tokens that need to be updated manually when expired. You can easily update an OAuth source access token.

1. On the Sources page, in the source list, select the OAuth source for which you want to update the access token.

2. In the Action bar, click More, and then select Update access token.

3. In the Update Access Token confirmation dialog:

   1. (For secured OAuth sources only) Optionally, click the Also update the token in the source identity security provider configuration check box.

      Ensure no source configurations will become invalid before updating the access token used by the source identity security provider.

   2. Click Update.

4. In the page that appears, enter the email address and password of the account that you created for the source to index your content, and then authorize update by clicking Authorize, Allow, or Log In, depending on the source system.

   By doing so, you grant Coveo Cloud read right to your source content using OAuth, a protocol that grants access to external applications without exposing the user’s real credentials. You can always revoke the Coveo Cloud organization OAuth authorization (e.g., see Manage OAuth Access for Your Connected Apps for Salesforce accounts and Revoke third-party access to your account information for Google accounts).

Manage Source Content Updates

Refresh, Rescan, or Rebuild Sources

While the best practice is to configure well thought out source update schedules (see Schedule Source Updates), you can also start a source update (refresh, rescan, or rebuild) manually. This is useful when you want to ensure that source content is up to date following a specific change in the indexed system.

1. On the Sources page, in the source list, select the source that you want to update.

2. In the Action bar, click Refresh, Rescan, or Rebuild.

   OR

   Click More, and select Refresh, Rescan, or Rebuild depending on what you want to achieve (see Refresh VS Rescan VS Rebuild).

   • Not all connectors support refresh.

   • Changes on items already processed by an ongoing source update are taken into account by the next scheduled update.

   • Source operations retrieve the latest version of unprocessed items.

     You start a rescan at 12:00 AM with a source of 100 items. At 12:01 AM, when the rescan processes the 50th item, the title of the 81st item changes. The rescan takes the change into account and thus updates the 81st item title.

3. Back on the source list, in the Status column for the selected source, monitor the progress for the operation that you selected.

   You can also get more update action information from the Activity panel (see Review the Activity of Individual Sources).

4. You can pause or cancel (rescan and rebuild only) an ongoing source update by selecting the source row > More menu > Pause/Cancel [Update_Operation].

   • Source configuration changes (e.g., mappings, fields) that occur when a source update is paused are taken account when the source operation is resumed.

     During a paused source rebuild, you add a mapping rule in the related source configuration. You resume the paused rebuild, which will take your new mapping rule into account.

   • Not all sources support the pause and cancel operations.

   • A paused source operation is canceled after a period of inactivity. The duration of this period depends on how the source operation was paused.

     Source status	Cause	Cancellation
     Paused on error	Issue with the source or the Coveo Cloud platform	After 7 days
     Paused	Operation paused manually	After 1 month

Schedule Source Updates

To edit a source update schedule, see Edit a Source Schedule.

Resume a Paused Source Update

When a source update is in the paused state, a button appears in the Action bar.

1. On the Sources page, in the source list, select the paused source for which you want to resume the update procedure (refresh, rebuild, or rescan).

2. In the Action bar, click Resume refresh, Resume rebuild, or Resume rescan, depending on the paused update.

Inspect Sources

Review the Status and Size of a Source

On the Sources page, the Status, Last Operation, and Content columns display the current source state, recent or upcoming activities, and the source size and number of items. See Reference for further information.

Review the Activity of Individual Sources

1. On the Sources page, in the table, select the source whose activity you want to review.

2. In the Action bar, click Activity.

   If the Activity option is missing, you do not have all of the required privileges to perform this action.

3. Review the Activity panel (see Review Events Related to Specific Coveo Cloud Administration Console Resources).

Browse the Content of a Source

You can easily review the content of a source using the Content Browser (see Inspect Items With the Content Browser).

1. On the Sources page, in the source list, select the source that you want to inspect.

2. In the Action bar, click More, and select Open in Content Browser.

   The Content Browser page opens, and the source you selected is automatically selected in the Source facet so that only associated items appear in the search results.

3. Use the Content Browser features to inspect your source content (see Inspect Items With the Content Browser).

Browse the Logs of a Source

You can review the log entries of a source using the Log Browser (see Review Item Logs).

1. On the Sources page, in the source list, select the source of which you want to inspect the logs.

2. In the Action bar, click More, and select Open in Log Browser.

   The Log Browser page opens up, and the source you selected is automatically selected in the Source facet so that only associated log entries appear in the search results.

3. Use the Log Browser features to inspect your source logs (see Review Item Logs).

Manage Source Notification Subscriptions

You can manage source notification subscriptions to specify whether you want to receive email notifications when certain activities are triggered by a specific source (see User Notifications).

1. On the Sources page, in the source list, select the source you want to receive, or cease receiving email notifications for.

2. In the Action bar, click More and select Subscribe to notifications (or Edit subscription to notifications) to access the subscription management panel, or select Unsubscribe from notifications to cease receiving notifications for that source.

Reference

The Sources page is essentially a table listing the sources defined in your Coveo organization.

• The Type column indicates:

  • The connector icon and name.

  • The source content retrieval method tag.

  • The selected content security option.

• The Name column indicates:

  • The source name that was entered when creating the source. A developer may need a source name when configuring the scope of a search interface.

  • The source ID, which is automatically generated by Coveo Cloud. This ID may be useful when browsing Crawling Module logs (see Coveo On-Premises Crawling Module and About Logs).

• The Status column indicates what the source is currently doing. A yellow or red indicator in this colum means that an action is required for a content retrieval operation to take place.

• The Last operation column indicates:

  • When the last update operation took place.

  • How many items were processed or the reason why the operation failed.

  • When the next update operation is due.

• The Content column indicates:

  • The number of items indexed in the source.

    The number of items in the Content column can be different from the number of processed items reported in the Status column.

    • Removed items

      Following a source activity completion, there were 1000 processed items in the Status column, but 10 items were deleted, 2 were added, the rest were updated or did not change. In the Content column, there are 992 items.

    • Attachments

      A source includes some items on which there are attachments. An item that has 3 attachments, counts as 1 processed item in the Status column, but counts as 4 items (1 parent item and 3 attachment items) in the Content column.

  • The original size of the source items.

  • The index in which these items are stored, if your organization has more than one (see Leverage Many Coveo Indexes).

    To see the index name in the Content column, you need the View access level on the Logical Index domain (see Privilege Management and Logical Indexes Domain).

Required Privileges

The following table indicates the privileges required to view or edit elements of the Sources page and associated panels (see Privilege Management and Privilege Reference).

Moreover, the following table lists privileges you can grant in addition to the required ones listed above. These privileges are not required to use the basic features of the Sources page, but they are required to use perform specific actions in this page.