Manage Sources

A source is a virtual container holding all items related to a specific system (Twitter®, Web, SharePoint®…). Specific data from items within the source is collected to form an index. Coveo Cloud connects to a specific system, extracts its data content to create a source, indexes that source, and maintains the source up to date. Sources can be made part of the scope of one or more search interfaces to make its content searchable by end-users that have appropriate permissions.

Use the Sources page to initially create your sources, and come back anytime to manage them.

Sources Page

Access the “Sources” Page

  1. If not already done, log in to the Coveo Cloud Platform as a member of a group with the required privileges to manage sources in the target Coveo Cloud organization (see Required Privileges).

  2. In the main menu on the left, under Content, select Sources.

Add a New Source

  • Use the appropriate source type.

  • While you initially build and tune a source configuration, consider:

    • Making the source Private so that only you can see search results from this source.

    • Indexing only one or a few typical items to speed up rebuilds and limit the impact on your indexed system.

  • When the source is ready, make it Shared or Secured.

Once a source is created, any organization member with appropriate privileges can edit and even delete the source. You have no particular source privileges because you created the source .

To add a new source

  1. In the Sources page, click Add Source (see Access the Sources Page).

    If the Add Source button is grayed and unresponsive, you do not have all required privileges to perform this action (see Required Privileges).

  2. In the panel that appears, select the type of source that you want to create.

    Sources labeled Unavailable are not allowed by your Coveo Cloud V2 license. Contact Coveo to upgrade your license.

  3. Refer to the appropriate topic for details on how to create each source type (see Available Coveo Cloud V2 Source Types).

    Plan and pick source names carefully. Once created, source names can only be changed by completely deleting and rebuilding the associated source, a cumbersome task, especially with large sources that require several hours to rebuild.

Filter Visible Table Elements

When there are numerous sources listed in the page and you want to find one or more specific sources more easily:

  1. In the Sources page, in the filter box on the Action bar, type characters corresponding to textual elements that you are looking for in the table (see Access the Sources Page).

    Sources with no matching textual elements disappear, leaving only those that match. The filter matches any text string entered in the table.

    In the filter box:

    To only show sources Start typing
    Of type Salesforce salesforce
    Of type Cloud cloud
    With the security type Shared shared
    With web in the Name web
    Where Content is expressed in GB gb

  2. Clear the filter box content to list all sources again.

Review the Status and Size of All Your Sources

  1. On the Sources page, in the table, review the content in the Status column to determine the current source state as well as details on recent or upcoming activity (see Access the Sources Page).

  2. In the table, review the number and size of source items in the Content column.

Edit an Existing Source

Once a source is created, you can always come back and edit its configuration.

  1. In the Sources page, in the source list (see Access the Sources Page):

    • Select the source that you want to edit, and then in the Action bar, click Edit.

      OR

    • Double-click the source that you want to edit.

    Grayed-out sources are sources for which you only have the View access level (see Understanding the Custom Access Level). You cannot edit these sources, but you can click View in the Action bar to review their configuration.

  2. In the Edit a [SourceType] Source: [SourceName] panel, make the desired changes, and then depending on the changes you made:

    • When the changes impact source items (e.g., you changed the source URL) or you want the changes to be effective as soon as possible, click Save and Rebuild.

      OR

    • When your changes does not affect source items (e.g., you updated source credentials) or you want to ensure your changes are correct before applying them, click Save.

    • Clicking Save defers the source rebuild operation, which is necessary to apply changes to the indexed items. Therefore, if you made changes that will affect the indexed items, it is recommended to rather use the Save and Rebuild option (see Save and Rebuild). However, if you are sure that your changes will not affect the indexed items in any way, such as changes relative to the source credentials, you may click Save to avoid an irrelevant rebuild operation. Since a source with several millions of items may take weeks to rebuild, choosing the Save option when possible prevents stressing the Coveo Cloud and system server resources.

    • If you change fields or mappings and do not rebuild the source, the changes will apply only to new source items following a source refresh.

  3. If you clicked Save, back in the Sources page, under the source Status column, click Start required rebuild when you are ready or wait for the next scheduled source rebuild to make the changes effective.

Review the Activity of Individual Sources

  1. In the Sources page, in the table, select the source whose activity you want to review (see Access the Sources Page).

  2. In the Action bar, click Activity.

    If the Activity option is missing, you do not have all required privileges to perform this action (see Required Privileges).

  3. Review the Activity panel (see Review Events Related to Specific Coveo Cloud Administration Console Resources).

Refresh, Rescan or Rebuild Sources

While the best practice is to configure well thought out source update schedules (see Schedule Source Updates), you can also start a source update (refresh, rescan, or rebuild) manually. This is useful when you want to ensure that source content is up to date following a specific change in the indexed system.

  • Your coworker just completed the import of a large number of articles into your knowledge base system. You manually start a rescan of your knowledge base source to immediately make these imported articles searchable.

  • You edit a field option and want this option to apply immediately. You manually launch a rebuild operation.

  1. In the Sources page, in the source list, select the source that you want to update (see Access the Sources Page).

  2. In the Action bar, click Refresh, Rescan, or Rebuild.

    OR

    Click More, and select Refresh, Rescan, or Rebuild depending on what you want to achieve (see Refresh VS Rescan VS Rebuild).

    • Not all source types support refresh.

    • Changes on items already processed by an ongoing source update are taken into account by the next scheduled update.

    • Source operations retrieve the latest version of unprocessed items.

      You start a rescan at 12:00 AM with a source of 100 items. At 12:01 AM, when the rescan processes the 50th item, the title of the 81st item changes. The rescan takes the change into account and thus updates the 81st item title.

  3. Back in the source list, in the Status column for the selected source, monitor the progress for the operation that you selected.

    You can also get more update action information from the Activity panel (see Review the Activity of Individual Sources).

  4. You can pause or cancel (rescan and rebuild only) an ongoing source update by selecting the source row > More menu > Pause/Cancel [Update_Operation].

    • Source configuration changes (e.g., mappings, fields) that occur during a source update is paused are only taken account by the next rebuild.

      During a paused source rebuild, you add a mapping rule in the related source configuration. So you cancel the paused rebuild, and then start a new rebuild to take account of your mapping rule.

    • Not all sources support the pause and cancel operations.

Resume a Paused Source Update

When a source update is in the paused state, a button appears in the Action bar.

Not all sources support the pause on error and pause states.

  1. In the Sources page, in the source list, select the paused source for which you want to resume the update procedure (refresh, rebuild, or rescan) (see Access the Sources Page).

  2. In the Action bar, click Resume refresh, Resume rebuild, or Resume rescan, depending on the paused update.

Browse the Content of a Source

You can easily review the content of a source using the Content Browser (see Inspect Items With the Content Browser).

  1. In the Sources page, in the source list, select the source that you want to inspect (see Access the Sources Page).

  2. In the Action bar, click More, and select Open in Content Browser.

    The Content Browser page opens, and the source you selected is automatically selected in the Source facet so that only associated items appear in the search results.

  3. Use the Content Browser features to inspect your source content (see Inspect Items With the Content Browser).

Browse the Logs of a Source

You can review the log entries of a source using the Log Browser (see Review Item Logs).

  1. In the Sources page, in the source list, select the source of which you want to inspect the logs (see Access the Sources Page).

  2. In the Action bar, click More, and select Open in Log Browser.

    The Log Browser page opens up, and the source you selected is automatically selected in the Source facet so that only associated log entries appear in the search results.

  3. Use the Log Browser features to inspect your source logs (see Review Item Logs).

Update Access Token

OAuth sources such as Salesforce and Google Drive use access tokens that need to be updated manually when expired. You can easily update an OAuth source access token.

  1. In the Sources page, in the source list, select the OAuth source for which you want to update the access token (see Access the Sources Page).

  2. In the Action bar, click More, and then select Update access token.

  3. In the Update Access Token confirmation dialog:

    1. (For secured OAuth sources only) Optionally, click the Also update the token in the source identity security provider configuration check box.

      Ensure no source configurations will become invalid before updating the access token used by the source identity security provider.

    2. Click Update.

  4. In the page that appears, enter the email address and password of the account that you created for the source to index your content, and then authorize update by clicking Authorize, Allow, or Log In, depending on the source system.

    By doing so, you grant Coveo Cloud read right to your source content using OAuth, a protocol that grants access to external applications without exposing the user’s real credentials. You can always revoke the Coveo Cloud organization OAuth authorization (e.g., see Managing Your Remote Access Applications for Salesforce accounts and Revoke third-party access to your account information for Google accounts).

Access the Mapping Management Panel

Mappings define the content of Coveo index fields for each source item (see Manage Source Mappings).

  1. In the Sources page, in the source list, select the source whose mappings you want to manage (see Access the Sources Page).

  2. In the Action bar, click More and select Manage mappings.

Access the Extension Management Panel

In the Edit Source Extensions: [SourceName] panel, you can apply indexing pipeline extensions to a source (see Manage Extensions and Edit Source Extensions).

  1. In the Sources page, in the source list, select the source whose extensions you want to manage (see Access the Sources Page).

  2. In the Action bar, click More, and then select Manage extensions (see Manage Source Extensions).

Schedule Source Updates

To edit a source update schedule, see Edit a Source Schedule.

Edit the Source Configuration in JSON Format

You can view and change the source configuration in JSON format. That format lists all the source parameters in key-value pairs, including the mappings (see Edit a Source JSON Configuration).

Manage Source Notification Subscriptions

You can manage source notification subscriptions to specify whether you want to receive email notifications when certain activities are triggered by a specific source (see User Notifications).

  1. In the Sources page, in the source list, select the source you want to receive, or cease receiving email notifications for (see Access the Sources Page).
  2. In the Action bar, click More and select Subscribe to notifications (or Edit subscription to notifications) to access the subscription management panel, or select Unsubscribe from notifications to cease receiving notifications for that source.

Delete an Existing Source

Deleting a source permanently erases the source configuration and its indexed content.

  • Deleting an item does not remove it from all the internal index structures, but the item is no longer accessible from Coveo powered search pages. The index completely wipes the item during the next recycling run, which occurs each time 50k items are deleted. You can contact Coveo Support to change that threshold.

  • Deleting items is significantly faster than adding items.

  • The bigger the source, the longer it takes the index to delete the source content.

Delete unused sources since it frees up resources. Also, deleting a source during off-peak hours is faster since the index has less operations to manage simultaneously (e.g., fewer queries are performed).

Repeatedly create and delete a source, particularly one that contains a large number of items, because the deletion process monopolizes a significant amount of resources. In this case, it is better to refresh or rebuild the source (see Refresh, Rescan or Rebuild Sources).

  1. In the Sources page, in the source list, select the source that you want to delete (see Access the Sources Page).

  2. In the Action bar, click More and select Delete.

  3. At the Are you sure? prompt, click Delete.

(When your organization contains more than 50 sources) In the Sources page, at the bottom-right of the table, click the left and right arrow icons, or a page number to navigate through pages (see Access the Sources Page).

Set the Number of Sources per Page

(When your organization contains more than 50 sources) In the Sources page, at the bottom-left of the page, select 25, 50, or 100 (see Access the Sources Page).

By default, the table shows 50 sources per page.

Reference

The body of the Sources page is essentially a table listing all the sources that are defined in your Coveo organization.

  • Type column indicates:

  • Name column indicates:
    • The source name that was entered when creating the source. A developer may need a source name when configuring the scope of a search interface.
    • The source ID, which is automatically generated by Coveo Cloud V2. This ID may be useful when browsing Crawling Module logs (see Coveo On-Premises Crawling Module and About Component Logs).

  • Status column

    • Ongoing or latest action state

      • Completed

      • Failed

      • Stopped

    • When the last action occurred and how many items were processed or the reason why it failed

    • When the next upcoming action will occur

  • Content column

    • Number of items indexed in the source

      The number of items in the Content column can be different from the number of processed items reported in the Status column.

      • Removed items

        Following a source activity completion, there were 1000 processed items in the Status column, but 10 items were deleted, 2 were added, the rest were updated or did not change. In the Content column, there are 992 items.

      • Attachments

        A source includes some items on which there are attachments. An item that has 3 attachments, counts as 1 processed item in the Status column, but counts as 4 items (1 parent item and 3 attachment items) in the Content column.

    • Original size of the source items

Required Privileges

The following table indicates the privileges required to view or edit elements of the Sources page and associated panels (see Privilege Management and Privilege Reference).

Unlike for other resources, the ability to create sources can be granted without the Edit all access level (see Understanding Privileges). When granting privileges, you can therefore grant a group or API key the View all or Custom access level for sources and check the Can Create checkbox to allow users to create sources (see Understanding the Custom Access Level).

Action Service - Privilege Required access level
View sources

Content - Fields

Content - Sources

Organization - Activities

Organization - Organization

 View
Edit sources

Organization - Activities

Organization - Organization

 View

Content - Fields

Content - Sources

 Edit