Connector building best practices

Typically, we recommend implementing the following capabilities:

A crawler that can conduct a build or rebuild operation can obtain the expected content from the repository and provide it to Coveo. This is the only capability that all crawlers should have, as is it essential for the crawler to work properly.

For further information on the build operation, see Rebuild.

It is often useful to have a configurable crawler, as it allows adapting to the settings of the repository without editing and testing the crawler code again.

We recommend taking a look at the options available in the source creation panels in the Coveo Administration Console to get an overview of the configuration capabilities Coveo offers.

Implementing some crawler options may require considerable work. Therefore, you could opt to start by making a non configurable crawler, and then add configuration options in a subsequent iteration of your project. However, if you choose to do so, you must ensure that the hard-coded configuration values in the original crawler are appropriately grouped.

Depending on the content to index, the crawler may need to authenticate to access it. Some repositories may offer more than one authentication option for this crucial step. Determining how to connect early in your project will help you configure your crawler down the line.

Selecting the item types you want to crawl and possibly making this selection configurable is another important decision to make. We recommend creating a crawler that retrieves only the content that will be relevant to your search interface end users. Not only does this makes the crawler easier to develop, but its work will be more efficient.

For instance, if you crawl the content of a blog, you could choose to index posts only and to ignore comments.

Retrieving metadata is likely to be a crucial task of your crawler. With Coveo, the crawled items are meant to become searchable, and search results can be filtered based on their metadata. It’s therefore important to index this information properly.

If you choose to retrieve metadata, you’ll also have to decide whether you want to retrieve everything or some specific pieces of information. The latter is more efficient, but retrieving all the metadata might make it easier to handle the retrieved items further down the line. In other words, if you don’t index all the available metadata, you might have fewer options when writing indexing pipeline extensions, for instance, which might lead you to change the crawler code.

Coveo can secure many content sources by replicating the repository permission system. As a result, through a Coveo-powered search interface, authenticated users only see the items that they’re allowed to access within the indexed repository.

Retrieving permissions and supporting secured search can be complex. If you choose to do so, you must determine in which format the crawler will retrieve permissions and how they will be used down the line.

For more information on how Coveo handles permissions, see Coveo management of security identities and item permissions.

When programming your crawler, you must also determine how robust it should be, that is, how many exceptions encountered while crawling it should handle.

If you intend to develop a simple crawler, a crawler for a very specific use case, or a crawler for a one-time use, it could make sense to handle fewer exceptions. However, if you want to distribute a connector that can connect to multiple configurations of a system, you need to plan time and tests to discover the exceptions susceptible to occur and adjust your crawler accordingly.

A few crawler capabilities depend on the capability to keep a state of the crawled items, that is, information regarding the last update operation. Upon the next update operation, the crawler uses this information to execute the task.

State keeping allows refresh operations and pausing the update process, among other things.

Discovering content that has been deleted and updating the index accordingly can be a useful capability of your crawler project, especially if it’s done as part of a refresh operation.

However, most APIs don’t provide a list of deleted items, therefore it can be complex to derive it from other data. A state can also be helpful to keep track of the items that have been crawled and list what is now gone, but this process may require a lot of computing.

First, the crawler must obtain and validate the configuration data. It will likely need user credentials to authenticate to your content repository. We recommend implementing a mechanism that lets the user know if the configuration is invalid or incomplete. It is best to have the operation fail immediately rather than to crawl and fail later due to an invalid configuration.

Once the crawler has validated the configuration information, it should authenticate and connect to the content repository you want to crawl. Implementing the authentication step may be complex if there are multiple scenarios to handle, that is, multiple ways to log in to your repository. As a result, testing how the crawler authenticates early in the project is a good idea.

Once the crawler has access to the content to index, it can operate iteratively, going through each item container, one after the other.

You should also structure the content to index. For example, you might want to prepare the data to send to Coveo’s Push API. Alternatively, you could need to aggregate metadata from the item’s parent container, for instance.

The last task that your connector has to accomplish is to provide Coveo with the content to index. This can be done through Coveo’s Push API, a folder on the computer hosting your crawler, etc.