Push data to Coveo (SAP Commerce Cloud 2011 or later)

This is for:

Developer System Administrator

This article explains how to push data from an SAP Commerce Cloud (version 2011 or later) to a Coveo organization.

Prerequisites

You’ll need the Coveo Push API client library for Java, so make sure that:

Install the Coveo SAP Commerce indexer push extension

The Coveo SAP Commerce indexer push extension allows your SAP project to exchange data with Coveo. To install the extension, perform the following steps:

  1. Copy the extension repository to the hybris/bin/custom directory of your project.

    Important

    Depending on your SAP Commerce version, download the extension from the appropriate branch:

    • For SAP Commerce Cloud 2011, use the v1 branch.

    • For SAP Commerce Cloud 2105 and 2205, use the v2 branch.

    • For SAP Commerce Cloud 2211, use the main branch.

  2. Open the hybris/config/localextensions.xml file and add the searchprovidercoveosearchservices extension:

    <extension name='searchprovidercoveosearchservices' />

  3. Install the Coveo Push API client library for Java:

    1. Open the Apache Maven settings.xml file. The file is located at ${user.home}/.m2/settings.xml.

    2. Add a repository definition for the library’s GitHub Package.

      <repository>
    <id>github</id>
    <url>https://maven.pkg.github.com/coveo/push-api-client.java</url>
    <snapshots>
        <enabled>true</enabled>
    </snapshots>
</repository>

    3. Specify your GitHub personal access token (classic) to be able to install packages from GitHub Packages.

      Important

      The token should have at least read:packages permission.

      		 
      <servers>
<server>
    <id>github</id>
    <username>USERNAME</username>
    <password>TOKEN</password>
</server>
</servers>

  4. Set up the Ant environment by running the script:

    . ./setantenv.sh

  5. From the root of your project directory, run the following command:

    ant clean all

  6. After the command execution, navigate to the hybris/bin/platform directory.

  7. Start your SAP Commerce installation by running the script:

    ./hybrisserver.sh

    By default, your local server would use ports 9001 (http) and 9002 (https).

  8. Open the Hybris Administration console at https://localhost:9002/platform/update. You can also navigate to the Update page by clicking Platform → Update in the top menu.

  9. Find and select the checkboxes for:

    • searchprovidercoveosearchservices (required to create the new search provider)

    • Update running system

    • Localize types

  10. At the top of the page, click Update.

Note

The extension creates new data types in the database, but it doesn’t import any data.
Tip
Semi-automatic configuration

In the extension code base, if you navigate to resources/examples, you’ll find the electronics-search-configuration-example.impex file. This is an example impex of how you can perform the SAP Commerce Cloud configuration in a programmatic way. This can be used as a template to fill with your data and configure your project faster.

You can also manually configure every source and your indexes via the Backoffice. Perform the following steps to do so.

Coveo setup

Step 1: Create a source in your Coveo organization

For storing the SAP catalog data, you need a Catalog source.

  1. Log in to the Coveo Administration Console.

  2. Add the Catalog sources that you’ll be pushing your data to.

    The guidelines for creating sources are:

    • Each combination of language, currency, and country existing in your data should have a separate Coveo Catalog source to push to.

    • Any availability data you have should also have it’s own dedicated Catalog source.

  3. Take note of the Stream API URL value for your source.

Step 2: Create an API key

Create one API key that you’ll use for all your sources. See Manage API Keys.

The key should have the edit privileges detailed here: Catalog source privileges. See the Edit sources, edit source update schedules, and view the View and map metadata page row in the privileges table.

SAP Commerce setup

Step 1: Create Coveo sources

For each source you created in the Coveo Administration Console you should perform the following steps in the SAP Backoffice.

  1. In the SAP Backoffice Administration Cockpit, go to System → API → Destinations → Coveo Source.

  2. Click the plus sign (+) above the search results.

  3. In the modal window that appears, enter a unique ID that represents your Catalog source. It’s recommended to use a human-readable value to assist with identifying a specific source.

  4. Click Finish.

  5. Select the newly created item from the search listing.

  6. On the General tab:

    1. Fill in the Name field.

    2. In the Object type menu, specify the type that matches your Coveo source:

      AVAILABILITY

      This type is for the availability source. See Source configuration approaches for availability channel and Catalog availability data.

      PRODUCTANDVARIANT

      This type is for the the products and variations of the catalog. See Catalog product data and Catalog variant data and product groupings.

      Important

      When selecting an object type of PRODUCTANDVARIANT, Language, Currency, and Country are mandatory for the system to determine the correct source for your product data.

  7. Click Save.

Step 2: Create a consumed destination

  1. Go to System → API → Destinations → Consumed Destinations.

  2. Click the plus sign (+) above the search results.

  3. In the modal window that appears:

    ID

    Enter a unique ID that represents your Catalog source.

    URL

    Paste the Stream API URL that you copied from the Coveo Catalog source.

    Details

    1. In your Coveo organization, open the Catalog source you created in step 2.

    2. On the Configuration tab, there’s a section called Stream API URL. Click Copy to clipboard to copy the URL.

    Active

    Verify that the Active checkbox is selected.

    Destination Target

    Click the field and select Default_Template.

    Note

    The destination target you select doesn’t matter as it’s not used in the push implementation and only needed to create the destination.

  4. Click Finish.

    The properties of a new consumed destination

Now you see the destination you created in the list of consumed destinations.

Step 3: Create a credential for the consumed destination

Important

If you have a number of sources, you only need to create one OAuth Credential and attach it to many Consumed Destinations.

  1. Click the destination you created and switch to the Destination Configuration tab.

  2. Double-click in the Credential field to open a dropdown menu and select Create Credential → Consumed OAuth Credential.

  3. Fill in the following fields:

    ID

    Enter a unique ID for a new credential. For example, CoveoCredential.

    Client ID

    Enter a client ID for a new credential. For example, CoveoClientID.

  4. Click Finish. You see the Destination Configuration tab again.

  5. Double-click in the Credential field to open the credential settings.

  6. In the Client Secret section, there are Password and Verify password fields. In both fields, paste the Coveo API key that you created earlier.

  7. Click Save to save and close the credential settings.

  8. Click Save to save the destination settings.

The properties of a Coveo credential with the password filled in

Step 4: Assign the consumed destination to the Coveo source

  1. Return to System → API → Destinations → Coveo Source.

  2. Click your Coveo source.

  3. On the General tab, find the Consumed Destination field.

  4. Specify the consumed destination that you created earlier.

  5. Click Save.

Step 5: Create a search provider

Now that you’ve mapped the Coveo sources to the SAP Commerce API destinations, you need to configure the search index to push the data to Coveo.

  1. Go to the System → Search and Navigation → Search Provider Configurations page.

  2. Next to the plus sign, click the dropdown menu and select Coveo Search Provider Configuration.

  3. Fill in the following fields:

    Identifier

    Enter a unique identifier for a new search provider, for example, CoveoSearchProviderID.

    Name

    Enter a name for a new search provider, for example, Coveo Search Provider Configuration.

    Coveo Source

    Select all the Coveo sources that you created.

  4. Click Finish. Now you see the provider you created in the list of search providers.

Step 6: Create the index configuration

  1. Go to System → Search and Navigation → Index Configurations.

  2. Next to the plus sign (+), click the dropdown menu and select CoveoSnIndexConfiguration.

  3. Fill in the following fields:

    Identifier

    Enter a unique identifier for a new configuration, for example, CoveoIndexID.

    Name

    Enter a name for a new search provider, for example, Coveo Index Configuration.

    Search Provider Configuration

    Select the Coveo Search Provider Configuration created in the previous step.

  4. Click Next to proceed to the next step, Index Types.

    Skip this step for now and click Next again.

  5. In the Session step, select all the Languages, Currencies, and Countries that are relevant for your data.

    Important

    You must fill in the User field if you have data with access restrictions. If that’s the case, specify the user name with the necessary permissions to access the data.

    Otherwise, you can leave the User field empty or set the user as Anonymous as it’s done in the example impex file in the extension code base.

    The Index configuration wizard in the SAP Commerce Backoffice

  6. Click Finish.

Step 7: Create the ServiceLayerJobs

Create the ServiceLayerJobs for the future cron jobs, one for full indexing and one for incremental indexing.

  1. Go to System → Types.

  2. In the search bar, type in ServicelayerJob and search.

  3. In the search results, click the ServicelayerJob.

    The properties of the ServicelayerJob are displayed on the page.

  4. In the upper part of the properties, click the search icon that’s titled Search by type.

  5. Click the plus sign (+) above the search listing.

  6. In the modal window that appears:

    Spring ID

    Paste fullCoveoSnIndexerJob.

    Code

    Paste fullCoveoSnIndexerJob.

  7. Click Finish.

  8. Perform the same steps to create another ServicelayerJob for which specify incrementalCoveoSnIndexerJob for the Code and the Spring ID attributes.

Step 8: Create the Index types

Create one Index type for full indexing and one for incremental indexing.

  1. Go to System → Search and Navigation → Index Types.

  2. Click the plus sign (+) above the search results.

  3. In the modal window that appears, in the Essential step, fill in the following fields:

    Identifier

    Enter a unique identifier for a new type, for example, CoveoFullIndexTypeID.

    Name

    Enter a name for a new type, for example, Coveo Full Index Type Configuration.

    Composed Type

    Specify the type that matches a Coveo Object type for the source you’ll use this to push data to.

    Example

    If this index type will be used to push availability objects, then you may want to select the Warehouse Composed type. This will be dependent on how your data is structured.

    Index Configuration

    Select the Coveo Index Configuration created in the previous steps.

  4. The next steps can be configured later, so you may skip them for now. Click Next to proceed to the Cron Jobs step.

  5. In the Cron Jobs step, click the plus sign (+) at the end of the dropdown menu.

  6. Select Full Indexer Cronjob.

  7. Fill in the following fields:

    • Code. Enter a unique code for the cron job, for example, coveoFullIndexType.

    • Job Definition. Select the fullCoveoSnIndexerJob system type. The chosen type should correspond to the type of cron job you’re creating.

    • Indexer Item Source. Click this field to create a Flexible Search Indexer Item Source.

      In the modal window that appears, fill in the Query field. It must be the FlexibleSearch query that will be used to retrieve the Composed Type. For example, if you selected Warehouse for your availability object, the following query would retrieve all warehouses for the electronics site:

      SELECT {w:PK}
FROM {Warehouse as w
  JOIN BaseStore2WarehouseRel as rel ON {w:PK} = {rel:target}
  JOIN BaseStore AS bs ON {rel:source} = {bs:PK}
}
WHERE {bs:uid}='electronics'

      For product indexing, the following query would retrieve all approved products in the catalog.

      SELECT {p:pk}
FROM {Product AS p
  LEFT JOIN ArticleApprovalStatus AS a ON {p:approvalStatus} = {a:pk}
}
WHERE {a:code} = 'approved'

  8. Click Finish for every modal window that has been opened.

  9. Select the newly created Index type.

  10. On the General tab, configure the Identity Provider as snIdentityProvider and link the catalog you want to index in the Catalogs field.

  11. Perform the same steps to create an Index type for incremental indexing, but adjust them as follows:

    • in the Essential step, update the Identifier and Name fields to reflect the incremental indexing type.

    • in the Create Cron Jobs step, select Incremental Indexer Cronjob. Use coveoIncrementalIndexType for the Code attribute and incrementalCoveoSnIndexerJob for the Job Definition attribute. Update the FlexibleSearch query to match the incremental indexing requirements.

Step 9: Create the objectType field

In both created Index types, create a field that will be used to specify the Coveo object type.

  1. Click the required Index type, either coveoFullIndexType or coveoIncrementalIndexType.

  2. Switch to the Fields tab.

  3. For Coveo Push solution, the Code, Name, and DocumentId fields are mandatory.

  4. Click in the Fields menu, to create a field specific for Coveo.

    Note

    You can create as many additional fields as you need to push to Coveo. In this example, you’ll create one field, objectType.

  5. In the modal window that appears, fill in the following fields:

    Identifier

    Enter objectType.

    Name

    Enter Object Type.

    Field Type

    Select TEXT.

  6. Click Next.

  7. In the Indexer step, paste coveoObjectTypeSnIndexerValueProvider in the Value Provider field.

  8. Click Finish.

  9. In your SAP Commerce Cloud project, open the hybris/bin/custom/searchprovidercoveosearchservices/project.properties file.

  10. There will be three properties, each maps one Coveo Source object with one or more Composed Index types.

    For example, if you created a Warehouse Composed index type for the availability object, add it to the coveo.availability.typecodes property.

    coveo.availability.typecodes=Warehouse
coveo.product.typecodes=Product
coveo.variant.typecodes=VariantProduct

Step 10: Use the value providers

Your Coveo organization has a set of expected fields to be indexed, see Standard commerce fields. The values for these fields should follow the Coveo index format.

However, your SAP Commerce Cloud project might generate values whose formats are different from what the Coveo index expects, so this will lead to errors during indexing. To handle this, you can employ value providers to retrieve and change field names from the SAP Commerce Cloud database.

The Coveo SAP Commerce indexer push extension includes two sets of value providers:

  • Standard providers that are used to retrieve values from the database as is. For the list of provider names, see the example impex file in the extension code base.

  • Coveo value providers that transform the values before they’re pushed to Coveo. Examine the table below to learn more about the Coveo-specific value providers.

Table 1. Coveo-specific value providers
Provider name Provider ID Description

coveoObjectType​SnIndexerValueProvider

Important

It’s a mandatory provider for the Coveo indexer to work.

objectType

This value provider uses the following properties:

  • coveo.availability.typecodes

  • coveo.product.typecodes

  • coveo.variant.typecodes

Each property can be set in the local.properties file and is a comma-separated list of the composed item types in SAP Commerce Cloud that map to one of the 3 object types in the Coveo catalog.

coveoDocument​IdSnIndexerValueProvider

Important

It’s a mandatory provider for the Coveo indexer to work.

coveoDocumentId

This value provider builds a document ID that follows the URI format required by the Coveo Platform, {prefix}://{uniqueId}. The provider requires the valueProviderParameters to include a prefix value.

For example, if the provider has the valueProviderParameters set as prefix → product|expression → code, it will use the expression to use the code attribute of the product and generate the unique value in the form of product://12345.

coveoSimpleClickable​ProductUriSnIndexer​ValueProvider

coveoClickableUri

This value provider extends the default ProductUrlSnIndexerValueProvider to provide a full, clickable URL for the index. The default provider only gets the path of the product, and not the domain. The SAP Commerce Cloud convention is to have a property in the form of website.{storeId}.https which holds the value of the storefronts domain. By adding this value to that returned by the ProductUrlSnIndexerValueProvider, it’s possible to generate the full clickable URL for the index.

To do that, set the valueProviderParameters to siteId → {yourSiteID} with the correct property set up in your local properties.

Not required, but recommended.

coveoWarehouse​AvailableSkus​SnIndexerProvider

coveoAvailableSkus

This value provider is an example of how to send an array of attributes to your Coveo index. The provider gets all the products available within a warehouse and returns them as an array.

For the fieldType attribute of this provider, use the value COVEOARRAY.

coveoProductPriceToUser​PriceGroupSnIndexer​ValueProvider

priceToPriceGroup

This value provider is an example of how to generate a dictionary object of localized values to the index. The example has a user pricing group to price dictionary, where the currency will determine what data to send to each source.

For the qualifierTypeId attribute of this provider, the value currency is used. This informs the indexer about the mapping where the key of the map is the currency.

The value of the map is the dictionary to push to the index. In this example, it’s a map of user price group to price value.

coveoProduct​StockLevel​SnIndexerValueProvider

stockLevels

This value provider is an example of how to generate a standard dictionary object. The example gets the stock levels of a product for the warehouses it can be available in.

For the qualifierTypeId attribute of this provider, the value warehouse is used. This informs the indexer that the keys for the map will be the Warehouse composed type. This requires to use the CoveoWarehouseSnQualifierProvider to determine all the possible warehouses that can be a key for the map returned by the value provider.

coveoProduct​CategoryHierarchy​AttributeSnIndex​ValueProvider

coveoProduct​CategoryHierarchy

This value provider returns a value with the category hierarchy in the specific format required by the Coveo index. It leverages the OOTB ProductCategoryAttribute​SnIndexerValueProvider to get the correct category data, and then reconstructs the category paths in the format required. To do that, it uses the valueProviderParameters in the form of rootCategory → {rootCategoryId}|expression → {attribute}.

By default, the root category name is excluded from the paths you push as this will be common to all products in the index.

Step 11: Map the catalog fields

For both created Index types, map catalog fields with the Coveo fields. Do the mapping only for the fields that you want to push to Coveo.

  1. Navigate to the System → Search and Navigation → Index Types page.

  2. In the list of index types, click the required index type.

  3. Switch to the Fields tab.

  4. Examine the fields.

    Important

    Make sure there are only fields that you want to push to Coveo. The rest should be removed as they will add to the indexing time.

    Although leaving them in won’t cause the indexing to fail, it’s recommended to remove them for optimization.

  5. Examine the Identifier column. The values in it should match the field name that was set within the source in the Coveo Administration Console.

    See Manage source mappings and Mapping rule syntax reference.

    Example

    For the SAP field name, you need to create the mapping %[name] in the Coveo Administration Console.

    The properties of an index type showing the available fields ←→ An example of mapping within a Coveo Catalog source

  6. If you need to create a new field, do so the same way you created the objectType field in the previous step.

Step 12: Run the indexers

Once you’ve configured your system correctly, you can run your indexers.

  1. Navigate to the System → Search and Navigation → Index Types page.

  2. In the list of index types, click the required index type.

  3. Switch to the Cron Jobs tab.

  4. Click the required cronjob.

    1. Make sure that the cronjob is enabled, see the True checkbox in the Enabled section.

    2. On the Time Schedule tab, you can set the time when the cronjob should run.

      Important

      For the full indexer, it’s recommended to run it relatively often to ensure the consistency of the data between your SAP Commerce Cloud project and the Coveo index. For example, you can schedule the full indexer to run every two or three days, but the exact timeframe will depend on your judgment based on the number of products in your catalog.

      The exact time depends on the amount of data you have and how often it changes.

  5. Click Save to save and close the cronjob settings.

  6. While having the index type open, switch to the Administration tab.

  7. Check that the Stores field is set to the store you want to index.

    The properties of an index type highlighting the Stores field with the value 'Powertools store'

  8. Above the index tabs, click Run indexer.

  9. In the modal window that appears, click the dropdown menu and select the cronjob that you’ve just enabled.

  10. Click Run.

You can track the indexing progress by clicking Processes at the top of the Administration Cockpit.