Push data to Coveo (SAP Commerce Cloud 2205 or later)

This is for:

Developer System Administrator

This article explains how to push data from an SAP Commerce Cloud (version 2205 or later) to a Coveo organization.

Prerequisites

You’ll need the Coveo Push API client library for Java, so make sure that:

Install the Coveo indexer push extension

The Coveo Search Provider SAP Commerce Extension allows your SAP project to exchange data with Coveo. To install the extension, perform the following steps:

  1. Copy the extension repository to the hybris/bin/custom directory of your project.

  2. Open the hybris/config/localextensions.xml file and add the searchprovidercoveosearchservices extension:

    <extension name='searchprovidercoveosearchservices' />

  3. Install the Coveo Push API client library for Java:

    1. Open the Apache Maven settings.xml file. The file is located at ${user.home}/.m2/settings.xml.

    2. Add a repository definition for the library’s GitHub Package.

      <repository>
    <id>github</id>
    <url>https://maven.pkg.github.com/coveo/push-api-client.java</url>
    <snapshots>
        <enabled>true</enabled>
    </snapshots>
</repository>

    3. Specify your GitHub personal access token (classic) to be able to install packages from GitHub Packages.

      Important

      The token should have at least read:packages permission.

      		 
      <servers>
<server>
    <id>github</id>
    <username>USERNAME</username>
    <password>TOKEN</password>
</server>
</servers>

  4. Set up the Ant environment by running the script:

    . ./setantenv.sh

  5. From the root of your project directory, run the following command:

    ant clean all

  6. After the command execution, navigate to the hybris/bin/platform directory.

  7. Start your SAP Commerce installation by running the script:

    ./hybrisserver.sh

    By default, your local server would use ports 9001 (http) and 9002 (https).

  8. Open the Hybris Administration console at https://localhost:9002/platform/update. You can also navigate to the Update page by clicking Platform → Update in the top menu.

  9. Find and select the checkboxes for:

    • searchprovidercoveosearchservices (required to create the new search provider)

    • Update running system

    • Localize types

  10. At the top of the page, click Update.

Note

The extension creates new data types in the database, but it doesn’t import any data.
Tip
Semi-automatic configuration

In the extension code base, if you navigate to resources/examples, you’ll find the electronics-search-configuration-example.impex file. This is an example impex of how you can perform the SAP Commerce Cloud configuration in a programmatic way. This can be used as a template to fill with your data and configure your project faster.

You can also manually configure every source and your indexes via the Backoffice. Perform the following steps to do so.

Coveo setup

Step 1: Create a source in your Coveo organization

For storing the SAP catalog data, you need a Catalog source.

  1. Log in to the Coveo Administration Console.

  2. Create the Catalog sources that you’ll be pushing your data to.

    The guidelines for creating sources are:

    • Each combination of language, currency, and country existing in your data should have a separate Coveo Catalog source to push to.

    • Any availability data you have should also have it’s own dedicated Catalog source.

  3. Take note of the Stream API URL value for your source.

Step 2: Create an API key

Create one API key that you’ll use for all your sources. See Manage API Keys.

The key should have the edit privileges detailed here: Catalog source privileges. See the Edit sources, edit source update schedules, and view the View Metadata page row in the privileges table.

SAP Commerce setup

Step 1: Create Coveo sources

For each source you created in the Coveo Administration Console you should perform the following steps in the SAP Backoffice.

  1. In the SAP Backoffice Administration Cockpit, go to System → API → Destinations → Coveo Source.

  2. Click the plus sign (+) above the search results.

  3. In the modal window that appears, enter a unique ID that represents your Catalog source. It’s recommended to use a human-readable value to assist with identifying a specific source.

  4. Click Finish.

  5. Select the newly created item from the search listing.

  6. On the General tab:

    1. Fill in the Name field.

    2. In the Object type menu, specify the type that matches your Coveo source:

      AVAILABILITY

      This type is for the availability source. See Source configuration approaches for availability channel and Catalog availability data.

      PRODUCTANDVARIANT

      This type is for the the products and variations of the catalog. See Catalog product data and Catalog variant data and product groupings.

      Important

      When selecting an object type of PRODUCTANDVARIANT, Language, Currency, and Country are mandatory for the system to determine the correct source for your product data.

  7. Click Save.

Step 2: Create a consumed destination

  1. Go to System → API → Destinations → Consumed Destinations.

  2. Click the plus sign (+) above the search results.

  3. In the modal window that appears:

    ID

    Enter a unique ID that represents your Catalog source.

    URL

    Paste the Stream API URL that you copied from the Coveo Catalog source.

    Details

    1. In your Coveo organization, open the Catalog source you created in step 2.

    2. On the Configuration tab, there’s a section called Stream API URL. Click Copy to clipboard to copy the URL.

    Active

    Verify that the Active checkbox is selected.

    Destination Target

    Click the field and select Default_Template.

    Note

    The destination target you select doesn’t matter as it’s not used in the push implementation and only needed to create the destination.

  4. Click Finish.

    The properties of a new consumed destination

Now you see the destination you created in the list of consumed destinations.

Step 3: Create a credential for the consumed destination

Important

If you have a number of sources, you only need to create one OAuth Credential and attach it to many Consumed Destinations.

  1. Click the destination you created and switch to the Destination Configuration tab.

  2. Double-click in the Credential field to open a dropdown menu and select Create Credential → Consumed OAuth Credential.

  3. Fill in the following fields:

    ID

    Enter a unique ID for a new credential. For example, CoveoCredential.

    Client ID

    Enter a client ID for a new credential. For example, CoveoClientID.

  4. Click Finish. You see the Destination Configuration tab again.

  5. Double-click in the Credential field to open the credential settings.

  6. In the Client Secret section, there are Password and Verify password fields. In both fields, paste the Coveo API key that you created earlier.

  7. Click Save to save and close the credential settings.

  8. Click Save to save the destination settings.

The properties of a Coveo credential with the password filled in

Step 4: Assign the consumed destination to the Coveo source

  1. Return to System → API → Destinations → Coveo Source.

  2. Click your Coveo source.

  3. On the General tab, find the Consumed Destination field.

  4. Specify the consumed destination that you created earlier.

  5. Click Save.

Step 5: Create a search provider

Now that you’ve mapped the Coveo sources to the SAP Commerce API destinations, you need to configure the search index to push the data to Coveo.

  1. Go to the System → Search and Navigation → Search Provider Configurations page.

  2. Next to the plus sign, click the dropdown menu and select Coveo Search Provider Configuration.

  3. Fill in the following fields:

    Identifier

    Enter a unique identifier for a new search provider, for example, CoveoSearchProviderID.

    Name

    Enter a name for a new search provider, for example, Coveo Search Provider Configuration.

    Coveo Source

    Select all the Coveo sources that you created.

  4. Click Finish. Now you see the provider you created in the list of search providers.

Step 6: Create the index configuration

  1. Go to System → Search and Navigation → Index Configurations.

  2. Next to the plus sign (+), click the dropdown menu and select CoveoSnIndexConfiguration.

  3. Fill in the following fields:

    Identifier

    Enter a unique identifier for a new configuration, for example, CoveoIndexID.

    Name

    Enter a name for a new search provider, for example, Coveo Index Configuration.

    Search Provider Configuration

    Select the Coveo Search Provider Configuration created in the previous step.

  4. Click Next to proceed to the next step, Index Types.

    Skip this step for now and click Next again.

  5. In the Session step, select all the Languages, Currencies, and Countries that are relevant for your data.

  6. Click Finish.

Step 7: Create the ServiceLayerJobs

Create the ServiceLayerJobs for the future cron jobs, one for full indexing and one for incremental indexing.

  1. Go to System → Types.

  2. In the search bar, type in ServicelayerJob and search.

  3. In the search results, click the ServicelayerJob.

    The properties of the ServicelayerJob are displayed on the page.

  4. In the upper part of the properties, click the search icon that’s titled Search by type.

  5. Click the plus sign (+) above the search listing.

  6. In the modal window that appears:

    Spring ID

    Paste fullCoveoSnIndexerJob.

    Code

    Paste fullCoveoSnIndexerJob.

  7. Click Finish.

  8. Perform the same steps to create another ServicelayerJob for which specify incrementalCoveoSnIndexerJob for the Code and the Spring ID attributes.

Step 8: Create the Index types

Create one Index type for full indexing and one for incremental indexing.

  1. Go to System → Search and Navigation → Index Types.

  2. Click the plus sign (+) above the search results.

  3. In the modal window that appears, in the Essential step, fill in the following fields:

    Identifier

    Enter a unique identifier for a new type, for example, CoveoFullIndexTypeID.

    Name

    Enter a name for a new type, for example, Coveo Full Index Type Configuration.

    Composed Type

    Specify the type that matches a Coveo Object type for the source you’ll use this to push data to.

    Example

    If this index type will be used to push availability objects, then you may want to select the Warehouse Composed type. This will be dependent on how your data is structured.

    Index Configuration

    Select the Coveo Index Configuration created in the previous steps.

  4. The next steps can be configured later, so you may skip them for now. Click Next to proceed to the Cron Jobs step.

  5. In the Cron Jobs step, click the plus sign (+) at the end of the dropdown menu.

  6. Select Full Indexer Cronjob.

  7. Fill in the following fields:

    • Code. Enter a unique code for the cron job, for example, coveoFullIndexType.

    • Job Definition. Select the fullCoveoSnIndexerJob system type. The chosen type should correspond to the type of cron job you’re creating.

    • Indexer Item Source. Click this field to create a Flexible Search Indexer Item Source.

      In the modal window that appears, fill in the Query field. It must be the FlexibleSearch query that will be used to retrieve the Composed Type. For example, if you selected Warehouse for your availability object, the following query would retrieve all warehouses for the electronics site:

      `SELECT {w:PK} FROM {Warehouse as w JOIN BaseStore2WarehouseRel as rel ON {w:PK} = {rel:target} JOIN BaseStore AS bs ON {rel:source} = {bs:PK} } WHERE {bs:uid}='electronics'`

      For product indexing, the following query would retrieve all approved products in the catalog.

      `SELECT {p:pk} FROM {Product AS p LEFT JOIN ArticleApprovalStatus AS a ON {p:approvalStatus} = {a:pk}} WHERE {a:code} = 'approved'`

  8. Click Finish for every modal window that has been opened.

  9. Select the newly created Index type.

  10. On the General tab, configure the Identity Provider as snIdentityProvider and link the catalog you want to index in the Catalogs field.

  11. Perform the same steps to create an Index type for incremental indexing, but adjust them as follows:

    • in the Essential step, update the Identifier and Name fields to reflect the incremental indexing type.

    • in the Create Cron Jobs step, select Incremental Indexer Cronjob. Use coveoIncrementalIndexType for the Code attribute and incrementalCoveoSnIndexerJob for the Job Definition attribute. Update the FlexibleSearch query to match the incremental indexing requirements.

Step 9: Create the objectType field

In both created Index types, create a field that will be used to specify the Coveo object type.

  1. Click the required Index type, either coveoFullIndexType or coveoIncrementalIndexType.

  2. Switch to the Fields tab.

  3. For Coveo Push solution, the Code, Name, and DocumentId fields are mandatory.

  4. Click in the Fields menu, to create a field specific for Coveo.

    Note

    You can create as many additional fields as you need to push to Coveo. In this example, you’ll create one field, objectType.

  5. In the modal window that appears, fill in the following fields:

    Identifier

    Enter objectType.

    Name

    Enter Object Type.

    Field Type

    Select TEXT.

  6. Click Next.

  7. In the Indexer step, paste coveoObjectTypeSnIndexerValueProvider in the Value Provider field.

  8. Click Finish.

  9. In your SAP Commerce Cloud project, open the hybris/bin/custom/searchprovidercoveosearchservices/project.properties file.

  10. There will be three properties, each maps one Coveo Source object with one or more Composed Index types.

    For example, if you created a Warehouse Composed index type for the availability object, add it to the coveo.availability.typecodes property.

    coveo.availability.typecodes=Warehouse
coveo.product.typecodes=Product
coveo.variant.typecodes=VariantProduct

Step 10: Map the catalog fields

For both created Index types, map catalog fields with the Coveo fields. Do the mapping only for the fields that you want to push to Coveo.

  1. Navigate to the System → Search and Navigation → Index Types page.

  2. In the list of index types, click the required index type.

  3. Switch to the Fields tab.

  4. Examine the Identifier column. The values in it should match the field name that was set within the source in the Coveo Administration Console.

    See Manage source mappings and Mapping rule syntax reference.

    Example

    For the SAP field name, you need to create the mapping %[name] in the Coveo Administration Console.

    The properties of an index type showing the available fields ←→ An example of mapping within a Coveo Catalog source

  5. If you need to create a new field, do so the same way you created the objectType field in the previous step.

Step 11: Run the indexers

Once you’ve configured your system correctly, you can run your indexers.

  1. Navigate to the System → Search and Navigation → Index Types page.

  2. In the list of index types, click the required index type.

  3. Switch to the Cron Jobs tab.

  4. Click the required cronjob.

  5. Make sure that the cronjob is enabled, see the True checkbox in the Enabled section.

  6. Click Save to save and close the cronjob settings.

  7. While having the index type open, switch to the Administration tab.

  8. Check that the Stores field is set to the store you want to index.

    The properties of an index type highlighting the Stores field with the value 'Powertools store'

  9. Above the index tabs, click Run indexer.

  10. In the modal window that appears, click the dropdown menu and select the cronjob that you’ve just enabled.

  11. Click Run.

You can track the indexing progress by clicking Processes at the top of the Administration Cockpit.