Push API Tutorial 2: Manage Secured Content

This is for:

Developer System Administrator

In the previous tutorial, you created a public Push source in which you added, and then deleted, a simple item.

In this tutorial, you’ll configure a secured Push source and its associated security identity provider. You’ll then perform a small mock content update, followed by a small mock security identity update.

Step 0: Prerequisites

For this tutorial, you need:

  • A Coveo organization in which:

  • A Coveo Platform access token (OAuth2 token or API key) that grants you the following privileges in that organization (see Create an API key and Get your Coveo access token):

      | `owner`      | `targetDomain`      | `type` | Corresponding privilege in the Coveo Administration Console |   |--------------|---------------------|--------|----------------------------------------------------------------------|   | `PLATFORM`   | `ACTIVITIES`        | `VIEW` | **Organization - Activities - View**                                 |   | `PLATFORM`   | `ORGANIZATION`      | `VIEW` | **Organization - Organization - View**                               |   | `PLATFORM`   | `SECURITY_PROVIDER` | `EDIT` | **Content - Security identity providers - Edit**                     |   | `PLATFORM`   | `SOURCE`            | `EDIT` | **Content- Sources - Edit**                                          |   | `SEARCH_API` | `EXECUTE_QUERY`     |        | **Search - Execute queries**                                         |   | `SEARCH_API` | `VIEW_ALL_CONTENT`  |        | **Search - View all content**                                        |

  • If you click Run in Postman, you must set appropriate values for the organizationId and apiKey environment variables so that the API requests are executed successfully in Postman (see Manage Globals).

If you have completed the introductory tutorial, and the Coveo organization you created for that tutorial is still available, you can (and should) re-use it.

Step 1: Configure a Secured Push Source

A secured Push source allows you to manage items with permission models. The index uses a permission model to evaluate who can or can’t see a given item in query results.

Rather than specifying a distinct permission model for each individual item in a secured Push source, you could configure that source with a generic permission model which applies to each of its items.

This is a legitimate practice when the original secured content repository that you’re indexing has no item-based permissions (that is, when the permissions are the same for every item in that source).

This tutorial assumes that the permissions in its fictitious content repository are entirely item-based, which is why it requires that you configure a secured Push source with no permission model.

This tutorial step depends on what you have done before:

  • If you have not completed the previous tutorial, or if the public Push source you created in that tutorial is no longer available, use the Source API to create a new Push source with the following configuration in your Coveo organization:

    Request body

      {
    "sourceType": "PUSH",
    "name": "My Push API Tutorial Source",
    "sourceVisibility": "SECURED",
    "pushEnabled": true
  }

  • Otherwise, you should modify the configuration of your existing Push source rather than create a new one. To do so, you first need to know the unique identifier of your existing source. Once you have this information, send a PUT request to:

    https://platform.cloud.coveo.com/rest/organizations/<MyOrganizationId>/sources/<MySourceId>

    In the request path, replace <MyOrganizationId> with the unique identifier of your Coveo organization, and <MySourceId> with the unique identifier of your Push source. Use the same request body shown above.

Step 2: Create a Security Identity Provider

The security identity provider of a secured source is a Coveo organization module whose main purpose is to crawl a specific secured enterprise system on a regular basis to retrieve its latest security identity relationships and forward this data to the security identity cache (see Security Identity Cache and Provider). A security identity provider allows the items in a secured source to reference existing and valid security identities in their permission models. This ensures that only the users who are allowed to see a specific item in its original secured repository can see this item in their query results (see Typical Coveo Secured Search).

Use the Security Cache API to create a security identity provider with the following configuration for the secured Push source you configured in step 1:

Request body (template)

{
  "name" : "My Push API Tutorial Security Identity Provider",
  "nodeRequired": false,
  "type": "EXPANDED",
  "referencedBy": [
    {
      "id": <MyPushSourceId>,
      "type": "SOURCE"
    }
  ],
  "cascadingSecurityProviders": {
      "EmailSecurityProvider": {
        "name": "Email Security Provider",
        "type": "EMAIL"
    }
  }
}

In the template above, you must replace <MyPushSourceId> with the actual ID of the secured Push source you configured in step 1.

Initially, your security identity provider has no means of automatically crawling its intended target secured content repository. You must eventually implement this behavior with your own code, but that’s beyond the scope of this tutorial.

Step 3: Add Items to Your Source

A secured source is a Coveo organization module whose main purpose is to crawl the content of a specific secured enterprise system on a regular basis to retrieve its items, along with their corresponding metadata and permission models, and forward this information to an index.

Perform three Push API calls to add the following items in the secured Push source you configured in step 1. Use the PUT /sources/{sourceId}/documents HTTP request.

  1. The permissions property of the following request body indicates that any anonymous user can see this item in their query results:

    Request body 1

     {
   "data": "",
   "permissions": [
     {
       "allowAnonymous": true
     }
   ]
 }

    When pushing the above item, you must set the documentId query parameter to file://folder/ (this URI doesn’t exist, but that doesn’t matter as you’re only pushing dummy items for this tutorial).

  2. The permissions property of the following request body indicates that only the users asmith@example.com and bjones@example.com can see this item in their query results (you’ll define these security identities in step 4):

    Request body 2 (template)

     {
   "data": "Item 1 raw textual data",
   "permissions": [
     {
       "allowAnonymous": false,
       "allowedPermissions": [
         {
           "identity": "asmith@example.com",
           "identityType": "User"
         },
         {
           "identity": "bjones@example.com",
           "identityType": "User"
         }
       ]
     }
   ]
 }

    When pushing the above item, you must set the documentId query parameter to file://folder/item1.txt (this URI doesn’t exist, but that doesn’t matter as you’re only pushing dummy items for this tutorial).

  3. The permissions property of the following request body indicates that only the user asmith@example.com can see this item in their query results (you’ll define security identities in step 4). The user bjones@example.com is both allowed and denied in the permission model, which means that this user is ultimately denied (see Simple Permission Model Definition Examples):

    Request body 3

     {
   "data": "Item 2 raw textual data",
   "permissions": [
     {
       "allowAnonymous": false,
       "allowedPermissions": [
         {
           "identity": "asmith@example.com",
           "identityType": "User"
         },
         {
           "identity": "bjones@example.com",
           "identityType": "User"
         }
       ],
       "deniedPermissions": [
         {
           "identity": "bjones@example.com",
           "identityType": "User"
         }
       ]
     }
   ]
 }

    When pushing the above item, you must set the documentId query parameter to file://folder/item2.txt (this URI doesn’t exist, but that doesn’t matter as you’re only pushing dummy items for this tutorial).

You may have to wait several minutes before these three items become available in your source (see About the Push API Processing Delay).

You’ll learn how to push batches of items along with metadata and complex permission models in the next tutorial.

Step 4: Add Security Identities in Your Security Identity Provider

The permission models of the second and third items you pushed in step 3 reference security identities (the users asmith@example.com and bjones@example.com) which have not yet been defined in the security identity provider you created in step 2. This means that once the indexing pipeline finishes processing those items, the security identities they refer to will be in error, and the allowed users won’t be able to see the items in their query results.

To fix this, you must first perform two distinct Push API calls to add the asmith@example.com and bjones@example.com mock security identities in the security identity provider you created in step 2 (see Add or Update a Single Security Identity):

  1. This request body defines the asmith@example.com user security identity:

    Request body 1

     {
   "identity": {
     "name": "asmith@example.com",
     "type": "USER"
   }
 }

  2. This request body defines the bjones@example.com user security identity:

    Request body 2

     {
   "identity": {
     "name": "bjones@example.com",
     "type": "USER"
   }
 }

You’ll learn how to define batches of security identities (including aliases, groups, and granted identities) in the next tutorial.

Step 5: (Optional) Validate the Indexing Status of the Items

Since the items you added in your secured Push source in step 3 have permission models, you can’t perform normal queries (either authenticated as your own security identity in Coveo, or as an anonymous user) to monitor their status. You would only be allowed to see the file://folder item, as its permission model allows anonymous users.

To see all three items in your query results, you must set the viewAllContent query parameter to 1 when you perform a query. Assuming that the access token you’re using has the privilege to view all of the content in the target organization (that is, the Allowed access level on the View all content domain), the query will bypass the item permission models. You’ll then be able to see all of the items in your query results, no matter which security identity you’re using to authenticate the query (see Manage privileges and View All Content Domain). The viewAllContent feature is useful when debugging.

Perform the following query to monitor the indexing status of your items (replace <MyOrganizationId> with the unique identifier of your Coveo organization):

https://platform.cloud.coveo.com/rest/search/v2?organizationId=<MyOrganizationId>&aq=@source==\"My Push API Tutorial Source\"&maximumAge=0&viewAllContent=1

Alternatively, you can use the Content Browser in the Coveo Administration Console to monitor your pushed items (see Browse All the Content of Your Organization Index).

Step 6: Delete the Items

To begin the next tutorial with a clean source, use the DELETE /sources/{sourceId}/documents HTTP request to delete the three items you added in step 3.

To do so, delete the file://folder item and set the deleteChildren parameter to true when you perform the delete operation, which recursively deletes every item whose documentId is nested under the documentId you delete (see Deleting an Item and Optionally, its Children in a Push Source). In this case, you’ll delete the file://folder item, along with all other items matching file://folder/* in the target Push source (that is, file://folder/item1.txt and file://folder/item2.txt). If there were additional matching subfolders/items in the Push source (for example, file://folder/subfolder, file://folder/subfolder/item3.txt, etc.), they would be deleted as well.

Once the call has successfully returned (202 Accepted), you can use the Source API to validate that the items have been deleted (see Get Detailed Information About a Source). If this is the case, the numberOfDocuments property value of your Push source should be back to 0.

Step 7: Disable the Security Identities

By default, when you perform a Push API operation, the service automatically sets the optional orderingId parameter of this operation to the current number of milliseconds since the UNIX epoch (see About the orderingId Parameter).

This means that each individual Push API operation has its own timestamp value, which can be compared to the mandatory orderingId parameter provided in a Delete old security identities or Delete old items operation.

To begin the next tutorial with a clean security identity provider, you must use the Push API to disable all security identities which are older than the current number of milliseconds since the UNIX epoch from the security identity provider you created in step 4.

You can then refresh your security identity provider.

What’s Next?

You’re now ready to proceed to Push API Tutorial 3: Using Batch Calls to Manage Complex Secured Content.