Push API Tutorial 3: Using Batch Calls to Manage Complex Secured Content
Push API Tutorial 3: Using Batch Calls to Manage Complex Secured Content
In the previous tutorial, you configured a secured Push source along with its associated security identity provider, and you learned how to push simple secured items and security identities individually.
You also learned how to delete an item and its children, as well as how to delete items or security identities based on the orderingId value of the last operation that was performed on them.
In this tutorial, you’ll create fields in your index and define custom mapping rules for your secured Push source. You’ll then use Push API batch operations to simulate a security identity update (including different types of security identities), followed by a source content update (including coherent item metadata and complex permission models). Finally, you’ll use the Coveo Administration Console to validate whether everything is working as expected.
Prerequisites
For this tutorial, you need:
-
A Coveo organization that minimally contains:
-
A secured Push source (see Create a Push source).
-
A security identity provider associated with that secured Push source, which must cascade to the
Email Security Provider(see Create a security identity provider for a secured Push source).
-
-
A Coveo Platform access token (OAuth2 token or API key) that grants you the following privileges in that organization (see Create an API key and Get your Coveo access token):
owner targetDomain type Coveo Administration Console equivalent PLATFORMORGANIZATIONVIEWOrganization - Organization - View
PLATFORMSECURITY_PROVIDEREDITContent - Security identity providers - Edit
PLATFORMSOURCEEDITContent - Sources - Edit
PLATFORMFIELDEDITContent - Fields - Edit
-
If you click Run in Postman, you must set appropriate values for the
organizationIdandapiKeyenvironment variables so that the API requests are executed successfully in Postman (see Manage Globals).
|
If you have completed the previous tutorial and the Coveo organization, secured Push source, security identity provider, and Coveo access token you used in that tutorial are still available, you can (and should) re-use them. You should also ensure that the secured Push source that you use for this tutorial contains no previously indexed items (see Deleting old items in a Push source). |
Step 1: Create fields
Fields let you store and leverage item metadata, allowing you to create facets, define custom query result sort criteria, etc.
It’s important to remember that fields are index-wide data containers, which implies that any source can theoretically populate a given field with item metadata.
The title standard field in your index typically gets populated by item metadata from many sources.
In this tutorial, the items you’ll add in your secured Push source are movie entries, so you’ll start by creating several fields to store and leverage movie-related metadata.
You’ll mark these as specific to your source by prefixing them with mmeps (which stands for My Movie Entry Push Source).
|
|
Leading practice: Prefix your fields
While prefixing a field doesn’t change the fact that it’s an index-wide container, it allows you to quickly identify which fields are semantically tied to a specific source. |
The JSON below defines 12 fields which you’ll populate with metadata when you add items in your secured Push source in step 3.
You won’t notice that additional Boolean options (for example, facet, sort, and useCacheForSort) are specified for some of these fields, but they’re not relevant in this tutorial.
For now, use the Fields API to create the following batch of fields in your index (see Creating fields):
|
Request body |
|
|
Note
In the request body above, some fields have the For example, since the |
Step 2: Create custom mapping rules
By default, a Push source automatically attempts to map incoming item metadata to a field of the corresponding name in your index. This means that when you push an item with a metadata key-value pair, the Coveo indexing pipeline will use the metadata value to populate a field whose name matches the metadata key, if such a field exists.
Assuming that your Push source has no custom mappings, pushing an item with the "mmepstitle": "Gone with the Wind" metadata key-value pair automatically populates the mmepstitle field in your index with the "Gone with the Wind" value for this item.
You’ll now create several custom mapping rules for your secured Push source. These rules will complement the default Push source mapping behavior when the indexing pipeline processes incoming items in your source.
The JSON below defines four custom common mapping rules:
-
The first rule maps the
titlestandard metadata to themmepstitlefield you created in step 1. This ensures that thetitlestandard field will also get populated by thetitlestandard metadata value. -
The second rule is similar to the first one, except that it maps the
durationstandard metadata to themmepsdurationfield you created in step 1.
|
|
Note
When these two rules apply, the It can be useful to duplicate standard fields this way, because it allows you to use prefixed fields to create facets whose values come only from a specific source. |
-
The third rule appends the
" million USD"string to the value of themmepsbudgetmetadata before mapping this metadata to the field of the same name, which you created in step 1. -
The fourth rule maps the
mmepsmpaafilmratingmetadata to the field of the same name, which you created in step 1. However, it also includes an alternative: if an incoming item doesn’t have anymmepsmpaafilmratingmetadata (that is, if a movie is unrated), themmepsmpaafilmratingfield for this item is populated by the"UR"string instead.
Use the Source API to configure the mappings of your Push source as follows (see Manage the mapping configuration of a source):
|
|
Request body |
Step 3: Add a batch of items
You’ll now use a batch operation to add several items at once in your secured Push source.
|
|
Note
In a real-life scenario, you would use your own code to retrieve items along with their corresponding metadata and permission models from a secured enterprise system. |
In the JSON below, each element in the addOrUpdate array represents a movie entry to add in your secured Push source.
Every entry contains standard metadata (for example, documenttype, filename, etc.), as well as custom metadata (for example, mmepsdirector, mmepsyear, etc.).
This metadata will populate fields in your index according to the default Push API mapping behavior and the custom mapping rules that you defined in step 2.
Since you’re pushing items in a secured source, each item must have a permission model. You’ll define the security identities on which these permission models rely in step 4.
In this tutorial, the permission model of an item can have many permission levels and permission sets. The first permission level is based on the default MPAA film rating. The second permission level is based on some other form of film rating, should a movie not yet have been rated by the MPAA. The third permission level contains parental approval information, and it’s further divided into two permission sets (one for each parent).
The movie whose title is 1900 (the last item in the JSON below) is unrated by the MPAA (that is, it doesn’t have any mmepsmpaamovierating metadata).
As a result, the first permission level, which is based on the MPAA film rating, only allows adults (Grace, Henry, and yourself). Permissions for young children, pre-teens, and teenagers are still unknown at this level.
The second permission level is based on some other form of film rating. It explicitly denies young children (Alice and Bob) and pre-teens (Chloe and David). Permissions for teenagers (Erin and Frank) are still unknown at this level.
The third permission level, which contains parental approval information, indicates that:
-
Chloe (a pre-teen) is explicitly allowed by both permission sets (that is, both of her parents).
However, since pre-teens were explicitly denied in the previous permission level, Chloe won’t be able to see the 1900 movie entry in her query results.
-
Erin (a teenager) is also explicitly allowed by both permission sets.
Since teenager permissions were unknown in previous permission levels, Erin will be able to see the 1900 movie entry in her query results.
-
Frank (also a teenager) is explicitly denied in the first permission set and explicitly allowed in the second one.
Since he isn’t allowed in all of the permission sets at this permission level (that is, one of his parents doesn’t want him to see this movie entry), Frank won’t be able to see the 1900 movie entry in his query results.
In conclusion, the only users who are allowed to see the 1900 movie entry in their query results are Erin, Grace, Henry, and yourself.
|
|
Note
Permission levels are always evaluated sequentially. This means that explicitly denying one of the adult users (Grace, Henry, or yourself) in the second or third permission levels wouldn’t ultimately result in this user being denied. As soon as a security identity is explicitly allowed by all permission sets of a lower permission level, it can’t be denied by any higher permission levels. |
Push the following batch of items in your secured Push source (see Manage batches of items in a Push source):
|
|
Now would be a good time to update the status of your Push source to If you do, remember to set it back to |
|
|
Request body |
You may have to wait several minutes before these items become available in your source (see About the Push API processing delay).
Step 4: Add a batch of security identities
Whenever you use the Push API to add, update, and/or delete items in a secured Push source, you should ensure that the security identity definitions to which these items refer in their permission models are up to date in the security identity cache of your Coveo organization. Otherwise, certain users may not be able to see some of the items for which they should have access. Even worse, other users may be able to see items for which they should no longer have access.
In this step, you’ll use a batch operation to define several security identities at once. The items that you added in your secured Push source in step 3 refer to these security identities in their permission models.
|
|
Note
In a real-life scenario, you would use your own code to extract security identities and security identity relationships from a secured enterprise system. |
The JSON below defines 9 individual users (Alice, Bob, Chloe, David, Erin, Frank, Grace, Henry, and yourself), four {groups} (YoungChildren, PreTeenagers, Teenagers, and Adults), and one granted identity (Everyone).
It also defines an alias to your own security identity in Coveo (for this to work, you must replace <MyEmailAddress> towards the end of this JSON with the actual email address with which you authenticate in Coveo).
|
Typically, you should define an alias relationship between each user security identity and its corresponding security identity in the Email Security Provider (see About the email security provider). This means that you’ll normally define each user in the This tutorial doesn’t define aliases for any of the fake users (for example, |
Every user belongs to a single group, as summarized in the following table:
| User | Group |
|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Each group also has the Everyone granted identity.
Push the following batch of security identities in the security identity provider of your secured Push source (see Manage batches of security identities):
|
|
Request body |
|
|
In the request body above, don’t forget to replace |
Step 5: (Optional) Validate item field values, effective permissions, and data
You’ll now use the Coveo Administration Console to manually verify that the metadata of the items you just pushed has populated the fields in your index as expected, and that the permission models of your items are being evaluated correctly.
-
In the Coveo Administration Console, access the Content Browser (platform-ca | platform-eu | platform-au) page.
-
In the search box, type
@source==<MyPushSource>(replace<MyPushSource>with the name of your secured Push source) and select Enter.The six items that you pushed in your source should appear in the result list.
-
Select the result whose title is
1900, and then click Properties in the Action bar.A panel should open in which the Fields tab is selected by default. You should see a list of mapped fields for this item, along with their corresponding values. You can now validate that:
-
The
titleandmmepstitlefields have the same value (1900), which means that the first mapping rule you defined in step 2 was correctly applied. -
The
durationandmmepsdurationfields have the same value (175.92), which means that the second mapping rule you defined in step 2 was correctly applied. -
The
" million USD"string was appended to the end ofmmepsbudget, which means that the third rule you defined in step 2 was correctly applied. -
The value of the
mmepsmpaafilmratingfield was set to the"UR"string, which implies that the fourth mapping rule you defined in step 2 was correctly applied, since this item was pushed without anymmepsmpaafimratingmetadata.
-
-
In the header of the panel, select the Permissions tab.
You should see the list of permissions for the
1900item. As explained in the example in step 3:-
The only allowed users are Erin, Grace, Henry, and yourself.
-
The
YoungChildrenandPreTeenagersgroups, as well as the user Frank, are explicitly denied.
-
-
In the header of the panel, select the Permission details tab.
You should see that this item has three permission levels. The first two levels have only one permission set, whereas the last one has two. As explained in the example in step 3:
-
Permission Level 0(that is, the MPAA film rating-based permission level) explicitly allows theAdultsgroup. -
Permission Level 1(that is, the additional film rating-based permission level) explicitly denies theYoungChildrenandPreTeenagersgroups. -
Permission Level 2(that is, the parental approval permission level) explicitly allows the user Erin, and denies the user Frank.
-
-
In the header of the panel, select the Quick view tab.
The HTML and CSS that was provided as the value of the
Dataproperty for this item should render as expected.
You can validate the other items that you added in your secured Push source in step 3 in the same fashion.
Very useful
Not really
Very useful
Not really