Push API Tutorial 3 - Using Batch Calls to Manage Complex Secured Content

In the previous tutorial (see Push API Tutorial 2 - Managing Secured Content), you configured a secured Push source along with its associated security identity provider, and learned how to push simple secured items and security identities individually. You also learned how to delete an item and its children, and how to delete items or security identities based on the orderingId value of the the last operation that was performed to add or update this content.

In this tutorial, you will create fields in your index and define custom mapping rules for your secured Push source. You will then use Push API batch operations to simulate a security identity update (including different types of security identities), followed by a source content update (including coherent item metadata and complex permission models). Finally, you will use the Coveo Cloud V2 administration console to validate whether everything is working as expected.

Step 0 - Prerequisites

For this tutorial, you need a Coveo Cloud V2 organization (see Creating a Coveo Cloud V2 Organization) that minimally contains:

owner
targetDomain
type
Corresponding privilege in the Coveo Cloud V2 administration console
PLATFORM ORGANIZATION VIEW Organization - Organization - View
PLATFORM SECURITY_PROVIDER VIEW Content - Security identity providers - View
PLATFORM SECURITY_PROVIDER EDIT Content - Security identity providers - Edit
PLATFORM SOURCE VIEW Content - Sources - View
PLATFORM SOURCE EDIT Content- Sources - Edit

you have completed the previous tutorial (see Push API Tutorial 2 - Managing Secured Content), you can (and should) re-use the Coveo Cloud V2 organization, secured Push source, security identity provider, and Coveo Cloud V2 platform access token you used in that tutorial, if they are still available.

Preferably, you should ensure that the secured Push source you use for this tutorial contains no previously index items (see Deleting Old Items in a Push Source).

Step 1 - Create Fields

A field allows you to store and leverage item metadata, which is especially useful when you want to create facets, sort query results, etc.

It is important to remember that fields are index-wide data containers, which implies that any source can theoretically populate a given field with item metadata.

The title standard field in your index typically gets populated by item metadata from multiple sources.

In this tutorial, the items you will add in your secured Push source are movie entries. Therefore, you will start by creating a few fields to store and leverage movie-related metadata. You will mark those as specific to your source by prefixing them with mmeps (which stands for My Movie Entry Push Source).

While prefixing a field does not actually change the fact that it is an index-wide container, doing so allows you to quickly identify which fields are semantically tied to a specific source.

The JSON below defines 12 fields which you will populate with metadata when you add items in your secured Push source in Step 3. You will notice that additional boolean options, such as facet, sort, or useCacheForSort, are specified for some of these fields. Do not worry about those options for now: they will be put to good use in more advanced tutorials.

For now, simply use the Fields API to create the following batch of fields in your index (see Creating Fields):

Field creation request body

[
  {
    "name": "mmepstitle",
    "description": "The title of the movie.",
    "type": "STRING",
    "facet": true,
    "sort": true,
    "useCacheForSort": true
  },
  {
    "name": "mmepsyear",
    "description": "The year the movie was released in.",
    "type": "LONG"
  },
  {
    "name": "mmepsduration",
    "description": "The duration of the movie, in seconds",
    "type": "DOUBLE"
  },
  {
    "name": "mmepssummary",
    "description": "A brief summary of the movie.",
    "type": "STRING"
  },
  {
    "name": "mmepsisseries",
    "description": "Whether the movie is part of a series.",
    "type": "STRING"
  },
  {
    "name": "mmepsmpaafilmrating",
    "description": "The MPAA film rating for the film. Can be either 'G', 'PG', 'PG-13', 'R', 'NC-17', or 'UR'.",
    "type": "STRING",
    "facet": true
  },
  {
    "name": "mmepsbudget",
    "description": "The movie budget, in millon USD.",
    "type": "STRING"
  },
  {
    "name": "mmepscriticsaverage",
    "description": "The average score that was given by critics to the movie, on a discrete scale of 1 to 100.",
    "type": "LONG",
    "useCacheForComputedFacet": true
  },
  {
    "name": "mmepsdirector",
    "description": "The director of the movie.",
    "type": "STRING",
    "facet": true,
    "sort": true,
    "useCacheForSort": true,
    "mergeWithLexicon": true,
    "useCacheForNestedQuery": true
  },
  {
    "name": "mmepscastoverview",
    "description": "The list of principal actors that appear in the movie.",
    "type": "STRING",
    "multiValueFacet": true,
    "multiValueFacetTokenizers": ";",
    "mergeWithLexicon": true,
    "useCacheForNestedQuery": true
  },
  {
    "name": "mmepsthumbnaillink",
    "description": "The URI of the thumbnail image for the movie.",
    "type": "STRING"
  },
  {
    "name": "mmepstrailerlink",
    "description": "The URI of the movie trailer.",
    "type": "STRING"
  }
]

Step 2 - Create Custom Mapping Rules

By default, a Push source automatically attempts to map incoming item metadata to a field of the corresponding name in your index. This means that if you push an item along with a certain metadata key-value pair, the Coveo Cloud V2 indexing pipeline will use the metadata value to populate a field whose name matches the metadata key, if such a field exists.

Assuming that your Push source has no custom mappings, pushing an item with the "mmepstitle": "Gone with the Wind" metadata key-value pair automatically populates the mmepstitle field in your index with the "Gone with the Wind" value for this item.

You will now create several custom mapping rules for your secured Push source. Those rules will complement the default Push source mapping behavior when the Coveo Cloud V2 indexing pipeline processes incoming items in your source.

The JSON below defines 4 custom common mapping rules:

  • The first rule maps the title standard metadata to the mmepstitle field you created at Step 1. This rule complements the default Push source mapping behavior, as the title standard field will also get populated by the title standard metadata value.

  • The second rule is similar to the first one, except that it maps the duration standard metadata to the mmepsduration field you created at Step 1.

    If those two rules apply, the mmepstitle and mmepsduration fields will essentially store the same values as the standard title and duration fields for items in this source.

    In fact, duplicating standard fields this way can be useful in some cases, as you can use prefixed fields to create facets whose values come exclusively from a specific source, for instance.

  • The third rule appends the " million USD" string to the value of the mmepsbudget metadata before mapping this metadata to the field of the same name, which you created at Step 1.
  • The fourth rule maps the mmepsmpaafilmrating metadata to the field of the same name, which you created at Step 1. However, this rule also includes an alternative: if an incoming item does not have an mmepsmpaafilmrating metadata (i.e., if a movie is unrated), the mmepsmpaafilmrating field is populated by the "UR" string for this item instead.

Use the Source API to configure the mappings of your Push source as follows (see Managing the Mapping Configuration of a Source):

Mapping configuration request body

{
  "common": {
    "rules": [
      {
        "content": [
          "%[title]"
        ],
        "field": "mmepstitle"
      },
      {
        "content": [
          "%[duration]"
        ],
        "field": "mmepsduration"
      },
      {
        "content": [
          "%[mmepsbudget] million USD"
        ],
        "field": "mmepsbudget"
      },
      {
        "content": [
          "%[mmepsmpaafilmrating]",
          "UR"
        ],
        "field": "mmepsmpaafilmrating"
      }
    ]
  },
  "types": []
}

Step 3 - Add a Batch of Items

You will now use a batch operation to add several items at once in your secured Push source.

In a real life scenario, you would actually use your own code to retrieve its items along with their corresponding metadata and permission models from a secured enterprise system.

In the JSON below, each element in the addOrUpdate array represents a movie entry to add in your secured Push source. Each entry contains standard metadata (documenttype, filename, etc.), as well as custom metadata (e.g., mmepsdirector, mmepsyear, etc.). This metadata will populate fields in your index according to the default Push API mapping behavior, and the custom mapping rules you have defined at Step 2.

Since you are pushing items in a secured source, each item must have a permission model. You will define the security identities on which these permission models rely in Step 4.

In this tutorial, the permission model of an item can have multiple permission levels and permission sets: the first permission level is based on the default MPAA movie rating. The second permission level is based on some other form of movie rating, should a movie be yet unrated by the MPAA. The third permission level contains parental approval information. This last level is further divided into two permission sets (one for each parent).

The movie whose title is 1900 (which the last item in the JSON below) is unrated by the MPAA (i.e., it does not have a mmepsmpaamovierating metadata).

As a result, the first permission level, which is based on the MPAA movie rating, only allows adults (i.e., Grace, Henry, and yourself). Permissions for young children, pre-teenagers, and teenagers are still unknown at this level.

The second permission level, which is based on some other form of movie rating, explicitly denies young children (i.e., Alice and Bob) and pre-teenagers (i.e., Chloe and David). Permissions for teenagers (i.e., Erin and Frank) are still unknown at this level.

Finally, the third permission level, which contains parental approval information, indicates that:

  • Chloe (a pre-teenager) is explicitly allowed by both permission sets (i.e., by both her parents).

    However, since pre-teenagers were explicitly denied in the previous permission level, Chloe will not be able to see the 1900 movie entry in her query results.

  • Erin (a teenager) is also explicitly allowed by both permissions sets.

    Since teenager permissions were unknown in previous permission levels, Erin will be able to see the 1900 movie entry in her query results.

  • Frank (also a teenager) is explicitly denied in the first permission set, whereas he is explicitly allowed in the second one.

    Since he is not allowed in all permission sets at this permission level (i.e., one of his parents does not want him to see that movie entry), Frank will not be able to see the 1900 movie entry in his query results.

In conclusion, the only users who are allowed to see the 1900 movie entry in their query results are Erin, Grace, Henry, and yourself.

Permission levels are always evaluated sequentially.

This means that explicitly denying an adult user (i.e., Grace, Henry, or yourself) in the second or third permission level would not result in this user being ultimately denied, since as soon as a security identity is explicitly allowed by all permission sets of a lower permission level, it cannot be denied by any higher permission levels.

Follow the instructions in Managing Batches of Items in a Push Source to push the following batch in your secured Push source:

Now would be a good time to set the status of your secured Push source to REBUILD.

If you do so, remember to set it back to IDLE once you have pushed your batch of items.

See Updating the Status of a Push Source.

Content update request body

{
  "addOrUpdate": [
    {
      "documenttype": "Movie Entry",
      "filename": "Gone-with-the-Wind.html",
      "permanentid": "gone9301020eenednc838389d2d",
      "sourcetype": "Push",
      "title": "Gone with the Wind",
      "duration": 14280.0,
      "mmepsdirector": "Victor Fleming",
      "mmepsyear": "1939",
      "mmepssummary": "A manipulative woman and a roguish man conduct a turbulent romance during the American Civil War and Reconstruction periods.",
      "mmepsisseries": false,
      "mmepsmpaafilmrating": "G",
      "mmepsbudget": 3.9,
      "mmepscriticsaverage": 97,
      "mmepscastoverview": [
        "Thomas Mitchell",
        "Barbara O'Neil",
        "Vivien Leigh",
        "Evelyn Keyes",
        "Ann Rutherford"
      ],
      "mmepsthumbnaillink": "https://images-na.ssl-images-amazon.com/images/M/MV5BYWQwOWVkMGItMDU2Yy00YjIzLWJkMjEtNmVkZjE3MjMwYzEzXkEyXkFqcGdeQXVyNjU0OTQ0OTY@._V1_SY1000_CR0,0,652,1000_AL_.jpg",
      "clickUri": "https://www.youtube.com/watch?v=8mM8iNarcRc",
      "data": "<html><head> <style>body{font-family: 'Lato', 'Helvetica Neue', Helvetica, Arial, sans-serif, sans-serif; font-size: 15px; color: #373737;}a{text-decoration: none; color: #009DDC; cursor: pointer;}a:hover{text-decoration: underline;}.left-column{float: left; width: 200px; padding: 5px 15px;}.right-column{overflow: hidden; padding: 5px 10px 10px 15px;}.style-summary{word-wrap: break-word;}.style-title h1{color: #67768B; text-transform: uppercase; white-space: normal; margin: 0px;}.director-year{color: #67768B;}.rating{width: 60px; text-align: center; border-width: 1px; border-style: solid; border-radius: 5px;}.rating span{margin: 0px; border-radius: 5px; display: block;}.rating span.g-rating{color: black; background: white;}.duration{font-size: 12px;}.trailer{border-bottom: 1px solid #BCC3CA; padding: 16px 0px 16px 0px;}</style></head><body> <div class='left-column'> <img src='https://images-na.ssl-images-amazon.com/images/M/MV5BYWQwOWVkMGItMDU2Yy00YjIzLWJkMjEtNmVkZjE3MjMwYzEzXkEyXkFqcGdeQXVyNjU0OTQ0OTY@._V1_SY1000_CR0,0,652,1000_AL_.jpg' width='200'> </div><div class='right-column'> <div class='style-title'> <h1>Gone with the Wind</h1> </div><div class='director-year'> <p>Victor Fleming - 1939</p></div><div class='duration'> <p>3h58</p></div><div class='rating'> <span class='g-rating'>G</span> </div><div class='summary'> <p>A manipulative woman and a roguish man conduct a turbulent romance during the American Civil War and Reconstruction periods.</p></div><div class='trailer'> <a href='https://www.youtube.com/watch?v=8mM8iNarcRc'>View trailer</a> </div></div></body></html>",
      "fileExtension": ".html",
      "documentId": "file://movie-entries/1939/Gone-with-the-Wind.html",
      "permissions": [
        {
          "name": "MPAA Default Permissions",
          "permissionSets": [
            {
              "allowAnonymous": false,
              "allowedPermissions": [
                {
                  "identity": "Everyone",
                  "identityType": "Group"
                }
              ],
              "deniedPermissions": []
            }
          ]
        }
      ]
    },
    {
      "documenttype": "Movie Entry",
      "filename": "The-Man-Who-Knew-Too-Much.html",
      "permanentid": "the940203m3mf0m0gv03333n4",
      "sourcetype": "Push",
      "title": "The Man Who Knew Too Much",
      "duration": 124.07,
      "mmepsdirector": "Alfred Hitchcock",
      "mmepsyear": "1956",
      "mmepssummary": "A family vacationing in Morocco accidentally stumble on to an assassination plot and the conspirators are determined to prevent them from interfering.",
      "mmepsisseries": false,
      "mmepsmpaafilmrating": "PG",
      "mmepsbudget": 1.2,
      "mmepscriticsaverage": 91,
      "mmepscastoverview": [
        "James Stewart",
        "Doris Day",
        "Brenda de Branzie",
        "Bernard Miles",
        "Ralph Truman"
      ],
      "mmepsthumbnaillink": "https://images-na.ssl-images-amazon.com/images/M/MV5BNjQ2OWFhMjItYmFmNC00OGQzLWE1ZTMtZGQwZDM3YWUzMDgxL2ltYWdlL2ltYWdlXkEyXkFqcGdeQXVyMDI2NDg0NQ@@._V1_SY1000_CR0,0,656,1000_AL_.jpg",
      "mmepstrailerlink": "https://www.youtube.com/watch?v=Qw0OYSBI71U",
      "data": "<html><head> <style>body{font-family: 'Lato', 'Helvetica Neue', Helvetica, Arial, sans-serif, sans-serif; font-size: 15px; color: #373737;}a{text-decoration: none; color: #009DDC; cursor: pointer;}a:hover{text-decoration: underline;}.left-column{float: left; width: 200px; padding: 5px 15px;}.right-column{overflow: hidden; padding: 5px 10px 10px 15px;}.summary{word-wrap: break-word;}.title h1{color: #67768B; text-transform: uppercase; white-space: normal; margin: 0px;}.director-year{color: #67768B;}.rating{width: 60px; text-align: center; border-width: 1px; border-style: solid; border-radius: 5px;}.rating span{margin: 0px; border-radius: 5px; display: block;}.rating span.pg-rating{color: black; background: yellow;}.duration{font-size: 12px;}.trailer{border-bottom: 1px solid #BCC3CA; padding: 16px 0px 16px 0px;}</style></head><body> <div class='left-column'> <img src='https://images-na.ssl-images-amazon.com/images/M/MV5BNjQ2OWFhMjItYmFmNC00OGQzLWE1ZTMtZGQwZDM3YWUzMDgxL2ltYWdlL2ltYWdlXkEyXkFqcGdeQXVyMDI2NDg0NQ@@._V1_SY1000_CR0,0,656,1000_AL_.jpg' width='200'> </div><div class='right-column'> <div class='title'> <h1>The Man Who Knew Too Much</h1> </div><div class='director-year'> <p>Alfred Hitchcock - 1956</p></div><div class='duration'> <p>2h</p></div><div class='rating'> <span class='pg-rating'>PG</span> </div><div class='summary'> <p>A family vacationing in Morocco accidentally stumble on to an assassination plot and the conspirators are determined to prevent them from interfering.</p></div><div class='trailer'> <a href='https://www.youtube.com/watch?v=Qw0OYSBI71U'>View trailer</a> </div></div></body></html>",
      "fileExtension": ".html",      
      "documentId": "file://movie-entries/1956/The-Man-Who-Knew-Too-Much.html",
      "permissions": [
        {
          "name": "MPAA Default Permissions",
          "permissionSets": [
            {
              "allowAnonymous": false,
              "allowedPermissions": [
                {
                  "identity": "Adults",
                  "identityType": "Group"
                },
                {
                  "identity": "Teenagers",
                  "identityType": "Group"
                },
                {
                  "identity": "PreTeenagers",
                  "identityType": "Group"
                }
              ],
              "deniedPermissions": []
            }
          ]
        },
        {
          "name": "Parental Approval Permissions",
          "permissionSets": [
            {
              "name": "First Parent Decision",
              "allowedPermissions": [
                {
                  "identity": "alice@example.com",
                  "identityType": "User"
                }
              ],
              "deniedPermissions": []
            },
            {
              "name": "Second Parent Decision",
              "allowedPermissions": [],
              "deniedPermissions": [
                {
                  "identity": "alice@example.com",
                  "identityType": "User"
                }
              ]
            }
          ]
        }
      ]
    },
    {
      "documenttype": "Movie Entry",
      "filename": "Spartacus.html",
      "permanentid": "spartacus3929393mnfvb30849923ncnv",
      "sourcetype": "Push",
      "title": "Spartacus",
      "duration": 163.72,
      "mmepsdirector": "Stanley Kubrick",
      "mmepsyear": "1960",
      "mmepssummary": "The slave Spartacus leads a violent revolt against the decadent Roman Republic.",
      "mmepsisseries": false,
      "mmepsmpaafilmrating": "PG-13",
      "mmepsbudget": 12.0,
      "mmepscriticsaverage": 87,
      "mmepscastoverview": [
        "Kirk Douglas",
        "Laurence Olivier",
        "Jean Simmons",
        "Charles Laughton",
        "Peter Ustinov"
      ],
      "mmepsthumbnaillink": "https://images-na.ssl-images-amazon.com/images/M/MV5BMjc4MTUxN2UtMmU1NC00MjQyLTk3YTYtZTQ0YzEzZDc0Njc0XkEyXkFqcGdeQXVyNjU0OTQ0OTY@._V1_SY999_CR0,0,660,999_AL_.jpg",
      "mmepstrailerlink": "https://www.youtube.com/watch?v=tFLCEDLCSHA",
      "data": "<html><head> <style>body{font-family: 'Lato', 'Helvetica Neue', Helvetica, Arial, sans-serif, sans-serif; font-size: 15px; color: #373737;}a{text-decoration: none; color: #009DDC; cursor: pointer;}a:hover{text-decoration: underline;}.left-column{float: left; width: 200px; padding: 5px 15px;}.right-column{overflow: hidden; padding: 5px 10px 10px 15px;}.summary{word-wrap: break-word;}.title h1{color: #67768B; text-transform: uppercase; white-space: normal; margin: 0px;}.director-year{color: #67768B;}.rating{width: 60px; text-align: center; border-width: 1px; border-style: solid; border-radius: 5px;}.rating span{margin: 0px; border-radius: 5px; display: block;}.rating span.pg-13-rating{color: white; background: purple;}.duration{font-size: 12px;}.trailer{border-bottom: 1px solid #BCC3CA; padding: 16px 0px 16px 0px;}</style></head><body> <div class='left-column'> <img src='https://images-na.ssl-images-amazon.com/images/M/MV5BMjc4MTUxN2UtMmU1NC00MjQyLTk3YTYtZTQ0YzEzZDc0Njc0XkEyXkFqcGdeQXVyNjU0OTQ0OTY@._V1_SY999_CR0,0,660,999_AL_.jpg' width='200'> </div><div class='right-column'> <div class='title'> <h1>Spartacus</h1> </div><div class='director-year'> <p>Stanley Kubrick - 1960</p></div><div class='duration'> <p>3h17</p></div><div class='rating'> <span class='pg-13-rating'>PG-13</span> </div><div class='summary'> <p>The slave Spartacus leads a violent revolt against the decadent Roman Republic.</p></div><div class='trailer'> <a href='https://www.youtube.com/watch?v=tFLCEDLCSHA'>View trailer</a> </div></div></body></html>",
      "documentId": "file://movie-entries/1960/Spartacus.html",
      "fileExtension": ".html",
      "permissions": [
        {
          "name": "MPAA Default Permissions",
          "permissionSets": [
            {
              "allowAnonymous": false,
              "allowedPermissions": [
                {
                  "identity": "Adults",
                  "identityType": "Group"
                },
                {
                  "identity": "Teenagers",
                  "identityType": "Group"
                }
              ],
              "deniedPermissions": [
                {
                  "identity": "YoungChildren",
                  "identityType": "Group"
                }
              ]
            }
          ]
        },
        {
          "name": "Parental Approval Permissions",
          "permissionSets": [
            {
              "name": "First Parent Decision",
              "allowedPermissions": [
                {
                  "identity": "bob@example.com",
                  "identityType": "User"
                },
                {
                  "identity": "david@example.com",
                  "identityType": "User"
                },
                {
                  "identity": "chloe@example.com",
                  "identityType": "User"
                }
              ],
              "deniedPermissions": []
            },
            {
              "name": "Second Parent Decision",
              "allowedPermissions": [
                {
                  "identity": "bob@example.com",
                  "identityType": "User"
                },
                {
                  "identity": "david@example.com",
                  "identityType": "User"
                }
              ],
              "deniedPermissions": []
            }
          ]
        }
      ]
    },
    {
      "documenttype": "Movie Entry",
      "filename": "Dangerous-Liaisons.html",
      "permanentid": "dangerous3920934n209823ncnvvn83",
      "sourcetype": "Push",
      "title": "Dangerous Liaisons",
      "duration": 88.47,
      "mmepsdirector": "Stephen Frears",
      "mmepsyear": "1988",
      "mmepssummary": "A scheming widow and her manipulative ex-lover make a bet regarding the corruption of a recently married woman.",
      "mmepsisseries": false,
      "mmepsmpaafilmrating": "R",
      "mmepsbudget": 14.0,
      "mmepscriticsaverage": 74,
      "mmepscastoverview": [
        "Glenn Close",
        "John Malkovich",
        "Michelle Pfeiffer",
        "Swoosie Kurtz",
        "Keanu Reeves"
      ],
      "mmepsthumbnaillink": "https://images-na.ssl-images-amazon.com/images/M/MV5BMDZmODc0MWQtYWVjOS00OGFiLWExNjItOTMyN2Q5ZTFmYzUzXkEyXkFqcGdeQXVyMTAwMzUyOTc@._V1_.jpg",
      "mmepstrailerlink": "https://www.youtube.com/watch?v=FbB2oBlP2uI",
      "data": "<html> <head> <style>body{font-family: 'Lato', 'Helvetica Neue', Helvetica, Arial, sans-serif, sans-serif; font-size: 15px; color: #373737;}a{text-decoration: none; color: #009DDC; cursor: pointer;}a:hover{text-decoration: underline;}.left-column{float: left; width: 200px; padding: 5px 15px;}.right-column{overflow: hidden; padding: 5px 10px 10px 15px;}.summary{word-wrap: break-word;}.title h1{color: #67768B; text-transform: uppercase; white-space: normal; margin: 0px;}.director-year{color: #67768B;}.rating{width: 60px; text-align: center; border-width: 1px; border-style: solid; border-radius: 5px;}.rating span{margin: 0px; border-radius: 5px; display: block;}.rating span.r-rating{color: white; background: red;}.duration{font-size: 12px;}.trailer{border-bottom: 1px solid #BCC3CA; padding: 16px 0px 16px 0px;}</style> </head> <body> <div class='left-column'> <img src='https://images-na.ssl-images-amazon.com/images/M/MV5BMDZmODc0MWQtYWVjOS00OGFiLWExNjItOTMyN2Q5ZTFmYzUzXkEyXkFqcGdeQXVyMTAwMzUyOTc@._V1_.jpg' width='200'> </div><div class='right-column'> <div class='title'> <h1>Dangerous Liaisons</h1> </div><div class='director-year'> <p>Stephen Frears - 1988</p></div><div class='duration'> <p>1h59</p></div><div class='rating'> <span class='r-rating'>R</span> </div><div class='summary'> <p>A scheming widow and her manipulative ex-lover make a bet regarding the corruption of a recently married woman.</p></div><div class='trailer'> <a href='https://www.youtube.com/watch?v=FbB2oBlP2uI'>View trailer</a> </div></div></body></html>",
      "documentId": "file://movie-entries/1988/Dangerous-Liaisons.html",
      "fileExtension": ".html",
      "permissions": [
        {
          "name": "MPAA Default Permissions",
          "permissionSets": [
            {
              "allowAnonymous": false,
              "allowedPermissions": [
                {
                  "identity": "Adults",
                  "identityType": "Group"
                },
                {
                  "identity": "Teenagers",
                  "identityType": "Group"
                }
              ],
              "deniedPermissions": [
                {
                  "identity": "YoungChildren",
                  "identityType": "Group"
                },
                {
                  "identity": "PreTeenagers",
                  "identityType": "Group"
                }
              ]
            }
          ]
        },
        {
          "name": "Parental Approval Permissions",
          "permissionSets": [
            {
              "name": "First Parent Decision",
              "allowedPermissions": [],
              "deniedPermissions": []
            },
            {
              "name": "Second Parent Decision",
              "allowedPermissions": [],
              "deniedPermissions": []
            }
          ]
        }
      ]
    },
    {
      "documenttype": "Movie Entry",
      "filename": "The-Evil-Dead.html",
      "permanentid": "th33vil39mmdmd93mc393309534m",
      "sourcetype": "Push",
      "title": "The Evil Dead",
      "duration": 125.37,
      "mmepsdirector": "Sam Raimi",
      "mmepsyear": "1981",
      "mmepssummary": "Five friends travel to a cabin in the woods, where they unknowingly release flesh-possessing demons.",
      "mmepsisseries": true,
      "mmepsmpaafilmrating": "NC-17",
      "mmepsbudget": 0.35,
      "mmepscriticsaverage": 70,
      "mmepscastoverview": [
        "Bruce Campbell",
        "Ellen Sandweiss",
        "Richard DeManincor",
        "Betsy Baker",
        "Theresa Tilly"
      ],
      "mmepsthumbnaillink": "https://images-na.ssl-images-amazon.com/images/M/MV5BODc2MmVjZmUtNjAzMS00MDNiLWIyM2YtOGEzMjg0YjRhMzRmXkEyXkFqcGdeQXVyMTQxNzMzNDI@._V1_UX182_CR0,0,182,268_AL_.jpg",
      "mmepstrailerlink": "https://www.youtube.com/watch?v=jgw4ZdC2XTE",
      "data": "<html><head> <style>body{font-family: 'Lato', 'Helvetica Neue', Helvetica, Arial, sans-serif, sans-serif; font-size: 15px; color: #373737;}a{text-decoration: none; color: #009DDC; cursor: pointer;}a:hover{text-decoration: underline;}.left-column{float: left; width: 200px; padding: 5px 15px;}.right-column{overflow: hidden; padding: 5px 10px 10px 15px;}.summary{word-wrap: break-word;}.title h1{color: #67768B; text-transform: uppercase; white-space: normal; margin: 0px;}.director-year{color: #67768B;}.rating{width: 60px; text-align: center; border-width: 1px; border-style: solid; border-radius: 5px;}.rating span{margin: 0px; border-radius: 5px; display: block;}.rating span.nc-17{color: white; background: black;}.duration{font-size: 12px;}.trailer{border-bottom: 1px solid #BCC3CA; padding: 16px 0px 16px 0px;}</style></head><body> <div class='left-column'> <img src='https://images-na.ssl-images-amazon.com/images/M/MV5BODc2MmVjZmUtNjAzMS00MDNiLWIyM2YtOGEzMjg0YjRhMzRmXkEyXkFqcGdeQXVyMTQxNzMzNDI@._V1_UX182_CR0,0,182,268_AL_.jpg' width='200'> </div><div class='right-column'> <div class='title'> <h1>The Evil Dead</h1> </div><div class='director-year'> <p>Sam Raimi - 1981</p></div><div class='duration'> <p>1h25</p></div><div class='rating'><span class='NC-17'>NC-17</span> </div><div class='summary'> <p>Five friends travel to a cabin in the woods, where they unknowingly release flesh-possessing demons.</p></div><div class='trailer'> <a href='https://www.youtube.com/watch?v=jgw4ZdC2XTE'>View trailer</a> </div></div></body></html>",
      "documentId": "file://movie-entries/1981/The-Evil-Dead.html",
      "fileExtension": ".html",
      "permissions": [
        {
          "name": "MPAA Default Permissions",
          "permissionSets": [
            {
              "allowAnonymous": false,
              "allowedPermissions": [
                {
                  "identity": "Adults",
                  "identityType": "Group"
                }
              ],
              "deniedPermissions": [
                {
                  "identity": "YoungChildren",
                  "identityType": "Group"
                },
                {
                  "identity": "PreTeenagers",
                  "identityType": "Group"
                },
                {
                  "identity": "Teenagers",
                  "identityType": "Group"
                }
              ]
            }
          ]
        }
      ]
    },
    {
      "documenttype": "Movie Entry",
      "filename": "1900.html",
      "permanentid": "19002ndndn3n39293092553ncf859802ncsdg",
      "sourcetype": "Push",
      "title": "1900",
      "duration": 175.92,
      "mmepsdirector": "Bernardo Bertolucci",
      "mmepsyear": 1976,
      "mmepssummary": "The epic tale of a class struggle in twentieth century Italy, as seen through the eyes of two childhood friends on opposing sides.",
      "mmepsisseries": false,
      "mmepsmpaafilmrating": "UR",
      "mmepsbudget": 9.0,
      "mmepscriticsaverage": 85,
      "mmepscastoverview": [
        "Robert De Niro",
        "Gérard Depardieu",
        "Dominique Sanda",
        "Francesca Bertini",
        "Laura Betti"
      ],
      "mmepsthumbnaillink": "https://upload.wikimedia.org/wikipedia/en/thumb/0/07/1900_Bertolluci.jpg/220px-1900_Bertolluci.jpg",
      "mmepstrailerlink": "https://www.youtube.com/watch?v=uGyH6Tt6qwY",
      "data": "<html><head> <style>body{font-family: 'Lato', 'Helvetica Neue', Helvetica, Arial, sans-serif, sans-serif; font-size: 15px; color: #373737;}a{text-decoration: none; color: #009DDC; cursor: pointer;}a:hover{text-decoration: underline;}.left-column{float: left; width: 200px; padding: 5px 15px;}.right-column{overflow: hidden; padding: 5px 10px 10px 15px;}.summary{word-wrap: break-word;}.title h1{color: #67768B; text-transform: uppercase; white-space: normal; margin: 0px;}.director-year{color: #67768B;}.rating{width: 60px; text-align: center; border-width: 1px; border-style: solid; border-radius: 5px;}.rating span{margin: 0px; border-radius: 5px; display: block;}.rating span.ur{color: white; background: blue;}.duration{font-size: 12px;}.trailer{border-bottom: 1px solid #BCC3CA; padding: 16px 0px 16px 0px;}</style></head><body> <div class='left-column'> <img src='https://upload.wikimedia.org/wikipedia/en/thumb/0/07/1900_Bertolluci.jpg/220px-1900_Bertolluci.jpg' width='200'> </div><div class='right-column'> <div class='title'> <h1>1900</h1> </div><div class='director-year'> <p>Bernardo Bertolucci - 1976</p></div><div class='duration'> <p>5h17</p></div><div class='rating'><span class='ur'>UR</span> </div><div class='summary'> <p>The epic tale of a class struggle in twentieth century Italy, as seen through the eyes of two childhood friends on opposing sides.</p></div><div class='trailer'> <a href='https://www.youtube.com/watch?v=uGyH6Tt6qwY'>View trailer</a> </div></div></body></html>",
      "documentId": "file://movie-entries/1976/1900.html",
      "fileExtension": ".html",
      "permissions": [
        {
          "name": "MPAA Default Permissions",
          "permissionSets": [
            {
              "allowAnonymous": false,
              "allowedPermissions": [
                {
                  "identity": "Adults",
                  "identityType": "Group"
                }
              ],
              "deniedPermissions": []
            }
          ]
        },
        {
          "name": "Additional Rating Permissions",
          "permissionSets": [
            {
              "allowAnonymous": false,
              "allowedPermissions": [],
              "deniedPermissions": [
                {
                  "identity": "YoungChildren",
                  "identityType": "Group"
                },
                {
                  "identity": "PreTeenagers",
                  "identityType": "Group"
                }
              ]
            }
          ]
        },
        {
          "name": "Parental Approval Permissions",
          "permissionSets": [
            {
              "name": "First Parent Decision",
              "allowedPermissions": [
                {
                  "identity": "chloe@example.com",
                  "identityType": "User"
                },
                {
                  "identity": "erin@example.com",
                  "identityType": "User"
                }
              ],
              "deniedPermissions": [
                {
                  "identity": "frank@example.com",
                  "identityType": "User"
                }
              ]
            },
            {
              "name": "Second Parent Decision",
              "allowedPermissions": [
                {
                  "identity": "chloe@example.com",
                  "identityType": "User"
                },
                {
                  "identity": "erin@example.com",
                  "identityType": "User"
                },
                {
                  "identity": "frank@example.com",
                  "identityType": "User"
                }
              ],
              "deniedPermissions": []
            }
          ]
        }
      ]
    }
  ],
  "delete": []
}

Remember that you may have to wait a few minutes before those three items become available in your source (see Understanding the Push API Processing Delay).

Step 4 - Add a Batch of Security Identities

Whenever you use the Push API to add, update, and/or delete items in a secured Push source, you should then ensure that the security identity definitions which these items refer to in their permission models are up to date in the security identity cache of your Coveo Cloud V2 organization. Otherwise, certain users might not be able to see some of the items for which they should have access (which is bad), while some others may be able to see items for which they should no longer have access (which is worse).

In this tutorial, you will use a batch operation to define several security identities at once. The items you added in your secured Push source at Step 3 refer to those security identities in their permission models.

In a real life scenario, you would actually use your own code to extract security identities and security identity relationships from a secured enterprise system.

The JSON below defines 9 individual users (Alice, Bob, Chloe, David, Erin, Frank, Grace, Henry, and yourself), along with four groups (YoungChildren, PreTeenagers, Teenagers, and Adults), and one granted identity (Everyone). It also defines an alias to your own security identity in the Coveo Cloud V2 platform, which implies that you must replace <MyEmailAddress> towards the end of this JSON by the actual email address you are authenticated as in the Coveo Cloud V2 platform.

Typically, you should define an alias relationship between each user security identity and its corresponding security identity in the Email Security Provider.

This means that you would normally define each user in both the members and mappings array (like the yourself user in this tutorial) when pushing a batch of security identities.

This tutorial does not define aliases for any of the fake users (alice@example.com, bob@example.com, etc.), since those users will only ever exist in the security identity provider of your secured Push source.

See Understanding the Email Security Provider.

Each user belongs to one and only one group as summarized in the following table:

User Group
alice@example.com YoungChildren
bob@example.com YoungChildren
chloe@example.com PreTeenagers
david@example.com PreTeenagers
erin@example.com Teenagers
frank@example.com Teenagers
grace@example.com Adults
henry@example.com Adults
yourself@example.com Adults

Moreover, each group has the Everyone granted identity.

Follow the instructions in Managing Batches of Security Identities to push the following batch of security identities in the security identity provider of your secured Push source.

Security identity update request body template

{
  "members": [
    {
      "identity": {
        "name": "Everyone",
        "type": "GROUP"
      }
    },
    {
      "identity": {
        "name": "alice@example.com",
        "type": "USER"
      }
    },
    {
      "identity": {
        "name": "bob@example.com",
        "type": "USER"
      }
    },
    {
      "identity": {
        "name": "chloe@example.com",
        "type": "USER"
      }
    },
    {
      "identity": {
        "name": "david@example.com",
        "type": "USER"
      }
    },
    {
      "identity": {
        "name": "erin@example.com",
        "type": "USER"
      }
    },
    {
      "identity": {
        "name": "frank@example.com",
        "type": "USER"
      }
    },
    {
      "identity": {
        "name": "grace@example.com",
        "type": "USER"
      }
    },
    {
      "identity": {
        "name": "henry@example.com",
        "type": "USER"
      }
    },
    {
      "identity": {
        "name": "yourself@example.com",
        "type": "USER"
      }
    },
    {
      "identity": {
        "name": "YoungChildren",
        "type": "GROUP"
      },
      "members": [
        {
          "name": "alice@example.com",
          "type": "USER"
        },
        {
          "name": "bob@example.com",
          "type": "USER"
        }
      ],
      "wellKnowns": [
        {
          "name": "Everyone",
          "type": "GROUP"
        }
      ]
    },
    {
      "identity": {
        "name": "PreTeenagers",
        "type": "GROUP"
      },
      "members": [
        {
          "name": "chloe@example.com",
          "type": "USER"
        },
        {
          "name": "david@example.com",
          "type": "USER"
        }
      ],
      "wellKnowns": [
        {
          "name": "Everyone",
          "type": "GROUP"
        }
      ]
    },
    {
      "identity": {
        "name": "Teenagers",
        "type": "GROUP"
      },
      "members": [
        {
          "name": "erin@example.com",
          "type": "USER"
        },
        {
          "name": "frank@example.com",
          "type": "USER"
        }
      ],
      "wellKnowns": [
        {
          "name": "Everyone",
          "type": "GROUP"
        }
      ]
    },
    {
      "identity": {
        "name": "Adults",
        "type": "GROUP"
      },
      "members": [
        {
          "name": "grace@example.com",
          "type": "USER"
        },
        {
          "name": "henry@example.com",
          "type": "USER"
        },
        {
          "name": "yourself@example.com",
          "type": "USER"
        }
      ],
      "wellKnowns": [
        {
          "name": "Everyone",
          "type": "GROUP"
        }
      ]
    }
  ],
  "mappings": [
    {
      "identity": {
        "name": "yourself@example.com",
        "type": "USER"
      },
      "mappings": [
        {
          "name": <MyEmailAddress>,
          "type": "USER",
          "provider": "Email Security Provider"
        }
      ]
    }
  ],
  "deleted": []
}

In the above request body template, do not forget to replace <MyEmailAddress> by the actual email address your are authenticated as in the Coveo Cloud V2 platform.

Step 5 - Refresh Your Security Identity Provider

You should now ensure that the security identity cache of your Coveo Cloud V2 organization fetches the security identities you added in your security identity provider at Step 4. Otherwise, those security identities will be in error when you push items whose permission models refer to them until the security identity cache automatically refreshes them (which is scheduled to happen once every two hours by default).

Use the Security Cache API to refresh the security identity provider of your secured Push source (see Refreshing a Security Identity Provider).

You may have to wait a few minutes before the security identity provider refresh is complete.

If you want, you can validate the status of your security identity provider refresh activity once every minute or so (see Validating the Status of a Security Identity Provider Refresh Activity).

Step 6 - (Optional) Validate Item Field Values, Effective Permissions, and Data

You will now use the Coveo Cloud V2 administration console to manually verify whether the metadata of the items you have just pushed has populated fields as expected in your index, and whether the permission models of your items are being evaluated correctly.

  1. In the Coveo Cloud V2 administration console, open the Content Browser.

  2. In the search box, type @source==<MyPushSource> (replace <MyPushSource> by the actual name of your secured Push source) and hit the Enter key.

    The six items you have pushed in your source should appear in the result list.

  3. Select the result whose title is 1900, and click the PROPERTIES link.

    A modal window should open in which the FIELDS tab is selected by default. You should see a list of mapped fields for this item along with their corresponding values. Among other things, you can now validate that:

    • The title and mmepstitle fields have the same value (1900), which means that the first mapping rule you defined in Step 2 was correctly applied.
    • The duration and mmepsduration fields have the same value (175.92), which means that the second mapping rule you defined at Step 2 was correctly applied.
    • The " million USD" string was correctly appended at the end of the mmepsbudget, which means that the third rule you defined at Step 2 was correctly applied.
    • The mmepsmpaafilmrating field value was correctly set to the "UR" string, which appears to indicate that the fourth mapping rule you defined at Step 2 was correctly applied, since this item was pushed without an mmepsmpaafimrating metadata.

  4. In the modal window header, click the PERMISSIONS tab.

    You should see the list of permissions for the 1900 item. As explained in the example in Step 3:

    • The only allowed users are Erin, Grace, Henry, and yourself.
    • The YoungChildren and PreTeenagers groups, as well as user Frank, are explicitely denied.

  5. In the modal window header, click the PERMISSION DETAILS tab. You should see that this item has three permission levels. The first two permission levels only have one permission set, wheras the last one has two. As explained in the example in Step 3:
    • Permission Level 0 (i.e., the MPAA film rating permission level) explicitly allows the Adults group.
    • Permission Level 1 (i.e., the additional rating-based permission level) the explicitly denies the YoungChildren and PreTeenagers groups.
    • Permission Level 2 (i.e., the parental approval permission level) ultimately allows user Erin, and denies user Frank.

  6. In the modal window header, click the QUICKVIEW tab.

    You should see that the HTML and CSS that was provided in the Data property value for this item is rendered as expected.

If you want, you can validate the other items you have added in your secured Push source at Step 3 in the same fashion.