Push API Tutorial 3: Using Batch Calls to Manage Complex Secured Content

This is for:

Developer System Administrator

In the previous tutorial, you configured a secured Push source along with its associated security identity provider, and you learned how to push simple secured items and security identities individually. You also learned how to delete an item and its children, as well as how to delete items or security identities based on the orderingId value of the last operation that was performed on them.

In this tutorial, you’ll create fields in your index and define custom mapping rules for your secured Push source. You’ll then use Push API batch operations to simulate a security identity update (including different types of security identities), followed by a source content update (including coherent item metadata and complex permission models). Finally, you’ll use the Coveo Administration Console to validate whether everything is working as expected.

Step 0: Prerequisites

For this tutorial, you need:

  • A Coveo organization that minimally contains:

  • A Coveo Platform access token (OAuth2 token or API key) that grants you the following privileges in that organization (see Create an API key and Get your Coveo access token):

      | `owner`      | `targetDomain`      | `type` | Corresponding privilege in the Coveo Administration Console |   |--------------|---------------------|--------|----------------------------------------------------------------------|   | `PLATFORM`   | `ORGANIZATION`      | `VIEW` | **Organization - Organization - View**                               |   | `PLATFORM`   | `SECURITY_PROVIDER` | `EDIT` | **Content - Security identity providers - Edit**                     |   | `PLATFORM`   | `SOURCE`            | `EDIT` | **Content- Sources - Edit**                                          |   | `PLATFORM`   | `FIELD`             | `EDIT` | **Content - Fields - Edit**                                          |

  • If you click Run in Postman, you must set appropriate values for the organizationId and apiKey environment variables so that the API requests are executed successfully in Postman (see Manage Globals).

If you have completed the previous tutorial and the Coveo organization, secured Push source, security identity provider, and Coveo access token you used in that tutorial are still available, you can (and should) re-use them.

You should also ensure that the secured Push source that you use for this tutorial contains no previously indexed items (see Deleting Old Items in a Push Source).

Step 1: Create Fields

Fields let you store and leverage item metadata, allowing you to create facets, define custom query result sort criteria, etc.

It’s important to remember that fields are index-wide data containers, which implies that any source can theoretically populate a given field with item metadata.

The title standard field in your index typically gets populated by item metadata from many sources.

In this tutorial, the items you’ll add in your secured Push source are movie entries, so you’ll start by creating several fields to store and leverage movie-related metadata. You’ll mark these as specific to your source by prefixing them with mmeps (which stands for My Movie Entry Push Source).

While prefixing a field doesn’t change the fact that it’s an index-wide container, it allows you to quickly identify which fields are semantically tied to a specific source.

The JSON below defines 12 fields which you’ll populate with metadata when you add items in your secured Push source in step 3. You won’t notice that additional Boolean options (for example, facet, sort, and useCacheForSort) are specified for some of these fields, but they’re not relevant in this tutorial.

For now, use the Fields API to create the following batch of fields in your index (see Creating Fields):

Field creation request body

[
  {
    "name": "mmepstitle",
    "description": "The title of the movie.",
    "type": "STRING",
    "facet": true,
    "sort": true,
    "useCacheForSort": true
  },
  {
    "name": "mmepsyear",
    "description": "The year in which the movie was released.",
    "type": "LONG"
  },
  {
    "name": "mmepsduration",
    "description": "The duration of the movie, in seconds.",
    "type": "DOUBLE"
  },
  {
    "name": "mmepssummary",
    "description": "A summary of the movie.",
    "type": "STRING"
  },
  {
    "name": "mmepsisseries",
    "description": "Whether the movie is part of a series.",
    "type": "STRING"
  },
  {
    "name": "mmepsmpaafilmrating",
    "description": "The MPAA film rating for the movie. Can be 'G', 'PG', 'PG-13', 'R', 'NC-17', or 'UR'.",
    "type": "STRING",
    "facet": true
  },
  {
    "name": "mmepsbudget",
    "description": "The budget of the movie, in million USD.",
    "type": "STRING"
  },
  {
    "name": "mmepscriticsaverage",
    "description": "The average score that was given to the movie by critics, on a discrete scale of 1 to 100.",
    "type": "LONG",
    "useCacheForComputedFacet": true
  },
  {
    "name": "mmepsdirector",
    "description": "The director of the movie.",
    "type": "STRING",
    "facet": true,
    "sort": true,
    "useCacheForSort": true,
    "mergeWithLexicon": true,
    "useCacheForNestedQuery": true
  },
  {
    "name": "mmepscastoverview",
    "description": "The list of principal actors that appear in the movie.",
    "type": "STRING",
    "multiValueFacet": true,
    "multiValueFacetTokenizers": ";",
    "mergeWithLexicon": true,
    "useCacheForNestedQuery": true
  },
  {
    "name": "mmepsthumbnaillink",
    "description": "The URI of the thumbnail image for the movie.",
    "type": "STRING"
  },
  {
    "name": "mmepstrailerlink",
    "description": "The URI of the movie trailer.",
    "type": "STRING"
  }
]

In the request body above, some fields have the mergeWithLexicon property set to true, which makes them free text searchable (see Determine Which Fields to Make Searchable).

For example, since the mmepsdirector field is free text searchable, an end user could submit a query such as Alfred Hitchcock from a search box and get matching results. However, since the mmepsmpaafilmrating field isn’t free text searchable, an end user would have to submit a field query such as @mmepsmpaafilmrating==G to retrieve matching items.

Step 2: Create Custom Mapping Rules

By default, a Push source automatically attempts to map incoming item metadata to a field of the corresponding name in your index. This means that when you push an item with a metadata key-value pair, the indexing pipeline will use the metadata value to populate a field whose name matches the metadata key, if such a field exists.

Assuming that your Push source has no custom mappings, pushing an item with the "mmepstitle": "Gone with the Wind" metadata key-value pair automatically populates the mmepstitle field in your index with the "Gone with the Wind" value for this item.

You’ll now create several custom mapping rules for your secured Push source. These rules will complement the default Push source mapping behavior when the indexing pipeline processes incoming items in your source.

The JSON below defines four custom common mapping rules:

  • The first rule maps the title standard metadata to the mmepstitle field you created in step 1. This ensures that the title standard field will also get populated by the title standard metadata value.

  • The second rule is similar to the first one, except that it maps the duration standard metadata to the mmepsduration field you created in step 1.

    When these two rules apply, the mmepstitle and mmepsduration fields will store the same values as the title and duration standard fields for items in this source.

    It can be useful to duplicate standard fields this way, because it allows you to use prefixed fields to create facets whose values come only from a specific source.

  • The third rule appends the " million USD" string to the value of the mmepsbudget metadata before mapping this metadata to the field of the same name, which you created in step 1.

  • The fourth rule maps the mmepsmpaafilmrating metadata to the field of the same name, which you created in step 1. However, it also includes an alternative: if an incoming item doesn’t have any mmepsmpaafilmrating metadata (that is, if a movie is unrated), the mmepsmpaafilmrating field for this item is populated by the "UR" string instead.

Use the Source API to configure the mappings of your Push source as follows (see Manage the Mapping Configuration of a Source):

Mapping configuration request body

{
  "common": {
    "rules": [
      {
        "content": [
          "%[title]"
        ],
        "field": "mmepstitle"
      },
      {
        "content": [
          "%[duration]"
        ],
        "field": "mmepsduration"
      },
      {
        "content": [
          "%[mmepsbudget] million USD"
        ],
        "field": "mmepsbudget"
      },
      {
        "content": [
          "%[mmepsmpaafilmrating]",
          "UR"
        ],
        "field": "mmepsmpaafilmrating"
      }
    ]
  },
  "types": []
}

Step 3: Add a Batch of Items

You’ll now use a batch operation to add several items at once in your secured Push source.

In a real-life scenario, you would use your own code to retrieve items along with their corresponding metadata and permission models from a secured enterprise system.

In the JSON below, each element in the addOrUpdate array represents a movie entry to add in your secured Push source. Every entry contains standard metadata (for example, documenttype, filename, etc.), as well as custom metadata (for example, mmepsdirector, mmepsyear, etc.). This metadata will populate fields in your index according to the default Push API mapping behavior and the custom mapping rules that you defined in step 2.

Since you’re pushing items in a secured source, each item must have a permission model. You’ll define the security identities on which these permission models rely in step 4.

In this tutorial, the permission model of an item can have many permission levels and permission sets. The first permission level is based on the default MPAA film rating. The second permission level is based on some other form of film rating, should a movie not yet have been rated by the MPAA. The third permission level contains parental approval information, and it’s further divided into two permission sets (one for each parent).

The movie whose title is 1900 (the last item in the JSON below) is unrated by the MPAA (that is, it doesn’t have any mmepsmpaamovierating metadata).

As a result, the first permission level, which is based on the MPAA film rating, only allows adults (Grace, Henry, and yourself). Permissions for young children, pre-teens, and teenagers are still unknown at this level.

The second permission level is based on some other form of film rating. It explicitly denies young children (Alice and Bob) and pre-teens (Chloe and David). Permissions for teenagers (Erin and Frank) are still unknown at this level.

The third permission level, which contains parental approval information, indicates that:

  • Chloe (a pre-teen) is explicitly allowed by both permission sets (that is, both of her parents).

    However, since pre-teens were explicitly denied in the previous permission level, Chloe won’t be able to see the 1900 movie entry in her query results.

  • Erin (a teenager) is also explicitly allowed by both permission sets.

    Since teenager permissions were unknown in previous permission levels, Erin will be able to see the 1900 movie entry in her query results.

  • Frank (also a teenager) is explicitly denied in the first permission set and explicitly allowed in the second one.

    Since he isn’t allowed in all of the permission sets at this permission level (that is, one of his parents doesn’t want him to see this movie entry), Frank won’t be able to see the 1900 movie entry in his query results.

In conclusion, the only users who are allowed to see the 1900 movie entry in their query results are Erin, Grace, Henry, and yourself.

Permission levels are always evaluated sequentially.

This means that explicitly denying one of the adult users (Grace, Henry, or yourself) in the second or third permission levels wouldn’t ultimately result in this user being denied. As soon as a security identity is explicitly allowed by all permission sets of a lower permission level, it can’t be denied by any higher permission levels.

Push the following batch of items in your secured Push source (see Manage batches of items in a Push source):

Now would be a good time to update the status of your Push source to REBUILD.

If you do, remember to set it back to IDLE after you push your batch of items.

Content update request body

{
  "addOrUpdate": [
    {
      "documenttype": "Movie Entry",
      "filename": "Gone-with-the-Wind.html",
      "permanentid": "gone9301020eenednc838389d2d",
      "sourcetype": "Push",
      "title": "Gone with the Wind",
      "duration": 14280.0,
      "mmepsdirector": "Victor Fleming",
      "mmepsyear": "1939",
      "mmepssummary": "A manipulative woman and a roguish man conduct a turbulent romance during the American Civil War and Reconstruction periods.",
      "mmepsisseries": false,
      "mmepsmpaafilmrating": "G",
      "mmepsbudget": 3.9,
      "mmepscriticsaverage": 97,
      "mmepscastoverview": [
        "Thomas Mitchell",
        "Barbara O'Neil",
        "Vivien Leigh",
        "Evelyn Keyes",
        "Ann Rutherford"
      ],
      "mmepsthumbnaillink": "https://images-na.ssl-images-amazon.com/images/M/MV5BYWQwOWVkMGItMDU2Yy00YjIzLWJkMjEtNmVkZjE3MjMwYzEzXkEyXkFqcGdeQXVyNjU0OTQ0OTY@._V1_SY1000_CR0,0,652,1000_AL_.jpg",
      "clickUri": "https://www.youtube.com/watch?v=8mM8iNarcRc",
      "data": "<html><head> <style>body{font-family: 'Lato', 'Helvetica Neue', Helvetica, Arial, sans-serif, sans-serif; font-size: 15px; color: #373737;}a{text-decoration: none; color: #009DDC; cursor: pointer;}a:hover{text-decoration: underline;}.left-column{float: left; width: 200px; padding: 5px 15px;}.right-column{overflow: hidden; padding: 5px 10px 10px 15px;}.style-summary{word-wrap: break-word;}.style-title h1{color: #67768B; text-transform: uppercase; white-space: normal; margin: 0px;}.director-year{color: #67768B;}.rating{width: 60px; text-align: center; border-width: 1px; border-style: solid; border-radius: 5px;}.rating span{margin: 0px; border-radius: 5px; display: block;}.rating span.g-rating{color: black; background: white;}.duration{font-size: 12px;}.trailer{border-bottom: 1px solid #BCC3CA; padding: 16px 0px 16px 0px;}</style></head><body> <div class='left-column'> <img src='https://images-na.ssl-images-amazon.com/images/M/MV5BYWQwOWVkMGItMDU2Yy00YjIzLWJkMjEtNmVkZjE3MjMwYzEzXkEyXkFqcGdeQXVyNjU0OTQ0OTY@._V1_SY1000_CR0,0,652,1000_AL_.jpg' width='200'> </div><div class='right-column'> <div class='style-title'> <h1>Gone with the Wind</h1> </div><div class='director-year'> <p>Victor Fleming - 1939</p></div><div class='duration'> <p>3h58</p></div><div class='rating'> <span class='g-rating'>G</span> </div><div class='summary'> <p>A manipulative woman and a roguish man conduct a turbulent romance during the American Civil War and Reconstruction periods.</p></div><div class='trailer'> <a href='https://www.youtube.com/watch?v=8mM8iNarcRc'>View trailer</a> </div></div></body></html>",
      "fileExtension": ".html",
      "documentId": "file://movie-entries/1939/Gone-with-the-Wind.html",
      "permissions": [
        {
          "name": "MPAA Default Permissions",
          "permissionSets": [
            {
              "allowAnonymous": false,
              "allowedPermissions": [
                {
                  "identity": "Everyone",
                  "identityType": "Group"
                }
              ],
              "deniedPermissions": []
            }
          ]
        }
      ]
    },
    {
      "documenttype": "Movie Entry",
      "filename": "The-Man-Who-Knew-Too-Much.html",
      "permanentid": "the940203m3mf0m0gv03333n4",
      "sourcetype": "Push",
      "title": "The Man Who Knew Too Much",
      "duration": 124.07,
      "mmepsdirector": "Alfred Hitchcock",
      "mmepsyear": "1956",
      "mmepssummary": "A family vacationing in Morocco accidentally stumble on to an assassination plot and the conspirators are determined to prevent them from interfering.",
      "mmepsisseries": false,
      "mmepsmpaafilmrating": "PG",
      "mmepsbudget": 1.2,
      "mmepscriticsaverage": 91,
      "mmepscastoverview": [
        "James Stewart",
        "Doris Day",
        "Brenda de Branzie",
        "Bernard Miles",
        "Ralph Truman"
      ],
      "mmepsthumbnaillink": "https://images-na.ssl-images-amazon.com/images/M/MV5BNjQ2OWFhMjItYmFmNC00OGQzLWE1ZTMtZGQwZDM3YWUzMDgxL2ltYWdlL2ltYWdlXkEyXkFqcGdeQXVyMDI2NDg0NQ@@._V1_SY1000_CR0,0,656,1000_AL_.jpg",
      "mmepstrailerlink": "https://www.youtube.com/watch?v=Qw0OYSBI71U",
      "data": "<html><head> <style>body{font-family: 'Lato', 'Helvetica Neue', Helvetica, Arial, sans-serif, sans-serif; font-size: 15px; color: #373737;}a{text-decoration: none; color: #009DDC; cursor: pointer;}a:hover{text-decoration: underline;}.left-column{float: left; width: 200px; padding: 5px 15px;}.right-column{overflow: hidden; padding: 5px 10px 10px 15px;}.summary{word-wrap: break-word;}.title h1{color: #67768B; text-transform: uppercase; white-space: normal; margin: 0px;}.director-year{color: #67768B;}.rating{width: 60px; text-align: center; border-width: 1px; border-style: solid; border-radius: 5px;}.rating span{margin: 0px; border-radius: 5px; display: block;}.rating span.pg-rating{color: black; background: yellow;}.duration{font-size: 12px;}.trailer{border-bottom: 1px solid #BCC3CA; padding: 16px 0px 16px 0px;}</style></head><body> <div class='left-column'> <img src='https://images-na.ssl-images-amazon.com/images/M/MV5BNjQ2OWFhMjItYmFmNC00OGQzLWE1ZTMtZGQwZDM3YWUzMDgxL2ltYWdlL2ltYWdlXkEyXkFqcGdeQXVyMDI2NDg0NQ@@._V1_SY1000_CR0,0,656,1000_AL_.jpg' width='200'> </div><div class='right-column'> <div class='title'> <h1>The Man Who Knew Too Much</h1> </div><div class='director-year'> <p>Alfred Hitchcock - 1956</p></div><div class='duration'> <p>2h</p></div><div class='rating'> <span class='pg-rating'>PG</span> </div><div class='summary'> <p>A family vacationing in Morocco accidentally stumble on to an assassination plot and the conspirators are determined to prevent them from interfering.</p></div><div class='trailer'> <a href='https://www.youtube.com/watch?v=Qw0OYSBI71U'>View trailer</a> </div></div></body></html>",
      "fileExtension": ".html",
      "documentId": "file://movie-entries/1956/The-Man-Who-Knew-Too-Much.html",
      "permissions": [
        {
          "name": "MPAA Default Permissions",
          "permissionSets": [
            {
              "allowAnonymous": false,
              "allowedPermissions": [
                {
                  "identity": "Adults",
                  "identityType": "Group"
                },
                {
                  "identity": "Teenagers",
                  "identityType": "Group"
                },
                {
                  "identity": "PreTeenagers",
                  "identityType": "Group"
                }
              ],
              "deniedPermissions": []
            }
          ]
        },
        {
          "name": "Parental Approval Permissions",
          "permissionSets": [
            {
              "name": "First Parent Decision",
              "allowedPermissions": [
                {
                  "identity": "alice@example.com",
                  "identityType": "User"
                }
              ],
              "deniedPermissions": []
            },
            {
              "name": "Second Parent Decision",
              "allowedPermissions": [],
              "deniedPermissions": [
                {
                  "identity": "alice@example.com",
                  "identityType": "User"
                }
              ]
            }
          ]
        }
      ]

    },
    {
      "documenttype": "Movie Entry",
      "filename": "Spartacus.html",
      "permanentid": "spartacus3929393mnfvb30849923ncnv",
      "sourcetype": "Push",
      "title": "Spartacus",
      "duration": 163.72,
      "mmepsdirector": "Stanley Kubrick",
      "mmepsyear": "1960",
      "mmepssummary": "The slave Spartacus leads a violent revolt against the decadent Roman Republic.",
      "mmepsisseries": false,
      "mmepsmpaafilmrating": "PG-13",
      "mmepsbudget": 12.0,
      "mmepscriticsaverage": 87,
      "mmepscastoverview": [
        "Kirk Douglas",
        "Laurence Olivier",
        "Jean Simmons",
        "Charles Laughton",
        "Peter Ustinov"
      ],
      "mmepsthumbnaillink": "https://images-na.ssl-images-amazon.com/images/M/MV5BMjc4MTUxN2UtMmU1NC00MjQyLTk3YTYtZTQ0YzEzZDc0Njc0XkEyXkFqcGdeQXVyNjU0OTQ0OTY@._V1_SY999_CR0,0,660,999_AL_.jpg",
      "mmepstrailerlink": "https://www.youtube.com/watch?v=tFLCEDLCSHA",
      "data": "<html><head> <style>body{font-family: 'Lato', 'Helvetica Neue', Helvetica, Arial, sans-serif, sans-serif; font-size: 15px; color: #373737;}a{text-decoration: none; color: #009DDC; cursor: pointer;}a:hover{text-decoration: underline;}.left-column{float: left; width: 200px; padding: 5px 15px;}.right-column{overflow: hidden; padding: 5px 10px 10px 15px;}.summary{word-wrap: break-word;}.title h1{color: #67768B; text-transform: uppercase; white-space: normal; margin: 0px;}.director-year{color: #67768B;}.rating{width: 60px; text-align: center; border-width: 1px; border-style: solid; border-radius: 5px;}.rating span{margin: 0px; border-radius: 5px; display: block;}.rating span.pg-13-rating{color: white; background: purple;}.duration{font-size: 12px;}.trailer{border-bottom: 1px solid #BCC3CA; padding: 16px 0px 16px 0px;}</style></head><body> <div class='left-column'> <img src='https://images-na.ssl-images-amazon.com/images/M/MV5BMjc4MTUxN2UtMmU1NC00MjQyLTk3YTYtZTQ0YzEzZDc0Njc0XkEyXkFqcGdeQXVyNjU0OTQ0OTY@._V1_SY999_CR0,0,660,999_AL_.jpg' width='200'> </div><div class='right-column'> <div class='title'> <h1>Spartacus</h1> </div><div class='director-year'> <p>Stanley Kubrick - 1960</p></div><div class='duration'> <p>3h17</p></div><div class='rating'> <span class='pg-13-rating'>PG-13</span> </div><div class='summary'> <p>The slave Spartacus leads a violent revolt against the decadent Roman Republic.</p></div><div class='trailer'> <a href='https://www.youtube.com/watch?v=tFLCEDLCSHA'>View trailer</a> </div></div></body></html>",
      "documentId": "file://movie-entries/1960/Spartacus.html",
      "fileExtension": ".html",
      "permissions": [
        {
          "name": "MPAA Default Permissions",
          "permissionSets": [
            {
              "allowAnonymous": false,
              "allowedPermissions": [
                {
                  "identity": "Adults",
                  "identityType": "Group"
                },
                {
                  "identity": "Teenagers",
                  "identityType": "Group"
                }
              ],
              "deniedPermissions": [
                {
                  "identity": "YoungChildren",
                  "identityType": "Group"
                }
              ]
            }
          ]

        },
        {
          "name": "Parental Approval Permissions",
          "permissionSets": [
            {
              "name": "First Parent Decision",
              "allowedPermissions": [
                {
                  "identity": "bob@example.com",
                  "identityType": "User"
                },
                {
                  "identity": "david@example.com",
                  "identityType": "User"
                },
                {
                  "identity": "chloe@example.com",
                  "identityType": "User"
                }
              ],
              "deniedPermissions": []
            },
            {
              "name": "Second Parent Decision",
              "allowedPermissions": [
                {
                  "identity": "bob@example.com",
                  "identityType": "User"
                },
                {
                  "identity": "david@example.com",
                  "identityType": "User"
                }
              ],
              "deniedPermissions": []
            }
          ]
        }
      ]
    },
    {
      "documenttype": "Movie Entry",
      "filename": "Dangerous-Liaisons.html",
      "permanentid": "dangerous3920934n209823ncnvvn83",
      "sourcetype": "Push",
      "title": "Dangerous Liaisons",
      "duration": 88.47,
      "mmepsdirector": "Stephen Frears",
      "mmepsyear": "1988",
      "mmepssummary": "A scheming widow and her manipulative ex-lover make a bet regarding the corruption of a recently married woman.",
      "mmepsisseries": false,
      "mmepsmpaafilmrating": "R",
      "mmepsbudget": 14.0,
      "mmepscriticsaverage": 74,
      "mmepscastoverview": [
        "Glenn Close",
        "John Malkovich",
        "Michelle Pfeiffer",
        "Swoosie Kurtz",
        "Keanu Reeves"
      ],
      "mmepsthumbnaillink": "https://images-na.ssl-images-amazon.com/images/M/MV5BMDZmODc0MWQtYWVjOS00OGFiLWExNjItOTMyN2Q5ZTFmYzUzXkEyXkFqcGdeQXVyMTAwMzUyOTc@._V1_.jpg",
      "mmepstrailerlink": "https://www.youtube.com/watch?v=FbB2oBlP2uI",
      "data": "<html> <head> <style>body{font-family: 'Lato', 'Helvetica Neue', Helvetica, Arial, sans-serif, sans-serif; font-size: 15px; color: #373737;}a{text-decoration: none; color: #009DDC; cursor: pointer;}a:hover{text-decoration: underline;}.left-column{float: left; width: 200px; padding: 5px 15px;}.right-column{overflow: hidden; padding: 5px 10px 10px 15px;}.summary{word-wrap: break-word;}.title h1{color: #67768B; text-transform: uppercase; white-space: normal; margin: 0px;}.director-year{color: #67768B;}.rating{width: 60px; text-align: center; border-width: 1px; border-style: solid; border-radius: 5px;}.rating span{margin: 0px; border-radius: 5px; display: block;}.rating span.r-rating{color: white; background: red;}.duration{font-size: 12px;}.trailer{border-bottom: 1px solid #BCC3CA; padding: 16px 0px 16px 0px;}</style> </head> <body> <div class='left-column'> <img src='https://images-na.ssl-images-amazon.com/images/M/MV5BMDZmODc0MWQtYWVjOS00OGFiLWExNjItOTMyN2Q5ZTFmYzUzXkEyXkFqcGdeQXVyMTAwMzUyOTc@._V1_.jpg' width='200'> </div><div class='right-column'> <div class='title'> <h1>Dangerous Liaisons</h1> </div><div class='director-year'> <p>Stephen Frears - 1988</p></div><div class='duration'> <p>1h59</p></div><div class='rating'> <span class='r-rating'>R</span> </div><div class='summary'> <p>A scheming widow and her manipulative ex-lover make a bet regarding the corruption of a recently married woman.</p></div><div class='trailer'> <a href='https://www.youtube.com/watch?v=FbB2oBlP2uI'>View trailer</a> </div></div></body></html>",
      "documentId": "file://movie-entries/1988/Dangerous-Liaisons.html",
      "fileExtension": ".html",
      "permissions": [
        {
          "name": "MPAA Default Permissions",
          "permissionSets": [
            {
              "allowAnonymous": false,
              "allowedPermissions": [
                {
                  "identity": "Adults",
                  "identityType": "Group"
                },
                {
                  "identity": "Teenagers",
                  "identityType": "Group"
                }
              ],
              "deniedPermissions": [
                {
                  "identity": "YoungChildren",
                  "identityType": "Group"
                },
                {
                  "identity": "PreTeenagers",
                  "identityType": "Group"
                }
              ]
            }
          ]
        },
        {
          "name": "Parental Approval Permissions",
          "permissionSets": [
            {
              "name": "First Parent Decision",
              "allowedPermissions": [],
              "deniedPermissions": []
            },
            {
              "name": "Second Parent Decision",
              "allowedPermissions": [],
              "deniedPermissions": []
            }
          ]
        }
      ]
    },
    {
      "documenttype": "Movie Entry",
      "filename": "The-Evil-Dead.html",
      "permanentid": "th33vil39mmdmd93mc393309534m",
      "sourcetype": "Push",
      "title": "The Evil Dead",
      "duration": 125.37,
      "mmepsdirector": "Sam Raimi",
      "mmepsyear": "1981",
      "mmepssummary": "Five friends travel to a cabin in the woods, where they unknowingly release flesh-possessing demons.",
      "mmepsisseries": true,
      "mmepsmpaafilmrating": "NC-17",
      "mmepsbudget": 0.35,
      "mmepscriticsaverage": 70,
      "mmepscastoverview": [
        "Bruce Campbell",
        "Ellen Sandweiss",
        "Richard DeManincor",
        "Betsy Baker",
        "Theresa Tilly"
      ],
      "mmepsthumbnaillink": "https://images-na.ssl-images-amazon.com/images/M/MV5BODc2MmVjZmUtNjAzMS00MDNiLWIyM2YtOGEzMjg0YjRhMzRmXkEyXkFqcGdeQXVyMTQxNzMzNDI@._V1_UX182_CR0,0,182,268_AL_.jpg",
      "mmepstrailerlink": "https://www.youtube.com/watch?v=jgw4ZdC2XTE",
      "data": "<html><head> <style>body{font-family: 'Lato', 'Helvetica Neue', Helvetica, Arial, sans-serif, sans-serif; font-size: 15px; color: #373737;}a{text-decoration: none; color: #009DDC; cursor: pointer;}a:hover{text-decoration: underline;}.left-column{float: left; width: 200px; padding: 5px 15px;}.right-column{overflow: hidden; padding: 5px 10px 10px 15px;}.summary{word-wrap: break-word;}.title h1{color: #67768B; text-transform: uppercase; white-space: normal; margin: 0px;}.director-year{color: #67768B;}.rating{width: 60px; text-align: center; border-width: 1px; border-style: solid; border-radius: 5px;}.rating span{margin: 0px; border-radius: 5px; display: block;}.rating span.nc-17{color: white; background: black;}.duration{font-size: 12px;}.trailer{border-bottom: 1px solid #BCC3CA; padding: 16px 0px 16px 0px;}</style></head><body> <div class='left-column'> <img src='https://images-na.ssl-images-amazon.com/images/M/MV5BODc2MmVjZmUtNjAzMS00MDNiLWIyM2YtOGEzMjg0YjRhMzRmXkEyXkFqcGdeQXVyMTQxNzMzNDI@._V1_UX182_CR0,0,182,268_AL_.jpg' width='200'> </div><div class='right-column'> <div class='title'> <h1>The Evil Dead</h1> </div><div class='director-year'> <p>Sam Raimi - 1981</p></div><div class='duration'> <p>1h25</p></div><div class='rating'><span class='NC-17'>NC-17</span> </div><div class='summary'> <p>Five friends travel to a cabin in the woods, where they unknowingly release flesh-possessing demons.</p></div><div class='trailer'> <a href='https://www.youtube.com/watch?v=jgw4ZdC2XTE'>View trailer</a> </div></div></body></html>",
      "documentId": "file://movie-entries/1981/The-Evil-Dead.html",
      "fileExtension": ".html",
      "permissions": [
        {
          "name": "MPAA Default Permissions",
          "permissionSets": [
            {
              "allowAnonymous": false,
              "allowedPermissions": [
                {
                  "identity": "Adults",
                  "identityType": "Group"
                }
              ],
              "deniedPermissions": [
                {
                  "identity": "YoungChildren",
                  "identityType": "Group"
                },
                {
                  "identity": "PreTeenagers",
                  "identityType": "Group"
                },
                {
                  "identity": "Teenagers",
                  "identityType": "Group"
                }
              ]
            }
          ]
        }
      ]
    },
    {
      "documenttype": "Movie Entry",
      "filename": "1900.html",
      "permanentid": "19002ndndn3n39293092553ncf859802ncsdg",
      "sourcetype": "Push",
      "title": "1900",
      "duration": 175.92,
      "mmepsdirector": "Bernardo Bertolucci",
      "mmepsyear": 1976,
      "mmepssummary": "The epic tale of a class struggle in twentieth century Italy, as seen through the eyes of two childhood friends on opposing sides.",
      "mmepsisseries": false,
      "mmepsmpaafilmrating": "UR",
      "mmepsbudget": 9.0,
      "mmepscriticsaverage": 85,
      "mmepscastoverview": [
        "Robert De Niro",
        "Gérard Depardieu",
        "Dominique Sanda",
        "Francesca Bertini",
        "Laura Betti"
      ],
      "mmepsthumbnaillink": "https://upload.wikimedia.org/wikipedia/en/thumb/0/07/1900_Bertolluci.jpg/220px-1900_Bertolluci.jpg",
      "mmepstrailerlink": "https://www.youtube.com/watch?v=uGyH6Tt6qwY",
      "data": "<html><head> <style>body{font-family: 'Lato', 'Helvetica Neue', Helvetica, Arial, sans-serif, sans-serif; font-size: 15px; color: #373737;}a{text-decoration: none; color: #009DDC; cursor: pointer;}a:hover{text-decoration: underline;}.left-column{float: left; width: 200px; padding: 5px 15px;}.right-column{overflow: hidden; padding: 5px 10px 10px 15px;}.summary{word-wrap: break-word;}.title h1{color: #67768B; text-transform: uppercase; white-space: normal; margin: 0px;}.director-year{color: #67768B;}.rating{width: 60px; text-align: center; border-width: 1px; border-style: solid; border-radius: 5px;}.rating span{margin: 0px; border-radius: 5px; display: block;}.rating span.ur{color: white; background: blue;}.duration{font-size: 12px;}.trailer{border-bottom: 1px solid #BCC3CA; padding: 16px 0px 16px 0px;}</style></head><body> <div class='left-column'> <img src='https://upload.wikimedia.org/wikipedia/en/thumb/0/07/1900_Bertolluci.jpg/220px-1900_Bertolluci.jpg' width='200'> </div><div class='right-column'> <div class='title'> <h1>1900</h1> </div><div class='director-year'> <p>Bernardo Bertolucci - 1976</p></div><div class='duration'> <p>5h17</p></div><div class='rating'><span class='ur'>UR</span> </div><div class='summary'> <p>The epic tale of a class struggle in twentieth century Italy, as seen through the eyes of two childhood friends on opposing sides.</p></div><div class='trailer'> <a href='https://www.youtube.com/watch?v=uGyH6Tt6qwY'>View trailer</a> </div></div></body></html>",
      "documentId": "file://movie-entries/1976/1900.html",
      "fileExtension": ".html",
      "permissions": [
        {
          "name": "MPAA Default Permissions",
          "permissionSets": [
            {
              "allowAnonymous": false,
              "allowedPermissions": [
                {
                  "identity": "Adults",
                  "identityType": "Group"
                }
              ],
              "deniedPermissions": []
            }
          ]
        },
        {
          "name": "Additional Rating Permissions",
          "permissionSets": [
            {
              "allowAnonymous": false,
              "allowedPermissions": [],
              "deniedPermissions": [
                {
                  "identity": "YoungChildren",
                  "identityType": "Group"
                },
                {
                  "identity": "PreTeenagers",
                  "identityType": "Group"
                }
              ]
            }
          ]
        },
        {
          "name": "Parental Approval Permissions",
          "permissionSets": [
            {
              "name": "First Parent Decision",
              "allowedPermissions": [
                {
                  "identity": "chloe@example.com",
                  "identityType": "User"
                },
                {
                  "identity": "erin@example.com",
                  "identityType": "User"
                }
              ],
              "deniedPermissions": [
                {
                  "identity": "frank@example.com",
                  "identityType": "User"
                }
              ]
            },
            {
              "name": "Second Parent Decision",
              "allowedPermissions": [
                {
                  "identity": "chloe@example.com",
                  "identityType": "User"
                },
                {
                  "identity": "erin@example.com",
                  "identityType": "User"
                },
                {
                  "identity": "frank@example.com",
                  "identityType": "User"
                }
              ],
              "deniedPermissions": []
            }
          ]
        }
      ]
    }
  ],
  "delete": []
}

You may have to wait several minutes before these items become available in your source (see About the Push API Processing Delay).

Step 4: Add a Batch of Security Identities

Whenever you use the Push API to add, update, and/or delete items in a secured Push source, you should ensure that the security identity definitions to which these items refer in their permission models are up to date in the security identity cache of your Coveo organization. Otherwise, certain users may not be able to see some of the items for which they should have access. Even worse, other users may be able to see items for which they should no longer have access.

In this step, you’ll use a batch operation to define several security identities at once. The items that you added in your secured Push source in step 3 refer to these security identities in their permission models.

In a real-life scenario, you would use your own code to extract security identities and security identity relationships from a secured enterprise system.

The JSON below defines 9 individual users (Alice, Bob, Chloe, David, Erin, Frank, Grace, Henry, and yourself), four groups (YoungChildren, PreTeenagers, Teenagers, and Adults), and one granted identity (Everyone). It also defines an alias to your own security identity in Coveo (for this to work, you must replace <MyEmailAddress> towards the end of this JSON with the actual email address with which you authenticate in Coveo).

Typically, you should define an alias relationship between each user security identity and its corresponding security identity in the Email Security Provider (see About the Email Security Provider).

This means that you’ll normally define each user in the mappings array (as the yourself user is defined in the JSON below) when pushing a batch of security identities, rather than in the members array.

This tutorial doesn’t define aliases for any of the fake users (for example, alice@example.com, bob@example.com, etc.), since they will only ever exist in the security identity provider of your secured Push source.

Every user belongs to a single group, as summarized in the following table:

User Group
alice@example.com YoungChildren
bob@example.com YoungChildren
chloe@example.com PreTeenagers
david@example.com PreTeenagers
erin@example.com Teenagers
frank@example.com Teenagers
grace@example.com Adults
henry@example.com Adults
yourself@example.com Adults

Each group also has the Everyone granted identity.

Push the following batch of security identities in the security identity provider of your secured Push source (see Manage Batches of Security Identities):

Security identity update request body template

{
  "members": [
    {
      "identity": {
        "name": "Everyone",
        "type": "GROUP"
      }
    },
    {
      "identity": {
        "name": "alice@example.com",
        "type": "USER"
      }
    },
    {
      "identity": {
        "name": "bob@example.com",
        "type": "USER"
      }
    },
    {
      "identity": {
        "name": "chloe@example.com",
        "type": "USER"
      }
    },
    {
      "identity": {
        "name": "david@example.com",
        "type": "USER"
      }
    },
    {
      "identity": {
        "name": "erin@example.com",
        "type": "USER"
      }
    },
    {
      "identity": {
        "name": "frank@example.com",
        "type": "USER"
      }
    },
    {
      "identity": {
        "name": "grace@example.com",
        "type": "USER"
      }
    },
    {
      "identity": {
        "name": "henry@example.com",
        "type": "USER"
      }
    },
    {
      "identity": {
        "name": "YoungChildren",
        "type": "GROUP"
      },
      "members": [
        {
          "name": "alice@example.com",
          "type": "USER"
        },
        {
          "name": "bob@example.com",
          "type": "USER"
        }
      ],
      "wellKnowns": [
        {
          "name": "Everyone",
          "type": "GROUP"
        }
      ]
    },
    {
      "identity": {
        "name": "PreTeenagers",
        "type": "GROUP"
      },
      "members": [
        {
          "name": "chloe@example.com",
          "type": "USER"
        },
        {
          "name": "david@example.com",
          "type": "USER"
        }
      ],
      "wellKnowns": [
        {
          "name": "Everyone",
          "type": "GROUP"
        }
      ]
    },
    {
      "identity": {
        "name": "Teenagers",
        "type": "GROUP"
      },
      "members": [
        {
          "name": "erin@example.com",
          "type": "USER"
        },
        {
          "name": "frank@example.com",
          "type": "USER"
        }
      ],
      "wellKnowns": [
        {
          "name": "Everyone",
          "type": "GROUP"
        }
      ]
    },
    {
      "identity": {
        "name": "Adults",
        "type": "GROUP"
      },
      "members": [
        {
          "name": "grace@example.com",
          "type": "USER"
        },
        {
          "name": "henry@example.com",
          "type": "USER"
        },
        {
          "name": "yourself@example.com",
          "type": "USER"
        }
      ],
      "wellKnowns": [
        {
          "name": "Everyone",
          "type": "GROUP"
        }
      ]
    }
  ],
  "mappings": [
    {
      "identity": {
        "name": "yourself@example.com",
        "type": "USER"
      },
      "mappings": [
        {
          "name": <MyEmailAddress>,
          "type": "USER",
          "provider": "Email Security Provider"
        }
      ]
    }
  ],
  "deleted": []
}

In the request body above, don’t forget to replace <MyEmailAddress> with the actual email address with which you authenticate in Coveo.

Step 5: (Optional) Validate Item Field Values, Effective Permissions, and Data

You’ll now use the Coveo Administration Console to manually verify that the metadata of the items you just pushed has populated the fields in your index as expected, and that the permission models of your items are being evaluated correctly.

  1. In the Coveo Administration Console, access the Content Browser (platform-ca | platform-eu | platform-au) page.

  2. In the search box, type @source==<MyPushSource> (replace <MyPushSource> with the name of your secured Push source) and select Enter.

    The six items that you pushed in your source should appear in the result list.

  3. Select the result whose title is 1900, and then click Properties in the Action bar.

    A panel should open in which the Fields tab is selected by default. You should see a list of mapped fields for this item, along with their corresponding values. You can now validate that:

    • The title and mmepstitle fields have the same value (1900), which means that the first mapping rule you defined in step 2 was correctly applied.

    • The duration and mmepsduration fields have the same value (175.92), which means that the second mapping rule you defined in step 2 was correctly applied.

    • The " million USD" string was appended to the end of mmepsbudget, which means that the third rule you defined in step 2 was correctly applied.

    • The value of the mmepsmpaafilmrating field was set to the "UR" string, which implies that the fourth mapping rule you defined in step 2 was correctly applied, since this item was pushed without any mmepsmpaafimrating metadata.

  4. In the header of the panel, select the Permissions tab.

    You should see the list of permissions for the 1900 item. As explained in the example in step 3:

    • The only allowed users are Erin, Grace, Henry, and yourself.

    • The YoungChildren and PreTeenagers groups, as well as the user Frank, are explicitly denied.

  5. In the header of the panel, select the Permission details tab.

    You should see that this item has three permission levels. The first two levels have only one permission set, whereas the last one has two. As explained in the example in step 3:

    • Permission Level 0 (that is, the MPAA film rating-based permission level) explicitly allows the Adults group.

    • Permission Level 1 (that is, the additional film rating-based permission level) explicitly denies the YoungChildren and PreTeenagers groups.

    • Permission Level 2 (that is, the parental approval permission level) explicitly allows the user Erin, and denies the user Frank.

  6. In the header of the panel, select the Quick view tab.

    The HTML and CSS that was provided as the value of the Data property for this item should render as expected.

You can validate the other items that you added in your secured Push source in step 3 in the same fashion.