REST API source permission configuration

This is for:

Developer System Administrator

When writing your REST API source JSON configuration, you can choose to index the access permissions securing your content and replicate them in your Coveo-powered search interface. As a result, through your search interface, authenticated users only see the items that they’re allowed to access within the indexed repository. Anonymous and unauthenticated users are impossible to relate to a security identity in this repository, so they can’t access the source content.

If, in your content system, access permissions are saved as item metadata, you can index them with one of the following methods. The method to use depends on your permission system.

  1. If your permission system doesn’t use group security identities and identifies security identities with email addresses, you could index your permission data through your source configuration only. This option works if your permission system is relatively straightforward and doesn’t change frequently, as an update operation is required whenever your permission system is modified (user addition/deletion or access permission modification) to keep your source’s permission system identical to your application’s. Check the source configuration documentation for an example of permissions indexed directly in the source configuration.

  2. If your permission system is more complex, in addition to a source configuration that retrieves item permissions, you must provide a permission configuration defining how Coveo should retrieve the relationships of the security identities indexed thanks to your source configuration. This allows Coveo to define the effective permissions of each item.

In either case, in the Content Security tab of the Add a REST API Source panel, you must select the Same users and groups as in your content system option. The difference is, if you choose option 1, you don’t need to provide a permission configuration. However, with option 2, you must instruct Coveo on how to extract the users from the groups that have been identified as allowed or denied access to your indexed items.

Your workflow with option 2 should be the following:

  1. In your source configuration, you provide a PermissionType value. This value identifies the configuration that the security identity provider should use to extract the relationships of the security identities indexed thanks to your source configuration.

  2. In the Content Security tab, you provide at least one permissionSubQueries object for each PermissionType specified in your source configuration. This object must contain the parameters of a query Coveo should make to your application to retrieve the relationships of the security identities indexed thanks to your source configuration.

  3. Depending on the information returned in your application’s JSON response, you may need to add additional permissionSubQueries objects or PermissionsFromMetadata objects to retrieve all user security identities in each group identity.

For example, let’s say you want to index a GitHub repository. With the following source configuration, you would index the repository specified under path as well as the list of teams that are allowed to access this repository.

{
  "services": [
    {
      "url": "https://api.github.com",
      "headers": {
            "accept": "application/vnd.github.v3+json",
            "User-Agent": "PostmanRuntime/7.29.0"
      },
      "authentication": {
        "username": "@username",
        "password": "@password",
        "forceBasicAuthentication": "true"
      },
      "endpoints": [
        {
          "path": "/repos/mycompany/teamsandbox",
          "method": "GET",
          "itemPath": "",
          "itemType": "Repo",
          "uri": "%[coveo_url]/repo/%[id]",
          "clickableUri": "%[url]",
          "title": "%[full_name]",
          "modifiedDate": "%[updated_at]",
          "body": "%[description]",
          "metadata": {
            "default_branch": "%[default_branch]"
          },
          "permissions": [
            {
              "name": "AllowedMemberLevel",
              "permissionsSets": [
                {
                  "name": "AllowedMembersSet",
                  "isAnonymousAllowed": "false",
                  "PermissionSubQueries": [
                    {
                      "path": "/repos/mycompany/teamsandbox/teams",
                      "method": "GET",
                      "itemPath": "",
                      "isAllowedMember": "true",
                      "name": "%[name]",
                      "permissionType": "GithubGroups",
                      "type": "Group",
                      "optional": "false",
                      "additionalInfo": {
                        "members_url": "%[url]/members"
                      }
                    }
                  ]
                }
              ]
            }
          ]
        }
      ]
    }
  ]
}

Next, you would need to extract the users of each team group with a PermissionSubQueries object, and then associate these users with an email address, either using another PermissionSubQueries object or with a PermissionsFromMetadata object. The following permission configurations showcase each option. In this specific case, the results of these two configurations are identical.

  • With an additional permissionSubQueries object:

    {
  "url": "https://api.github.com",
  "headers": {
          "accept": "application/vnd.github.v3+json",
          "User-Agent": "PostmanRuntime/7.29.0"
  },
  "authentication": {
    "username": "@username",
    "password": "@password",
    "forceBasicAuthentication": "true"
  },
  "Identities": {
    "GithubGroups": {
      "PermissionSubQueries": [{
        "ItemPath": "",
        "Path": "%[coveo_parent.members_url]",
        "Method": "GET",
        "Name": "%[login]",
        "additionalInfo": {
          "url": "%[url]"
        },
        "Type": "User",
        "PermissionType": "GithubUsers"
      }
      ]
    },
      "GithubUsers": {
        "PermissionSubQueries": [{
          "ItemPath": "",
          "Path": "%[coveo_parent.url]",
          "Method": "GET",
          "Name": "%[login]@example.com",
          "Type": "User"
        }
        ]
      }
    }
}

  • With a PermissionsFromMetadata object:

    {
  "url": "https://api.github.com",
  "headers": {
          "accept": "application/vnd.github.v3+json",
          "User-Agent": "PostmanRuntime/7.29.0"
  },
  "authentication": {
    "username": "@username",
    "password": "@password",
    "forceBasicAuthentication": "true"
  },
  "Identities": {
    "GithubGroups": {
      "PermissionSubQueries": [{
        "ItemPath": "",
        "Path": "%[coveo_parent.members_url]",
        "Method": "GET",
        "Name": "%[id]",
        "additionalInfo": {
          "userlogin": "%[login]"
        },
        "Type": "User",
        "PermissionType": "GithubUsers"
      }
      ]
    },
    "GithubUsers": {
      "PermissionsFromMetadata": [{
        "Name": "%[coveo_parent.userlogin]@example.com",
        "Type": "User"
      }
      ]
    }
  }
}

Url (String, Required)

The Url value is the service URL of the web application from which you want to retrieve permission data.

Example: "Url": "http://example.com/api/v1"

Identities (Identities Object, Required)

The Identities object contains objects named after the PermissionType parameter values specified in your source configuration. Their name allows Coveo to link the identities retrieved thanks to the source configuration Permission Set and Permission Subquery arrays with the desired extraction instructions.

Each of these objects details how to extract the relationships of a security identity retrieved with the source configuration. The security provider will use this configuration when processing the security identity.

The Identities object includes:

  • Everything Coveo needs to make an API call and retrieve the permission data.

  • Mapping information, that is, how Coveo should store this data.

Example:

In your source JSON configuration, you specified that retrieved security identities should be extracted according to the MyPermissionType1 configuration:

"AllowedMembers": [
  {
    "Name": "%[author]",
    "Type": "User",
    "PermissionType": "MyPermissionType1",
    "Condition": "%[field] == value",
    "AdditionalInfo": {
      "title": "%[customMetadata]"
    }
  }
]

So, in your permission JSON configuration, you provide the MyPermissionType1 object.

The MyPermissionType1 object details how to extract the relationships of the security identities retrieved thanks to your source JSON configuration. It contains the "permissionType": "MyPermissionType2" parameter, meaning that to extract the relationships of the extracted identities, the security identity provider must use the MyPermissionType2 configuration.

"identities": {
  "MyPermissionType1": {
    "permissionSubQueries": [
      {
        "path": "string",
        "method": "GET",
        "headers": {
          "key": "value"
        },
        "queryParameters": {
          "key": "value"
        },
        "itemPath": "string | %[string]",
        "paging": {
          "key": "value"
        },
        "name": "%[fullname]",
        "permissionType": "MyPermissionType2",
        "type": "group",
        "additionalInfo": {
          "key": "value"
        }
      }
    ]
},
  "MyPermissionType2": {
    "permissionSubQueries": [
      {
        "path": "string",
        "method": "GET",
        "headers": {
          "key": "value"
        },
        "queryParameters": {
          "key": "value"
        },
        "itemPath": "string | %[string]",
        "paging": {
          "key": "value"
        },
        "name": "%[fullname]",
        "type": "user",
        "additionalInfo": {
          "key": "value"
        }
      }
    ]
  }
}

Authentication (Authentication Object)

If the service to crawl requires authentication, your JSON configuration must include your authentication properties.

See the source configuration Authentication object documentation for details on the properties in this object. Both Authentication objects work similarly, so you should use placeholders such as @Username to retrieve the value specified in the Add a REST API Source panel.

Identities Object

Each object in the Identities object is a configuration for the security provider to use when extracting the relationships of a security identity.

PermissionType (PermissionType Object, Required)

PermissionType is a placeholder for the key that you specified in the source configuration. This key identifies the configuration the security identity provider should use to extract the relationships of the security identity retrieved with the source configuration.

Dynamic values are supported.

Paging (Paging Object)

In the Paging object, specify how you want the permission data to be paged.

This property is inheritable.

See the source configuration Paging object documentation for details on the properties in this object. Both Paging objects work similarly.

PermissionsFromMetadata Object

The PermissionsFromMetadata object is an alternative to the PermissionSubQueries object, as it fetches security identity relationship data, typically the security identities that are children of a group. This data is crucial to retrieve if you want to replicate the application’s permission system in Coveo.

Include a PermissionsFromMetadata object in your source permission configuration when the two following conditions are met:

  1. The permission data extracted with your source configuration is susceptible to contain group security identities.

  2. The metadata of these identities contains information on their children.

In other words, the PermissionsFromMetadata object is useful if, when Coveo queries your application to get the children of a group security identity indexed as part of the permission data of an item, the response contains the identity’s children as email addresses. Leverage PermissionsFromMetadata to avoid unnecessary new queries to the application you’re indexing.

The following is an example of a permission configuration in such a scenario:

{
  "url": "https://api.github.com",
  "authentication": {
    "username": "@username",
    "password": "@password",
    "forceBasicAuthentication": "true"
  },
  "Identities": {
    "GithubGroups": {
      "PermissionSubQueries": [{
        "ItemPath": "",
        "Path": "%[coveo_parent.members_url]",
        "Method": "GET",
        "headers": {
          "accept": "application/vnd.github.v3+json",
          "User-Agent": "PostmanRuntime/7.29.0"
        },
        "Name": "%[id]",
        "additionalInfo": {
          "userlogin": "%[login]"
        },
        "Type": "User",
        "PermissionType": "GithubUsers"
      }
      ]
    },
    "GithubUsers": {
      "PermissionsFromMetadata": [{
        "Name": "%[coveo_parent.userlogin]@example.com",
        "Type": "User"
      }
      ]
    }
  }
}

Conversely, a second PermissionSubQueries object should be used when Coveo only receives a PermissionType value and must make a new query to retrieve the children of a group security identity.

For more information on sources that index permissions and on how Coveo handles these permissions, see Coveo management of security identities and item permissions.

In a PermissionsFromMetadata object, you must use the coveo_parent dynamic value. In this context, the parent item is the security identity indexed thanks to your source configuration.

The PermissionsFromMetadata object contains a subset of the fields from the PermissionSubQueries object configuration:

PermissionSubQueries Object

In the PermissionSubQueries object, you must specify the parameters of the query to make to retrieve the relationships of a security identity.

Example:

{
  "url": "https://api.github.com",
  "authentication": {
    "username": "@username",
    "password": "@password",
    "forceBasicAuthentication": "true"
  },
  "Identities": {
    "GithubGroups": {
      "PermissionSubQueries": [{
        "ItemPath": "",
        "Path": "%[coveo_parent.members_url]",
        "Method": "GET",
        "headers": {
          "accept": "application/vnd.github.v3+json",
          "User-Agent": "PostmanRuntime/7.29.0"
        },
        "Name": "%[login]",
        "additionalInfo": {
          "url": "%[url]"
        },
        "Type": "User",
        "PermissionType": "GithubUsers"
      }
      ]
    },
      "GithubUsers": {
        "PermissionSubQueries": [{
          "ItemPath": "",
          "Path": "%[coveo_parent.url]",
          "Method": "GET",
          "headers": {
            "accept": "application/vnd.github.v3+json",
            "User-Agent": "PostmanRuntime/7.29.0"
          },
          "Name": "%[login]@example.com",
          "Type": "User"
        }
        ]
      }
    }
}

Method (String Enum, Required)

The HTTP method to use to fetch the resource. Possible values are GET, POST, and PUT. See the API documentation of the application to make searchable to determine which one you should use.

Example: "method": "GET"

Member Name (String, Required)

The name of the extracted member security identity. Email addresses and Active Directory usernames are supported.

Dynamic values are supported.

Note

By default, the source uses an Email Security Provider and therefore expects an email address under Name.

Path (String, Required)

The relative path to the desired permission data endpoint.

Example: "Path": "accesspermissions/groups/supportagents"

Type (String Enum, Required)

The type of member. Allowed values are User, Group, and VirtualGroup.

For more information on these types of security identities, see Coveo management of security identities and item permissions.

AdditionalInfo Object

A key-value list of the additional information required by some security providers. Each key represents the metadata name of the repository item, while its value is the value path (simple path or JSONPath) in the JSON response.

Example:

Your source configuration contains the following Permissions object, which indicates that members of the Support Agents group should be allowed to access the indexed content:

"Permissions": [
  {
    "Name": "Permission Level 1",
    "PermissionsSets": [
      {
        "Name": "Permission Set 1",
        "AllowedMembers": [
          {
            "Name": "Support Agents",
            "Type": "Group",
            "PermissionType": "MyPermissionType2",
            "AdditionalInfo": {
              "group_id": "00841"
            }
          }
        ],
      }
    ]
  }
]

To extract the members of this group, you provide the following permission configuration. Therefore, the security identity provider will use the group_id you specified in the source configuration to make an API request and obtain the group members.

Then, the security provider will refer to the user instructions to retrieve the desired information on the extracted group members.

"identities": {
  "group": {
    "permissionSubQueries": [
      {
        "path": "/group/%[group_id]/members",
        "method": "GET",
        "itemPath": "members",
        "permissionType": "user",
        "name": "%[username]",
        "type": "User",
        "additionalInfo": {
          "user_id": "%[user_id]"
        }
      }
    ]
  },
  "user": {
    "permissionSubQueries": [
      {
        "path": "/users/%[user_id]",
        "method": "GET",
        "itemPath": "user",
        "name": "%[email]",
        "type": "User"
      }
    ]
  }
}

Condition (String)

The Condition value is a condition that must resolve to true for the security identity extracted from the security identity retrieved with the source configuration to be resolved, that is, for its permission information to be extracted by the security identity provider. As a result, the permission applies.

When the condition resolves to false, the security identity is not resolved and the corresponding permission doesn’t apply.

To write your condition, use the same syntax as for indexing pipeline extension conditions. Conditions can be assembled using the following operators: AND, OR, Exists, NOT, >, and <. Parentheses are also supported to specify operation order.

The metadata fields in your condition must either be defined in the source configuration Metadata object or referenced with raw. You can specify an array if the metadata refers to a multi-value field in your repository. Dynamic values are also supported.

Headers Object

A key-value list of HTTP headers to add to the query. Each value can be either a number, string, or Boolean.

Example:

"Headers": {
  "Custom-Header-Name": "myValue",
  "X-Api-Key": "1234567890"
}

ItemPath (String)

The path to the item in the JSON response returned by the application.

When ItemPath is omitted, Coveo assumes that the permission data is at the root level of the JSON response.

Dynamic values are supported.

Note

ItemPath and StringItemPath are mutually exclusive. In other words, if you use ItemPath in a permission subquery, you can’t also use StringItemPath, and vice versa.

Optional (Boolean)

The Optional value indicates whether the specified Condition is optional when there is no match. In other words, if the condition doesn’t resolve to true for any of the members extracted of a group, the security identity provider either:

  • Resolves no security identity if you set "optional": false. As a result, none of the group member identities is considered as allowed to access the indexed content.

  • Resolves all security identities in the group if you set "optional": true, and therefore all security identities in the group are considered as allowed to access the indexed content.

Default value is false.

PermissionSubQueries Paging (Paging Object)

In the Paging object, specify how you want the permission data to be paged.

This paging configuration overrides any general paging configuration you may have provided.

See the source configuration Paging object documentation for details on the properties in this object. All Paging objects work similarly.

QueryParameters Object

A key-value list of HTTP parameters to add to the query. Each value can be either a number, string, or Boolean.

Dynamic values are supported.

Example:

"QueryParameters": {
  "type": "post",
  "expand": "true",
  "id": 120
}

StringItemPath (String)

The path to the item in the JSON response returned by your API.

When indexing permissions with PermissionSubQueries, use StringItemPath in place of ItemPath if your API returns the list of identities in an array rather than as a series of objects.

Then, use value %[item] to tell Coveo where you want to store each of the identities extracted from the permission array. The Name property is an appropriate place.

Read more on StringItemPath vs. ItemPath

Most APIs return permissions as a series of objects such as:

{
  "allowed": [
    {
      "email": "jsmith@example.com"
    },
    {
      "email": "ballen@example.com"
    }
  ]
}

In such a case, you would use ItemPath to extract permissions.

However, if your API rather returns permissions as follows, you should use StringItemPath instead.

{
  "allowed": ["jsmith@example.com", "ballen@example.com"]
}
Example

If you expect your API to return permissions in an array such as:

{
  "allowedusers": ["jsmith@example.com", "ballen@example.com"]
}

Include the following in your permission subquery: "StringItemPath": "allowedusers".

In addition, to index the extracted user identities under Name, add: "Name": %[item].

Your permission subquery therefore looks as follows:

"permissionSubQueries": [
  {
    "StringItemPath": "allowedusers",
    "Path": "item/location",
    "Method": "GET",
    "Name": "%[item]",
    "Type": "User",
    "IsAllowedMember": true
  }
]

StringItemPath can appear in PermissionSubQueries objects only.

Note

ItemPath and StringItemPath are mutually exclusive. In other words, if you use StringItemPath in a subquery, you can’t also use ItemPath, and vice versa.

PermissionType Object

The object identified with the key that you specified in the source configuration is the configuration the security identity provider should use to extract the relationships of a security identity.

Example:

In your source JSON configuration, you specified that retrieved security identities are group security identities and that they should be extracted according to the extractgroupmembers configuration ("PermissionType": "extractgroupmembers").

"PermissionsSets": [
  {
    "Name": "Permission Set 1",
    "AllowedMembers": [
      {
        "Name": "%[email]",
        "Type": "group",
        "PermissionType": "extractgroupmembers"
      }
    ],
    "DeniedMembers": null,
    "IsAnonymousAllowed": false
  }
]

So, in your permission JSON configuration, you provide the extractgroupmembers object. This object details how to extract the members of each group retrieved thanks to your source JSON configuration.

"identities": {
  "extractgroupmembers": {
    "permissionSubQueries": [
      {
        "path": "string",
        "method": "GET",
        "headers": {
          "key": "value"
        },
        "queryParameters": {
          "key": "value"
        },
        "itemPath": "string | %[string]",
        "paging": {
          "key": "value"
        },
        "name": "%[fullname]",
        "type": "user",
        "additionalInfo": {
          "key": "value"
        }
      }
    ]
  }
}

PermissionSubqueries (Array of PermissionSubQueries Object, Required)

The PermissionSubQueries object contains the queries to make in order to retrieve your permission data and extract the relationships of a security identity. It is an alternative to the PermissionsFromMetadata object.

PermissionsFromMetadata (Array of PermissionsFromMetadata Object)

The PermissionsFromMetadata object is an alternative to the PermissionSubQueries object, as it fetches security identity relationship data, typically the security identities that are children of a group.

SkippableErrorCodes (String)

When crawling your permission data, Coveo may encounter an HTTP error and stop. If you expect the web application API to return an error and want it to be ignored so that the crawling process continues, specify the error code as the SkippableErrorCodes property value.

Use a semicolon (;) to separate two error codes.

Example: "SkippableErrorCodes": "500;404;403"