Manage organization snapshots

This is for:

Developer

The Coveo CLI can greatly facilitate resource snapshot management. While you could accomplish the same operations through the Administration Console or the Migration API, the CLI reduces manual intervention as much as possible, simplifying your deployments with modern infrastructure-as-code.

This article explains how to use the CLI to pull and push snapshots. You can use it to manage snapshots in a single Coveo organization or in multiple ones.

Single vs multi org management

Prerequisites

  • The Coveo CLI installed on your machine.

  • One or more organizations in which you have the privileges to push/pull snapshots.

Pull an Organization Snapshot

This section explains how to use the CLI to import a project constituting a snapshot of a target organization. Concretely, the project contains two folders:

  • .coveo/: the project metadata.

  • resources/: all the resources pulled from the organization.

Pull All Resources

To pull all resources available in your organization, use the coveo org:resources:pull command.

coveo org:resources:pull

Pull Specific Resources Types

If you only want to pull specific types of resources, for example because you have limited access or no interest in other resources, use the --resourceTypes flag. In addition to pulling the target resources, it also pulls all other resources that depend on them. For example, pulling a query pipeline would also retrieve the rules of this pipeline and the machine learning (ML) models associated with it.

To give another example, the following command pulls all the fields and all the sources available in your organization, along with the dependent resources, but nothing else.

coveo org:resources:pull --resourceTypes SOURCE FIELD

Pull Resources Using a Snapshot Model

You can also provide a snapshot model to specify individual resources to pull from the organization. The snapshot model is a JSON file containing the target resource IDs. Using a snapshot model can be useful when multiple teams are working on the same organization and you would like to pull only the resources you’re working on. It can also be helpful to automate tasks on a subset of resources in a CI/CD environment.

coveo org:resources:pull --model path/to/model.json

To create a snapshot model, run the following command.

coveo org:resources:model:create

This redirects you to the Administration Console. From there, select the specific resources to include in your snapshot model and click Save for CLI.

The following JSON model only pulls the myorgid-catalog and myorgid-inventory sources regardless of the other sources in the organization.

{
   "orgId": "myorgid",
   "resourcesToExport": {
       "SOURCE": [
           "myorgid-catalog"
           "myorgid-inventory"
       ]
   },
   "developerNotes": "Commerce Snapshot",
   "includeChildrenResources": true
}

Output

If the org:resources:pull command is successful, the terminal displays the ID of the organization from which you just pulled resources as well as a confirmation that a project in the current directory was updated:

Creating Snapshot from myorgid... ✔
Updating project with Snapshot... ✔

You will see two folders in your working directory:

  • .coveo/: contains the project metadata.

  • resources/: contains all the resources pulled from the organization.

Each resource type is saved in its respective JSON file. For instance, you will find all your source configurations in the resources/SOURCE.json file.

Push an Organization Snapshot

From the directory containing the target resources/ folder, run the coveo org:resources:push command to upload the local resources to the destination organization.

coveo org:resources:push -o <DESTINATION_ORG_ID>

The CLI will return a preview of the changes you’re about to apply to the destination organization. You will always be prompted for confirmation before applying any changes to an organization.

Notes

  • If you only want to view the changes of a snapshot with no side effect to your organization, use the coveo org:resources:preview command instead.

  • After it has applied a snapshot in the destination organization, the CLI deletes the snapshot from that destination organization, to avoid reaching the limit on the number of organization snapshots.

Handle Missing Vault Entries

After running the push command, you might receive a message asking you to transfer vault entries. Vault entries are sensitive and restricted key-value pairs stored in the Vault Resource of the Coveo Migration API. The API uses these entries to migrate sensitive information leveraged by your resources, such as credentials used to index content.

You will see this message if your snapshot contains at least one resource that depends on sensitive information missing in the destination organization. This is due to the fact that sensitive information isn’t transferred with the snapshot.

The message will look something like the following.

Missing vault entries in destination organization:
  • some_source-configuration.parameters.ConsumerSecret.value
  • some_source-configuration.parameters.ConsumerKey.value
  • some_source-configuration.parameters.AccessTokenSecret.value
  • some_source-configuration.parameters.AccessToken.value

Would you like to try transferring the vault entries from some_source to the destination organization destinationorg? (y/n):

In this example, the source some_source has four secrets not yet present in the destination organization. If you answer "yes", the CLI will try to transfer missing vault entries from the origin to the destination organization.

If the transfer isn’t possible, you will be asked to manually populate the missing vault entries.

Would you like to create the missing vault entries in the destination organization destinationorg? (y/n):

Answering "yes" to the above question opens a JSON file in your default editor. From there, fill the empty strings with the target vault entry values.

{
   "some_source-configuration.parameters.ConsumerSecret.value": "",
   "some_source-configuration.parameters.ConsumerKey.value": "",
   "some_source-configuration.parameters.AccessTokenSecret.value": "",
   "some_source-configuration.parameters.AccessToken.value": ""
}

Finally, you should see a confirmation that your snapshot has been applied to the target organization.