Evaluate and manage Relevance Generative Answering (RGA) answers

This is for:

Developer System Administrator
Important

The Coveo Knowledge Hub and Answer Manager are currently available as a beta to early-access customers only. Contact your Customer Success Manager for early access to this feature.

Evaluating the answers that are generated by Relevance Generative Answering (RGA) is an important part of your RGA implementation, just as with any generative answering system.

At a basic level, RGA relies on data and algorithms to generate answers. Evaluating RGA-generated answers for specific user queries helps you assess the performance and effectiveness of your RGA implementation in relation to your dataset. Evaluating answers is crucial when testing your RGA implementation in a pre-production environment. Post-production evaluations are equally valuable, as they help you fine-tune and improve your implementation as your dataset evolves.

You can use the Answer Manager in the Coveo Knowledge Hub to evaluate the RGA-generated answers, and create rules to manage and improve the generated answers.

To evaluate and manage your RGA-generated answers

  1. Create an answer configuration in the Answer Manager for your RGA-enabled search interface.

  2. Use the answer configuration to evaluate and manage your RGA-generated answers.

Answer configuration setup

To evaluate and manage the RGA-generated answers for an RGA-enabled search interface, create an answer configuration in the Answer Manager and then associate the answer configuration with the search interface.

An answer configuration is the link between your search interface and the Answer Manager. The answer configuration is where you’ll see the feedback that was collected for the generated answers, and where you’ll create rules to manage the answers.

To set up an answer configuration

  1. Create an answer configuration.

  2. Associate the answer configuration with your RGA-enabled search interface.

  3. Add an answerAPI condition to the query pipeline.

Create an answer configuration

Create an answer configuration to evaluate and manage the answers that are generated from an RGA-enabled search interface.

Note

We recommend using a separate answer configuration for each search interface, even if you use the same RGA model for multiple search interfaces. This makes evaluating and managing the RGA-generated answers easier and more focused.

  1. In the Coveo Administration Console header, use the application picker to select Knowledge Hub.

    Application picker in the Administration Console header | Coveo

  2. In the Knowledge Hub navigation menu, select Answer Manager.

  3. Click Add configuration.

  4. Enter a descriptive Name for your configuration.

  5. Click Add.

  6. If your RGA-enabled search interface is a custom search interface that was created using the Coveo Atomic library, Coveo Headless library, or Quantic library, copy the Configuration ID of the newly created answer configuration. You’ll need the configuration ID to associate the answer configuration with your search interface. The configuration ID isn’t required if you created a Coveo hosted search interface using the Search Page builder, In-Product Experience (IPX) builder, or Insight Panel builder.

    Answer configuration list in the Administration Console | Coveo

  7. Associate your answer configuration with your RGA-enabled search interface.

Edit an answer configuration

You can edit an existing answer configuration to change its name.

  1. In the Coveo Administration Console header, use the application picker to select Knowledge Hub.

    Application picker in the Administration Console header | Coveo

  2. In the Knowledge Hub navigation menu, select Answer Manager.

  3. Click the answer configuration you want to edit from the list.

  4. In the Action bar, click Edit.

  5. Modify the Name as needed.

  6. Click Save to apply your changes.

Delete an answer configuration

You can delete an answer configuration when it’s no longer needed.

Important

  • All the corresponding feedback and rules will be lost.

  • If the configuration is associated with a custom search interface, remove the association or replace it with another configuration to prevent errors in your search interface.

  • If an RGA-enabled search interface is no longer associated with an answer configuration, remove the answerAPI condition for your RGA model in your query pipeline. Otherwise, your search interface won’t use the RGA model and answers won’t be generated.

  1. In the Coveo Administration Console header, use the application picker to select Knowledge Hub.

    Application picker in the Administration Console header | Coveo

  2. In the Knowledge Hub navigation menu, select Answer Manager.

  3. Click the answer configuration you want to delete from the list.

  4. In the Action bar, click Delete.

  5. In the confirmation prompt that appears, click Delete again to confirm.

Associate an answer configuration with a search interface

Associating an answer configuration with an RGA-enabled search interface lets you review the feedback for the answers that are generated in that search interface. The feedback is collected by Coveo and made available in the associated answer configuration in the Answer Manager.

An answer configuration also lets you create rules to manage the search interface’s generated answers. For example, you can create a rule that prevents an answer from being generated based on the presence of specific words in a query. Rules that you create impact the generated answers only in the associated search interface.

How to associate an answer configuration with a search interface depends on how you created your RGA-enabled search interface:

  • If you created a Coveo hosted search interface using one of the search interface builders (Search Page builder, In-Product Experience (IPX) builder, or Hosted Insight Panel builder), see Coveo hosted search interface.

  • If you created a custom search page using the Coveo Atomic library, Coveo Headless library, or Quantic library, see Custom search interface.

Coveo hosted search interface

If you created a Coveo hosted search interface using one of the Coveo builders, the association is made directly in the builder settings.

  1. Select the answer configuration that you want to associate with the search interface in the appropriate builder settings:

  2. Add an answerAPI condition to the search interface’s query pipeline.

Custom search interface

If you created a custom search page using the Coveo Atomic library, Coveo Headless library, or Quantic library, associate the answer configuration with the search interface using the configuration ID.

  1. Enter the Configuration ID of the answer configuration in the search interface’s configuration:

    • For Atomic search pages, enter the configuration ID in the answer-configuration-id property of the atomic-generated-answer component.

      Note

      Association with an answer configuration requires Atomic v2.79.1+.

    • For Quantic search pages, enter the configuration ID in the answerConfigurationId property of the QuanticGeneratedAnswer component.

      Note

      Association with an answer configuration requires Quantic v3.10.0+.

    • For Headless search pages, enter the configuration ID in the answerConfigurationId? property of the GeneratedAnswer controller.

      Note

      Association with an answer configuration requires Headless v2.80.6+.

  2. Add an answerAPI condition to the search interface’s query pipeline.

Add an answerAPI condition to the query pipeline

When an RGA-enabled search interface is associated with an answer configuration, add an answerAPI condition for the RGA model in the query pipeline. This condition ensures that the generative queries per month (GQPM) aren’t duplicated for queries that use your RGA model.

For information on how to add the condition, see Prevent duplicate GQPM when using the Answer Manager.

Important

If an RGA-enabled search interface is no longer associated with an answer configuration, remove the answerAPI condition for your RGA model in your query pipeline. Otherwise, your search interface won’t use the RGA model and answers won’t be generated.

Answer evaluation and management

Once an answer configuration is created and associated with your RGA-enabled search interface, you can see the feedback that’s collected for the generated answers. Optionally, you can use the Chunk Inspector in the Knowledge Hub to inspect the exact segments of text (chunks) that RGA used to generate a specific answer. Based on the feedback and chunk information, you can then choose to create rules and modify other RGA-specific settings to improve the answers that are generated by RGA.

Before diving into how to evaluate and manage your RGA answers, let’s review the suggested evaluation and answer management flow for RGA, which is one of continuous evaluation and refinement.

Relevance Generative Answering evaluation flow diagram | Coveo

1

When an answer is generated by RGA for a user query in a search interface, the user can click the thumbs-up or thumbs-down icon in the RGA component to provide feedback on the generated answer. Coveo collects this feedback and makes it available in the associated answer configuration for answer-evaluation purposes.

2

In the Answer Manager, use the answer configuration that’s associated with the search interface to review the answer feedback. You’ll be able to see the query, the generated answer, and any additional feedback information that was provided.

3

In the Answer Manager, use the Chunk Inspector to see the segments of text (chunks) that RGA used to generate the answer. Inspecting the chunks used by RGA provides insight as to why certain segments of text were used over others. This is especially useful when evaluating answers with negative feedback or answers that have been flagged as needing improvement. Combining answer feedback with chunk inspection gives you the information that you need to better manage your RGA answers.

4

Based on the information gathered from reviewing the answer feedback and inspecting the chunks used by RGA, you can manage your RGA answers by creating answer rules and modifying the RGA model configurations.

Review RGA answer evaluations

Once feedback is collected for your search interface’s RGA-generated answers using the RGA answer feedback modal, you can view the feedback in the search interface’s associated answer configuration in the Answer Manager.

Note

An answer configuration shows only the feedback that’s collected after the association with the search interface is established.

  1. In the Coveo Administration Console header, use the application picker to select Knowledge Hub.

    App picker | Coveo

  2. In the Knowledge Hub navigation menu, select Answer Manager.

  3. Click the answer configuration that’s associated with the search interface for which you want to review the answer feedback, and then click View in the Action bar.

  4. Select the Evaluations tab. A list of the feedback collected for answers generated in the corresponding search interface appears.

    Note

    An answer’s evaluation is listed as either Helpful or Unhelpful, based on whether the thumbs-up or thumbs-down icon was clicked in the RGA component.
    Tip

    You can filter the evaluation list by Helpful or Unhelpful, by keywords in the question (query), and a date range.

  5. To view the evaluation details for a specific answer, click the answer evaluation from the list, and then click View in the Action bar.

    Answer Manager evaluation list | Coveo

  6. The Evaluation details show the following:

    Answer Manager evaluation details | Coveo

    • The Question and answer set.

    • The Evaluation date and time.

    • Any Additional notes that were provided in the feedback.

    • If the answer was Evaluated as Helpful or Unhelpful.

    • The Answer ID, which is needed if you want to inspect the chunks used by RGA when generating the answer.

  7. After having reviewed the feedback for the generated answers, you can choose to do the following:

Inspect the RGA chunks

For a given RGA-generated answer, you can review the segments of text (chunks) that were identified by the RGA model as being the most relevant to the query, and the chunks that were used to generate the answer. This is especially useful when evaluating answers with negative feedback or answers that have been flagged as needing improvement. Inspecting the chunks used by RGA gives you the information that you need to better manage your RGA answers.

To inspect the chunks used by RGA, use the Chunk Inspector in the Coveo Knowledge Hub.

Manage your RGA answer

After reviewing the feedback of an RGA-generated answer, and inspecting the chunks used by RGA to generate the answer, you may want to make adjustments to improve the question-answer experience of your RGA implementation.

You can do one or more of the following:

  • Create an answer rule to manage the RGA-generated answers for a specific search interface. For example, you can create a rule that prevents an answer from being generated based on the presence of specific words in a query.

  • Modify the maximum number of items that the RGA model considers when retrieving the relevant segments of text (chunks) from which to generate answers. For example, you can decrease this value if you find that answers are being generated using text from low relevance items or documents.

  • Modify the relevancy threshold that’s used to determine whether a segment of text (chunk) is relevant enough to be considered by the RGA model for answer generation. For example, you can increase the threshold value if you find that answers are being generated using chunks of lower relevance.

Create answer rules

You can create rules in an answer configuration that impact the answers that are generated in the associated search interface. For example, you can create a rule that prevents an answer from being generated based on the presence of specific words in a query.

Note

The rules in an answer configuration apply only to the configuration’s associated RGA-enabled search interface.

  1. In the Coveo Administration Console header, use the application picker to select Knowledge Hub.

    App picker | Coveo

  2. In the Knowledge Hub navigation menu, select Answer Manager.

  3. Click the answer configuration that’s associated with the search interface for which you want to create an answer rule, and then click View in the Action bar.

  4. Select the Rules tab.

  5. Click Add rule.

  6. Enter a descriptive Name for the rule.

  7. In the Rule type list, select the desired type of rule to create.

    Note

    Currently, you can only create a blocking rule.

  8. Enter the Word or phrase to block. A blocking rule prevents RGA from generating an answer when the query contains the specified word or phrase.

  9. Click Add.

Required privileges

The following table indicates the privileges required to access the Coveo Knowledge Hub, and view or edit elements of the Answer Manager and associated panels (see Manage privileges and Privilege reference).

Important

Both the Edit and View access level on the Answer manager domain allows the member to view all generated answers for which feedback exists in an answer configuration in the Answer Manager. This may include content to which the member may not have access to otherwise through the repository’s permission system.

Exercise caution when granting this privilege to members.
Action Service - Domain Required access level

Access the Coveo Knowledge Hub

Knowledge - Knowledge hub

Enable

Organization - Organization

View

  • View the generated-answer feedback for an answer configuration in the Answer Manager

  • View the rules for an answer configuration in the Answer Manager

Knowledge - Answer manager

View

  • Add, edit, or delete answer configurations and rules in the Answer Manager

  • View the generated-answer feedback for an answer configuration in the Answer Manager

Knowledge - Answer manager

Edit