Relevance Generative Answering (RGA) reports and UA events

This is for:

Developer System Administrator

To track the usage of Relevance Generative Answering (RGA), Coveo Usage Analytics (Coveo UA) logs events for actions performed on the RGA component in your search interface.

This article provides instructions on how to create a report to view the performance of your RGA implementation, and provides a reference for the RGA usage analytics events that are tracked by Coveo UA.

Add custom dimensions

To be able to create a report to view data on the RGA custom UA events, you must add the following custom dimensions to your Coveo organization:

  • c_answergenerated

  • To view event data collected from the RGA feedback modal:

    • c_helpful

    • c_readable

    • c_documented

    • c_correctTopic

    • c_hallucinationFree

    • c_details

    • c_documentUrl

To add a custom dimension for RGA

Add a dimension using the following criteria:

  • For each of the custom dimensions, API name is the name of the dimension. For example, when creating the c_answergenerated dimension, API name is c_answergenerated.

  • For each of the custom dimensions, Related events must be set to Custom event.

Example

To create the c_answergenerated dimension, the settings are as follows:

Relevance Generative Answering custom dimension | Coveo

Create an RGA report

You can quickly create a report to analyze the performance of your Relevance Generative Answering (RGA) implementation by using the preconfigured Generative Answering Performance usage analytics (UA) dashboard template.

The template creates a report with three tabs. The Search Metrics tab contains report cards that show metrics such as the number of generated answers, click-through rate, and top queries with and without generated answers. The User Feedback tab displays data collected through the RGA feedback modal. The Self Service tab lets you see the impact of RGA on case submission, and by extension on self-service success, if you’re logging a custom event when a user submits a case during the same visit when accessing the RGA-enabled search interface.

Relevance Generative Answering report template | Coveo

  1. If you haven’t done so already, add the custom dimensions to your Coveo organization.

  2. Add a usage analytics dashboard using the Generative Answering Performance template.

RGA usage analytics events

This section lists the custom usage analytics events for RGA.

Value Event

  • eventType: generatedAnswer

  • eventValue: generatedAnswerStreamEnd

The answer was successfully streamed in the search interface. This event won’t be logged if the user initiates another search while the answer is being generated.

In customData:
answerGenerated: true

An answer was successfully generated for the query.

In customData:
answerGenerated: false

An answer wasn’t generated for the query because there wasn’t enough relevant information to generate an answer. In this case, a message saying that no answer was created appears instead of the generated answer.

Note

If RGA wasn’t able to generate an answer, such as due to an empty query, empty search results, or an error, no UA event is logged and no message appears in the search interface.

  • eventType: generatedAnswer

  • eventValue: openGeneratedAnswerSource

  • In customData:

    • generativeQuestionAnsweringID: The ID of the generated answer.

    • citationId: The ID of the citation that was clicked.

    • permanentId: The permanent ID of the item corresponding to the citation that was clicked.

A citation link was clicked.

  • eventType: generatedAnswer

  • eventValue: generatedAnswerSourceHover

  • In customData:

    • generativeQuestionAnsweringID: The ID of the generated answer.

    • citationId: The ID of the citation that was clicked.

    • permanentId: The permanent ID of the item corresponding to the citation that was clicked.

    • citationHoverTimeMs: The duration of the hover in milliseconds (ms).

A user hovered over a citation link.

  • eventType: generatedAnswer

  • eventValue: likeGeneratedAnswer

  • In customData:

    • generativeQuestionAnsweringID: The ID of the generated answer.

A user clicked the thumbs-up icon to provide positive feedback for the generated answer.

  • eventType: generatedAnswer

  • eventValue: dislikeGeneratedAnswer

  • In customData:

    • generativeQuestionAnsweringID: The ID of the generated answer.

A user clicked the thumbs-down icon to provide negative feedback for the generated answer.

  • eventType: generatedAnswer

  • eventValue: generatedAnswerFeedbackSubmitV2

  • In customData:

    • generativeQuestionAnsweringID: The ID of the generated answer.

    • helpful: The feedback icon that was clicked. Shows "true" for thumbs-up, and "false" for thumbs-down.

    • readable: The feedback option selected for if the generated answer is in a readable format ("yes", "unknown", or "no").

    • documented: The feedback option selected for if the answer is present in the existing dataset ("yes", "unknown", or "no").

    • correctTopic: The feedback option selected for if the generated answer relates to the correct topic ("yes", "unknown", or "no").

    • hallucinationFree: The feedback option selected for if the generated answer is free of non-factual (hallucinated) content ("yes", "unknown", or "no").

    • details: Shows any additional notes entered by the user.

    • documentUrl: Shows the URL entered by the user that points to the dataset item that contains the answer.

If the thumbs-up or thumbs-down icon is clicked, the user submits feedback details using the RGA feedback modal.

  • eventType: generatedAnswer

  • eventValue: generatedAnswerCopyToClipboard

  • In customData:

    • generativeQuestionAnsweringID: The ID of the generated answer.

A user clicked the copy button to copy the generated answer to their clipboard.

  • eventType: generatedAnswer

  • eventValue: generatedAnswerHideAnswers

  • In customData:

    • generativeQuestionAnsweringID: The ID of the generated answer.

A user closed (hid) the RGA component.

  • eventType: generatedAnswer

  • eventValue: generatedAnswerShowAnswers

  • In customData:

    • generativeQuestionAnsweringID: The ID of the generated answer.

A user opened (showed) the RGA component.

  • eventType: generatedAnswer

  • eventValue: generatedAnswerExpand

  • In customData:

    • generativeQuestionAnsweringID: The ID of the generated answer.

A user expanded the generated answer using Show more.

Note

This feature is enabled using the collapsible property in the Coveo Atomic Library or Coveo Quantic Library.

  • eventType: generatedAnswer

  • eventValue: generatedAnswerCollapse

  • In customData:

    • generativeQuestionAnsweringID: The ID of the generated answer.

A user collapsed the generated answer using Show less.

Note

This feature is enabled using the collapsible property in the Coveo Atomic Library or Coveo Quantic Library.

Inspect your search page

Use your browser developer tools to retrieve what’s being tracked:

  1. Open a Coveo-powered search interface with RGA enabled.

  2. Open your web browser’s developer tools.

    Note

    The examples in this article use the Google Chrome developer tools. For browser-specific information, see:

  3. Select the Network tab.

  4. On the search page, perform a query that will generate an answer.

  5. Perform an action in the RGA component.

  6. Back in the developer tools window, under the Name column, click the latest custom call to Coveo UA.

  7. Select the Payload tab. You should now see the tracked usage analytics event.