Relevance Generative Answering (RGA) reports and UA events

This is for:

Developer System Administrator

To track the usage of Relevance Generative Answering (RGA), Coveo Usage Analytics (Coveo UA) logs events for actions performed on the RGA component in your search interface.

This article provides instructions on how to create a report to view the performance of your RGA implementation, and provides a reference for the RGA usage analytics events that are tracked by Coveo UA.

Add a custom dimension

To be able to create usage analytics (UA) reports on the RGA custom UA events, you must add a custom dimension to your Coveo organization.

Follow the instructions to add a dimension on custom metadata using the following criteria:

  • For API name, enter c_answergenerated.

  • For Related events, select Custom event.

Create an RGA report

You can quickly create a report to analyze the performance of your Relevance Generative Answering (RGA) implementation by using the preconfigured Generative Answering Performance usage analytics (UA) dashboard template.

The template creates a report with two tabs. The Search Metrics tab contains report cards that show metrics such as the number of generated answers, click-through rate, feedback, and top queries with and without generated answers. The Self Service tab lets you see the impact of RGA on case submission, and by extension on self-service success, if you’re logging a custom event when a user submits a case during the same visit when accessing the RGA-enabled search interface.

Relevance Generative Answering report template | Coveo

  1. If you haven’t done so already, add the c_answergenerated custom dimension to your Coveo organization.

  2. Add a usage analytics dashboard using the Generative Answering Performance template.

RGA usage analytics events

This section lists the custom usage analytics events for RGA.

Value Event

  • eventType: generatedAnswer

  • eventValue: generatedAnswerStreamEnd

The answer was successfully streamed in the search interface. This event won’t be logged if the user initiates another search while the answer is being generated.

In customData:
answerGenerated: true

An answer was successfully generated for the query.

In customData:
answerGenerated: false

An answer wasn’t generated for the query because there wasn’t enough relevant information to generate an answer. In this case, a message saying that no answer was created appears instead of the generated answer.

Note

If RGA wasn’t able to generate an answer, such as due to an empty query, empty search results, or an error, no UA event is logged and no message appears in the search interface.

  • eventType: generatedAnswer

  • eventValue: openGeneratedAnswerSource

  • In customData:

    • generativeQuestionAnsweringID: The ID of the generated answer.

    • citationId: The ID of the citation that was clicked.

    • permanentId: The permanent ID of the item corresponding to the citation that was clicked.

A citation link was clicked.

  • eventType: generatedAnswer

  • eventValue: generatedAnswerSourceHover

  • In customData:

    • generativeQuestionAnsweringID: The ID of the generated answer.

    • citationId: The ID of the citation that was clicked.

    • permanentId: The permanent ID of the item corresponding to the citation that was clicked.

    • citationHoverTimeMs: The duration of the hover in milliseconds (ms).

A user hovered over a citation link.

  • eventType: generatedAnswer

  • eventValue: likeGeneratedAnswer

  • In customData:

    • generativeQuestionAnsweringID: The ID of the generated answer.

A user clicked the thumbs-up icon to provide positive feedback for the generated answer.

  • eventType: generatedAnswer

  • eventValue: dislikeGeneratedAnswer

  • In customData:

    • generativeQuestionAnsweringID: The ID of the generated answer.

A user clicked the thumbs-down icon to provide negative feedback for the generated answer.

  • eventType: generatedAnswer

  • eventValue: generatedAnswerFeedbackSubmit

  • In customData:

    • reason: The option selected in the feedback dialog ("irrelevant", "notAccurate", "outOfDate", "harmful", or "other").

    • details: When the reason is "other", shows the free text entered by the user.

If the thumbs-down icon is clicked, the user specifies and submits a reason for the negative feedback using the feedback dialog.

  • eventType: generatedAnswer

  • eventValue: generatedAnswerCopyToClipboard

  • In customData:

    • generativeQuestionAnsweringID: The ID of the generated answer.

A user clicked the copy button to copy the generated answer to their clipboard.

  • eventType: generatedAnswer

  • eventValue: generatedAnswerHideAnswers

  • In customData:

    • generativeQuestionAnsweringID: The ID of the generated answer.

A user closed (hid) the RGA component.

  • eventType: generatedAnswer

  • eventValue: generatedAnswerShowAnswers

  • In customData:

    • generativeQuestionAnsweringID: The ID of the generated answer.

A user opened (showed) the RGA component.

Inspect your search page

Use your browser developer tools to retrieve what’s being tracked:

  1. Open a Coveo-powered search interface with RGA enabled.

  2. Open your web browser’s developer tools.

    Note

    The examples in this article use the Google Chrome developer tools. For browser-specific information, see:

  3. Select the Network tab.

  4. On the search page, perform a query that will generate an answer.

  5. Perform an action in the RGA component.

  6. Back in the developer tools window, under the Name column, click the latest custom call to Coveo UA.

  7. Select the Payload tab. You should now see the tracked usage analytics event.