--- title: Relevance Generative Answering (RGA) model card slug: o5g90043 canonical_url: https://docs.coveo.com/en/o5g90043/ collection: leverage-machine-learning source_format: adoc --- # Relevance Generative Answering (RGA) model card ## What's a model card? A [model](https://docs.coveo.com/en/1012/) card is a document that provides a summary of key information about a [Coveo Machine Learning (Coveo ML)](https://docs.coveo.com/en/188/) [model](https://docs.coveo.com/en/1012/). It details the model's purpose, intended use, performance, and limitations. ## Model details The Coveo [Relevance Generative Answering (RGA)](https://docs.coveo.com/en/nbtb6010/) [model](https://docs.coveo.com/en/1012/) provides Coveo customers' end users with generative answers to queries performed in real time. [RGA](https://docs.coveo.com/en/nbtb6010/) is primarily designed to enhance end-user search experience in Coveo-powered search solutions. The RGA model uses text data from a customer's index to generate answers that are relevant, personalized, and secure. * **Development team**: Coveo ML team * **Initial release date**: December 14, 2023. Major changes can occur and are communicated via Coveo release notes. * **Activation**: The RGA model is created and assigned to query pipelines using the Coveo Administration Console. ## Intended use * **Intended purpose**: To enhance an end user's search experience by providing a generated answer to a search query using natural language. * **Intended output**: The answer is generated using only the customer's content that's indexed to the Coveo Platform. The indexing process is managed by the customer's administrator. * **Intended users**: End users of Coveo customers. ## Factors The [RGA](https://docs.coveo.com/en/nbtb6010/) [model](https://docs.coveo.com/en/1012/) generates an answer through a combination of factors. The first set of factors involves content retrieval. This includes retrieving the right documents using a hybrid approach that integrates lexical and semantic search, business rules, and behavioral analytics, as well as retrieving the appropriate text chunks using semantic search. The second set of factors pertains to generating the answer based on prompt instructions and the retrieved text chunks. ## Training data The data used within the [RGA](https://docs.coveo.com/en/nbtb6010/) [model](https://docs.coveo.com/en/1012/) is tailored to each Coveo customer's organization. Specifically, the [RGA](https://docs.coveo.com/en/nbtb6010/) [model](https://docs.coveo.com/en/1012/) uses text data from selected content within the customer's index. [RGA](https://docs.coveo.com/en/nbtb6010/) breaks down each document into text chunks and creates embeddings from these chunks. Customers retain ownership of their data. When using the [RGA](https://docs.coveo.com/en/nbtb6010/) [model](https://docs.coveo.com/en/1012/), customers grant Coveo a non-exclusive license to use, display, and create derivative works based on such content during the subscription term. These datasets may also include personal information or aggregate consumer information, if provided by customers, which Coveo processes on their behalf. The selected content and its quality directly impact the quality of the answers generated by [RGA](https://docs.coveo.com/en/nbtb6010/). Higher quality data results in better embeddings and more relevant answers. For optimal results, it's crucial to adhere to the requirements specified in Coveo's documentation and best practices. [RGA](https://docs.coveo.com/en/nbtb6010/)'s generated answers and the utilized text chunks can be [inspected in Snowflake](https://docs.coveo.com/en/o42b0517/). ## Evaluation data To support the offline development and evaluation of the [RGA](https://docs.coveo.com/en/nbtb6010/) [model](https://docs.coveo.com/en/1012/), Coveo uses a combination of publicly available datasets and a subset of customer datasets. Public datasets include question-answering and adversarial datasets used to evaluate the [RGA](https://docs.coveo.com/en/nbtb6010/) [model](https://docs.coveo.com/en/1012/), including retrieval effectiveness, grounding/citation accuracy, and abstention behavior. We may subsample or construct negative examples to evaluate abstention and citation behavior. These datasets are typically not modified unless required to improve robustness testing. For more information on offline evaluation, see to [Coveo machine learning model development and evaluation](https://docs.coveo.com/en/o6jc0313/). ## Data use Coveo collects and processes the datasets during the term of an active subscription between Coveo and its customers. ## Performance The quality and performance of the [RGA](https://docs.coveo.com/en/nbtb6010/) [model](https://docs.coveo.com/en/1012/) is measured by examining how well the [model](https://docs.coveo.com/en/1012/) operates based on retrieval and generation metrics performed offline and online. Coveo also uses internal performance metrics to measure the overall reliability of the [RGA](https://docs.coveo.com/en/nbtb6010/) [model](https://docs.coveo.com/en/1012/), such as response time and average build time. * **Retrieval metrics**: Coveo uses offline retrieval metrics based on publicly available datasets (for example, [MTEB dataset](https://huggingface.co/spaces/mteb/leaderboard)) to assess the effectiveness and relevance of the [RGA](https://docs.coveo.com/en/nbtb6010/) [model](https://docs.coveo.com/en/1012/) under controlled conditions without the variability of real-time user interaction. * **Generation metrics**: Coveo uses generation metrics to evaluate the performance, quality, and accuracy of the RGA model's outputs: ** Coveo uses offline generation metrics based on internal or public datasets (for example, [ASQA public dataset](https://github.com/google-research/language/tree/master/language/asqa)) to assess the answering capabilities of the [RGA](https://docs.coveo.com/en/nbtb6010/) [model](https://docs.coveo.com/en/1012/). In practice Coveo uses the following: *** The weighted mean[.footnote]^[[1](#weighted-mean)]^ metric to assess when the [RGA](https://docs.coveo.com/en/nbtb6010/) [model](https://docs.coveo.com/en/1012/): **** refrains from answering when the chunks don't contain the answer, using soft-negative[.footnote]^[[2](#soft-negative)]^ and hard-negative[.footnote]^[[3](#hard-negative)]^ samples. **** answers when expected, or the [RGA](https://docs.coveo.com/en/nbtb6010/) [model](https://docs.coveo.com/en/1012/)’s ability to answer when the chunk contains the answer. *** The precision[.footnote]^[[4](#precision-metric)]^, recall[.footnote]^[[5](#recall-metric)]^, and F1 score[.footnote]^[[6](#f1-score)]^ metrics to evaluate the RGA model's ability to accurately cite the chunks that are used. *** The weighted mean[.footnote]^[[1](#weighted-mean)]^ metric to evaluate the repeatability of answers over time for the same end user's question. ** Coveo uses aggregated information to assess the average online answer rate of queries performed on the Coveo Platform, based on the weighted mean metric. -- 1. A measure that calculates the average value of a set of data points, where each data point contributes to the final average in proportion to its assigned weight. This is particularly important when certain data points are considered more important or relevant than others, which allows for a more accurate representation of the overall data. 2. Soft negatives are examples that are somewhat similar to the positive answers, but aren't exactly correct. 3. Hard negatives are examples that are very similar to the positive answers, and more challenging to distinguish from positive answers. 4. Precision measures the correctness of the generated outputs. 5. Recall measures the completeness of the generated outputs, or the ability of the model to not miss any important element of the target output. 6. The harmonic mean of precision and recall, which provides a single metric to evaluate the balance between precision and recall. -- ## Limitations Some factors might degrade the [RGA](https://docs.coveo.com/en/nbtb6010/) [model](https://docs.coveo.com/en/1012/)’s performance: * **Quality of training data**: The effectiveness of the [RGA](https://docs.coveo.com/en/nbtb6010/) [model](https://docs.coveo.com/en/1012/) is largely dependent on the quality of the training data. If the customer's selected documents that form the dataset are biased, non-representative, irrelevant, incomplete, or inadequate, the model's performance will be affected. For instance, the RGA model's performance might be sub-optimal if trained on non-factual documents. * **Risk of AI hallucinations**: The output of the [RGA](https://docs.coveo.com/en/nbtb6010/) [model](https://docs.coveo.com/en/1012/) is based on a customer's internal content. Therefore, if a customer's dataset contains meticulously curated informational content that's accurate and up-to-date, the risk of AI hallucination is drastically reduced. Conversely, if a customer's internal content contains false or inaccurate information, the risk of AI hallucination increases. * **Language limitations**: The [RGA](https://docs.coveo.com/en/nbtb6010/) [model](https://docs.coveo.com/en/1012/) provides generated answers in multiple languages, including English. > **Tip** > > By default, only English content is supported. > However, Coveo offers beta support for languages other than English. > Learn more about [multilingual content retrieval and answer generation](https://docs.coveo.com/en/p5ne0024/). * **Indirect feedback loop**: The [RGA](https://docs.coveo.com/en/nbtb6010/) [model](https://docs.coveo.com/en/1012/) doesn't directly take end user feedback (thumbs up/thumbs down) into account when generating the answer. However, all behavioral signals from an end user will be taken into account by other ML [models](https://docs.coveo.com/en/1012/) ([ART](https://docs.coveo.com/en/1013/), [DNE](https://docs.coveo.com/en/2907/)) that influence the ranking of documents that [RGA](https://docs.coveo.com/en/nbtb6010/) uses to extract text chunks and generate answers. ## Best practices Best practices for the [RGA](https://docs.coveo.com/en/nbtb6010/) [model](https://docs.coveo.com/en/1012/) are documented in [Relevance Generative Answering (RGA) content requirements and best practices](https://docs.coveo.com/en/nb6a0008/). All usage must comply with Coveo's [Acceptable Use Policy](https://www.coveo.com/en/company/legal/terms-and-agreements/acceptable-use-policy).