Altering Search Results Before They Are Displayed in a Search Interface

There are many different ways to change the appearance of your search results. For example:

  • You can choose which fields and data are indexed for each Sitecore item, thus effectively changing what data is available to be displayed in your search interface. This mostly involves custom code running in a server-side .NET context.
  • You can customize the layout of your search page so that you only display the fields and data that you are interested in (even though many more fields are available to be displayed). This mostly involves adding custom, client-side JavaScript code in your layout.

Now, an alternative approach is to use the coveoProcessParsedRestResponse and coveoProcessRestJsonResponse pipelines. Here is how they work:

  1. A user performs a query in a search interface.
  2. The query is handled by the Coveo REST endpoint running in the Sitecore context.
  3. The REST endpoint calls CES and receives a JSON response with the search results.
  4. The coveoProcessRestJsonResponse pipeline kicks in and deserializes the JSON response into .NET objects.
  5. The coveoProcessParsedRestResponse pipeline is called, thus altering search results.
  6. The search results are serialized in JSON again.
  7. The new JSON response is sent to the search interface.

Understanding the coveoProcessParsedRestResponse Pipeline

This pipeline allows you to modify search results using .NET objects; you do not have to worry about the serialization process. The pipeline is defined in the Coveo.SearchProvider.Rest.config file.

It calls its processors by passing them an instance of type Coveo.SearchProvider.Rest.Pipelines.CoveoProcessParsedRestResponseArgs. You can access the response information with the ResponseContent property of the CoveoProcessParsedRestResponseArgs instance. The property returns an object of type Coveo.SearchProvider.Rest.Serialization.SearchResponse, which is a dictionary with shorthand properties to get and set the most commonly used values.

Creating a Custom Processor for the coveoProcessParsedRestResponse Pipeline

  1. Assuming that you already have set up a C# project, you need to add these assembly references:
    1. Coveo.Framework.dll 
    2. Coveo.SearchProvider.Rest.dll 
    3. Sitecore.Kernel.dll 
  2. Create a processor based on the sample code below. This one adds a new field on every result. It simply returns the number of fields that are indexed for the item.

    ComputeNumberOfIndexedFields.cs

    using Coveo.Framework.Processor;
    using Coveo.SearchProvider.Rest.Pipelines;
    using Coveo.SearchProvider.Rest.Serialization;
    namespace Tutorials.Lib.Processors.coveoProcessParsedRestResponse
    {
        /// <summary>
        /// Custom processor that computes the number of indexed fields for every search result and
        /// add it to the response.
        /// </summary>
        public class ComputeNumberOfIndexedFields : IProcessor<CoveoProcessParsedRestResponseArgs>
        {
            /// <summary>
            /// The "Process" method is called by the pipeline. The "p_Args" instance
            /// is transferred from one processor to another until the pipeline ends.
            /// </summary>
            /// <param name="p_Args">The pipeline arguments.</param>
            public void Process(CoveoProcessParsedRestResponseArgs p_Args)
            {
                SearchResponse response = p_Args.ResponseContent;
                foreach (SearchResult result in response.Results) {
                    // The "Raw" property returns all the fields that are indexed on the item.
                    int numberOfIndexedFields = result.Raw.Count;
                    // Simply define a new field name and set its value.
                    result["numberOfIndexedFields"] = numberOfIndexedFields;
                }
            }
        }
    }
    
  3. Build the assembly and copy the DLL file to the website bin folder. For this tutorial, the assembly is Tutorials.Lib.dll.

  4. Finally, modify the configuration file to tell Sitecore when the new processor must be called:

    1. Open the Coveo.SearchProvider.Rest.Custom.config file.

      In a more formal project, you would want to use a separate include file and register the processor there.

    2. Under sitecore, add the following element, and insert your processor definition.

      <pipelines>
        <coveoProcessParsedRestResponse>
          <processor type="Tutorials.Lib.Processors.coveoProcessParsedRestResponse.ComputeNumberOfIndexedFields, Tutorials.Lib" />
        </coveoProcessParsedRestResponse>
      </pipelines>
      
  5. Validate that you can now access the numberOfIndexedFields field from a search interface.

For more details on handling field name translation in this pipeline, see Removing Fields From Search Results.

Understanding the coveoProcessRestJsonResponse Pipeline

This pipeline is used to process the whole JSON response at once; modifying it manually would be error-prone. There are however situations where you may want to use a specific library to handle JSON serialization. This is what this pipeline is used for. Coveo for Sitecore comes shipped with the ParseJsonRestSearchResponseProcessor processor, already integrated to the pipeline out of the box. This processor internally uses the .NET System.Web.Script.Serialization.JavaScriptSerializer class to process JSON data. However, you can inherit from this processor and define the JSON library that you want to use. You need to understand that when executing the Process method, the default processor performs these operations:

  1. It deserializes the JSON response received from CES.
  2. It calls the coveoProcessParsedRestResponse pipeline.
  3. It serializes the .NET results to JSON and returns the new response.

Replacing the JSON Library

Here is a sample showing you how to replace the JSON library that performs serialization/deserialization.

  1. Assuming that you already have set up a C# project, you need to add these assembly references:
    1. Coveo.Framework.dll
    2. Coveo.SearchProvider.Rest.dll
    3. Newtonsoft.Json.dll (this one is only mandatory for this specific example)
    4. Sitecore.Kernel.dll
  2. Since you want to replace the JSON library used for serializing data, you do not need to redefine the whole behavior of the ParseJsonRestSearchResponseProcessor class. Instead, you can create a serialization class that uses the Newtonsoft.Json library.

    There are unfortunately some differences between JavaScriptSerializer and Newtonsoft.Json in the way they deserialize JSON. Newtonsoft.Json uses some wrappers to represent JSON values, and they can break some of the shorthand properties defined on the Coveo.SearchProvider.Rest.Serialization.SearchResponse type. To work around this issue, you need to implement some more deserialization logic as shown in the example below.

    NewtonsoftJsonSerializer.cs

    using System;
    using System.Collections.Generic;
    using System.IO;
    using Coveo.Framework.Utils;
    using Coveo.SearchProvider.Rest.Serialization;
    using Newtonsoft.Json;
    using Newtonsoft.Json.Linq;
    using JsonSerializer = Newtonsoft.Json.JsonSerializer;
    namespace Tutorials.Lib.Processors.coveoProcessRestJsonResponse
    {
        /// <summary>
        /// Custom serializer that rely on Newtonsoft.Json instead of JavaScriptSerializer.
        /// </summary>
        public class NewtonsoftJsonSerializer : ISerializer
        {
            /// <summary>
            /// Deserializes a JSON string into .NET objects of the given type.
            /// </summary>
            /// <typeparam name="T">The type in which JSON is deserialized.</typeparam>
            /// <param name="p_Stream">The stream that contains the JSON.</param>
            /// <returns>The deserialized object structure.</returns>
            public T DeserializeObjectFromStream<T>(Stream p_Stream) where T : class
            {
                T result = default(T);
                using (StreamReader reader = new StreamReader(p_Stream)) {
                    // When deserializing, Newtonsoft.Json encapsulate the values in its own types.
                    // Unfortunately it causes issues with the shorthand properties of the "SearchResponse" type.
                    // The workaround is to ensure that values are converted in the right type and then
                    // create the "SearchResponse" instance.
                    Dictionary<string, object>  parsed = DeserializeToDictionary(reader.ReadToEnd());
                    // Since this serializer is used only with the "SearchResponse" type, the type
                    // can be created directly.
                    result = new SearchResponse(parsed) as T;
                }
                return result;
            }
            /// <summary>
            /// Serializes an object structure into a JSON stream.
            /// </summary>
            /// <typeparam name="T">The type of the object to serialize.</typeparam>
            /// <param name="p_Stream">The stream in which the JSON is written.</param>
            /// <param name="p_Object">The object to serialize.</param>
            public void SerializeToStream<T>(Stream p_Stream, T p_Object)
            {
                JsonSerializer serializer = JsonSerializer.Create();
                using (JsonTextWriter writer = new JsonTextWriter(new StreamWriter(p_Stream))) {
                    serializer.Serialize(writer, p_Object);
                }
            }
            /// <summary>
            /// Deserializes a JSON string into a dictionary. It will convert the
            /// dictionary values recursively to use the expected .NET types instead
            /// of the Newtonsoft.Json wrappers.
            /// </summary>
            /// <param name="p_Json">A JSON string that represents the value to deserialize.</param>
            /// <returns>The deserialized dictionary.</returns>
            private Dictionary<string, object> DeserializeToDictionary(string p_Json)
            {
                // This call creates a dictionary only for the first level.
                Dictionary<string, object> result = JsonConvert.DeserializeObject<Dictionary<string, object>>(p_Json);
                // This dictionary will contain the converted values.
                Dictionary<string, object> converted = new Dictionary<string, object>();
                foreach (KeyValuePair<string, object> pair in result) {
                    converted.Add(pair.Key, ConvertValue(pair.Value));
                }
                return converted;
            }
            /// <summary>
            /// Deserializes a JSON string into a list. It will convert the
            /// list values recursively to use the expected .NET types instead
            /// of the Newtonsoft.Json wrappers.
            /// </summary>
            /// <param name="p_Json">A JSON string that represents the value to deserialize.</param>
            /// <returns>The deserialized dictionary.</returns>
            private IList<object> DeserializeToList(string p_Json)
            {
                // This call creates a list only for the first level.
                List<object> result = JsonConvert.DeserializeObject<List<object>>(p_Json);
                // This list will contain the converted values.
                List<object> converted = new List<object>();
                foreach (object value in result) {
                    converted.Add(ConvertValue(value));
                }
                return converted;
            }
            /// <summary>
            /// Converts a value in the expected type.
            /// </summary>
            /// <param name="p_Value">The value to convert.</param>
            /// <returns>The converted value.</returns>
            private object ConvertValue(object p_Value)
            {
                object convertedValue;
                if (p_Value is JArray) {
                    convertedValue = DeserializeToList(p_Value.ToString());
                } else if (p_Value is JObject) {
                    convertedValue = DeserializeToDictionary(p_Value.ToString());
                } else if (p_Value is Int64) {
                    // When deserializing an integer value as an object, Newtonsoft.Json
                    // creates an Int64 instance. But the SearchResponse class is expecting
                    // Int32. Values that are too large to fit in an Int32 instance will still
                    // be Int64.
                    try {
                        convertedValue = Convert.ToInt32(p_Value);
                    } catch (OverflowException) {
                        convertedValue = p_Value;
                    }
                } else {
                    convertedValue = p_Value;
                }
                return convertedValue;
            }
        }
    }
    
  3. You now need to implement a custom processor that uses the custom serializer. This processor simply inherits from ParseJsonRestSearchResponseProcessor, which is the default processor, and passes an instance of the custom serializer to the constructor of the base class.

    ParseJsonResponseWithNewtonsoft.cs

    using Coveo.Framework.Pipelines;
    using Coveo.SearchProvider.Rest.Processors.CoveoProcessRestJsonResponse;
    namespace Tutorials.Lib.Processors.coveoProcessRestJsonResponse
    {
        /// <summary>
        /// Custom processor that will use the new JSON serializer.
        /// </summary>
        public class ParseJsonResponseWithNewtonsoft : ParseJsonRestSearchResponseProcessor
        {
            /// <summary>
            /// Simply calls the base constructor by setting the specific JSON serializer.
            /// </summary>
            public ParseJsonResponseWithNewtonsoft()
                : base(new PipelineRunnerHandler(new PipelineRunner()),
                       new NewtonsoftJsonSerializer())
            {
            }
        }
    }
    
  4. Build the assembly containing your custom code and copy the DLL file to your website bin folder.

  5. Finally, modify the configuration file to use the custom processor instead of the default one:

    1. Open the Coveo.SearchProvider.Rest.Custom.config file.

      In a more formal project, you would want to use a separate include file and register the processor there.

    2. Under sitecore, add the following element, and replace the processor definition by your own.

      <pipelines>
        <coveoProcessRestJsonResponse>
          <processor type="Tutorials.Lib.Processors.coveoProcessRestJsonResponse.ParseJsonResponseWithNewtonsoft, Tutorials.Lib" />
        </coveoProcessRestJsonResponse>
      </pipelines>
      
  6. Validate that the Coveo Search REST Endpoint still works properly. If it does, then that it properly uses Newtonsoft.Json to return search results.