Advanced Configuration

Once you have installed and configured the Coveo On-Premises Crawling Module, you can further configure it so that it meets your specific needs. The instructions in this article are optional, and there is no obligation to make these changes if your environment does not require them.

Changing the Default DNS Server Address

Domain Name System (DNS) servers translate hostnames to public IP addresses. For instance, if you type coveo.com in your browser, a DNS server will translate it to the Coveo website server address.

Once you have authenticated to the platform and started a worker, you may notice that your worker cannot communicate with the Coveo Cloud platform properly. The worker log should show the following error: System.Net.WebException: The remote name could not be resolved: 'api.cloud.coveo.com' (see About Component Logs).

To address this issue, you can change the default DNS server address.

In the Maestro Swagger, use the Modify Maestro’s configuration operation described in the following request template to change the default DNS server address.

Request template

PUT http://localhost:5000/api/config HTTP/1.1
Content-Type: application/json-patch+json

Payload

{
  "CustomDnsServers": ["<IPAddress>", "<IPAddress>"]
}

Replace <IPAddress> with the desired DNS server IP addresses.

You can enter several addresses, which will be used in order, i.e., if the first one fails to communicate with Coveo Cloud as expected, the next one will be used, and so on.

For testing purposes, you can use the Google DNS server address: 8.8.8.8.

The body of a successful response is an empty JSON object ({}).

When you change the DNS server address, your workers automatically stop and restart. They will then try to communicate with the Coveo Cloud platform using the specified DNS server IP address.

If you ever need to revert to the default DNS server IP address, use the same API call and, in the request payload, use an empty array for the CustomDnsServers parameter value.

Adding a DNS Suffix

A DNS suffix is added to an unresolved domain name when searching for an IP address. For instance, if you have a coveo.com DNS suffix and want your Crawling Module source to index https://docs.coveo.com, you can simply provide the short form https://docs as the source starting address, and it will be resolved to the full address.

In the Maestro Swagger, use the Modify Maestro’s configuration operation described in the following request template to add a DNS suffix.

Request template

PUT http://localhost:5000/api/config HTTP/1.1
Content-Type: application/json-patch+json

Payload

{
  "CustomDnsSuffixes": ["<Suffix>", "<Suffix>"]
}

Replace <Suffix> with the desired DNS suffixes.

URL short forms are not allowed in the Coveo Cloud administration console source configuration panels. Therefore, to create a source with a short form starting address, you must use the Coveo Platform API.

Using a Proxy

Due to your network configuration, you may want the Crawling Module to download new Docker images and to communicate with Coveo Cloud through a proxy.

  1. In the Maestro Swagger, use the Modify Maestro’s configuration operation described in the following request template to provide a proxy address starting with http:// or https://.

    Request template

     PUT http://localhost:5000/api/config HTTP/1.1
     Content-Type: application/json-patch+json
    

    Payload

     {
       "ProxyAddress": "<Address>"
     }
    

    Replace <Address> with the desired proxy address.

    The body of a successful response is an empty JSON object ({}).

  2. Configure Docker so that it uses the same proxy (see Proxy Configuration).

When you change the ProxyAddress setting, your workers will automatically stop and restart. The process may cause a brief downtime.

To stop using the proxy and return to a direct connection, enter an empty ProxyAddress value ("") in Maestro’s JSON configuration and configure Docker accordingly.

You cannot use a proxy to crawl content through an external connection. If you want to do so, you instead need either a cloud source or a Crawling Module setup in a different location (see Content Retrieval Methods).