--- title: Add a Database source slug: '1885' canonical_url: https://docs.coveo.com/en/1885/ collection: index-content source_format: adoc --- # Add a Database source A Database [source](https://docs.coveo.com/en/246/) indexes the content of a database. You can then make this content searchable through a Coveo-powered search interface. For example, your company developer may have created a custom database to manage the parts used in your facilities, their location in your warehouse, and purchase orders. You could decide to index data regarding purchase orders only so that your buyers can find this content via your Coveo-powered employee portal. ## Source key characteristics The following table presents the main characteristics of a Database source. [%header,cols="2,3,1,3"] |=== 2+|Features ^|Supported |Additional information .3+|[Content update operations](https://docs.coveo.com/en/2039/) |[refresh](https://docs.coveo.com/en/2710/) ^|[check] |See [Enabling the Refresh Capability](https://docs.coveo.com/en/l15b5020#enabling-the-refresh-capability). |[rescan](https://docs.coveo.com/en/2711/) ^|[check] |[Takes place every day by default](https://docs.coveo.com/en/1933/). |[rebuild](https://docs.coveo.com/en/2712/) ^|[check] | .3+|[Content security](https://docs.coveo.com/en/1779/) options |[Same users and groups as in your content system](https://docs.coveo.com/en/1779#same-users-and-groups-as-in-your-content-system) ^|[check] | |[Specific users and groups](https://docs.coveo.com/en/1779#specific-users-and-groups) ^|[check] | |[Everyone](https://docs.coveo.com/en/1779#everyone) ^|[check] | .3+|[Metadata indexing for search](#index-metadata) |Automatic mapping of [metadata](https://docs.coveo.com/en/218/) to [fields](https://docs.coveo.com/en/200/) that have the same name 2+a|[Enabled by default](https://docs.coveo.com/en/1640#about-the-performfieldmappingusingallorigins-setting). |Automatically indexed [metadata](https://docs.coveo.com/en/218/) 2+a|Examples of [auto-populated fields](https://docs.coveo.com/en/1833#field-origin): * `date` * `filetype` (autodetected from [`` or ``](https://docs.coveo.com/en/l15b4451#fields)) * `language` (autodetected from [`` or ``](https://docs.coveo.com/en/l15b4451#fields)) * `size`   Sample of auto-populated fields (when the source configuration XML includes the corresponding element): * `author` (using a [](https://docs.coveo.com/en/l15b4451#customfield) element) * [`title`](https://docs.coveo.com/en/l15b4451#title) * [`uri`](https://docs.coveo.com/en/l15b4451#uri-required)   After a content update, [inspect your item field values](https://docs.coveo.com/en/2053#inspect-search-results) in the **Content Browser**. |Extracted but not indexed metadata 2+a|Source configuration `` elements specified in the source whose `name` attribute values don't match a Coveo default field name are extracted, but not indexed by default. After a rebuild, review the [**View and map metadata**](https://docs.coveo.com/en/m9ti0339#view-and-map-metadata-subpage) subpage for the list of indexed metadata, and [index additional metadata](https://docs.coveo.com/en/m9ti0339#index-metadata). |=== > **Leading practice** > > The number of [items](https://docs.coveo.com/en/210/) that a source processes per hour (crawling speed) depends on various factors, such as network bandwidth and source configuration. > See [About crawling speed](https://docs.coveo.com/en/2078/) for information on what can impact crawling speed, as well as possible solutions. ## Supported database types The cloud version (icon:cloud-icon[alt=cloud-icon,width=16]) of the source supports the following database types: * Databricks * Microsoft SQL Server * MySQL * Postgres * Oracle * Redshift The [Crawling Module](https://docs.coveo.com/en/3260/) version ([crawlingmodule]) supports the following database types: * Microsoft SQL Server * MySQL * Postgres * Oracle * Any database using an [ODBC driver](#about-odbc-drivers). This driver must be installed on your Crawling Module server. ## Commerce requirements When using a Database [source](https://docs.coveo.com/en/246/) to [index](https://docs.coveo.com/en/204/) [commerce-specific content](https://docs.coveo.com/en/n8of5561/), such as products, [variants](https://docs.coveo.com/en/mc7f0326/), and [availabilities](https://docs.coveo.com/en/mc7e9096/), you have to undergo a catalog configuration process to benefit from all commerce-related capabilities. More specifically, you must: * [Enable Coveo Personalization-as-you-go (PAYG) capabilities](#enable-coveo-personalization-as-you-go) in your source. * Associate your source with a [catalog configuration](#catalog-configuration). ### Enable Coveo Personalization-as-you-go [Coveo Machine Learning](https://docs.coveo.com/en/188/) tools include [Coveo Personalization-as-you-go](https://docs.coveo.com/en/m5kd0347/) (PAYG) capabilities for commerce use cases. This suite of advanced features learns from a user's intent and reacts within a few clicks. PAYG models require the building of a [product vector space](https://docs.coveo.com/en/l9gg3565/) to represent the products contained in your source. For Database sources, Coveo PAYG needs to be enabled in order to produce the product vector space. Contact your Coveo representative to discuss your options. ### Catalog configuration Behind the scenes, the Database source uses the [Stream API](https://docs.coveo.com/en/12#tag/Stream) to push content to the Coveo [index](https://docs.coveo.com/en/204/). Therefore, Database sources must be associated with a [catalog entity](https://docs.coveo.com/en/3143/) to ensure a complete configuration. This allows the source to accurately build a [product vector](https://docs.coveo.com/en/l9gg3565/) space. For instructions on how to create a catalog entity, see [Commerce catalog entity](https://docs.coveo.com/en/3139/). ## Add a Database source Follow the instructions below to add a Database source using the desired [content retrieval method](https://docs.coveo.com/en/1612/). . On the [**Sources**](https://platform.cloud.coveo.com/admin/#/orgid/content/sources/) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/content/sources/) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/content/sources/) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/content/sources/)) page, click **Add source**. . In the **Add a source of content** panel, click the **Cloud** (icon:cloud-icon[alt=cloud-icon,width=16]) or **Crawling Module** ([crawlingmodule]) tab, depending on your [content retrieval context](https://docs.coveo.com/en/1612/). With the latter, you must [install the Crawling Module](https://docs.coveo.com/en/3263/) to make your source operational. . Click the **Database** tile. . Configure your source. > **Leading practice** > > It's best to create or edit your source in your sandbox organization first. > Once you've confirmed that it indexes the desired content, you can copy your source configuration to your production organization, either [with a snapshot](https://docs.coveo.com/en/3239/) or manually. > > See [About non-production organizations](https://docs.coveo.com/en/2959/) for more information and best practices regarding sandbox organizations. ### "Configuration" tab On the **Add a Database Source** page, the **Configuration** tab is selected by default. It contains your source's general and authentication information, as well as other parameters. #### "Identification" subtab The **Identification** subtab contains general information about the source. ## Name The source name. It can't be modified once it's saved. ##### Name Enter a name for your source. > **Leading practice** > > A source name can't be modified once it's saved, therefore be sure to use a short and descriptive name, using letters, numbers, hyphens (`-`), and underscores (`_`). Avoid spaces and other special characters. ##### Project Use the **Project** selector to associate your source with one or more Coveo [projects](https://docs.coveo.com/en/n7ef0517/). #### "Connection parameters" subtab Enter your database connection parameters or a connection string. Coveo will use this information to access your database. Since the connection string syntax differs from one database type to another, you might want to see [The connection strings reference](https://www.connectionstrings.com/) for details. When providing credentials to access your database, enter the credentials of a dedicated user account that has access to the content you want to index. See [Source credentials leading practices](https://docs.coveo.com/en/1920/). > **Notes** > > When you paste a connection string into the **Connection string** box: > > * You may notice that Coveo slightly reformats your connection string or reorders parameters. > > * Coveo automatically parses your connection string to fill connection parameter boxes. > Depending on the **Data provider type**, only the **Username** and **Password** boxes may be displayed and filled. > > * Coveo replaces the credentials in the connection string with placeholders `@uid` and `@pwd`. > Since connection strings aren't encrypted, they should never contain credentials in plain text. > > ![Coveo Database Connection String Credential Security](:https://docs.coveo.com/en/assets/images/index-content/database-connection-string-credentials.gif) > > These behaviors are normal and don't indicate that your connection string is invalid. ##### About ODBC drivers If you're using a Crawling Module source to index content from a database using an ODBC driver, ensure that you have the appropriate [ODBC driver](https://en.wikipedia.org/wiki/Open_Database_Connectivity#Drivers) installed on your Crawling Module server. A driver acts as a intermediary between Coveo's connector and your database server. For example, to connect Coveo to an Oracle source, you must install an Oracle driver. ODBC drivers are typically provided by the database vendor. You can also use ODBC Data Source Administrator to review the drivers installed on your server. The [Crawling Module REST API](https://docs.coveo.com/en/3266/) offers a [call that lists the drivers that are installed and available on your server](https://docs.coveo.com/en/3266#getting-the-available-drivers-for-an-odbc-source). #### "Content to index" subtab The **Content to index** subtab lets you define the content that you want to make available as search results. ##### Item types Enter the name of the tables or view objects you want to index. ##### XML configuration For each item you entered above, add the corresponding XML read query to define what to index and how. Specify the data to copy from database fields to Coveo [fields](https://docs.coveo.com/en/200/). See the [reference page](https://docs.coveo.com/en/l15b4451/) to help you in the process. > **Important** > > Your XML queries should be read queries only, or else you could make unwanted changes to your database. If needed, you can [run subqueries](https://docs.coveo.com/en/l15b5020#complement-information-retrieval-using-subqueries) on every item to complete the information with more complex queries. See also [Further configuration](https://docs.coveo.com/en/l15b5020/) for other options you can implement. #### "Advanced settings" subtab The advanced settings are optional. They let you fine-tune the behavior of your source in some specific situations. ##### Command timeout Try increasing the timeout value when: * Your database is slow to respond to queries. * Your queries take long to execute. * You encounter database timeout errors. First increase the value to 120 seconds. If the issue persists, you can either increase the timeout value again or use paged SQL queries instead. ##### Maximum response page size This is the maximum number of items a database can return at once. If you know your query will return a large number of items and this may affect your database's stability, try decreasing this value. #### "Crawling Module" subtab If your source is a [Crawling Module source](https://docs.coveo.com/en/1612/), and if you haven't already installed the Coveo Crawling Module on a server that has access to the database of which you want to retrieve the content, click **Download** in the information box to do so. If you have [more than one Crawling Module linked to this organization](https://docs.coveo.com/en/3271#deploying-multiple-crawling-module-instances), select the one with which you want to pair your source. If you change the Crawling Module instance paired with your source, a successful [rebuild](https://docs.coveo.com/en/3390#refresh-rescan-or-rebuild-sources) is required for your change to apply. ### "Items" tab On the **Items** tab, you can specify how the source handles items based on their file type or content type. #### File types File types let you define how the source handles [items](https://docs.coveo.com/en/210/) based on their file extension or content type. For each file type, you can specify whether to index the item content and [metadata](https://docs.coveo.com/en/218/), only the item metadata, or neither. You should fine-tune the file type configurations with the objective of indexing only the content that's relevant to your users. **Example** Your repository contains `.pdf` files, but you don't want them to appear in search results. You click **Extensions** and then, for the `.pdf` extension, you change the **Default action** and **Action on error** values to `Ignore item`. For more details about this feature, see [File type handling](https://docs.coveo.com/en/l3qg9275/). #### Content and images If you want Coveo to extract text from image files or PDF files containing images, enable the appropriate option. The extracted text is processed as item data, meaning that it's fully searchable and will appear in the item [Quick view](https://docs.coveo.com/en/2760#search-result-quick-view). > **Note** > > When OCR is enabled, ensure the source's relevant [file type configurations](https://docs.coveo.com/en/l3qg9275/) index the item content. > Indexing the item's metadata only or ignoring the item will prevent OCR from being applied. See [Enable optical character recognition](https://docs.coveo.com/en/2937/) for details on this feature. ### "Content security" tab Select who will be able to access the source items through a Coveo-powered [search interface](https://docs.coveo.com/en/2741/). For details on the content security options, see [Content security](https://docs.coveo.com/en/1779/). If you select **Everyone** or **Specific users or groups**, skip to <>. If you select the **Same users and groups as in your current permission system** option to index a permission system, ensure that you've defined `AllowedUsers` nodes in your <>. Moreover, check the **Permission system is Active Directory** box if applicable. If so, you'll need to enter your Active Directory details. **Active Directory details**
Details **Username and password** Enter credentials to grant Coveo access to your Active Directory. See [Source credentials leading practices](https://docs.coveo.com/en/1920/). **Email attributes** By default, Coveo retrieves the email address associated to each security identity from the `mail` attribute. Optionally, you can specify additional or different attributes to check. Should an attribute contain more than one value, Coveo uses the first one. **Use TLS protocol to connect to Active Directory** Select this option to use a TLS protocol to retrieve your security identities. If you do, we strongly recommend selecting StartTLS if you can. Since LDAPS is a much older protocol, you should only select this value if StartTLS is incompatible with your environment. **The database content is secured with well-known SIDs** Select this option if you want the users that are included in your Active Directory [well-known](https://docs.coveo.com/en/1603#granted-security-identities) security identifiers to be granted access to the indexed content. Supported well-known SIDs are: `Everyone`, `Authenticated Users`, `Domain Admins`, `Domain Users`, and `Anonymous Users`. When enabling this option, you can expect an increase in the duration of the security identity provider refresh operation. [TIP] **Leading practice**
If your entire content is secured with `Everyone` or `Authenticated users`, we recommend selecting the [**Everyone** content security option](https://docs.coveo.com/en/1779#everyone) instead. The result will be the same, that is, all users will be able to access the database content through your search interface, and Coveo's update operations will be more efficient. #### **Bidirectional relationship between the root domain of your Crawling Module server and your other domains** Select this option to have Coveo connect to your root domain to get the security identities of your other domains through the root domain. If your environment contains more than one domain, you can establish a bidirectional or outbound cross-link relationship between the root domain of your Crawling Module server and your additional domains. When you do so, these domains trust your root domain, and Coveo can get their security identities through this root domain. When enabling this option, you can expect an increase in the duration of the security identity provider refresh operation. Moreover, if a linked domain is unreachable, Coveo stops the security identity provider refresh operation. ##### === "Access" tab . On the **Access** tab, specify whether each group (and API key, if applicable) in your [Coveo organization](https://docs.coveo.com/en/185/) can view or edit the current source. For example, when creating a new source, you could decide that members of Group A can edit its configuration, while Group B can only view it. For more information, see [Custom access level](https://docs.coveo.com/en/3151#custom-access-level). On the **Access** tab, specify whether each group (and API key, if applicable) in your [Coveo organization](https://docs.coveo.com/en/185/) can view or edit the current source. For example, when creating a new source, you could decide that members of Group A can edit its configuration, while Group B can only view it. For more information, see [Custom access level](https://docs.coveo.com/en/3151#custom-access-level). ### Build the source . Finish adding or editing your source: ** When you're done editing the source and want to make your changes effective, click **Add and build source**/**Save and rebuild source**. ** When you want to save your source configuration changes without starting a build/rebuild, such as when you know you want to make other changes soon, click **Add source**/**Save**. On the [**Sources**](https://platform.cloud.coveo.com/admin/#/orgid/content/sources/) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/content/sources/) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/content/sources/) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/content/sources/)) page, click **Launch build** or **Start required rebuild** when you're ready to make your changes effective and index your content. > **Leading practice** > > By default, a Jira Software source indexes the entire Jira Software instance content. > To index only certain projects, click **Save**, and then specify the desired address patterns in your [source JSON configuration](https://docs.coveo.com/en/1685/) before launching the initial build. > See [Add source filters](https://docs.coveo.com/en/2006#add-source-filters) for further information. . On the [**Sources**](https://platform.cloud.coveo.com/admin/#/orgid/content/sources/) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/content/sources/) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/content/sources/) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/content/sources/)) page, follow the progress of your source addition or modification. . Once the source is built or rebuilt, [review its content in the Content Browser](https://docs.coveo.com/en/2053/). . Optionally, consider [editing or adding mappings](https://docs.coveo.com/en/1640/). > **Note** > > If you selected **Specific URLs** or **User profiles** in the [**Content**](https://docs.coveo.com/en/1739#content) section, some additional items will appear in the Content Browser. > To retrieve user profiles, Coveo must crawl your SharePoint Online instance, including your host site collection and the documents it contains. > Items encountered during this process are also retrieved and therefore appear in the Content Browser. ### Index metadata To use [metadata](https://docs.coveo.com/en/218/) values in [search interface](https://docs.coveo.com/en/2741/) [facets](https://docs.coveo.com/en/198/) or result templates, the metadata must be [mapped](https://docs.coveo.com/en/217/) to [fields](https://docs.coveo.com/en/200/). With recently created Database sources, [Coveo automatically maps metadata to fields with the same name](https://docs.coveo.com/en/1640#automatically-and-manually-created-mappings). > **Note** > > To enable this auto-mapping behavior on older Database sources, set the [`performFieldMappingUsingAllOrigins`](https://docs.coveo.com/en/1640#about-the-performfieldmappingusingallorigins-setting) parameter to `true` in the source JSON configuration. Coveo has some [default fields](https://docs.coveo.com/en/1833#field-origin) for commonly extracted metadata (for example, `author`, `date`). For any custom metadata defined in your source [XML configuration](https://docs.coveo.com/en/l15b4451/), you must create a field with the same name to store the metadata values. For example, if you've defined a `department` metadata, you must have a `department` field to store the metadata values. . Review the [](https://docs.coveo.com/en/l15b4451#customfields) element in your source [XML configuration](https://docs.coveo.com/en/l15b4451/) for the list of metadata you're currently extracting from your content. . On the [**Fields**](https://platform.cloud.coveo.com/admin/#/orgid/content/fields/) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/content/fields/) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/content/fields/) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/content/fields/)) page, for each metadata you want to use in facets or result templates, [add a field](https://docs.coveo.com/en/1833#add-a-field) with the same name, unless one already exists. > **Note** > > Fields are shared across sources in your [Coveo organization](https://docs.coveo.com/en/185/). > If a field with the same name as the metadata you want to index already exists and its [configuration](https://docs.coveo.com/en/1833#field-options) suits you, use it for the metadata you want to index. > Otherwise, you can [create a mapping](https://docs.coveo.com/en/1640#manage-mappings) to index the metadata in a new field with a different name. **Example** You decided to retrieve picture URIs from your content using the following metadata definition in your source [XML configuration](https://docs.coveo.com/en/l15b4451/): `%[pictureuri]` Since there's no Coveo `pictureuri` default field, you need to create the `pictureuri` field, unless it already exists. . Return to the [**Sources**](https://platform.cloud.coveo.com/admin/#/orgid/content/sources/) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/content/sources/) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/content/sources/) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/content/sources/)) page. . To reindex your source with your new mappings, click your source, and then click **More** > **Rebuild** in the Action bar. . Once the source is rebuilt, review your item field values. They should now include the values of the metadata you selected to index. .. On the [**Sources**](https://platform.cloud.coveo.com/admin/#/orgid/content/sources/) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/content/sources/) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/content/sources/) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/content/sources/)) page, click your source, and then click **More** > **Open in Content Browser** in the Action bar. .. Select the card of the item for which you want to inspect properties, and then click **Properties** in the Action bar. .. In the panel that appears, select the **Fields** tab. ## Crawling Module time zone adjustment If you're using the Crawling Module to index your content and if this Crawling Module is running in a different time zone than your database, [edit the source JSON configuration](https://docs.coveo.com/en/1685/) and add the `TzdbTimeZoneId` [hidden parameter](https://docs.coveo.com/en/2006#add-a-hidden-source-parameter) as follows. The expected time zone format is [IANA TZDB](https://en.wikipedia.org/wiki/Tz_database). **Example** The time zone of your database is America/New_York, therefore you add the following to the source JSON configuration: ```json "TzdbTimeZoneId": { "sensitive": false, "value": "America/New_York" } ``` ## Required privileges You can assign privileges to allow access to specific tools in the [Coveo Administration Console](https://docs.coveo.com/en/183/). The following table indicates the privileges required to view or edit elements of the [**Sources**](https://platform.cloud.coveo.com/admin/#/orgid/content/sources/) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/content/sources/) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/content/sources/) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/content/sources/)) page and associated panels. See [Manage privileges](https://docs.coveo.com/en/3151/) and [Privilege reference](https://docs.coveo.com/en/1707/) for more information. > **Note** > > The **Edit all** privilege isn't required to create sources. > When granting privileges for the [Sources](https://docs.coveo.com/en/1707#sources-domain) domain, you can grant a group or API key the **View all** or [**Custom**](https://docs.coveo.com/en/3151#custom-access-level) access level, instead of **Edit all**, and then select the **Can Create** checkbox to allow users to create sources. > See [Can Create ability dependence](https://docs.coveo.com/en/3151#can-create-ability-dependence) for more information. ## What's next? * See [Further configuration](https://docs.coveo.com/en/l15b5020/) for options you can implement to make your source more efficient. * [Schedule source updates](https://docs.coveo.com/en/1933/). * If you encounter database timeout errors, try adjusting the [Command timeout](#command-timeout). * If you're [using the Crawling Module to retrieve your content](https://docs.coveo.com/en/1612/), consider [subscribing to deactivation notifications](https://docs.coveo.com/en/3271#subscribing-to-crawling-module-notifications) to receive an alert when a Crawling Module component becomes obsolete and stops the content crawling process. If you're [using the Crawling Module to retrieve your content](https://docs.coveo.com/en/1612/), consider [subscribing to deactivation notifications](https://docs.coveo.com/en/3271#subscribing-to-crawling-module-notifications) to receive an alert when a Crawling Module component becomes obsolete and stops the content crawling process. * Consider [subscribing to deactivation notifications](https://docs.coveo.com/en/3271#subscribing-to-crawling-module-notifications) to receive an alert when a Crawling Module component becomes obsolete and stops the content crawling process.