--- title: Creating a Crawling Module source using the Source API slug: '3268' canonical_url: https://docs.coveo.com/en/3268/ collection: index-content source_format: adoc --- # Creating a Crawling Module source using the Source API Once you have started your workers, you can [log in](https://docs.coveo.com/en/1697/) to the [Coveo Administration Console](https://docs.coveo.com/en/1841/) to create a [source](https://docs.coveo.com/en/246/) for the content you want to [index](https://docs.coveo.com/en/204/). As an alternative to the Administration Console, you can use the [Coveo Platform API](https://platform.cloud.coveo.com/docs?urls.primaryName=Source#/Sources/rest_organizations_paramId_sources_post) to provide Coveo with the content you want to make searchable. See the general [Source API documentation](https://docs.coveo.com/en/169/) for details. > **Note** > > Coveo only supports Swagger for source creation via the [Source API](https://docs.coveo.com/en/169/). > If you want to use a different tool such as Postman or PowerShell, keep in mind that the [Coveo Support](https://connect.coveo.com/s/case/Case/Default) team offers help with Swagger only. To create a content source, in the [Create a source from simple configuration](https://platform.cloud.coveo.com/docs?urls.primaryName=Source#/Sources/rest_organizations_paramId_sources_post) call, under **Parameters**, provide the [organization ID](https://docs.coveo.com/en/1562/) of the organization you [linked to Maestro](https://docs.coveo.com/en/3266#linking-the-crawling-module-to-an-organization). This replaces `+{organizationId}+` in the request path. You must also provide a JSON source configuration. Each source has different JSON configuration parameters to enter in the request body. Therefore, this article provides source-specific JSON configuration samples for you to use when making the [Create a source from simple configuration](https://platform.cloud.coveo.com/docs?urls.primaryName=Source#/Sources/rest_organizations_paramId_sources_post) call. If you need a more advanced configuration, contact the [Coveo Support](https://connect.coveo.com/s/case/Case/Default) team for help. Once you've created a source, you can [manage it](https://docs.coveo.com/en/3390/) through the [**Sources**](https://platform.cloud.coveo.com/admin/#/orgid/content/sources/) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/content/sources/) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/content/sources/) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/content/sources/)) page of the [Administration Console](https://docs.coveo.com/en/1841/). > **Tip** > > [Subscribe to notifications](https://docs.coveo.com/en/3271#subscribing-to-crawling-module-notifications) to receive an alert when the [Crawling Module](https://docs.coveo.com/en/3260/) stops reporting to Coveo. ## Basic Properties The following basic properties are common to all or most of the Coveo Crawling Module source configurations: * The `name` property value is the source name as it should appear in the [Coveo Administration Console](https://platform.cloud.coveo.com/). Replace `` with the desired source name. > **Note** > > You can't change the name of a source once it has been created, so make sure the name you choose fits the content you intend to index with that source. * The [`sourceVisibility`](https://docs.coveo.com/en/1779/) property determines who can see the source items in their search results. Options are `SHARED` (everyone), `PRIVATE` (some specific users and groups), or `SECURED` (the same users as in the repository's permission system). If you choose to create a [source that indexes permissions](https://docs.coveo.com/en/1779#same-users-and-groups-as-in-your-content-system), see [Indexing Secured Content](https://docs.coveo.com/en/3267#indexing-secured-content) and ensure to add [security workers](https://docs.coveo.com/en/3278/) to your [Crawling Module configuration](https://docs.coveo.com/en/3266#editing-the-crawling-module-configuration). * The `sourceType` value determines the type of source you create. Check [the list](https://docs.coveo.com/en/100/) for the value to use. * Boolean properties `pushEnabled` and `onPremisesEnabled` must always be `true`. * The `username` and `password` values allow the crawler to access the secured content to index. Replace the value placeholders with the corresponding credentials. * The `crawlingModuleId` value identifies the Crawling Module deployment you want to use to retrieve your content. Crawling Module IDs are listed on the [**Crawling Modules**](https://platform.cloud.coveo.com/admin/#/orgid/content/crawling-module/) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/content/crawling-module/) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/content/crawling-module/) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/content/crawling-module/)) page. Also in the [Create a source from simple configuration](https://platform.cloud.coveo.com/docs?urls.primaryName=Source#/Sources/rest_organizations_paramId_sources_post) call, leave the `updateSecurityProviders` and `rebuild` parameter values to `true` unless otherwise instructed by the [Coveo Support](https://connect.coveo.com/s/case/Case/Default) team. For an example of a request and of a response body, see [Create a basic public Web source](https://docs.coveo.com/en/79/) in the Source API documentation. ![Source API Swagger Parameters Section | Coveo](https://docs.coveo.com/en/assets/images/coveo-platform/attachments/37589182/38142326.png) ## Confluence Data Center > **Note** > > Coveo recommends you [install its plugin for Confluence](https://docs.coveo.com/en/1716/) before creating a Confluence Data Center source. ```json { "name": "", "sourceVisibility": "", "sourceType": "CONFLUENCE2_HOSTED", "pushEnabled": true, "onPremisesEnabled": true, "urls": ["", ""], "username": "", "password": "", "crawlingModuleId": "" } ``` In the request body, beside providing an adequate value for the [basic properties](#basic-properties) listed above, make sure to replace `<["URL", "URL"]>` with the addresses to crawl. ## Database ```json { "name": "", "sourceVisibility": "", "sourceType": "DATABASE", "pushEnabled": true, "onPremisesEnabled": true, "username": "", "password": "", "crawlingModuleId": "", "connectionString": "", "ConfigFileContent": "", "DriverType": "Odbc|SqlClient|MySql|Postgres|Oracle|Redshift|Databricks", "ItemType": "" } ``` In the request body, beside providing an adequate value for the [basic properties](#basic-properties) listed above, make sure to: * For `connectionString`, replace `` with the connection string used to connect to the database. See the [Administration Console documentation](https://docs.coveo.com/en/1885#connection-parameters-subtab) for details on connection strings. > **Notes** > > * The connection string syntax differs from one database type to another. > See [Connection Strings](http://www.connectionstrings.com/) for the format to use for your database type. > > * The same connection string can be used for different sources. > However, there can only be one connection string per source. > > * If applicable, specify the name of the desired [ODBC driver](https://docs.coveo.com/en/1885#about-odbc-drivers). * Provide an [XML configuration file](https://docs.coveo.com/en/l15b4451/) with the desired content, mappings, allowed users, etc. to enable Coveo to retrieve and copy the data from record fields to Coveo default and standard source fields. This configuration file must however be base64-encoded for your JSON source configuration to be valid. Use the [Base64 Encode and Decode](https://www.base64decode.org/) online tool, and then replace `` with the encoded output. See also the Administration Console documentation for details on how to [enable the refresh capability](https://docs.coveo.com/en/l15b5020#enabling-the-refresh-capability) and [enable the pausing/resuming update operations](https://docs.coveo.com/en/l15b5020#enable-pausingresuming-update-operations). * For `DriverType`, specify the data provider that provides access to the database. * For `ItemType`, replace `` with the `Mapping type` values from your XML configuration file above. These values are the table or object names to retrieve and must be separated by commas. ## File System ```json { "name": "", "sourceVisibility": "", "sourceType": "FILE", "pushEnabled": true, "onPremisesEnabled": true, "username": "", "password": "", "crawlingModuleId": "", "startingAddresses": ["", "", ...], "expandMailArchives": , "indexSharePermissions": } ``` In the request body, beside providing an adequate value for the [basic properties](#basic-properties) listed above, make sure to: * Replace `+` with the address to crawl. * Indicate `true` or `false` for: ** `expandMailArchives`, which determines whether the content of mail archives (.pst) should be indexed. Default value is `false`. ** `indexSharePermissions`, which determines whether share and NTFS permissions should be taken into account and applied in Coveo. > **Note** > > To take permissions into account, contact the [Coveo Support](https://connect.coveo.com/s/case/Case/Default) team. ## Jira Software Data Center > **Note** > > To support indexing all [permission](https://docs.coveo.com/en/223/) types, [install Coveo's plugin on your instance](https://docs.coveo.com/en/1918/) before you create a Jira Software Data Center source. ```json { "name": "", "sourceVisibility": "", "sourceType": "JIRA2_HOSTED", "pushEnabled": true, "onPremisesEnabled": true, "username": "", "password": "", "crawlingModuleId": "", "serverUrl": "", "indexAttachments": "", "indexComments": "", "indexWorkLogs": "", "supportCommentPermissions": "" } ``` In the request body, beside providing an adequate value for the [basic properties](#basic-properties) listed above, make sure to: * Replace `` with the address of your Jira Software Data Center instance. **Example** `+http://MyJiraServer:8080/+` * Indicate `true` or `false` for: ** `indexAttachments`, which determines whether binary files attached to an issue should be indexed. Attachments are indexed with the same level and sets of their parent issue. ** `indexComments`, which determines whether comments on an issue should be indexed. Comments are indexed with the same level and sets of their parent issue. When permissions on the comments are supported, if a comment is restricted to a group or a project role, an additional set with the group or the role is added. ** `indexWorkLogs`, which determines whether time entry on a issue should be indexed. Work logs are indexed with the same level and sets of their parent issue. If a work log is restricted to a group or a project role, an additional set with the group or the role is added. ** `supportCommentPermissions`, which determines whether only users allowed to see a comment in Jira can also see it in their search results. If this property value is true, an issue and its comments are indexed as separate items, leading to lower search relevance. If the value is false, the issue and its comments are indexed as one item, allowing to find an item via either an issue or its comments. However, there are no restrictions on users seeing comments on an issue. ## SharePoint Server You can use a SharePoint Server source to make your SharePoint 2019, 2016, 2013, or 2010 content searchable. The following general configuration works for most use cases. However, to index SharePoint content that's secured with an Active Directory security identity provider, use [this configuration](#sharepoint-server-with-an-active-directory-security-identity-provider) instead. ```json { "name": "", "sourceVisibility": "", "sourceType": "SHAREPOINT", "pushEnabled": true, "onPremisesEnabled": true, "username": "", "password": "", "crawlingModuleId": "", "urls": <["URL", "URL"]>, "authenticationType": "WindowsUnderClaims|AdfsUnderClaims", "crawlScope": "WebApplication|SiteCollection|WebAndSubWebs|List" } ``` In the request body, beside providing an adequate value for the [basic properties](#basic-properties) listed above, make sure to: * Replace `<["URL", "URL"]>` with the site collection, list, website, or subsite addresses to crawl. **Examples** * For a specific web application: `+https://site:8080/+` * For a specific site collection: `+https://site:8080/sites/support+` * For a specific website: `+https://site:8080/sites/support/subsite+` * For a specific list: `+https://site:8080/sites/support/lists/contacts/allItems.aspx+` > **Important** > > Indexing a specific folder in a list isn't supported. * Indicate, under `authenticationType`, the authentication type value corresponding to your SharePoint environment. Available values are: [cols="30%,70%"] |=== |Value |Description |`WindowsUnderClaims` |Windows authentication mode under claims (includes Windows Classic) |`AdfsUnderClaims` |Authentication for Trusted Security Providers |=== * Indicate, under `crawlScope`, the content type that you want to crawl in relation with the source `urls` that you specified. Indicate `WebApplication`, the default value and highest element type in the SharePoint farm hierarchy to crawl everything. [cols="30%,70%"] |=== |Value | Content to crawl |`WebApplication` |All site collections of the specified web application |`SiteCollection` |All web sites of the specified site collection |`WebAndSubWebs` |Only the specified web site and its sub webs |`List` |Only the specified list or document library |=== **Examples** * To crawl a web application: ```json { "name": "My SharePoint Web Application", "sourceVisibility": "SECURED", "sourceType": "SHAREPOINT", "urls": ["http://mysharepointserver:35318/"], "authenticationType": "AdfsUnderClaims", "username": "john.smith@mycompany.com.com", "password": "MyPassword", "crawlingModuleId": "", "AdfsServerUrl": "https://adfs.server.com/", "SharePointTrustIdentifier": "urn:federation:MicrosoftOnline", "crawlScope": "WebApplication", "loadUserProfiles": true, "loadPersonalSites": true, "indexListFolders": true } ``` * To crawl a sub web: ```json { "name": "My SharePoint Sub Web", "sourceVisibility": "SECURED", "sourceType": "SHAREPOINT", "username": "mycompany\john.smith", "password": "MyPassword", "crawlingModuleId": "myorganizationmf767fwm-1j862vd92-j8g1-9223-pa89-bd72m0a9b0d", "urls": ["http://mysharepointserver:35318/site/web/subweb"], "authenticationType": "WindowsUnderClaims", "crawlScope": "WebAndSubWebs" } ``` ### SharePoint Server with an Active Directory security identity provider To retrieve SharePoint 2019, 2016, 2013, or 2010 content that's secured with an Active Directory security identity provider, use the following configuration. This configuration is similar to that of other [SharePoint Server](#sharepoint-server) sources, with additional Active Directory parameters. ```json { "name": "", "sourceVisibility": "SECURED", "sourceType": "SHAREPOINT", "pushEnabled": true, "onPremisesEnabled": true, "username": "", "password": "", "urls": <["URL", "URL"]>, "authenticationType": "WindowsClassic", "crawlingModuleId": "YOUR CRAWLING MODULE ID", "crawlScope": "WebApplication|SiteCollection|WebAndSubWebs|List", "activeDirectoryUsername": "", "activeDirectoryPassword": "", "activeDirectoryEmailAttributes": <["attribute", "attribute"]>, "activeDirectoryExpandWellKnowns": , "activeDirectoryTlsMode": "" } ``` In the request body, beside providing an adequate value for the [basic properties](#basic-properties) as well as the general SharePoint Server properties listed [above](#sharepoint-server), make sure to: * Replace `` and `` with the credentials required to access your Active Directory. * Replace `<["attribute", "attribute"]>` with a list of attributes under which Coveo can find the email address associated to a security identity. Should an attribute contain more than one value, Coveo uses the first one. If you don't provide attributes, Coveo retrieves email addresses from the `mail` attribute by default. * Indicate `true` for `activeDirectoryExpandWellKnowns` if you want the users included in your Active Directory [well-known](https://docs.coveo.com/en/1603#granted-security-identities) security identifiers to be granted access to the indexed content. Expect an increase in the duration of the security provider refresh operation. Supported well-known SIDs are: `Everyone`, `Authenticated Users`, `Domain Admins`, `Domain Users`, and `Anonymous Users`. > **Tip** > > If your entire site collection is secured with the `Everyone` or `Authenticated users` well-known, it's more cost-effective resource-wise to index its content with a source whose content is accessible to everyone (`"sourceVisibility": "SHARED"`) than to expand this well-known with a source with `"sourceVisibility": "SECURED"`. * If you want to use a TLS protocol to retrieve your security identities, the `activeDirectoryTlsMode` value should be the protocol to use: `StartTls` or `Ldaps`. Coveo strongly recommends using StartTLS if you can. Since Ldaps is a much older protocol, you should only select this value if StartTLS is incompatible with your environment. ## Sitemap You can use a Sitemap source to make searchable the content of listed web pages from a Sitemap (Sitemap file or a Sitemap index file). ```json { "name": "", "sourceVisibility": "", "sourceType": "SITEMAP", "pushEnabled": true, "onPremisesEnabled": true, "crawlingModuleId": "", "urls": <["URL", "URL"]>, "userAgent": "<>", "enableJavaScript": , "scrapingConfiguration": <""|SCRAPING_CONFIG> } ``` In the request body, beside providing an adequate value for the [basic properties](#basic-properties) listed above, make sure to: * Replace `<["URL", "URL"]>` with the addresses to crawl. * Provide the value of the user-agent HTTP header to use as userAgent. This is the identifier used when downloading web pages. Default is `Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.110 Safari/537.36`. * Indicate `true` or `false` for `enableJavaScript`, which determines whether JavaScript should be evaluated and rendered before indexing. This option is useful when you want to index the dynamically rendered content of crawled pages. However, activating this option has a significant impact on the crawling performance. * Provide a [JSON `scrapingConfiguration`](https://docs.coveo.com/en/mc1f3573/) or leave the quotation marks empty. ## Web You can use the Coveo Crawling Module to crawl an internal website, that is, pages available on a certain network only. To crawl a public website, that is, a website that's globally available on the Internet, use the [Coveo Administration Console](https://docs.coveo.com/en/1841/) to create a [cloud](https://docs.coveo.com/en/1612/)-type [Web source](https://docs.coveo.com/en/malf0160/). ```json { "name": "", "sourceVisibility": "", "sourceType": "WEB2", "pushEnabled": true, "onPremisesEnabled": true, "urls": [""], "crawlingModuleId": "" } ```