---
title: Index with a generic connector
slug: l89f0171
canonical_url: https://docs.coveo.com/en/l89f0171/
collection: adobe
source_format: adoc
---
# Index with a generic connector
The [Coveo Platform](https://docs.coveo.com/en/186/) provides two [generic connectors](https://docs.coveo.com/en/pa6h0282/) that may be used to index website content, namely the [Sitemap](https://docs.coveo.com/en/1967/) and [Web](https://docs.coveo.com/en/malf0160/) connectors.

The goals of this article are:

- To present the characteristics and features of the Sitemap and Web connectors.
This information will help you determine the optimal indexing strategy for your use case.

- To guide you in creating [sources](https://docs.coveo.com/en/246/) in your Coveo organization once you've determined your indexing strategy.

> **Note**
>
> Though they represent different concepts, the terms _source_ and _connector_ are often used interchangeably in Coveo terminology.

## Sitemap and Web connector comparison table
Consider the characteristics of the Sitemap and Web connectors in the table below when deciding how to index your Adobe Experience Manager content.
Green check marks in the table below highlight the advantages of each connector over the other.

[%header,cols="d,d,d"]
|===
|Criteria
|Sitemap connector
|Web connector

|**Prerequisites**

a|An existing sitemap or a new, Coveo-specific sitemap.

Learn about [AEM out-of-the-box sitemaps](https://experienceleague.adobe.com/docs/experience-manager-learn/sites/seo/sitemaps.html?lang=en), which are available starting with AEM 6.5.9.

a|[check] None.

a|**Content coverage**

{nbsp}

(Ability to index all content from AEM. Ease of scoping the AEM content to be indexed.)

a|Covers all content.

{nbsp}

DAM is supported if the sitemap includes links to assets.

{nbsp}

Content can be filtered using inclusion/exclusion rules.

a|Covers all content.

{nbsp}

DAM is supported if assets are linked to parent web pages.

{nbsp}

Content can be filtered using inclusion/exclusion rules.

|**Indexing speed**

|[check] Faster than with the Web connector since the Sitemap connector simply fetches the web pages listed in the sitemap file.

|Slower than with the Sitemap connector because the Web connector has to discover content, reading a web page to find links to other pages.

a|**Metadata**

{nbsp}

(What kind of metadata can you index?)

a|Using a [Web scraping configuration](#using-a-web-scraping-configuration)

{nbsp}

Enabling the indexing of the [web page <meta> tag `content` attribute values](#indexing-web-page-head-section-metadata-in-a-sitemap-source)

{nbsp}

Enabling the indexing of [JSON-LD <script> tags in your web pages](#using-json-ld-script-tags-in-your-web-pages)

{nbsp}

[check] Indexing [standard sitemap metadata](https://docs.coveo.com/en/1833#field-origin) (for example, `loc`, `lastmod`)

{nbsp}

[check] Adding [Coveo-specific metadata in the sitemap file](#adding-coveo-metadata-tags-directly-in-your-sitemap-file)

{nbsp}

[check] Using [third-party sitemap file extensions](https://docs.coveo.com/en/2656#third-party-extensions).

a|Using a [Web scraping configuration](#using-a-web-scraping-configuration)

{nbsp}

Indexing of the [web page <meta> tag `content` attribute values](https://docs.coveo.com/en/mc1f0259/)

{nbsp}

Enabling the indexing of [JSON-LD <script> tags in your web pages](#using-json-ld-script-tags-in-your-web-pages)

a|**Partial update support**

{nbsp}

(Can the connector index only what has changed in the source since the last indexing operation? Is this indexing triggered  manually or can it be scheduled?)

a|[check] Manual and scheduled [refresh](https://docs.coveo.com/en/2710/) is supported, provided the target sitemap file defines the optional `Last Modification Date` (for example, `lastmod` for XML sitemaps).

{nbsp}

The maximum refresh schedule frequency is every 5 minutes.

{nbsp}

A [rescan](https://docs.coveo.com/en/2711/) (either manual or scheduled) or [rebuild](https://docs.coveo.com/en/2712/) operation is required to take into account deleted and new sitemap entries.

|Refresh isn't supported.
Only rescans (either manual or scheduled) and rebuilds are available.

|===

The Coveo Sitemap connector is the ideal choice for Adobe Experience Manager content, not only from an indexing performance perspective but also because of the many metadata indexing options it provides.
The Coveo Web connector should only be considered as a fallback solution.


## Adobe Experience Manager sitemaps

Sitemaps are an easy way for webmasters to inform search engines about pages on their sites that are available for crawling.
In its simplest form, a Sitemap is an XML file that lists URLs for a site along with additional metadata about each URL, so that search engines can more intelligently crawl the site.
[Here’s an example of a Sitemap](https://corporate.barca.group/content/barca-corporate.sitemap.en-sitemap.xml).

[Sitemaps](https://experienceleague.adobe.com/en/docs/experience-manager-learn/sites/seo/sitemaps) in AEM are considered an opt-in feature.
That is, we need to explicitly enable them.

There are two ways to do it:

* **Configure Sitemap Scheduler**: This generates sitemap files at a defined frequency (for example, once a day).
Using this approach, a background process creates the sitemap and caches it in AEM.
It is then served from cache, when requested.
This approach is recommended by Adobe for large websites, such as those indexed with Coveo.

* **On-demand Sitemap Generation**: In this method, a sitemap is generated whenever a request for Sitemap is made.
This approach is suitable only for small sites in production.
[Enable on-demand sitemap generation](#enable-on-demand-sitemap-generation) during development so you don't have to wait until the scheduler kicks in.

### AEM sitemap customization

Customizing sitemaps in AEM provides greater control over content indexing, enhancing SEO and search engine visibility.
By leveraging the AEM SEO - Page Tree Sitemap Generator and extending the Apache Sling Sitemap Generator with Java, developers can enrich sitemap entries with custom metadata, optimized update frequencies, and additional SEO-friendly elements.
These enhancements not only refine sitemap structures but also ensure that search engines, including Coveo, can efficiently index custom sitemap elements, improving content discoverability.

* AEM offers the flexibility to enable or disable options such as `Add Last Modified` and `Add Language Alternates` through the **Adobe AEM SEO - Page Tree Sitemap Generator** configuration panel, which can be accessed via the OSGi Configuration Manager at `http://[aem_server]:[port]/system/console/configMgr`.
This feature allows users to customize sitemaps according to specific SEO requirements, enhancing the management of page attributes directly within AEM's interface.

![Sources | Adobe Experience Manager Sitemap Generator](coveo-for-adobe/ae-sitemap-generator.png)
However, while injecting data via Java-based Sling extensions in a CMS is feasible, it may pose challenges related to implementation complexity, performance, and security.

### Enable on-demand sitemap generation

. In your browser, open the AEM SDK author instance `http://localhost:4502/system/console/bundles` page.

. In the main menu, select **OSGi** > **Configuration**.
This will display a list of all configurable services in your AEM instance.

![Sources | Adobe Experience Manager Web Console Configuration](coveo-for-adobe/ae-web-config.png)

. In the **Apache Sling Sitemap - Sitemap Generator Manager** configuration, enable **All on-demand**.

![Sources | Adobe Experience Manager All on-demand option to enable dynamic sitemap generation](coveo-for-adobe/ae-option-all-on-demand.png)

. Open the AEM Sites management interface at `http://localhost:4502/sites.html/content`.
. Select the level (for example, site, page) where you want to enable sitemap generation.

![Sources | Adobe Experience Manager Select where you want to enable sitemap generation](coveo-for-adobe/ae-select-entity.png)
. Click **Properties** in the context menu.

. Select the **Advanced** tab.
Here, you’ll find an option to **Generate Sitemap**.
Check this box to activate sitemap generation for the chosen level.

![Sources | Adobe Experience Manager Activate sitemap generation](coveo-for-adobe/ae-advanced-tab.png)

. Save and publish your changes.

Browse your sitemap to validate it's being generated as expected.
Typically, you can access the sitemap by appending `/sitemap.xml` to the base URL of your site or specific page.

### Key learnings
. Sitemaps can be generated by AEM at any level.
The choice lies with the website author.
By level, think of locale-based websites (for example, `website.com/en/`), subdirectory-level websites (for example, `website.com/en/articles/`), or country-based websites (for example, `website.com/ca/` or `website.com/gb/`).

.. An AEM website can have multiple sitemaps, which are reflected in the root-level sitemap file.
This file points to the other sitemap files under it.

. AEM internally uses the open-source library [Apache Sling](https://github.com/apache/sling-org-apache-sling-sitemap) to generate sitemaps.

## Adobe Experience Manager metadata indexing options

Depending on the way your Adobe Experience Manager website metadata is organized, one of the options below (or a combination thereof) will fulfill your needs.
The options are presented in order of performance.

### Adding Coveo metadata tags directly in your sitemap file
By default, when using the Sitemap connector, Coveo doesn't index the content of the `<meta>` tags in the `<head>` of the web pages.
This operation is costly resource-wise and may therefore impact the indexing performance.

Instead, by default, the Coveo Sitemap connector is coded to look for item metadata added directly inside the website sitemap file `<url>` elements.
The connector expects this metadata to be included in a `<coveo:metadata>` tag. A developer, therefore, needs to extend the Sitemap protocol, and to modify or generate the sitemap file with the necessary Coveo metadata tag structure and content (see [Coveo-Specific Custom Metadata](https://docs.coveo.com/en/2656#coveo-specific-custom-metadata)).

### Using JSON-LD script tags in your web pages
If you're already using JSON-LD `<script>` tags in your web pages as your metadata implementation format, Coveo has an [**Extract JSON-LD metadata**](https://docs.coveo.com/en/1967#extract-json-ld-metadata) Sitemap source option you can enable to extract that metadata.

> **Note**
>
> Coveo also provides the [`IndexJsonLdMetadata` parameter in its Web connector](https://docs.coveo.com/en/mc1f0219#indexjsonldmetadata-boolean).
> However, the Web connector also automatically parses the entire document which is a drawback from a performance standpoint.

For general information on how to enable or configure connector parameters, see [Edit a source JSON configuration](https://docs.coveo.com/en/1685/).

### Indexing web page head section metadata in a Sitemap source
As mentioned in [Adding Coveo metadata tags directly in your sitemap file](#adding-coveo-metadata-tags-directly-in-your-sitemap-file), the Sitemap connector doesn't fetch the content of the `<meta>` tags in the `<head>` of web pages by default.
However, the Sitemap connector has the [IndexHtmlMetadata](https://docs.coveo.com/en/3158#indexhtmlmetadata-boolean) parameter you can enable to do just that.
You can then create a [field and mapping](https://docs.coveo.com/en/mc1f0259/) to store that metadata.

> **Note**
>
> AEM supports advanced customization of sitemaps by [extending the Apache Sling sitemap generator](https://experienceleaguecommunities.adobe.com/t5/adobe-experience-manager-blogs/extending-the-apache-sling-sitemap-generator-for-aem-6-5-11-and/ba-p/433849) with Java code.
> This enables developers to enhance SEO by introducing custom tags, modifying update frequencies, or adding detailed metadata to sitemap entries.
> For those looking to implement such customizations, the blog "[AEM Simplified by Nikhil](https://aemsimplifiedbynikhil.wordpress.com/2021/12/01/extending-the-apache-sling-sitemap-generator-for-aem-6-5-11-and-aemaacs/)" provides a practical guide with code examples and detailed instructions.
> This feature makes AEM more flexible and effective for specific organizational SEO needs.
> 
> . Titles and descriptions significantly enhance the quality of indexed metadata.
> Ensuring that each new page created in the CMS includes a default meta `jcr:description` tag can be useful.
> Developers can utilize Sling to set up an event listener that triggers when a new page is created, automatically injecting a default description into the page’s metadata.
> This custom implementation can then be deployed by logging into the AEM OSGi console and using the **Install**/**Update** option to upload and install the bundle.

### Using a web scraping configuration
Unlike the previous options that automatically capture a site's metadata because it's presented in a standard format, setting up a [web scraping configuration](https://docs.coveo.com/en/mahe0350/) requires more work on your part.
Moreover, the web scraping configuration may vary from one page to another.
However, a web scraping configuration is more flexible than the previous metadata extraction options.

The Sitemap and Web connectors both support web scraping configurations.
Once again, you should favor the Sitemap connector for performance considerations.

To more easily create and test web scraping configurations, consider using the [Coveo Labs Web Scraper Helper](http://url.coveo.com/webscraper) Chrome extension.

## Indexing sitemap alternate URLs
The Sitemap source supports alternate language links but, by default, these links aren't parsed.
To enable this feature, set the `ParseSitemapAlternateLinks` parameter to `true` (see [ParseSitemapAlternateLinks](https://docs.coveo.com/en/3158#parsesitemapalternatelinks-boolean)).

## Create your source

As a prerequisite, you need a Coveo organization.
If you don't have one, you can start a [free trial](https://www.coveo.com/en/free-trial).

> **Note**
>
> You can also create a [test organization](https://docs.coveo.com/en/2959#definitions-and-use-cases) afterward.

To create a source

. Access the [**Sources**](https://platform.cloud.coveo.com/admin/#/orgid/content/sources/) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/content/sources/) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/content/sources/) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/content/sources/)) page in your Coveo organization.

. Click **Add source** in the upper-right corner of the screen.

. In the **Add a source of content** dialog, select the source type you've chosen to use.

. Name and configure your source.

. Click **Add and build source**.

You can now browse the content you've indexed in the [**Content Browser**](https://platform.cloud.coveo.com/admin/#/orgid/content/browser/) ([platform-ca](https://platform-ca.cloud.coveo.com/admin/#/orgid/content/browser/) | [platform-eu](https://platform-eu.cloud.coveo.com/admin/#/orgid/content/browser/) | [platform-au](https://platform-au.cloud.coveo.com/admin/#/orgid/content/browser/)).

> **Notes**
>
> * If necessary, refer to the [Add a Sitemap source](https://docs.coveo.com/en/1967#add-a-sitemap-source) or [Add a Web source](https://docs.coveo.com/en/malf0160#add-a-web-source) documentation sections for help.
> 
> * Coveo provides numerous other [native and generic connectors](https://docs.coveo.com/en/1702/) you may use to index content.