Maestro release notes of 2024

This release includes a modern .NET implementation of the Worker Service in charge of worker configuration and communication, which is named NodeAgent Core. This release includes both the previous and new Worker Service, but will use the new one by default.

A temporary option has been added to Maestro’s settings file. This option lets you roll back the Worker Service while troubleshooting a source operation issue, if needed. To do so:

This release removes the BatchSize parameter from the connector. No customer impact is expected from this change.

Added logging when the crawler encounters long paths.

Added error code FILE_CRAWLER_NOT_ENOUGH_DISK_SPACE to display when running out of disk space.

Improved retry logic when encountering locked files.

These connectors no longer index comments and attachments by default when the corresponding attributes aren’t specified in the source configuration. These attributes are ShallIndexComments and ShallIndexAttachments for Jira Software, IndexComments and IndexAttachments for Confluence.

As these values are set at source creation, no customer impact is expected following this change.

This release includes the migration of the Connector State to a new JSON serialized format in the State Store. This migration is done automatically and requires no downtime.

Improved error handling when the RootSpaceRegex configuration parameter is invalid.

Fixed an issue where a redirect adding the www prefix to the starting address would lead to it being skipped during crawling.

Improved handling of trailing slashes in inclusion and exclusion rules.

The migration script provided to upgrade the embedded State Store to version 10.4 could fail on hosts that don’t have the Get-WmiObject commandlet installed. The script now uses Get-CimInstance in these instances.

Maestro’s installation wizard now offers Canada as a Coveo Platform region to pair to.

Fixed an issue where Maestro couldn’t be reinstalled on the same host after uninstallation.

Fixed an issue where Maestro could sometimes fail to kill its sub-processes during shutdown.

The source now validates that its configuration contains the user credentials required by secured sources.

Added the operation error code CONNECTOR_INVALID_CUSTOMER_CONFIGURATION, which indicates that the source configuration should be reviewed.

Trailing slashes are now omitted when comparing the URI of crawled pages.

Fixed an issue where items were not sent to the index after being inaccessible during the previous indexing operation.

Improved handling of erroneous status codes when JavaScript rendering is enabled.

The Chrome driver used by these sources has been updated to version 128.0.6613.85. This version contains important security fixes.

This release contains version 10.11 of the embedded MariaDB State Store. Binaries for previous version 10.4 are still available in the installation folder.

On new installations of Maestro version 1.96 onward, MariaDB 10.11 will be used by default for the State Store.

However, when automatically updating Maestro, the State Store isn’t automatically replaced. Coveo recommends you manually upgrade your State Store with a PowerShell script provided in Maestro’s installation folder, under /Scripts/mariadb.ps1. From an elevated PowerShell window, you can run the script using ./mariadb.ps1 -Migrate.

To confirm the script ran successfully, run ./mariadb.ps1 -Info. In the service information returned, you should see the new version of MariaDB in the Executable Path, for example "C:\Program Files\Coveo\Maestro\MariaDb\10.11.8\bin\mysqld.exe".

Coveo plans to automate the upgrade process in a future release.

Added the DoNotInherit property to disable paging inheritance.

Added support for single-use refresh tokens.

This version removes support for the PhantomJS web driver type.

Added Boolean property LogResponseBodyWhenUnsuccessful to troubleshoot REST API responses that don’t contain the expected object.

Added support for defining the user agent.

Improved Chrome Driver disposal to avoid process leaks.

Fixed an issue where ETag metadata could be skipped over instead of being pushed to the index.

Maestro’s status call now includes the MariaDB version and the operating system version for better visibility on the upgrade progress.

Fixed an issue where, if failing checksum validation, the rejected files downloaded by the Updater wouldn’t be deleted from disk.

Fixed an issue where the Updater wouldn’t log to disk.

Fixed an issue where a worker wouldn’t flush its current items when a source operation was manually paused.

Initial support for a new Connector State format. This release only performs compatibility checks. It doesn’t migrate to the new format.

Retry space API calls without description on internal server errors.

Support the upcoming new version of the Confluence plugin. This plugin will support read-only crawling of user permissions.

Fixed an issue where ChromeDrivers were spawned per starting address.

Fixed an issue where Sitemap items aren’t deleted when they should be.

More logging was added to the Pre-Push Extension Runner.

Fixed an issue caused by a new version of Pip removing support for Python 3.7.

Improved logging for the Platform SDK and the REST API crawler.

Fixed an issue where a missing plugin can cause public Confluence Server source operations to fail when using personal token authentication.

The Chrome driver used by these sources has been updated to version 125. This version contains important security fixes.

Dynamic expressions can no longer be used to populate the Uri metadata key on documents. This change was made to improve troubleshooting source operations, as it ensures this piece of metadata is reliably indexed.

URL source filters no longer apply to starting addresses. This change reflects the expected behavior since starting sitemaps aren’t indexed in the first place.

Improved source resuming logic by applying source filters before crawling site collections.

Performance has been improved when extracting HTML page content.

Improved monitoring of security provider operations.

Validation of the web scraping configuration has been added to these sources. Web scraping performance has been improved as well.

The "INDEX_DOCUMENT_LIMIT_EXCEEDED" error code has been split into more precise "TOO_MANY_DOCUMENTS_DAILY" and "DOCUMENT_LIMIT_EXCEEDED" errors.

Various dependency updates for stability and security.

The SharePoint Server security provider is now able to map Claims users to Active Directory users when the "Classic" permissions mode is used.

This feature will be released progressively. Contact Coveo Support if you need to enable it in your organization before general availability.

Changed disposing logic of web drivers to limit risks of memory leakage.

Improved handling of connection errors with the local database.

Improved handling of generic exceptions in the framework to provide clearer errors in the Coveo Administration Console.

Fixed an issue where a crawler could become deadlocked after throwing a fatal exception.

This release introduces support for the DateTime data type in MySQL.

Pagination support has also been added to payload parameters and the JSON content.

The error message returned when the Crawling Module’s database is unreachable has been improved.

This release removes the .NET Framework version of the SharePoint Server connector. Since all customers have been migrated to use the .NET version of the connector, this will be a transparent change.

The Chromium driver has been updated to version 120.

The Coveo indexing pipeline now skips items when a failed dynamic expression would create an invalid URI. This behavior is acceptable for this source type, as database content is expected to be reliable and constant. The previous behavior was to stop the source update operation.

An API request returning an invalidUri exception now interrupts the source update operation. Ignoring this exception could’ve led to missed content, which would’ve been removed at the end of a build operation.

Maestro now runs on .NET 8. Various dependency updates are also included.

This isn’t expected to be a breaking change, provided customers follow the operating system requirements for Maestro (Windows Server 2016 or later).

Optimized web scraping by reducing the frequency of page bodies being read while scraping.

Improved the error message shown when a source configuration contains both an Authorization header and the ForceBasicAuthentication property.