Maestro Release Notes

This release includes the migration of the Connector State to a new JSON serialized format in the State Store. This migration is done automatically and requires no downtime.

Improved error handling when the RootSpaceRegex configuration parameter is invalid.

Fixed an issue where a redirect adding the www prefix to the starting address would lead to it being skipped during crawling.

Improved handling of trailing slashes in inclusion and exclusion rules.

The migration script provided to upgrade the embedded State Store to version 10.4 could fail on hosts that don’t have the Get-WmiObject commandlet installed. The script now uses Get-CimInstance in these instances.

Maestro’s installation wizard now offers Canada as a Coveo Platform region to pair to.

Fixed an issue where Maestro couldn’t be reinstalled on the same host after uninstallation.

Fixed an issue where Maestro could sometimes fail to kill its sub-processes during shutdown.

The source now validates that its configuration contains the user credentials required by secured sources.

Added the operation error code CONNECTOR_INVALID_CUSTOMER_CONFIGURATION, which indicates that the source configuration should be reviewed.

Trailing slashes are now omitted when comparing the URI of crawled pages.

Fixed an issue where items were not sent to the index after being inaccessible during the previous indexing operation.

Improved handling of erroneous status codes when JavaScript rendering is enabled.

The Chrome driver used by these sources has been updated to version 128.0.6613.85. This version contains important security fixes.

This release contains version 10.11 of the embedded MariaDB State Store. Binaries for previous version 10.4 are still available in the installation folder.

On new installations of Maestro version 1.96 onward, MariaDB 10.11 will be used by default for the State Store.

However, when automatically updating Maestro, the State Store isn’t automatically replaced. Coveo recommends you manually upgrade your State Store with a PowerShell script provided in Maestro’s installation folder, under /Scripts/mariadb.ps1. From an elevated PowerShell window, you can run the script using ./mariadb.ps1 -Migrate.

To confirm the script ran successfully, run ./mariadb.ps1 -Info. In the service information returned, you should see the new version of MariaDB in the Executable Path, for example "C:\Program Files\Coveo\Maestro\MariaDb\10.11.8\bin\mysqld.exe".

Coveo plans to automate the upgrade process in a future release.

Added the DoNotInherit property to disable paging inheritance.

Added support for single-use refresh tokens.

This version removes support for the PhantomJS web driver type.

Added Boolean property LogResponseBodyWhenUnsuccessful to troubleshoot REST API responses that don’t contain the expected object.

Added support for defining the user agent.

Improved Chrome Driver disposal to avoid process leaks.

Fixed an issue where ETag metadata could be skipped over instead of being pushed to the index.

Maestro’s status call now includes the MariaDB version and the operating system version for better visibility on the upgrade progress.

Fixed an issue where, if failing checksum validation, the rejected files downloaded by the Updater wouldn’t be deleted from disk.

Fixed an issue where the Updater wouldn’t log to disk.

Fixed an issue where a worker wouldn’t flush its current items when a source operation was manually paused.

Initial support for a new Connector State format. This release only performs compatibility checks. It doesn’t migrate to the new format.

Retry space API calls without description on internal server errors.

Support the upcoming new version of the Confluence plugin. This plugin will support read-only crawling of user permissions.

Fixed an issue where ChromeDrivers were spawned per starting address.

Fixed an issue where Sitemap items aren’t deleted when they should be.

More logging was added to the Pre-Push Extension Runner.

Fixed an issue caused by a new version of Pip removing support for Python 3.7.

Improved logging for the Platform SDK and the REST API crawler.

Fixed an issue where a missing plugin can cause public Confluence Server source operations to fail when using personal token authentication.

The Chrome driver used by these sources has been updated to version 125. This version contains important security fixes.

Dynamic expressions can no longer be used to populate the Uri metadata key on documents. This change was made to improve troubleshooting source operations, as it ensures this piece of metadata is reliably indexed.

URL source filters no longer apply to starting addresses. This change reflects the expected behavior since starting sitemaps aren’t indexed in the first place.

Improved source resuming logic by applying source filters before crawling site collections.

Performance has been improved when extracting HTML page content.

Improved monitoring of security provider operations.

Validation of the web scraping configuration has been added to these sources. Web scraping performance has been improved as well.

The "INDEX_DOCUMENT_LIMIT_EXCEEDED" error code has been split into more precise "TOO_MANY_DOCUMENTS_DAILY" and "DOCUMENT_LIMIT_EXCEEDED" errors.

Various dependency updates for stability and security.

The SharePoint Server security provider is now able to map Claims users to Active Directory users when the "Classic" permissions mode is used.

This feature will be released progressively. Contact Coveo Support if you need to enable it in your organization before general availability.

Changed disposing logic of web drivers to limit risks of memory leakage.

Improved handling of connection errors with the local database.

Improved handling of generic exceptions in the framework to provide clearer errors in the Coveo Administration Console.

Fixed an issue where a crawler could become deadlocked after throwing a fatal exception.

This release introduces support for the DateTime data type in MySQL.

Pagination support has also been added to payload parameters and the JSON content.

The error message returned when the Crawling Module’s database is unreachable has been improved.

This release removes the .NET Framework version of the SharePoint Server connector. Since all customers have been migrated to use the .NET version of the connector, this will be a transparent change.

The Chromium driver has been updated to version 120.

The Coveo indexing pipeline now skips items when a failed dynamic expression would create an invalid URI. This behavior is acceptable for this source type, as database content is expected to be reliable and constant. The previous behavior was to stop the source update operation.

An API request returning an invalidUri exception now interrupts the source update operation. Ignoring this exception could’ve led to missed content, which would’ve been removed at the end of a build operation.

Maestro now runs on .NET 8. Various dependency updates are also included.

This isn’t expected to be a breaking change, provided customers follow the operating system requirements for Maestro (Windows Server 2016 or later).

Optimized web scraping by reducing the frequency of page bodies being read while scraping.

Improved the error message shown when a source configuration contains both an Authorization header and the ForceBasicAuthentication property.

Added support for the multi-match web scraping mode, where multiple web scraping configurations are applied to the same pages.

Added support for sending permissions as a string array, instead of the typical array of JSON objects.

Added stricter validation of item size to avoid scenarios where items are rejected as they go through the indexing pipeline without visibility in logs.

This validation doesn’t include the metadata added in the indexing pipeline through extensions.

Added a validation step to the configuration ensuring that the URI has a dynamic expression, to avoid scenarios where multiple items would override each other.

The DateFormat attribute now applies to all dates returned by the API.

Fixed a bug that caused the File crawler to freeze if a source operation was canceled during a long OS operation (over an hour). Additional logs are also written while listing folder contents.

Fixed a bug where the operating system’s default proxy would be used to download attachments even when the Crawling Module wasn’t configured to use one.

We improved the handling of security provider configuration exceptions. The output should be clearer when looking into the security provider activities.

Improve logging available for throttling, pagination and OAuth authentication in the REST API connector.

The connector, when using the TotalCount feature to determine when crawling ends, will now complete the current operation after retrieving 10 consecutive empty pages. Previously, when the TotalCountKey value retrieved was much higher than the amount of content available, the connector would keep retrieving empty pages for a significant period of time.

Throttling errors coming from the Coveo Platform will be retried for an extended period of time, increasing the resiliency of Crawling Module sources.

Full security provider refreshes now prioritize the processing of new identities.

We improved the error reporting framework by sending over the stack trace of internal exceptions to our reporting service. This will help us better understand and troubleshoot errors coming from Crawling Module sources.

We fixed an issue where pages with different fragments under the same base URL could render identically, despite the parameter IgnoreUrlFragment being set to false.

Handling of empty document bodies during web scraping has also been improved.

The connector now correctly handles errors caused by the wrong content type being returned by the SOAP API.

These errors can happen when SharePoint returns an error page as HTML, instead of the expected XML response. The connector now retries these calls as expected.

We improved troubleshooting logs when fetching new pages in the GenericRestConnection.

We removed the .NET 4.8 version of the Job Handler from installed binaries, as the software has been running on multiplatform .NET for over a year without need to use this as a fallback.

Features such as pagination and handling of exceptions returned in response bodies have been added with this release. Maestro is now ready for the beta release of the GraphQL API connector, which will take place in the coming weeks.

Uniqueness of names given to web scraping configurations will now be enforced at the connector level. This was already enforced in the Administration Console.

Fixed concurrency issues with metadata collection in the connector.

We fixed an issue with Maestro 1.81 that prevented Web and Sitemap source operations when JavaScript rendering was enabled. This was an unintended consequence of the cleanup of the .NET Framework 4.8 version of the Sitemap connector. The cleanup removed necessary shared binaries that are now successfully brought back to the .NET 6 version. This issue also prevented Maestro’s System Tray from working as intended.

Following this major issue of the previous version, we’ve taken the opportunity to improve our test suite and our error reporting, so that this kind of error can be caught earlier in our release process.

This release resumes "shadow mode" tests between .NET Framework and .NET 6 versions of this connector.

The number of added API calls is small and is not expected to make a notable difference in crawling performance.

This release removes the .NET Framework 4.8 version of the Sitemap connector, now that upgrade to .NET 6 is complete.

The REST API connector now has a rate limiter to manage throttling limits on different APIs. You can use this feature by setting the RequestsIntervalInMs parameter in the source JSON configuration.

We’ve fixed a regression that has been introduced in Maestro 1.76 where we increased the amount of time a source should wait and the retry count on Push API exceptions. These retry strategies were applied to all exceptions of the SDK rather than only the exceptions specific to the Push API.

We’re added monitoring of resource usage by Web crawlers. This release introduces logging the memory usage of Chromium drivers.

Chromium processes are now recycled after their current item if the memory usage passes a critical threshold. This doesn’t interrupt the operation.

Pinned the version of the requests library to fix a compatibility issue with urllib3 that caused issues with pre-push extensions.

Fixed a bug where cancelling a rescan operation could delete all documents not processed yet by this operation.

You can now add the SkipUncrawledDocumentsDeletionOnRescan parameter to the JSON configuration of Crawling Module sources to prevent any accidental content deletion in sources that aren’t expected to delete content over time.

Maestro 1.76 turned off sending monitoring alerts to the Connectivity Service by default, limiting adoption of this new feature. The feature is now turned on by default. Note that this is a separate toggle from log downloads, which remain opt-in.

We’ve revamped the release process of Maestro to limit the impact of potential future outages. Our previous approach rolled out a new version of the software to every organization at once, and forced us to release a new version whenever we wanted to fix issues.

Instead, we now have an internal history of release versions, as well as the ability to serve to Maestro different versions to update to depending on the situation. This will be used, for example, to seamlessly revert to a previous version in the case of an outage, and to update sandbox organizations first.

When on-premises connectors and security identity providers run into unexpected errors, alerts are now sent to the Coveo Connectivity Service, where we can easily review and act upon them. We aim to significantly improve our ability to respond quickly during outages thanks to this feature.

This release allows us to kickstart the upgrade of the SharePoint Server crawler to .NET 6, which is our last step towards full .NET 6 releases of Maestro. With this release, we’ll progressively enable "Shadow mode", comparing the results of some calls between .NET Framework and .NET 6. The number of added API calls is small, and is not expected to make a notable difference in crawling performance.

We’ve tweaked how a source changes status when throttled by the PushAPI due to item counts. Retries in this scenario used to be very strict, with sources failing after only five minutes of being unable to send items. Now, they’ll only be paused on error after constant throttling over several hours.