Maestro Release Notes

The SharePoint Server security provider is now able to map Claims users to Active Directory users when the “Classic” permissions mode is used.

This feature will be released progressively. Contact Coveo Support if you need to enable it in your organization before general availability.

Changed disposing logic of web drivers to limit risks of memory leakage.

Improved handling of connection errors with the local database.

Improved handling of generic exceptions in the framework to provide clearer errors in the Coveo Administration Console.

Fixed an issue where a crawler could become deadlocked after throwing a fatal exception.

This release introduces support for the DateTime data type in MySQL.

Pagination support has also been added to payload parameters and the JSON content.

The error message returned when the Crawling Module’s database is unreachable has been improved.

Tis release removes the .NET Framework version of the SharePoint Server connector. Since all customers have been migrated to use the .NET version of the connector, this will be a transparent change.

The Chromium driver has been updated to version 120.

The Coveo indexing pipeline now skips items when a failed dynamic expression would create an invalid URI. This behavior is acceptable for this source type, as database content is expected to be reliable and constant. The previous behavior was to stop the source update operation.

An API request returning an invalidUri exception now interrupts the source update operation. Ignoring this exception could’ve led to missed content, which would’ve been removed at the end of a build operation.

Maestro now runs on .NET 8. Various dependency updates are also included.

This isn’t expected to be a breaking change, provided customers follow the operating system requirements for Maestro (Windows Server 2016 or later).

Optimized web scraping by reducing the frequency of page bodies being read while scraping.

Improved the error message shown when a source configuration contains both an Authorization header and the ForceBasicAuthentication property.

Added support for the multi-match web scraping mode, where multiple web scraping configurations are applied to the same pages.

Added support for sending permissions as a string array, instead of the typical array of JSON objects.

Added stricter validation of item size to avoid scenarios where items are rejected as they go through the indexing pipeline without visibility in logs.

This validation doesn’t include the metadata added in the indexing pipeline through extensions.

Added a validation step to the configuration ensuring that the URI has a dynamic expression, to avoid scenarios where multiple items would override each other.

The DateFormat attribute now applies to all dates returned by the API.

Fixed a bug that caused the File crawler to freeze if a source operation was canceled during a long OS operation (over an hour). Additional logs are also written while listing folder contents.

Fixed a bug where the operating system’s default proxy would be used to download attachments even when the Crawling Module wasn’t configured to use one.

We improved the handling of security provider configuration exceptions. The output should be clearer when looking into the security provider activities.

Improve logging available for throttling, pagination and OAuth authentication in the REST API connector.

The connector, when using the TotalCount feature to determine when crawling ends, will now complete the current operation after retrieving 10 consecutive empty pages. Previously, when the TotalCountKey value retrieved was much higher than the amount of content available, the connector would keep retrieving empty pages for a significant period of time.

Throttling errors coming from the Coveo Platform will be retried for an extended period of time, increasing the resiliency of Crawling Module sources.

Full security provider refreshes now prioritize the processing of new identities.

We improved the error reporting framework by sending over the stack trace of internal exceptions to our reporting service. This will help us better understand and troubleshoot errors coming from Crawling Module sources.

We fixed an issue where pages with different fragments under the same base URL could render identically, despite the parameter IgnoreUrlFragment being set to false.

Handling of empty document bodies during web scraping has also been improved.

The connector now correctly handles errors caused by the wrong content type being returned by the SOAP API.

These errors can happen when SharePoint returns an error page as HTML, instead of the expected XML response. The connector now retries these calls as expected.

We improved troubleshooting logs when fetching new pages in the GenericRestConnection.

We removed the .NET 4.8 version of the Job Handler from installed binaries, as the software has been running on multiplatform .NET for over a year without need to use this as a fallback.

Features such as pagination and handling of exceptions returned in response bodies have been added with this release. Maestro is now ready for the beta release of the GraphQL API connector, which will take place in the coming weeks.

Uniqueness of names given to web scraping configurations will now be enforced at the connector level. This was already enforced in the Administration Console.

Fixed concurrency issues with metadata collection in the connector.

We fixed an issue with Maestro 1.81 that prevented Web and Sitemap source operations when JavaScript rendering was enabled. This was an unintended consequence of the cleanup of the .NET Framework 4.8 version of the Sitemap connector. The cleanup removed necessary shared binaries that are now successfully brought back to the .NET 6 version. This issue also prevented Maestro’s System Tray from working as intended.

Following this major issue of the previous version, we’ve taken the opportunity to improve our test suite and our error reporting, so that this kind of error can be caught earlier in our release process.

This release resumes "shadow mode" tests between .NET Framework and .NET 6 versions of this connector.

The number of added API calls is small and is not expected to make a notable difference in crawling performance.

This release removes the .NET Framework 4.8 version of the Sitemap connector, now that upgrade to .NET 6 is complete.

The REST API connector now has a rate limiter to manage throttling limits on different APIs. You can use this feature by setting the RequestsIntervalInMs parameter in the source JSON configuration.

We’ve fixed a regression that has been introduced in Maestro 1.76 where we increased the amount of time a source should wait and the retry count on Push API exceptions. These retry strategies were applied to all exceptions of the SDK rather than only the exceptions specific to the Push API.

We’re added monitoring of resource usage by Web crawlers. This release introduces logging the memory usage of Chromium drivers.

Chromium processes are now recycled after their current item if the memory usage passes a critical threshold. This doesn’t interrupt the operation.

Pinned the version of the requests library to fix a compatibility issue with urllib3 that caused issues with pre-push extensions.

Fixed a bug where cancelling a rescan operation could delete all documents not processed yet by this operation.

You can now add the SkipUncrawledDocumentsDeletionOnRescan parameter to the JSON configuration of Crawling Module sources to prevent any accidental content deletion in sources that aren’t expected to delete content over time.

Maestro 1.76 turned off sending monitoring alerts to the Connectivity Service by default, limiting adoption of this new feature. The feature is now turned on by default. Note that this is a separate toggle from log downloads, which remain opt-in.

We’ve revamped the release process of Maestro to limit the impact of potential future outages. Our previous approach rolled out a new version of the software to every organization at once, and forced us to release a new version whenever we wanted to fix issues.

Instead, we now have an internal history of release versions, as well as the ability to serve to Maestro different versions to update to depending on the situation. This will be used, for example, to seamlessly revert to a previous version in the case of an outage, and to update sandbox organizations first.

When on-premises connectors and security identity providers run into unexpected errors, alerts are now sent to the Coveo Connectivity Service, where we can easily review and act upon them. We aim to significantly improve our ability to respond quickly during outages thanks to this feature.

This release allows us to kickstart the upgrade of the SharePoint Server crawler to .NET 6, which is our last step towards full .NET 6 releases of Maestro. With this release, we’ll progressively enable “Shadow mode”, comparing the results of some calls between .NET Framework and .NET 6. The number of added API calls is small, and is not expected to make a notable difference in crawling performance.

We’ve tweaked how a source changes status when throttled by the PushAPI due to item counts. Retries in this scenario used to be very strict, with sources failing after only five minutes of being unable to send items. Now, they’ll only be paused on error after constant throttling over several hours.