Maestro Release Notes

This page summarizes the changes delivered with every update of Maestro, the software that manages and monitors your local workers, State Store, etc. Updates are automatic to allow you to benefit from the latest features and bug fixes. See Updates for details.

September 7, 2023 Update: Maestro 1.84

What’s New

General

We improved the error reporting framework by sending over the stack trace of internal exceptions to our reporting service. This will help us better understand and troubleshoot errors coming from Crawling Module sources.

Other Changes

Web and Sitemap

We fixed an issue where pages with different fragments under the same base URL could render identically, despite the parameter IgnoreUrlFragment being set to false.

Handling of empty document bodies during web scraping has also been improved.

SharePoint Server

The connector now correctly handles errors caused by the wrong content type being returned by the SOAP API.

These errors can happen when SharePoint returns an error page as HTML, instead of the expected XML response. The connector now retries these calls as expected.

Generic Rest

We improved troubleshooting logs when fetching new pages in the GenericRestConnection.

General

We removed the .NET 4.8 version of the Job Handler from installed binaries, as the software has been running on multiplatform .NET for over a year without need to use this as a fallback.

July 26, 2023 Update: Maestro 1.83

What’s New

GraphQL

Features such as pagination and handling of exceptions returned in response bodies have been added with this release. Maestro is now ready for the beta release of the GraphQL API connector, which will take place in the coming weeks.

Other Changes

Web and Sitemap

Uniqueness of names given to web scraping configurations will now be enforced at the connector level. This was already enforced in the Administration Console.

Database

Fixed concurrency issues with metadata collection in the connector.

June 21, 2023 Update: Maestro 1.82

This is a small release that mainly fixes a major issue with Maestro 1.81.

What’s New

Web

We fixed an issue with Maestro 1.81 that prevented Web and Sitemap source operations when JavaScript rendering was enabled. This was an unintended consequence of the cleanup of the .NET Framework 4.8 version of the Sitemap connector. The cleanup removed necessary shared binaries that are now successfully brought back to the .NET 6 version. This issue also prevented Maestro’s System Tray from working as intended.

Following this major issue of the previous version, we’ve taken the opportunity to improve our test suite and our error reporting, so that this kind of error can be caught earlier in our release process.

Other Changes

SharePoint Server

This release resumes "shadow mode" tests between .NET Framework and .NET 6 versions of this connector.

The number of added API calls is small and is not expected to make a notable difference in crawling performance.

May 31, 2023 Update: Maestro 1.81

What’s New

Web

This release removes the .NET Framework 4.8 version of the Sitemap connector, now that upgrade to .NET 6 is complete.

REST API

The REST API connector now has a rate limiter to manage throttling limits on different APIs. You can use this feature by setting the RequestsIntervalInMs parameter in the source JSON configuration.

Other Changes

General

We’ve fixed a regression that has been introduced in Maestro 1.76 where we increased the amount of time a source should wait and the retry count on Push API exceptions. These retry strategies were applied to all exceptions of the SDK rather than only the exceptions specific to the Push API.

May 18, 2023 Update: Maestro 1.78

What’s New

Web

We’re added monitoring of resource usage by Web crawlers. This release introduces logging the memory usage of Chromium drivers.

Chromium processes are now recycled after their current item if the memory usage passes a critical threshold. This doesn’t interrupt the operation.

General

Pre-push extensions

Pinned the version of the requests library to fix a compatibility issue with urllib3 that caused issues with pre-push extensions.

Rescans

Fixed a bug where cancelling a rescan operation could delete all documents not processed yet by this operation.

May 4, 2023 Update: Maestro 1.77

What’s New

Crawling

You can now add the SkipUncrawledDocumentsDeletionOnRescan parameter to the JSON configuration of Crawling Module sources to prevent any accidental content deletion in sources that aren’t expected to delete content over time.

Other Changes

Maestro

Maestro 1.76 turned off sending monitoring alerts to the Connectivity Service by default, limiting adoption of this new feature. The feature is now turned on by default. Note that this is a separate toggle from log downloads, which remain opt-in.

April 21, 2023 Update: Maestro 1.76

What’s New

This release marks a significant milestone for Maestro, as it includes two key improvements to the uptime and reliability of the software.

Gradual Release Process

We’ve revamped the release process of Maestro to limit the impact of potential future outages. Our previous approach rolled out a new version of the software to every organization at once, and forced us to release a new version whenever we wanted to fix issues.

Instead, we now have an internal history of release versions, as well as the ability to serve to Maestro different versions to update to depending on the situation. This will be used, for example, to seamlessly revert to a previous version in the case of an outage, and to update sandbox organizations first.

Improved Monitoring

When on-premises connectors and security identity providers run into unexpected errors, alerts are now sent to the Coveo Connectivity Service, where we can easily review and act upon them. We aim to significantly improve our ability to respond quickly during outages thanks to this feature.

Other Changes

SharePoint Server

This release allows us to kickstart the upgrade of the SharePoint Server crawler to .NET 6, which is our last step towards full .NET 6 releases of Maestro. With this release, we’ll progressively enable “Shadow mode”, comparing the results of some calls between .NET Framework and .NET 6. The number of added API calls is small, and is not expected to make a notable difference in crawling performance.

General

We’ve tweaked how a source changes status when throttled by the PushAPI due to item counts. Retries in this scenario used to be very strict, with sources failing after only five minutes of being unable to send items. Now, they’ll only be paused on error after constant throttling over several hours.

Web and Sitemap

  • Improved cleanup of web drivers to lower memory usage.

  • Improved logging during authentication issues.

  • Fixed an issue where multiple threads would attempt to add the same credentials to the credential cache, leading to an Argument exception.

February 7, 2023 Update: Maestro 1.74

What’s New

  • Added support for Active Directory authentication to the .NET 6 version of the Database connector, enabling us to update the framework of this connector.

  • Many features and fixes that were included in 1.72 had to be reverted in 1.73 due to a bug that caused crawlers to refuse to connect to local addresses, despite it being a valid use case for Crawling Module sources. The following changes are brought back, now that this behavior has been corrected:

    • Fixes for NTLM/Kerberos/Negotiate authentications in the Sitemap crawler.

    • Updating the Chromium driver to 108.0.5359.72.

    • The AdditionalHeaders property in Web and Sitemap source configuration no longer accepts authorization headers.

January 20, 2023 Update: Maestro 1.73

What’s New

Reverted some of the changes of version 1.72. The previous release had an issue where some sources wouldn’t crawl local addresses. This was undesirable and breaking behavior for Crawling Module sources.

January 19, 2023 Update: Maestro 1.72

What’s New

Maestro is now running on .NET 7. This isn’t expected to be a breaking change, provided customers are following the operating system requirements for Maestro (Windows Server 2016 and above).

Other Changes

  • Fix for NTLM/Kerberos/Negotiate authentications. The ForceBasicAuthorizationHeader parameter is now respected again. Previously, the header was automatically added regardless of the parameter’s value.

  • The AdditionalHeaders property in the Web and Sitemap source configuration no longer accepts authorization headers.

  • Updated the Chromium and Chrome driver to 108.0.5359.72 for the Sitemap and Web connectors.

December 8, 2022 Update: Maestro 1.71

What’s New

The EnableLogRequests parameter has been improved!

  • Maestro now handles the case where no logs can be found for a requested operation. It now returns a status to the Coveo platform rather than returning an empty archive.

  • Polling for log requests is now done every 30 seconds, which means that, in the worst case, a user will wait 30 seconds (plus the upload duration) after requesting logs remotely.

This update is the last one required for the general availability of the log access feature. We aim for a public release of this feature by December 16.

Other Changes

  • Fixed an issue in the Database connector that could sometimes delete indexed content.

  • Fixed an issue with resumes of the REST API connector where total counts weren’t properly set.

  • Fixed an issue in the REST API connector where headers specified to the Service level didn’t apply to the subqueries.

  • Improved handling of disposed web responses in the Web, Sitemap, and RSS connectors.