Maestro Release Notes
Maestro Release Notes
- September 30, 2024, Update: Maestro 1.101
- August 27, 2024, Update: Maestro 1.96
- July 4, 2024 update: Maestro 1.95
- June 6, 2024 update: Maestro 1.94
- June 4, 2024 update: Maestro 1.93
- April 17, 2024 update: Maestro 1.90
- February 29, 2024 update: Maestro 1.89
- January 31, 2024 update: Maestro 1.88
- January 17, 2024 update: Maestro 1.87
- December 14, 2023 update: Maestro 1.86
- October 19, 2023 update: Maestro 1.85
- September 7, 2023 update: Maestro 1.84
- July 26, 2023 update: Maestro 1.83
- June 21, 2023 update: Maestro 1.82
- May 31, 2023 update: Maestro 1.81
- May 18, 2023 update: Maestro 1.78
- May 4, 2023 update: Maestro 1.77
- April 21, 2023 update: Maestro 1.76
- February 7, 2023 update: Maestro 1.74
- January 20, 2023 update: Maestro 1.73
- January 19, 2023 update: Maestro 1.72
- December 8, 2022 update: Maestro 1.71
This page summarizes the changes delivered with every update of Maestro, the software that manages and monitors your local workers, State Store, etc. Updates are automatic to allow you to benefit from the latest features and bug fixes. See Updates for details.
September 30, 2024, Update: Maestro 1.101
What’s New
General
The migration script provided to upgrade the embedded State Store to version 10.4 could fail on hosts that don’t have the Get-WmiObject commandlet installed. The script now uses Get-CimInstance in these instances.
Maestro’s installation wizard now offers Canada as a Coveo Platform region to pair to.
Fixed an issue where Maestro couldn’t be reinstalled on the same host after uninstallation.
Fixed an issue where Maestro could sometimes fail to kill its sub-processes during shutdown.
Jira
The source now validates that its configuration contains the user credentials required by secured sources.
Confluence
Added the operation error code CONNECTOR_INVALID_CUSTOMER_CONFIGURATION
, which indicates that the source configuration should be reviewed.
Web
Trailing slashes are now omitted when comparing the URI of crawled pages.
Sitemap
Fixed an issue where items were not sent to the index after being inaccessible during the previous indexing operation.
Web and Sitemap
Improved handling of erroneous status codes when JavaScript rendering is enabled.
The Chrome driver used by these sources has been updated to version 128.0.6613.85. This version contains important security fixes.
August 27, 2024, Update: Maestro 1.96
What’s New
Action required
This release contains changes to the checksum validation in the automatic updater. To ensure continued automatic updates, update to Maestro 1.96 or a newer version by March 1, 2025. If you don’t update by this date, automatic updates will no longer be available. Coveo will contact customers who haven’t migrated to version 1.96 by February 2025. |
State Store
This release contains version 10.11 of the embedded MariaDB State Store. Binaries for previous version 10.4 are still available in the installation folder.
On new installations of Maestro version 1.96 onward, MariaDB 10.11 will be used by default for the State Store.
However, when automatically updating Maestro, the State Store isn’t automatically replaced.
Coveo recommends you manually upgrade your State Store with a PowerShell script provided in Maestro’s installation folder, under /Scripts/mariadb.ps1
.
From an elevated PowerShell window, you can run the script using ./mariadb.ps1 -Migrate
.
To confirm the script ran successfully, run ./mariadb.ps1 -Info
.
In the service information returned, you should see the new version of MariaDB in the Executable Path
, for example "C:\Program Files\Coveo\Maestro\MariaDb\10.11.8\bin\mysqld.exe"
.
Coveo plans to automate the upgrade process in a future release.
REST API
Added the DoNotInherit
property to disable paging inheritance.
Added support for single-use refresh tokens.
Web and Sitemap
This version removes support for the PhantomJS web driver type.
Other Changes
REST API
Added Boolean property LogResponseBodyWhenUnsuccessful
to troubleshoot REST API responses that don’t contain the expected object.
RSS
Added support for defining the user agent.
Web and Sitemap
Improved Chrome Driver disposal to avoid process leaks.
Fixed an issue where ETag metadata could be skipped over instead of being pushed to the index.
General
Maestro’s status call now includes the MariaDB version and the operating system version for better visibility on the upgrade progress.
Fixed an issue where, if failing checksum validation, the rejected files downloaded by the Updater wouldn’t be deleted from disk.
Fixed an issue where the Updater wouldn’t log to disk.
Fixed an issue where a worker wouldn’t flush its current items when a source operation was manually paused.
Initial support for a new Connector State format. This release only performs compatibility checks. It doesn’t migrate to the new format.
July 4, 2024 update: Maestro 1.95
What’s New
Confluence Server
Retry space API calls without description on internal server errors.
Support the upcoming new version of the Confluence plugin. This plugin will support read-only crawling of user permissions.
Web and Sitemap
Fixed an issue where ChromeDrivers were spawned per starting address.
Fixed an issue where Sitemap items aren’t deleted when they should be.
General
More logging was added to the Pre-Push Extension Runner.
Fixed an issue caused by a new version of Pip removing support for Python 3.7.
Improved logging for the Platform SDK and the REST API crawler.
June 6, 2024 update: Maestro 1.94
What’s New
Confluence Server
Fixed an issue where a missing plugin can cause public Confluence Server source operations to fail when using personal token authentication.
Web and Sitemap
The Chrome driver used by these sources has been updated to version 125. This version contains important security fixes.
June 4, 2024 update: Maestro 1.93
What’s New
Database
Dynamic expressions can no longer be used to populate the Uri
metadata key on documents.
This change was made to improve troubleshooting source operations, as it ensures this piece of metadata is reliably indexed.
Sitemap
URL source filters no longer apply to starting addresses. This change reflects the expected behavior since starting sitemaps aren’t indexed in the first place.
SharePoint Server
Improved source resuming logic by applying source filters before crawling site collections.
Other Changes
Web and Sitemap
Performance has been improved when extracting HTML page content.
General
Improved monitoring of security provider operations.
April 17, 2024 update: Maestro 1.90
What’s New
Web and Sitemap
Validation of the web scraping configuration has been added to these sources. Web scraping performance has been improved as well.
General
The "INDEX_DOCUMENT_LIMIT_EXCEEDED" error code has been split into more precise "TOO_MANY_DOCUMENTS_DAILY" and "DOCUMENT_LIMIT_EXCEEDED" errors.
Various dependency updates for stability and security.
February 29, 2024 update: Maestro 1.89
What’s New
SharePoint Server
The SharePoint Server security provider is now able to map Claims users to Active Directory users when the “Classic” permissions mode is used.
This feature will be released progressively. Contact Coveo Support if you need to enable it in your organization before general availability.
Other Changes
Web
Changed disposing logic of web drivers to limit risks of memory leakage.
General
Improved handling of connection errors with the local database.
Improved handling of generic exceptions in the framework to provide clearer errors in the Coveo Administration Console.
Fixed an issue where a crawler could become deadlocked after throwing a fatal exception.
January 31, 2024 update: Maestro 1.88
What’s New
General
This release introduces support for the DateTime data type in MySQL.
Pagination support has also been added to payload parameters and the JSON content.
The error message returned when the Crawling Module’s database is unreachable has been improved.
Other Changes
SharePoint Server
This release removes the .NET Framework version of the SharePoint Server connector. Since all customers have been migrated to use the .NET version of the connector, this will be a transparent change.
Web and Sitemap
The Chromium driver has been updated to version 120.
Database
The Coveo indexing pipeline now skips items when a failed dynamic expression would create an invalid URI. This behavior is acceptable for this source type, as database content is expected to be reliable and constant. The previous behavior was to stop the source update operation.
REST API
An API request returning an invalidUri
exception now interrupts the source update operation.
Ignoring this exception could’ve led to missed content, which would’ve been removed at the end of a build operation.
January 17, 2024 update: Maestro 1.87
What’s New
General
Maestro now runs on .NET 8. Various dependency updates are also included.
This isn’t expected to be a breaking change, provided customers follow the operating system requirements for Maestro (Windows Server 2016 or later).
Other Changes
Web and Sitemap
Optimized web scraping by reducing the frequency of page bodies being read while scraping.
REST API
Improved the error message shown when a source configuration contains both an Authorization
header and the ForceBasicAuthentication
property.
December 14, 2023 update: Maestro 1.86
What’s New
Web and Sitemap
Added support for the multi-match web scraping mode, where multiple web scraping configurations are applied to the same pages.
REST API
Added support for sending permissions as a string array, instead of the typical array of JSON objects.
Other Changes
General
Added stricter validation of item size to avoid scenarios where items are rejected as they go through the indexing pipeline without visibility in logs.
This validation doesn’t include the metadata added in the indexing pipeline through extensions.
REST API and Database
Added a validation step to the configuration ensuring that the URI has a dynamic expression, to avoid scenarios where multiple items would override each other.
REST API
The DateFormat attribute now applies to all dates returned by the API.
File
Fixed a bug that caused the File crawler to freeze if a source operation was canceled during a long OS operation (over an hour). Additional logs are also written while listing folder contents.
SharePoint Server
Fixed a bug where the operating system’s default proxy would be used to download attachments even when the Crawling Module wasn’t configured to use one.
October 19, 2023 update: Maestro 1.85
What’s New
Active Directory and Database
We improved the handling of security provider configuration exceptions. The output should be clearer when looking into the security provider activities.
REST API
Improve logging available for throttling, pagination and OAuth authentication in the REST API connector.
The connector, when using the TotalCount feature to determine when crawling ends, will now complete the current operation after retrieving 10 consecutive empty pages. Previously, when the TotalCountKey value retrieved was much higher than the amount of content available, the connector would keep retrieving empty pages for a significant period of time.
General
Throttling errors coming from the Coveo Platform will be retried for an extended period of time, increasing the resiliency of Crawling Module sources.
Full security provider refreshes now prioritize the processing of new identities.
September 7, 2023 update: Maestro 1.84
What’s New
General
We improved the error reporting framework by sending over the stack trace of internal exceptions to our reporting service. This will help us better understand and troubleshoot errors coming from Crawling Module sources.
Other Changes
Web and Sitemap
We fixed an issue where pages with different fragments under the same base URL could render identically, despite the parameter IgnoreUrlFragment
being set to false
.
Handling of empty document bodies during web scraping has also been improved.
SharePoint Server
The connector now correctly handles errors caused by the wrong content type being returned by the SOAP API.
These errors can happen when SharePoint returns an error page as HTML, instead of the expected XML response. The connector now retries these calls as expected.
Generic Rest
We improved troubleshooting logs when fetching new pages in the GenericRestConnection
.
General
We removed the .NET 4.8 version of the Job Handler from installed binaries, as the software has been running on multiplatform .NET for over a year without need to use this as a fallback.
July 26, 2023 update: Maestro 1.83
What’s New
GraphQL
Features such as pagination and handling of exceptions returned in response bodies have been added with this release. Maestro is now ready for the beta release of the GraphQL API connector, which will take place in the coming weeks.
Other Changes
Web and Sitemap
Uniqueness of names given to web scraping configurations will now be enforced at the connector level. This was already enforced in the Administration Console.
Database
Fixed concurrency issues with metadata collection in the connector.
June 21, 2023 update: Maestro 1.82
This is a small release that mainly fixes a major issue with Maestro 1.81.
What’s New
Web
We fixed an issue with Maestro 1.81 that prevented Web and Sitemap source operations when JavaScript rendering was enabled. This was an unintended consequence of the cleanup of the .NET Framework 4.8 version of the Sitemap connector. The cleanup removed necessary shared binaries that are now successfully brought back to the .NET 6 version. This issue also prevented Maestro’s System Tray from working as intended.
Following this major issue of the previous version, we’ve taken the opportunity to improve our test suite and our error reporting, so that this kind of error can be caught earlier in our release process.
Other Changes
SharePoint Server
This release resumes "shadow mode" tests between .NET Framework and .NET 6 versions of this connector.
The number of added API calls is small and is not expected to make a notable difference in crawling performance.
May 31, 2023 update: Maestro 1.81
What’s New
Web
This release removes the .NET Framework 4.8 version of the Sitemap connector, now that upgrade to .NET 6 is complete.
REST API
The REST API connector now has a rate limiter to manage throttling limits on different APIs.
You can use this feature by setting the RequestsIntervalInMs
parameter in the source JSON configuration.
Other Changes
General
We’ve fixed a regression that has been introduced in Maestro 1.76 where we increased the amount of time a source should wait and the retry count on Push API exceptions. These retry strategies were applied to all exceptions of the SDK rather than only the exceptions specific to the Push API.
May 18, 2023 update: Maestro 1.78
What’s New
Web
We’re added monitoring of resource usage by Web crawlers. This release introduces logging the memory usage of Chromium drivers.
Chromium processes are now recycled after their current item if the memory usage passes a critical threshold. This doesn’t interrupt the operation.
General
Pre-push extensions
Pinned the version of the requests library to fix a compatibility issue with urllib3 that caused issues with pre-push extensions.
Rescans
Fixed a bug where cancelling a rescan operation could delete all documents not processed yet by this operation.
May 4, 2023 update: Maestro 1.77
What’s New
Crawling
You can now add the SkipUncrawledDocumentsDeletionOnRescan
parameter to the JSON configuration of Crawling Module sources to prevent any accidental content deletion in sources that aren’t expected to delete content over time.
Other Changes
Maestro
Maestro 1.76 turned off sending monitoring alerts to the Connectivity Service by default, limiting adoption of this new feature. The feature is now turned on by default. Note that this is a separate toggle from log downloads, which remain opt-in.
April 21, 2023 update: Maestro 1.76
What’s New
This release marks a significant milestone for Maestro, as it includes two key improvements to the uptime and reliability of the software.
Gradual Release Process
We’ve revamped the release process of Maestro to limit the impact of potential future outages. Our previous approach rolled out a new version of the software to every organization at once, and forced us to release a new version whenever we wanted to fix issues.
Instead, we now have an internal history of release versions, as well as the ability to serve to Maestro different versions to update to depending on the situation. This will be used, for example, to seamlessly revert to a previous version in the case of an outage, and to update sandbox organizations first.
Improved Monitoring
When on-premises connectors and security identity providers run into unexpected errors, alerts are now sent to the Coveo Connectivity Service, where we can easily review and act upon them. We aim to significantly improve our ability to respond quickly during outages thanks to this feature.
Other Changes
SharePoint Server
This release allows us to kickstart the upgrade of the SharePoint Server crawler to .NET 6, which is our last step towards full .NET 6 releases of Maestro. With this release, we’ll progressively enable “Shadow mode”, comparing the results of some calls between .NET Framework and .NET 6. The number of added API calls is small, and is not expected to make a notable difference in crawling performance.
General
We’ve tweaked how a source changes status when throttled by the PushAPI due to item counts. Retries in this scenario used to be very strict, with sources failing after only five minutes of being unable to send items. Now, they’ll only be paused on error after constant throttling over several hours.
Web and Sitemap
-
Improved cleanup of web drivers to lower memory usage.
-
Improved logging during authentication issues.
-
Fixed an issue where multiple threads would attempt to add the same credentials to the credential cache, leading to an Argument exception.
February 7, 2023 update: Maestro 1.74
What’s New
-
Added support for Active Directory authentication to the .NET 6 version of the Database connector, enabling us to update the framework of this connector.
-
Many features and fixes that were included in 1.72 had to be reverted in 1.73 due to a bug that caused crawlers to refuse to connect to local addresses, despite it being a valid use case for Crawling Module sources. The following changes are brought back, now that this behavior has been corrected:
-
Fixes for NTLM/Kerberos/Negotiate authentications in the Sitemap crawler.
-
Updating the Chromium driver to 108.0.5359.72.
-
The AdditionalHeaders property in Web and Sitemap source configuration no longer accepts authorization headers.
-
January 20, 2023 update: Maestro 1.73
What’s New
Reverted some of the changes of version 1.72. The previous release had an issue where some sources wouldn’t crawl local addresses. This was undesirable and breaking behavior for Crawling Module sources.
January 19, 2023 update: Maestro 1.72
What’s New
Maestro is now running on .NET 7. This isn’t expected to be a breaking change, provided customers are following the operating system requirements for Maestro (Windows Server 2016 and above).
Other Changes
-
Fix for NTLM/Kerberos/Negotiate authentications. The
ForceBasicAuthorizationHeader
parameter is now respected again. Previously, the header was automatically added regardless of the parameter’s value. -
The
AdditionalHeaders
property in the Web and Sitemap source configuration no longer accepts authorization headers. -
Updated the Chromium and Chrome driver to 108.0.5359.72 for the Sitemap and Web connectors.
December 8, 2022 update: Maestro 1.71
What’s New
The EnableLogRequests parameter has been improved!
-
Maestro now handles the case where no logs can be found for a requested operation. It now returns a status to the Coveo platform rather than returning an empty archive.
-
Polling for log requests is now done every 30 seconds, which means that, in the worst case, a user will wait 30 seconds (plus the upload duration) after requesting logs remotely.
This update is the last one required for the general availability of the log access feature. We aim for a public release of this feature by December 16.
Other Changes
-
Fixed an issue in the Database connector that could sometimes delete indexed content.
-
Fixed an issue with resumes of the REST API connector where total counts weren’t properly set.
-
Fixed an issue in the REST API connector where headers specified to the Service level didn’t apply to the subqueries.
-
Improved handling of disposed web responses in the Web, Sitemap, and RSS connectors.