Maestro release notes

Improved handling of DAILY_DOCUMENT_LIMIT_EXCEEDED error.

Removed support for error fields that aren’t at the top level of the API response to better align with the standard. Nested error fields in GraphQL API responses will be ignored during crawling. The operation will complete successfully, potentially hiding previously visible issues when crawling non-standard APIs.

Added support for JavaScript-rendered sitemaps.

Fix for a deadlock issue that could occur in long-running sources.

Fixed Database connector ending in a fatal exception when an operation is canceled.

Improved error handling.

This release introduces the new Pre-Push Extension Runner in shadow mode. Coveo will ensure the new implementation produces identical results to the legacy version. If you observe any errors or degraded performance, please reach out to your Coveo representative.

Clarified the error code returned when an invalid execution condition is used in the configuration.

Improved error handling regarding the plugin being missing. Customers that are intentionally not using the Confluence plugin will no longer get errors and warnings about its absence.

The communication port selection logic between Crawling Module components has been improved.

This release of Maestro introduces a new implementation of the Pre-Push Extension Runner, which will be gradually rolled out over the coming weeks.

The new Extension Runner is built as a Minimal API that hosts a lightweight local server on http://localhost, using a randomly selected open port. This updated architecture is modern, more reliable, and easier to maintain, and it is now cross-platform, removing the previous limitation to Windows systems.

The new Extension Runner will be rolled out in shadow mode for the first few weeks. Since it will run in parallel with the existing implementation, you should not experience any performance impact.

Early testing indicates that the new Extension Runner delivers performance comparable to or better than the previous implementation. We’ll be monitoring exceptions coming out of the new implementation and any difference in the items returned. Some differences may occur if a Python extension script produces non-deterministic results. For example if random elements, GUIDs, or the working folder (which contains a GUID) are used in added metadata.

To help us investigate any discrepancies and identify potential bugs, please enable diagnostic log downloads so we can analyze the Crawling Module’s logs.

You can verify the current mode of the Extension Runner by checking the logs for the line The new Extension Runner service is running in mode. The mode will be compare during the shadow rollout. If you observe any errors or degraded performance, please reach out to your Coveo representative.

The new Extension Runner now uses a cache for Python modules, stored in the PythonModuleCache directory located alongside the Crawling Module’s binaries. This cache significantly reduces initialization time for subsequent runs. However, if a module’s version isn’t explicitly pinned in requirements.txt, it may not update automatically when a new version is released.

The cache is normally deleted with the Crawling Module’s installation folder during updates. If you encounter unexpected behavior in a script that depends on a newer module version, delete the PythonModuleCache folder. The modules will then be refreshed on the next run.

In contrast, under the previous implementation, Python modules were typically never updated when their versions weren’t explicitly specified in requirements.txt. As a result, the new implementation may trigger sudden upgrades of those modules, potentially causing script incompatibilities if newer versions aren’t backward compatible.

During the transition period, documents in error that are dumped by the new Extension Runner will include a -new suffix in their filenames. This allows them to be easily distinguished from those generated by the legacy Extension Runner.

The new Extension Runner’s logs are now included with the crawler logs, simplifying troubleshooting compared to the previous implementation.

The Extension Runner initialization has a hardcoded 10-minute timeout, which also covers the execution of the extension on the first item. If the extension is particularly slow to start, the first item may timeout, causing the original item to be indexed instead.

Since initialization persists between runs, if you notice that an extension’s results are missing from some items, running a rebuild should resolve the issue and apply the extension as expected.

As a best practice, use as few Python packages as possible, as installing dependencies can significantly increase initialization time.

Improved SID handling in the File system source by adding a connection lookup fallback.

Introduced the IgnoreMetadataWithNullOrEmptyValues parameter, allowing metadata fields that are null or empty to be omitted rather than sent with an empty value.

Worker monitoring logic is now simpler and more efficient. This reduces the software’s resource requirements, particularly lowering CPU usage during worker startup.

If installing Maestro on a new host, it is crucial that you start from a version of Maestro’s installer from September 2024 or after. To ensure you have the latest version, download the installer from either the Installing the Crawling Module article or the Crawling Modules (platform-ca | platform-eu | platform-au) page of the Coveo Administration Console.

The embedded browser packaged with Maestro now uses Microsoft Edge WebView2, replacing Chromium and Chrome for Testing. We expect this change to greatly improve maintainability of the embedded browser.

Maestro’s installer automatically checks for the presence of the WebView2 runtime and will install it if it’s not present on the host. As such, no manual action is required from users.

Additional startup validation is now performed to bring visibility to invalid configurations:

The Chrome driver used for JavaScript rendering has been updated to Chrome for Testing v134.

It’s now possible to skip request timeout errors caused by unresponsive API calls (taking over 5 minutes by default), instead of stopping the source update operation.

To do so, in your source configuration, add error code 408 under SkippableErrorCodes.

Added support for indexing a root space from the starting addresses in the new URL format [URL]/spaces/[KEY] that is introduced in Confluence Data Center 9.

Existing formats remain supported:

Handling of space type filtering parameters has been improved as well.

The Chrome driver used for JavaScript rendering has been changed from Chromium to Chrome for Testing (v133).

Chrome process management has been improved as well.

Improved support for error codes returned by GraphQL APIs.

Added checkpoint progress reporting in the security identity providers to avoid timeouts when expanding large identities.

This release includes fixes for all connectors:

Extended metadata actions are now available for REST API and GraphQL sources.

Improved handling of HTTP request errors in the Web source.

The Chrome driver used by these sources has been updated to major version 131.

Fixed an issue where the HTML Sitemap parser would fail on URLs containing encoding characters.