Python Modules Available to Indexing Pipeline Extensions

A large number of modules are available for import in indexing pipeline extension (IPE) scripts. The following Python 3 modules may be especially useful:

  • This list is maintained manually, and might therefore be slightly out of date (e.g., new modules may have been added, or existing modules may have been updated to a newer version since the last edit). Last updated: October 2019.

    Validate the complete list of modules and their versions programmatically (see Listing Available Modules Programmatically).

  • Modules from the Python 3 Standard Library.

  • beautifulsoup4 (4.6.0) - A Python library for pulling data out of HTML and XML files (see beautifulsoup4).

  • boto3 (1.9.0) - Amazon Web Services (AWS) Software Development Kit (SDK) for Python (see boto3).

  • cryptography (2.7) - A cryptographic library (see cryptography).

  • python-dateutil (2.8.0) - Provides powerful extensions to the datetime module (see python-dateutil).

  • requests (2.22.0) - HTTP library for Python (see requests).

  • urllib3 (1.25.5) - An HTTP client (see urllib3).

Listing Available Modules Programmatically

To get a log message containing the current, exhaustive, up-to-date list of all Python 3 modules/versions available for import in any given Python 3 indexing pipeline extension script, you can run a Python 3 indexing pipeline extension that has the following script:

import pkg_resources
modules = pkg_resources.working_set
modules_list = sorted(["%s, version %s" % (i.project_name, i.version) for i in modules])
log(str(modules_list))

Getting the Python Version Programmatically

To get a log message containing the current Python version used for indexing pipeline extensions, you can run or test an indexing pipeline extension that has the following script:

import sys
myPythonVersion = sys.version_info
log(str(myPythonVersion))
Recommended Articles