Use the Extensions API
- Document Object Python API Reference
- Condition Syntax Reference
- Python Modules Available to Extensions
- Extension Parameters
- Extension ActionOnError
- Supported Character Sets
- Extension Script Samples
- Extension State
- Use Vault Parameters in Your Extension
- Create an Extension With the API
- Log Messages From an Extension
- Test Strategies and Good Practices
Indexing Pipeline Extension State
An IPE may need to perform web requests to access an external resource required for its execution, such as an authorization token or a catalog.
In order not to flood the remote server, you can keep and share an IPE state between executions using the
state must be a Python dictionary and has a 32kb size limit.
Oversized state or invalid types will be dropped.
The state is shared between the instances of a single IPE bound to a single source.
For example, a state stored for
source_1 cannot be accessed from
source_2, and neither can a state stored for
IPE_1 be accessed from
The state is cleared every 24 hours. You must implement your own logic in order to support a shorter lifetime. Since IPEs are executed in parallel, it may take up to a minute to propagate the updates to all running instances of the IPE.
token = state.get('token') if token: # Use existing token in request # ... else: # Set the token in the state, and move on state['token'] = 'new_token’ # ...