Indexing Pipeline Extension State

This is for:

An IPE may need to perform web requests to access an external resource required for its execution, such as an authorization token or a catalog. In order not to flood the remote server, you can keep and share an IPE state between executions using the state keyword.

The state must be a Python dictionary and has a 32kb size limit. Oversized state or invalid types will be dropped.

The state is shared between the instances of a single IPE bound to a single source. For example, a state stored for source_1 cannot be accessed from source_2, and neither can a state stored for IPE_1 be accessed from IPE_2.

The state is cleared every 24 hours. You must implement your own logic in order to support a shorter lifetime. Since IPEs are executed in parallel, it may take up to a minute to propagate the updates to all running instances of the IPE.

Example

token = state.get('token')
if token:
    # Use existing token in request
    # ...
else:
    # Set the token in the state, and move on
    state['token'] = 'new_token'
    # ...