Each component type has clearly defined roles with responsibilities and constraints that should be followed in order to fully benefit from functionality offered by Jetstream.
Here are some notes on how to choose the right component type for each use case, and how to combine components to achieve desired result when it’s not possible to get that from a single component.
An Input connects to a data source and produces a data stream. It does nothing else. Given that, an input component is typically set as the first component of the pipe. In that case, there is no data pipe passed to it as an input parameter.
However, if a pipe is configured so that the input component is not the first one, it receives a data pipe, and can then be used to implement following actions on received data:
An input MAY add records that differ in format from those passed to it; those records can later be separated, if needed, using the input id associated with each data item.
Todo
add reference to retrieving and using input id label
An input MAY NOT modify the structure or content of data passed to it. To modify data in any way, always use a transformer.
For developing inputs, see Input components.
An Inspector reads the data but does not touch (or output) it. However it has control over the processing of the data stream, and may include parts of data within its own non-data output. An inspector can thus perform for example one or more of the following actions:
For developing inspectors, see Inspector components.
Transformer modifies data and passes it on, and may also drop records. Thus a transformer can be naturally used to:
A transformer may also add to the data. It is however limited to using the data it has received as input, including any information available within the Jetstream system, such as configuration information passed to the transformer. It may not fetch data on its own from any other, external data source.
For developing Transformers, see Transformer components.
An Output connects to a :term:`data destination` and writes data to it. So the end of the pipe always has some sort of Output configured, even if it is simply some code that reads and prints out the data, for example.
When not configured as pipe end, an output component can be used to:
Todo
add loop-back data source/destination functionality to Jetstream
For developing Outputs, see Output components.