Preprocessor

The preprocessor is the second step in the pipeline. It is responsible for cleaning and transforming your raw uploads into a format that is ready for data analyis and text generation.

You control the behaviour of the preprocessor with JavaScript code. The simplest preprocessor that just passes an upload through into data storage looks like this (this is also the default for each new pipeline):

export default function (object) {
	return object
}

While uploads have no format requirements, after preprocessing, the returned data object MUST have a uid attribute. This is used to identify the object in the data storage and is required for the rest of the pipeline and to match object updates. Without a uid, you would never be able to update your data. We also recommend to keep your object data structure flat, to make analysis and text generation easier.

JS Examples

TODO