preprocessor Class

preprocessor Class

Preprocessors are a specific type of LHF pipeline function that only run before the other pipelines. These pipe functions augment the input data before the remainder of the pipeline, such as clustering the data and replacing with representative centroids. Other functions, such as scaling or normalizing the data, may also be useful as future preprocessors.

Every preprocessor inherits the preprocessor class which defines the standard input and output methods to execute and retrieve data from the algorithm. See the pipePacket page for definition of the data passed into and retrieved from each preprocessor function.

---

Virtual Functions

The preprocessor class defines several virtual functions to be overridden by a designed pipe class.

virtual bool configPreprocessor(std::map<std::string, std::string> configMap);
virtual pipePacket runPreprocessor(pipePacket inData);
virtual void outputData(std::vector<unsigned>);
virtual void outputData(std::vector<std::vector<double>>);

preprocessor::configPreprocessor

A created pipeline that inherits the preprocessor class must have a configPreprocessor function to grab relevant configurations from the std::map object passed. This may include the dimensions, clusters to generate, iterations, whether to use special arguments, or other controlling parameters needed for the preprocessor.

To find an argument that is not required, use:

auto pipe = configMap.find("debug");
if(pipe != configMap.end())
    debug = std::atoi(configMap["debug"].c_str());

To find a required argument, use:

pipe = configMap.find("clusters");
if(pipe != configMap.end())
    maxEpsilon = std::atof(configMap["clusters"].c_str());
else return false;

Returning false indicates the configuration was unsuccessful and will not allow the preprocessor function to be executed.

preprocessor::runPreprocessor

The runPreprocessor function executes the designed algorithm against the input data from the pipePacket object. This function carries out all required steps and returns the data stored into the pipePacket object for further processing.

In the preprocessor class, the runPreprocessorWrapper function actually makes the call to runPreprocessor. The runPreprocessorWrapper function handles timing and memory evaluation, along with several other setup and cleanup steps that are consistent over all pipes.

The runPreprocessor function may also call methods within the class, the utilities functions, or from other libraries linked to LHF on compilation. The preprocessor structure gives a quick way to build and run a preprocessor in the LHF architecture.

basePipe::outputData

The outputData function is used to track a preprocessor's output during runtime. This function is only called when debug mode is enabled, in which case the data is written to the output folder and labeled with the name of the preprocessor.

Depending on the transformation or algorithm a pipeline is performing, one or several data sets may need to be output for transparency into the preprocessor function. If data is stored within the pipePacket, it can be accessed from the outputData function. However, transient data used in the preprocessor needs to be stored into a class-object to persist for output in the outputData function.

Each preprocessor should, at minimum, output the working data set for consistency between pipelined segments. The outputData function is not included in the timing of the preprocessor, so only output and statistical information should be evaluated during these points.