Tutorial Overview
Introduction
This tutorial gives you an overview of the QA/QC framework. You will learn how you can use and adapt it. In the beginning, each module of the framework will be shortly described. Then you will learn how to perform a single test and understand when to use mappers. Based on it, you will learn how to use the existing QA/QC components. Finally, the tutorial explains how you can implement your solutions and adapt the framework.
Note
The framework is based on pandas because it has been one of the most popular and favorite data science tools used in Python programming language. It’s highly recommended to have a solid knowledge of the package. If you aren’t familiar with pandas, check the docs https://pandas.pydata.org/docs/ before.
Modules Overview
autom8qc.core
The module autom8qc.core contains base classes that are isolated and can be used by any other module. It comes up with the submodule autom8qc.core.parameters that provides the classes Parameter and ParameterList to define the supported parameters of tests, mappers, functions, rules and measures. Moreover, it contains the submodule autom8qc.core.structures that defines the supported data structures (e.g., Series, DataFrame). And finally, it also includes the submodule autom8qc.core.validities that allows you to define your own validities or flags. The standard validities are Good, Limited, Erroneous, Not validate and Missing.
autom8qc.functions
Sometimes it’s required that a series must be processed before or after the execution of a test. For example, if you have two series with different resolutions and want to combine them, you have to resample one series to make them compatible. For that reason, the framework provides the module autom8qc.functions that implements functions (e.g., linear interpolation).
autom8qc.mappers
Mappers allow you to map values from a domain to another domain. For example, if you don’t want to use probabilities, you can use a mapper to map the probabilities to validities.
autom8qc.measures
If you use a group or a sequence with several tests, each test will return its probabilities or mapped values. To combine them to a total result, you have to use a measure. An example of a measure is the weighted mean of the probabilities.
autom8qc.qaqc.base
This module provides all about the QA/QC tests. It provides the classes TestManager, TestGroup, TestSequence and QAQCContainer. Moreover, all tests are implemented in that module. For sake of overview, the tests are divided into submodules like outlier, peak, limit, etc.
autom8qc.rules
Rules can be used to define rules for the test that will be applied on the results. An example is that all data points are set to invalid if over 20% of the data points are invalid.