Tutorial Overview

Introduction

This tutorial gives you an overview of the QA/QC framework. You will learn how you can use and adapt it. In the beginning, each module of the framework will be shortly described. Then you will learn how to perform a single test and understand when to use mappers. Based on it, you will learn how to use the existing QA/QC components. Finally, the tutorial explains how you can implement your solutions and adapt the framework.

Note

The framework is based on pandas because it has been one of the most popular and favorite data science tools used in Python programming language. It’s highly recommended to have a solid knowledge of the package. If you aren’t familiar with pandas, check the docs https://pandas.pydata.org/docs/ before.

Modules Overview

autom8qc.core

The module autom8qc.core contains base classes that are isolated and can be used by any other module. It comes up with the submodule autom8qc.core.parameters that provides the classes Parameter and ParameterList to define the supported parameters of tests, mappers, functions, rules and measures. Moreover, it contains the submodule autom8qc.core.structures that defines the supported data structures (e.g., Series, DataFrame). And finally, it also includes the submodule autom8qc.core.validities that allows you to define your own validities or flags. The standard validities are Good, Limited, Erroneous, Not validate and Missing.

autom8qc.functions

Sometimes it’s required that a series must be processed before or after the execution of a test. For example, if you have two series with different resolutions and want to combine them, you have to resample one series to make them compatible. For that reason, the framework provides the module autom8qc.functions that implements functions (e.g., linear interpolation).

autom8qc.mappers

Mappers allow you to map values from a domain to another domain. For example, if you don’t want to use probabilities, you can use a mapper to map the probabilities to validities.

autom8qc.measures

If you use a group or a sequence with several tests, each test will return its probabilities or mapped values. To combine them to a total result, you have to use a measure. An example of a measure is the weighted mean of the probabilities.

autom8qc.qaqc.base

This module provides all about the QA/QC tests. It provides the classes TestManager, TestGroup, TestSequence and QAQCContainer. Moreover, all tests are implemented in that module. For sake of overview, the tests are divided into submodules like outlier, peak, limit, etc.

autom8qc.rules

Rules can be used to define rules for the test that will be applied on the results. An example is that all data points are set to invalid if over 20% of the data points are invalid.