autom8qc.mappers

autom8qc.mappers.actris

ACTRISStandardMapper

Class

class autom8qc.mappers.actris.ACTRISStandardMapper(invalid_flag)

Bases: autom8qc.mappers.base.BaseMapper

This class provides a mapper that maps probabilities to the standardized ACTRIS flags. If the probability is 1, the flag is “000”. The user has to pass a flag to the constructor, which will set if the probability is 0. NaN values have the flag “999”.

Parameters

NAME (string) – Name of the test
DESCRIPTION (string) – Description of the test
parameters (ParameterList) – Supported parameters (default: None)

Supported parameters:

invalid_flag (str): Flag for invalid values. The flag has to contain 3 digits (e.g., 666)

map(values)

Maps the probabilities to flags.

Raises: InvalidType – If probabilities isn’t a Series.dtype
Parameters: values (pd.Series) – Probabilities of a test.
Returns: Mapped values
Return type: pd.Series

plot(data, values=None, mapped_values=None, description=None, *args, **kwargs)

Plots the mapped values. If the parameter mapped_values is not passed, the given values will be mapped before.

Parameters

data (object) – Values
values (pd.Series) – Values that should be mapped (e.g., probabilities)
mapped_values (pd.Series) – Mapped values (optional)
description (str) – Description of the data (e.g., parameter name)

Returns

None

Return type

None

savefig(filename, data, values=None, mapped_values=None, description=None, min_val=None, max_val=None, colormap='RdYlGn_r', title=None)

Saves the plot.

Important

The extension of the filename has to be the format. For example, if you want to store a SVG figure, the filename has to be ./example.svg

Parameters

filename – Name of the file
data (object) – Values
values (pd.Series) – Values that should be mapped (e.g., probabilities)
mapped_values (pd.Series) – Mapped values (optional)
description (str) – Description of the data (e.g., parameter name)
min_val (float) – Minium value (lower limit)
max_val (float) – Maximum value (upper limit)
colormap (str) – Colormap
title (string) – Title of the plot (default: NAME)

Returns

None

Return type

None

static supported_parameters()

Returns the supported parameters of the mapper.

Returns: Supported parameters
Return type: ParameterList

Example

# Generate sample data
import numpy as np
import pandas as pd
np.random.seed(42)
mu, sigma = 50, 5
values = np.random.normal(mu, sigma, 300)
values[[7, 19, 42, 93, 211]] = np.nan
index = pd.date_range(start="1/1/2021", periods=300, freq="min")
series = pd.Series(values, index=index)

# Perform test
from autom8qc.qaqc.limit import GlobalMinimumTest
test = GlobalMinimumTest(min_val=42, min_lim=40)
probabilities = test.perform(series)

# Map the probabilities
from autom8qc.mappers.actris import ACTRISStandardMapper
mapper = ACTRISStandardMapper(invalid_flag="666")
mapper.plot(series, probabilities)

Visualization

Logical2ACTRISMapper

Class

class autom8qc.mappers.actris.Logical2ACTRISMapper(invalid_flag)

Bases: autom8qc.mappers.base.BaseMapper

This class provides a mapper that maps bools to the standardized ACTRIS flags. If the value is True, it will be mapped to “000”. If the value is False, it will be mapped to the passed flag. If the value is NaN, the value will be mapped to “999”.

Parameters

NAME (string) – Name of the test

DESCRIPTION (string) – Description of the test

parameters (ParameterList) – Supported parameters

Supported parameters:

invalid_flag (str): Flag for invalid values. The flag has to contain 3 digits (e.g., 666)

map(values)

Maps the probabilities to flags.

Raises

InvalidType – If probabilities isn’t a Series.dtype

Parameters

values (pd.Series) – Probabilities of a test.

Returns

Mapped values

Return type

pd.Series

plot(data, values=None, mapped_values=None, description=None, *args, **kwargs)

Plots the mapped values. If the parameter mapped_values is not passed, the given values will be mapped before.

Parameters

data (object) – Values

values (pd.Series) – Values that should be mapped (e.g., probabilities)

mapped_values (pd.Series) – Mapped values (optional)

description (str) – Description of the data (e.g., parameter name)

Returns

None

Return type

None

savefig(filename, data, values=None, mapped_values=None, description=None, min_val=None, max_val=None, colormap='RdYlGn_r', title=None)

Saves the plot.

Important

The extension of the filename has to be the format. For example, if you want to store a SVG figure, the filename has to be ./example.svg

Parameters

filename – Name of the file

data (object) – Values

values (pd.Series) – Values that should be mapped (e.g., probabilities)

mapped_values (pd.Series) – Mapped values (optional)

description (str) – Description of the data (e.g., parameter name)

min_val (float) – Minium value (lower limit)

max_val (float) – Maximum value (upper limit)

colormap (str) – Colormap

title (string) – Title of the plot (default: NAME)

Returns

None

Return type

None

static supported_parameters()

Returns the supported parameters of the mapper.

Returns

Supported parameters

Return type

ParameterList

Example

# Generate sample data
import numpy as np
import pandas as pd
values = np.random.randint(2, size=100).astype(bool)
index = pd.date_range(start="1/1/2021", periods=100, freq="min")
series = pd.Series(values, index=index)

# Map the probabilities
from autom8qc.mappers.actris import Logical2ACTRISMapper
mapper = Logical2ACTRISMapper(invalid_flag="666")
mapper.plot(series, series)

Visualization

autom8qc.mappers.base

BaseMapper

class autom8qc.mappers.base.BaseMapper

Bases: autom8qc.core.components.BaseComponent

Abstract base class that defines the interface for mappers. Each mapper has to provide its information (NAME, DESCRIPTION) and has to implement the abstract method map. The method maps the given values to another domain (e.g., probabilities to validities)

Warning

If you inherit from this class, make sure that you call the super constructor and implement the abstract method map.

ListMapper

Class

class autom8qc.mappers.base.ListMapper(*mappers)

Bases: autom8qc.mappers.base.BaseMapper

This class allows you to concatenate mappers. The mappers must be pass to the constructor and will be stored in a list. When you apply the list to the data, all mappers of the list will be executed behind each other (the results will pass to the following mapper). The mappers will be performed in the same order as you passed them to the constructor.

Important

This mapper can be treated like a normal mapper.

Parameters

NAME (str) – Name of the mapper
DESCRIPTION (str) – Description of the mapper
parameters (ParameterList) – Supported parameters (default: None)

Supported parameters:

mappers (list): Mappers

append(mapper)

Append the given mapper.

Parameters: mapper (BaseMapper) – Mapper
Returns: None
Return type: None

map(values)

Maps the given values to other values (e.g. probabilities to flags).

Raises: NotImplementedError – This is an abstract method
Parameters: values (pd.Series) – Values that should be mapped (e.g. probabilities)
Returns: Mapped values
Return type: pd.Series

prepend(mapper)

Insert the given mapper at the beginning of the list.

Parameters: mapper (BaseMapper) – Mapper
Returns: None
Return type: None

static supported_parameters()

Returns the supported parameters.

Returns: Supported parameters
Return type: ParameterList

Example

# Generate sample data
import numpy as np
import pandas as pd
np.random.seed(42)
mu, sigma = 50, 5
values = np.random.normal(mu, sigma, 300)
index = pd.date_range(start="1/1/2021", periods=300, freq="min")
series = pd.Series(values, index=index)

# Perform test
from autom8qc.qaqc.limit import GlobalMinimumTest
test = GlobalMinimumTest(min_val=42, min_lim=40)
probabilities = test.perform(series)

# Map the probabilities
from autom8qc.core.validities import StandardValidities
from autom8qc.core.validities import ValidityContainer
from autom8qc.mappers.base import ListMapper
from autom8qc.mappers.validities import StandardValidityMapper
from autom8qc.mappers.validities import ValidityContainerMapper
iagos_validities = ValidityContainer(0, 2, 3, 4, 7)
mapper_a = StandardValidityMapper(good_limit=0.8, limited_limit=0.3)
mapper_b = ValidityContainerMapper(StandardValidities(), iagos_validities)
mapper = ListMapper(mapper_a, mapper_b)
mapper.plot(series, probabilities)

Visualization

autom8qc.mappers.logical

Logical2ValidityMapper

Class

class autom8qc.mappers.logical.Logical2ValidityMapper(true_validity=None, false_validity=None, missing_validity=None)

Bases: autom8qc.mappers.base.BaseMapper

Maps bool values to validities.

Note

The supported parameters are optional. If you don’t pass them, the standard validities will be used.

Parameters

NAME (str) – Name of the test
DESCRIPTION (str) – Description of the test
parameters (ParameterList) – Supported parameters

Supported parameters:

true_validity (Validity): Validity for True values (default: GOOD)
false_validity (Validity): Validity for False values (default: ERRONEOUS)
missing_validity (Validity): Validity for nan values (default: MISSING)

map(values)

Maps the probabilities to bools.

Raises: InvalidType – If probabilities isn’t a Series.dtype
Parameters: probabilities (pd.Series) – Probabilities of a test.
Returns: Mapped values
Return type: pd.Series

plot(data, values=None, mapped_values=None, description=None, *args, **kwargs)

Plots the mapped values. If the parameter mapped_values is not passed, the given values will be mapped before.

Parameters

data (object) – Values
values (pd.Series) – Values that should be mapped (e.g., probabilities)
mapped_values (pd.Series) – Mapped values (optional)
description (str) – Description of the data (e.g., parameter name)

Returns

None

Return type

None

savefig(filename, data, values=None, mapped_values=None, description=None, min_val=None, max_val=None, colormap='RdYlGn_r', title=None)

Saves the plot.

Important

The extension of the filename has to be the format. For example, if you want to store a SVG figure, the filename has to be ./example.svg

Parameters

filename – Name of the file
data (object) – Values
values (pd.Series) – Values that should be mapped (e.g., probabilities)
mapped_values (pd.Series) – Mapped values (optional)
description (str) – Description of the data (e.g., parameter name)
min_val (float) – Minium value (lower limit)
max_val (float) – Maximum value (upper limit)
colormap (str) – Colormap
title (string) – Title of the plot (default: NAME)

Returns

None

Return type

None

static supported_parameters()

Returns the supported parameters of the mapper.

Returns: Supported parameters
Return type: ParameterList

Example

# Generate sample data
import numpy as np
import pandas as pd
np.random.seed(42)
mu, sigma = 50, 4
values = np.random.normal(mu, sigma, 300) < 50
index = pd.date_range(start="1/1/2021", periods=300, freq="min")
series = pd.Series(values, index=index, dtype=bool)

# Map the probabilities
from autom8qc.mappers.logical import Logical2ValidityMapper
mapper = Logical2ValidityMapper()
mapper.plot(series, series)

Visualization

LogicalThresholdMapper

Class

class autom8qc.mappers.logical.LogicalThresholdMapper(threshold=0.5, force_nan=False)

Bases: autom8qc.mappers.base.BaseMapper

Implements a mapper that maps the probabilities that are lower equals then the defined threshold to False and probabilitiies above the threshold to True.

Parameters

NAME (str) – Name of the test
DESCRIPTION (str) – Description of the test
parameters (ParameterList) – Supported parameters

Supported parameters:

threshold (float): Limit for the probabilities (default: 0.5)
force_nan (bool): NaN will not be casted to False (default: False)

map(values)

Maps the probabilities to bools.

Warning

If you set the parameter force_nan to True, the method doesn’t return booleans because NaN can not be represented as a bool. The method will return floats (1=True, 0=False, np.nan).

Raises: InvalidType – If probabilities isn’t a Series.dtype
Parameters: values (pd.Series) – Probabilities of a test.
Returns: Mapped values
Return type: pd.Series

static supported_parameters()

Returns the supported parameters of the mapper.

Returns: Supported parameters
Return type: ParameterList

Example

# Generate sample data
import numpy as np
import pandas as pd
np.random.seed(42)
mu, sigma = 50, 4
values = np.random.normal(mu, sigma, 300)
index = pd.date_range(start="1/1/2021", periods=300, freq="min")
series = pd.Series(values, index=index)

# Perform test
from autom8qc.qaqc.limit import GlobalMinimumTest
test = GlobalMinimumTest(min_val=50)
probabilities = test.perform(series)

# Map the probabilities
from autom8qc.mappers.logical import LogicalThresholdMapper
mapper = LogicalThresholdMapper(threshold=0.5, force_nan=False)
mapper.plot(series, probabilities)

Visualization

autom8qc.mappers.validities

StandardValidityMapper

Class

class autom8qc.mappers.validities.StandardValidityMapper(validities=None, good_limit=1, limited_limit=0)

Bases: autom8qc.mappers.base.BaseMapper

This class provides a validity mapper that doesn’t need to be configured. If a probability is 0, the validity is Erroneous. If the probability is between 0 and 1, the validity is Limited. And if the probability is 1, the validity is Good. If the value is np.nan, the validity is Missing. Optional, you can set the lower limit for Good (e.g., 0.8) and Limited (e.g., 0.3) data points.

Warning

The following equations will be used to map the probabilities:

probabilities <= limited_limit to Erroneous
(probabilities > limited_limit) & (probabilities < good_limit) to Limited
probabilities >= good_limit to Good

Parameters

NAME (str) – Name of the test
DESCRIPTION (str) – Description of the test
parameters (ParameterList) – Supported parameters (default: None)

Supported parameters:

validities (StandardValidities): Validities that will be use by the mapper (optional)
good_limit (float): Lower limit for good data points (default=1)
limited_limit (float): Lower limit for limited data points (default=0)

map(values)

Maps the probabilities to validities.

Raises: InvalidType – If probabilities isn’t a Series.dtype
Parameters: values (pd.Series) – Probabilities of a test.
Returns: Mapped values
Return type: pd.Series

static supported_parameters()

Returns the supported parameters of the mapper.

Returns: Supported parameters
Return type: ParameterList

Example

# Generate sample data
import numpy as np
import pandas as pd
np.random.seed(42)
mu, sigma = 50, 5
values = np.random.normal(mu, sigma, 300)
index = pd.date_range(start="1/1/2021", periods=300, freq="min")
series = pd.Series(values, index=index)

# Perform test
from autom8qc.qaqc.limit import GlobalMinimumTest
test = GlobalMinimumTest(min_val=42, min_lim=40)
probabilities = test.perform(series)

# Map the probabilities
from autom8qc.mappers.validities import StandardValidityMapper
mapper = StandardValidityMapper(good_limit=0.8, limited_limit=0.3)
mapper.plot(series, probabilities)

Visualization

ValidityContainerMapper

Class

class autom8qc.mappers.validities.ValidityContainerMapper(origin_container, target_container)

Bases: autom8qc.mappers.base.BaseMapper

This class provides a mapper that maps the validities from an origin container to the valdities of a target container.

Example

# Generate sample data
import numpy as np
import pandas as pd
np.random.seed(42)
probabilities = np.random.rand(100)
index = pd.date_range(start="1/1/2021", periods=100, freq="min")
series = pd.Series(probabilities, index=index)

# Map probabilities to standard validities
from autom8qc.mappers.validities import StandardValidityMapper
mapper = StandardValidityMapper(good_limit=0.6, limited_limit=0.3)
validities = mapper.map(series)

# Create container for IAGOS validities
from autom8qc.core.validities import ValidityContainer
iagos_validities = ValidityContainer(0, 2, 3, 4, 7)

# Map the probabilities
from autom8qc.core.validities import StandardValidities
from autom8qc.mappers.validities import ValidityContainerMapper
mapper = ValidityContainerMapper(StandardValidities(), iagos_validities)
mapper.plot(validities, validities)

Visualization