autom8qc.functions

autom8qc.functions.base

BaseFunction

class autom8qc.functions.base.BaseFunction

Bases: autom8qc.core.components.BaseComponent

Abstract base class that defines the interface for functions. Each function has to provide its information (NAME, DESCRIPTION) and has to implement the abstract method apply. Functions can be used for preprocessing and postprocessing.

See also

  • autom8qc.core.components.BaseComponent

  • autom8qc.core.parameters.ParameterList

Warning

If you inherit from this class, make sure that you call the super constructor and implement the abstract method apply.

Parameters

  • NAME (str) – Name of the function

  • DESCRIPTION (str) – Description of the function

  • parameters (ParameterList) – Supported parameters (default: None)

abstract apply(data)

Applies the function to the given data.

Warning

Make sure that you don’t override data points of the data. For effiency reasons the data won’t be copied.

Raises

NotImplementedError – This is an abstract method

Parameters

data (BaseStructure) – Series, DataFrame, or mapped values

Returns

Results of the function

Return type

pd.Series

plot(data, mapped_values=None, style='.-', title=None)

Plots the mapped values. If the parameter mapped_values is not passed, the given values will be mapped before.

Parameters

  • data (object) – Values

  • mapped_values (pd.Series) – Mapped values (optional)

  • style (string) – Style of the line (default: line with points)

  • title (string) – Title of the plot (default: NAME)

Returns

None

Return type

None

savefig(filename, data, mapped_values=None, style='.-', title=None)

Saves the plot.

Important

The extension of the filename has to be the format. For example, if you want to store a SVG figure, the filename has to be ./example.svg

Parameters

  • filename (string) – Name of the file

  • data (object) – Values

  • mapped_values (pd.Series) – Mapped values (optional)

  • style (string) – Style of the line (default: line with points)

  • title (string) – Title of the plot (default: NAME)

Returns

None

Return type

None

ListFunction

Class

class autom8qc.functions.base.ListFunction(*functions)

Bases: autom8qc.functions.base.BaseFunction

This class allows you to concatenate functions. The functions must be pass to the constructor and will be stored in a list. When you apply the list to the data, all functions of the list will be executed behind each other (the results will pass to the following function). The functions will be performed in the same order as you passed them to the constructor.

Important

This function can be treated like a normal function.

Parameters

  • NAME (str) – Name of the function

  • DESCRIPTION (str) – Description of the function

  • parameters (ParameterList) – Supported parameters (default: None)

Supported parameters:

  • functions (list): Functions

append(function)

Append the given function.

Parameters

function (BaseFunction) – Function

Returns

None

Return type

None

apply(data)

Performs the functions behind each other.

Parameters

data (pd.Series or pd.DataFrame) – Series data

Returns

Results of the function

Return type

pd.Series or pd.DataFrame

plot(data, mapped_values=None, style='.-', title=None)

Plots the mapped values. If the parameter mapped_values is not passed, the given values will be mapped before.

Parameters

  • data (object) – Values

  • mapped_values (pd.Series) – Mapped values (optional)

  • style (string) – Style of the line (default: line with points)

  • title (string) – Title of the plot (default: NAME)

Returns

None

Return type

None

prepend(function)

Insert the given function at the beginning of the list.

Parameters

function (BaseFunction) – Function

Returns

None

Return type

None

savefig(filename, data, mapped_values=None, style='.-', title=None)

Saves the plot.

Important

The extension of the filename has to be the format. For example, if you want to store a SVG figure, the filename has to be ./example.svg

Parameters

  • filename (string) – Name of the file

  • data (object) – Values

  • mapped_values (pd.Series) – Mapped values (optional)

  • style (string) – Style of the line (default: line with points)

  • title (string) – Title of the plot (default: NAME)

Returns

None

Return type

None

static supported_parameters()

Returns the supported parameters.

Returns

Supported parameters

Return type

ParameterList

Example

# Generate sample data
import numpy as np
import pandas as pd
np.random.seed(42)
mu, sigma = 50, 3
values = np.random.normal(mu, sigma, 20)
values[[2, 3, 5, 7]] = -9999
index = pd.date_range(start="1/1/2021", periods=20, freq="2min")
series = pd.Series(values, index=index)

from autom8qc.functions.base import ListFunction
from autom8qc.functions.general import ReplaceMissingsFunction
from autom8qc.functions.interpolation import InterpolationFunction
function_a = ReplaceMissingsFunction(value=-9999)
function_b = InterpolationFunction(method="nearest")
function = ListFunction(function_a, function_b)
function.plot(series)

Visualization

../_images/ListFunction.svg

autom8qc.functions.general

ReplaceMissingsFunction

Class

class autom8qc.functions.general.ReplaceMissingsFunction(value=- 9999)

Bases: autom8qc.functions.base.BaseFunction

This class implements a function that replace missing values with NaN values. Within the IAGOS project the value -9999 is used to mark that the value is missing. In this case, the function replaces all -9999 values with NaN.

Parameters

  • NAME (str) – Name of the function

  • DESCRIPTION (str) – Description of the function

  • parameters (ParameterList) – Supported parameters (default: None)

Supported parameters:

  • value (float): Value that marks missing values (default: -9999)

apply(data)

Replaces the values value with NaN values.

Important

If the data doesn’t contain the missing value, a copy will be created with the exact values!

Parameters

data (pd.Series or pd.DataFrame) – Series data

Returns

Results of the function

Return type

pd.Series or pd.DataFrame

plot(data, mapped_values=None, style='.-', title=None)

Plots the mapped values. If the parameter mapped_values is not passed, the given values will be mapped before.

Parameters

  • data (object) – Values

  • mapped_values (pd.Series) – Mapped values (optional)

  • style (string) – Style of the line (default: line with points)

  • title (string) – Title of the plot (default: NAME)

Returns

None

Return type

None

savefig(filename, data, mapped_values=None, style='.-', title=None)

Saves the plot.

Important

The extension of the filename has to be the format. For example, if you want to store a SVG figure, the filename has to be ./example.svg

Parameters

  • filename (string) – Name of the file

  • data (object) – Values

  • mapped_values (pd.Series) – Mapped values (optional)

  • style (string) – Style of the line (default: line with points)

  • title (string) – Title of the plot (default: NAME)

Returns

None

Return type

None

static supported_parameters()

Returns the supported parameters.

Returns

Supported parameters

Return type

ParameterList

Example

# Generate sample data
import numpy as np
import pandas as pd
np.random.seed(42)
mu, sigma = 50, 3
values = np.random.normal(mu, sigma, 20)
values[[2, 3, 5, 7]] = -9999
index = pd.date_range(start="1/1/2021", periods=20, freq="2min")
series = pd.Series(values, index=index)

from autom8qc.functions.general import ReplaceMissingsFunction
function = ReplaceMissingsFunction(value=-9999)
function.plot(series)

Visualization

../_images/ReplaceMissingsFunction.svg

autom8qc.functions.interpolation

FillValidityGapsFunction

Class

class autom8qc.functions.interpolation.FillValidityGapsFunction(time_delta, validities, method)

Bases: autom8qc.functions.base.BaseFunction

An instance of the function interpolates gaps.

Important

The function supports only pd.Series!

Parameters

  • NAME (str) – Name of the function

  • DESCRIPTION (str) – Description of the function

  • parameters (ParameterList) – Supported parameters

Supported parameters:

  • time_delta (int): Number of seconds to detect gaps

  • validities (object): Validities

  • method (str): Method to fill the gaps

Supported methods:

  • worst (str): Fill the gaps with the worst validity around the gaps.

apply(data)

Fills the gaps.

Raises

InvalidType – If data is not an instance of the class pd.Series

Parameters

data (pd.Series) – Series data or mapped values

Returns

Results of the function

Return type

pd.Series

static supported_parameters()

Returns the supported parameters.

Returns

Supported parameters

Return type

ParameterList

worst_validity(data)

Applies the interpolation to fill the gaps. During the interpolation, the last flag before the gap and the first flag after the gap will be considered, and the worst flag will be used to fill the gap. If the series has missing values in the beginning or at the end, the gap will not fill.

Parameters

data (pd.Series) – Given values

Returns

Interpolated series

Return type

pd.Series

Example

# Generate sample data
import numpy as np
import pandas as pd
from autom8qc.core.validities import StandardValidities
np.random.seed(42)
validities = StandardValidities()
ids = np.random.randint(4, size=50)
values = np.array([validities.ALL_VALIDITIES[id] for id in ids])
values[10:20] = validities.MISSING
index = pd.date_range(start="1/1/2021", periods=50, freq="1min")
series = pd.Series(values, index=index)

from autom8qc.functions.interpolation import FillValidityGapsFunction
function = FillValidityGapsFunction(time_delta=5, validities=validities, method="worst")
function.plot(series)

Visualization

../_images/FillValidityGapsFunction.svg

InterpolationFunction

Class

class autom8qc.functions.interpolation.InterpolationFunction(resolution=None, method=None)

Bases: autom8qc.functions.base.BaseFunction

An instance of the class can be used to interpolate values.

See also

For supported methods take a look at the pandas documentation: pd.DataFrame.interpolate

Important

The function supports only pd.Series!

Parameters

  • NAME (str) – Name of the function

  • DESCRIPTION (str) – Description of the function

  • parameters (ParameterList) – Supported parameters

Supported parameters:

  • resolution (str): Lower limit for valid points

  • method (str): Lower limit for doubtful points (optional)

apply(data)

Applies the liner interpolation to the series.

Raises

InvalidType – If data is not an instance of the class pd.Series

Parameters

data (pd.Series) – Series data or mapped values

Returns

Results of the function

Return type

pd.Series

static supported_parameters()

Returns the supported parameters.

Returns

Supported parameters

Return type

ParameterList

Example

# Generate sample data
import numpy as np
import pandas as pd
np.random.seed(42)
mu, sigma = 50, 3
values = np.random.normal(mu, sigma, 100)
index = pd.date_range(start="1/1/2021", periods=100, freq="2min")
series = pd.Series(values, index=index)

from autom8qc.functions.interpolation import InterpolationFunction
function = InterpolationFunction(method="linear", resolution="30s")
function.plot(series)

Visualization

../_images/InterpolationFunction.svg