Evaluation wrapper#

class TSCC.assessment.evaluation_handler.EvaluationHandler[source]#

The EvaluationHandler class manages the evaluation of machine learning models, specifically handling metrics for both classification and regression tasks. It includes functionality to append new rows of evaluation data, and retrieve stored evaluations. It also supports cross-validation and train-test evaluations.

Parameters:

Configobject: Configuration object that defines model settings for evaluation.
dataSetHandlerobject: Object responsible for managing the dataset during evaluation.

Attributes:

COLUMN_LIST_EVAL_CATlist: List of columns for storing categorical evaluation metrics (e.g., tn, fp, fn, tp, recall, precision).
COLUMN_LIST_ERRCLASS_EVAL_CATlist: List of columns for storing evaluation results based on error classes (e.g., tnr).
COLUMN_LIST_EVAL_REGlist: List of columns for storing regression evaluation metrics (e.g., mse, rmse, mae, nse).
det_evalpandas DataFrame: DataFrame that stores categorical evaluation results.
det_errorclass_evalpandas DataFrame: DataFrame that stores evaluation results based on error classes.
corr_evalpandas DataFrame: DataFrame that stores regression evaluation results.

Methods

get_det_eval()	Retrieves the current categorical evaluation DataFrame (det_eval).
get_det_errorclass_eval()	Retrieves the current error class evaluation DataFrame (det_errorclass_eval).
append_row_to_det_eval(new_row)	Appends a new row of categorical evaluation data to the det_eval DataFrame.
append_row_to_det_errorclass_eval(new_row)	Appends a new row of error class evaluation data to the det_errorclass_eval DataFrame.
append_row_to_corr_eval(new_row)	Appends a new row of regression evaluation data to the corr_eval DataFrame.

__init__(Config, dataSetHandler)[source]#

append_row_to_corr_eval(new_row)[source]#: Append a new row to the DataFrame. The new row should be in a format that can be converted to a DataFrame.

append_row_to_det_errorclass_eval(new_row)[source]#: Append a new row to the DataFrame. The new row should be in a format that can be converted to a DataFrame.

append_row_to_det_eval(new_row)[source]#: Append a new row to the DataFrame. The new row should be in a format that can be converted to a DataFrame.

get_det_errorclass_eval()[source]#: Return the current DataFrame.

get_det_eval()[source]#: Return the current DataFrame.

TSCC.assessment.evaluation_handler.correction_evaluation_wrapper(config, dataSetHandler, dataSetHandlerColumn_errorDetection, corr_ML_methods=[], corr_stat_methods=[])[source]#

Wraps error correction modeling and evaluation, applying machine learning and statistical methods for correcting detected errors. This function manages the application of correction models, performs cross-validation if configured, and stores the results for further evaluation.

Parameters:

configclass instance of Config: A mutable class instance that contains configuration settings for correction models, such as column names, cross-validation flags, and other relevant options.
dataSetHandlerclass instance of DataSetHandler: An instance responsible for storing and handling the dataset, managing both the feature and target data for training and testing.
dataSetHandlerColumn_errorDetectionstr: The column in the dataset that marks whether an error has been detected. This column is used to identify erroneous entries for correction.
corr_ML_methodslist, optional: A list of machine learning methods to be applied for error correction modeling (default is an empty list).
corr_stat_methodslist, optional: A list of statistical methods to be applied for error correction modeling (default is an empty list).

Returns:

ddict: A nested dictionary where each key corresponds to a correction method (either machine learning or statistical). The value is a sub-dictionary that contains the train and test predictions, the corrected target values, and other evaluation results.
evaluationHandlerEvaluationHandler: An instance of EvaluationHandler that contains the evaluation results for the applied correction models.

Notes

For each correction method, the results from multiple runs (if applicable) are combined using averaging for both train and test datasets.

TSCC.assessment.evaluation_handler.detection_evaluation_wrapper(config, dataSetHandler, det_ML_methods=[], det_stat_methods=[])[source]#

Wraps error detection modeling and evaluation, applying both machine learning and statistical methods for anomaly detection. This function manages the execution of models, performs cross-validation if configured, and stores the results in a nested dictionary structure and a class instance of the evaluationHandler.

Parameters:

configclass instance of Config: A mutable class instance containing the configuration settings for model evaluation, such as column names, cross-validation settings, and other options.
dataSetHandlerclass instance of DataSetHandler: An instance responsible for storing and handling the dataset. It manages both the features and targets for training and testing data.
det_ML_methodslist, optional: A list of machine learning methods for detection (default is an empty list). These methods will be applied for error detection modeling.
det_stat_methodslist, optional: A list of statistical methods for detection (default is an empty list). These methods will be applied for error detection modeling.

Returns:

ddict: A nested dictionary, where each key corresponds to a machine learning or statistical method. The value is a sub-dictionary that contains the train and test predictions (as pandas Series) and the generated machine learning models.
evaluationHandlerobject: A list containing the generated machine learning models for each method used in the detection.