dquality.data

Please make sure the installation Pre-requisites are met.

This module verifies a configured hd5 file. The data is verified against configured “limits” file. The limits are applied by processes performing specific quality calculations.

The results is a detail report of calculated values. The indexes of slices that did not pass quality check are returned back.

Functions:

init(config)	This function initializes global variables.
process_data(data_type, aggregateq, fp, ...)	This method creates and starts a new handler process.
verify_file(logger, file, data_tags, limits, ...)	This method creates and starts a new handler process.
verify(conf, file)	This invokes sequentially data verification processes for data_dark, data_white, and data that is contained in configured hd5 file.

dquality.data.init(config)

This function initializes global variables. It gets values from the configuration file, evaluates and processes the values. If mandatory file or directory is missing, the script logs an error and exits.

Parameters:config (str) – configuration file name, including path Returns:

• logger (Logger) – logger instance
• data_tags (dict) – a dictionary od data_type/hdf tag
• limits (dictionary) – a dictionary containing limit values read from the configured ‘limit’ file
• report_file (str) – a report file configured in a given configuration file

dquality.data.process_data(data_type, aggregateq, fp, data_tag, limits)

This method creates and starts a new handler process. The handler is initialized with data queue, the data type, and a result queue. The data type can be ‘data_dark’, ‘data_white’ or ‘data’. After starting the process the function enqueues queue slice by slice into data, until all data is queued. The last enqueued element is end of the data marker.

Parameters:

• data_type (str) – string indicating what type of data is processed.
• aggregateq (Queue) – result queue; used by handler to enqueue results
• fp (file pointer) – a pointer to the file that is verifier
• data_tag (str) – a data tag to look for
• limits (dict) – a dictionary of limits values

Returns:

None

dquality.data.verify_file(logger, file, data_tags, limits, report_file)

This method creates and starts a new handler process. The handler is initialized with data queue, the data type, and a result queue. The data type can be ‘data_dark’, ‘data_white’ or ‘data’. After starting the process the function enqueues queue slice by slice into data, until all data is queued. The last enqueued element is end of the data marker.

Parameters:

• logger (Logger) – Logger instance.
• file (str) – a filename including path that will be verified
• data_tags (dict) – a dictionary od data_type/hdf tag
• limits (dict) – a dictionary of limits values
• report_file (str) – a name of a report file

Returns:

bad_indexes (dict) – a dictionary of bad indexes per data type

dquality.data.verify(conf, file)

This invokes sequentially data verification processes for data_dark, data_white, and data that is contained in configured hd5 file. The calculated values are check against limits, configured in a file. Each process gets two queues parameters; one queue to get the data, and second queue to pass back the results. The function awaits results from the response queues in the matching sequence to how the processes started. The results are retrieved as json object. The indexes of slices that did not pass quality check are reported to the calling function in form of dictionary.

Parameters:

• conf (str) – name of the configuration file including path. If contains only directory, ‘dqconfig_test.ini’ will a default file name.
• file (str) – file name to do the quality checks on

Returns:

bad_indexes (Dict) – A dictionary containing indexes of slices that did not pass quality check. The key is a type of data. (i.e. data_dark, data_white,data)