dquality.accumulator

Please make sure the installation Pre-requisites are met.

The application monitors given directory for new/modified files of the given pattern. Each of the detected file is verified according to schema configuration and for each of the file several new processes are started, each process performing specific quality calculations.

The results will be sent to an EPICS PV (printed on screen for now).

Functions:

init(config)	This function initializes global variables.
verify(conf, folder, data_type, num_files[, ...])	This is the main function called when the verifier application starts.
directory(directory, patterns)	This method monitors a directory given by the “directory” parameter.

dquality.accumulator.init(config)

This function initializes global variables. It gets values from the configuration file, evaluates and processes the values. If mandatory file or directory is missing, the script logs an error and exits.

Parameters:config (str) – configuration file name, including path Returns:

• logger (Logger) – logger instance
• limits (dictionary) – a dictionary containing limit values read from the configured ‘limit’ file
• report_file (str) – a report file configured in a given configuration file
• extensions (list) – a list containing extensions of files to be monitored read from the configuration file

dquality.accumulator.verify(conf, folder, data_type, num_files, report_by_files=True)

This is the main function called when the verifier application starts. It reads the configuration for the directory to monitor, for pattern that represents a file extension to look for, and for a number of files that are expected for the experiment. The number of files configuration parameter is added for experiments that generate multiple files. In some cases the experiment data is collected into a single file, which is organized with data sets.

The function calls directory function that sets up the monitoring and returns notifier. After the monitoring is initialized, it starts a loop that reads the global “files” queue and then the global “results” queue. If there is any new file, the file is removed from the queue, and the data in the file is validated by a sequence of validation methods. If there is any new result, the result is removed from the queue, corresponding process is terminated, and the result is presented. (currently printed on console, later will be pushed into an EPICS process variable)

The loop is interrupted when all expected processes produced results. The number of expected processes is determined by number of files and number of validation functions.

Parameters:

• conf (str) – configuration file name, including path
• folder (str) – monitored directory
• data_type (str) – defines which data type is being evaluated
• num_files (int) – number of files that will be processed
• report_by_files (boolean) – this variable directs how to present the bad indexes in a report. If True, the indexes are related to the files, and a filename is included in the report. Otherwise, the report contains a list of bad indexes.

Returns:

bad_indexes (dict) – a dictionary or list containing bad indexes

dquality.accumulator.directory(directory, patterns)

This method monitors a directory given by the “directory” parameter. It creates a notifier object. The notifier is registered to await the “CLOSE_WRITE” event on a new file that matches the “pattern” parameter. If there is no such event, it yields control on timeout, defaulted to 1 second. It returns the created notifier.

Parameters:

• file (str) – File Name including path
• patterns (list) – A list of strings representing file extension

Returns:

None