calibpipe.core package

Provide core classes and methods for CalibPipe.

exception calibpipe.core.CorruptedInputDataError(message='Input data is corrupted.')

Bases: CalibrationError

Raise when the input data file is available but cannot be correctly processed.

exit_code = 104

__init__(message='Input data is corrupted.')

exception calibpipe.core.ConfigurationError(message='Problem with the input configuration file.', *args)

Bases: CalibrationError

Raise for issues with the configuration files.

exit_code = 101

__init__(message='Problem with the input configuration file.', *args)

exception calibpipe.core.MissingInputDataError(message='Input data is missing.')

Bases: CalibrationError

Raise when expected input data is missing.

exit_code = 105

__init__(message='Input data is missing.')

exception calibpipe.core.InsufficientStatisticsError(message='Quality of input data is not sufficient.')

Bases: CalibrationError

Raise when data quality is found to be insufficient after processing.

exit_code = 106

__init__(message='Quality of input data is not sufficient.')

exception calibpipe.core.IntermittentError(message='Intermittent runtime error.')

Bases: CalibrationError

Indicate an intermittent known issue that might be resolved by re-submitting the job.

exit_code = 100

__init__(message='Intermittent runtime error.')

exception calibpipe.core.DBStorageError(message='Storage in the database cannot be performed.')

Bases: CalibrationStorageError

Raise for issues with data storage in the database.

exit_code = 102

__init__(message='Storage in the database cannot be performed.')

exception calibpipe.core.FileStorageError(message='Storage in the output file cannot be performed.')

Bases: CalibrationStorageError

Raise for issues with data storage in ECSV files.

exit_code = 102

__init__(message='Storage in the output file cannot be performed.')

exception calibpipe.core.CalibrationStorageError(message='Calibration coefficient cannot be stored.')

Bases: CalibrationError

Raise for database/ECSV table storage related exceptions.

exit_code = 102

__init__(message='Calibration coefficient cannot be stored.')

class calibpipe.core.ReferenceMetadataContainer(prefix=None, **fields)

Bases: Container

Container to store reference metadata.

Serves as a central node for other metadata containers. This is the essential metadata that must be attached to all data products to uniquely describe the data product.

Attributes:

version_atmospheric_modelField(default=None, type=str)

Atmospheric Model version value to serve as a foreign key

versionField(default=None, type=int)

Version of the reference metadata schema used in the data product

ID_optical_throughputField(default=None, type=int)

Optical throughput ID to serve as a foreign key

metadict

dict of attached metadata

prefixstr

Prefix attached to column names when saved to a table or file

version_atmospheric_model

version

ID_optical_throughput

meta

prefix

default_prefix = 'referencemetadata'

fields = {'ID_optical_throughput': Field(default=None, type=int), 'version': Field(default=None, type=int), 'version_atmospheric_model': Field(default=None, type=str)}

class calibpipe.core.ProductReferenceMetadataContainer(prefix=None, **fields)

Bases: Container

Container to store product-related reference metadata.

Describes the details of the data product, including its type and links to the data model definition.

Attributes:

IDField(default=None, type=int)

Autoincremented value to serve as primary/foreign key

descriptionField(default=None, type=str)

Human-readable description of data product

creation_timeField(default=None, type=str)

Human-readable date and time of file creation, in ISO format, UTC

product_idField(default=None, type=str)

A fixed-id to identify this product, e.g. UUID or VFN

data_categoryField(default=None, type=str)

A,B,C

data_levelField(default=None, type=str)

R0, R1, DL0, DL1, etc

data_associationField(default=None, type=str)

CTAO, Site, Subarray, Telescope, etc

data_typeField(default=None, type=str)

Event, Monitoring, Service, etc

data_model_nameField(default=None, type=str)

Identifying name of the data model used

data_model_versionField(default=None, type=str)

Version of the data model used

data_model_urlField(default=None, type=str)

Link to definition of data model, if available, and preferably a DOI

formatField(default=None, type=str)

File format of the data product

metadict

dict of attached metadata

prefixstr

Prefix attached to column names when saved to a table or file

ID

description

creation_time

product_id

data_category

data_level

data_association

data_type

data_model_name

data_model_version

data_model_url

format

meta

prefix

default_prefix = 'productreferencemetadata'

fields = {'ID': Field(default=None, type=int), 'creation_time': Field(default=None, type=str), 'data_association': Field(default=None, type=str), 'data_category': Field(default=None, type=str), 'data_level': Field(default=None, type=str), 'data_model_name': Field(default=None, type=str), 'data_model_url': Field(default=None, type=str), 'data_model_version': Field(default=None, type=str), 'data_type': Field(default=None, type=str), 'description': Field(default=None, type=str), 'format': Field(default=None, type=str), 'product_id': Field(default=None, type=str)}

class calibpipe.core.ContactReferenceMetadataContainer(prefix=None, **fields)

Bases: Container

Container to store contact reference metadata.

Describes the person or institution that is responsible for this data product.

Attributes:

IDField(default=None, type=int)

Autoincremented value to serve as primary/foreign key

organizationField(default=None, type=str)

Organization to which this data product is associated

nameField(default=None, type=str)

Name of contact within organization

emailField(default=None, type=str)

Contact email address

metadict

dict of attached metadata

prefixstr

Prefix attached to column names when saved to a table or file

ID

organization

name

email

meta

prefix

default_prefix = 'contactreferencemetadata'

fields = {'ID': Field(default=None, type=int), 'email': Field(default=None, type=str), 'name': Field(default=None, type=str), 'organization': Field(default=None, type=str)}

class calibpipe.core.ProcessReferenceMetadataContainer(prefix=None, **fields)

Bases: Container

Container to store process reference metadata.

Describes the top-level process to which the activity that generated this product belongs.

Attributes:

IDField(default=None, type=int)

Autoincremented value to serve as primary/foreign key

typeField(default=None, type=str)

General type of the process

subtypeField(default=None, type=str)

More specific class of the process if the class is not sufficient to describe it

subtype_idField(default=None, type=str)

Unique identifier of the process, e.g. if the type is observation this is the obs_id

metadict

dict of attached metadata

prefixstr

Prefix attached to column names when saved to a table or file

ID

subtype

subtype_id

meta

prefix

default_prefix = 'processreferencemetadata'

fields = {'ID': Field(default=None, type=int), 'subtype': Field(default=None, type=str), 'subtype_id': Field(default=None, type=str), 'type': Field(default=None, type=str)}

class calibpipe.core.ActivityReferenceMetadataContainer(prefix=None, **fields)

Bases: Container

Container to store activity reference metadata.

Describes the specific software or task that generated this particular data product.

Attributes:

IDField(default=None, type=int)

Autoincremented value to serve as primary/foreign key

activity_idField(default=None, type=str)

Unique identifier of the instance of this activity, if software a UUID

nameField(default=None, type=str)

Name of activity that produced this data product, e.g. the software/script name

typeField(default=None, type=str)

General type of the activity

startField(default=None, type=str)

Starting date/time of activity, in ISO format, UTC

endField(default=None, type=str)

Ending date/time of activity, in ISO format, UTC

software_nameField(default=None, type=str)

Name of software framework/library that was used if the activity if it involved software

software_versionField(default=None, type=str)

Version of software used

metadict

dict of attached metadata

prefixstr

Prefix attached to column names when saved to a table or file

ID

activity_id

name

start

end

software_name

software_version

meta

prefix

default_prefix = 'activityreferencemetadata'

fields = {'ID': Field(default=None, type=int), 'activity_id': Field(default=None, type=str), 'end': Field(default=None, type=str), 'name': Field(default=None, type=str), 'software_name': Field(default=None, type=str), 'software_version': Field(default=None, type=str), 'start': Field(default=None, type=str), 'type': Field(default=None, type=str)}

class calibpipe.core.InstrumentReferenceMetadataContainer(prefix=None, **fields)

Bases: Container

Container to store instrument reference metadata.

Describes the subset of CTAO Instrument Description to which this data product is associated, which could be e.g. a Sub-array, or a small part such as a photo-sensor.

Attributes:

IDField(default=None, type=int)

Autoincremented value to serve as primary/foreign key

siteField(default=None, type=str)

CTAO-South, CTAO-North, or other site associated with the data product

typeField(default=None, type=str)

(‘The specific type of instrument in the class. E.g. if class=camera, ‘, “the type might be CHEC, or if class=telescope, the type might be ‘SST’”)

subtypeField(default=None, type=str)

Sub-type of the instrument. For example if the type is MST, this might be ‘NectarCAM’ or ‘FlashCAM’

instrument_idField(default=None, type=str)

The unique ID of the specific instrument in the class. Depending on the instrument class,this might be for example is a telescope id, nominal subarray name, camera_id, or part serial number

metadict

dict of attached metadata

prefixstr

Prefix attached to column names when saved to a table or file

ID

site

subtype

instrument_id

meta

prefix

default_prefix = 'instrumentreferencemetadata'

fields = {'ID': Field(default=None, type=int), 'instrument_id': Field(default=None, type=str), 'site': Field(default=None, type=str), 'subtype': Field(default=None, type=str), 'type': Field(default=None, type=str)}

Submodules

calibpipe.core.common_metadata_containers module

Common metadata containers.

class calibpipe.core.common_metadata_containers.ReferenceMetadataContainer(prefix=None, **fields)

Bases: Container

Container to store reference metadata.

Serves as a central node for other metadata containers. This is the essential metadata that must be attached to all data products to uniquely describe the data product.

Attributes:

version_atmospheric_modelField(default=None, type=str)

Atmospheric Model version value to serve as a foreign key

versionField(default=None, type=int)

Version of the reference metadata schema used in the data product

ID_optical_throughputField(default=None, type=int)

Optical throughput ID to serve as a foreign key

metadict

dict of attached metadata

prefixstr

Prefix attached to column names when saved to a table or file

version_atmospheric_model

version

ID_optical_throughput

meta

prefix

default_prefix = 'referencemetadata'

fields = {'ID_optical_throughput': Field(default=None, type=int), 'version': Field(default=None, type=int), 'version_atmospheric_model': Field(default=None, type=str)}

class calibpipe.core.common_metadata_containers.ProductReferenceMetadataContainer(prefix=None, **fields)

Bases: Container

Container to store product-related reference metadata.

Describes the details of the data product, including its type and links to the data model definition.

Attributes:

IDField(default=None, type=int)

Autoincremented value to serve as primary/foreign key

descriptionField(default=None, type=str)

Human-readable description of data product

creation_timeField(default=None, type=str)

Human-readable date and time of file creation, in ISO format, UTC

product_idField(default=None, type=str)

A fixed-id to identify this product, e.g. UUID or VFN

data_categoryField(default=None, type=str)

A,B,C

data_levelField(default=None, type=str)

R0, R1, DL0, DL1, etc

data_associationField(default=None, type=str)

CTAO, Site, Subarray, Telescope, etc

data_typeField(default=None, type=str)

Event, Monitoring, Service, etc

data_model_nameField(default=None, type=str)

Identifying name of the data model used

data_model_versionField(default=None, type=str)

Version of the data model used

data_model_urlField(default=None, type=str)

Link to definition of data model, if available, and preferably a DOI

formatField(default=None, type=str)

File format of the data product

metadict

dict of attached metadata

prefixstr

Prefix attached to column names when saved to a table or file

ID

description

creation_time

product_id

data_category

data_level

data_association

data_type

data_model_name

data_model_version

data_model_url

format

meta

prefix

default_prefix = 'productreferencemetadata'

fields = {'ID': Field(default=None, type=int), 'creation_time': Field(default=None, type=str), 'data_association': Field(default=None, type=str), 'data_category': Field(default=None, type=str), 'data_level': Field(default=None, type=str), 'data_model_name': Field(default=None, type=str), 'data_model_url': Field(default=None, type=str), 'data_model_version': Field(default=None, type=str), 'data_type': Field(default=None, type=str), 'description': Field(default=None, type=str), 'format': Field(default=None, type=str), 'product_id': Field(default=None, type=str)}

class calibpipe.core.common_metadata_containers.ContactReferenceMetadataContainer(prefix=None, **fields)

Bases: Container

Container to store contact reference metadata.

Describes the person or institution that is responsible for this data product.

Attributes:

IDField(default=None, type=int)

Autoincremented value to serve as primary/foreign key

organizationField(default=None, type=str)

Organization to which this data product is associated

nameField(default=None, type=str)

Name of contact within organization

emailField(default=None, type=str)

Contact email address

metadict

dict of attached metadata

prefixstr

Prefix attached to column names when saved to a table or file

ID

organization

name

email

meta

prefix

default_prefix = 'contactreferencemetadata'

fields = {'ID': Field(default=None, type=int), 'email': Field(default=None, type=str), 'name': Field(default=None, type=str), 'organization': Field(default=None, type=str)}

class calibpipe.core.common_metadata_containers.ProcessReferenceMetadataContainer(prefix=None, **fields)

Bases: Container

Container to store process reference metadata.

Describes the top-level process to which the activity that generated this product belongs.

Attributes:

IDField(default=None, type=int)

Autoincremented value to serve as primary/foreign key

typeField(default=None, type=str)

General type of the process

subtypeField(default=None, type=str)

More specific class of the process if the class is not sufficient to describe it

subtype_idField(default=None, type=str)

Unique identifier of the process, e.g. if the type is observation this is the obs_id

metadict

dict of attached metadata

prefixstr

Prefix attached to column names when saved to a table or file

ID

subtype

subtype_id

meta

prefix

default_prefix = 'processreferencemetadata'

fields = {'ID': Field(default=None, type=int), 'subtype': Field(default=None, type=str), 'subtype_id': Field(default=None, type=str), 'type': Field(default=None, type=str)}

class calibpipe.core.common_metadata_containers.ActivityReferenceMetadataContainer(prefix=None, **fields)

Bases: Container

Container to store activity reference metadata.

Describes the specific software or task that generated this particular data product.

Attributes:

IDField(default=None, type=int)

Autoincremented value to serve as primary/foreign key

activity_idField(default=None, type=str)

Unique identifier of the instance of this activity, if software a UUID

nameField(default=None, type=str)

Name of activity that produced this data product, e.g. the software/script name

typeField(default=None, type=str)

General type of the activity

startField(default=None, type=str)

Starting date/time of activity, in ISO format, UTC

endField(default=None, type=str)

Ending date/time of activity, in ISO format, UTC

software_nameField(default=None, type=str)

Name of software framework/library that was used if the activity if it involved software

software_versionField(default=None, type=str)

Version of software used

metadict

dict of attached metadata

prefixstr

Prefix attached to column names when saved to a table or file

ID

activity_id

name

start

end

software_name

software_version

meta

prefix

default_prefix = 'activityreferencemetadata'

fields = {'ID': Field(default=None, type=int), 'activity_id': Field(default=None, type=str), 'end': Field(default=None, type=str), 'name': Field(default=None, type=str), 'software_name': Field(default=None, type=str), 'software_version': Field(default=None, type=str), 'start': Field(default=None, type=str), 'type': Field(default=None, type=str)}

class calibpipe.core.common_metadata_containers.InstrumentReferenceMetadataContainer(prefix=None, **fields)

Bases: Container

Container to store instrument reference metadata.

Describes the subset of CTAO Instrument Description to which this data product is associated, which could be e.g. a Sub-array, or a small part such as a photo-sensor.

Attributes:

IDField(default=None, type=int)

Autoincremented value to serve as primary/foreign key

siteField(default=None, type=str)

CTAO-South, CTAO-North, or other site associated with the data product

typeField(default=None, type=str)

(‘The specific type of instrument in the class. E.g. if class=camera, ‘, “the type might be CHEC, or if class=telescope, the type might be ‘SST’”)

subtypeField(default=None, type=str)

Sub-type of the instrument. For example if the type is MST, this might be ‘NectarCAM’ or ‘FlashCAM’

instrument_idField(default=None, type=str)

The unique ID of the specific instrument in the class. Depending on the instrument class,this might be for example is a telescope id, nominal subarray name, camera_id, or part serial number

metadict

dict of attached metadata

prefixstr

Prefix attached to column names when saved to a table or file

ID

site

subtype

instrument_id

meta

prefix

default_prefix = 'instrumentreferencemetadata'

fields = {'ID': Field(default=None, type=int), 'instrument_id': Field(default=None, type=str), 'site': Field(default=None, type=str), 'subtype': Field(default=None, type=str), 'type': Field(default=None, type=str)}

calibpipe.core.exceptions module

Calibpipe-specific exceptions.

Exit code values are following the ICD.

exception calibpipe.core.exceptions.CalibrationError

Bases: Exception

Base class for all CalibPipe exceptions.

exit_code = 1

exception calibpipe.core.exceptions.IntermittentError(message='Intermittent runtime error.')

Bases: CalibrationError

Indicate an intermittent known issue that might be resolved by re-submitting the job.

exit_code = 100

__init__(message='Intermittent runtime error.')

exception calibpipe.core.exceptions.CorruptedInputDataError(message='Input data is corrupted.')

Bases: CalibrationError

Raise when the input data file is available but cannot be correctly processed.

exit_code = 104

__init__(message='Input data is corrupted.')

exception calibpipe.core.exceptions.MissingInputDataError(message='Input data is missing.')

Bases: CalibrationError

Raise when expected input data is missing.

exit_code = 105

__init__(message='Input data is missing.')

exception calibpipe.core.exceptions.InsufficientStatisticsError(message='Quality of input data is not sufficient.')

Bases: CalibrationError

Raise when data quality is found to be insufficient after processing.

exit_code = 106

__init__(message='Quality of input data is not sufficient.')

exception calibpipe.core.exceptions.ConfigurationError(message='Problem with the input configuration file.', *args)

Bases: CalibrationError

Raise for issues with the configuration files.

exit_code = 101

__init__(message='Problem with the input configuration file.', *args)

exception calibpipe.core.exceptions.CalibrationStorageError(message='Calibration coefficient cannot be stored.')

Bases: CalibrationError

Raise for database/ECSV table storage related exceptions.

exit_code = 102

__init__(message='Calibration coefficient cannot be stored.')

exception calibpipe.core.exceptions.DBStorageError(message='Storage in the database cannot be performed.')

Bases: CalibrationStorageError

Raise for issues with data storage in the database.

exit_code = 102

__init__(message='Storage in the database cannot be performed.')

exception calibpipe.core.exceptions.FileStorageError(message='Storage in the output file cannot be performed.')

Bases: CalibrationStorageError

Raise for issues with data storage in ECSV files.

exit_code = 102

__init__(message='Storage in the output file cannot be performed.')