Input/Output

datapipe_testbench.storable

Classes for managing processed data.

class MetricStatus(*values)

Bases: StrEnum

Reasons why a benchmark failed or passed.

class ReqReport(name: str, value: float = nan, domain: str = 'unknown', status: MetricStatus = MetricStatus.UNKNOWN)

Bases: Storable

Stores the final report of a benchmark.

domain: str = 'unknown'

to which (telescope, array, etc) this report applies

classmethod load(init)

Load object from file or input dictionary.

Parameters:

initdict

initialization parameters

save(output_file: Path)

Save object to file.

Parameters:

output_filePath

Output filename.

class Storable

Bases: object

Base class for metric data that can be inserted into a DataSet.

abstractmethod classmethod load(init)

Load this object from a file.

init : file path, file object, dict

• file path: Initialize from the given file (FITS, JSON or ASDF)

• readable file object: Initialize from the given file object

• dict: The object model tree for the data model

abstractmethod save(output_file: Path)

Serialize this object to output_file.

update_name_with_store(output_store)

Update the name of a storable.

Appends the name of a supplied output store.

datapipe_testbench.store

Defines DataSets: collections of stored Data objects related to a single analysis.

class Metadata(path)

Bases: object

Handle metadata storage and retrieval from JSON.

Init function.

has_key(k)

Has key.

update(indict)

Update function.

class MetricsStore(base_path, *args, **kw)

Bases: Store

Manage storage of multiple related Data objects.

This class manages a directory structure defined by the base_path attribute under which data objects are stored and retrieved by identifier, which are simply relative paths. Additionally, this class manages global metadata that apply to all Data inside.

Store designed for Metrics, i.e. reduced bins that were or will be produced using input_data.

Parameters:

label

By default, None.

namestr, optional

By default, None.

str base_path

Base path On the hard drive where all the store structure will be save.

classmethod from_path_and_input_dataset(base_path: Path, input_dataset: InputDataset | dict)

Create MetricsStore from path and dict.

Parameters:

base_pathPath | str

Path on the hard drive where all the store structure will be save.

input_datasetInputDataset | dict

Dict that can instantiate an InputDataset, requires a “name” key and one of the InputDataset datalevel strings.

get_inputdata()

Get all the necessary information for the input dataset, and provenance (if any).

Returns:

InputDataset

Get the input dataset defined for this MetricsStore instance.

index_metrics()

Return all identified metrics in the store.

Structure of output dict is as follow:

• key: data_level/BenchmarkClass

• value: dict of histograms:

  • key: tuple of all column names

  • value: Metric instance

Returns:

dict[str,dict[str,Metric]]

Dict of all Metrics in the store, sorted by data_level/Benchmark.

set_inputdata(dataset: InputDataset)

Add input dataset metadata to metric store.

Parameters:

datasetInputDataset

Dataset object used for the benchmarks of this store. Subsequent use of set_inputdata will only work if preexisting sub_files in InputDataset are still the same in the new one.

class RequirementStore(base_path, *args, **kw)

Bases: Store

Store for requirement inputs.

Init function.

class ResultStore(base_path, *args, **kw)

Bases: Store

Storage of plots and reports.

Init function.

from_path(path: Path | str)

Create MetricsStore from path to an existing store.

Deprecated since version Use: constructor with path of pre-existing store as argument instead.

list_store()

List files in the store matching a supplied glob.

Default glob: *.asdf.

Deprecated since version Use: Store.contents() instead. (filetype can’t be given, this new method list ALL files in the store)

class Store(base_path, *args, **kw)

Bases: object

Base class for storage of sets of items and common metadata.

Init function.

contents(pattern='**')

List all files in the store, ordered by extension.

data_exists(path: Path) → bool

Check if the corresponding path exist in this store.

Parameters:

pathPath

Path to an existing store to test.

metadata_repr(indent=0)

List metadata in store as string representation.

Parameters:

indent

By default, 0.

int indent

Number of space to add to each line.

property name

The name of this dataset, used as a label when comparing.

retrieve_data(path: Path) → Storable

Retrieve data from this store corresponding to the input path.

Parameters:

pathPath

Returns:

storable.Storable

store_data(data: Storable | FigureBase, path: Path | str)

Store data in this Store at the given relative path.

datapipe_testbench.inputdataset

InputDataset class that wrap everything you need to define related to input files necessary to process a benchmark.

class InputDataset(name: str, dl0: Path | None = None, dl1: Path | None = None, dl1_images: Path | None = None, dl2: Path | None = None, dl2_signal: Path | None = None, dl2_background: Path | None = None, dl3_irf: Path | None = None, dl3_benchmark: Path | None = None)

Bases: object

An inputDataset defines common inputs that can be used by Benchmarks.

property dtypes

Return list of field names.

classmethod from_dict(input_dict)

Create instance from input dictionary (from serialisation).

Parameters:

input_dict

cls

dict input_dict

Input dictionary with all necessary information for a valid constructor.

Returns:

InputDataset

Instance corresponding to the input dictionary.

to_dict() → dict

Represent as a dict.