API¶
The following functions mirror the command-line tools, and can be called from a Python workflow to run the same stages (see Data model: building your own tools).
Sources¶
- accessionflow.sources.folder.api.collate(input_paths: list[str], output_dir: ~pathlib.Path, datatypes: ~typing.Sequence[type[~fileformats.core.fileset.FileSet]] = (<class 'fileformats.medimage.dicom.DicomSeries'>,), accession_uid: ~typing.Sequence[~accessionflow.helpers.arg_types.IDSpec] = (IDSpec(specifier='StudyInstanceUID', datatype=<class 'fileformats.medimage.dicom.DicomCollection'>),), acquisition_id: ~typing.Sequence[~accessionflow.helpers.arg_types.IDSpec] = (IDSpec(specifier='SeriesNumber', datatype=<class 'fileformats.medimage.dicom.DicomCollection'>),), resource: ~typing.Sequence[~accessionflow.helpers.arg_types.IDSpec] = (), path_metadata_regex: ~typing.Sequence[~accessionflow.helpers.arg_types.PathMetadataRegex] = (), path_metadata: ~typing.Sequence[~accessionflow.helpers.arg_types.PathMetadataPattern] = (), unlink_source: str | None = None, raise_errors: bool = False, copy_mode: ~fileformats.core.fileset.FileSet.CopyMode = CopyMode.hardlink_or_copy, wait_period: int = 0, collation_map: dict[type[~fileformats.core.fileset.FileSet], ~fileformats.core.fileset.FileSet.CopyCollation] | None = None, conversion_map: dict[type[~fileformats.core.fileset.FileSet], tuple[type[~fileformats.core.fileset.FileSet], dict[str, str]]] | None = None, allow_unrecognised: ~typing.Sequence[str] = (), exclude_paths: ~typing.Sequence[str] = (), ignore_datatypes: ~typing.Sequence[type[~fileformats.core.fileset.FileSet]] = (), on_resource_clash: ~typing.Literal['error', 'avoid', 'merge', 'overwrite'] | ~typing.Sequence[~accessionflow.helpers.arg_types.ClashSpec] = 'error', metadata_tables: list[~accessionflow.helpers.arg_types.MetadataTable] | None = None, recursive: bool = False, dont_read_metadata: ~typing.Sequence[type[~fileformats.core.fileset.FileSet]] = ()) list[str][source]¶
Collates the input files into accessions/acquisitions/resources and stages them into the staging directory. Project/subject/visit IDs and scan descriptions are not labelled at this point, see the 'label' function for that.
- Parameters:
input_paths (list[str]) -- List of paths to search for input files: directories, or glob patterns that select the files directly. They must be local or on a mounted file system.
output_dir (Path) -- Path to the staging directory where the collated sessions will be saved. This should be a local path.
datatypes (ty.Sequence[type[FileSet]]) -- FileSet types to look for in the input files. Only files with these datatypes will be considered for staging. Defaults to
(DicomSeries,)to mirror the CLI.accession_uid (ty.Sequence[IDSpec]) -- Field specifications to extract the accession UIDs from the input files, to collate them into separate accessions. Defaults to
DEFAULT_ACCESSION_UID_FIELD(StudyInstanceUIDscoped to DICOM collections; the same object backs the CLI's--accession); other fileset types need an explicit spec.acquisition_id (ty.Sequence[IDSpec]) -- Field specifications to extract the acquisition IDs from the input files, to collate them into acquisitions. Defaults to
DEFAULT_ACQUISITION_ID_FIELD(SeriesNumberscoped to DICOM collections; shared with the CLI's--acquisition); for a fileset whose type is not matched by any spec here the acquisition is named after that fileset's resource.resource (ty.Sequence[IDSpec]) -- List of field specifications to use for extracting the resource IDs from the input files to collate them into resources. If empty, each resource is labelled with the mime-like rendering of its fileset type name, e.g. 'vectra-export', 'sqlite3-db'
path_metadata_regex (ty.Sequence[PathMetadataRegex]) -- Regular expressions to extract "metadata" values from resource file paths as named groups. The named groups are used as metadata fields for the resource files, and the extracted values will be used to populate the corresponding metadata fields to complement the metadata read from the file headers.
path_metadata (ty.Sequence[PathMetadataPattern]) -- A simpler, glob-like alternative to
path_metadata_regex, where{name}placeholders mark the parts of the path to extract as metadata fields, e.g.'{subject}/{session}'. Each pattern is translated into a regex (seepath_metadata_pattern_to_regex) and applied before thepath_metadata_regexones, so a regex wins where both set the same field.unlink_source (str | None) -- If "all" or "keep-metadata", the input files will be unlinked one by one after staging (both behave the same here, since the source isn't a directory tree that accessionflow owns). If None, the input files will be left in place.
raise_errors (bool) -- If True, any errors encountered during staging will raise an exception. If False, errors will be logged and the staging process will continue for the remaining sessions.
copy_mode (FileSet.CopyMode) -- The copy mode to use when saving the sessions. This determines whether files are copied, moved or symlinked when saving the sessions to the staging directory.
collation_map (dict[ty.Type[FileSet], FileSet.CopyCollation] | None) -- A mapping of FileSet types to CopyCollation objects that specify how to collate files of that type when saving the sessions. If None, the default collation behavior for each FileSet type will be used.
conversion_map (dict[ty.Type[FileSet], tuple[ty.Type[FileSet], dict[str, str]]] | None) -- A mapping of source FileSet types to (target FileSet types, conversion options). When a resource matches a source type, it will be converted to the target type during save, with the options passed through to
convert().allow_unrecognised (ty.Sequence[str]) -- Regexes matched against the basename of any input path that no datatype recognised; matches are skipped instead of raising
FormatMismatchError.[".*"]tolerates all unrecognised files. Does not affect recognised filesets.exclude_paths (ty.Sequence[str]) -- Globs matched against each input path relative to its input directory, applied before classification so a match is dropped even if a datatype would claim it (e.g. a vendor thumbnail that is a valid
image/png).*does not cross/,**does.ignore_datatypes (ty.Sequence[type[FileSet]]) -- Datatypes expected in the input but not wanted: recognised filesets of these types are dropped rather than raising, and (with
recursive) matching directories are skipped without descending. A path matching neitherdatatypesnorignore_datatypes(norallow_unrecognised/exclude_paths) still raises.dont_read_metadata (ty.Sequence[type[FileSet]]) -- Datatypes whose files aren't read for metadata, so their only metadata fields are those from
path_metadata/path_metadata_regexandmetadata_tables. Saves reading the header of every file when none of its fields are used.wait_period (int) -- If provided, this is the number of seconds that must have passed since the last modification time of the session before it will be staged. This can be used to avoid staging sessions that are still being modified or created.
on_resource_clash (OnResourceClash or Sequence[ClashSpec]) -- Behaviour when two filesets resolve to the same scan/resource name. A bare policy string ("error"/"avoid"/"merge"/"overwrite") applies to any clash. A sequence of
ClashSpec(policy + datatype scope) resolves each clash with the first spec whose scope covers both filesets - a clash no spec covers raises. "avoid" suffixes, "merge" folds into oneSetOf, "overwrite" replaces. Default "error".recursive (bool) -- If True, the input paths will be searched recursively for files to stage. If False, only the files directly within the input paths will be considered for staging.
metadata_tables (list[MetadataTable] | None) --
Specify metadata tables to extract and join metadata from input files (ACFLOW_METADATA_TABLES env. var). The 'path' arg specifies the location of the metadata table file. Its format is auto-detected as CSV or TSV from the file extension; a different format can be forced by appending its mime-type in square brackets, e.g. 'path/to/table.dat[text/csv]'. The "row frequency" arg specifies what each row in the metadata table corresponds to in the data hierarchy, and can be one of 'accession', 'acquisition', 'resource', 'fileset', 'fileset[<mime-type>]'. When one or more mime-types are given in square brackets after 'fileset' they restrict the join to input files of those types (multiple mime-types can be '|'-separated, e.g. 'fileset[image/png|image/jpeg]'); a bare 'fileset' matches any input file. The 'join-exprs' arg is a comma-separated list of '<column-name>=<cell-value>' expressions; a row is a match when every expression holds. The '<cell-value>' is either the name of an existing metadata field or a Python format string over one or more metadata fields, e.g. '{PatientID}_{SessionID}'. All columns of the matched row are then merged into the target's metadata. The example below extracts the relative path of an image file with path_metadata_regex and uses it to join a table whose 'ImagePath' column holds spreadsheet HYPERLINK() formulas:
collate( ..., path_metadata_regex=[ PathMetadataRegex( regex=r".*/(?P<relpath>[\w-]+/[\w-]+\.(?:png|jpg))", datatype="image/png|image/jpeg", ) ], metadata_tables=[ MetadataTable( table_file="path/to/table.csv[text/csv]", row_frequency="fileset[image/png|image/jpeg]", join_exprs='ImagePath=HYPERLINK("{relpath}")', ) ], )
- accessionflow.sources.orthanc.api.collate(url: str, store_dir: Path, output_dir: Path, user: str, password: str, to_process_label: str | None = None, processed_label: str | None = None, unlink_source: str | None = None, raise_errors: bool = False, copy_mode: CopyMode = CopyMode.hardlink_or_copy, wait_period: int = 0, conversion_map: dict[type[FileSet], tuple[type[FileSet], dict[str, str]]] | None = None) list[str][source]¶
Collates the DICOM studies held by an Orthanc server into accessions in the staging directory, hardlinking the files from Orthanc's storage directory.
- Parameters:
url (str) -- the base URL of the Orthanc REST API, e.g. 'http://orthanc:8042'
store_dir (Path) -- Orthanc's StorageDirectory as mounted, which must be on the same file system as output_dir for the files to be hardlinked
output_dir (Path) -- the staging directory the collated accessions are saved in
user (str) -- the Orthanc user to log in as
password (str) -- the password of the Orthanc user
to_process_label (str | None) -- the label applied to studies in Orthanc, by something else, to mark them for collation. If None, all studies are collated
processed_label (str | None) -- the label applied to studies in Orthanc once they have been collated, so that they aren't collated again
unlink_source (str | None) -- if "all" or "keep-metadata", the studies would be removed from Orthanc once they have been collated. Not yet implemented
raise_errors (bool) -- whether to raise errors instead of logging them. Not yet implemented
copy_mode (FileSet.CopyMode) -- how the files are copied. Only hardlinks are implemented
wait_period (int) -- the number of seconds since Orthanc last updated a study before it is collated, so that studies still being received aren't collated partially
conversion_map (dict[type[FileSet], tuple[type[FileSet], dict[str, str]]] | None) -- the types series are converted to before the study is labelled as processed, with the options passed through to
convert()
- Returns:
the IDs of the studies that couldn't be collated
- Return type:
- accessionflow.api.save_accessions_to_dir(accessions: list[Accession], msg: str, build_dir: Path, copy_mode: CopyMode, output_dir: Path, wait_period: int = 0, collation_map: dict[type[FileSet], CopyCollation] | None = None, conversion_map: dict[type[FileSet], tuple[Type[FileSet], dict[str, str]]] | None = None, unlink_source: str | None = None, raise_errors: bool = False) list[str][source]¶
Save collated accessions into a build directory, then move each one into the staging directory once it is complete, so that a later stage never sees a partly saved accession.
- Parameters:
accessions (list[Accession]) -- the collated accessions, referencing their files at the source
msg (str) -- the description of the progress bar
build_dir (Path) -- the directory the accessions are saved into before they are moved
copy_mode (FileSet.CopyMode) -- how the files are copied from the source
output_dir (Path) -- the staging directory the accessions are moved into
wait_period (int) -- the number of seconds since an accession's files were last modified before it is saved, so that accessions still being transferred are skipped
collation_map (dict[type[FileSet], FileSet.CopyCollation], optional) -- how files of each type are laid out in their resource directories
conversion_map (dict[type[FileSet], tuple[type[FileSet], dict[str, str]]], optional) -- the types resources are converted to as they are saved, with the options passed to
convert()unlink_source (str, optional) -- if "all" or "keep-metadata", each accession's source files are removed once it is saved
raise_errors (bool) -- whether to raise errors instead of logging them and moving on to the next accession
- Returns:
the errors, one per accession that couldn't be saved
- Return type:
Stages¶
- accessionflow.api.attach(input_dir: Path, output_dir: Path, datatype: Type[FileSet], glob: str, identity_pattern: str, spaces_to_underscores: bool = False, on_resource_clash: Literal['error', 'avoid', 'merge', 'overwrite'] = 'error', raise_errors: bool = False, require_manifest: bool = True, copy_mode: CopyMode = CopyMode.copy, unlink_source: str | None = None) list[str][source]¶
- accessionflow.api.label(input_dir: Path, output_dir: Path, label_fields: Mapping[str, str], acquisition_field: str | None = None, constant_labels: Mapping[str, str] | None = None, store_type: str | None = None, copy_mode: CopyMode = CopyMode.hardlink_or_copy, unlink_source: str | None = None, raise_errors: bool = False, include: Sequence[type[FileSet]] = (), single_resource: bool = False) list[str][source]¶
Sorts the input files into sessions and stages them into the staging directory.
- Parameters:
input_dir (Path) -- List of paths to search for input files. Can be local paths or S3 paths.
output_dir (Path) -- Path to the staging directory where the sorted sessions will be saved. This should be a local path.
label_fields (Mapping[str, str]) -- The metadata field (or a format string over several fields) to extract each label from, keyed by the name of its level in the data store, in order from the top level, e.g.
{"project": "StudyComments", "subject": "PatientID", "session": "AccessionNumber"}for XNATacquisition_field (str | None) -- Field name to use for extracting a description for each acquisition (e.g. the XNAT scan type). Acquisitions for which the field can't be resolved are left without a description.
constant_labels (Mapping[str, str] | None) -- Labels to use for all sessions instead of extracting them from the metadata, keyed by level name, e.g.
{"project": "MYPROJECT"}. Useful for instruments that deposit in a single project.store_type (str | None) -- The name of the data store the labels are for (e.g. 'xnat'), recorded with the session so it isn't deposited in a different kind of store
copy_mode (FileSet.CopyMode) -- The copy mode to use when saving the sessions. This determines whether files are copied, moved or symlinked when saving the sessions to the staging directory.
unlink_source (str | None) -- If "all" or "keep-metadata", record files present before label loads the session. After a successful save, remove those files and empty directories. With "keep-metadata", remove files only inside loaded resource directories; leave session and scan metadata in place. Files at new paths remain. If None, the collated session directory is left in place.
raise_errors (bool) -- If True, any errors encountered during staging will raise an exception. If False, errors will be logged and the staging process will continue for the remaining sessions.
include (Sequence[type[FileSet]]) -- Only stage resources matching at least one of these datatypes. An empty sequence stages all resources.
single_resource (bool) -- If True, the store holds a single resource per acquisition, so a session with an acquisition that would be saved with more than one resource (after filtering by include) is an error, and is left in place to be reprocessed (e.g. with include set to select one).
Notes
If any of a session's labels can't be resolved from its metadata, it is saved with placeholder IDs (see Accession.label) under an INVALID_NAME_DEFAULT ('__invalid__') subdirectory of output_dir instead of the regular output location, so it can be found and manually reprocessed rather than being lost.
- accessionflow.api.deidentify(input_dir: Path, output_dir: Path, recipe_dir: Path, reid_dir: Path | None = None, on_resource_clash: Literal['error', 'avoid', 'merge', 'overwrite'] = 'error', raise_errors: bool = False, copy_mode: CopyMode = CopyMode.copy, require_manifest: bool = True, unlink_source: str | None = None, reid_encrypt_key: bytes | None = None, max_workers: int | None = None, metadata_allowlist: Collection[str] = {'__uid__'}, hash_uid_key: str | None = None) list[str][source]¶
- Parameters:
reid_dir (Path, optional) -- the directory to write the re-identification metadata to, one JSON file per session. If not given, the re-identification metadata is discarded rather than written to disk (use this when retaining the original<->deidentified mapping would itself be a security concern).
max_workers (int, optional) -- the number of threads handed to a resource's own deidentify implementation to parallelise work within that resource (e.g. the per-file loop for a DICOM series). Ignored by formats that don't support it.
hash_uid_key (str, optional) -- the secret key used to replace each session's UID (by default its DICOM StudyInstanceUID) with a keyed hash of it in the output. The same UID and key always give the same hash, so a session is still recognised when it is uploaded again. If not given, the original UID is kept in the output.
- accessionflow.api.package(input_dir: Path, output_dir: Path, source_datatype: type[FileSet], output_resources: Mapping[str, tuple[type[FileSet], Mapping[str, Any]]], on_resource_clash: Literal['error', 'avoid', 'merge', 'overwrite'] = 'error', raise_errors: bool = False, require_manifest: bool = True, copy_mode: CopyMode = CopyMode.hardlink_or_copy, unlink_source: str | None = None) list[str][source]¶
Replace matching scan resources with multiple named converted resources.
Conversions are materialised only after a staged session has been loaded, so this stage can be placed after deidentification. All output conversions are independent and receive the original source fileset.
- accessionflow.api.annotate_api.annotate(input_dir: Path, output_dir: Path, extract: Mapping[str, IDSpec], raise_errors: bool = False) list[str][source]¶
Annotates the accessions with fields extracted from their metadata, which the data store records alongside them (e.g. as custom variables on XNAT sessions).
Data stores¶
- accessionflow.stores.folder.api.store(input_dir: str | Path, dest: str, storage_options: Mapping[str, Any] | None = None, wait_period: int = 0, require_manifest: bool = True, overwrite: bool = False, raise_errors: bool = False, store_credentials: StoreCredentials | None = None, temp_dir: Path | None = None, s3_max_workers: int | None = None) list[str][source]¶
Deposits the labelled accessions in a staging directory in a folder store: nested label folders on a local path, a mount, or a remote file system such as S3 or SFTP (see accessionflow.stores.folder.layout for the layout).
Depositing again is cheap: a resource whose manifest in the store matches the staged one is skipped, and one left without a manifest by an interrupted deposit is deposited again.
- Parameters:
input_dir (str or Path) -- the staging directory holding the labelled accessions (the output of 'label folder', or of 'deidentify' after it), or an S3 bucket ('s3://...') to download them from
dest (str) -- the root of the store, a path or an fsspec URL, e.g. '/mnt/archive', 's3://bucket/archive' or 'sftp://host/archive'
storage_options (Mapping[str, Any], optional) -- options for the fsspec file system of the store, e.g. credentials
wait_period (int) -- the number of seconds since an accession was last modified before it is deposited, so one that is still being written isn't
require_manifest (bool) -- whether the staged resources must have manifests. If False, the checksums of resources without one are worked out from their files
overwrite (bool) -- whether to replace a resource that is already in the store with different files. If False, it is reported as an error and left as it is
raise_errors (bool) -- whether to raise errors instead of logging them and continuing with the next accession
store_credentials (StoreCredentials, optional) -- the credentials of the S3 bucket the accessions are staged in, if they are
temp_dir (Path, optional) -- the directory to download accessions staged in S3 to
s3_max_workers (int, optional) -- the number of files to download from an S3 staging bucket at once
- Returns:
the errors encountered, one per accession that couldn't be deposited
- Return type:
- accessionflow.stores.folder.api.check_store(input_dir: str | Path, dest: str, storage_options: Mapping[str, Any] | None = None, checksums: bool = True, require_manifest: bool = True, raise_errors: bool = False, store_credentials: StoreCredentials | None = None, temp_dir: Path | None = None, s3_max_workers: int | None = None) list[str][source]¶
Checks what has been deposited in a folder store against the staging directory it was deposited from, and logs every resource that is missing, incomplete or different.
- Parameters:
input_dir (str or Path) -- the staging directory the accessions were deposited from, or an S3 bucket
dest (str) -- the root of the store, a path or an fsspec URL, e.g. 's3://bucket/archive'
storage_options (Mapping[str, Any], optional) -- options for the fsspec file system of the store, e.g. credentials
checksums (bool) -- whether to check the files' checksums, which reads every file back from the store. If False, only their sizes are compared, which is much cheaper
require_manifest (bool) -- whether the staged resources must have manifests
raise_errors (bool) -- whether to raise errors instead of logging them
store_credentials (StoreCredentials, optional) -- the credentials of the S3 bucket the accessions are staged in, if they are
temp_dir (Path, optional) -- the directory to download accessions staged in S3 to
s3_max_workers (int, optional) -- the number of files to download from an S3 staging bucket at once
- Returns:
a description of each problem found
- Return type:
- accessionflow.stores.xnat.api.store(input_dir: str, xnat_repo: Xnat, always_include: Sequence[str | FileSet] = (), store_credentials: StoreCredentials | None = None, require_manifest: bool = True, methods: Sequence[UploadMethod] = (), wait_period: int = 0, num_files_per_batch: int = 0, check_checksums: bool = True, s3_cache_dir: Path | None = None, raise_errors: bool = False, dry_run: bool = False, s3_max_workers: int = 10, xnat_max_workers: int = 10, modality: str | None = None, date_field: str | None = None, skip_uid_check: bool = False) list[str][source]¶
Upload sorted sessions in the given staging directory to XNAT
- Parameters:
input_dir (Path) -- The directory containing the sessions to upload. Each session should be in a separate subdirectory.
xnat_repo (Xnat) -- The XNAT repository to upload to
always_include (Sequence[str]) -- A sequence of scan types or file paths to always include in the upload regardless of whether they are explicitly specified in the frameset definition
raise_errors (bool) -- Whether to raise errors that occur during upload or to log them and continue with the next session
store_credentials (StoreCredentials) -- Whether to store credentials for accessing staging directories that require authentication (e.g. S3)
require_manifest (bool) -- Whether to require a manifest file in each session directory that specifies the resources to upload and their checksums
methods (Sequence[UploadMethod]) -- The upload method to use for each datatype (e.g. 'tgz_file' or 'directory')
wait_period (int) -- The minimum age in seconds of session directories to upload (only applicable for local staging directories)
num_files_per_batch (int) -- The number of files to upload in each batch when uploading resources with the 'directory' method (if 0, all files will be uploaded in a single batch)
check_checksums (bool) -- Whether to check checksums of uploaded resources against the checksums specified in the manifest file and the checksums of the files in the staged resources (if available) to verify that they were uploaded correctly
dry_run (bool) -- Whether to list the sessions that would be uploaded instead of actually uploading them
s3_max_workers (int, optional) -- The maximum number of concurrent S3 object downloads. Defaults to 10.
xnat_max_workers (int, optional) -- The maximum number of concurrent XNAT resource uploads. Defaults to 10.
modality (str, optional) -- The modality of the sessions to upload (e.g. 'MR', 'CT'). Defaults to the "Modality" metadata field if present otherwise an error is raised.
skip_uid_check (bool, optional) -- don't compare a session's UID with the one recorded on an existing XNAT session, for use while moving to hashed UIDs. The subject is still checked where known.
date_field (str, optional) -- Session metadata field to use as the XNAT session date when creating a session. Existing XNAT sessions are not modified.
- accessionflow.stores.xnat.api.check_store(input_dir: str, server: str, user: str, password: str, store_credentials: ~accessionflow.helpers.arg_types.StoreCredentials, always_include: ~typing.Sequence[str | ~typing.Type[~fileformats.core.fileset.FileSet]] = (<class 'fileformats.medimage.dicom.DicomSeries'>, ), temp_dir: ~pathlib.Path | None = None, verify_ssl: bool = True, use_curl_jsession: bool = False, disable_progress: bool = False, s3_max_workers: int = 10, xnat_max_workers: int = 10, skip_uid_check: bool = False) None[source]¶
Checks the staged sessions against the XNAT server to check for any issues before upload.
This is intended to be used as a pre-upload check to identify any issues with the staged sessions before attempting to upload them to XNAT. It checks that the project, session, scan and resource exist on XNAT and that the checksums of the files in the resource match those on XNAT. If any issues are found, they are logged and a summary of the issues is logged at the end. If the --raise-errors flag is set, then any issues will raise an exception instead of being logged.
- Parameters:
input_dir (str) -- The directory containing the staged sessions to check.
server (str) -- The URL of the XNAT server to check against.
user (str) -- The username to use when connecting to the XNAT server.
password (str) -- The password to use when connecting to the XNAT server.
loggers (list[LoggerConfig]) -- The logger configurations to use for logging the output of the function.
always_include (list[str]) -- A list of MIME types to always include in the check, even if they aren't defined in the frameset on XNAT. This can be used to include additional file formats that aren't defined in the frameset, or to include all file formats by using "all".
s3_max_workers (int, optional) -- the maximum number of concurrent S3 object downloads. Defaults to 10.
xnat_max_workers (int, optional) -- the maximum number of concurrent XNAT checksum requests. Defaults to 10.
Model classes¶
These classes represent an accession as it moves through the stages: an accession contains acquisitions, each of which contains one or more resources, e.g. the original DICOM files and a derived NIfTI conversion (see Accessions, acquisitions and resources).
- class accessionflow.model.accession.Accession(uid: str, labels: Mapping[str, str] = NOTHING, store_type: str | None = None, acquisitions: Sequence[Acquisition] | Dict[str, Acquisition] = NOTHING, session_resources: Dict[str, Resource] = NOTHING, run_uid: str | None = None)[source]¶
Bases:
objectRepresentation of an accession to be deposited in a data store: the data from one experiment (e.g. an imaging session of a patient, a preclinical scan, or the imaging of a specimen or phantom), as a set of acquisitions that belong together under the same labels.
- Parameters:
labels (dict[str, str]) -- The labels the accession is deposited under in the data store, keyed by the names of the store's label levels in order from the top, e.g.
{"project": "P1", "subject": "S1", "session": "V1"}for XNAT. Empty until the accession is labelledstore_type (str, optional) -- The name of the data store the accession was labelled for, e.g. 'xnat'
acquisitions (ty.Dict[str, Acquisition]) -- The acquisitions in the accession
run_uid (ty.Optional[str]) -- The run UID of the session, if it exists
- acquisitions: Dict[str, Acquisition]¶
- attach_files(patterns: list[AttachedFiles], spaces_to_underscores: bool = True, on_resource_clash: Literal['error', 'avoid', 'merge', 'overwrite'] = 'error') list[FileSet][source]¶
Adds files associated with the primary files to the session
- deidentify(dest_dir: Path, recipe_files: dict[type[FileSet], Path] | None = None, copy_mode: CopyMode = CopyMode.hardlink_or_copy, on_resource_clash: Literal['error', 'avoid', 'merge', 'overwrite'] = 'error', require_matching_recipe_file: bool = True, max_workers: int | None = None) tuple[Self, dict[str, Any]][source]¶
Creates a new session with deidentified images
- Parameters:
dest_dir (Path) -- the directory to save the deidentified files into
recipe_files (dict[type[FileSet], Path], optional) -- project-specific deidentification recipe files for the different file types within the imaging session. The keys are the file types (see https://arcanaframework.github.io/fileformats/) and the values are paths to the recipe files, which are loaded into the format expected by the deidentify implementation for each file type (e.g.
DeidRecipefor DICOM)copy_mode (FileSet.CopyMode, optional) -- the mode to use to copy the files that don't need to be deidentified, by default FileSet.CopyMode.hardlink_or_copy
on_resource_clash (OnResourceClash, optional) -- when copying a file that doesn't need to be deidentified, if "avoid", if a resource with the same name already exists in the scan, increment the resource name by appending _1, _2 etc. to the name until a unique name is found, by default "avoid" if "merge", existing sessions with the same name will be merged. if "error", an error will be raised if a session with the same name already exists in the staging directory. if "overwrite", an existing resource with the same name will be overwritten.
require_matching_recipe_file (bool, optional) -- whether to require a matching recipe file for each fileset whose deidentify implementation accepts a recipe, by default True
max_workers (int, optional) -- passed through as max_workers to each resource's FileSet.deidentify, for formats whose deidentification implementation can parallelise work within a single resource (e.g. the per-file loop for a DICOM series) using threads. Formats that don't accept/use it just ignore it. Resources themselves are deidentified/copied sequentially, one at a time, to keep failures easy to trace back to the resource that caused them.
- Returns:
Accession -- a new session with deidentified images
dict[str, Any] -- a mapping containing the original values of metadata fields that have been removed or modified
- fields: Fields¶
- classmethod from_paths(files_path: str | Path | Sequence[str | Path], datatypes: type[FileSet] | Sequence[type[FileSet]], accession_uid_field: Sequence[IDSpec], acquisition_id_field: Sequence[IDSpec] = (), resource_field: Sequence[IDSpec] = (), recursive: bool = False, on_resource_clash: Literal['error', 'avoid', 'merge', 'overwrite'] | Sequence[ClashSpec] = 'error', allow_unrecognised: Sequence[str] | None = None, exclude_paths: Sequence[str] | None = None, ignore_datatypes: Sequence[type[FileSet]] | None = None, path_metadata_regex: Sequence[PathMetadataRegex] = (), metadata_tables: list[MetadataTable] | None = None, dont_read_metadata: Sequence[type[FileSet]] = ()) list[Self][source]¶
Loads all imaging sessions from a list of DICOM files
- Parameters:
files_path (str or Path) -- Path to a directory containing the resources to load the sessions from, or a glob string that selects the paths
datatypes (type or list[type]) -- the fileformats to load from the paths, e.g. DicomSeries or [DicomSeries, NiftiGz]
accession_uid_field (ty.Sequence[IDSpec]) -- the metadata field that uniquely identifies the accession, used to collate files together before they are labelled (e.g. StudyInstanceUID)
acquisition_id_field (ty.Sequence[IDSpec]) -- the value of this field is used to collate resources under single acquisitions. For a fileset whose type is not matched by any spec here (including the empty default), the acquisition is named after that fileset's resource.
resource_field (ty.Sequence[IDSpec]) -- the value of this field is used to identify resources. If empty, the resource is labelled with the mime-like rendering of the fileset's type name, e.g. 'vectra-export', 'sqlite3-db'
recursive (bool, optional) -- recurse into directories passed as file paths. For file datatypes this flattens the tree; when any
datatypes/ignore_datatypesentry is a directory format the walk stops descending into a directory as soon as it validates as one of them (yielding adatatypesmatch whole, skipping anignore_datatypesmatch whole).generic/directoryandgeneric/file-setcannot be used asdatatypeswithrecursive. By default Falseon_resource_clash (OnResourceClash or Sequence[ClashSpec], optional) -- how to handle two filesets resolving to the same scan/resource name. A bare policy string ("error"/"avoid"/"merge"/"overwrite") applies to any clash. A sequence of
ClashSpec(policy + datatype scope) resolves each clash with the first spec whose scope covers both filesets; "merge" folds them into aSetOf, "avoid" suffixes, "overwrite" replaces; a clash no spec covers raises. Default "error".allow_unrecognised (Sequence[str] or None, optional) -- regular expressions matched against the basename of any input path that no datatype recognised - matches are skipped instead of raising
FormatMismatchError. Does not affect recognised filesets.exclude_paths (Sequence[str] or None, optional) -- glob patterns matched against each input path relative to its input directory, applied before classification so a match is dropped even if a datatype would claim it (e.g. a vendor thumbnail that is a valid
image/png).*does not cross/,**does.ignore_datatypes (ty.Sequence[type[FileSet]] or None, optional) -- datatypes that are expected in the input but not wanted: recognised filesets of these types are dropped from the result rather than raising, and (when
recursive) matching directories are skipped without descending. An input path matching neitherdatatypesnorignore_datatypes(norallow_unrecognised/exclude_paths) still raises.path_metadata_regex (ty.Sequence[PathMetadataRegex], optional) -- Regular expressions to extract "metadata" values from resource file paths as named groups. The named groups are used as metadata fields for the resource files, and the extracted values will be used to populate the corresponding metadata fields to complement the metadata read from the file headers.
metadata_tables (list[MetadataTable] or None, optional) -- a list of MetadataTable objects that define how to extract metadata from input files (e.g. CSV files and spreadsheets) and join them with the sessions. If None, no metadata tables will be used.
dont_read_metadata (ty.Sequence[type[FileSet]], optional) -- datatypes whose files aren't read for metadata (matched with
isinstance), so their only metadata fields are those frompath_metadata_regexandmetadata_tables. Reading the headers of many files can be slow, e.g. on a network file system, and is wasted when none of their fields are used.
- Returns:
all imaging sessions that are present in list of dicom paths
- Return type:
- Raises:
AccessionParseError -- if values extracted from IDs across the DICOM scans are not consistent across DICOM files within the session
- label(label_fields: Mapping[str, str], acquisition_field: str | None = None, constant_labels: Mapping[str, str] | None = None, store_type: str | None = None) dict[str, str][source]¶
Assigns the labels the accession is deposited under in a data store, extracted from its metadata. Also resolves a description for each acquisition in the accession, if 'acquisition_field' is provided.
- Parameters:
label_fields (Mapping[str, str]) -- the metadata field (or a format string over several fields) to extract each label from, keyed by the name of its level in the data store, in order from the top level, e.g.
{"project": "StudyComments", "subject": "PatientID", "session": "AccessionNumber"}for XNATacquisition_field (str, optional) -- metadata field to extract a description for each acquisition from. Acquisitions for which the field can't be resolved are left without a description (saved with a trailing-dot '<scan_id>.' directory name)
constant_labels (Mapping[str, str], optional) -- labels to use as given rather than extract from the metadata, keyed by level name, e.g.
{"project": "MYPROJECT"}store_type (str, optional) -- the name of the data store the labels are for, e.g. 'xnat', recorded so the accession isn't deposited in a different kind of store
- Returns:
the metadata fields that couldn't be resolved, mapped to the placeholder labels used instead
- Return type:
Notes
If a label's field can't be resolved from the accession's metadata, a unique 'INVALID_MISSING_<FIELD>_<random>' placeholder is used instead of raising, so the accession can still be saved (see invalid_ids) for manual review/reprocessing rather than being silently dropped.
- classmethod load(session_dir: Path, require_manifest: bool = True, check_checksums: bool = True) Self[source]¶
Loads a session from a directory. Assumes that the name of the directory is the name of the session dir and the parent directory is the subject ID and the grandparent directory is the project ID. The scan information is loaded from a YAML along with the scan type, resources and fileformats. If the YAML file is not found or use_manifest is set to True, the session is loaded based on the directory structure.
- Parameters:
session_dir (Path) -- the path to the directory where the session is saved
require_manifiest (bool, optional) -- whether a manifest file is required to load the resources in the session, if true, resources will only be loaded if the manifest file is found, if false, resources will be loaded as FileSet types and checksums will not be checked, by default True
check_checksums (bool, optional) -- whether to check the checksums of the files in the session, by default True
- Returns:
the loaded session
- Return type:
- metadata: Metadata¶
- save(dest_dir: Path, copy_mode: CopyMode = CopyMode.hardlink_or_copy, collation_map: dict[type[FileSet], CopyCollation] | None = None, conversion_map: dict[type[FileSet], tuple[Type[FileSet], dict[str, str]]] | None = None, metadata_allowlist: Collection[str] | None = None, include: Sequence[type[FileSet]] = ()) tuple[Self, Path][source]¶
Saves the session to a directory, in a subdirectory named after its labels (see
staging_dirname()), along with the scans manifest- Parameters:
dest_dir (Path) -- destination directory to save the files to. The session is saved in a subdirectory of it named after its labels (see
staging_dirname())copy_mode (FileSet.CopyMode, optional) -- the mode to use to copy the files that don't need to be deidentified, by default FileSet.CopyMode.hardlink_or_copy
collation_map (dict[type[FileSet], FileSet.CopyCollation] or None) -- maps file-set types to the collation mode used when copying matching resources. If None, the file format's default collation behavior is used.
conversion_map (dict[type[FileSet], tuple[type[FileSet], dict[str, str]]] or None) -- maps source file-set types to a target type and conversion options. Matching resources are converted before they are saved. If None, no conversion is performed.
metadata_allowlist (Collection[str] or None) -- metadata keys to include in the JSON file. If None, all metadata is written. An empty collection writes an empty metadata object. Filtering affects only the saved data; the in-memory metadata is unchanged.
include (sequence[type[FileSet]], optional) -- only save resources matching at least one of these datatypes. An empty sequence saves all resources.
- Returns:
Accession -- a deidentified session with updated paths
Path -- the path to the directory where the session is saved
- class accessionflow.model.acquisition.Acquisition(id: str, type: str | None = None, resources: dict[str, Resource | FileSet] = NOTHING, attached: AttachedFiles | None = None, accession: Accession = None)[source]¶
Bases:
objectRepresentation of a scan to be uploaded to XNAT
- Parameters:
id (str) -- the ID of the scan on XNAT
type (str | None) -- the scan type/description
resources (dict[str, Resource]) -- the list of the resources associated with the scan
attached (AttachedFiles, optional) -- Secondary files that are associated with the given imaging scan (but may not have the necessary metadata to identify the scan itself)
- classmethod load(scan_dir: Path, require_manifest: bool = True, check_checksums: bool = True) Self[source]¶
- metadata: Metadata¶
- save(dest_dir: Path, copy_mode: CopyMode = CopyMode.hardlink_or_copy, collation_map: dict[Type[FileSet], CopyCollation] | None = None, conversion_map: dict[Type[FileSet], tuple[Type[FileSet], dict[str, str]]] | None = None, metadata_allowlist: Collection[str] | None = None, include: Sequence[Type[FileSet]] = (), build_dir: Path | None = None) Self[source]¶
- class accessionflow.model.resource.Resource(name: str, fileset: ~fileformats.core.fileset.FileSet, checksums: dict[str, str] | None | ~accessionflow.model.resource._Unset = <accessionflow.model.resource._Unset object>, acquisition: Acquisition = None)[source]¶
Bases:
objectRepresentation of a resource to be uploaded to XNAT, which is a set of files associated with a scan. There can be multiple resources associated with a scan, e.g. the original DICOM files, a BIDS-converted version of the scan, and a brain mask generated from the scan.
- Parameters:
name (str) -- The name of the resource
fileset (FileSet) -- The set of files associated with the resource
checksums (dict[str, str], optional) -- The checksums of the files in the resource. If not given they are calculated the first time they are needed, so that resources that are saved straight away (e.g. by 'collate') don't hash their files before the copy is hashed
acquisition (Acquisition) -- The acquisition that the resource is associated with
- acquisition: Acquisition¶
- property checksums: dict[str, str] | None¶
The checksums of the files, calculated the first time they are needed if they weren't given. None if they are unknown, i.e. loaded without a manifest
- fileset: FileSet¶
- classmethod load(resource_dir: Path, require_manifest: bool = True, check_checksums: bool = True) Self[source]¶
Load a resource from a directory, reading the manifest file if it exists. If the manifest file doesn't exist and 'require_manifest' is True then an exception is raised, if it is False, then a generic FileSet object is loaded from the files that were found
- metadata: Metadata¶
- save(dest_dir: Path, copy_mode: CopyMode = CopyMode.copy, collation_map: dict[Type[FileSet], CopyCollation] | None = None, conversion_map: dict[Type[FileSet], tuple[Type[FileSet], dict[str, str]]] | None = None, metadata_allowlist: Collection[str] | None = None, calculate_checksums: bool = True, overwrite: bool | None = None, build_dir: Path | None = None) Self[source]¶
Save the resource to a directory
- Parameters:
dest_dir (Path) -- The directory to save the resource
copy_mode (FileSet.CopyMode) -- The method to copy the files
collation_map (dict[ty.Type[FileSet], FileSet.CopyCollation] | None) -- A mapping of FileSet types to CopyCollation objects that specify how to collate files of that type when saving the sessions. If None, the default collation behavior for each FileSet type will be used.
conversion_map (dict[ty.Type[FileSet], tuple[ty.Type[FileSet], dict[str, str]]] | None) -- A mapping of source FileSet types to (target FileSet types, conversion options). When a resource matches a source type, it will be converted to the target type during save, with the options passed through to
convert().metadata_allowlist (Collection[str] or None) -- Metadata keys to include in the JSON file. If None, all metadata is written. An empty collection writes an empty metadata object. Filtering affects only the saved data; the in-memory metadata is unchanged.
calculate_checksums (bool) -- Whether to calculate the checksums of the files
overwrite (bool) -- Whether to overwrite the resource if it already exists, if None then the files are overwritten if they are newer than the ones saved, otherwise a warning is issued, if False an exception will be raised, if True then the resource is saved regardless of the files being newer. An existing resource that is incomplete (no manifest, a manifest that cannot be read, or files that do not match the manifest) is always replaced
build_dir (Path, optional) -- Directory for complete resource builds on the destination filesystem. Session.save supplies a directory outside the visible session. Standalone saves use a hidden directory inside dest_dir.
- Returns:
The saved resource
- Return type:
- Raises:
FileExistsError -- If a complete resource already exists and overwrite is False or None and the files are not newer
- accessionflow.model.accession.recipe_formats_for(datatype: type[FileSet]) list[type[FileSet]] | None[source]¶
Determines the formats that deidentification recipes for the datatype can be loaded from, i.e. the formats given by the
Loaded[<recipe format>]annotation (or union of them, in order of preference) of therecipeargument of its deidentify implementation (e.g.Loaded[DeidRecipeX] | Loaded[DeidRecipe]for DICOM)- Parameters:
datatype (type[FileSet]) -- the type of the filesets to be deidentified
- Returns:
list[type[FileSet]] | None -- the formats the recipe can be loaded from, in order of preference, or None if the implementation doesn't accept a recipe (i.e. its
recipeargument is annotated withNone)Formats whose
contains_phiis "nested" and don't have their own implementationare deidentified by passing the recipe on to the formats nested within them (see
MedicalImagingData.deidentify_nested), so their recipe formats are the ones
common to the nested formats that may contain PHI, as given by the return
annotations of their validated properties.
- Raises:
NotImplementedError -- if there is no deidentify implementation registered for the fileset type (or for one of the formats nested within it if its
contains_phiis "nested")TypeError -- if the implementation doesn't have a
recipeargument annotated with eitherLoaded[<recipe format>](or a union of them) orNone, or the formats nested within a "nested" format don't accept a common recipe format
Plugin configuration¶
- class accessionflow.model.source.Source(name: str, title: str)[source]¶
Bases:
objectWhere the files that are collated into accessions come from.
- class accessionflow.model.store.Store(name: str, title: str, labels: tuple[LabelLevel, ...] | None, acquisition: LabelLevel | None = None, multiple_resources: bool = True, envvar_prefix: str = 'ACFLOW_')[source]¶
Bases:
objectHow sessions are labelled for, and named within, a store.
- Parameters:
name (str) -- the plugin name, used as the subcommand name, e.g. 'xnat' for 'label xnat'
title (str) -- the name of the store for help text, e.g. 'XNAT'
labels (tuple[LabelLevel, ...] or None) -- the levels of the store's hierarchy, from the top, e.g. XNAT's project, subject and session. A labelled session is saved in a folder named by its labels joined by '.', in this order, and 'deidentify' selects recipes by the first label. None if the store has no fixed hierarchy, in which case the levels are named when labelling (with 'label <store> --level NAME FIELD ...')
acquisition (LabelLevel, optional) -- the store's name for an acquisition, and the metadata field to describe each one by, e.g. XNAT's 'scan' described by 'SeriesDescription'. None if the store does not describe acquisitions
multiple_resources (bool) -- whether an acquisition can hold more than one resource (e.g. the DICOM files and a NIfTI conversion of them), as XNAT scans can. If False, each acquisition must hold exactly one, which 'label' checks
envvar_prefix (str) -- the prefix of the environment variables that the options can be set by
- class accessionflow.model.store.LabelLevel(name: str, default: str | None = None, description: str = '')[source]¶
Bases:
objectA level of a store's hierarchy that a session is labelled with, or the store's name for an acquisition within the session.
- Parameters:
name (str) -- the store's name for the level, which names its command-line option (e.g. 'project' gives '--project') and environment variable (e.g. 'ACFLOW_PROJECT')
default (str, optional) -- the metadata field, or a format string over several fields, that the label is extracted from by default
description (str) -- what the label is in the store's terms, for the help text, e.g. 'XNAT project ID'