API

The following functions mirror the command-line tools, and can be called from a Python workflow to run the same stages (see Data model: building your own tools).

Sources

accessionflow.sources.folder.api.collate(input_paths: list[str], output_dir: ~pathlib.Path, datatypes: ~typing.Sequence[type[~fileformats.core.fileset.FileSet]] = (<class 'fileformats.medimage.dicom.DicomSeries'>,), accession_uid: ~typing.Sequence[~accessionflow.helpers.arg_types.IDSpec] = (IDSpec(specifier='StudyInstanceUID', datatype=<class 'fileformats.medimage.dicom.DicomCollection'>),), acquisition_id: ~typing.Sequence[~accessionflow.helpers.arg_types.IDSpec] = (IDSpec(specifier='SeriesNumber', datatype=<class 'fileformats.medimage.dicom.DicomCollection'>),), resource: ~typing.Sequence[~accessionflow.helpers.arg_types.IDSpec] = (), path_metadata_regex: ~typing.Sequence[~accessionflow.helpers.arg_types.PathMetadataRegex] = (), path_metadata: ~typing.Sequence[~accessionflow.helpers.arg_types.PathMetadataPattern] = (), unlink_source: str | None = None, raise_errors: bool = False, copy_mode: ~fileformats.core.fileset.FileSet.CopyMode = CopyMode.hardlink_or_copy, wait_period: int = 0, collation_map: dict[type[~fileformats.core.fileset.FileSet], ~fileformats.core.fileset.FileSet.CopyCollation] | None = None, conversion_map: dict[type[~fileformats.core.fileset.FileSet], tuple[type[~fileformats.core.fileset.FileSet], dict[str, str]]] | None = None, allow_unrecognised: ~typing.Sequence[str] = (), exclude_paths: ~typing.Sequence[str] = (), ignore_datatypes: ~typing.Sequence[type[~fileformats.core.fileset.FileSet]] = (), on_resource_clash: ~typing.Literal['error', 'avoid', 'merge', 'overwrite'] | ~typing.Sequence[~accessionflow.helpers.arg_types.ClashSpec] = 'error', metadata_tables: list[~accessionflow.helpers.arg_types.MetadataTable] | None = None, recursive: bool = False, dont_read_metadata: ~typing.Sequence[type[~fileformats.core.fileset.FileSet]] = ()) → list[str][source]

Collates the input files into accessions/acquisitions/resources and stages them into the staging directory. Project/subject/visit IDs and scan descriptions are not labelled at this point, see the 'label' function for that.

Parameters:
  • input_paths (list[str]) -- List of paths to search for input files: directories, or glob patterns that select the files directly. They must be local or on a mounted file system.

  • output_dir (Path) -- Path to the staging directory where the collated sessions will be saved. This should be a local path.

  • datatypes (ty.Sequence[type[FileSet]]) -- FileSet types to look for in the input files. Only files with these datatypes will be considered for staging. Defaults to (DicomSeries,) to mirror the CLI.

  • accession_uid (ty.Sequence[IDSpec]) -- Field specifications to extract the accession UIDs from the input files, to collate them into separate accessions. Defaults to DEFAULT_ACCESSION_UID_FIELD (StudyInstanceUID scoped to DICOM collections; the same object backs the CLI's --accession); other fileset types need an explicit spec.

  • acquisition_id (ty.Sequence[IDSpec]) -- Field specifications to extract the acquisition IDs from the input files, to collate them into acquisitions. Defaults to DEFAULT_ACQUISITION_ID_FIELD (SeriesNumber scoped to DICOM collections; shared with the CLI's --acquisition); for a fileset whose type is not matched by any spec here the acquisition is named after that fileset's resource.

  • resource (ty.Sequence[IDSpec]) -- List of field specifications to use for extracting the resource IDs from the input files to collate them into resources. If empty, each resource is labelled with the mime-like rendering of its fileset type name, e.g. 'vectra-export', 'sqlite3-db'

  • path_metadata_regex (ty.Sequence[PathMetadataRegex]) -- Regular expressions to extract "metadata" values from resource file paths as named groups. The named groups are used as metadata fields for the resource files, and the extracted values will be used to populate the corresponding metadata fields to complement the metadata read from the file headers.

  • path_metadata (ty.Sequence[PathMetadataPattern]) -- A simpler, glob-like alternative to path_metadata_regex, where {name} placeholders mark the parts of the path to extract as metadata fields, e.g. '{subject}/{session}'. Each pattern is translated into a regex (see path_metadata_pattern_to_regex) and applied before the path_metadata_regex ones, so a regex wins where both set the same field.

  • unlink_source (str | None) -- If "all" or "keep-metadata", the input files will be unlinked one by one after staging (both behave the same here, since the source isn't a directory tree that accessionflow owns). If None, the input files will be left in place.

  • raise_errors (bool) -- If True, any errors encountered during staging will raise an exception. If False, errors will be logged and the staging process will continue for the remaining sessions.

  • copy_mode (FileSet.CopyMode) -- The copy mode to use when saving the sessions. This determines whether files are copied, moved or symlinked when saving the sessions to the staging directory.

  • collation_map (dict[ty.Type[FileSet], FileSet.CopyCollation] | None) -- A mapping of FileSet types to CopyCollation objects that specify how to collate files of that type when saving the sessions. If None, the default collation behavior for each FileSet type will be used.

  • conversion_map (dict[ty.Type[FileSet], tuple[ty.Type[FileSet], dict[str, str]]] | None) -- A mapping of source FileSet types to (target FileSet types, conversion options). When a resource matches a source type, it will be converted to the target type during save, with the options passed through to convert().

  • allow_unrecognised (ty.Sequence[str]) -- Regexes matched against the basename of any input path that no datatype recognised; matches are skipped instead of raising FormatMismatchError. [".*"] tolerates all unrecognised files. Does not affect recognised filesets.

  • exclude_paths (ty.Sequence[str]) -- Globs matched against each input path relative to its input directory, applied before classification so a match is dropped even if a datatype would claim it (e.g. a vendor thumbnail that is a valid image/png). * does not cross /, ** does.

  • ignore_datatypes (ty.Sequence[type[FileSet]]) -- Datatypes expected in the input but not wanted: recognised filesets of these types are dropped rather than raising, and (with recursive) matching directories are skipped without descending. A path matching neither datatypes nor ignore_datatypes (nor allow_unrecognised / exclude_paths) still raises.

  • dont_read_metadata (ty.Sequence[type[FileSet]]) -- Datatypes whose files aren't read for metadata, so their only metadata fields are those from path_metadata/path_metadata_regex and metadata_tables. Saves reading the header of every file when none of its fields are used.

  • wait_period (int) -- If provided, this is the number of seconds that must have passed since the last modification time of the session before it will be staged. This can be used to avoid staging sessions that are still being modified or created.

  • on_resource_clash (OnResourceClash or Sequence[ClashSpec]) -- Behaviour when two filesets resolve to the same scan/resource name. A bare policy string ("error"/"avoid"/"merge"/"overwrite") applies to any clash. A sequence of ClashSpec (policy + datatype scope) resolves each clash with the first spec whose scope covers both filesets - a clash no spec covers raises. "avoid" suffixes, "merge" folds into one SetOf, "overwrite" replaces. Default "error".

  • recursive (bool) -- If True, the input paths will be searched recursively for files to stage. If False, only the files directly within the input paths will be considered for staging.

  • metadata_tables (list[MetadataTable] | None) --

    Specify metadata tables to extract and join metadata from input files (ACFLOW_METADATA_TABLES env. var). The 'path' arg specifies the location of the metadata table file. Its format is auto-detected as CSV or TSV from the file extension; a different format can be forced by appending its mime-type in square brackets, e.g. 'path/to/table.dat[text/csv]'. The "row frequency" arg specifies what each row in the metadata table corresponds to in the data hierarchy, and can be one of 'accession', 'acquisition', 'resource', 'fileset', 'fileset[<mime-type>]'. When one or more mime-types are given in square brackets after 'fileset' they restrict the join to input files of those types (multiple mime-types can be '|'-separated, e.g. 'fileset[image/png|image/jpeg]'); a bare 'fileset' matches any input file. The 'join-exprs' arg is a comma-separated list of '<column-name>=<cell-value>' expressions; a row is a match when every expression holds. The '<cell-value>' is either the name of an existing metadata field or a Python format string over one or more metadata fields, e.g. '{PatientID}_{SessionID}'. All columns of the matched row are then merged into the target's metadata. The example below extracts the relative path of an image file with path_metadata_regex and uses it to join a table whose 'ImagePath' column holds spreadsheet HYPERLINK() formulas:

    collate(
        ...,
        path_metadata_regex=[
            PathMetadataRegex(
                regex=r".*/(?P<relpath>[\w-]+/[\w-]+\.(?:png|jpg))",
                datatype="image/png|image/jpeg",
            )
        ],
        metadata_tables=[
            MetadataTable(
                table_file="path/to/table.csv[text/csv]",
                row_frequency="fileset[image/png|image/jpeg]",
                join_exprs='ImagePath=HYPERLINK("{relpath}")',
            )
        ],
    )
    

accessionflow.sources.orthanc.api.collate(url: str, store_dir: Path, output_dir: Path, user: str, password: str, to_process_label: str | None = None, processed_label: str | None = None, unlink_source: str | None = None, raise_errors: bool = False, copy_mode: CopyMode = CopyMode.hardlink_or_copy, wait_period: int = 0, conversion_map: dict[type[FileSet], tuple[type[FileSet], dict[str, str]]] | None = None) → list[str][source]

Collates the DICOM studies held by an Orthanc server into accessions in the staging directory, hardlinking the files from Orthanc's storage directory.

Parameters:
  • url (str) -- the base URL of the Orthanc REST API, e.g. 'http://orthanc:8042'

  • store_dir (Path) -- Orthanc's StorageDirectory as mounted, which must be on the same file system as output_dir for the files to be hardlinked

  • output_dir (Path) -- the staging directory the collated accessions are saved in

  • user (str) -- the Orthanc user to log in as

  • password (str) -- the password of the Orthanc user

  • to_process_label (str | None) -- the label applied to studies in Orthanc, by something else, to mark them for collation. If None, all studies are collated

  • processed_label (str | None) -- the label applied to studies in Orthanc once they have been collated, so that they aren't collated again

  • unlink_source (str | None) -- if "all" or "keep-metadata", the studies would be removed from Orthanc once they have been collated. Not yet implemented

  • raise_errors (bool) -- whether to raise errors instead of logging them. Not yet implemented

  • copy_mode (FileSet.CopyMode) -- how the files are copied. Only hardlinks are implemented

  • wait_period (int) -- the number of seconds since Orthanc last updated a study before it is collated, so that studies still being received aren't collated partially

  • conversion_map (dict[type[FileSet], tuple[type[FileSet], dict[str, str]]] | None) -- the types series are converted to before the study is labelled as processed, with the options passed through to convert()

Returns:

the IDs of the studies that couldn't be collated

Return type:

list[str]

accessionflow.api.save_accessions_to_dir(accessions: list[Accession], msg: str, build_dir: Path, copy_mode: CopyMode, output_dir: Path, wait_period: int = 0, collation_map: dict[type[FileSet], CopyCollation] | None = None, conversion_map: dict[type[FileSet], tuple[Type[FileSet], dict[str, str]]] | None = None, unlink_source: str | None = None, raise_errors: bool = False) → list[str][source]

Save collated accessions into a build directory, then move each one into the staging directory once it is complete, so that a later stage never sees a partly saved accession.

Parameters:
  • accessions (list[Accession]) -- the collated accessions, referencing their files at the source

  • msg (str) -- the description of the progress bar

  • build_dir (Path) -- the directory the accessions are saved into before they are moved

  • copy_mode (FileSet.CopyMode) -- how the files are copied from the source

  • output_dir (Path) -- the staging directory the accessions are moved into

  • wait_period (int) -- the number of seconds since an accession's files were last modified before it is saved, so that accessions still being transferred are skipped

  • collation_map (dict[type[FileSet], FileSet.CopyCollation], optional) -- how files of each type are laid out in their resource directories

  • conversion_map (dict[type[FileSet], tuple[type[FileSet], dict[str, str]]], optional) -- the types resources are converted to as they are saved, with the options passed to convert()

  • unlink_source (str, optional) -- if "all" or "keep-metadata", each accession's source files are removed once it is saved

  • raise_errors (bool) -- whether to raise errors instead of logging them and moving on to the next accession

Returns:

the errors, one per accession that couldn't be saved

Return type:

list[str]

Stages

accessionflow.api.attach(input_dir: Path, output_dir: Path, datatype: Type[FileSet], glob: str, identity_pattern: str, spaces_to_underscores: bool = False, on_resource_clash: Literal['error', 'avoid', 'merge', 'overwrite'] = 'error', raise_errors: bool = False, require_manifest: bool = True, copy_mode: CopyMode = CopyMode.copy, unlink_source: str | None = None) → list[str][source]
accessionflow.api.label(input_dir: Path, output_dir: Path, label_fields: Mapping[str, str], acquisition_field: str | None = None, constant_labels: Mapping[str, str] | None = None, store_type: str | None = None, copy_mode: CopyMode = CopyMode.hardlink_or_copy, unlink_source: str | None = None, raise_errors: bool = False, include: Sequence[type[FileSet]] = (), single_resource: bool = False) → list[str][source]

Sorts the input files into sessions and stages them into the staging directory.

Parameters:
  • input_dir (Path) -- List of paths to search for input files. Can be local paths or S3 paths.

  • output_dir (Path) -- Path to the staging directory where the sorted sessions will be saved. This should be a local path.

  • label_fields (Mapping[str, str]) -- The metadata field (or a format string over several fields) to extract each label from, keyed by the name of its level in the data store, in order from the top level, e.g. {"project": "StudyComments", "subject": "PatientID", "session": "AccessionNumber"} for XNAT

  • acquisition_field (str | None) -- Field name to use for extracting a description for each acquisition (e.g. the XNAT scan type). Acquisitions for which the field can't be resolved are left without a description.

  • constant_labels (Mapping[str, str] | None) -- Labels to use for all sessions instead of extracting them from the metadata, keyed by level name, e.g. {"project": "MYPROJECT"}. Useful for instruments that deposit in a single project.

  • store_type (str | None) -- The name of the data store the labels are for (e.g. 'xnat'), recorded with the session so it isn't deposited in a different kind of store

  • copy_mode (FileSet.CopyMode) -- The copy mode to use when saving the sessions. This determines whether files are copied, moved or symlinked when saving the sessions to the staging directory.

  • unlink_source (str | None) -- If "all" or "keep-metadata", record files present before label loads the session. After a successful save, remove those files and empty directories. With "keep-metadata", remove files only inside loaded resource directories; leave session and scan metadata in place. Files at new paths remain. If None, the collated session directory is left in place.

  • raise_errors (bool) -- If True, any errors encountered during staging will raise an exception. If False, errors will be logged and the staging process will continue for the remaining sessions.

  • include (Sequence[type[FileSet]]) -- Only stage resources matching at least one of these datatypes. An empty sequence stages all resources.

  • single_resource (bool) -- If True, the store holds a single resource per acquisition, so a session with an acquisition that would be saved with more than one resource (after filtering by include) is an error, and is left in place to be reprocessed (e.g. with include set to select one).

Notes

If any of a session's labels can't be resolved from its metadata, it is saved with placeholder IDs (see Accession.label) under an INVALID_NAME_DEFAULT ('__invalid__') subdirectory of output_dir instead of the regular output location, so it can be found and manually reprocessed rather than being lost.

accessionflow.api.deidentify(input_dir: Path, output_dir: Path, recipe_dir: Path, reid_dir: Path | None = None, on_resource_clash: Literal['error', 'avoid', 'merge', 'overwrite'] = 'error', raise_errors: bool = False, copy_mode: CopyMode = CopyMode.copy, require_manifest: bool = True, unlink_source: str | None = None, reid_encrypt_key: bytes | None = None, max_workers: int | None = None, metadata_allowlist: Collection[str] = {'__uid__'}, hash_uid_key: str | None = None) → list[str][source]
Parameters:
  • reid_dir (Path, optional) -- the directory to write the re-identification metadata to, one JSON file per session. If not given, the re-identification metadata is discarded rather than written to disk (use this when retaining the original<->deidentified mapping would itself be a security concern).

  • max_workers (int, optional) -- the number of threads handed to a resource's own deidentify implementation to parallelise work within that resource (e.g. the per-file loop for a DICOM series). Ignored by formats that don't support it.

  • hash_uid_key (str, optional) -- the secret key used to replace each session's UID (by default its DICOM StudyInstanceUID) with a keyed hash of it in the output. The same UID and key always give the same hash, so a session is still recognised when it is uploaded again. If not given, the original UID is kept in the output.

accessionflow.api.package(input_dir: Path, output_dir: Path, source_datatype: type[FileSet], output_resources: Mapping[str, tuple[type[FileSet], Mapping[str, Any]]], on_resource_clash: Literal['error', 'avoid', 'merge', 'overwrite'] = 'error', raise_errors: bool = False, require_manifest: bool = True, copy_mode: CopyMode = CopyMode.hardlink_or_copy, unlink_source: str | None = None) → list[str][source]

Replace matching scan resources with multiple named converted resources.

Conversions are materialised only after a staged session has been loaded, so this stage can be placed after deidentification. All output conversions are independent and receive the original source fileset.

accessionflow.api.annotate_api.annotate(input_dir: Path, output_dir: Path, extract: Mapping[str, IDSpec], raise_errors: bool = False) → list[str][source]

Annotates the accessions with fields extracted from their metadata, which the data store records alongside them (e.g. as custom variables on XNAT sessions).

Data stores

accessionflow.stores.folder.api.store(input_dir: str | Path, dest: str, storage_options: Mapping[str, Any] | None = None, wait_period: int = 0, require_manifest: bool = True, overwrite: bool = False, raise_errors: bool = False, store_credentials: StoreCredentials | None = None, temp_dir: Path | None = None, s3_max_workers: int | None = None) → list[str][source]

Deposits the labelled accessions in a staging directory in a folder store: nested label folders on a local path, a mount, or a remote file system such as S3 or SFTP (see accessionflow.stores.folder.layout for the layout).

Depositing again is cheap: a resource whose manifest in the store matches the staged one is skipped, and one left without a manifest by an interrupted deposit is deposited again.

Parameters:
  • input_dir (str or Path) -- the staging directory holding the labelled accessions (the output of 'label folder', or of 'deidentify' after it), or an S3 bucket ('s3://...') to download them from

  • dest (str) -- the root of the store, a path or an fsspec URL, e.g. '/mnt/archive', 's3://bucket/archive' or 'sftp://host/archive'

  • storage_options (Mapping[str, Any], optional) -- options for the fsspec file system of the store, e.g. credentials

  • wait_period (int) -- the number of seconds since an accession was last modified before it is deposited, so one that is still being written isn't

  • require_manifest (bool) -- whether the staged resources must have manifests. If False, the checksums of resources without one are worked out from their files

  • overwrite (bool) -- whether to replace a resource that is already in the store with different files. If False, it is reported as an error and left as it is

  • raise_errors (bool) -- whether to raise errors instead of logging them and continuing with the next accession

  • store_credentials (StoreCredentials, optional) -- the credentials of the S3 bucket the accessions are staged in, if they are

  • temp_dir (Path, optional) -- the directory to download accessions staged in S3 to

  • s3_max_workers (int, optional) -- the number of files to download from an S3 staging bucket at once

Returns:

the errors encountered, one per accession that couldn't be deposited

Return type:

list[str]

accessionflow.stores.folder.api.check_store(input_dir: str | Path, dest: str, storage_options: Mapping[str, Any] | None = None, checksums: bool = True, require_manifest: bool = True, raise_errors: bool = False, store_credentials: StoreCredentials | None = None, temp_dir: Path | None = None, s3_max_workers: int | None = None) → list[str][source]

Checks what has been deposited in a folder store against the staging directory it was deposited from, and logs every resource that is missing, incomplete or different.

Parameters:
  • input_dir (str or Path) -- the staging directory the accessions were deposited from, or an S3 bucket

  • dest (str) -- the root of the store, a path or an fsspec URL, e.g. 's3://bucket/archive'

  • storage_options (Mapping[str, Any], optional) -- options for the fsspec file system of the store, e.g. credentials

  • checksums (bool) -- whether to check the files' checksums, which reads every file back from the store. If False, only their sizes are compared, which is much cheaper

  • require_manifest (bool) -- whether the staged resources must have manifests

  • raise_errors (bool) -- whether to raise errors instead of logging them

  • store_credentials (StoreCredentials, optional) -- the credentials of the S3 bucket the accessions are staged in, if they are

  • temp_dir (Path, optional) -- the directory to download accessions staged in S3 to

  • s3_max_workers (int, optional) -- the number of files to download from an S3 staging bucket at once

Returns:

a description of each problem found

Return type:

list[str]

accessionflow.stores.xnat.api.store(input_dir: str, xnat_repo: Xnat, always_include: Sequence[str | FileSet] = (), store_credentials: StoreCredentials | None = None, require_manifest: bool = True, methods: Sequence[UploadMethod] = (), wait_period: int = 0, num_files_per_batch: int = 0, check_checksums: bool = True, s3_cache_dir: Path | None = None, raise_errors: bool = False, dry_run: bool = False, s3_max_workers: int = 10, xnat_max_workers: int = 10, modality: str | None = None, date_field: str | None = None, skip_uid_check: bool = False) → list[str][source]

Upload sorted sessions in the given staging directory to XNAT

Parameters:
  • input_dir (Path) -- The directory containing the sessions to upload. Each session should be in a separate subdirectory.

  • xnat_repo (Xnat) -- The XNAT repository to upload to

  • always_include (Sequence[str]) -- A sequence of scan types or file paths to always include in the upload regardless of whether they are explicitly specified in the frameset definition

  • raise_errors (bool) -- Whether to raise errors that occur during upload or to log them and continue with the next session

  • store_credentials (StoreCredentials) -- Whether to store credentials for accessing staging directories that require authentication (e.g. S3)

  • require_manifest (bool) -- Whether to require a manifest file in each session directory that specifies the resources to upload and their checksums

  • methods (Sequence[UploadMethod]) -- The upload method to use for each datatype (e.g. 'tgz_file' or 'directory')

  • wait_period (int) -- The minimum age in seconds of session directories to upload (only applicable for local staging directories)

  • num_files_per_batch (int) -- The number of files to upload in each batch when uploading resources with the 'directory' method (if 0, all files will be uploaded in a single batch)

  • check_checksums (bool) -- Whether to check checksums of uploaded resources against the checksums specified in the manifest file and the checksums of the files in the staged resources (if available) to verify that they were uploaded correctly

  • dry_run (bool) -- Whether to list the sessions that would be uploaded instead of actually uploading them

  • s3_max_workers (int, optional) -- The maximum number of concurrent S3 object downloads. Defaults to 10.

  • xnat_max_workers (int, optional) -- The maximum number of concurrent XNAT resource uploads. Defaults to 10.

  • modality (str, optional) -- The modality of the sessions to upload (e.g. 'MR', 'CT'). Defaults to the "Modality" metadata field if present otherwise an error is raised.

  • skip_uid_check (bool, optional) -- don't compare a session's UID with the one recorded on an existing XNAT session, for use while moving to hashed UIDs. The subject is still checked where known.

  • date_field (str, optional) -- Session metadata field to use as the XNAT session date when creating a session. Existing XNAT sessions are not modified.

accessionflow.stores.xnat.api.check_store(input_dir: str, server: str, user: str, password: str, store_credentials: ~accessionflow.helpers.arg_types.StoreCredentials, always_include: ~typing.Sequence[str | ~typing.Type[~fileformats.core.fileset.FileSet]] = (<class 'fileformats.medimage.dicom.DicomSeries'>, ), temp_dir: ~pathlib.Path | None = None, verify_ssl: bool = True, use_curl_jsession: bool = False, disable_progress: bool = False, s3_max_workers: int = 10, xnat_max_workers: int = 10, skip_uid_check: bool = False) → None[source]

Checks the staged sessions against the XNAT server to check for any issues before upload.

This is intended to be used as a pre-upload check to identify any issues with the staged sessions before attempting to upload them to XNAT. It checks that the project, session, scan and resource exist on XNAT and that the checksums of the files in the resource match those on XNAT. If any issues are found, they are logged and a summary of the issues is logged at the end. If the --raise-errors flag is set, then any issues will raise an exception instead of being logged.

Parameters:
  • input_dir (str) -- The directory containing the staged sessions to check.

  • server (str) -- The URL of the XNAT server to check against.

  • user (str) -- The username to use when connecting to the XNAT server.

  • password (str) -- The password to use when connecting to the XNAT server.

  • loggers (list[LoggerConfig]) -- The logger configurations to use for logging the output of the function.

  • always_include (list[str]) -- A list of MIME types to always include in the check, even if they aren't defined in the frameset on XNAT. This can be used to include additional file formats that aren't defined in the frameset, or to include all file formats by using "all".

  • s3_max_workers (int, optional) -- the maximum number of concurrent S3 object downloads. Defaults to 10.

  • xnat_max_workers (int, optional) -- the maximum number of concurrent XNAT checksum requests. Defaults to 10.

Model classes

These classes represent an accession as it moves through the stages: an accession contains acquisitions, each of which contains one or more resources, e.g. the original DICOM files and a derived NIfTI conversion (see Accessions, acquisitions and resources).

class accessionflow.model.accession.Accession(uid: str, labels: Mapping[str, str] = NOTHING, store_type: str | None = None, acquisitions: Sequence[Acquisition] | Dict[str, Acquisition] = NOTHING, session_resources: Dict[str, Resource] = NOTHING, run_uid: str | None = None)[source]

Bases: object

Representation of an accession to be deposited in a data store: the data from one experiment (e.g. an imaging session of a patient, a preclinical scan, or the imaging of a specimen or phantom), as a set of acquisitions that belong together under the same labels.

Parameters:
  • labels (dict[str, str]) -- The labels the accession is deposited under in the data store, keyed by the names of the store's label levels in order from the top, e.g. {"project": "P1", "subject": "S1", "session": "V1"} for XNAT. Empty until the accession is labelled

  • store_type (str, optional) -- The name of the data store the accession was labelled for, e.g. 'xnat'

  • acquisitions (ty.Dict[str, Acquisition]) -- The acquisitions in the accession

  • run_uid (ty.Optional[str]) -- The run UID of the session, if it exists

acquisitions: Dict[str, Acquisition]
attach_files(patterns: list[AttachedFiles], spaces_to_underscores: bool = True, on_resource_clash: Literal['error', 'avoid', 'merge', 'overwrite'] = 'error') → list[FileSet][source]

Adds files associated with the primary files to the session

Parameters:
  • patterns (list[AttachedFiles]) -- list of patterns to attach files to the primary files in the session

  • spaces_to_underscores (bool, optional) -- when building attached file globs, convert spaces underscores in fields extracted from source file metadata, false by default

deidentify(dest_dir: Path, recipe_files: dict[type[FileSet], Path] | None = None, copy_mode: CopyMode = CopyMode.hardlink_or_copy, on_resource_clash: Literal['error', 'avoid', 'merge', 'overwrite'] = 'error', require_matching_recipe_file: bool = True, max_workers: int | None = None) → tuple[Self, dict[str, Any]][source]

Creates a new session with deidentified images

Parameters:
  • dest_dir (Path) -- the directory to save the deidentified files into

  • recipe_files (dict[type[FileSet], Path], optional) -- project-specific deidentification recipe files for the different file types within the imaging session. The keys are the file types (see https://arcanaframework.github.io/fileformats/) and the values are paths to the recipe files, which are loaded into the format expected by the deidentify implementation for each file type (e.g. DeidRecipe for DICOM)

  • copy_mode (FileSet.CopyMode, optional) -- the mode to use to copy the files that don't need to be deidentified, by default FileSet.CopyMode.hardlink_or_copy

  • on_resource_clash (OnResourceClash, optional) -- when copying a file that doesn't need to be deidentified, if "avoid", if a resource with the same name already exists in the scan, increment the resource name by appending _1, _2 etc. to the name until a unique name is found, by default "avoid" if "merge", existing sessions with the same name will be merged. if "error", an error will be raised if a session with the same name already exists in the staging directory. if "overwrite", an existing resource with the same name will be overwritten.

  • require_matching_recipe_file (bool, optional) -- whether to require a matching recipe file for each fileset whose deidentify implementation accepts a recipe, by default True

  • max_workers (int, optional) -- passed through as max_workers to each resource's FileSet.deidentify, for formats whose deidentification implementation can parallelise work within a single resource (e.g. the per-file loop for a DICOM series) using threads. Formats that don't accept/use it just ignore it. Resources themselves are deidentified/copied sequentially, one at a time, to keep failures easy to trace back to the resource that caused them.

Returns:

  • Accession -- a new session with deidentified images

  • dict[str, Any] -- a mapping containing the original values of metadata fields that have been removed or modified

fields: Fields
classmethod from_paths(files_path: str | Path | Sequence[str | Path], datatypes: type[FileSet] | Sequence[type[FileSet]], accession_uid_field: Sequence[IDSpec], acquisition_id_field: Sequence[IDSpec] = (), resource_field: Sequence[IDSpec] = (), recursive: bool = False, on_resource_clash: Literal['error', 'avoid', 'merge', 'overwrite'] | Sequence[ClashSpec] = 'error', allow_unrecognised: Sequence[str] | None = None, exclude_paths: Sequence[str] | None = None, ignore_datatypes: Sequence[type[FileSet]] | None = None, path_metadata_regex: Sequence[PathMetadataRegex] = (), metadata_tables: list[MetadataTable] | None = None, dont_read_metadata: Sequence[type[FileSet]] = ()) → list[Self][source]

Loads all imaging sessions from a list of DICOM files

Parameters:
  • files_path (str or Path) -- Path to a directory containing the resources to load the sessions from, or a glob string that selects the paths

  • datatypes (type or list[type]) -- the fileformats to load from the paths, e.g. DicomSeries or [DicomSeries, NiftiGz]

  • accession_uid_field (ty.Sequence[IDSpec]) -- the metadata field that uniquely identifies the accession, used to collate files together before they are labelled (e.g. StudyInstanceUID)

  • acquisition_id_field (ty.Sequence[IDSpec]) -- the value of this field is used to collate resources under single acquisitions. For a fileset whose type is not matched by any spec here (including the empty default), the acquisition is named after that fileset's resource.

  • resource_field (ty.Sequence[IDSpec]) -- the value of this field is used to identify resources. If empty, the resource is labelled with the mime-like rendering of the fileset's type name, e.g. 'vectra-export', 'sqlite3-db'

  • recursive (bool, optional) -- recurse into directories passed as file paths. For file datatypes this flattens the tree; when any datatypes / ignore_datatypes entry is a directory format the walk stops descending into a directory as soon as it validates as one of them (yielding a datatypes match whole, skipping an ignore_datatypes match whole). generic/directory and generic/file-set cannot be used as datatypes with recursive. By default False

  • on_resource_clash (OnResourceClash or Sequence[ClashSpec], optional) -- how to handle two filesets resolving to the same scan/resource name. A bare policy string ("error"/"avoid"/"merge"/"overwrite") applies to any clash. A sequence of ClashSpec (policy + datatype scope) resolves each clash with the first spec whose scope covers both filesets; "merge" folds them into a SetOf, "avoid" suffixes, "overwrite" replaces; a clash no spec covers raises. Default "error".

  • allow_unrecognised (Sequence[str] or None, optional) -- regular expressions matched against the basename of any input path that no datatype recognised - matches are skipped instead of raising FormatMismatchError. Does not affect recognised filesets.

  • exclude_paths (Sequence[str] or None, optional) -- glob patterns matched against each input path relative to its input directory, applied before classification so a match is dropped even if a datatype would claim it (e.g. a vendor thumbnail that is a valid image/png). * does not cross /, ** does.

  • ignore_datatypes (ty.Sequence[type[FileSet]] or None, optional) -- datatypes that are expected in the input but not wanted: recognised filesets of these types are dropped from the result rather than raising, and (when recursive) matching directories are skipped without descending. An input path matching neither datatypes nor ignore_datatypes (nor allow_unrecognised/exclude_paths) still raises.

  • path_metadata_regex (ty.Sequence[PathMetadataRegex], optional) -- Regular expressions to extract "metadata" values from resource file paths as named groups. The named groups are used as metadata fields for the resource files, and the extracted values will be used to populate the corresponding metadata fields to complement the metadata read from the file headers.

  • metadata_tables (list[MetadataTable] or None, optional) -- a list of MetadataTable objects that define how to extract metadata from input files (e.g. CSV files and spreadsheets) and join them with the sessions. If None, no metadata tables will be used.

  • dont_read_metadata (ty.Sequence[type[FileSet]], optional) -- datatypes whose files aren't read for metadata (matched with isinstance), so their only metadata fields are those from path_metadata_regex and metadata_tables. Reading the headers of many files can be slow, e.g. on a network file system, and is wasted when none of their fields are used.

Returns:

all imaging sessions that are present in list of dicom paths

Return type:

list[Accession]

Raises:

AccessionParseError -- if values extracted from IDs across the DICOM scans are not consistent across DICOM files within the session

label(label_fields: Mapping[str, str], acquisition_field: str | None = None, constant_labels: Mapping[str, str] | None = None, store_type: str | None = None) → dict[str, str][source]

Assigns the labels the accession is deposited under in a data store, extracted from its metadata. Also resolves a description for each acquisition in the accession, if 'acquisition_field' is provided.

Parameters:
  • label_fields (Mapping[str, str]) -- the metadata field (or a format string over several fields) to extract each label from, keyed by the name of its level in the data store, in order from the top level, e.g. {"project": "StudyComments", "subject": "PatientID", "session": "AccessionNumber"} for XNAT

  • acquisition_field (str, optional) -- metadata field to extract a description for each acquisition from. Acquisitions for which the field can't be resolved are left without a description (saved with a trailing-dot '<scan_id>.' directory name)

  • constant_labels (Mapping[str, str], optional) -- labels to use as given rather than extract from the metadata, keyed by level name, e.g. {"project": "MYPROJECT"}

  • store_type (str, optional) -- the name of the data store the labels are for, e.g. 'xnat', recorded so the accession isn't deposited in a different kind of store

Returns:

the metadata fields that couldn't be resolved, mapped to the placeholder labels used instead

Return type:

dict[str, str]

Notes

If a label's field can't be resolved from the accession's metadata, a unique 'INVALID_MISSING_<FIELD>_<random>' placeholder is used instead of raising, so the accession can still be saved (see invalid_ids) for manual review/reprocessing rather than being silently dropped.

labels: dict[str, str]
classmethod load(session_dir: Path, require_manifest: bool = True, check_checksums: bool = True) → Self[source]

Loads a session from a directory. Assumes that the name of the directory is the name of the session dir and the parent directory is the subject ID and the grandparent directory is the project ID. The scan information is loaded from a YAML along with the scan type, resources and fileformats. If the YAML file is not found or use_manifest is set to True, the session is loaded based on the directory structure.

Parameters:
  • session_dir (Path) -- the path to the directory where the session is saved

  • require_manifiest (bool, optional) -- whether a manifest file is required to load the resources in the session, if true, resources will only be loaded if the manifest file is found, if false, resources will be loaded as FileSet types and checksums will not be checked, by default True

  • check_checksums (bool, optional) -- whether to check the checksums of the files in the session, by default True

Returns:

the loaded session

Return type:

Accession

metadata: Metadata
property name: str | None
run_uid: str | None
save(dest_dir: Path, copy_mode: CopyMode = CopyMode.hardlink_or_copy, collation_map: dict[type[FileSet], CopyCollation] | None = None, conversion_map: dict[type[FileSet], tuple[Type[FileSet], dict[str, str]]] | None = None, metadata_allowlist: Collection[str] | None = None, include: Sequence[type[FileSet]] = ()) → tuple[Self, Path][source]

Saves the session to a directory, in a subdirectory named after its labels (see staging_dirname()), along with the scans manifest

Parameters:
  • dest_dir (Path) -- destination directory to save the files to. The session is saved in a subdirectory of it named after its labels (see staging_dirname())

  • copy_mode (FileSet.CopyMode, optional) -- the mode to use to copy the files that don't need to be deidentified, by default FileSet.CopyMode.hardlink_or_copy

  • collation_map (dict[type[FileSet], FileSet.CopyCollation] or None) -- maps file-set types to the collation mode used when copying matching resources. If None, the file format's default collation behavior is used.

  • conversion_map (dict[type[FileSet], tuple[type[FileSet], dict[str, str]]] or None) -- maps source file-set types to a target type and conversion options. Matching resources are converted before they are saved. If None, no conversion is performed.

  • metadata_allowlist (Collection[str] or None) -- metadata keys to include in the JSON file. If None, all metadata is written. An empty collection writes an empty metadata object. Filtering affects only the saved data; the in-memory metadata is unchanged.

  • include (sequence[type[FileSet]], optional) -- only save resources matching at least one of these datatypes. An empty sequence saves all resources.

Returns:

  • Accession -- a deidentified session with updated paths

  • Path -- the path to the directory where the session is saved

session_resources: Dict[str, Resource]
store_type: str | None
uid: str
class accessionflow.model.acquisition.Acquisition(id: str, type: str | None = None, resources: dict[str, Resource | FileSet] = NOTHING, attached: AttachedFiles | None = None, accession: Accession = None)[source]

Bases: object

Representation of a scan to be uploaded to XNAT

Parameters:
  • id (str) -- the ID of the scan on XNAT

  • type (str | None) -- the scan type/description

  • resources (dict[str, Resource]) -- the list of the resources associated with the scan

  • attached (AttachedFiles, optional) -- Secondary files that are associated with the given imaging scan (but may not have the necessary metadata to identify the scan itself)

attached: AttachedFiles | None
id: str
classmethod load(scan_dir: Path, require_manifest: bool = True, check_checksums: bool = True) → Self[source]
metadata: Metadata
resources: Dict[str, Resource]
save(dest_dir: Path, copy_mode: CopyMode = CopyMode.hardlink_or_copy, collation_map: dict[Type[FileSet], CopyCollation] | None = None, conversion_map: dict[Type[FileSet], tuple[Type[FileSet], dict[str, str]]] | None = None, metadata_allowlist: Collection[str] | None = None, include: Sequence[Type[FileSet]] = (), build_dir: Path | None = None) → Self[source]
type: str
class accessionflow.model.resource.Resource(name: str, fileset: ~fileformats.core.fileset.FileSet, checksums: dict[str, str] | None | ~accessionflow.model.resource._Unset = <accessionflow.model.resource._Unset object>, acquisition: Acquisition = None)[source]

Bases: object

Representation of a resource to be uploaded to XNAT, which is a set of files associated with a scan. There can be multiple resources associated with a scan, e.g. the original DICOM files, a BIDS-converted version of the scan, and a brain mask generated from the scan.

Parameters:
  • name (str) -- The name of the resource

  • fileset (FileSet) -- The set of files associated with the resource

  • checksums (dict[str, str], optional) -- The checksums of the files in the resource. If not given they are calculated the first time they are needed, so that resources that are saved straight away (e.g. by 'collate') don't hash their files before the copy is hashed

  • acquisition (Acquisition) -- The acquisition that the resource is associated with

acquisition: Acquisition
property checksums: dict[str, str] | None

The checksums of the files, calculated the first time they are needed if they weren't given. None if they are unknown, i.e. loaded without a manifest

fileset: FileSet
classmethod load(resource_dir: Path, require_manifest: bool = True, check_checksums: bool = True) → Self[source]

Load a resource from a directory, reading the manifest file if it exists. If the manifest file doesn't exist and 'require_manifest' is True then an exception is raised, if it is False, then a generic FileSet object is loaded from the files that were found

metadata: Metadata
name: str
save(dest_dir: Path, copy_mode: CopyMode = CopyMode.copy, collation_map: dict[Type[FileSet], CopyCollation] | None = None, conversion_map: dict[Type[FileSet], tuple[Type[FileSet], dict[str, str]]] | None = None, metadata_allowlist: Collection[str] | None = None, calculate_checksums: bool = True, overwrite: bool | None = None, build_dir: Path | None = None) → Self[source]

Save the resource to a directory

Parameters:
  • dest_dir (Path) -- The directory to save the resource

  • copy_mode (FileSet.CopyMode) -- The method to copy the files

  • collation_map (dict[ty.Type[FileSet], FileSet.CopyCollation] | None) -- A mapping of FileSet types to CopyCollation objects that specify how to collate files of that type when saving the sessions. If None, the default collation behavior for each FileSet type will be used.

  • conversion_map (dict[ty.Type[FileSet], tuple[ty.Type[FileSet], dict[str, str]]] | None) -- A mapping of source FileSet types to (target FileSet types, conversion options). When a resource matches a source type, it will be converted to the target type during save, with the options passed through to convert().

  • metadata_allowlist (Collection[str] or None) -- Metadata keys to include in the JSON file. If None, all metadata is written. An empty collection writes an empty metadata object. Filtering affects only the saved data; the in-memory metadata is unchanged.

  • calculate_checksums (bool) -- Whether to calculate the checksums of the files

  • overwrite (bool) -- Whether to overwrite the resource if it already exists, if None then the files are overwritten if they are newer than the ones saved, otherwise a warning is issued, if False an exception will be raised, if True then the resource is saved regardless of the files being newer. An existing resource that is incomplete (no manifest, a manifest that cannot be read, or files that do not match the manifest) is always replaced

  • build_dir (Path, optional) -- Directory for complete resource builds on the destination filesystem. Session.save supplies a directory outside the visible session. Standalone saves use a hidden directory inside dest_dir.

Returns:

The saved resource

Return type:

Resource

Raises:

FileExistsError -- If a complete resource already exists and overwrite is False or None and the files are not newer

accessionflow.model.accession.recipe_formats_for(datatype: type[FileSet]) → list[type[FileSet]] | None[source]

Determines the formats that deidentification recipes for the datatype can be loaded from, i.e. the formats given by the Loaded[<recipe format>] annotation (or union of them, in order of preference) of the recipe argument of its deidentify implementation (e.g. Loaded[DeidRecipeX] | Loaded[DeidRecipe] for DICOM)

Parameters:

datatype (type[FileSet]) -- the type of the filesets to be deidentified

Returns:

  • list[type[FileSet]] | None -- the formats the recipe can be loaded from, in order of preference, or None if the implementation doesn't accept a recipe (i.e. its recipe argument is annotated with None)

  • Formats whose contains_phi is "nested" and don't have their own implementation

  • are deidentified by passing the recipe on to the formats nested within them (see

  • MedicalImagingData.deidentify_nested), so their recipe formats are the ones

  • common to the nested formats that may contain PHI, as given by the return

  • annotations of their validated properties.

Raises:
  • NotImplementedError -- if there is no deidentify implementation registered for the fileset type (or for one of the formats nested within it if its contains_phi is "nested")

  • TypeError -- if the implementation doesn't have a recipe argument annotated with either Loaded[<recipe format>] (or a union of them) or None, or the formats nested within a "nested" format don't accept a common recipe format

Plugin configuration

class accessionflow.model.source.Source(name: str, title: str)[source]

Bases: object

Where the files that are collated into accessions come from.

Parameters:
  • name (str) -- the plugin name, used as the subcommand name, e.g. 'orthanc' for 'collate orthanc'

  • title (str) -- the name of the source for help text, e.g. 'Orthanc'

class accessionflow.model.store.Store(name: str, title: str, labels: tuple[LabelLevel, ...] | None, acquisition: LabelLevel | None = None, multiple_resources: bool = True, envvar_prefix: str = 'ACFLOW_')[source]

Bases: object

How sessions are labelled for, and named within, a store.

Parameters:
  • name (str) -- the plugin name, used as the subcommand name, e.g. 'xnat' for 'label xnat'

  • title (str) -- the name of the store for help text, e.g. 'XNAT'

  • labels (tuple[LabelLevel, ...] or None) -- the levels of the store's hierarchy, from the top, e.g. XNAT's project, subject and session. A labelled session is saved in a folder named by its labels joined by '.', in this order, and 'deidentify' selects recipes by the first label. None if the store has no fixed hierarchy, in which case the levels are named when labelling (with 'label <store> --level NAME FIELD ...')

  • acquisition (LabelLevel, optional) -- the store's name for an acquisition, and the metadata field to describe each one by, e.g. XNAT's 'scan' described by 'SeriesDescription'. None if the store does not describe acquisitions

  • multiple_resources (bool) -- whether an acquisition can hold more than one resource (e.g. the DICOM files and a NIfTI conversion of them), as XNAT scans can. If False, each acquisition must hold exactly one, which 'label' checks

  • envvar_prefix (str) -- the prefix of the environment variables that the options can be set by

class accessionflow.model.store.LabelLevel(name: str, default: str | None = None, description: str = '')[source]

Bases: object

A level of a store's hierarchy that a session is labelled with, or the store's name for an acquisition within the session.

Parameters:
  • name (str) -- the store's name for the level, which names its command-line option (e.g. 'project' gives '--project') and environment variable (e.g. 'ACFLOW_PROJECT')

  • default (str, optional) -- the metadata field, or a format string over several fields, that the label is extracted from by default

  • description (str) -- what the label is in the store's terms, for the help text, e.g. 'XNAT project ID'