Command-line interface

AccessionFlow's command-line interface is a set of subcommands of the accessionflow command (or its shorter alias, acflow). See Concepts for how they fit together into an accession flow.

The collate stage has a subcommand for each source, e.g. accessionflow collate folder, and the label, store and check-store stages have one for each data store, e.g. accessionflow label xnat. Most options can also be set by environment variables, listed in Environment variables.

Sources

accessionflow collate folder

Collates the files found in folders, on a local path or a mounted file system, into accessions, acquisitions and resources

INPUT_PATHS are either paths to directories containing the files to collate, or glob patterns that select the paths directly

OUTPUT_DIR is the staging directory the files of each accession are collated to, ready to be labelled

Usage

accessionflow collate folder [OPTIONS] [INPUT_PATHS]... OUTPUT_DIR

Options

--accession <specifier> <datatype>

The metadata field whose value identifies the accession each file is collated into, before it is labelled, so it must be unique across all accessions (ACFLOW_ACCESSION env. var). Defaults to 'StudyInstanceUID' for DICOM collections; other fileset types need an explicit spec. Can also be a Python format string over several fields, e.g. '{PatientID}_{StudyDate:%Y%m%d}', to compose one. Not to be confused with DICOM's AccessionNumber, which is just another field that can be used here or to label the accession.

--acquisition <specifier> <datatype>

The keyword of the metadata field to extract the acquisition ID from, which must be unique within its accession, or a Python format string over several fields (see --accession). Defaults to 'SeriesNumber' for DICOM collections (one acquisition per series); for any fileset type not matched by an --acquisition spec the acquisition is named after the resource (ACFLOW_ACQUISITION env. var)

--resource <specifier> <datatype>

The keywords of the metadata field to extract the resource ID from for different datatypes (use generic/file-set as a catch-all if required), or a Python format string over several fields (see --accession). If not given, each resource is labelled with the mime-like rendering of its fileset type name, e.g. 'vectra-export', 'sqlite3-db' (ACFLOW_RESOURCE env. var)

--datatype <mime-type>

The MIME-type(s) (or "MIME-like" see FileFormats docs) of potential datatype(s) of the primary files to to upload, defaults to 'medimage/dicom-series'. Any formats implemented in the FileFormats Python package (https://github.com/ArcanaFramework/fileformats) that implement the 'read_metadata' "extra" are supported, see FF docs on how to add support for new formats.

--path-metadata-regex <regex> <datatype>

Regular expressions to extract "metadata" values from resource file paths as named groups. using Python regular expression syntax. The named groups are used as metadata fields for the resource files, and the extracted values will be used to populate the corresponding metadata fields to complement the metadata read from the file headers.

--path-metadata <pattern> <datatype>

A simpler, glob-like alternative to --path-metadata-regex. '{name}' placeholders mark the parts of the resource's path to extract into the metadata field 'name', e.g. '{subject}/{session}'. '*' matches within a path segment, '**' across segments and '?' a single character; '{{' and '}}' match literal braces. Unless the pattern starts with '/', it only needs to match the trailing segments of the path. Applied before any --path-metadata-regex (ACFLOW_PATH_METADATA env. var)

--on-resource-clash <policy> <scope>

How to handle two filesets resolving to the same acquisition/resource name: a <policy> ('avoid' appends a _2/_3 suffix, 'merge' folds both into one SetOf, 'overwrite' replaces) plus the datatype <scope> it applies to (a mime-like, a '|'-union, or 'all'). Repeatable. A clash is resolved by the first spec whose scope covers BOTH filesets; a clash no spec covers raises. (ACFLOW_ON_RESOURCE_CLASH env. var)

--allow-unrecognised, --allow-unrecognized <allow_unrecognised>

Regular expressions matched against the basename of any input path that no --datatype recognised; matches are skipped instead of raising. Use '.*' to tolerate all unrecognised files. Does not affect recognised filesets. Repeatable; the ACFLOW_ALLOW_UNRECOGNISED env. var takes a ';'-separated list.

--exclude-path <exclude_paths>

Glob patterns matched against each input path relative to <input-dir>, applied before classification so a match is dropped even if a --datatype would claim it (e.g. a vendor thumbnail that is a valid image/png). '*' does not cross '/', '**' does. Repeatable; the ACFLOW_EXCLUDE_PATH env. var takes a ';'-separated list.

--ignore-datatype <ignore_datatypes>

Datatypes expected in the input but not wanted: recognised files of these types are dropped instead of raising, and with --recursive a matching directory is skipped without descending into it. A path matching neither --datatype nor --ignore-datatype nor --allow-unrecognised/--exclude-path still raises. (ACFLOW_IGNORE_DATATYPE env. var)

--dont-read-metadata <datatype>

Datatypes whose files aren't read for metadata, so their only metadata fields are those from --path-metadata/--path-metadata-regex and --metadata-table. Saves reading the header of every file when none of its fields are used, e.g. when the IDs all come from the path. Matched with isinstance, so 'medimage/nifti' also covers 'medimage/nifti-gz'. Repeatable. (ACFLOW_DONT_READ_METADATA env. var)

--recursive, --not-recursive

Whether to recursively search input directories for input files

Whether to unlink the source files after they have been successfully collated. 'all' and 'keep-metadata' behave the same here (individual source files are removed one by one) since 'collate's source isn't a directory tree that accessionflow owns and can't have a metadata skeleton left behind in it (ACFLOW_UNLINK_SOURCE env. var)

Options:

all | keep-metadata

--collate-resources <mime-type> <collation>

Flatten files of the given datatype into the resource directory during collation, regardless of source directory structure. Collation level is one of 'any', 'siblings', or 'adjacent' (default 'siblings').

--metadata-table <path> <row-frequency> <join-exprs>

Specify metadata tables to extract and join metadata from input files (ACFLOW_METADATA_TABLES env. var). The 'path' arg specifies the location of the metadata table file. Its format is auto-detected as CSV or TSV from the file extension; a different format can be forced by appending its mime-type in square brackets, e.g. 'path/to/table.dat[text/csv]'. The "row frequency" arg specifies what each row in the metadata table corresponds to in the data hierarchy, and can be one of 'accession', 'acquisition', 'resource', 'fileset', 'fileset[<mime-type>]'. When one or more mime-types are given in square brackets after 'fileset' they restrict the join to input files of those types (multiple mime-types can be '|'-separated, e.g. 'fileset[image/png|image/jpeg]'); a bare 'fileset' matches any input file. The 'join-exprs' arg is a comma-separated list of '<column-name>=<cell-value>' expressions; a row is a match when every expression holds. The '<cell-value>' is either the name of an existing metadata field or a Python format string over one or more metadata fields, e.g. '{PatientID}_{SessionID}'. All columns of the matched row are then merged into the target's metadata. The example below extracts the relative path of an image file with --path-metadata-regex and uses it to join a table whose 'ImagePath' column holds spreadsheet HYPERLINK() formulas.

accessionflow collate folder ...

--path-metadata-regex '.*/(?P<relpath>[w-]+/[w-]+.(?:png|jpg))' image/png|image/jpeg --metadata-table path/to/table.csv[text/csv] fileset[image/png|image/jpeg] 'ImagePath=HYPERLINK("{relpath}")'

--loop <loop>

Run the collation continuously every LOOP seconds (ACFLOW_LOOP env. var)

--wait-period <wait_period>

The number of seconds since an accession was last modified at the source before it is collated, so that accessions that are still being transferred aren't collated partially (ACFLOW_WAIT_PERIOD env. var)

--copy-mode <copy_mode>

The method to use for copying files (ACFLOW_COPY_MODE env. var)

--convert <src-mime-like> <tgt-spec>

Convert resources of <src-mime-like> to <tgt-spec> during collation. <tgt-spec> may be either '<tgt-mime-like>' or '<tgt-mime-like>:key=value[,key=value...]', e.g. 'application/zip:compression=ZIP_STORED' (ACFLOW_CONVERT env. var)

--logger <logtype> <loglevel> <location>

Setup handles to capture logs that are generated (ACFLOW_LOGGERS env. var)

--additional-logger <additional_loggers>

Additional loggers to capture the logs of, besides 'accessionflow' (e.g. 'xnat') (ACFLOW_ADDITIONAL_LOGGERS env. var)

--raise-errors, --dont-raise-errors

Whether to raise errors instead of logging them (typically for debugging)

Arguments

INPUT_PATHS

Optional argument(s)

OUTPUT_DIR

Required argument

Environment variables

ACFLOW_INPUT_PATHS

Provide a default for INPUT_PATHS

ACFLOW_OUTPUT_DIR

Provide a default for OUTPUT_DIR

ACFLOW_ACCESSION

Provide a default for --accession

ACFLOW_ACCESSION_UID

Provide a default for --accession-uid

ACFLOW_ACQUISITION

Provide a default for --acquisition

ACFLOW_ACQUISITION_ID

Provide a default for --acquisition-id

ACFLOW_RESOURCE

Provide a default for --resource

ACFLOW_DATATYPES

Provide a default for --datatype

ACFLOW_PATH_METADATA_REGEX

Provide a default for --path-metadata-regex

ACFLOW_PATH_METADATA

Provide a default for --path-metadata

ACFLOW_ON_RESOURCE_CLASH

Provide a default for --on-resource-clash

ACFLOW_ALLOW_UNRECOGNISED

Provide a default for --allow-unrecognised

ACFLOW_EXCLUDE_PATH

Provide a default for --exclude-path

ACFLOW_IGNORE_DATATYPE

Provide a default for --ignore-datatype

ACFLOW_DONT_READ_METADATA

Provide a default for --dont-read-metadata

ACFLOW_RECURSIVE

Provide a default for --recursive

ACFLOW_COLLATE_RESOURCES

Provide a default for --collate-resources

ACFLOW_METADATA_TABLES

Provide a default for --metadata-table

ACFLOW_LOOP

Provide a default for --loop

ACFLOW_WAIT_PERIOD

Provide a default for --wait-period

ACFLOW_COPY_MODE

Provide a default for --copy-mode

ACFLOW_CONVERT

Provide a default for --convert

ACFLOW_LOGGERS

Provide a default for --logger

ACFLOW_ADDITIONAL_LOGGERS

Provide a default for --additional-logger

accessionflow collate orthanc

Collates the DICOM studies held by an Orthanc server into accessions in a staging directory, ready to be labelled. The files are hardlinked from Orthanc's storage directory, so it must be on the same file system as the staging directory.

URL of the Orthanc instance to connect to (ACFLOW_ORTHANC_URL env. var)

STORE_DIR is the path to Orthanc's StorageDirectory as mounted (ACFLOW_ORTHANC_STORE_DIR env. var)

OUTPUT_DIR is the staging directory the files of each accession are collated to

USER is the Orthanc user (ACFLOW_ORTHANC_USER env. var)

PASSWORD is the password of the Orthanc user (ACFLOW_ORTHANC_PASSWORD env. var)

Usage

accessionflow collate orthanc [OPTIONS] URL STORE_DIR OUTPUT_DIR USER PASSWORD

Options

--to-process-label <to_process_label>

Label applied to Orthanc studies after staging to prevent re-processing. Can be removed via the Orthanc UI (ACFLOW_ORTHANC_TO_PROCESS env. var)

--processed-label <processed_label>

Label applied to Orthanc studies after staging to prevent re-processing. Can be removed via the Orthanc UI (ACFLOW_ORTHANC_PROCESSED env. var)

Whether to unlink the source studies in Orthanc after they have been successfully collated. Not yet implemented (ACFLOW_UNLINK_SOURCE env. var)

Options:

all | keep-metadata

--loop <loop>

Run the collation continuously every LOOP seconds (ACFLOW_LOOP env. var)

--wait-period <wait_period>

The number of seconds since an accession was last modified at the source before it is collated, so that accessions that are still being transferred aren't collated partially (ACFLOW_WAIT_PERIOD env. var)

--copy-mode <copy_mode>

The method to use for copying files (ACFLOW_COPY_MODE env. var)

--convert <src-mime-like> <tgt-spec>

Convert resources of <src-mime-like> to <tgt-spec> during collation. <tgt-spec> may be either '<tgt-mime-like>' or '<tgt-mime-like>:key=value[,key=value...]', e.g. 'application/zip:compression=ZIP_STORED' (ACFLOW_CONVERT env. var)

--logger <logtype> <loglevel> <location>

Setup handles to capture logs that are generated (ACFLOW_LOGGERS env. var)

--additional-logger <additional_loggers>

Additional loggers to capture the logs of, besides 'accessionflow' (e.g. 'xnat') (ACFLOW_ADDITIONAL_LOGGERS env. var)

--raise-errors, --dont-raise-errors

Whether to raise errors instead of logging them (typically for debugging)

Arguments

URL

Required argument

STORE_DIR

Required argument

OUTPUT_DIR

Required argument

USER

Required argument

PASSWORD

Required argument

Environment variables

ACFLOW_ORTHANC_URL

Provide a default for URL

ACFLOW_ORTHANC_STORE_DIR

Provide a default for STORE_DIR

ACFLOW_ORTHANC_USER

Provide a default for USER

ACFLOW_ORTHANC_PASSWORD

Provide a default for PASSWORD

ACFLOW_ORTHANC_TO_PROCESS

Provide a default for --to-process-label

ACFLOW_ORTHANC_PROCESSED

Provide a default for --processed-label

ACFLOW_LOOP

Provide a default for --loop

ACFLOW_WAIT_PERIOD

Provide a default for --wait-period

ACFLOW_COPY_MODE

Provide a default for --copy-mode

ACFLOW_CONVERT

Provide a default for --convert

ACFLOW_LOGGERS

Provide a default for --logger

ACFLOW_ADDITIONAL_LOGGERS

Provide a default for --additional-logger

Stages

accessionflow attach

Stages images found in the input directories into separate directories for each imaging acquisition session

FILES_PATH is either the path to a directory containing the files to upload, or a glob pattern that selects the paths directly

OUTPUT_DIR is the directory that the files for each session are collated to before they are uploaded to XNAT

Usage

accessionflow attach [OPTIONS] INPUT_DIR OUTPUT_DIR DATATYPE GLOB ID_PATTERN

Options

--copy-mode <copy_mode>

The method to use for copying files (ACFLOW_COPY_MODE env. var)

Whether to unlink the source files matched by GLOB after they have been attached. 'all' and 'keep-metadata' behave the same here (individual source files are removed one by one), since these files live at an arbitrary external location rather than a directory tree accessionflow owns. If not set, the source files are left in place (ACFLOW_UNLINK_SOURCE env. var)

Options:

all | keep-metadata

--logger <logtype> <loglevel> <location>

Setup handles to capture logs that are generated (ACFLOW_LOGGERS env. var)

--additional-logger <additional_loggers>

The loggers to use for logging. By default just the 'accessionflow' logger is used. But additional loggers can be included (e.g. 'xnat') can be specified here (ACFLOW_ADDITIONAL_LOGGERS env. var)

--raise-errors, --dont-raise-errors

Whether to raise errors instead of logging them (typically for debugging)

--spaces-to-underscores, --no-spaces-to-underscores

Whether to replace spaces with underscores in the filenames of attached files (ACFLOW_SPACES_TO_UNDERSCORES env. var)

--loop <loop>

Run the staging process continuously every LOOP seconds (ACFLOW_LOOP env. var).

--on-resource-clash <on_resource_clash>

Determines the behavior when a resource with the same name already exists in the scan. Options are 'merge', 'avoid', 'error' (ACFLOW_ON_RESOURCE_CLASH env. var)Default: 'error'

Options:

error | avoid | merge | overwrite

--temp-dir <temp_dir>

The directory to use for temporary downloads (i.e. from s3)

--require-manifest, --dont-require-manifest

Whether to require manifest files in the staged resources or not (default: required)

Arguments

INPUT_DIR

Required argument

OUTPUT_DIR

Required argument

DATATYPE

Required argument

GLOB

Required argument

ID_PATTERN

Required argument

Environment variables

ACFLOW_INPUT_DIR

Provide a default for INPUT_DIR

ACFLOW_OUTPUT_DIR

Provide a default for OUTPUT_DIR

ACFLOW_DATATYPE

Provide a default for DATATYPE

ACFLOW_GLOB

Provide a default for GLOB

ACFLOW_ID_PATTERN

Provide a default for ID_PATTERN

ACFLOW_COPY_MODE

Provide a default for --copy-mode

ACFLOW_LOGGERS

Provide a default for --logger

ACFLOW_ADDITIONAL_LOGGERS

Provide a default for --additional-logger

ACFLOW_SPACES_TO_UNDERSCORES

Provide a default for --spaces-to-underscores

ACFLOW_LOOP

Provide a default for --loop

ACFLOW_ON_RESOURCE_CLASH

Provide a default for --on-resource-clash

ACFLOW_TEMPDIR

Provide a default for --temp-dir

ACFLOW_REQUIRE_MANIFEST

Provide a default for --require-manifest

accessionflow deidentify

Stages images found in the input directories into separate directories for each imaging acquisition session

INPUT_DIR is the path to the directory containing the labelled session directories to de-identify (the output of the 'label' stage). Each is named after the session's labels, e.g. <project>.<subject>.<session> for XNAT, and contains subdirectories for each acquisition, which in turn contain its resource files.

OUTPUT_DIR is the directory the de-identified sessions are written to, ready for the 'store' stage

RECIPE_DIR is the directory containing the deidentification recipes. Recipes are selected by the session's top-level label (e.g. the XNAT project), so it should contain one subdirectory per top-level label, plus an optional "__default__" subdirectory used as a fallback for those without their own. Within each of these subdirectories, the directory structure mirrors the MIME-like hierarchy: a subdirectory per category containing one recipe file per format (e.g. 'medimage/dicom-series'). Any file extension (or none) is accepted. Side-cars of the recipe format can sit alongside the recipe file, e.g. for DICOM a '<format>.transforms.yaml' file (e.g. 'medimage/dicom-series.transforms.yaml') declaring the values of the 'var:' and 'func:' references in the recipe (see the deid transforms spec in fileformats-medimage), and a '<format>.salt' file with a key the transforms can salt hashed values with (only readable by the user running the deidentification). Formats without a matching recipe file are only deidentified if one is found for a broader/parent format (e.g. a 'medimage/dicom-collection' recipe also covers 'medimage/dicom-series'). Metadata side-cars in the deidentified output retain only __uid__, where present. Other metadata fields are omitted; source metadata is unchanged. __uid__ holds the original session UID (by default the DICOM StudyInstanceUID) and is copied to the data store by the store stage, unless it is replaced with a keyed hash using --hash-uid-key.

The re-identification metadata (a mapping from original to de-identified identifiers for each resource in the session, plus any DICOM tags modified during de-identification) is only written if --reid-dir is given; see that option.

Usage

accessionflow deidentify [OPTIONS] INPUT_DIR OUTPUT_DIR RECIPE_DIR

Options

--reid-dir <reid_dir>

The directory to save the re-identification metadata to, which can be used to re-identify the de-identified data if needed. One JSON file is written per session, named after the session's directory, <labels>.json (or <labels>.json.enc if encrypted with --reid-encrypt-key). If this option is not given, the re-identification metadata is discarded rather than written to disk, which is the safer default when retaining the original<->deidentified mapping would itself be a security concern (ACFLOW_REID_DIR env. var)

Whether to unlink the labelled session directories after they have been deidentified. 'all' removes the whole labelled session directory; 'keep-metadata' removes the resource data but leaves the session/scan-level metadata behind, so a lightweight skeleton of the session survives (e.g. for 'attach' to use later). If not set, the labelled directories are left in place (ACFLOW_UNLINK_SOURCE env. var)

Options:

all | keep-metadata

--logger <logtype> <loglevel> <location>

Setup handles to capture logs that are generated (ACFLOW_LOGGERS env. var)

--additional-logger <additional_loggers>

The loggers to use for logging. By default just the 'accessionflow' logger is used. But additional loggers can be included (e.g. 'xnat') can be specified here (ACFLOW_ADDITIONAL_LOGGERS env. var)

--raise-errors, --dont-raise-errors

Whether to raise errors instead of logging them (typically for debugging)

--require-manifest, --no-require-manifest

Whether to require a MANIFEST.json file in each resource directory. If False, resources are loaded as generic FileSets without checksum validation (ACFLOW_REQUIRE_MANIFEST env. var)

--copy-mode <copy_mode>

The method to use for copying files (ACFLOW_COPY_MODE env. var)

--loop <loop>

Run the staging process continuously every LOOP seconds (ACFLOW_LOOP env. var).

--on-resource-clash <on_resource_clash>

Determines the behavior when a resource with the same name already exists in the scan. Options are 'merge', 'avoid', 'error' (ACFLOW_ON_RESOURCE_CLASH env. var).Default: 'error'

Options:

error | avoid | merge | overwrite

--reid-encrypt-key <reid_encrypt_key>

An optional encryption key to use for encrypting the re-identification metadata (ACFLOW_REID_ENCRYPT_KEY env. var). This should be a URL-safe base64-encoded 32-byte key, e.g. generated using Fernet.generate_key() from the cryptography package

--metadata-allowlist <metadata_allowlist>

Comma-separated metadata keys to retain in output sidecars (ACFLOW_METADATA_ALLOWLIST env. var). Defaults to __uid__. __uid__, __labels__ and __store_type__ are always retained, as they are needed to identify a session.

--hash-uid-key <hash_uid_key>

Secret key used to replace each session's UID (by default its DICOM StudyInstanceUID) with a keyed hash of it in the output. The original and hashed UIDs are both recorded in the re-identification metadata if --reid-dir is given. Keep the key private and don't change it, as the hash identifies the session if it is uploaded again. Sessions already on XNAT hold their previous UID, see --no-uid-check of the upload stage. If not given, the original UID is kept. Prefer the env. var so that the key isn't visible in the process list (ACFLOW_HASH_UID_KEY env. var)

Arguments

INPUT_DIR

Required argument

OUTPUT_DIR

Required argument

RECIPE_DIR

Required argument

Environment variables

ACFLOW_INPUT_DIR

Provide a default for INPUT_DIR

ACFLOW_OUTPUT_DIR

Provide a default for OUTPUT_DIR

['ACFLOW_RECIPE_DIR', 'ACFLOW_SPEC_DIR']

Provide a default for RECIPE_DIR

ACFLOW_REID_DIR

Provide a default for --reid-dir

ACFLOW_LOGGERS

Provide a default for --logger

ACFLOW_ADDITIONAL_LOGGERS

Provide a default for --additional-logger

ACFLOW_REQUIRE_MANIFEST

Provide a default for --require-manifest

ACFLOW_COPY_MODE

Provide a default for --copy-mode

ACFLOW_LOOP

Provide a default for --loop

ACFLOW_ON_RESOURCE_CLASH

Provide a default for --on-resource-clash

ACFLOW_REID_ENCRYPT_KEY

Provide a default for --reid-encrypt-key

ACFLOW_METADATA_ALLOWLIST

Provide a default for --metadata-allowlist

ACFLOW_HASH_UID_KEY

Provide a default for --hash-uid-key

accessionflow package

Replace staged scan resources with one or more named conversions.

INPUT_DIR is normally the output of the deidentify command. Every OUTPUT_RESOURCE conversion receives the original matching SOURCE_DATATYPE resource, allowing an archive and a sample to be produced independently without reopening the archive.

Usage

accessionflow package [OPTIONS] INPUT_DIR OUTPUT_DIR SOURCE_DATATYPE

Options

--output-resource <label> <target-spec>

Required Resource label and target datatype to produce from each matching source. May be repeated. Target specs accept converter options after ':', e.g. 'application/zip:allowZip64=false'.

--copy-mode <copy_mode>

How converted and pass-through resources are copied to the output.

Remove the input session after successful packaging, either entirely or leaving its session/scan metadata skeleton.

Options:

all | keep-metadata

--on-resource-clash <on_resource_clash>

How to handle output labels that clash within the same scan.

Options:

error | avoid | merge | overwrite

--require-manifest, --dont-require-manifest

Whether staged input resources must contain manifests.

--temp-dir <temp_dir>

Directory in which converters create temporary data.

--loop <loop>

Run repeatedly every LOOP seconds; a negative value runs once.

--logger <logtype> <loglevel> <location>
--additional-logger <additional_loggers>
--raise-errors, --dont-raise-errors

Arguments

INPUT_DIR

Required argument

OUTPUT_DIR

Required argument

SOURCE_DATATYPE

Required argument

Environment variables

ACFLOW_INPUT_DIR

Provide a default for INPUT_DIR

ACFLOW_OUTPUT_DIR

Provide a default for OUTPUT_DIR

ACFLOW_SOURCE_DATATYPE

Provide a default for SOURCE_DATATYPE

ACFLOW_OUTPUT_RESOURCES

Provide a default for --output-resource

ACFLOW_COPY_MODE

Provide a default for --copy-mode

ACFLOW_ON_RESOURCE_CLASH

Provide a default for --on-resource-clash

ACFLOW_REQUIRE_MANIFEST

Provide a default for --require-manifest

ACFLOW_TEMPDIR

Provide a default for --temp-dir

ACFLOW_LOOP

Provide a default for --loop

ACFLOW_LOGGERS

Provide a default for --logger

ACFLOW_ADDITIONAL_LOGGERS

Provide a default for --additional-logger

accessionflow annotate

Annotates accessions with fields extracted from their metadata, which the data store records alongside them (e.g. as custom variables on XNAT sessions).

Usage

accessionflow annotate [OPTIONS] INPUT_DIR OUTPUT_DIR

Options

--extract <xnat-field> <specifier>

Required Expose session metadata as an XNAT field. May be repeated; separate entries with ';' in ACFLOW_FIELDS.

--loop <loop>
--logger <logtype> <loglevel> <location>
--additional-logger <additional_loggers>
--raise-errors, --dont-raise-errors

Arguments

INPUT_DIR

Required argument

OUTPUT_DIR

Required argument

Environment variables

ACFLOW_INPUT_DIR

Provide a default for INPUT_DIR

ACFLOW_OUTPUT_DIR

Provide a default for OUTPUT_DIR

ACFLOW_FIELDS

Provide a default for --extract

ACFLOW_LOOP

Provide a default for --loop

ACFLOW_LOGGERS

Provide a default for --logger

ACFLOW_ADDITIONAL_LOGGERS

Provide a default for --additional-logger

Folder store

accessionflow label folder

Labels accessions for folder data stores with their user-defined (--level) labels, extracted from accession metadata, after they have been collated into acquisitions/resources

INPUT_DIR is the path to the directory containing the collated-but-not-yet-labelled accessions (the output of the 'collate' command)

OUTPUT_DIR is the directory that the labelled accessions will be written to, each in a directory named '<label1>.<label2>...'

If a label can't be resolved from an accession's metadata, it is given a unique 'INVALID_MISSING_<FIELD>_<random>' placeholder instead of being dropped, and the whole accession is saved under an '__invalid__' subdirectory of OUTPUT_DIR instead of alongside normally-labelled ones, so it can be found and manually reviewed/reprocessed rather than silently lost.

Usage

accessionflow label folder [OPTIONS] INPUT_DIR OUTPUT_DIR

Options

--level <name> <field>

Required A level of the folder hierarchy the sessions are labelled with, and the keyword of the metadata field to extract its label from (or a Python format string over several fields, e.g. '{PatientID}_{StudyDate:%Y%m%d}'). Repeat for each level, in order from the top, e.g. '--level site SiteID --level subject PatientID' (ACFLOW_LEVELS env. var, as space-separated pairs)

--acquisition <acquisition_field>

The keyword of the metadata field to extract the description of each acquisition from. Acquisitions for which the field can't be resolved are left without one (ACFLOW_ACQUISITION env. var)

--constant-label <name> <value>

Fix the label of a level given by --level as a constant for all data matched by this command, e.g. '--constant-label site SYDNEY'. May be repeated (ACFLOW_CONSTANT_LABELS env. var)

--include <mime-type>

Only label resources matching this fileformats MIME-like datatype. May be repeated to include multiple datatypes (ACFLOW_INCLUDE env. var)

--loop <loop>

Run the label process continuously every LOOP seconds (ACFLOW_LOOP env. var).

--copy-mode <copy_mode>

The method to use for copying files (ACFLOW_COPY_MODE env. var)

Whether to unlink the collated session directories after they have been labelled. 'all' removes the whole collated session directory; 'keep-metadata' removes the resource data but leaves the session/acquisition-level metadata behind, so a lightweight skeleton of the session survives (e.g. for 'attach' to use later). If not set, the collated directories are left in place (ACFLOW_UNLINK_SOURCE env. var)

Options:

all | keep-metadata

--logger <logtype> <loglevel> <location>

Setup handles to capture logs that are generated (ACFLOW_LOGGERS env. var)

--additional-logger <additional_loggers>

The loggers to use for logging. By default just the 'accessionflow' logger is used. But additional loggers can be included (e.g. 'xnat') can be specified here (ACFLOW_ADDITIONAL_LOGGERS env. var)

--raise-errors, --dont-raise-errors

Whether to raise errors instead of logging them (typically for debugging)

Arguments

INPUT_DIR

Required argument

OUTPUT_DIR

Required argument

Environment variables

ACFLOW_INPUT_DIR

Provide a default for INPUT_DIR

ACFLOW_OUTPUT_DIR

Provide a default for OUTPUT_DIR

ACFLOW_LEVELS

Provide a default for --level

ACFLOW_ACQUISITION

Provide a default for --acquisition

ACFLOW_CONSTANT_LABELS

Provide a default for --constant-label

ACFLOW_INCLUDE

Provide a default for --include

ACFLOW_LOOP

Provide a default for --loop

ACFLOW_COPY_MODE

Provide a default for --copy-mode

ACFLOW_LOGGERS

Provide a default for --logger

ACFLOW_ADDITIONAL_LOGGERS

Provide a default for --additional-logger

accessionflow store folder

Deposits the labelled accessions in a staging directory (as prepared by 'label folder') in a folder data store: nested label folders, with a folder per acquisition and per resource, on a local path, a mount, or a remote file system such as S3 or SFTP.

STAGED is the staging directory, or an S3 bucket to download the accessions from.

DEST is the root of the store, a path or an fsspec URL, e.g. '/mnt/archive', 's3://bucket/archive' or 'sftp://host/archive'. Remote file systems need their fsspec package installed, e.g. 'accessionflow[s3]' for S3 or 'accessionflow[ssh]' for SFTP.

The resources' manifests and the accessions' fields are deposited with them, but not their metadata. A resource already in the store with the same files is skipped, so depositing again only copies what has changed.

Usage

accessionflow store folder [OPTIONS] STAGED DEST

Options

--storage-option <key> <value>

An option for the fsspec file system of the store, e.g. '--storage-option anon true' or '--storage-option key <access-key>'. Values that are JSON are decoded. May be repeated (ACFLOW_STORAGE_OPTIONS env. var, as space-separated pairs)

--require-manifest, --dont-require-manifest

Whether the staged resources must have manifests. If not, the checksums of resources without one are worked out from their files (ACFLOW_REQUIRE_MANIFEST env. var)

--store-credentials <access-key> <secret-key>

Credentials of the S3 bucket the accessions are staged in, if STAGED is one (ACFLOW_STORE_CREDENTIALS env. var)

--temp-dir <temp_dir>

The directory to download accessions staged in S3 to

--s3-max-workers <s3_max_workers>

The number of files to download from an S3 staging bucket at once (ACFLOW_S3_MAX_WORKERS env. var)

--logger <logtype> <loglevel> <location>

Setup handles to capture logs that are generated (ACFLOW_LOGGERS env. var)

--additional-logger <additional_loggers>

Additional loggers to capture the logs of, besides 'accessionflow' (e.g. 's3fs') (ACFLOW_ADDITIONAL_LOGGERS env. var)

--raise-errors, --dont-raise-errors

Whether to raise errors instead of logging them (typically for debugging)

--overwrite, --no-overwrite

Whether to replace a resource that is already in the store with different files. If not, it is reported as an error (ACFLOW_OVERWRITE env. var)

--wait-period <wait_period>

The number of seconds since an accession was last modified before it is deposited (ACFLOW_WAIT_PERIOD env. var)

--loop <loop>

Run continuously, every LOOP seconds (ACFLOW_LOOP env. var)

Arguments

STAGED

Required argument

DEST

Required argument

Environment variables

ACFLOW_STAGED

Provide a default for STAGED

ACFLOW_DEST

Provide a default for DEST

ACFLOW_STORAGE_OPTIONS

Provide a default for --storage-option

ACFLOW_REQUIRE_MANIFEST

Provide a default for --require-manifest

ACFLOW_STORE_CREDENTIALS

Provide a default for --store-credentials

ACFLOW_TEMPDIR

Provide a default for --temp-dir

ACFLOW_S3_MAX_WORKERS

Provide a default for --s3-max-workers

ACFLOW_LOGGERS

Provide a default for --logger

ACFLOW_ADDITIONAL_LOGGERS

Provide a default for --additional-logger

ACFLOW_OVERWRITE

Provide a default for --overwrite

ACFLOW_WAIT_PERIOD

Provide a default for --wait-period

ACFLOW_LOOP

Provide a default for --loop

accessionflow check-store folder

Checks what has been deposited in a folder data store against the staging directory it was deposited from, and logs every resource that is missing, incomplete or different. Exits with a non-zero status if there are any.

STAGED is the staging directory, or an S3 bucket to download the accessions from.

DEST is the root of the store, a path or an fsspec URL, e.g. 's3://bucket/archive'.

Usage

accessionflow check-store folder [OPTIONS] STAGED DEST

Options

--storage-option <key> <value>

An option for the fsspec file system of the store, e.g. '--storage-option anon true' or '--storage-option key <access-key>'. Values that are JSON are decoded. May be repeated (ACFLOW_STORAGE_OPTIONS env. var, as space-separated pairs)

--require-manifest, --dont-require-manifest

Whether the staged resources must have manifests. If not, the checksums of resources without one are worked out from their files (ACFLOW_REQUIRE_MANIFEST env. var)

--store-credentials <access-key> <secret-key>

Credentials of the S3 bucket the accessions are staged in, if STAGED is one (ACFLOW_STORE_CREDENTIALS env. var)

--temp-dir <temp_dir>

The directory to download accessions staged in S3 to

--s3-max-workers <s3_max_workers>

The number of files to download from an S3 staging bucket at once (ACFLOW_S3_MAX_WORKERS env. var)

--logger <logtype> <loglevel> <location>

Setup handles to capture logs that are generated (ACFLOW_LOGGERS env. var)

--additional-logger <additional_loggers>

Additional loggers to capture the logs of, besides 'accessionflow' (e.g. 's3fs') (ACFLOW_ADDITIONAL_LOGGERS env. var)

--raise-errors, --dont-raise-errors

Whether to raise errors instead of logging them (typically for debugging)

--checksums, --sizes-only

Whether to check the files' checksums, which reads every file back from the store, or only compare their sizes, which is much cheaper (ACFLOW_CHECK_CHECKSUMS env. var)

Arguments

STAGED

Required argument

DEST

Required argument

Environment variables

ACFLOW_STAGED

Provide a default for STAGED

ACFLOW_DEST

Provide a default for DEST

ACFLOW_STORAGE_OPTIONS

Provide a default for --storage-option

ACFLOW_REQUIRE_MANIFEST

Provide a default for --require-manifest

ACFLOW_STORE_CREDENTIALS

Provide a default for --store-credentials

ACFLOW_TEMPDIR

Provide a default for --temp-dir

ACFLOW_S3_MAX_WORKERS

Provide a default for --s3-max-workers

ACFLOW_LOGGERS

Provide a default for --logger

ACFLOW_ADDITIONAL_LOGGERS

Provide a default for --additional-logger

ACFLOW_CHECK_CHECKSUMS

Provide a default for --checksums

XNAT

accessionflow label xnat

Labels accessions for XNAT data stores with their project/subject/session labels, extracted from accession metadata, after they have been collated into scans/resources

INPUT_DIR is the path to the directory containing the collated-but-not-yet-labelled accessions (the output of the 'collate' command)

OUTPUT_DIR is the directory that the labelled accessions will be written to, each in a directory named '<project>.<subject>.<session>'

If a label can't be resolved from an accession's metadata, it is given a unique 'INVALID_MISSING_<FIELD>_<random>' placeholder instead of being dropped, and the whole accession is saved under an '__invalid__' subdirectory of OUTPUT_DIR instead of alongside normally-labelled ones, so it can be found and manually reviewed/reprocessed rather than silently lost.

Usage

accessionflow label xnat [OPTIONS] INPUT_DIR OUTPUT_DIR

Options

--project <label_0>

The keyword of the metadata field to extract the XNAT project ID from, or a Python format string over several fields, e.g. '{PatientID}_{StudyDate:%Y%m%d}', to compose one (ACFLOW_PROJECT env. var)

--subject <label_1>

The keyword of the metadata field to extract the XNAT subject ID from, or a Python format string over several fields, e.g. '{PatientID}_{StudyDate:%Y%m%d}', to compose one (ACFLOW_SUBJECT env. var)

--session <label_2>

The keyword of the metadata field to extract the XNAT session ID from, or a Python format string over several fields, e.g. '{PatientID}_{StudyDate:%Y%m%d}', to compose one (ACFLOW_SESSION env. var)

--scan <acquisition_field>

The keyword of the metadata field to extract the XNAT scan type of each scan from. Scans for which the field can't be resolved are left without one (ACFLOW_SCAN env. var)

--constant-project-id <constant_top_label>

Fix the XNAT project ID as a constant for all data matched by this command (ACFLOW_CONSTANT_PROJECT_ID env. var)

--include <mime-type>

Only label resources matching this fileformats MIME-like datatype. May be repeated to include multiple datatypes (ACFLOW_INCLUDE env. var)

--loop <loop>

Run the label process continuously every LOOP seconds (ACFLOW_LOOP env. var).

--copy-mode <copy_mode>

The method to use for copying files (ACFLOW_COPY_MODE env. var)

Whether to unlink the collated session directories after they have been labelled. 'all' removes the whole collated session directory; 'keep-metadata' removes the resource data but leaves the session/acquisition-level metadata behind, so a lightweight skeleton of the session survives (e.g. for 'attach' to use later). If not set, the collated directories are left in place (ACFLOW_UNLINK_SOURCE env. var)

Options:

all | keep-metadata

--logger <logtype> <loglevel> <location>

Setup handles to capture logs that are generated (ACFLOW_LOGGERS env. var)

--additional-logger <additional_loggers>

The loggers to use for logging. By default just the 'accessionflow' logger is used. But additional loggers can be included (e.g. 'xnat') can be specified here (ACFLOW_ADDITIONAL_LOGGERS env. var)

--raise-errors, --dont-raise-errors

Whether to raise errors instead of logging them (typically for debugging)

Arguments

INPUT_DIR

Required argument

OUTPUT_DIR

Required argument

Environment variables

ACFLOW_INPUT_DIR

Provide a default for INPUT_DIR

ACFLOW_OUTPUT_DIR

Provide a default for OUTPUT_DIR

ACFLOW_PROJECT

Provide a default for --project

ACFLOW_SUBJECT

Provide a default for --subject

ACFLOW_SESSION

Provide a default for --session

ACFLOW_SCAN

Provide a default for --scan

ACFLOW_CONSTANT_PROJECT_ID

Provide a default for --constant-project-id

ACFLOW_INCLUDE

Provide a default for --include

ACFLOW_LOOP

Provide a default for --loop

ACFLOW_COPY_MODE

Provide a default for --copy-mode

ACFLOW_LOGGERS

Provide a default for --logger

ACFLOW_ADDITIONAL_LOGGERS

Provide a default for --additional-logger

accessionflow store xnat

Deposits the labelled accessions in a staging directory (as prepared by the label xnat sub-command) in an XNAT data store, by uploading them to it.

STAGED is either a directory that the files for each session are collated to before they are uploaded to XNAT or an S3 bucket to download the files from.

SERVER is address of the XNAT server to upload the scans up to. Can alternatively provided by setting the "ACFLOW_HOST" environment variable.

Usage

accessionflow store xnat [OPTIONS] STAGED SERVER

Options

--user <user>

the XNAT user to connect with (alternatively the "ACFLOW_USER" env. variable can be used.

--password <password>

the password for the XNAT user, alternatively "ACFLOW_PASS" env. var

--logger <logtype> <loglevel> <location>

Setup handles to capture logs that are generated (ACFLOW_LOGGERS env. var)

--additional-logger <additional_loggers>

The loggers to use for logging. By default just the 'accessionflow' logger is used. But additional loggers can be included (e.g. 'xnat') can be specified here (ACFLOW_ADDITIONAL_LOGGERS env. var).

-i, --always-include <always_include>

Scan types to always include in the upload, regardless of whether they arespecified in a column or not. Specified using the scan types IANA mime-type or fileformats "mime-like" (see https://arcanaframework.github.io/fileformats/), e.g. 'application/json', 'medimage/dicom-series', 'image/jpeg'). Use 'all' to include all file-types in the session (ACFLOW_ALWAYSINCLUDE env. var).

--date <date_field>

Session metadata field to use as the XNAT session date when creating a session. The value must resolve to one calendar date (ACFLOW_DATE env. var).

--raise-errors, --dont-raise-errors

Whether to raise errors instead of logging them (typically for debugging)

--store-credentials <access-key> <secret-key>

Credentials to use to access of data stored in remote stores (e.g. AWS S3) (ACFLOW_STORE_CREDENTIALS env. var)

--temp-dir <temp_dir>

The directory to use for temporary downloads (i.e. from s3)

--s3-max-workers <s3_max_workers>

Maximum concurrent S3 object downloads.

Default:

10

--xnat-max-workers <xnat_max_workers>

Maximum concurrent XNAT resource uploads.

Default:

10

--require-manifest, --dont-require-manifest

Whether to require manifest files in the staged resources or not (default: required)

--verify-ssl, --dont-verify-ssl

Whether to verify the SSL certificate of the XNAT server

--use-curl-jsession, --dont-use-curl-jsession

Whether to use CURL to create a JSESSION token to authenticate with XNAT. This is used to work around a strange authentication issue when running within a Kubernetes cluster and targeting the XNAT Tomcat directly (ACFLOW_USE_CURL_JSESSION env. var).

--method <method> <datatype>

The methods to use to upload the file types to XNAT with. Passed through to XNATPy and controls whether directories are tarred and/or gzipped before they are uploaded, by default 'tgz_file' is used (ACFLOW_METHODS env. var).

--wait-period <wait_period>

The number of seconds to wait since the last file modification in sessions in the S3 bucket or source file-system directory before uploading them to avoid uploading partial sessions (ACFLOW_WAIT_PERIOD env. var).

--loop <loop>

Run the staging process continuously every LOOP seconds (ACFLOW_LOOP env. var).

--num-files-per-batch <num_files_per_batch>

When uploading files to XNAT, the number of files to upload in each batch. The number of files that are uploaded in a single batch can be limited to avoid overloading the building of the catalog file. If <= 0 (the default), then all files are uploaded in a single batch (ACFLOW_NUM_FILES_PER_BATCH env. var).

--check-checksums, --dont-check-checksums

Whether to check the checksums of the files in the staged resources against the checksums of both the checksums saved in the manifests and verify after upload to verify that they were uploaded correctly (if checksums are enabled site-wide on XNAT, i.e. enableChecksums is set to true in the XNAT configuration)

--dry-run, --no-dry-run

List the sessions that will be uploaded instead of the actually uploading them

--modality <modality>

Specify the modality of the sessions to upload (ACFLOW_MODALITY env. var).

--no-uid-check

Don't compare a session's UID with the one recorded on an existing XNAT session. Only for the transition to hashed UIDs (see --hash-uid-key of the deidentify stage), as sessions already on XNAT hold their previous UID. The subject of the session is still checked, where it is known, and the UID on XNAT isn't changed (ACFLOW_NO_UID_CHECK env. var)

Arguments

STAGED

Required argument

SERVER

Required argument

Environment variables

ACFLOW_STAGED

Provide a default for STAGED

ACFLOW_HOST

Provide a default for SERVER

ACFLOW_USER

Provide a default for --user

ACFLOW_PASS

Provide a default for --password

ACFLOW_LOGGERS

Provide a default for --logger

ACFLOW_ADDITIONAL_LOGGERS

Provide a default for --additional-logger

ACFLOW_ALWAYSINCLUDE

Provide a default for --always-include

ACFLOW_DATE

Provide a default for --date

ACFLOW_STORE_CREDENTIALS

Provide a default for --store-credentials

ACFLOW_TEMPDIR

Provide a default for --temp-dir

ACFLOW_S3_MAX_WORKERS

Provide a default for --s3-max-workers

ACFLOW_XNAT_MAX_WORKERS

Provide a default for --xnat-max-workers

ACFLOW_REQUIRE_MANIFEST

Provide a default for --require-manifest

ACFLOW_VERIFY_SSL

Provide a default for --verify-ssl

ACFLOW_USE_CURL_JSESSION

Provide a default for --use-curl-jsession

ACFLOW_METHODS

Provide a default for --method

ACFLOW_WAIT_PERIOD

Provide a default for --wait-period

ACFLOW_LOOP

Provide a default for --loop

ACFLOW_NUM_FILES_PER_BATCH

Provide a default for --num-files-per-batch

ACFLOW_CHECK_CHECKSUMS

Provide a default for --check-checksums

ACFLOW_DRY_RUN

Provide a default for --dry-run

ACFLOW_MODALITY

Provide a default for --modality

ACFLOW_NO_UID_CHECK

Provide a default for --no-uid-check

accessionflow check-store xnat

Checks what has been deposited in an XNAT data store against the staging directory it was deposited from, and logs all files that haven't been

STAGED is either a directory that the files for each session are collated to before they are uploaded to XNAT or an S3 bucket to download the files from.

SERVER is address of the XNAT server to upload the scans up to. Can alternatively provided by setting the "ACFLOW_HOST" environment variable.

Usage

accessionflow check-store xnat [OPTIONS] INPUT_DIR SERVER

Options

--user <user>

the XNAT user to connect with (alternatively the "ACFLOW_USER" env. variable can be used.

--password <password>

the password for the XNAT user, alternatively "ACFLOW_PASS" env. var

--logger <logtype> <loglevel> <location>

Setup handles to capture logs that are generated (ACFLOW_LOGGERS env. var)

--additional-logger <additional_loggers>

The loggers to use for logging. By default just the 'accessionflow' logger is used. But additional loggers can be included (e.g. 'xnat') can be specified here (ACFLOW_ADDITIONAL_LOGGERS env. var).

-i, --always-include <always_include>

Scan types to always include in the upload, regardless of whether they arespecified in a column or not. Specified using the scan types IANA mime-type or fileformats "mime-like" (see https://arcanaframework.github.io/fileformats/), e.g. 'application/json', 'medimage/dicom-series', 'image/jpeg'). Use 'all' to include all file-types in the session (ACFLOW_ALWAYSINCLUDE env. var).

--store-credentials <access-key> <secret-key>

Credentials to use to access of data stored in remote stores (e.g. AWS S3)

--temp-dir <temp_dir>

The directory to use for temporary downloads (i.e. from s3) (ACFLOW_TEMPDIR env. var)

--s3-max-workers <s3_max_workers>

Maximum concurrent S3 object downloads.

Default:

10

--xnat-max-workers <xnat_max_workers>

Maximum concurrent XNAT checksum requests.

Default:

10

--verify-ssl, --dont-verify-ssl

Whether to verify the SSL certificate of the XNAT server (ACFLOW_VERIFY_SSL env. var)

--use-curl-jsession, --dont-use-curl-jsession

Whether to use CURL to create a JSESSION token to authenticate with XNAT. This is used to work around a strange authentication issue when running within a Kubernetes cluster and targeting the XNAT Tomcat directly (ACFLOW_USE_CURL_JSESSION env. var).

--disable-progress, --enable-progress

Disable the progress bar

--no-uid-check

Don't compare a session's UID with the one recorded on an existing XNAT session. Only for the transition to hashed UIDs (see --hash-uid-key of the deidentify stage), as sessions already on XNAT hold their previous UID. The subject of the session is still checked, where it is known, and the UID on XNAT isn't changed (ACFLOW_NO_UID_CHECK env. var)

Arguments

INPUT_DIR

Required argument

SERVER

Required argument

Environment variables

ACFLOW_STAGED

Provide a default for INPUT_DIR

ACFLOW_HOST

Provide a default for SERVER

ACFLOW_USER

Provide a default for --user

ACFLOW_PASS

Provide a default for --password

ACFLOW_LOGGERS

Provide a default for --logger

ACFLOW_ADDITIONAL_LOGGERS

Provide a default for --additional-logger

ACFLOW_ALWAYSINCLUDE

Provide a default for --always-include

ACFLOW_STORE_CREDENTIALS

Provide a default for --store-credentials

ACFLOW_TEMPDIR

Provide a default for --temp-dir

ACFLOW_S3_MAX_WORKERS

Provide a default for --s3-max-workers

ACFLOW_XNAT_MAX_WORKERS

Provide a default for --xnat-max-workers

ACFLOW_VERIFY_SSL

Provide a default for --verify-ssl

ACFLOW_USE_CURL_JSESSION

Provide a default for --use-curl-jsession

ACFLOW_DISABLE_PROGRESS

Provide a default for --disable-progress

ACFLOW_NO_UID_CHECK

Provide a default for --no-uid-check