Command-line interface

XNAT Ingest's command line interface consists of a number of sub-commands under the xnat-ingest command. See Basic ingest workflow for how these fit together into a pipeline.

xnat-ingest group

Groups images found in the input paths into separate resources, grouped into scans and acquisition sessions

INPUT_PATHS are either paths to directories containing the files to upload, or glob patterns that select the paths directly

OUTPUT_DIR is the directory that the files for each session are collated to before they are uploaded to XNAT

Usage

xnat-ingest group [OPTIONS] [INPUT_PATHS]... OUTPUT_DIR

Options

--session <session>

The metadata field used to group files into the same session before IDs are extracted (XINGEST_SESSION env. var). Defaults to StudyInstanceUID.

--scan <specifier> <datatype>

The keyword of the metadata field to extract the XNAT imaging scan ID from (XINGEST_SCAN env. var)

--resource <specifier> <datatype>

The keywords of the metadata field to extract the XNAT imaging resource ID from for different datatypes (use generic/file-set as a catch-all if required). (XINGEST_RESOURCE env. var)

--datatype <mime-type>

The MIME-type(s) (or "MIME-like" see FileFormats docs) of potential datatype(s) of the primary files to to upload, defaults to 'medimage/dicom-series'. Any formats implemented in the FileFormats Python package (https://github.com/ArcanaFramework/fileformats) that implement the 'read_metadata' "extra" are supported, see FF docs on how to add support for new formats.

--path-metadata-regex <regex> <datatype>

Regular expressions to extract "metadata" values from resource file paths as named groups. using Python regular expression syntax. The named groups are used as metadata fields for the resource files, and the extracted values will be used to populate the corresponding metadata fields to complement the metadata read from the file headers.

--loop <loop>

Run the staging process continuously every LOOP seconds (XINGEST_LOOP env. var).

--wait-period <wait_period>

The number of seconds to wait since the last file modification in sessions in the S3 bucket or source file-system directory before uploading them to avoid uploading partial sessions (XINGEST_WAIT_PERIOD env. var).

--recursive, --not-recursive

Whether to recursively search input directories for input files

--copy-mode <copy_mode>

The method to use for copying files (XINGEST_COPY_MODE env. var)

Whether to unlink the source files after they have been successfully grouped. 'all' and 'keep-metadata' behave the same here (individual source files are removed one by one) since 'group's source isn't a directory tree that xnat-ingest owns and can't have a metadata skeleton left behind in it (XINGEST_UNLINK_SOURCE env. var)

Options:

all | keep-metadata

--logger <logtype> <loglevel> <location>

Setup handles to capture logs that are generated (XINGEST_LOGGERS env. var)

--additional-logger <additional_loggers>

The loggers to use for logging. By default just the 'xnat-ingest' logger is used. But additional loggers can be included (e.g. 'xnat') can be specified here (XINGEST_ADDITIONAL_LOGGERS env. var)

--raise-errors, --dont-raise-errors

Whether to raise errors instead of logging them (typically for debugging)

--collate-resources <mime-type> <collation>

Flatten files of the given datatype into the resource directory during grouping, regardless of source directory structure (e.g. when grouping from Orthanc). Collation level is one of 'any', 'siblings', or 'adjacent' (default 'siblings').

Arguments

INPUT_PATHS

Optional argument(s)

OUTPUT_DIR

Required argument

Environment variables

XINGEST_INPUT_PATHS

Provide a default for INPUT_PATHS

XINGEST_OUTPUT_DIR

Provide a default for OUTPUT_DIR

XINGEST_SESSION

Provide a default for --session

XINGEST_SCAN

Provide a default for --scan

XINGEST_RESOURCE

Provide a default for --resource

XINGEST_DATATYPES

Provide a default for --datatype

XINGEST_PATH_METADATA_REGEX

Provide a default for --path-metadata-regex

XINGEST_LOOP

Provide a default for --loop

XINGEST_WAIT_PERIOD

Provide a default for --wait-period

XINGEST_COPY_MODE

Provide a default for --copy-mode

XINGEST_LOGGERS

Provide a default for --logger

XINGEST_ADDITIONAL_LOGGERS

Provide a default for --additional-logger

XINGEST_COLLATE_RESOURCES

Provide a default for --collate-resources

xnat-ingest group-orthanc

Groups images stored within an Orthanc instance into directories that can be processed by subsequent processing steps.

URL of the Orthanc instance to connect to

STORE_DIR path to Orthanc's " "StorageDirectory as mounted in pod. DICOM files are hardlinked from the storage " "directory directly to the staging directory. (XINGEST_ORTHANC_STORE_DIR env. var)"

OUTPUT_DIR is the directory that the files for each session are collated to before they are uploaded to XNAT

USERNAME for the Orthanc user

PASSWORD for the Orthanc user

Usage

xnat-ingest group-orthanc [OPTIONS] URL STORE_DIR OUTPUT_DIR USER PASSWORD

Options

--to-process-label <to_process_label>

Label applied to Orthanc studies after staging to prevent re-processing. Can be removed via the Orthanc UI (XINGEST_ORTHANC_TO_PROCESS env. var)

--processed-label <processed_label>

Label applied to Orthanc studies after staging to prevent re-processing. Can be removed via the Orthanc UI (XINGEST_ORTHANC_PROCESSED env. var)

Whether to unlink the source studies in Orthanc after they have been successfully grouped. Not yet implemented (XINGEST_UNLINK_SOURCE env. var)

Options:

all | keep-metadata

--loop <loop>

Run the staging process continuously every LOOP seconds (XINGEST_LOOP env. var).

--wait-period <wait_period>

The number of seconds to wait since the last file modification in sessions in the S3 bucket or source file-system directory before uploading them to avoid uploading partial sessions (XINGEST_WAIT_PERIOD env. var).

--copy-mode <copy_mode>

The method to use for copying files (XINGEST_COPY_MODE env. var)

--logger <logtype> <loglevel> <location>

Setup handles to capture logs that are generated (XINGEST_LOGGERS env. var)

--additional-logger <additional_loggers>

The loggers to use for logging. By default just the 'xnat-ingest' logger is used. But additional loggers can be included (e.g. 'xnat') can be specified here (XINGEST_ADDITIONAL_LOGGERS env. var)

--raise-errors, --dont-raise-errors

Whether to raise errors instead of logging them (typically for debugging)

Arguments

URL

Required argument

STORE_DIR

Required argument

OUTPUT_DIR

Required argument

USER

Required argument

PASSWORD

Required argument

Environment variables

XINGEST_ORTHANC_URL

Provide a default for URL

XINGEST_ORTHANC_STORE_DIR

Provide a default for STORE_DIR

XINGEST_ORTHANC_USER

Provide a default for USER

XINGEST_ORTHANC_PASSWORD

Provide a default for PASSWORD

XINGEST_ORTHANC_TO_PROCESS

Provide a default for --to-process-label

XINGEST_ORTHANC_PROCESSED

Provide a default for --processed-label

XINGEST_LOOP

Provide a default for --loop

XINGEST_WAIT_PERIOD

Provide a default for --wait-period

XINGEST_COPY_MODE

Provide a default for --copy-mode

XINGEST_LOGGERS

Provide a default for --logger

XINGEST_ADDITIONAL_LOGGERS

Provide a default for --additional-logger

xnat-ingest assign

Assigns project, subject and session IDs, extracted from session metadata, to sessions that have already been grouped into scans/resources

INPUT_DIR is the path to the directory containing the grouped-but-not-yet-assigned sessions (the output of the 'group' command)

OUTPUT_DIR is the directory that the assigned sessions will be written to

Usage

xnat-ingest assign [OPTIONS] INPUT_DIR OUTPUT_DIR

Options

--project <project_field>

The keyword of the metadata field to extract the XNAT project ID from (XINGEST_PROJECT env. var)

--subject <subject_field>

The keyword of the metadata field to extract the XNAT subject ID from (XINGEST_SUBJECT env. var)

--session <session_field>

The keyword of the metadata field to extract the XNAT session ID from (XINGEST_SESSION env. var)

--scan <scan_field>

The keyword of the metadata field to extract a description for each scan from. Scans for which the field can't be resolved are left without a description (XINGEST_SCAN_DESC env. var)

--constant-project-id <constant_project_id>

Fix the project ID as a constant for all data matched by this command

--loop <loop>

Run the assign process continuously every LOOP seconds (XINGEST_LOOP env. var).

--copy-mode <copy_mode>

The method to use for copying files (XINGEST_COPY_MODE env. var)

Whether to unlink the grouped session directories after they have been assigned. 'all' removes the whole grouped session directory; 'keep-metadata' removes the resource data but leaves the session/scan-level metadata behind, so a lightweight skeleton of the session survives (e.g. for 'associate' to use later). If not set, the grouped directories are left in place (XINGEST_UNLINK_SOURCE env. var)

Options:

all | keep-metadata

--logger <logtype> <loglevel> <location>

Setup handles to capture logs that are generated (XINGEST_LOGGERS env. var)

--additional-logger <additional_loggers>

The loggers to use for logging. By default just the 'xnat-ingest' logger is used. But additional loggers can be included (e.g. 'xnat') can be specified here (XINGEST_ADDITIONAL_LOGGERS env. var)

--raise-errors, --dont-raise-errors

Whether to raise errors instead of logging them (typically for debugging)

Arguments

INPUT_DIR

Required argument

OUTPUT_DIR

Required argument

Environment variables

XINGEST_INPUT_DIR

Provide a default for INPUT_DIR

XINGEST_OUTPUT_DIR

Provide a default for OUTPUT_DIR

XINGEST_PROJECT

Provide a default for --project

XINGEST_SUBJECT

Provide a default for --subject

XINGEST_SESSION

Provide a default for --session

XINGEST_SCAN_DESC

Provide a default for --scan

XINGEST_LOOP

Provide a default for --loop

XINGEST_COPY_MODE

Provide a default for --copy-mode

XINGEST_LOGGERS

Provide a default for --logger

XINGEST_ADDITIONAL_LOGGERS

Provide a default for --additional-logger

xnat-ingest deidentify

Stages images found in the input directories into separate directories for each imaging acquisition session

INPUT_DIR is the path to the directory containing the session directories to de-identify. Each session directory should be named in the format <project_id>.<subject_id>.<session_id> and contain subdirectories for each scan, which in turn contain the resource files for each scan.

OUTPUT_DIR is the directory that the files for each session are collated to before they are uploaded to XNAT

SPEC_DIR is the directory containing the project-specific deidentification specifications. It should contain one subdirectory per project, named <project_id>, plus an optional "__default__" subdirectory used as a fallback for projects that don't have their own. Within each of these subdirectories, there is one JSON spec file per file format that requires deidentification in that project, named after the format's MIME-like identifier with '/' replaced by '@' (e.g. 'medimage/dicom-series' -> 'medimage@dicom-series.json'). Formats without a matching spec file are only deidentified if a spec is found for a broader/parent format (e.g. a 'medimage/dicom-collection' spec also covers 'medimage/dicom-series').

REID_DIR is the directory to save the re-identification metadata to, which can be used to re-identify the de-identified data if needed. The re-identification metadata is saved in JSON format, with one JSON file per session, containing a list of mappings from original to de-identified identifiers for each resource in the session, as well as any additional metadata needed for re-identification (e.g. DICOM tags that were modified during de-identification). The re-identification metadata files are named <session_id>.json (or <session_id>.json.enc if encrypted by --reid-encrypt-key option) and saved in the REID_DIR.

Usage

xnat-ingest deidentify [OPTIONS] INPUT_DIR OUTPUT_DIR SPEC_DIR REID_DIR

Options

Whether to unlink the assigned session directories after they have been deidentified. 'all' removes the whole assigned session directory; 'keep-metadata' removes the resource data but leaves the session/scan-level metadata behind, so a lightweight skeleton of the session survives (e.g. for 'associate' to use later). If not set, the assigned directories are left in place (XINGEST_UNLINK_SOURCE env. var)

Options:

all | keep-metadata

--logger <logtype> <loglevel> <location>

Setup handles to capture logs that are generated (XINGEST_LOGGERS env. var)

--additional-logger <additional_loggers>

The loggers to use for logging. By default just the 'xnat-ingest' logger is used. But additional loggers can be included (e.g. 'xnat') can be specified here (XINGEST_ADDITIONAL_LOGGERS env. var)

--raise-errors, --dont-raise-errors

Whether to raise errors instead of logging them (typically for debugging)

--require-manifest, --no-require-manifest

Whether to require a MANIFEST.json file in each resource directory. If False, resources are loaded as generic FileSets without checksum validation (XINGEST_REQUIRE_MANIFEST env. var)

--copy-mode <copy_mode>

The method to use for copying files (XINGEST_COPY_MODE env. var)

--loop <loop>

Run the staging process continuously every LOOP seconds (XINGEST_LOOP env. var).

--avoid-clashes, --dont-avoid-clashes

If a resource with the same name already exists in the scan, increment the resource name by appending _1, _2 etc. to the name until a unique name is found (XINGEST_AVOID_CLASHES env. var)

--reid-encrypt-key <reid_encrypt_key>

An optional encryption key to use for encrypting the re-identification metadata (XINGEST_REID_ENCRYPT_KEY env. var). This should be a URL-safe base64-encoded 32-byte key, e.g. generated using Fernet.generate_key() from the cryptography package

Arguments

INPUT_DIR

Required argument

OUTPUT_DIR

Required argument

SPEC_DIR

Required argument

REID_DIR

Required argument

Environment variables

XINGEST_INPUT_DIR

Provide a default for INPUT_DIR

XINGEST_OUTPUT_DIR

Provide a default for OUTPUT_DIR

XINGEST_SPEC_DIR

Provide a default for SPEC_DIR

XINGEST_REID_DIR

Provide a default for REID_DIR

XINGEST_LOGGERS

Provide a default for --logger

XINGEST_ADDITIONAL_LOGGERS

Provide a default for --additional-logger

XINGEST_REQUIRE_MANIFEST

Provide a default for --require-manifest

XINGEST_COPY_MODE

Provide a default for --copy-mode

XINGEST_LOOP

Provide a default for --loop

XINGEST_AVOID_CLASHES

Provide a default for --avoid-clashes

XINGEST_REID_ENCRYPT_KEY

Provide a default for --reid-encrypt-key

xnat-ingest upload

uploads all sessions found in the staging directory (as prepared by the stage sub-command) to XNAT.

STAGED is either a directory that the files for each session are collated to before they are uploaded to XNAT or an S3 bucket to download the files from.

SERVER is address of the XNAT server to upload the scans up to. Can alternatively provided by setting the "XNAT_INGEST_HOST" environment variable.

Usage

xnat-ingest upload [OPTIONS] STAGED SERVER

Options

--user <user>

the XNAT user to connect with (alternatively the "XINGEST_USER" env. variable can be used.

--password <password>

the password for the XNAT user, alternatively "XINGEST_PASS" env. var

--logger <logtype> <loglevel> <location>

Setup handles to capture logs that are generated (XINGEST_LOGGERS env. var)

--additional-logger <additional_loggers>

The loggers to use for logging. By default just the 'xnat-ingest' logger is used. But additional loggers can be included (e.g. 'xnat') can be specified here (XINGEST_ADDITIONAL_LOGGERS env. var).

-i, --always-include <always_include>

Scan types to always include in the upload, regardless of whether they arespecified in a column or not. Specified using the scan types IANA mime-type or fileformats "mime-like" (see https://arcanaframework.github.io/fileformats/), e.g. 'application/json', 'medimage/dicom-series', 'image/jpeg'). Use 'all' to include all file-types in the session (XINGEST_ALWAYSINCLUDE env. var).

--raise-errors, --dont-raise-errors

Whether to raise errors instead of logging them (typically for debugging)

--store-credentials <access-key> <secret-key>

Credentials to use to access of data stored in remote stores (e.g. AWS S3) (XINGEST_STORE_CREDENTIALS env. var)

--temp-dir <temp_dir>

The directory to use for temporary downloads (i.e. from s3)

--require-manifest, --dont-require-manifest

Whether to require manifest files in the staged resources or not

--verify-ssl, --dont-verify-ssl

Whether to verify the SSL certificate of the XNAT server

--use-curl-jsession, --dont-use-curl-jsession

Whether to use CURL to create a JSESSION token to authenticate with XNAT. This is used to work around a strange authentication issue when running within a Kubernetes cluster and targeting the XNAT Tomcat directly (XINGEST_USE_CURL_JSESSION env. var).

--method <method> <datatype>

The methods to use to upload the file types to XNAT with. Passed through to XNATPy and controls whether directories are tarred and/or gzipped before they are uploaded, by default 'tgz_file' is used (XINGEST_METHODS env. var).

--wait-period <wait_period>

The number of seconds to wait since the last file modification in sessions in the S3 bucket or source file-system directory before uploading them to avoid uploading partial sessions (XINGEST_WAIT_PERIOD env. var).

--loop <loop>

Run the staging process continuously every LOOP seconds (XINGEST_LOOP env. var).

--num-files-per-batch <num_files_per_batch>

When uploading files to XNAT, the number of files to upload in each batch. The number of files that are uploaded in a single batch can be limited to avoid overloading the building of the catalog file. If <= 0 (the default), then all files are uploaded in a single batch (XINGEST_NUM_FILES_PER_BATCH env. var).

--check-checksums, --dont-check-checksums

Whether to check the checksums of the files in the staged resources against the checksums of both the checksums saved in the manifests and verify after upload to verify that they were uploaded correctly (if checksums are enabled site-wide on XNAT, i.e. enableChecksums is set to true in the XNAT configuration)

--dry-run, --no-dry-run

List the sessions that will be uploaded instead of the actually uploading them

Arguments

STAGED

Required argument

SERVER

Required argument

Environment variables

XINGEST_STAGED

Provide a default for STAGED

XINGEST_HOST

Provide a default for SERVER

XINGEST_USER

Provide a default for --user

XINGEST_PASS

Provide a default for --password

XINGEST_LOGGERS

Provide a default for --logger

XINGEST_ADDITIONAL_LOGGERS

Provide a default for --additional-logger

XINGEST_ALWAYSINCLUDE

Provide a default for --always-include

XINGEST_STORE_CREDENTIALS

Provide a default for --store-credentials

XINGEST_TEMPDIR

Provide a default for --temp-dir

XINGEST_REQUIRE_MANIFEST

Provide a default for --require-manifest

XINGEST_VERIFY_SSL

Provide a default for --verify-ssl

XINGEST_USE_CURL_JSESSION

Provide a default for --use-curl-jsession

XINGEST_METHODS

Provide a default for --method

XINGEST_WAIT_PERIOD

Provide a default for --wait-period

XINGEST_LOOP

Provide a default for --loop

XINGEST_NUM_FILES_PER_BATCH

Provide a default for --num-files-per-batch

XINGEST_CHECK_CHECKSUMS

Provide a default for --check-checksums

XINGEST_DRY_RUN

Provide a default for --dry-run

xnat-ingest check-upload

Checks staging directory against uploaded files and logs all files that aren't uploaded

STAGED is either a directory that the files for each session are collated to before they are uploaded to XNAT or an S3 bucket to download the files from.

SERVER is address of the XNAT server to upload the scans up to. Can alternatively provided by setting the "XNAT_INGEST_HOST" environment variable.

Usage

xnat-ingest check-upload [OPTIONS] INPUT_DIR SERVER

Options

--user <user>

the XNAT user to connect with (alternatively the "XINGEST_USER" env. variable can be used.

--password <password>

the password for the XNAT user, alternatively "XINGEST_PASS" env. var

--logger <logtype> <loglevel> <location>

Setup handles to capture logs that are generated (XINGEST_LOGGERS env. var)

--additional-logger <additional_loggers>

The loggers to use for logging. By default just the 'xnat-ingest' logger is used. But additional loggers can be included (e.g. 'xnat') can be specified here (XINGEST_ADDITIONAL_LOGGERS env. var).

-i, --always-include <always_include>

Scan types to always include in the upload, regardless of whether they arespecified in a column or not. Specified using the scan types IANA mime-type or fileformats "mime-like" (see https://arcanaframework.github.io/fileformats/), e.g. 'application/json', 'medimage/dicom-series', 'image/jpeg'). Use 'all' to include all file-types in the session (XINGEST_ALWAYSINCLUDE env. var).

--store-credentials <access-key> <secret-key>

Credentials to use to access of data stored in remote stores (e.g. AWS S3)

--temp-dir <temp_dir>

The directory to use for temporary downloads (i.e. from s3) (XINGEST_TEMPDIR env. var)

--verify-ssl, --dont-verify-ssl

Whether to verify the SSL certificate of the XNAT server (XINGEST_VERIFY_SSL env. var)

--use-curl-jsession, --dont-use-curl-jsession

Whether to use CURL to create a JSESSION token to authenticate with XNAT. This is used to work around a strange authentication issue when running within a Kubernetes cluster and targeting the XNAT Tomcat directly (XINGEST_USE_CURL_JSESSION env. var).

--disable-progress, --enable-progress

Disable the progress bar

Arguments

INPUT_DIR

Required argument

SERVER

Required argument

Environment variables

XINGEST_STAGED

Provide a default for INPUT_DIR

XINGEST_HOST

Provide a default for SERVER

XINGEST_USER

Provide a default for --user

XINGEST_PASS

Provide a default for --password

XINGEST_LOGGERS

Provide a default for --logger

XINGEST_ADDITIONAL_LOGGERS

Provide a default for --additional-logger

XINGEST_ALWAYSINCLUDE

Provide a default for --always-include

XINGEST_STORE_CREDENTIALS

Provide a default for --store-credentials

XINGEST_TEMPDIR

Provide a default for --temp-dir

XINGEST_VERIFY_SSL

Provide a default for --verify-ssl

XINGEST_USE_CURL_JSESSION

Provide a default for --use-curl-jsession

XINGEST_DISABLE_PROGRESS

Provide a default for --disable-progress

xnat-ingest associate

Stages images found in the input directories into separate directories for each imaging acquisition session

FILES_PATH is either the path to a directory containing the files to upload, or a glob pattern that selects the paths directly

OUTPUT_DIR is the directory that the files for each session are collated to before they are uploaded to XNAT

Usage

xnat-ingest associate [OPTIONS] INPUT_DIR OUTPUT_DIR DATATYPE GLOB ID_PATTERN

Options

--copy-mode <copy_mode>

The method to use for copying files (XINGEST_COPY_MODE env. var)

Whether to unlink the source files matched by GLOB after they have been associated. 'all' and 'keep-metadata' behave the same here (individual source files are removed one by one), since these files live at an arbitrary external location rather than a directory tree xnat-ingest owns. If not set, the source files are left in place (XINGEST_UNLINK_SOURCE env. var)

Options:

all | keep-metadata

--logger <logtype> <loglevel> <location>

Setup handles to capture logs that are generated (XINGEST_LOGGERS env. var)

--additional-logger <additional_loggers>

The loggers to use for logging. By default just the 'xnat-ingest' logger is used. But additional loggers can be included (e.g. 'xnat') can be specified here (XINGEST_ADDITIONAL_LOGGERS env. var)

--raise-errors, --dont-raise-errors

Whether to raise errors instead of logging them (typically for debugging)

--spaces-to-underscores, --no-spaces-to-underscores

Whether to replace spaces with underscores in the filenames of associated files (XINGEST_SPACES_TO_UNDERSCORES env. var)

--loop <loop>

Run the staging process continuously every LOOP seconds (XINGEST_LOOP env. var).

--avoid-clashes, --dont-avoid-clashes

If a resource with the same name already exists in the scan, increment the resource name by appending _1, _2 etc. to the name until a unique name is found (XINGEST_AVOID_CLASHES env. var)

--temp-dir <temp_dir>

The directory to use for temporary downloads (i.e. from s3)

--require-manifest, --dont-require-manifest

Whether to require manifest files in the staged resources or not

Arguments

INPUT_DIR

Required argument

OUTPUT_DIR

Required argument

DATATYPE

Required argument

GLOB

Required argument

ID_PATTERN

Required argument

Environment variables

XINGEST_INPUT_DIR

Provide a default for INPUT_DIR

XINGEST_OUTPUT_DIR

Provide a default for OUTPUT_DIR

XINGEST_DATATYPE

Provide a default for DATATYPE

XINGEST_GLOB

Provide a default for GLOB

XINGEST_ID_PATTERN

Provide a default for ID_PATTERN

XINGEST_COPY_MODE

Provide a default for --copy-mode

XINGEST_LOGGERS

Provide a default for --logger

XINGEST_ADDITIONAL_LOGGERS

Provide a default for --additional-logger

XINGEST_SPACES_TO_UNDERSCORES

Provide a default for --spaces-to-underscores

XINGEST_LOOP

Provide a default for --loop

XINGEST_AVOID_CLASHES

Provide a default for --avoid-clashes

XINGEST_TEMPDIR

Provide a default for --temp-dir

XINGEST_REQUIRE_MANIFEST

Provide a default for --require-manifest