Command-line interface

XNAT Ingest's command line interface consists of a number of sub-commands under the xnat-ingest command. See Basic ingest workflow for how these fit together into a pipeline.

xnat-ingest group

Groups images found in the input paths into separate resources, grouped into scans and acquisition sessions

INPUT_PATHS are either paths to directories containing the files to upload, or glob patterns that select the paths directly

OUTPUT_DIR is the directory that the files for each session are collated to before they are uploaded to XNAT

Usage

xnat-ingest group [OPTIONS] [INPUT_PATHS]... OUTPUT_DIR

Options

--session <session>

The metadata field used to group files into the same session before IDs are extracted (XINGEST_SESSION env. var). Defaults to 'StudyInstanceUID' for DICOM collections; other fileset types need an explicit spec. Can also be a Python format string over several fields, e.g. '{PatientID}_{StudyDate:%Y%m%d}', to compose one.

--scan <specifier> <datatype>

The keyword of the metadata field to extract the XNAT imaging scan ID from, or a Python format string over several fields (see --session). Defaults to 'SeriesNumber' for DICOM collections (one scan per acquisition series); for any fileset type not matched by a --scan spec the scan is named after the resource. (XINGEST_SCAN env. var)

--resource <specifier> <datatype>

The keywords of the metadata field to extract the XNAT imaging resource ID from for different datatypes (use generic/file-set as a catch-all if required), or a Python format string over several fields (see --session). If not given, each resource is labelled with the mime-like rendering of its fileset type name, e.g. 'vectra-export', 'sqlite3-db'. (XINGEST_RESOURCE env. var)

--datatype <mime-type>

The MIME-type(s) (or "MIME-like" see FileFormats docs) of potential datatype(s) of the primary files to to upload, defaults to 'medimage/dicom-series'. Any formats implemented in the FileFormats Python package (https://github.com/ArcanaFramework/fileformats) that implement the 'read_metadata' "extra" are supported, see FF docs on how to add support for new formats.

--path-metadata-regex <regex> <datatype>

Regular expressions to extract "metadata" values from resource file paths as named groups. using Python regular expression syntax. The named groups are used as metadata fields for the resource files, and the extracted values will be used to populate the corresponding metadata fields to complement the metadata read from the file headers.

--on-resource-clash <policy> <scope>

How to handle two filesets resolving to the same scan/resource name: a <policy> ('avoid' appends a _2/_3 suffix, 'merge' folds both into one SetOf, 'overwrite' replaces) plus the datatype <scope> it applies to (a mime-like, a '|'-union, or 'all'). Repeatable. A clash is resolved by the first spec whose scope covers BOTH filesets; a clash no spec covers raises. (XINGEST_ON_RESOURCE_CLASH env. var)

--allow-unrecognised, --allow-unrecognized <allow_unrecognised>

Regular expressions matched against the basename of any input path that no --datatype recognised; matches are skipped instead of raising. Use '.*' to tolerate all unrecognised files. Does not affect recognised filesets. Repeatable; the XINGEST_ALLOW_UNRECOGNISED env. var takes a ';'-separated list.

--exclude-path <exclude_paths>

Glob patterns matched against each input path relative to <input-dir>, applied before classification so a match is dropped even if a --datatype would claim it (e.g. a vendor thumbnail that is a valid image/png). '*' does not cross '/', '**' does. Repeatable; the XINGEST_EXCLUDE_PATH env. var takes a ';'-separated list.

--ignore-datatype <ignore_datatypes>

Datatypes expected in the input but not wanted: recognised files of these types are dropped instead of raising, and with --recursive a matching directory is skipped without descending into it. A path matching neither --datatype nor --ignore-datatype nor --allow-unrecognised/--exclude-path still raises. (XINGEST_IGNORE_DATATYPE env. var)

--loop <loop>

Run the staging process continuously every LOOP seconds (XINGEST_LOOP env. var).

--wait-period <wait_period>

The number of seconds to wait since the last file modification in sessions in the S3 bucket or source file-system directory before uploading them to avoid uploading partial sessions (XINGEST_WAIT_PERIOD env. var).

--recursive, --not-recursive

Whether to recursively search input directories for input files

--copy-mode <copy_mode>

The method to use for copying files (XINGEST_COPY_MODE env. var)

Whether to unlink the source files after they have been successfully grouped. 'all' and 'keep-metadata' behave the same here (individual source files are removed one by one) since 'group's source isn't a directory tree that xnat-ingest owns and can't have a metadata skeleton left behind in it (XINGEST_UNLINK_SOURCE env. var)

Options:

all | keep-metadata

--logger <logtype> <loglevel> <location>

Setup handles to capture logs that are generated (XINGEST_LOGGERS env. var)

--additional-logger <additional_loggers>

The loggers to use for logging. By default just the 'xnat-ingest' logger is used. But additional loggers can be included (e.g. 'xnat') can be specified here (XINGEST_ADDITIONAL_LOGGERS env. var)

--raise-errors, --dont-raise-errors

Whether to raise errors instead of logging them (typically for debugging)

--collate-resources <mime-type> <collation>

Flatten files of the given datatype into the resource directory during grouping, regardless of source directory structure (e.g. when grouping from Orthanc). Collation level is one of 'any', 'siblings', or 'adjacent' (default 'siblings').

--convert <src-mime-like> <tgt-spec>

Convert resources of <src-mime-like> to <tgt-spec> during save. <tgt-spec> may be either '<tgt-mime-like>' or '<tgt-mime-like>:key=value[,key=value...]', e.g. 'application/zip:compression=ZIP_STORED'.

--metadata-table <path> <row-frequency> <join-exprs>

Specify metadata tables to extract and join metadata from input files (XINGEST_METADATA_TABLES env. var). The 'path' arg specifies the location of the metadata table file. Its format is auto-detected as CSV or TSV from the file extension; a different format can be forced by appending its mime-type in square brackets, e.g. 'path/to/table.dat[text/csv]'. The "row frequency" arg specifies what each row in the metadata table corresponds to in the data hierarchy, and can be one of 'session', 'scan', 'resource', 'fileset', 'fileset[<mime-type>]'. When one or more mime-types are given in square brackets after 'fileset' they restrict the join to input files of those types (multiple mime-types can be '|'-separated, e.g. 'fileset[image/png|image/jpeg]'); a bare 'fileset' matches any input file. The 'join-exprs' arg is a comma-separated list of '<column-name>=<cell-value>' expressions; a row is a match when every expression holds. The '<cell-value>' is either the name of an existing metadata field or a Python format string over one or more metadata fields, e.g. '{PatientID}_{SessionID}'. All columns of the matched row are then merged into the target's metadata. The example below extracts the relative path of an image file with --path-metadata-regex and uses it to join a table whose 'ImagePath' column holds spreadsheet HYPERLINK() formulas.

xnat-ingest group ...

--path-metadata-regex '.*/(?P<relpath>[w-]+/[w-]+.(?:png|jpg))' image/png|image/jpeg --metadata-table path/to/table.csv[text/csv] fileset[image/png|image/jpeg] 'ImagePath=HYPERLINK("{relpath}")'

Arguments

INPUT_PATHS

Optional argument(s)

OUTPUT_DIR

Required argument

Environment variables

XINGEST_INPUT_PATHS

Provide a default for INPUT_PATHS

XINGEST_OUTPUT_DIR

Provide a default for OUTPUT_DIR

XINGEST_SESSION

Provide a default for --session

XINGEST_SCAN

Provide a default for --scan

XINGEST_RESOURCE

Provide a default for --resource

XINGEST_DATATYPES

Provide a default for --datatype

XINGEST_PATH_METADATA_REGEX

Provide a default for --path-metadata-regex

XINGEST_ON_RESOURCE_CLASH

Provide a default for --on-resource-clash

XINGEST_ALLOW_UNRECOGNISED

Provide a default for --allow-unrecognised

XINGEST_EXCLUDE_PATH

Provide a default for --exclude-path

XINGEST_IGNORE_DATATYPE

Provide a default for --ignore-datatype

XINGEST_LOOP

Provide a default for --loop

XINGEST_WAIT_PERIOD

Provide a default for --wait-period

XINGEST_RECURSIVE

Provide a default for --recursive

XINGEST_COPY_MODE

Provide a default for --copy-mode

XINGEST_LOGGERS

Provide a default for --logger

XINGEST_ADDITIONAL_LOGGERS

Provide a default for --additional-logger

XINGEST_COLLATE_RESOURCES

Provide a default for --collate-resources

XINGEST_CONVERT

Provide a default for --convert

XINGEST_METADATA_TABLES

Provide a default for --metadata-table

xnat-ingest group-orthanc

Groups images stored within an Orthanc instance into directories that can be processed by subsequent processing steps.

URL of the Orthanc instance to connect to

STORE_DIR path to Orthanc's " "StorageDirectory as mounted in pod. DICOM files are hardlinked from the storage " "directory directly to the staging directory. (XINGEST_ORTHANC_STORE_DIR env. var)"

OUTPUT_DIR is the directory that the files for each session are collated to before they are uploaded to XNAT

USERNAME for the Orthanc user

PASSWORD for the Orthanc user

Usage

xnat-ingest group-orthanc [OPTIONS] URL STORE_DIR OUTPUT_DIR USER PASSWORD

Options

--to-process-label <to_process_label>

Label applied to Orthanc studies after staging to prevent re-processing. Can be removed via the Orthanc UI (XINGEST_ORTHANC_TO_PROCESS env. var)

--processed-label <processed_label>

Label applied to Orthanc studies after staging to prevent re-processing. Can be removed via the Orthanc UI (XINGEST_ORTHANC_PROCESSED env. var)

Whether to unlink the source studies in Orthanc after they have been successfully grouped. Not yet implemented (XINGEST_UNLINK_SOURCE env. var)

Options:

all | keep-metadata

--loop <loop>

Run the staging process continuously every LOOP seconds (XINGEST_LOOP env. var).

--wait-period <wait_period>

The number of seconds to wait since the last file modification in sessions in the S3 bucket or source file-system directory before uploading them to avoid uploading partial sessions (XINGEST_WAIT_PERIOD env. var).

--copy-mode <copy_mode>

The method to use for copying files (XINGEST_COPY_MODE env. var)

--logger <logtype> <loglevel> <location>

Setup handles to capture logs that are generated (XINGEST_LOGGERS env. var)

--additional-logger <additional_loggers>

The loggers to use for logging. By default just the 'xnat-ingest' logger is used. But additional loggers can be included (e.g. 'xnat') can be specified here (XINGEST_ADDITIONAL_LOGGERS env. var)

--raise-errors, --dont-raise-errors

Whether to raise errors instead of logging them (typically for debugging)

Arguments

URL

Required argument

STORE_DIR

Required argument

OUTPUT_DIR

Required argument

USER

Required argument

PASSWORD

Required argument

Environment variables

XINGEST_ORTHANC_URL

Provide a default for URL

XINGEST_ORTHANC_STORE_DIR

Provide a default for STORE_DIR

XINGEST_ORTHANC_USER

Provide a default for USER

XINGEST_ORTHANC_PASSWORD

Provide a default for PASSWORD

XINGEST_ORTHANC_TO_PROCESS

Provide a default for --to-process-label

XINGEST_ORTHANC_PROCESSED

Provide a default for --processed-label

XINGEST_LOOP

Provide a default for --loop

XINGEST_WAIT_PERIOD

Provide a default for --wait-period

XINGEST_COPY_MODE

Provide a default for --copy-mode

XINGEST_LOGGERS

Provide a default for --logger

XINGEST_ADDITIONAL_LOGGERS

Provide a default for --additional-logger

xnat-ingest assign

Assigns project, subject and session IDs, extracted from session metadata, to sessions that have already been grouped into scans/resources

INPUT_DIR is the path to the directory containing the grouped-but-not-yet-assigned sessions (the output of the 'group' command)

OUTPUT_DIR is the directory that the assigned sessions will be written to

If a project/subject/session ID can't be resolved from a session's metadata, it is assigned a unique 'INVALID_MISSING_<FIELD>_<random>' placeholder instead of being dropped, and the whole session is saved under an '__invalid__' subdirectory of OUTPUT_DIR instead of alongside normally-assigned ones, so it can be found and manually reviewed/reprocessed rather than silently lost.

Usage

xnat-ingest assign [OPTIONS] INPUT_DIR OUTPUT_DIR

Options

--project <project_field>

The keyword of the metadata field to extract the XNAT project ID from, or a Python format string over several fields, e.g. '{PatientID}_{StudyDate:%Y%m%d}', to compose one (XINGEST_PROJECT env. var)

--subject <subject_field>

The keyword of the metadata field to extract the XNAT subject ID from, or a Python format string over several fields (see --project) (XINGEST_SUBJECT env. var)

--session <session_field>

The keyword of the metadata field to extract the XNAT session ID from, or a Python format string over several fields (see --project) (XINGEST_SESSION env. var)

--scan <scan_field>

The keyword of the metadata field to extract a description for each scan from. Scans for which the field can't be resolved are left without a description (XINGEST_SCAN env. var)

--constant-project-id <constant_project_id>

Fix the project ID as a constant for all data matched by this command (XINGEST_CONSTANT_PROJECT_ID env. var)

--include <mime-type>

Only assign resources matching this fileformats MIME-like datatype. May be repeated to include multiple datatypes (XINGEST_INCLUDE env. var)

--loop <loop>

Run the assign process continuously every LOOP seconds (XINGEST_LOOP env. var).

--copy-mode <copy_mode>

The method to use for copying files (XINGEST_COPY_MODE env. var)

Whether to unlink the grouped session directories after they have been assigned. 'all' removes the whole grouped session directory; 'keep-metadata' removes the resource data but leaves the session/scan-level metadata behind, so a lightweight skeleton of the session survives (e.g. for 'associate' to use later). If not set, the grouped directories are left in place (XINGEST_UNLINK_SOURCE env. var)

Options:

all | keep-metadata

--logger <logtype> <loglevel> <location>

Setup handles to capture logs that are generated (XINGEST_LOGGERS env. var)

--additional-logger <additional_loggers>

The loggers to use for logging. By default just the 'xnat-ingest' logger is used. But additional loggers can be included (e.g. 'xnat') can be specified here (XINGEST_ADDITIONAL_LOGGERS env. var)

--raise-errors, --dont-raise-errors

Whether to raise errors instead of logging them (typically for debugging)

Arguments

INPUT_DIR

Required argument

OUTPUT_DIR

Required argument

Environment variables

XINGEST_INPUT_DIR

Provide a default for INPUT_DIR

XINGEST_OUTPUT_DIR

Provide a default for OUTPUT_DIR

XINGEST_PROJECT

Provide a default for --project

XINGEST_SUBJECT

Provide a default for --subject

XINGEST_SESSION

Provide a default for --session

XINGEST_SCAN

Provide a default for --scan

XINGEST_CONSTANT_PROJECT_ID

Provide a default for --constant-project-id

XINGEST_INCLUDE

Provide a default for --include

XINGEST_LOOP

Provide a default for --loop

XINGEST_COPY_MODE

Provide a default for --copy-mode

XINGEST_LOGGERS

Provide a default for --logger

XINGEST_ADDITIONAL_LOGGERS

Provide a default for --additional-logger

xnat-ingest deidentify

Stages images found in the input directories into separate directories for each imaging acquisition session

INPUT_DIR is the path to the directory containing the session directories to de-identify. Each session directory should be named in the format <project_id>.<subject_id>.<session_id> and contain subdirectories for each scan, which in turn contain the resource files for each scan.

OUTPUT_DIR is the directory that the files for each session are collated to before they are uploaded to XNAT

SPEC_DIR is the directory containing the project-specific deidentification specifications. It should contain one subdirectory per project, named <project_id>, plus an optional "__default__" subdirectory used as a fallback for projects that don't have their own. Within each of these subdirectories, the directory structure mirrors the MIME-like hierarchy: a subdirectory per category containing one spec file per format (e.g. 'medimage/dicom-series'). Any file extension (or none) is accepted. Optionally, a transforms file named '<format>.transforms.py' (e.g. 'medimage/dicom-series.transforms.py') can sit alongside the spec to define callable transforms for computed replacement values. Formats without a matching spec file are only deidentified if a spec is found for a broader/parent format (e.g. a 'medimage/dicom-collection' spec also covers 'medimage/dicom-series').

The re-identification metadata (a mapping from original to de-identified identifiers for each resource in the session, plus any DICOM tags modified during de-identification) is only written if --reid-dir is given; see that option.

Usage

xnat-ingest deidentify [OPTIONS] INPUT_DIR OUTPUT_DIR SPEC_DIR

Options

--reid-dir <reid_dir>

The directory to save the re-identification metadata to, which can be used to re-identify the de-identified data if needed. One JSON file is written per session, named <session_id>.json (or <session_id>.json.enc if encrypted with --reid-encrypt-key). If this option is not given, the re-identification metadata is discarded rather than written to disk, which is the safer default when retaining the original<->deidentified mapping would itself be a security concern (XINGEST_REID_DIR env. var)

Whether to unlink the assigned session directories after they have been deidentified. 'all' removes the whole assigned session directory; 'keep-metadata' removes the resource data but leaves the session/scan-level metadata behind, so a lightweight skeleton of the session survives (e.g. for 'associate' to use later). If not set, the assigned directories are left in place (XINGEST_UNLINK_SOURCE env. var)

Options:

all | keep-metadata

--logger <logtype> <loglevel> <location>

Setup handles to capture logs that are generated (XINGEST_LOGGERS env. var)

--additional-logger <additional_loggers>

The loggers to use for logging. By default just the 'xnat-ingest' logger is used. But additional loggers can be included (e.g. 'xnat') can be specified here (XINGEST_ADDITIONAL_LOGGERS env. var)

--raise-errors, --dont-raise-errors

Whether to raise errors instead of logging them (typically for debugging)

--require-manifest, --no-require-manifest

Whether to require a MANIFEST.json file in each resource directory. If False, resources are loaded as generic FileSets without checksum validation (XINGEST_REQUIRE_MANIFEST env. var)

--copy-mode <copy_mode>

The method to use for copying files (XINGEST_COPY_MODE env. var)

--loop <loop>

Run the staging process continuously every LOOP seconds (XINGEST_LOOP env. var).

--on-resource-clash <on_resource_clash>

Determines the behavior when a resource with the same name already exists in the scan. Options are 'merge', 'avoid', 'error' (XINGEST_ON_RESOURCE_CLASH env. var).Default: 'error'

Options:

error | avoid | merge | overwrite

--reid-encrypt-key <reid_encrypt_key>

An optional encryption key to use for encrypting the re-identification metadata (XINGEST_REID_ENCRYPT_KEY env. var). This should be a URL-safe base64-encoded 32-byte key, e.g. generated using Fernet.generate_key() from the cryptography package

Arguments

INPUT_DIR

Required argument

OUTPUT_DIR

Required argument

SPEC_DIR

Required argument

Environment variables

XINGEST_INPUT_DIR

Provide a default for INPUT_DIR

XINGEST_OUTPUT_DIR

Provide a default for OUTPUT_DIR

XINGEST_SPEC_DIR

Provide a default for SPEC_DIR

XINGEST_REID_DIR

Provide a default for --reid-dir

XINGEST_LOGGERS

Provide a default for --logger

XINGEST_ADDITIONAL_LOGGERS

Provide a default for --additional-logger

XINGEST_REQUIRE_MANIFEST

Provide a default for --require-manifest

XINGEST_COPY_MODE

Provide a default for --copy-mode

XINGEST_LOOP

Provide a default for --loop

XINGEST_ON_RESOURCE_CLASH

Provide a default for --on-resource-clash

XINGEST_REID_ENCRYPT_KEY

Provide a default for --reid-encrypt-key

xnat-ingest upload

uploads all sessions found in the staging directory (as prepared by the stage sub-command) to XNAT.

STAGED is either a directory that the files for each session are collated to before they are uploaded to XNAT or an S3 bucket to download the files from.

SERVER is address of the XNAT server to upload the scans up to. Can alternatively provided by setting the "XNAT_INGEST_HOST" environment variable.

Usage

xnat-ingest upload [OPTIONS] STAGED SERVER

Options

--user <user>

the XNAT user to connect with (alternatively the "XINGEST_USER" env. variable can be used.

--password <password>

the password for the XNAT user, alternatively "XINGEST_PASS" env. var

--logger <logtype> <loglevel> <location>

Setup handles to capture logs that are generated (XINGEST_LOGGERS env. var)

--additional-logger <additional_loggers>

The loggers to use for logging. By default just the 'xnat-ingest' logger is used. But additional loggers can be included (e.g. 'xnat') can be specified here (XINGEST_ADDITIONAL_LOGGERS env. var).

-i, --always-include <always_include>

Scan types to always include in the upload, regardless of whether they arespecified in a column or not. Specified using the scan types IANA mime-type or fileformats "mime-like" (see https://arcanaframework.github.io/fileformats/), e.g. 'application/json', 'medimage/dicom-series', 'image/jpeg'). Use 'all' to include all file-types in the session (XINGEST_ALWAYSINCLUDE env. var).

--raise-errors, --dont-raise-errors

Whether to raise errors instead of logging them (typically for debugging)

--store-credentials <access-key> <secret-key>

Credentials to use to access of data stored in remote stores (e.g. AWS S3) (XINGEST_STORE_CREDENTIALS env. var)

--temp-dir <temp_dir>

The directory to use for temporary downloads (i.e. from s3)

--require-manifest, --dont-require-manifest

Whether to require manifest files in the staged resources or not

--verify-ssl, --dont-verify-ssl

Whether to verify the SSL certificate of the XNAT server

--use-curl-jsession, --dont-use-curl-jsession

Whether to use CURL to create a JSESSION token to authenticate with XNAT. This is used to work around a strange authentication issue when running within a Kubernetes cluster and targeting the XNAT Tomcat directly (XINGEST_USE_CURL_JSESSION env. var).

--method <method> <datatype>

The methods to use to upload the file types to XNAT with. Passed through to XNATPy and controls whether directories are tarred and/or gzipped before they are uploaded, by default 'tgz_file' is used (XINGEST_METHODS env. var).

--wait-period <wait_period>

The number of seconds to wait since the last file modification in sessions in the S3 bucket or source file-system directory before uploading them to avoid uploading partial sessions (XINGEST_WAIT_PERIOD env. var).

--loop <loop>

Run the staging process continuously every LOOP seconds (XINGEST_LOOP env. var).

--num-files-per-batch <num_files_per_batch>

When uploading files to XNAT, the number of files to upload in each batch. The number of files that are uploaded in a single batch can be limited to avoid overloading the building of the catalog file. If <= 0 (the default), then all files are uploaded in a single batch (XINGEST_NUM_FILES_PER_BATCH env. var).

--check-checksums, --dont-check-checksums

Whether to check the checksums of the files in the staged resources against the checksums of both the checksums saved in the manifests and verify after upload to verify that they were uploaded correctly (if checksums are enabled site-wide on XNAT, i.e. enableChecksums is set to true in the XNAT configuration)

--dry-run, --no-dry-run

List the sessions that will be uploaded instead of the actually uploading them

Arguments

STAGED

Required argument

SERVER

Required argument

Environment variables

XINGEST_STAGED

Provide a default for STAGED

XINGEST_HOST

Provide a default for SERVER

XINGEST_USER

Provide a default for --user

XINGEST_PASS

Provide a default for --password

XINGEST_LOGGERS

Provide a default for --logger

XINGEST_ADDITIONAL_LOGGERS

Provide a default for --additional-logger

XINGEST_ALWAYSINCLUDE

Provide a default for --always-include

XINGEST_STORE_CREDENTIALS

Provide a default for --store-credentials

XINGEST_TEMPDIR

Provide a default for --temp-dir

XINGEST_REQUIRE_MANIFEST

Provide a default for --require-manifest

XINGEST_VERIFY_SSL

Provide a default for --verify-ssl

XINGEST_USE_CURL_JSESSION

Provide a default for --use-curl-jsession

XINGEST_METHODS

Provide a default for --method

XINGEST_WAIT_PERIOD

Provide a default for --wait-period

XINGEST_LOOP

Provide a default for --loop

XINGEST_NUM_FILES_PER_BATCH

Provide a default for --num-files-per-batch

XINGEST_CHECK_CHECKSUMS

Provide a default for --check-checksums

XINGEST_DRY_RUN

Provide a default for --dry-run

xnat-ingest check-upload

Checks staging directory against uploaded files and logs all files that aren't uploaded

STAGED is either a directory that the files for each session are collated to before they are uploaded to XNAT or an S3 bucket to download the files from.

SERVER is address of the XNAT server to upload the scans up to. Can alternatively provided by setting the "XNAT_INGEST_HOST" environment variable.

Usage

xnat-ingest check-upload [OPTIONS] INPUT_DIR SERVER

Options

--user <user>

the XNAT user to connect with (alternatively the "XINGEST_USER" env. variable can be used.

--password <password>

the password for the XNAT user, alternatively "XINGEST_PASS" env. var

--logger <logtype> <loglevel> <location>

Setup handles to capture logs that are generated (XINGEST_LOGGERS env. var)

--additional-logger <additional_loggers>

The loggers to use for logging. By default just the 'xnat-ingest' logger is used. But additional loggers can be included (e.g. 'xnat') can be specified here (XINGEST_ADDITIONAL_LOGGERS env. var).

-i, --always-include <always_include>

Scan types to always include in the upload, regardless of whether they arespecified in a column or not. Specified using the scan types IANA mime-type or fileformats "mime-like" (see https://arcanaframework.github.io/fileformats/), e.g. 'application/json', 'medimage/dicom-series', 'image/jpeg'). Use 'all' to include all file-types in the session (XINGEST_ALWAYSINCLUDE env. var).

--store-credentials <access-key> <secret-key>

Credentials to use to access of data stored in remote stores (e.g. AWS S3)

--temp-dir <temp_dir>

The directory to use for temporary downloads (i.e. from s3) (XINGEST_TEMPDIR env. var)

--verify-ssl, --dont-verify-ssl

Whether to verify the SSL certificate of the XNAT server (XINGEST_VERIFY_SSL env. var)

--use-curl-jsession, --dont-use-curl-jsession

Whether to use CURL to create a JSESSION token to authenticate with XNAT. This is used to work around a strange authentication issue when running within a Kubernetes cluster and targeting the XNAT Tomcat directly (XINGEST_USE_CURL_JSESSION env. var).

--disable-progress, --enable-progress

Disable the progress bar

Arguments

INPUT_DIR

Required argument

SERVER

Required argument

Environment variables

XINGEST_STAGED

Provide a default for INPUT_DIR

XINGEST_HOST

Provide a default for SERVER

XINGEST_USER

Provide a default for --user

XINGEST_PASS

Provide a default for --password

XINGEST_LOGGERS

Provide a default for --logger

XINGEST_ADDITIONAL_LOGGERS

Provide a default for --additional-logger

XINGEST_ALWAYSINCLUDE

Provide a default for --always-include

XINGEST_STORE_CREDENTIALS

Provide a default for --store-credentials

XINGEST_TEMPDIR

Provide a default for --temp-dir

XINGEST_VERIFY_SSL

Provide a default for --verify-ssl

XINGEST_USE_CURL_JSESSION

Provide a default for --use-curl-jsession

XINGEST_DISABLE_PROGRESS

Provide a default for --disable-progress

xnat-ingest associate

Stages images found in the input directories into separate directories for each imaging acquisition session

FILES_PATH is either the path to a directory containing the files to upload, or a glob pattern that selects the paths directly

OUTPUT_DIR is the directory that the files for each session are collated to before they are uploaded to XNAT

Usage

xnat-ingest associate [OPTIONS] INPUT_DIR OUTPUT_DIR DATATYPE GLOB ID_PATTERN

Options

--copy-mode <copy_mode>

The method to use for copying files (XINGEST_COPY_MODE env. var)

Whether to unlink the source files matched by GLOB after they have been associated. 'all' and 'keep-metadata' behave the same here (individual source files are removed one by one), since these files live at an arbitrary external location rather than a directory tree xnat-ingest owns. If not set, the source files are left in place (XINGEST_UNLINK_SOURCE env. var)

Options:

all | keep-metadata

--logger <logtype> <loglevel> <location>

Setup handles to capture logs that are generated (XINGEST_LOGGERS env. var)

--additional-logger <additional_loggers>

The loggers to use for logging. By default just the 'xnat-ingest' logger is used. But additional loggers can be included (e.g. 'xnat') can be specified here (XINGEST_ADDITIONAL_LOGGERS env. var)

--raise-errors, --dont-raise-errors

Whether to raise errors instead of logging them (typically for debugging)

--spaces-to-underscores, --no-spaces-to-underscores

Whether to replace spaces with underscores in the filenames of associated files (XINGEST_SPACES_TO_UNDERSCORES env. var)

--loop <loop>

Run the staging process continuously every LOOP seconds (XINGEST_LOOP env. var).

--on-resource-clash <on_resource_clash>

Determines the behavior when a resource with the same name already exists in the scan. Options are 'merge', 'avoid', 'error' (XINGEST_ON_RESOURCE_CLASH env. var)Default: 'error'

Options:

error | avoid | merge | overwrite

--temp-dir <temp_dir>

The directory to use for temporary downloads (i.e. from s3)

--require-manifest, --dont-require-manifest

Whether to require manifest files in the staged resources or not

Arguments

INPUT_DIR

Required argument

OUTPUT_DIR

Required argument

DATATYPE

Required argument

GLOB

Required argument

ID_PATTERN

Required argument

Environment variables

XINGEST_INPUT_DIR

Provide a default for INPUT_DIR

XINGEST_OUTPUT_DIR

Provide a default for OUTPUT_DIR

XINGEST_DATATYPE

Provide a default for DATATYPE

XINGEST_GLOB

Provide a default for GLOB

XINGEST_ID_PATTERN

Provide a default for ID_PATTERN

XINGEST_COPY_MODE

Provide a default for --copy-mode

XINGEST_LOGGERS

Provide a default for --logger

XINGEST_ADDITIONAL_LOGGERS

Provide a default for --additional-logger

XINGEST_SPACES_TO_UNDERSCORES

Provide a default for --spaces-to-underscores

XINGEST_LOOP

Provide a default for --loop

XINGEST_ON_RESOURCE_CLASH

Provide a default for --on-resource-clash

XINGEST_TEMPDIR

Provide a default for --temp-dir

XINGEST_REQUIRE_MANIFEST

Provide a default for --require-manifest