Folders¶
collate folder reads files from directories on a local path or a mounted file
system, reads each file's metadata (DICOM tags by default), and collates the files into
accessions, acquisitions and resources in OUTPUT_DIR, ready to be labelled (see
Staging directories for the layout).
$ accessionflow collate folder /data/incoming/*.dcm /data/staging/collated
INPUT_PATHS— one or more directories, or glob patterns that select the files directly (ACFLOW_INPUT_PATHS)OUTPUT_DIR— where the collated accessions are written (ACFLOW_OUTPUT_DIR)
Add --unlink-source all to remove each source file once it has been collated (see
Cleaning up previous stages).
Grouping files into accessions, acquisitions and resources¶
Three options decide which accession, acquisition and resource each file belongs to, each taking a specifier and the datatype it applies to:
--accession— files with the same value are collated into the same accession, and the value is its UID. Defaults toStudyInstanceUIDfor DICOM. This isn't DICOM'sAccessionNumber, which is just another field that can be used here or to label the accession--acquisition— files with the same value are collated into the same acquisition, and the value is its ID. Defaults toSeriesNumberfor DICOM. If no--acquisitionspec matches a fileset, its acquisition is named after its resource--resource— names the resource. If no--resourcespec matches a fileset, its resource is named after its type, e.g.dicom-seriesorvectra-export
There are no defaults for formats other than DICOM. Each option can be repeated to treat datatypes differently.
Non-DICOM data (raw exports, image files, vendor directory formats) needs --datatype
set to the relevant FileFormats
MIME-like type(s), and usually its own --accession/--acquisition/--resource specs,
since the DICOM defaults won't apply. Any format that FileFormats can read metadata from
can be collated (see Adding support for new file types).
Pulling extra metadata from file and directory paths¶
Not everything you need is necessarily readable from a file's own headers: a raw format
might carry no metadata at all, or your incoming data might already be organised into
directories that encode something useful (a cohort label, a scanner ID, ...) that never
made it into the DICOM tags. --path-metadata-regex pulls extra metadata fields out
of a resource's own file or directory path instead, through the named groups of a Python
regular expression. It is often the easiest way to get a field that the data store needs
(e.g. a project ID) when the instrument itself doesn't write it anywhere in the file:
$ accessionflow collate folder /data/incoming/cohort-A /data/staging/collated \
--path-metadata-regex '.*/(?P<cohort>[^/]+)$' medimage/dicom-series
Here, anything found under a .../cohort-A/... directory gets a cohort metadata
field set to cohort-A, alongside whatever is read from its DICOM headers. It can be
used anywhere a metadata field can, e.g. in a --project or --level specifier
when labelling.
The regex is matched (with
re.match, so from the start of the string, but not necessarily the whole of it unless you anchor it with$as above) against the resource's own directory, or its file path for single-file formats, not against the wholeINPUT_PATHSargument.The second argument restricts which datatype the pattern applies to. It is matched with
isinstance, so a broader datatype likemedimage/dicom-collectionalso matchesmedimage/dicom-series.It can be repeated for different datatypes and patterns.
If a resource matches the datatype but its path doesn't match the regex, that's a hard error: it aborts the whole run rather than skipping that one accession, so keep the pattern narrow enough that it only applies to data you know follows the convention.
If you don't need the full power of a regular expression, --path-metadata takes a
simpler, glob-like pattern instead, with a {name} placeholder for each part of the
path to extract into the metadata field name. The example above becomes:
$ accessionflow collate folder /data/incoming/cohort-A /data/staging/collated \
--path-metadata '{cohort}' medimage/dicom-series
{name}matches all or part of a single path segment (it doesn't match/).namemust be a valid Python identifier, and a name repeated in the pattern must have the same value each time.*matches anything within a segment,**anything across segments (**/zero or more whole segments),?a single character and[...]/[!...]a (negated) set of characters, as in a shell glob.{{and}}match literal braces, and every other character matches itself.Unless it starts with
/, a pattern only needs to match the trailing segments of the path, so{cohort}matches the last segment of/data/incoming/cohort-A. It always has to match up to the end of the path.
Each pattern is translated into the equivalent regular expression, so the same rules
about the datatype, repeating the option and a path that doesn't match apply. For
example, {subject}_{session}/resource/*.dcm becomes:
(?:.*/)?(?P<subject>[^/]+)_(?P<session>[^/]+)/resource/[^/]*\.dcm$
The two options can be used together: the --path-metadata patterns are applied
first, so a --path-metadata-regex wins where both set the same field.
What the pattern is matched against¶
The pattern (or regex) is matched against the path of the resource, not of each of its
files. A single-file datatype like medimage/dicom-image or generic/file is
matched by the file's own path, but a multi-file one like medimage/dicom-series is
matched by the directory that holds its files. So for files in
/data/sub01_ses02/resource/:
# a DICOM series is matched by its directory, so the pattern ends there
--path-metadata '{subject}_{session}/resource' medimage/dicom-series
# a single file is matched by its own path, so the pattern includes the file name
--path-metadata '{subject}_{session}/resource/*.dcm' medimage/dicom-image
Both set subject to sub01 and session to ses02. A pattern ending in
*.dcm used with medimage/dicom-series never matches, so the run fails.
Placeholders sharing a segment¶
Where two placeholders share a segment, the first takes as much of it as it can, so
{subject}_{session} splits a directory named sub01_ses_02 into subject
sub01_ses and session 02. That's fine as long as the values themselves
never contain _. If they can, use --path-metadata-regex with a group that
excludes it instead, e.g. to stop subject at the first _:
--path-metadata-regex '.*/(?P<subject>[^_/]+)_(?P<session>[^/]+)/resource$' \
medimage/dicom-series
Skipping the file headers¶
When every field you need comes from the path (or a metadata table), reading each
file's header is wasted work, which can add up over many files on a network file
system. --dont-read-metadata <datatype> stops the files of that datatype being read
for metadata, so the only fields they have are the ones set from the path or a table:
$ accessionflow collate folder /data/incoming /data/staging/collated --recursive \
--datatype medimage/nifti-gz \
--path-metadata '{subject}/{subject}_*.nii.gz' medimage/nifti-gz \
--accession subject all \
--dont-read-metadata medimage/nifti-gz
The datatype is matched with isinstance, like --path-metadata's, and the option
can be repeated. A field that would only have come from the header, e.g. one used in a
later --project specifier or by store xnat --date, is then missing.
Joining metadata from a table¶
--metadata-table <path> <row-frequency> <join-exprs> merges the columns of a CSV or
TSV table into the metadata, e.g. a spreadsheet exported alongside the images:
<path>— the table. Its format is detected from its extension; a different format can be forced by appending its MIME type in square brackets, e.g.path/to/table.dat[text/csv]<row-frequency>— what each row describes:accession,acquisition,resource,fileset, orfileset[<mime-type>]to join only files of those types (several can be|-separated, e.g.fileset[image/png|image/jpeg])<join-exprs>— a comma-separated list of<column-name>=<cell-value>expressions, all of which must hold for a row to match. A<cell-value>is a metadata field, or a format string over several, e.g.{PatientID}_{SessionID}
All the columns of the matching row are merged into the target's metadata. This
example extracts the relative path of each image with --path-metadata-regex and
uses it to join a table whose ImagePath column holds spreadsheet HYPERLINK()
formulas:
$ accessionflow collate folder ... \
--path-metadata-regex '.*/(?P<relpath>[\w-]+/[\w-]+\.(?:png|jpg))' 'image/png|image/jpeg' \
--metadata-table path/to/table.csv[text/csv] 'fileset[image/png|image/jpeg]' \
'ImagePath=HYPERLINK("{relpath}")'
Recursing into nested directory formats¶
--recursive descends into the input directories. For plain file datatypes it just
flattens the tree. When a --datatype (or --ignore-datatype) is a directory
format, the walk instead stops descending into a directory as soon as it validates as
one of them: a --datatype match is taken whole (its contents are never looked at
individually), an --ignore-datatype match is skipped whole, and anything else is
descended. This is how you pull a nested directory format out of a larger vendor
export:
$ accessionflow collate folder /data/incoming /data/staging/collated --recursive \
--datatype medimage/vnd.canfield.whole-body-analysis-dir \
--datatype medimage/vnd.canfield.dexi-data-dir \
--exclude-path '*/*/*.png'
generic/directory and generic/file-set can't be used as a --datatype with
--recursive, since they match every directory, at every depth.
Deciding what to skip¶
Anything found under the input paths that no --datatype recognises is a hard error
by default. Three options relax that, from most to least targeted:
--exclude-path <glob>drops a path before it is classified, matching the path relative to its input directory. So it removes a path even if a--datatypewould have claimed it, e.g. a vendor thumbnail that happens to be a validimage/pngthree directories deep, while keeping your realimage/pngtwo deep.*does not cross/,**does. Repeatable.--ignore-datatype <mime>names a datatype that is recognised but isn't wanted: its filesets are dropped from the result, and (with--recursive) matching directories are skipped without descending. Repeatable.--allow-unrecognised <regex>matches the basename of any leftover path that no datatype recognised, and skips it instead of raising.--allow-unrecognised '.*'tolerates every stray file. It never affects a recognised fileset.
A path that matches none of these, nor a --datatype, still aborts the run.
Handling resource-name clashes¶
If two filesets resolve to the same acquisition and resource name, the run aborts unless
you say what to do about it. --on-resource-clash takes <policy> <scope> and is
repeatable; a clash is resolved by the first entry whose <scope> (a MIME-like, a
|-union, or all) covers both filesets:
mergefolds them into a singleSetOf[...]resource, e.g. the PNG and JPEG views of one lesion, with--on-resource-clash merge 'image/png|image/jpeg'avoidappends a__2/__3suffixoverwritereplaces the existing resource
A clash that no entry's scope covers (e.g. a PNG colliding with a directory format)
always raises. Tighten --acquisition/--resource so the two don't collide, or add a scope
that deliberately covers both.
Laying out the files of a resource¶
--collate-resources <mime-type> <collation> flattens the files of a datatype into
their resource directory as they are collated, regardless of how they were laid out at
the source. <collation> is one of any, siblings or adjacent (the default
is siblings).
Series without numbers¶
The default --acquisition spec, SeriesNumber, must resolve for every DICOM series. For
sources that don't number their series, give an explicit
--acquisition SeriesInstanceUID medimage/dicom-collection instead. (collate orthanc
falls back to the SeriesInstanceUID automatically, see Missing series tags.)