Attach files without relevant metadata¶
collate/label sort files using metadata read out of the files themselves (DICOM
tags, by default). Some data that belongs in an accession doesn't work that way — e.g.
raw list-mode or count-rate files from a Siemens PET scanner, which have no readable
patient/session metadata of their own, or live in a completely separate export
location to the DICOMs. If those files can still be found some other way (a shared
naming convention with the primary DICOMs, in the same folder or elsewhere) and you
know which acquisition/resource they belong to from their path, attach can link them
into an accession that's already been through collate (see Sources).
It can run at any point after collate — on collate's own output, on
label's, or later — since it only needs the accession's own metadata (read from the
primary files' own headers), not its labels. The one real ordering constraint is deidentify: run
attach before it if GLOB relies on identifying metadata, as in the
example below ({PatientName.family_name}) — deidentify replaces
PatientName with a fixed placeholder rather than removing it, so the glob would
still evaluate afterward, just silently against the wrong (anonymised) name, and
never find the real files:
$ accessionflow attach /data/staging/labelled /data/staging/attached \
"medimage/vnd.siemens.syngo-mi.vr20b.count-rate|medimage/vnd.siemens.syngo-mi.vr20b.list-mode" \
"/data/raw-exports/{PatientName.family_name}_{PatientName.given_name}*.ptd" \
".*/[^.]+\.[^.]+\.[^.]+\.(?P<id>\d+)\.[A-Z]+_(?P<resource>[^.]+).*"
INPUT_DIR— the directory of accessions to attach files into, from whichever stage you're running this after (ACFLOW_INPUT_DIR)OUTPUT_DIR— where accessions are written with the attached files attached (ACFLOW_OUTPUT_DIR)DATATYPE— the format of the files being linked in, as one or more|-separated MIME-like identifiers (ACFLOW_DATATYPE) — multiple formats can be matched by a single call, as in the example aboveGLOB— a glob pattern used to find the files, with{field}placeholders filled in from the accession's own metadata (e.g.{PatientName.family_name}) (ACFLOW_GLOB). A relative pattern is resolved against the directory the primary DICOMs came from; an absolute one (as above) searches a separate location entirely, ignoring where the DICOMs liveID_PATTERN— a regular expression matched against each found file's path (ACFLOW_ID_PATTERN), with named groupsid(the acquisition ID to attach it to) andresource(the resource name to file it under), and optionallytype(the acquisition type/description, which otherwise defaults to the acquisition ID)
Attaching files after cleanup (metadata skeletons)¶
Attached files sometimes turn up late — after an accession's primary DICOMs have
already been through label/deidentify/store and had their local copies
cleaned up. attach doesn't need to know or care whether that's happened: if
label (and/or deidentify) was run with --unlink-source keep-metadata
rather than all, the accession and acquisition directories are still there — just without
the (now redundant) image data — since --unlink-source keep-metadata only removes
each resource's files, never the accession's or acquisition's own __METADATA__.json
(see Cleaning up previous stages).
attach loads an accession the same way either way. An acquisition with no resources at all
is an ordinary, valid state (it's what a freshly-collated acquisition looks like before
anything's been added to it), so a metadata-only skeleton reloads and matches files
against it exactly as it would for an accession whose data is still fully present. The
GLOB template is filled in from the accession's own metadata (preserved regardless),
and ID_PATTERN only ever looks at the new file's path — neither depends on the
primary resource data actually being there.
In short: run label/deidentify with --unlink-source keep-metadata instead
of all if you expect attach to need to attach files sometime after the bulk
of an accession's data has already moved on, and you don't want to keep the full image
data around indefinitely just in case.