Deidentify¶
For sites where the label-ed accessions still contain identifiable data (e.g. a
clinical scanner staging DICOMs before upload), insert a deidentify step between
label and store. It strips patient-identifying fields from each accession and
writes the deidentified copy, and — only if --reid-dir is given — a
re-identification mapping so the process can be reversed later if needed (e.g. to look
up a scan for a clinical follow-up).
$ accessionflow deidentify /data/staging/labelled /data/staging/deidentified \
/etc/accessionflow/deid-recipes --reid-dir /data/staging/reid
INPUT_DIR— thelabeloutput directory (ACFLOW_INPUT_DIR)OUTPUT_DIR— where deidentified accessions are written; pointstoreat this directory instead of thelabeloutput (ACFLOW_OUTPUT_DIR)RECIPE_DIR— the deidentification recipes, one directory per project (see below) (ACFLOW_RECIPE_DIR, orACFLOW_SPEC_DIRas it was previously named)--reid-dir— optional; where the re-identification mappings are written, one JSON file per accession (ACFLOW_REID_DIR). Omit it to discard the mapping rather than persist it, which is the safer choice when holding an original↔deidentified lookup on disk is itself a concern.
--unlink-source all/--unlink-source keep-metadata clean up the label output
once an accession has been deidentified (see Cleaning up previous stages).
The metadata side-cars in the output keep only the accession's UID (__uid__) and
the fields that identify its labels and store; every other metadata field is dropped,
while the input's metadata is unchanged. To keep other fields, e.g. for
Annotate to read, list them with --metadata-allowlist. The accession's
fields (__FIELDS__.json) are carried through to the output unchanged: they are not
de-identified, so annotate should only extract fields that can't identify the
subject (see Annotate). The UID is kept so that the data store can recognise an accession
that is deposited again; to replace it with a keyed hash, see
Session identity on reuse.
Every input accession with resources is processed, even when its output already exists. A late series is saved into the existing output accession. Earlier acquisitions remain there. A format's deidentification method can change bytes, remove files, or remove resources. The stage does not compare input and output file counts or use a format list to decide whether a run is complete. The method must raise an exception if it cannot complete its work.
Accession.load_for_processing records the input file paths before loading
the accession. After processing, saving the output and writing any requested
re-identification mapping succeed, --unlink-source all removes only those
recorded files, then empty directories. --unlink-source keep-metadata removes
only recorded files inside the loaded resource directories. It keeps accession and
acquisition metadata. A new path that arrives during the run stays for the next cycle,
including a file that arrives during cleanup. A processing or save failure keeps
the input. Without --unlink-source, the input stays and is processed on each
cycle. Metadata-only skeletons from keep-metadata are skipped.
New accessions are built outside the visible accession directory and published by
rename. When Accession.save adds or replaces a resource in an existing
accession, it also builds the resource, manifest and metadata under __build__
on the destination filesystem before publishing the complete resource by rename.
This prevents a reader from seeing a partly copied file. A delay alone would not
protect against a writer that pauses mid-copy. An unchanged resource is retained
without changing its file times.
This protection is per resource. It is not a transaction for the whole accession. Replacing an existing resource has a brief interval when its directory is absent; a reader can fail and retry or see only the other resources. A reader can also see a mix of complete resources from different saves. The input path snapshot is not a writer lock: writers must not replace a recorded path during processing. Files written by external programs must also be published only after they are complete. Do not run two deidentify workers on the same input directory.
The file format's contains_phi flag selects deidentification. DICOM uses this
path by default. Other formats can provide their own deidentification method;
formats without that flag are copied. If the method requires a recipe and there
is no applicable recipe (see below), the accession fails and its input stays.
Laying out the deidentification recipes¶
RECIPE_DIR holds one subdirectory per project, named to match the accession's first
label (e.g. the XNAT project ID that label xnat gave it), plus an optional
__default__ subdirectory used as a fallback for any project without its own:
deid-recipes/
__default__/
medimage/
dicom-series.json
dicom-series.transforms.yaml
MYPROJECT/
medimage/
dicom-series.json
Within each project directory, there's one recipe file per format, at the path given by
the format's MIME-like identifier (e.g. medimage/dicom-series is
medimage/dicom-series.json), alongside any side-car files the recipe format uses
(e.g. dicom-series.transforms.yaml). A recipe also covers
more specific sub-formats — e.g. a recipe for the broader medimage/dicom-collection
format applies to medimage/dicom-series accessions too if there's no more specific
match.
Note
A recipe's contents decide how the format is deidentified for that project: for
DICOM, a deid recipe saying which tags to
remove, blank or replace, with the values of its var: references declared in
the <format>.transforms.yaml side-car. accessionflow ships an example in
accessionflow/assets/deid_recipes/__default__, which can be copied as a
starting point. Check any recipe with your data's governance requirements before
relying on it.
Re-identifying data later¶
If --reid-dir is given, each accession's original identifying values (before they
were stripped) are written there as <labels>.json, named after the accession's
directory. If --reid-encrypt-key
is also set to a URL-safe base64-encoded 32-byte key (e.g. from
Fernet.generate_key() in the cryptography package), the file is written
encrypted instead, as <labels>.json.enc, and can only be read back with that
same key — keep it somewhere separate from the --reid-dir directory itself.
Without --reid-dir the mapping is never written and the deidentification cannot be
reversed (--reid-encrypt-key then has no effect and is warned about).