.. _file-formats:
Adding support for new file types
=================================
*AccessionFlow* doesn't have built-in knowledge of DICOM, NIfTI, or any other specific
format baked into its core logic. Instead, every file on disk is represented as a
typed ``FileSet`` from the `FileFormats
`_ package (e.g. ``DicomSeries``,
``NiftiGz``), and format-specific behaviour is provided by that type rather than by
``accessionflow`` itself. This is what lets the same code handle a wildly
different data — clinical DICOMs, derived NIfTIs, proprietary raw PET data — without
a format-specific branch for each one.
Collating files into resources
----------------------------------
``collate folder``'s ``--datatype`` option (see :doc:`/sources/folder`) is a `FileFormats MIME-like
identifier `_ (or a ``|``-separated union of several) that says which types of file to
look for in the input paths at all. Within a matched accession, ``--acquisition``/
``--resource`` (:class:`~accessionflow.helpers.arg_types.IDSpec`) then decide which
acquisition and resource each file belongs to, based on values read out of the file's own
metadata (e.g. DICOM ``SeriesNumber``, ``ImageType``) — see
:class:`~accessionflow.model.resource.Resource` and
:class:`~accessionflow.model.acquisition.Acquisition`.
Reading metadata
---------------------
Metadata is read via FileFormats' ``read_metadata`` "extra" — a method declared with
``@extra`` on ``FileSet`` itself (so it applies to every format), with the actual
implementation registered separately, per format, via ``@extra_implementation``. This
indirection is what lets ``accessionflow`` call ``fileset.metadata`` (or
``fileset.read_metadata()``) generically, regardless of what the underlying format
actually is, and is why adding support for a new file type is a matter of writing an
``extra_implementation`` for it in a `FileFormats extras package
`_ (e.g.
``fileformats-medimage-extras``), rather than modifying ``accessionflow`` itself.
Deidentifying via extra implementations
--------------------------------------------
The ``deidentify`` command works the same way: ``MedicalImagingData.deidentify`` is
declared as an ``@extra`` (in ``fileformats-medimage``), and the concrete
implementation for each format is registered with ``@extra_implementation`` in an
extras package (e.g. ``dicom_deidentify`` in ``fileformats-medimage-extras``).
``Accession.deidentify`` (see :class:`~accessionflow.model.accession.Accession`)
only decides *which* recipe file applies to a given resource — the actual
deidentification logic is entirely delegated to whatever's registered for that
resource's type.
Typing the ``recipe`` argument
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
*AccessionFlow* never parses recipe files itself. It reads the annotation of the
``recipe`` argument of the registered implementation to work out which format(s) the
recipe file should be loaded as, loads it, and passes the loaded object in (see
:func:`~accessionflow.model.accession.recipe_formats_for`). That annotation is therefore
part of the implementation's runtime behaviour, not just a hint for type checkers,
and must be one of:
* ``Loaded[]`` — the recipe file is loaded with
``(path).load()``, e.g. ``Loaded[DeidRecipe]``
* a union of ``Loaded[...]`` hints, in order of preference — the first format the
recipe file matches is used. For DICOM this is
``Loaded[DeidRecipeX] | Loaded[DeidRecipe]``, so a recipe with its
``.transforms.py``/``.salt`` side-cars present is loaded together with them, and a
bare recipe is loaded on its own
* ``None`` — the implementation doesn't take a recipe. A recipe file found in the
spec directory for such a format is treated as an error rather than silently
ignored
``| None`` can be appended to the first two (with a default of ``None``) to match the
base signature. For example:
.. code-block:: python
@extra_implementation(MedicalImagingData.deidentify)
def dicom_deidentify(
dicom: DicomImage,
out_dir: os.PathLike[str],
recipe: Loaded[DeidRecipeX] | Loaded[DeidRecipe] | None = None,
**kwargs: ty.Any,
) -> DicomImage:
...
If the argument is left unannotated, or annotated with a plain type (e.g.
``recipe: DeidRecipe`` or ``recipe: ty.Any``), *AccessionFlow* can't tell what to load
the file as, so ``deidentify`` fails with a ``TypeError`` for every accession with that
format rather than guessing. Things to watch out for:
* Annotate with the *file format* class (a ``FileSet`` subclass) wrapped in
``Loaded[...]``, not the in-memory type it loads into. ``Loaded[X]`` resolves to
``Annotated[X.loaded_type, LoadedMarker(X)]`` at runtime, which is what carries the
format through to *AccessionFlow*. Static type checkers just see ``Any``, so mypy
won't catch a wrong annotation.
* String annotations (e.g. from ``from __future__ import annotations``) are fine, but
only the ``recipe`` annotation is evaluated, in the implementation module's
globals. The recipe format classes therefore have to be imported at module level,
not only under ``if TYPE_CHECKING:``.
* The order of a union matters: put the most specific format (e.g. the one requiring
side-cars) first. If a broader format comes first, it matches the recipe file
before the specific one is tried, so the side-cars are never loaded.
* Where the implementation relies on something only the loaded type provides, check
it at the top of the function with ``check_loaded(, recipe)``
(as ``dicom_deidentify`` does), so a recipe passed in by other callers fails with a
clear error.
Only formats flagged ``contains_phi = True`` (the ``MedicalImagingData`` default) are
run through ``deidentify`` at all — formats known not to carry patient information
(e.g. derived NIfTIs) set ``contains_phi = False`` and are just copied through
unchanged.