Raw Dataset
===========

*BLISS* will register dataset with ICAT automatically. If there are datasets
that are not confirmed to be received by ICAT when switching to a different
investigation, *BLISS* will save the data metadata in a folder called ``__icat__``.

To register those datasets

.. code-block:: bash

    icat-store-from-file /data/visitor/.../RAW_DATA/__icat__/*.xml

ICAT Registration
-----------------

In case you need to register a directory which is neither in ICAT nor in ``__icat__``

.. code-block:: bash

    icat-store-raw --beamline id00 \
        --proposal id002207 \
        --path /data/visitor/.../RAW_DATA/collection/collection_dataset1 \
        --dataset dataset1 \
        --sample mysample \
        -p FIELD1=value1 \
        -p FIELD2=value2

The equivalent in *python* (metadata is optional)

.. code-block:: python

    from pyicat_plus.client.main import IcatClient
    from pyicat_plus.client import defaults

    client = IcatClient(metadata_urls=args.metadata_urls)

    metadata = {"FIELD1": "value1", "FIELD2": "value2"}

    client.store_dataset(
        beamline="id00",
        proposal="id002207",
        dataset=,
        path="/data/visitor/.../RAW_DATA/collection/collection_dataset1",
        metadata=metadata,
    )
    client.disconnect()


ICAT Synchronization
--------------------

.. warning:: This requires expert knowledge and is normally done by ICAT admins.

The raw data on disk of one of more investigations can be synchronized with ICAT as follows

.. code-block:: bash

    icat-sync-raw --beamline id27 --proposal blc14904 --session 20230829 \
                  --save-dir /tmp/icat/summary --cache-dir /tmp/icat/cache \
                  --format esrfv3 --register --invalidate-cache

* *--save-dir*: generate CSV files and bash scripts to resolves issues later
* *--cache-dir*: store session information in JSON files
* *--register*: ask for ICAT registration when a session is not properly registered with ICAT and removes the session from the cache when answering "yes"
* *--auto-register*: same as `--register` but then without prompting for validation and only for datasets that are safe to register unsupervised

To update the cache periodically, this command is appropriate

.. code-block:: bash

    icat-sync-raw --save-dir /tmp/icat/summary --cache-dir /tmp/icat/cache \
                  --invalidate-cache --no-print

* *--invalidate-cache*: remove session from the cache that no longer exist on disk or have changed
* *--no-print*: do not print a summary for each session