metadata de-serialization fails

[Flexi23]

The latest release version (5.2.3) of the Python API does not properly load project's metadata.

This seems directly related to an issue that I ran into a month ago: https://github.com/elabftw/elabftw/issues/5721 (The openapi specification of the metadata field type is inconsistent with the actually provided string value.)

In C# code, I got some hard exceptions and experiments could not get loaded at all, but the Python code just goes over the error and returns None values in the metadata properties where the API actually provides the expected value as string but not as schema-specified objects.

[HannahJuerss]

To be more precise: I have an experiment in eLabFTW with the extra field Temperature. The Python function read_experiments returns the metadata field correctly in version 5.1.0: 'metadata': '{"extra_fields": {"Temperature": {"type": "text", "value": ""}}}' If I run the same code using elabapi_python version 5.2.3, it retuns: 'metadata': {'elabftw': None, 'extra_fields': None}

[nap84]

I can report the same problem.

(using requests to directly interact with the API works)

[buschalx]

I stumbled upon the same issue. I noticed it when using the read_experiments_templates function. I'll probably use the requests library to interact with the API instead, since the problem doesn't occur there.

[mhxion]

The Python function read_experiments returns the metadata field correctly in version 5.1.0

I could be wrong, but it seems metadata is one of those fields that is expected to be read from read_experiments(id="<experiment ID>") instead of read_experiments. See: https://github.com/elabftw/elabapi-python/issues/37#issuecomment-3094794488

elAPI works fine similar to requests too.

from elapi.api import GETRequest

experiments_session = GETRequest()
experiment_response = experiments_session()  # id = None gives all experiment list by deault
# experiment_response is a similar object to request library's response
all_expriments_data = experiment_response.json() 
for experiment in all_expriments_data:
    print(experiment.get("metadata"))

[MoustaphaCamara]

but the Python code just goes over the error and returns None values in the metadata properties where the API actually provides the expected value as string but not as schema-specified objects.

If I run the same code using elabapi_python version 5.2.3, it retuns: 'metadata': {'elabftw': None, 'extra_fields': None}

When using elabapi_python version 5.2.3, calling experimentsApi.read_experiments() returns a metadata field like:

'metadata': {'elabftw': None, 'extra_fields': None}

However, as you mentioned, the API response does include values for these fields in JSON string format. The issue seems to be with the client deserialization logic, which silently fails to parse the values (likely due to a schema mismatch) and substitutes None.

To bypass this deserialization problem, I’m using the _preload_content=False option in the API call. This prevents the client from trying to automatically decode the response:

response = experimentsApi.read_experiments(_preload_content=False, limit=2)

I then manually decode the raw HTTP response:

import json
experiments = json.loads(response.data.decode("utf-8"))

Here's a full code sample to try out:

# need api client configuration though, https://github.com/elabftw/elabapi-python?tab=readme-ov-file#basic-concepts
import elabapi_python
import json
import sys

experimentsApi = elabapi_python.ExperimentsApi(api_client)

response = experimentsApi.read_experiments(_preload_content=False, limit=2)
experiments = json.loads(response.data.decode("utf-8"))

for exp in experiments:
    metadata_raw = exp.get("metadata")
    print(metadata_raw)
    sys.exit()

[anotherska]

I also struggled with this and realized that there is already a decoded metadata included in the response data which contains the same information as metadata but decoded as a dictionary. Adding metadata_decoded to the openapi.yaml using the same schema

...
metadata:
          $ref: '#/components/schemas/metadata'
metadata_decoded:
          $ref: '#/components/schemas/metadata'
...

and regenerating the API using swagger codegen worked form me. Afterwards I can directly access the metadata_decoded property.

[anotherska]

Doing further tests I found out that the decoded_metadata is not consistently included in the response data. E.g. it is included in ExperimentTemplates but not in Experiments. Also the definition of $ref: '#/components/schemas/metadata' in the openapi.yaml is only valid for dictionaries and thus metadata_decoded.

My current solution therefore is to use

...
        metadata:
          type: string
        metadata_decoded:
          $ref: '#/components/schemas/metadata'
...

Still this feels like a bad workaround and it would be better if all responses already include decoded_metadata.

[NicolasCARPi]

is not consistently included in the response data

metadata_decoded in included all the time, but only if the metadata is not empty.

[anotherska]

Thanks Nicolas for the clarification. I could not find anything about metadata_decoded in the documentation. Personally, I would prefer metadata_decoded over metadata and would even suggest to replace the current metadata by the decoded version. But I understand that this would be a breaking change which is to be avoided.

[MariaF203]

We are also experiencing the same issue with the Python API. The metadata fields are set to null. Is there any fix or update planned to address this problem?

[JohannesBaller]

I also ran into this issue and would be glad if an upstream fix could be implemented :)