Skip to content

Can't tiled .export from DatasetClient with different shaped data #879

Description

@evvaaaa

In my client I am trying to export data

In [50]: db["a638fd91-28fc-4378-9bca-9170e35563d5"]["primary"]["data"]["pfy_bruker_mca_spectrum"]
Out[50]: <ArrayClient shape=(81, 1024) chunks=((81,), (1024,)) dtype=float64 dims=('time', 'dim_0')>

In [51]: db["a638fd91-28fc-4378-9bca-9170e35563d5"]["primary"]["data"]["pfy_bruker_mca_roi0_count"]
Out[51]: <ArrayClient shape=(81,) chunks=((81,)) dtype=float64 dims=('time',)>

into a csv.

I run

In [53]: db["a638fd91-28fc-4378-9bca-9170e35563d5"]["primary"]["data"].export("meh/meh.spec")

or

In [54]: db["a638fd91-28fc-4378-9bca-9170e35563d5"]["primary"]["data"].export("meh/meh.csv")

and get internal server errors

Scheduled retry in 0.37 seconds due to HTTPStatusError("Server error '500 Internal Server Error' for url '<....>/api/v1/container/full/a638fd91-28fc-4378-9bca-9170e35563d5/primary/data?format=spec'\nFor more information, server admin can search server logs for correlation ID 6ff861b57ceeefe5.") (attempt 1)
Scheduled retry in 0.54 seconds due to HTTPStatusError("Server error '500 Internal Server Error' for url '<....>/api/v1/container/full/a638fd91-28fc-4378-9bca-9170e35563d5/primary/data?format=spec'\nFor more information, server admin can search server logs for correlation ID 02ffcfd49f7c9d3c.") (attempt 2)

of the form

  File "/home/bluesky/.venv/lib/python3.11/site-packages/fastapi/routing.py", line 430, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bluesky/.venv/lib/python3.11/site-packages/fastapi/routing.py", line 316, in run_endpoint_function
    return await dependant.call(**values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bluesky/.venv/lib/python3.11/site-packages/tiled/server/router.py", line 1077, in get_container_full
    return await container_full(
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/bluesky/.venv/lib/python3.11/site-packages/tiled/server/router.py", line 1147, in container_full
    data = await ensure_awaitable(entry.read, fields=field)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bluesky/.venv/lib/python3.11/site-packages/tiled/utils.py", line 742, in ensure_awaitable
    return await anyio.to_thread.run_sync(functools.partial(func, **kwargs), *args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bluesky/.venv/lib/python3.11/site-packages/anyio/to_thread.py", line 61, in run_sync
    return await get_async_backend().run_sync_in_worker_thread(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bluesky/.venv/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 2525, in run_sync_in_worker_thread
    return await future
           ^^^^^^^^^^^^
  File "/home/bluesky/.venv/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 986, in run
    result = context.run(func, *args)
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bluesky/.venv/lib/python3.11/site-packages/databroker/mongo_normalized.py", line 741, in read
    columns.update(self.get_columns(keys_to_fetch, slices=None))
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bluesky/.venv/lib/python3.11/site-packages/databroker/mongo_normalized.py", line 882, in get_columns
    to_stack = self._inner_get_columns(tuple(keys), min_seq_num, max_seq_num)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bluesky/.venv/lib/python3.11/site-packages/databroker/mongo_normalized.py", line 997, in _inner_get_columns
    populate_columns(tuple(scalars), min_, max_)
  File "/home/bluesky/.venv/lib/python3.11/site-packages/databroker/mongo_normalized.py", line 954, in populate_columns
    validated_column = list(
                       ^^^^^
  File "/home/bluesky/.venv/lib/python3.11/site-packages/databroker/mongo_normalized.py", line 956, in <lambda>
    lambda item: self.validate_shape(key, numpy.asarray(item), expected_shape)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bluesky/.venv/lib/python3.11/site-packages/databroker/mongo_normalized.py", line 2138, in default_validate_shape
    raise BadShapeMetadata(
databroker.mongo_normalized.BadShapeMetadata: For data key pfy_bruker_mca_roi0_count shape () does not match expected shape (1024,).

From this, I think databroker is mixing up rows with columns.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions