Check the
documentation
Ask the
Community
Take a look
at
Academy
Cognite
Status
Page
Contact
Cognite Support
Solved
CDF Files block writes `ReadError`
I have implemented block-updates to work with CDF Files. This by itself works quite well, as documented here.
However, for very large files there are some problems. After some time, typically around 4 minutes, one of the block updates fail, with a simple `ReadError`. There’s no response from the API.
Some logs from our application:
DEBUG:odp.odcat.storage.cdffs.async_spec:Uploaded 133120 bytes to https://westeurope-1.cognitedata.com/api/v1/files/storage/cognite/6723545981847516/7406155183547467/wod_gld_2013.nc?sig=REDACTED&rsct=application%2Foctet-stream&sp=cw&sr=b
DEBUG:odp.odcat.storage.cdffs.async_spec:Total bytes uploaded: 2660989807
DEBUG:odp.odcat.storage.cdffs.async_spec:Data: len=5376000, MD5=6a0c0db1b66dac18ee28e8b082830667
DEBUG:odp.odcat.storage.cdffs.async_spec:Buffer cursors: start=0, end=5242880, final=False
DEBUG:httpx._client:HTTP Request: PUT https://westeurope-1.cognitedata.com/api/v1/files/storage/cognite/6723545981847516/7406155183547467/wod_gld_2013.nc?sig=REDACTED&rsct=application%2Foctet-stream&sp=cw&sr=b&comp=block&blockid=gAAAAAAAA94= "HTTP/1.1 201 Created"
DEBUG:odp.odcat.storage.cdffs.async_http_client:status code returned: 201
DEBUG:odp.odcat.storage.cdffs.async_spec:Uploaded 5242880 bytes to https://westeurope-1.cognitedata.com/api/v1/files/storage/cognite/6723545981847516/7406155183547467/wod_gld_2013.nc?sig=REDACTED&rsct=application%2Foctet-stream&sp=cw&sr=b
DEBUG:odp.odcat.storage.cdffs.async_spec:Total bytes uploaded: 2666232687
DEBUG:odp.odcat.storage.cdffs.async_spec:Data: len=5376000, MD5=6a0c0db1b66dac18ee28e8b082830667
DEBUG:odp.odcat.storage.cdffs.async_spec:Buffer cursors: start=5242880, end=10485760, final=False
DEBUG:httpx._client:HTTP Request: PUT https://westeurope-1.cognitedata.com/api/v1/files/storage/cognite/6723545981847516/7406155183547467/wod_gld_2013.nc?sig=REDACTED&rsct=application%2Foctet-stream&sp=cw&sr=b&comp=block&blockid=gAAAAAAAA98= "HTTP/1.1 201 Created"
DEBUG:odp.odcat.storage.cdffs.async_http_client:status code returned: 201
DEBUG:odp.odcat.storage.cdffs.async_spec:Uploaded 133120 bytes to https://westeurope-1.cognitedata.com/api/v1/files/storage/cognite/6723545981847516/7406155183547467/wod_gld_2013.nc?sig=REDACTED&rsct=application%2Foctet-stream&sp=cw&sr=b
DEBUG:odp.odcat.storage.cdffs.async_spec:Total bytes uploaded: 2666365807
DEBUG:odp.odcat.storage.cdffs.async_spec:Data: len=4772470, MD5=474504b7e60e0163b07bdd85bb47bfe2
DEBUG:odp.odcat.storage.cdffs.async_spec:Buffer cursors: start=0, end=4772470, final=True
ERROR:odp.odcat.storage.cdffs.async_files:A transport error occurred while requesting https://westeurope-1.cognitedata.com/api/v1/files/storage/cognite/6723545981847516/7406155183547467/wod_gld_2013.nc?sig=REDACTED&rsct=application%2Foctet-stream&sp=cw&sr=b&comp=block&blockid=gAAAAAAAA+A=.
WARNING:odp.odcat.storage.cdffs.async_spec:, retrying in 1.0...
ERROR:odp.odcat.storage.cdffs.async_files:A transport error occurred while requesting https://westeurope-1.cognitedata.com/api/v1/files/storage/cognite/6723545981847516/7406155183547467/wod_gld_2013.nc?sig=REDACTED&rsct=application%2Foctet-stream&sp=cw&sr=b&comp=block&blockid=gAAAAAAAA+A=.
WARNING:odp.odcat.storage.cdffs.async_spec:, retrying in 2.0...
ERROR:odp.odcat.storage.cdffs.async_files:A transport error occurred while requesting https://westeurope-1.cognitedata.com/api/v1/files/storage/cognite/6723545981847516/7406155183547467/wod_gld_2013.nc?sig=REDACTED&rsct=application%2Foctet-stream&sp=cw&sr=b&comp=block&blockid=gAAAAAAAA+A=.
WARNING:odp.odcat.storage.cdffs.async_spec:, retrying in 4.0...
ERROR:odp.odcat.storage.cdffs.async_files:A transport error occurred while requesting https://westeurope-1.cognitedata.com/api/v1/files/storage/cognite/6723545981847516/7406155183547467/wod_gld_2013.nc?sig=REDACTED&rsct=application%2Foctet-stream&sp=cw&sr=b&comp=block&blockid=gAAAAAAAA+A=.
WARNING:odp.odcat.storage.cdffs.async_spec:, retrying in 8.0...
ERROR:odp.odcat.storage.cdffs.async_files:A transport error occurred while requesting https://westeurope-1.cognitedata.com/api/v1/files/storage/cognite/6723545981847516/7406155183547467/wod_gld_2013.nc?sig=REDACTED&rsct=application%2Foctet-stream&sp=cw&sr=b&comp=block&blockid=gAAAAAAAA+A=.
ERROR:blacksheep.server:Unhandled exception - "PATCH /data/catalog.hubocean.io/dataset/test-dataset/wod_gld_2013.nc"
Traceback (most recent call last):
File "blacksheep/baseapp.pyx", line 84, in blacksheep.baseapp.BaseApplication.handle
File "/home/tom/project/odf/BlackSheep/blacksheep/middlewares.py", line 6, in middleware_wrapper
return await handler(request, next_handler)
File "/home/tom/project/odf/app-odcat/sdk/odp/odcat/api/middlewares/prometheus_middleware.py", line 116, in __call__
raise e from None
File "/home/tom/project/odf/app-odcat/sdk/odp/odcat/api/middlewares/prometheus_middleware.py", line 113, in __call__
response = await handler(request)
File "/home/tom/project/odf/BlackSheep/blacksheep/server/normalization.py", line 459, in handler
response = await method(*values)
File "/home/tom/project/odf/app-odcat/controllers/storage-raw/odcat/storage_raw/handlers/raw_storage_handler.py", line 380, in upload_file_by_dataset_qname
return await self._upload_file(
File "/home/tom/project/odf/app-odcat/sdk/odp/odcat/utils/http_exc.py", line 31, in handler
return ensure_response(await next_handler(*args, **kwargs))
File "/home/tom/project/odf/app-odcat/controllers/storage-raw/odcat/storage_raw/handlers/raw_storage_handler.py", line 172, in _upload_file
async with file_store.open(file_info, "wb") as fd:
File "/usr/lib64/python3.10/contextlib.py", line 206, in __aexit__
await anext(self.gen)
File "/home/tom/project/odf/app-odcat/sdk/odp/odcat/storage/cdf/cdf_file_store.py", line 48, in open
async with super().open(
File "/usr/lib64/python3.10/contextlib.py", line 206, in __aexit__
await anext(self.gen)
File "/home/tom/project/odf/app-odcat/sdk/odp/odcat/storage/file_store.py", line 66, in open
async with fs.open_async(str(fname), mode, **kwargs) as fd:
File "/home/tom/.cache/pypoetry/virtualenvs/app-odcat-Dj0wURcA-py3.10/lib/python3.10/site-packages/fsspec/asyn.py", line 1020, in __aexit__
await self.close()
File "/home/tom/.cache/pypoetry/virtualenvs/app-odcat-Dj0wURcA-py3.10/lib/python3.10/site-packages/fsspec/asyn.py", line 979, in close
await self.flush(force=True)
File "/home/tom/.cache/pypoetry/virtualenvs/app-odcat-Dj0wURcA-py3.10/lib/python3.10/site-packages/fsspec/asyn.py", line 1012, in flush
if await self._upload_chunk(final=force) is not False:
File "/home/tom/project/odf/app-odcat/sdk/odp/odcat/storage/cdffs/async_spec.py", line 969, in _upload_chunk
raise RuntimeError from cognite_exp
RuntimeError
INFO: 127.0.0.1:58422 - "PATCH /data/catalog.hubocean.io/dataset/test-dataset/wod_gld_2013.nc HTTP/1.1" 500 Internal Server ErrorMy first thought was that the upload url expired, which is causing this issue. However after conferring with a colleague, I learned the upload URL should be valid for a week, so that shouldn’t be an issue.
Any idea what may be happening here?
