CDF Files and cloud optimized storage

Forum|Forum|4 years ago
August 13, 2021
3 replies
161 views

We are looking into cloud-optimized storage formats such as GeoTIFF, Parquet, etc. One of the things we are trying to determine is whether we could fully utilize such cloud optimized formats with CDF Files.

Will the download-link returned by CDF Files allow us to do seek-operations and only download parts of the file?

Best answer by erlend.vollset

The Range header is part of HTTP, as specified by RFC7233 (https://datatracker.ietf.org/doc/html/rfc7233). Most blob stores will return an `Accept-Ranges` response header indicating support for Range requests. I’m not familiar with Pangeo, but if it conforms to this standard it should work fine.

E

erlend.vollset

Senior Software Engineer

Hi Thomas,

Thanks for reaching out!

The download endpoint supports supplying the Range header to specify which byte range to download, e.g.

"Range": "bytes:1000-2000"

Please not that both start and end are inclusive.

Does that solve your problem?

thomafredAuthor

Seasoned ⭐️⭐️⭐️

Allowing for a range-argument is a nice feature.

Is this part of a propriatary protocol, or can I (in python) use something like gcsfs as described here:

https://pangeo.io/data.html

E

erlend.vollsetAnswer

Senior Software Engineer

The Range header is part of HTTP, as specified by RFC7233 (https://datatracker.ietf.org/doc/html/rfc7233). Most blob stores will return an `Accept-Ranges` response header indicating support for Range requests. I’m not familiar with Pangeo, but if it conforms to this standard it should work fine.

Sign up

Welcome to Cognite Hub