Skip to main content
Solved

CDF Files and cloud optimized storage


thomafred
Seasoned
Forum|alt.badge.img

We are looking into cloud-optimized storage formats such as GeoTIFF, Parquet, etc. One of the things we are trying to determine is whether we could fully utilize such cloud optimized formats with CDF Files.

Will the download-link returned by CDF Files allow us to do seek-operations and only download parts of the file?

Best answer by erlend.vollset

The Range header is part of HTTP, as specified by RFC7233 (https://datatracker.ietf.org/doc/html/rfc7233). Most blob stores will return an `Accept-Ranges` response header indicating support for Range requests. I’m not familiar with Pangeo, but if it conforms to this standard it should work fine.

View original
Did this topic help you find an answer to your question?

3 replies

  • Senior Software Engineer
  • 6 replies
  • August 13, 2021

Hi Thomas,

Thanks for reaching out!

The download endpoint supports supplying the Range header to specify which byte range to download, e.g.

"Range": "bytes:1000-2000"

Please not that both start and end are inclusive.

Does that solve your problem?


thomafred
Seasoned
Forum|alt.badge.img
  • Author
  • Seasoned
  • 18 replies
  • August 13, 2021

Allowing for a range-argument is a nice feature.

 

Is this part of a propriatary protocol, or can I (in python) use something like gcsfs as described here:

 

https://pangeo.io/data.html


  • Senior Software Engineer
  • 6 replies
  • Answer
  • August 13, 2021

The Range header is part of HTTP, as specified by RFC7233 (https://datatracker.ietf.org/doc/html/rfc7233). Most blob stores will return an `Accept-Ranges` response header indicating support for Range requests. I’m not familiar with Pangeo, but if it conforms to this standard it should work fine.


Reply


Cookie Policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie Settings