Skip to main content
Answer

Protobuf Warning

  • April 5, 2023
  • 4 replies
  • 85 views
B
Committed

 

Just going to post the full warning. Is this something that is wrong with my installation (I updated the SDK a month or so ago)? Or something that is on the SDK side of a to be implemented feature? I’m curious because it says that this is causing the data fetching to run much slower than it can.

 

~\Anaconda3\lib\site-packages\cognite\client\_api\datapoints.py:755: UserWarning: Your installation of 'protobuf' is missing compiled C binaries, and will run in pure-python mode, which causes datapoints fetching to be ~5x slower. To verify, set the environment variable `PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=cpp` before running (this will cause the code to fail). The easiest fix is probably to pin your 'protobuf' dependency to major version 4 (or higher), see: https://developers.google.com/protocol-buffers/docs/news/2022-05-06#python-updates

Best answer by Anita Hæhre

Great @Brendan Buckbee seems it resolved so I’ll go ahead and close this thread for now. Feel free to open it again should you have more questions related to the same. 

Anita Hæhre

4 replies

H
Practitioner
Forum|alt.badge.img

Hi Brendan,

A little bit of background first: starting with v5 of the SDK, datapoints fetching was changed from JSON to protobuf (initially with version >=4 ) as this is by far the heaviest and most used operation.

For compatibility with important packages like TensorFlow and Streamlit (to name a few) which unfortunately still depend on an old version of protobuf (^3.x i.e. <4), we (or rather I) went to great lengths to support both major versions >=3. One of the issues with v3 is that it lacks the compiled C binaries for certain platforms like M1 macs which I’m guessing you are on.

To solve this, verify that your project does not have a strict requirement on v3 of protobuf and simply upgrade the version to v4 or higher.

poetry add "protobuf>=4"
# or
pip install "protobuf>=4"

In case you must stay on v3, feel free to silence the warning by doing: (it is still plenty fast for small-to-medium sized jobs, i.e. less than a few tens of millions of datapoints)

import warnings

warnings.filterwarnings(
    "ignore",
    message="Your installation of 'protobuf' is missing compiled C binaries",
    category=UserWarning,
)

 

K
Anita Hæhre
Seasoned Practitioner
Forum|alt.badge.img+1
  • Anita Hæhre
  • Head of Community
  • April 13, 2023

Hi @Brendan Buckbee did @Håkon V. Treiders reply give you what you were looking for? 

 

Anita Hæhre
B
Committed

Yes, I installed a protobuf version >4 and am no longer receiving the errors. I’m not sure why I ended up with <4 to begin with as I’m not on an M1 mac like Hakon suggested.

Anita Hæhre
Seasoned Practitioner
Forum|alt.badge.img+1
  • Anita Hæhre
  • Head of Community
  • Answer
  • April 13, 2023

Great @Brendan Buckbee seems it resolved so I’ll go ahead and close this thread for now. Feel free to open it again should you have more questions related to the same. 

Anita Hæhre
Terms & ConditionsCookie SettingsAccessibility statement