Skip to main content
B
Seasoned ⭐️⭐️
Brendan Buckbee
Seasoned ⭐️⭐️
April 5, 2023
Solved

Protobuf Warning

  • April 5, 2023
  • 4 replies
  • 128 views

 

Just going to post the full warning. Is this something that is wrong with my installation (I updated the SDK a month or so ago)? Or something that is on the SDK side of a to be implemented feature? I’m curious because it says that this is causing the data fetching to run much slower than it can.

 

~\Anaconda3\lib\site-packages\cognite\client\_api\datapoints.py:755: UserWarning: Your installation of 'protobuf' is missing compiled C binaries, and will run in pure-python mode, which causes datapoints fetching to be ~5x slower. To verify, set the environment variable `PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=cpp` before running (this will cause the code to fail). The easiest fix is probably to pin your 'protobuf' dependency to major version 4 (or higher), see: https://developers.google.com/protocol-buffers/docs/news/2022-05-06#python-updates

    Best answer by Anita Hæhre

    Great @Brendan Buckbee seems it resolved so I’ll go ahead and close this thread for now. Feel free to open it again should you have more questions related to the same. 

    4 replies

    H
    Practitioner ⭐️⭐️⭐️
    Håkon V. Treider
    Practitioner ⭐️⭐️⭐️
    April 6, 2023

    Hi Brendan,

    A little bit of background first: starting with v5 of the SDK, datapoints fetching was changed from JSON to protobuf (initially with version >=4 ) as this is by far the heaviest and most used operation.

    For compatibility with important packages like TensorFlow and Streamlit (to name a few) which unfortunately still depend on an old version of protobuf (^3.x i.e. <4), we (or rather I) went to great lengths to support both major versions >=3. One of the issues with v3 is that it lacks the compiled C binaries for certain platforms like M1 macs which I’m guessing you are on.

    To solve this, verify that your project does not have a strict requirement on v3 of protobuf and simply upgrade the version to v4 or higher.

    poetry add "protobuf>=4"
    # or
    pip install "protobuf>=4"

    In case you must stay on v3, feel free to silence the warning by doing: (it is still plenty fast for small-to-medium sized jobs, i.e. less than a few tens of millions of datapoints)

    import warnings

    warnings.filterwarnings(
        "ignore",
        message="Your installation of 'protobuf' is missing compiled C binaries",
        category=UserWarning,
    )

     

      Anita Hæhre
      Expert ⭐️⭐️⭐️⭐️
      Anita Hæhre
      Head of Community
      April 13, 2023

      Hi @Brendan Buckbee did @Håkon V. Treiders reply give you what you were looking for? 

       

      Anita Hæhre
      B
      Seasoned ⭐️⭐️
      Brendan BuckbeeAuthor
      Seasoned ⭐️⭐️
      April 13, 2023

      Yes, I installed a protobuf version >4 and am no longer receiving the errors. I’m not sure why I ended up with <4 to begin with as I’m not on an M1 mac like Hakon suggested.

      Anita Hæhre
      Expert ⭐️⭐️⭐️⭐️
      Anita HæhreAnswer
      Head of Community
      April 13, 2023

      Great @Brendan Buckbee seems it resolved so I’ll go ahead and close this thread for now. Feel free to open it again should you have more questions related to the same. 

      Anita Hæhre
      Terms & ConditionsAccessibility statement