Skip to main content
Question

CDF Spark datasource is broken from version


 

Hello, 

I’m having this error while using CDF latest Spark datasource. I was only able to make it work using 2.1.10 version (com.cognite.spark.datasource:cdf-spark-datasource_2.12:2.1.10). 

 

Could you please take a look? 

 

Thank you !

 

Mithila Jayalath
Seasoned Practitioner
Forum|alt.badge.img

@Marwen TALEB I’ll look into this and get back to you with an update.


Mithila Jayalath
Seasoned Practitioner
Forum|alt.badge.img

@JaimeSilva will you be able to help out here?


Forum|alt.badge.img

Hi ​@Marwen TALEB ! Sorry for the delay, could you please provide us with some more details on how are you using the datasource? I’m assuming this is pyspark on a databricks notebook, but would like to confirm.

If that’s the case, could you share a small snippet that we can use to attempt and reproduce the issue? In concrete, I would like to know how is `assets` defined, that could be very useful to see if there was an undocumented breaking change or something of sorts.


Sure

assets = spark.read.format("cognite.spark.v1") \
    .option("tokenUri", "https://login.microsoftonline.com/xx/oauth2/v2.0/token") \
    .option("clientId", "xxx") \
    .option("clientSecret", "xxx") \
    .option("project", "xxx") \
    .option("scopes", "xx") \
    .option("baseUrl", "xx") \
    .option("type", "assets") \
    .load()  

 


Forum|alt.badge.img

do you know which version of spark is your cluster running?


Forum|alt.badge.img

Also, if you can get the continuation of the error stacktrace, that would be amazing 

I mean the rest of the exception, after

(promise.scala:87)...

 


Reply


Cookie Policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie Settings