Skip to main content
Question

CDF Spark datasource is broken from version

  • March 19, 2025
  • 6 replies
  • 60 views

Marwen TALEB
MVP

 

Hello, 

I’m having this error while using CDF latest Spark datasource. I was only able to make it work using 2.1.10 version (com.cognite.spark.datasource:cdf-spark-datasource_2.12:2.1.10). 

 

Could you please take a look? 

 

Thank you !

 

6 replies

Mithila Jayalath
Seasoned Practitioner
Forum|alt.badge.img+8

@Marwen TALEB I’ll look into this and get back to you with an update.


Mithila Jayalath
Seasoned Practitioner
Forum|alt.badge.img+8

@JaimeSilva will you be able to help out here?


Forum|alt.badge.img
  • Practitioner
  • April 1, 2025

Hi ​@Marwen TALEB ! Sorry for the delay, could you please provide us with some more details on how are you using the datasource? I’m assuming this is pyspark on a databricks notebook, but would like to confirm.

If that’s the case, could you share a small snippet that we can use to attempt and reproduce the issue? In concrete, I would like to know how is `assets` defined, that could be very useful to see if there was an undocumented breaking change or something of sorts.


Marwen TALEB
MVP
  • Author
  • MVP
  • April 1, 2025

Sure

assets = spark.read.format("cognite.spark.v1") \
.option("tokenUri", "https://login.microsoftonline.com/xx/oauth2/v2.0/token") \
.option("clientId", "xxx") \
.option("clientSecret", "xxx") \
.option("project", "xxx") \
.option("scopes", "xx") \
.option("baseUrl", "xx") \
.option("type", "assets") \
.load()

 


Forum|alt.badge.img
  • Practitioner
  • April 1, 2025

do you know which version of spark is your cluster running?


Forum|alt.badge.img
  • Practitioner
  • April 1, 2025

Also, if you can get the continuation of the error stacktrace, that would be amazing 

I mean the rest of the exception, after

(promise.scala:87)...