Impact 2024: The Industrial Data and AI Conference for and by Users | Nominate Speakers Now for a Ch...
Ok, it seemed like I needed to start PySpark with this packagepyspark --packages com.cognite.spark.datasource:cdf-spark-datasource-fat_2.12:2.0.10My system Scala installation is 2.13 but apache-spark has its own installed using HomeBrew (well, that’s what Brew does I guess). After this everything else works and I can download datapoints :) Thanks again @Håkon Trømborg
Some more progress :) I get initially no error if I start PySpark with this commandpyspark --packages com.cognite.spark.datasource:cdf-spark-datasource-fat_2.13:2.0.10and no, I have 2.13.8>> scala --version>> Scala code runner version 2.13.8 -- Copyright 2002-2021, LAMP/EPFL and Lightbend, IncI appreciate the support so far @Håkon Trømborg But I still have problems loading the data. I am trying to load some datapoints with this commandspark.read.format("cognite.spark.v1").option("type", "datapoints").option("apiKey", MY_API_KEY).option("project", MY_PROJECT).load() where I originally have MY_API_KEY and MY_PROJECT as clear text, to avoid any bugs. is this command correct? The same command works in Azure Databricks notebook though, with the same values for apiKey and projectThe error message is very long but here is a summaryTraceback (most recent call last): File "<stdin>", line 1, in <module> File "/opt/homebrew/Cellar/apache-spark/3.2.1/libexec/python/pyspar
Some progress here, but still an error. It finds the package now, but new complaints. First some warnings which are suspicious WARNING: An illegal reflective access operation has occurredWARNING: Illegal reflective access by org.apache.spark.unsafe.Platform (file:/opt/homebrew/Cellar/apache-spark/3.2.1/libexec/jars/spark-unsafe_2.12-3.2.1.jar) to constructor java.nio.DirectByteBuffer(long,int)WARNING: Please consider reporting this to the maintainers of org.apache.spark.unsafe.PlatformWARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operationsWARNING: All illegal access operations will be denied in a future release and then this error :: loading settings :: url = jar:file:/opt/homebrew/Cellar/apache-spark/3.2.1/libexec/jars/ivy-2.5.0.jar!/org/apache/ivy/core/settings/ivysettings.xmlException in thread "main" java.lang.IllegalArgumentException: requirement failed: Provided Maven Coordinates must be in the form 'groupId:artifactId:version'. The
Does this help? File "/opt/homebrew/Cellar/apache-spark/3.2.1/libexec/python/pyspark/java_gateway.py", line 108, in launch_gateway raise RuntimeError("Java gateway process exited before sending its port number")RuntimeError: Java gateway process exited before sending its port numberI can otherwise use Python3, Scala, and Java on my Mac (M1)
Yes, it does not find the dependency. :: com.cognite.spark.datasource#cdf-spark-datasource_2.13;2.0.10: not found is the top error message.
Thanks for the reply @Håkon Trømborg . I got a dependency error when I ran this command, but I got a dependency error. My Scala version is 2.13.8. I can install Scala 2.12, but where can I get a list of all com.cognite.spark.datasource releases?
Already have an account? Login
Enter your username or e-mail address. We'll send you an e-mail with instructions to reset your password.
Sorry, we're still checking this file's contents to make sure it's safe to download. Please try again in a few minutes.
Sorry, our virus scanner detected that this file isn't safe to download.