Impact 2024: The Industrial Data and AI Conference for and by Users | Nominate Speakers Now for a Ch...
Thanks for the reply @Håkon Trømborg . I got a dependency error when I ran this command, but I got a dependency error. My Scala version is 2.13.8. I can install Scala 2.12, but where can I get a list of all com.cognite.spark.datasource releases?
Yes, it does not find the dependency. :: com.cognite.spark.datasource#cdf-spark-datasource_2.13;2.0.10: not found is the top error message.
Does this help? File "/opt/homebrew/Cellar/apache-spark/3.2.1/libexec/python/pyspark/java_gateway.py", line 108, in launch_gateway raise RuntimeError("Java gateway process exited before sending its port number")RuntimeError: Java gateway process exited before sending its port numberI can otherwise use Python3, Scala, and Java on my Mac (M1)
Some progress here, but still an error. It finds the package now, but new complaints. First some warnings which are suspicious WARNING: An illegal reflective access operation has occurredWARNING: Illegal reflective access by org.apache.spark.unsafe.Platform (file:/opt/homebrew/Cellar/apache-spark/3.2.1/libexec/jars/spark-unsafe_2.12-3.2.1.jar) to constructor java.nio.DirectByteBuffer(long,int)WARNING: Please consider reporting this to the maintainers of org.apache.spark.unsafe.PlatformWARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operationsWARNING: All illegal access operations will be denied in a future release and then this error :: loading settings :: url = jar:file:/opt/homebrew/Cellar/apache-spark/3.2.1/libexec/jars/ivy-2.5.0.jar!/org/apache/ivy/core/settings/ivysettings.xmlException in thread "main" java.lang.IllegalArgumentException: requirement failed: Provided Maven Coordinates must be in the form 'groupId:artifactId:version'. The
Some more progress :) I get initially no error if I start PySpark with this commandpyspark --packages com.cognite.spark.datasource:cdf-spark-datasource-fat_2.13:2.0.10and no, I have 2.13.8>> scala --version>> Scala code runner version 2.13.8 -- Copyright 2002-2021, LAMP/EPFL and Lightbend, IncI appreciate the support so far @Håkon Trømborg But I still have problems loading the data. I am trying to load some datapoints with this commandspark.read.format("cognite.spark.v1").option("type", "datapoints").option("apiKey", MY_API_KEY).option("project", MY_PROJECT).load() where I originally have MY_API_KEY and MY_PROJECT as clear text, to avoid any bugs. is this command correct? The same command works in Azure Databricks notebook though, with the same values for apiKey and projectThe error message is very long but here is a summaryTraceback (most recent call last): File "<stdin>", line 1, in <module> File "/opt/homebrew/Cellar/apache-spark/3.2.1/libexec/python/pyspar
Ok, it seemed like I needed to start PySpark with this packagepyspark --packages com.cognite.spark.datasource:cdf-spark-datasource-fat_2.12:2.0.10My system Scala installation is 2.13 but apache-spark has its own installed using HomeBrew (well, that’s what Brew does I guess). After this everything else works and I can download datapoints :) Thanks again @Håkon Trømborg
Already have an account? Login
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.
Sorry, we're still checking this file's contents to make sure it's safe to download. Please try again in a few minutes.
Sorry, our virus scanner detected that this file isn't safe to download.