Compatibility matrix for Elasticsearch and Spark is missing

[sergeykad]

Feature description

Right now it requires a significant effort to understand what versions combinations of Elasticksearch and Spark are supported and what version of 'elasticsearch-hadoop' (or 'elasticsearch-spark' ?) is needed. A compatibility matrix with clear supported versions ranges will be really helpful.

There is a partial list here, but it does not list supported Spark versions at all.
This is also related to #1346

[masseyke]

Hi @sergeykad I agree with you that we need this. And adding scala version to the combinations makes it even more complicated. I started putting that matrix together a few months ago, but haven't published it yet. For the moment there is this (which is not what you are looking for, but is useful) -- https://www.elastic.co/guide/en/elasticsearch/hadoop/current/install.html. Until we get the matrix published, we can answer any specific questions you have at https://discuss.elastic.co/.

[sergeykad]

Thanks for the reply. I just want to add that although documentation at https://www.elastic.co/guide/en/elasticsearch/hadoop/current/install.html gives some general guidance it is not accurate.

For example, it specifies that elasticsearch-spark-30_2.12 compatible with Spark 3.0+. In practice, the earlier elasticsearch-spark-30_2.12 versions depend on Spark 3.0.1 while the latter versions use Spark 3.2.0. These two Spark versions are not compatible and using the incorrect elasticsearch-spark-30_2.12 version causes strange issues due to two different spark-core versions available in the classpath.

Also, from what I have seen, there is no version that supports Spark 3.1.x

[ekhc]

Oh no version supports Spark 3.1.x?

[masseyke]

We support spark 3.1 with scala 2.12 with elasticsearch-spark-30_2.12 -- https://discuss.elastic.co/t/issue-using-the-connector-from-pyspark-in-7-17-3/304084/3.

These two Spark versions are not compatible and using the incorrect elasticsearch-spark-30_2.12 version causes strange issues due to two different spark-core versions available in the classpath.

Do you have an example of something that causes this? I don't think we'd seen this problem.

[sergeykad]

I do not remember the exact errors, but I think we can agree that having two different versions of the same jar in classpath is never a good thing. In this case, it is spark-core.jar that is duplicated. One version is part of Spark itself and another comes as a dependency of elasticsearch-spark-30_2.12.jar

[jbaiera]

@sergeykad are you by chance running into effects caused by #1541?

[sergeykad]

@jbaiera Yes, it looks like the problem described in #1541, but I used a relatively new jar version - "org.elasticsearch:elasticsearch-spark-30_2.12:7.16.2" and newer.
According to the MVNRepository sparc-core is a compile dependency and not provided. This was also my experience.

I can manually exclude the transitive dependency on the sparc-core, but #1541 should have prevented exactly this issue. In addition, if I go this route, how far I can deviate from the version that elasticsearch-spark was compiled with?

[vipulgabhane]

i am able to read the data but when i tried with df.show() getting some error . unable to understand the error

.option("es.resource", "test_index_name")
... .option("es.nodes.wan.only", "true")
... .load()
2024-01-16 12:05:49,133 WARN rest.RestClient: Could not verify server is Elasticsearch! ES-Hadoop will require server validation when connecting to an Elasticsearch cluster if that Elasticsearch cluster is v7.14 and up.

df
DataFrame[coordinates: bigint, field1: string, field10: bigint, field11: bigint, field12: float, field2: string, field3: string, field4: bigint, field5: bigint, field6: float, field7: boolean, field8: timestamp, field9: string, fieldtime: timestamp]
df.show()
2024-01-16 12:05:58,280 WARN rest.RestClient: Could not verify server is Elasticsearch! ES-Hadoop will require server validation when connecting to an Elasticsearch cluster if that Elasticsearch cluster is v7.14 and up.
2024-01-16 12:06:07,018 WARN scheduler.TaskSetManager: Lost task 0.0 in stage 1.0 (TID 4) (ip-172-31-12-49.us-west-2.compute.internal executor 2): java.lang.RuntimeException: Error while encoding: java.lang.RuntimeException: scala.collection.convert.Wrappers$JListWrapper is not a valid external type for schema of bigint
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 0, coordinates), LongType) AS coordinates#169L
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 1, field1), StringType), true, false) AS field1#170

[vipulgabhane]

i am using this connector org.elasticsearch:elasticsearch-spark-30_2.12:7.17.3

[jbaiera]

@vipulgabhane this does not seem to be related to the current issue. Please avoid posting unrelated questions on existing issues.

This problem seems to be some kind of issue with your spark schema. Please make sure your Elasticsearch data is structured in a homogenous manner. If you are working with geospatial data all documents should have the same format.

There's an active community in the forums that may be able to help get an answer to your question. If you are experiencing a bug with the connector instead of a configuration problem please open a new issue.

[bytebilly]

@jbaiera @dakrone is there anything we can do here to improve our documentation?