Upgrade Hadoop version in order to support Cloud Storage connector

Description

The customer wants to use Google Cloud Storage connector. See here: https://cloud.google.com/dataproc/docs/concepts/connectors/cloud-storage
But got:
java.lang.NoSuchMethodError: org.apache.hadoop.conf.Configuration.getPropsWithPrefix(Ljava/lang/StringLjava/util/Map;
at com.google.cloud.hadoop.repackaged.gcs.com.google.cloud.hadoop.util.HadoopConfigurationProperty.lambda$getPropsWithPrefix$3(HadoopConfigurationProperty.java:106)
at com.google.cloud.hadoop.repackaged.gcs.com.google.cloud.hadoop.util.HadoopConfigurationProperty.getLookupKey(HadoopConfigurationProperty.java:120)
at com.google.cloud.hadoop.repackaged.gcs.com.google.cloud.hadoop.util.HadoopConfigurationProperty.getPropsWithPrefix(HadoopConfigurationProperty.java:106)
at com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemConfiguration.getGcsOptionsBuilder(GoogleHadoopFileSystemConfiguration.java:417)
at com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemConfiguration.getGcsFsOptionsBuilder(GoogleHadoopFileSystemConfiguration.java:379)
at com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase.createGcsFs(GoogleHadoopFileSystemBase.java:1516)
at com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase.configure(GoogleHadoopFileSystemBase.java:1486)
at com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase.initialize(GoogleHadoopFileSystemBase.java:541)
at com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase.initialize(GoogleHadoopFileSystemBase.java:494)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2669)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2703)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2685)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:373)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295)
... 49 elided

According to this page: https://github.com/GoogleCloudDataproc/hadoop-connectors/issues/323
This feature is supported since Hadoop 2.8.3
We are using Hadoop 2.7.3 See:
{IE_HOME}/insightedge/spark/jars/
{IE_HOME}/insightedge/zeppelin/lib/
{IE_HOME}/spark/python/pyspark.egg-info/SOURCES.txt

Acceptance Test

None

Assignee

Unassigned

Reporter

Yuval Dori

Labels

None

Priority

Critical

SalesForce Case ID

12942

Fix versions

None

Commitment Version/s

None

Due date

None

Product

InsightEdge

Edition

Enterprise

Platform

All
Configure