Customer cannot submit spark job in IE version 15.0 with specific k8s versions
GS-13982
logback files are not being created in version 14.5
GS-13923
WARNING replicating a packet which is bigger than the batch size - Who is the target?
GS-13918
Durable Notification with FIFO order cause memory leak in rare conditions
GS-13808
IllegalArgumentException: A metric named process_cpu_time-total already exists when deploy a pu.
GS-13187
increase/decrease number of partitions for MX
GS-14041
Heap Dump analyzer for space
GS-14023
Javadoc improvement
GS-14021
InsightEdge JDBC Driver code throwing exception for object not found
GS-14001
Tool for generating code from a JDBC connection
GS-13992
sparkContext.gridSql doesn't support nested objects
GS-13924
Enforcing per core XAP license for XAP users
GS-13898
After network recovery reconnected Space instance was stuck verifying last primary state
GS-13782
web-ui directories are exposed by this url: http://localhost:8099/images/
GS-13605
Static Analysis improvements
GS-13507
xap demo CLI command sometimes does not performs space deploy on busy machine
GS-13504
RocksDB Off-heap cache update optimization
GS-13435
Abridged deployment of Space with MemoryXtend
GS-13426
Native support for Scala Case Class
GS-13425
XAP_GSM_OPTIONS are not honored
GS-13298
mvn install doesn't work for 10.1.0-12630-RELEASE
GS-13272
Notifications are not received when using SSL and slow consumers
GS-13150
CLI for Grid Management
GS-13025
New MemoryXtend Metrics
GS-13024
Web UI doesn't start because of pu-common
GS-13014
Deploy data connection as a project
GS-14046
Tableau 2019.4 integration - Aligning com.google.guava versions
GS-14045
View the indexes in Object type exploration
GS-14043
PU is pending deployment when new cli is used(no sla option) and deterministic deployment
GS-14042
Object type data distribution
GS-14038
Object type exploration
GS-14034
Avoid duplication in maven dependencies
GS-14033
Default Hibernate Data Source cause warning message in the log
GS-14032
SQL Query should support named parameters or JPA-style positional parameters
GS-14029
XAP won't start when running Java 11 on Windows 10
GS-14022
During primary space restart when direct persistence is used, records updated in DB might not be replicated and will be missing in the new primary
GS-14015
Customer would like to have a simplified way to set JMX options
GS-14013
Support Quotes and white spaces in JavaCommandBuilder#optionsFromEnv
GS-14012
DeSerialization problems reading from blobstore after upgrade from 11 to 12.3 version in hybrid mode
GS-14011
Deterministic deployment should use demote instead of restarts
GS-14009
java.lang.IllegalArgumentException: The constructor's class loader cannot find the bytecode that defined the constructor's class
GS-14007
Calling GigaSpace#aggregate sometimes causes a NPE when type definition is missing in one of the parttions
GS-14006
Simplify Lucene properties configuration
GS-14005
Failure to get pus using rest api when primary backup zones are defined
GS-14002
Dynamic Service Orchestration
GS-13997
xap-spatial add support for : MultiPoint, MultiLineString and MultiPolygon are not supported at all
GS-13996
Introduce a new space iterator implementation
GS-13995
Web-UI - drop 2
GS-13994
ClassCastException when deploy/un-deploy the same jar three times
GS-13988
Relook cluster proxy in client side automatically when space number pf partitions are changed
GS-13985
issue 1 of 229

Customer cannot submit spark job in IE version 15.0 with specific k8s versions

Description

  • Originally customer was getting '403 Forbidden' error

2019-12-04 20:37:58,474 [OkHttp https://kubernetes.default.svc/...] WARN - Exec Failure: HTTP 403, Status: 403 -
java.net.ProtocolException: Expected HTTP 101 response but was '403 Forbidden'
at okhttp3.internal.ws.RealWebSocket.checkResponse(RealWebSocket.java:216)
at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:183)
at okhttp3.RealCall$AsyncCall.execute(RealCall.java:141)
at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2019-12-04 20:37:58,478 [main] WARN - Kubernetes client has been closed (this is expected if the application is shutting down.)

For example (snippet from Dockerfile):

FROM gigaspaces/insightedge-enterprise:15.0

RUN rm /opt/gigaspaces/insightedge/spark/jars/kubernetes-client-4.1.2.jar
RUN rm /opt/gigaspaces/insightedge/spark/jars/kubernetes-model-4.1.2.jar
RUN rm /opt/gigaspaces/insightedge/spark/jars/kubernetes-model-common-4.1.2.jar

ADD kubernetes-client-4.4.2.jar /opt/gigaspaces/insightedge/spark/jars/
ADD kubernetes-model-4.4.2.jar /opt/gigaspaces/insightedge/spark/jars/
ADD kubernetes-model-common-4.4.2.jar /opt/gigaspaces/insightedge/spark/jars/

  • Once this was done the customer did not get the '403 Forbidden' error.
    However, they would get the following error:

2019-12-05 20:46:16,357 [task-result-getter-0] WARN - Lost task 63.0 in stage 54.0 (TID 13033, 172.30.107.18, executor 72): java.lang.NoClassDefFoundError: org/openspaces/core/space/SpaceConfigurer

at org.insightedge.spark.utils.GridProxyFactory$.org$insightedge$spark$utils$GridProxyFactory$$createSpaceProxy(GridProxyFactory.scala:41)
at org.insightedge.spark.utils.GridProxyFactory$$anonfun$getOrCreateClustered$1.apply(GridProxyFactory.scala:35)
at org.insightedge.spark.utils.GridProxyFactory$$anonfun$getOrCreateClustered$1.apply(GridProxyFactory.scala:35)
at org.insightedge.spark.utils.LocalCache.org$insightedge$spark$utils$LocalCache$$updateIfRequired(LocalCache.scala:43)
at org.insightedge.spark.utils.LocalCache$$anonfun$getOrElseUpdate$1.apply(LocalCache.scala:31)
at scala.collection.MapLike$class.getOrElse(MapLike.scala:128)
at scala.collection.AbstractMap.getOrElse(Map.scala:59)
at org.insightedge.spark.utils.LocalCache.getOrElseUpdate(LocalCache.scala:31)
at org.insightedge.spark.utils.GridProxyFactory$.getOrCreateClustered(GridProxyFactory.scala:35)
at org.insightedge.spark.rdd.InsightEdgeRDDFunctions$$anonfun$saveToGrid$1.apply(InsightEdgeRDDFunctions.scala:50)
at org.insightedge.spark.rdd.InsightEdgeRDDFunctions$$anonfun$saveToGrid$1.apply(InsightEdgeRDDFunctions.scala:49)

  • Our response was to suggest to revert to the previous version of k8s. However, it's not possible on this customer's infrastructure.

  • Now we are at an impasse and customer would like this resolved post haste.

Workaround

None

Acceptance Test

None

Assignee

Unassigned

Reporter

Dixson Huie

Labels

None

Priority

Critical

SalesForce Case ID

None

Fix versions

None

Commitment Version/s

None

Due date

None

Product

None

Edition

Open Source

Platform

All
Configure