Customer cannot submit spark job in IE version 15.0 with specific k8s versions
GS-13982
logback files are not being created in version 14.5
GS-13923
WARNING replicating a packet which is bigger than the batch size - Who is the target?
GS-13918
ReplicaNoProgressException timing is wrong
GS-13874
Durable Notification with FIFO order cause memory leak in rare conditions
GS-13808
IllegalArgumentException: A metric named process_cpu_time-total already exists when deploy a pu.
GS-13187
InsightEdge JDBC Driver code throwing exception for object not found
GS-14001
Tool for generating code from a JDBC connection
GS-13992
sparkContext.gridSql doesn't support nested objects
GS-13924
Enforcing per core XAP license for XAP users
GS-13898
Aggregation sum ignores task transactions
GS-13850
After network recovery reconnected Space instance was stuck verifying last primary state
GS-13782
web-ui directories are exposed by this url: http://localhost:8099/images/
GS-13605
Static Analysis improvements
GS-13507
xap demo CLI command sometimes does not performs space deploy on busy machine
GS-13504
RocksDB Off-heap cache update optimization
GS-13435
Abridged deployment of Space with MemoryXtend
GS-13426
Native support for Scala Case Class
GS-13425
XAP_GSM_OPTIONS are not honored
GS-13298
mvn install doesn't work for 10.1.0-12630-RELEASE
GS-13272
Notifications are not received when using SSL and slow consumers
GS-13150
CLI for Grid Management
GS-13025
New MemoryXtend Metrics
GS-13024
Web UI doesn't start because of pu-common
GS-13014
Customer would like to have a simplified way to set JMX options
GS-14013
Support Quotes and white spaces in JavaCommandBuilder#optionsFromEnv
GS-14012
DeSerialization problems reading from blobstore after upgrade from 11 to 12.3 version in hybrid mode
GS-14011
Deterministic deployment should use demote instead of restarts
GS-14009
java.lang.IllegalArgumentException: The constructor's class loader cannot find the bytecode that defined the constructor's class
GS-14007
Calling GigaSpace#aggregate sometimes causes a NPE when type definition is missing in one of the parttions
GS-14006
Simplify Lucene properties configuration
GS-14005
Generate heap dump from ui fail with java 11
GS-14004
Failure to get pus using rest api when primary backup zones are defined
GS-14002
Dynamic Service Orchestration
GS-13997
xap-spatial add support for : MultiPoint, MultiLineString and MultiPolygon are not supported at all
GS-13996
Introduce a new space iterator implementation
GS-13995
Web-UI - drop 2
GS-13994
ClassCastException when deploy/un-deploy the same jar three times
GS-13988
Relook cluster proxy in client side automatically when space number pf partitions are changed
GS-13985
Add custom security to Rest api, new cli, new ui
GS-13983
Package Tableau connector as a .taco file
GS-13975
Fail to deploy elastic pu due to incorrect memory calculation
GS-13973
New gs.sh cli tool does not have deploy-application functionality
GS-13968
Add an option to access rest manger web only from manger machine
GS-13960
Collect cluster memory usage info and print in GSM log daily
GS-13959
Add cli and rest command to generate cluster dump
GS-13957
Allow getting partial results from broadcast
GS-13954
Querying space text index and space index takes very long time in some condition
GS-13953
Deployment Properties do not support semicolon (";") in name or value
GS-13945
Add CPU memory and NW statistics when Severe messages are logged and dump is preformed
GS-13943
issue 1 of 218

Customer cannot submit spark job in IE version 15.0 with specific k8s versions

Description

  • Originally customer was getting '403 Forbidden' error

2019-12-04 20:37:58,474 [OkHttp https://kubernetes.default.svc/...] WARN - Exec Failure: HTTP 403, Status: 403 -
java.net.ProtocolException: Expected HTTP 101 response but was '403 Forbidden'
at okhttp3.internal.ws.RealWebSocket.checkResponse(RealWebSocket.java:216)
at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:183)
at okhttp3.RealCall$AsyncCall.execute(RealCall.java:141)
at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2019-12-04 20:37:58,478 [main] WARN - Kubernetes client has been closed (this is expected if the application is shutting down.)

For example (snippet from Dockerfile):

FROM gigaspaces/insightedge-enterprise:15.0

RUN rm /opt/gigaspaces/insightedge/spark/jars/kubernetes-client-4.1.2.jar
RUN rm /opt/gigaspaces/insightedge/spark/jars/kubernetes-model-4.1.2.jar
RUN rm /opt/gigaspaces/insightedge/spark/jars/kubernetes-model-common-4.1.2.jar

ADD kubernetes-client-4.4.2.jar /opt/gigaspaces/insightedge/spark/jars/
ADD kubernetes-model-4.4.2.jar /opt/gigaspaces/insightedge/spark/jars/
ADD kubernetes-model-common-4.4.2.jar /opt/gigaspaces/insightedge/spark/jars/

  • Once this was done the customer did not get the '403 Forbidden' error.
    However, they would get the following error:

2019-12-05 20:46:16,357 [task-result-getter-0] WARN - Lost task 63.0 in stage 54.0 (TID 13033, 172.30.107.18, executor 72): java.lang.NoClassDefFoundError: org/openspaces/core/space/SpaceConfigurer

at org.insightedge.spark.utils.GridProxyFactory$.org$insightedge$spark$utils$GridProxyFactory$$createSpaceProxy(GridProxyFactory.scala:41)
at org.insightedge.spark.utils.GridProxyFactory$$anonfun$getOrCreateClustered$1.apply(GridProxyFactory.scala:35)
at org.insightedge.spark.utils.GridProxyFactory$$anonfun$getOrCreateClustered$1.apply(GridProxyFactory.scala:35)
at org.insightedge.spark.utils.LocalCache.org$insightedge$spark$utils$LocalCache$$updateIfRequired(LocalCache.scala:43)
at org.insightedge.spark.utils.LocalCache$$anonfun$getOrElseUpdate$1.apply(LocalCache.scala:31)
at scala.collection.MapLike$class.getOrElse(MapLike.scala:128)
at scala.collection.AbstractMap.getOrElse(Map.scala:59)
at org.insightedge.spark.utils.LocalCache.getOrElseUpdate(LocalCache.scala:31)
at org.insightedge.spark.utils.GridProxyFactory$.getOrCreateClustered(GridProxyFactory.scala:35)
at org.insightedge.spark.rdd.InsightEdgeRDDFunctions$$anonfun$saveToGrid$1.apply(InsightEdgeRDDFunctions.scala:50)
at org.insightedge.spark.rdd.InsightEdgeRDDFunctions$$anonfun$saveToGrid$1.apply(InsightEdgeRDDFunctions.scala:49)

  • Our response was to suggest to revert to the previous version of k8s. However, it's not possible on this customer's infrastructure.

  • Now we are at an impasse and customer would like this resolved post haste.

Workaround

None

Acceptance Test

None

Assignee

Unassigned

Reporter

Dixson Huie

Labels

None

Priority

Critical

SalesForce Case ID

None

Fix versions

None

Commitment Version/s

None

Due date

None

Product

None

Edition

Open Source

Platform

All
Configure