Redolog can't be read and will keep growing if redolog went to disk and change operations were done by 14.0 or older clients on 14.2 or newer server

Description

The same issue with 14.2 servers and 14.0.0 clients, The issue is not reproduced with 14.0.0 server and 12.3 client

Reproduction:
1. generate my-stateful-with-db bluprint (add this to space properties) properties.setProperty("cluster-config.groups.group.repl-policy.redo-log-memory-capacity","0");
strat the demo
2. undeploy mirror
3. run a client from 12.3 that writes and change objects
4. redeploy the mirror
see exceptions in space log:
2020-12-08 16:45:38,106 my-app-space.1 [1] SEVERE [com.gigaspaces.replication.backlog.primary-backup-reliable-async-mirror-1] - Redo log file integrity validation failed; Caused by: com.gigaspaces.internal.server.space.redolog.storage.bytebuffer.ByteBufferStorageCompromisedException: Error while checking data integrity, reached packet index 100
at com.gigaspaces.internal.server.space.redolog.storage.bytebuffer.ByteBufferRedoLogFileStorage.validateIntegrity(ByteBufferRedoLogFileStorage.java:781)
at com.gigaspaces.internal.server.space.redolog.storage.BufferedRedoLogFileStorageDecorator.validateIntegrity(BufferedRedoLogFileStorageDecorator.java:173)
at com.gigaspaces.internal.server.space.redolog.storage.CacheLastRedoLogFileStorageDecorator.validateIntegrity(CacheLastRedoLogFileStorageDecorator.java:130)
at com.gigaspaces.internal.server.space.redolog.FixedSizeSwapRedoLogFile.validateIntegrity(FixedSizeSwapRedoLogFile.java:271)
at com.gigaspaces.internal.cluster.node.impl.backlog.AbstractSingleFileGroupBacklog.validateIntegrity(AbstractSingleFileGroupBacklog.java:1304)
at com.gigaspaces.internal.cluster.node.impl.backlog.AbstractSingleFileGroupBacklog.getPacketsUnsafe(AbstractSingleFileGroupBacklog.java:943)
at com.gigaspaces.internal.cluster.node.impl.backlog.multisourcesinglefile.MultiSourceSingleFileReliableAsyncGroupBacklog.getReliableAsyncPackets(MultiSourceSingleFileReliableAsyncGroupBacklog.java:265)
at com.gigaspaces.internal.cluster.node.impl.groups.reliableasync.ReliableAsyncReplicationSourceChannel.getPendingPackets(ReliableAsyncReplicationSourceChannel.java:97)
at com.gigaspaces.internal.cluster.node.impl.groups.async.AsyncReplicationSourceChannel$AsyncDispatcher.call(AsyncReplicationSourceChannel.java:250)
at com.gigaspaces.internal.cluster.node.impl.groups.async.AsyncReplicationSourceChannel$AsyncDispatcher.call(AsyncReplicationSourceChannel.java:211)
at com.gigaspaces.internal.utils.concurrent.ScheduledThreadPoolAsyncHandlerProvider$AsyncRunnableWrapper$ExecutionTask.run(ScheduledThreadPoolAsyncHandlerProvider.java:297)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: Unknown class code: 3
at com.gigaspaces.internal.io.IOUtils.readObject(IOUtils.java:736)
at com.gigaspaces.internal.client.mutators.SetValueSpaceEntryMutator.readExternal(SetValueSpaceEntryMutator.java:63)
at java.io.ObjectInputStream.readExternalData(ObjectInputStream.java:2118)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2067)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431)
at java.util.LinkedList.readObject(LinkedList.java:1149)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1170)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2178)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431)
at com.gigaspaces.internal.serialization.ObjectClassSerializer.read(ObjectClassSerializer.java:37)
at com.gigaspaces.internal.io.IOUtils.readObject(IOUtils.java:738)
at com.gigaspaces.internal.cluster.node.impl.packets.data.operations.ChangeReplicationPacketData.readExternalImpl(ChangeReplicationPacketData.java:346)
at com.gigaspaces.internal.cluster.node.impl.packets.data.operations.ChangeReplicationPacketData.readFromSwap(ChangeReplicationPacketData.java:383)
at com.gigaspaces.internal.io.IOUtils.readSwapExternalizableObject(IOUtils.java:895)
at com.gigaspaces.internal.cluster.node.impl.backlog.globalorder.GlobalOrderOperationPacket.readFromSwap(GlobalOrderOperationPacket.java:106)
at com.gigaspaces.internal.io.IOUtils.readSwapExternalizableObject(IOUtils.java:895)
at com.gigaspaces.internal.server.space.redolog.storage.bytebuffer.SwapPacketStreamSerializer.readPacketFromStream(SwapPacketStreamSerializer.java:43)
at com.gigaspaces.internal.cluster.node.impl.backlog.AbstractSingleFileGroupBacklog$1.readPacketFromStream(AbstractSingleFileGroupBacklog.java:383)
at com.gigaspaces.internal.cluster.node.impl.backlog.AbstractSingleFileGroupBacklog$1.readPacketFromStream(AbstractSingleFileGroupBacklog.java:373)
at com.gigaspaces.internal.server.space.redolog.storage.bytebuffer.PacketSerializer$InputStreamsResource.readPacket(PacketSerializer.java:132)
at com.gigaspaces.internal.server.space.redolog.storage.bytebuffer.PacketSerializer.deserializePacket(PacketSerializer.java:100)
at com.gigaspaces.internal.server.space.redolog.storage.bytebuffer.ByteBufferRedoLogFileStorage.validateIntegrity(ByteBufferRedoLogFileStorage.java:756)
... 17 more

Attachments

2
  • 10 Dec 2020, 06:17 AM
  • 10 Dec 2020, 06:06 AM

Activity

Show:

Moran Avigdor February 24, 2022 at 9:49 AM

Notes:

  • backwards in ChangeReplicationPacketData between client in version 12.3 vs newer server version 14.2

Ester Atsmon December 30, 2020 at 1:30 PM

might be related Jira : https://insightedge.atlassian.net/browse/GS-13776 Added ability to disable caching of the change path for the Change API

Fixed

Details

Assignee

Reporter

Labels

Participants of an issue

David Freedman
Ester Atsmon
Moran Avigdor

Priority

Fix versions

Edition

Open Source

Platform

All

Acceptance Test

test, skip message - new implementation of redolog swap

Freshdesk Support

Open Freshdesk Support

Created December 8, 2020 at 2:59 PM
Updated May 18, 2022 at 11:25 AM
Resolved May 2, 2022 at 6:33 AM
Freshdesk Support