Aggregation sum ignores task transactions
Description
Attachments
- 30 Sep 2020, 01:54 PM
- 30 Sep 2020, 01:54 PM
- 22 Sep 2020, 08:51 AM
Activity
Ester Atsmon May 23, 2023 at 10:51 AM
This issue was worked around by using one object that includes all related objects that should be committed in one transaction. Changing the object structures and making the queries and report code more complicated. and each object weight much more.
And solution can’t scale when more objects are involved
The customer is waiting for MVCC solution to have consistent query and aggregation results.
Yuval Dori September 30, 2020 at 2:24 PM(edited)
On Sep 30 Niv & I had a session with Wasim & Fahd. In this session Niv suggested 2 options to solve this issue.
Pessimistic concurrency - In the aggregation do read multiple with exclusive read lock.
Optimistic concurrency - scan the aggregate entries before and after the aggregation, verify that the entries haven't been modified by the change operations. This can be done by verifying that the entries version hasn't been changed. If it been changed, the query should be run again and again.
Wasim & Fahd prefered the second option and will try it out. The presentation and the code sample has been attache to FD4855
Niv Ingberg September 30, 2020 at 1:55 PM
Attached slides and solution code.
Please close this issue.
Yuval Dori July 13, 2020 at 8:26 AM
The test is doing the following:
prepare 1K objects (properties: Integer id, Double risk) while the sum of all risk value is 0. This is done by setting 0 - 500 to half of them and -500 - 0 to the other half. so 1 + (-1) + 2 (-2) + ... = 0
write multiple the 1k objects.
start 5 threads. each run a loop from 0 to Integer MAX Value. In each iteration sum is performed: final Double sum = QueryExtension.sum(gigaSpace, new SQLQuery<>(TestBean.class, ""), "risk");
then a while true loop is executed. each iteration starts a transaction and commits at the end. Inside the transaction in each iteration, a transactional Task that received a Map with a changeSet: Map<Integer, ChangeSet> changeSetMap) is executed. The ChangeSet that passed to each Task is a change (by doubled with -1) of the risk values. So still the sun of the whole 1K risks is always 0.
the expectation is that the 5 thread (from point 3) will always receive sum = 0
worked with 1 partition.
running the test today I always got sum = 0 but it did reproduce a few months ago.
the test is attached.
can you think of a reason why the sum will not always return 0?
Yuval Dori February 12, 2020 at 6:36 PM
This is a WAD and the customer has a workaround.
When a change operation is done within a transactional task, the sum aggregation not always return the expected result and seems to read locked data.
There is a T grid test
com.gigaspaces.test.support.morgan.case12437.AggregationUnderTransactionUsingMultiThreadTest
And I have also attached a standalone test MorganGS13850.zip
Please run the standalone test as follows:
use 15.5
./gs.sh host run-agent --manager --gsc=10
deploy the embedded space inside the Space module.
run the {{TestTransactionn.java }}receiving Non-zero sum value returned! sum=-xxx.x means that it reproduced