Aggregation sum ignores task transactions

Description

When a change operation is done within a transactional task, the sum aggregation not always return the expected result and seems to read locked data.

There is a T grid test
com.gigaspaces.test.support.morgan.case12437.AggregationUnderTransactionUsingMultiThreadTest

And I have also attached a standalone test MorganGS13850.zip

Please run the standalone test as follows:

  1. use 15.5

  2. ./gs.sh host run-agent --manager --gsc=10

  3. deploy the embedded space inside the Space module.

  4. run the {{TestTransactionn.java }}receiving Non-zero sum value returned! sum=-xxx.x means that it reproduced

Attachments

3
  • 30 Sep 2020, 01:54 PM
  • 30 Sep 2020, 01:54 PM
  • 22 Sep 2020, 08:51 AM

Activity

Ester Atsmon 
May 23, 2023 at 10:51 AM

This issue was worked around by using one object that includes all related objects that should be committed in one transaction. Changing the object structures and making the queries and report code more complicated. and each object weight much more.

And solution can’t scale when more objects are involved

The customer is waiting for MVCC solution to have consistent query and aggregation results.

Yuval Dori 
September 30, 2020 at 2:24 PM
(edited)

On Sep 30 Niv & I had a session with Wasim & Fahd. In this session Niv suggested 2 options to solve this issue.

  1. Pessimistic concurrency - In the aggregation do read multiple with exclusive read lock.

  2. Optimistic concurrency - scan the aggregate entries before and after the aggregation, verify that the entries haven't been modified by the change operations. This can be done by verifying that the entries version hasn't been changed. If it been changed, the query should be run again and again.

Wasim & Fahd prefered the second option and will try it out. The presentation and the code sample has been attache to FD4855

Niv Ingberg 
September 30, 2020 at 1:55 PM

Attached slides and solution code.

Please close this issue.

Yuval Dori 
July 13, 2020 at 8:26 AM

The test is doing the following:

  1. prepare 1K objects (properties: Integer id, Double risk) while the sum of all risk value is 0. This is done by setting 0 - 500 to half of them and -500 - 0 to the other half. so 1 + (-1) + 2 (-2) + ... = 0

  2. write multiple the 1k objects.

  3. start 5 threads. each run a loop from 0 to Integer MAX Value. In each iteration sum is performed: final Double sum = QueryExtension.sum(gigaSpace, new SQLQuery<>(TestBean.class, ""), "risk");

  4. then a while true loop is executed. each iteration starts a transaction and commits at the end. Inside the transaction in each iteration, a transactional Task that received a Map with a changeSet: Map<Integer, ChangeSet> changeSetMap) is executed. The ChangeSet that passed to each Task is a change (by doubled with -1) of the risk values. So still the sun of the whole 1K risks is always 0.

  5. the expectation is that the 5 thread (from point 3) will always receive sum = 0

  6. worked with 1 partition.

  7. running the test today I always got sum = 0 but it did reproduce a few months ago.

  8. the test is attached.

  9. can you think of a reason why the sum will not always return 0?

Yuval Dori 
February 12, 2020 at 6:36 PM

This is a WAD and the customer has a workaround.

Won't Fix

Details

Assignee

Reporter

Participants of an issue

Ester Atsmon
Niv Ingberg
Yuval Dori

Priority

SalesForce Case ID

Edition

Platform

All

Freshdesk Support

Open Freshdesk Support
Created June 3, 2019 at 6:21 AM
Updated May 23, 2023 at 10:51 AM
Resolved February 12, 2020 at 6:36 PM
Freshdesk Support