Implement ready space storage mode for vinyl engine for utube and utubettl

[DerekBum]

This issue will become relevant after merging this PR: https://github.com/tarantool/queue/pull/229

Right now new STORAGE_MODE_UTUBE_READY_SPACE storage mode for utube and utubettl implemented only for memtx engine. With vinyl there is a problem with yields inside of space and space.index methods. Conflict ER_TRANSACTION_CONFLICT: Transaction has been aborted by conflict may be received in a transaction, resulting in incorrect work of the tube.

Possible fix: repeat the transaction in case of a conflict several times.

[DerekBum]

Ready space.

Original issue #228 was fixed by creating new ready space mode for utube and utubettl. The main idea is to store in the new space only first (in priority order) READY task from each ready queue (queues without any TAKEN tasks) at any given moment. And this invariant is crucial for correct queue work.

This mode was implemented only for memtx engine.

Issue overview.

The main problem with vinyl is that it might yield on almost every space method: https://www.tarantool.io/en/doc/latest/concepts/coop_multitasking/#implicit-yields. And this issue is what prevents us from implementing ready space for vinyl.

Right now every operation on a queue requires working with both spaces: main queue space with all tasks and ready space. And in a nutshell, if vinyl will yield between working with different spaces, that will result in breaking invariants of the ready space.

To be more precise.

Utube.

Lets review some examples.

• Lets say we delete the TAKEN task from the queue. This will result in search for the first READY task to put into ready space: https://github.com/tarantool/queue/blob/df24f171690e9128e94d4ced938cebd8a547eadb/queue/abstract/driver/utube.lua#L310-L311 https://github.com/tarantool/queue/blob/df24f171690e9128e94d4ced938cebd8a547eadb/queue/abstract/driver/utube.lua#L134-L150 Lets simulate yield between lines 138 and 139:

    if taken == nil or taken[2] ~= state.TAKEN then
        require('fiber').sleep(10)
        local next_task = self.space.index.utube:min{state.READY, utube}

After that lets open two terminals:

> queue = require('queue')
---
...
> box.cfg()
---
...
> test_queue = queue.create_tube('test_queue', 'utube', {engine = 'vinyl', storage_mode =  queue.driver.utube.STORAGE_MODE_READY_BUFFER})
---
...
> test_queue:put('0', {utube = tostring(1)})
---
- [0, 'r', '0']
...
> test_queue:put('1', {utube = tostring(1)})
---
- [1, 'r', '1']
...
> test_queue:put('2', {utube = tostring(1)})
---
- [2, 'r', '2']
...
> test_queue:take(0)
---
- [0, 't', '0']
...
> test_queue:delete(0)
// sleep here! Need to call take from the second terminal!
---
- [0, '-', '0']
...

Second terminal (after delete call)

> test_queue:take(1)
---
- [1, 't', '1']
...

After all that we will have two taken tasks from the single queue (task 1 and task 2).

• We can also break this method by putting sleep before line 148 (imitating yield inside insert call). This way we can insert TAKEN or even deleted task into space ready.
• Same errors could be done for bury and release (they all the same method).
• Similar thing could be done with an ordinary put. https://github.com/tarantool/queue/blob/df24f171690e9128e94d4ced938cebd8a547eadb/queue/abstract/driver/utube.lua#L152-L164 We can delete the task after checks on line 156, but before line 162. Thus, we will insert deleted task into ready space.
• If we take the task and then release it, this could result in absence of utube in the ready space. https://github.com/tarantool/queue/blob/df24f171690e9128e94d4ced938cebd8a547eadb/queue/abstract/driver/utube.lua#L216-L248 All we need to do is simulate an yield between lines 235 and 236. While yielding we need to release taken task from the second terminal. Thus this task will be READY (as well as this whole utube), but this task will also be deleted from the ready space (without inserting any new tasks).

As we can see, the main problem here is the necessity to work with two separate spaces. Firstly we need to check that all invariants are fulfilled (using main space) and then to update the ready space. And vinyl could yield between these two steps and break those invariants. But given changes will still affect ready space.

Utubettl.

All yield conflicts from above will also work for utubettl. Moreover, there are more nasty bugs while working with ttl and ttr timeouts.

Proposition.

We cannot fix vinyl for this mode implementation. To make it work we need to write another one, without using several spaces (but only one), because any other solution will result in the same bugs as before (some breaking changes between working with different spaces). This is the hard way.

The easy way is to leave vinyl unsupported.