cameron314
When there is lot of enqueue , it fails
I have been using queue like this
parallelQueue::ConcurrentQueue<myDS,mYTraits> *pMPSC_consume;
pMPSC_consume = new ConcurrentQueue(PARALLEL_QUEUE_MAX_NODES, 0, enteredNoOfThreads);
and when i call pMPSC_consume->try_enqueue() , its failing at times ....
I have given PARALLEL_QUEUE_MAX_NODES and even at times there are close to like max 10K to 20K in loop, but even then the try_enqueue fails , please let me know what i need to do to handle this
thanks
typedef struct mYTraits: public parallelQueue::ConcurrentQueueDefaultTraits { static const size_t BLOCK_SIZE = 128; static const size_t INITIAL_IMPLICIT_PRODUCER_HASH_SIZE = 128; static const size_t IMPLICIT_INITIAL_INDEX_SIZE = 128; };
and these are the Index sizes
I am just producing from single thread and consuming this queue from single thread, without re-allocating further memory
as per the code / Computes the correct amount of pre-allocated blocks for you based // on the minimum number of elements you want available at any given // time, and the maximum concurrent number of each type of producer.
Then isnt a producer thread should have 100K nodes ? but its failing when it touched 16K in a loop
the the enteredNoOfThread is just 1
pMPSC_consume = new ConcurrentQueue(PARALLEL_QUEUE_MAX_NODES, 0, enteredNoOfThreads);
As documented, with an IMPLICIT_INITIAL_INDEX_SIZE of 128 and a block size of 128, that gives a max of 16k elements per implicit producer. Increase IMPLICIT_INITIAL_INDEX_SIZE or use enqueue instead of try_enqueue. The arguments given to the constructor allocate enough blocks for more elements than that in advance, but the index size is your limitation here.
ohh ConcurrentQueue(PARALLEL_QUEUE_MAX_NODES, 0, enteredNoOfThreads), the no of thread are like no parallel producers, which means that each thread will be having Max 128 x 128 thats total of approx 16K nodes as per of size myDS struct, But in this case i was creating queue of like its producing from single thread only , hence
i did something like BLOCK_SIZE = 1024, and IMPLICIT_INITIAL_INDEX_SIZE = 2048
Yes , this i had made the change last sat . I am observing more ... Do you have any scribbling or any doc that visualizes the parallel queue, i get that its parallel queues as per no of producers, and each producer has array of nodes to access ....
Not more than what's in the comments, README, and blog post. Yes, each producer is mapped to a sub-queue which has an index or list of blocks of elements.