robberlang

Queue batch prediction tasks collectively

1

This PR allows inputs of batch queries (using `input_batch`) to be sent to the model container all together (limited to the maximum batch size), rather than individually. It does this...

Fix potential deadlock when deleting queue

1

This fixes Issue #758 by using an upgrade lock. As well, it makes `RPCService::send_message` thread-safe by using atomic int, and removes a redundant call to `TaskExecutionThreadPool::create_queue` and uses `std::unique_ptr` rather...

A deadlock can occur when a model container replica is removed, crippling all communication between the frontend and all model containers. Any requests that are not in the cache will...

No quantization of parameter with name ending in _Wt

1

### Description marian-conv does not work properly with models trained with `tied-embeddings` and `tied-embeddings-all` both set to false. This PR removes quantization of parameters with name ending in _Wt, which...

Error in decoding using packed binary model (packed8avx512)

6

### Bug description A fatal error occurs when decoding with a model that was converted to `packed8avx512` GEMM type from a .npz formatted model using `marian-conv`. I have a few...