Does the QPL library have an API compatible with Rocksdb CRC32C

[damoncui1993]

My company used to use CRC32C from the Rocksdb library for data validation. Currently, I want to use the API interface of the QPL library to accelerate CRC verification, but I have encountered another problem:

1. The result of QPL API verification data is inconsistent with CRC32C, which will cause the business to be unable to use it. I need to use QPL API to be compatible with CRC32C. (Similar to crc32_icsi in Intel ISAL library)
2. The QPL API will report an error when the data exceeds 1MB, as follows An error 57 acquired during CRC calculation.

[damoncui1993]

My test code is as follows:

#include <boost/crc.hpp>
#include <iostream>
#include <vector>
#include <chrono>
#include <cstdlib> // for std::rand and std::srand
#include <ctime>   // for std::time
#include "isa-l.h" // 引入ISA-L CRC实现
#include "base/crc32c.h"  // 引入Rocksdb_base_common CRC实现
#include <memory>  // Add this line
#include "qpl/qpl.h"

// QPL_API
uint64_t qpl_crc64(const std::vector<uint8_t>& data, uint32_t source_size, qpl_path_t execution_path = qpl_path_auto) {
    uint64_t crc_value = 0;
    std::unique_ptr<uint8_t[]> job_buffer;
    uint32_t size = 0;

    // Job initialization
    qpl_status status = qpl_get_job_size(execution_path, &size);
    if (status != QPL_STS_OK) {
        std::cout << "An error " << status << " acquired during job size getting.\n";
        return 1;
    }

    job_buffer = std::make_unique<uint8_t[]>(size);
    qpl_job *job = reinterpret_cast<qpl_job *>(job_buffer.get());

    status = qpl_init_job(execution_path, job);
    if (status != QPL_STS_OK) {
        std::cout << "An error " << status << " acquired during job initializing.\n";
        return 1;
    }

    // Performing an operation
    job->op           = qpl_op_crc64;
    job->next_in_ptr  = const_cast<uint8_t*>(data.data());
    job->available_in = source_size;
    job->crc64_poly   = 0x1EDC6F4100000000;  //poly_crc32_iscsi

    // execute job
    status = qpl_execute_job(job);
    if (status != QPL_STS_OK) {
        std::cout << "An error " << status << " acquired during CRC calculation.\n";
        return 1;
    }

    crc_value = job->crc64;

    return crc_value;
}

int main() {
    // 使用当前时间作为随机种子
    std::srand(std::time(nullptr));

    // 生成一些数据
    std::vector<uint8_t> data(100000000); // 100 MB 的数据
    for (auto& byte : data) {
        byte = std::rand() % 256; // 填充随机数据
    }

    // 使用 Rocksdb CRC32 测试
    auto start_custom = std::chrono::high_resolution_clock::now();
    uint32_t crc_custom = base::crc32c::Extend(0, reinterpret_cast<const char*>(data.data()), data.size());
    auto end_custom = std::chrono::high_resolution_clock::now();
    std::chrono::duration<double, std::milli> elapsed_custom = end_custom - start_custom;
    std::cout << "Rocksdb CRC32: " << crc_custom << ", Time: " << elapsed_custom.count() << " ms" << std::endl;

    // ISA-L crc32_iscsi
    auto start_isal_iscsi = std::chrono::high_resolution_clock::now();
    uint32_t crc_isal_iscsi = ~crc32_iscsi((uint8_t *) data.data(), data.size(), 0xFFFFFFFF);
    auto end_isal_iscsi = std::chrono::high_resolution_clock::now();
    std::chrono::duration<double, std::milli> elapsed_isal_iscsi = end_isal_iscsi - start_isal_iscsi;
    std::cout << "ISA-L CRC32_ISCSI: " << crc_isal_iscsi << ", Time: " << elapsed_isal_iscsi.count() << " ms" << std::endl;

    // QPL_CRC64
    auto start_qpl_crc64 = std::chrono::high_resolution_clock::now();
    uint64_t crc_qpl_crc64 = 0;
    uint32_t source_size = data.size();
    crc_qpl_crc64 = qpl_crc64(data, source_size);
    auto end_qpl_crc64 = std::chrono::high_resolution_clock::now();
    std::chrono::duration<double, std::milli> elapsed_qpl_crc64 = end_qpl_crc64 - start_qpl_crc64;
    std::cout << "QPL CRC64: " << crc_qpl_crc64 << ", Time: " << elapsed_qpl_crc64.count() << " ms" << std::endl;

    return 0;
}

Result: Rocksdb CRC32: 2477611959, Time: 0.123651 ms ISA-L CRC32_ISCSI: 2477611959, Time: 0.017069 ms QPL CRC64: 18188091449403244544, Time: 4.58579 ms

[abdelrahim-hentabli]

Hey @damoncui1993 could you try adding this line to your usage?

job->flags        = QPL_FLAG_CRC64_INV | QPL_FLAG_CRC64_BE

currently, you are inverting the isal result, and not the QPL result. While also, you are having the result in the form of a 64 bit value: so something like

QPL CRC64: 18188091449403244544 = 0xFC6914CF00000000 

and so the value your result is incorrect.

Here is what I got by adding the correct flags:

[mzhukova]

I'm closing the issue since there is no response from the user for a long time and the issue was in a missing flag in the user's code. Please, feel free to re-open if any questions.