FEMU icon indicating copy to clipboard operation
FEMU copied to clipboard
verified publisher icon MoatLab

Cannot write OCSSD2.0 of FEMU using SPDK

Open lovers-under-moon opened this issue 4 years ago • 9 comments

When I run the hello_world(https://github.com/spdk/spdk/tree/v20.10/examples/nvme/hello_world) of SPDK(https://github.com/spdk/spdk/tree/v20.10) in FEMU's OCSSD2.0 mode,I didn't see “Hello world!” being printed.But when I use the FEMU's ZNS mode,the hello_world example runs successfully.The hello_world example will write the "Hello world!" string to the device and then read it out.

- OCSSD2.0 Behavior

femu@fvm:~/spdk$ sudo ./scripts/setup.sh [sudo] password for femu: 0000:00:04.0 (1af4 1004): Active mountpoints on /dev/sda2, so not binding PCI dev 0000:00:05.0 (1d1d 1f1f): nvme -> uio_pci_generic femu@fvm:~/spdk$ sudo build/examples/hello_world [2021-03-25 12:33:54.422615] Starting SPDK v20.10 git sha1 e5d26ecc2 / DPDK 20.08.0 initialization... [2021-03-25 12:33:54.422893] [ DPDK EAL parameters: [2021-03-25 12:33:54.423293] hello_world [2021-03-25 12:33:54.4] EAL: No available hugepages reported in hugepages-1048576kB EAL: No legacy callbacks, legacy socket not created Initializing NVMe Controllers Attaching to 0000:00:05.0 [2021-03-25 12:33:54.557991] nvme_qpair.c: 248:nvme_admin_qpair_print_command: NOTICE: ASYNC EVENT REQUEST (0c) q [2021-03-25 12:33:54.558158] nvme_qpair.c: 451:spdk_nvme_print_completion: NOTICE: INVALID OPCODE (00/01) qid:0 c1 [2021-03-25 12:33:54.558705] nvme_qpair.c: 248:nvme_admin_qpair_print_command: NOTICE: ASYNC EVENT REQUEST (0c) q [2021-03-25 12:33:54.559213] nvme_qpair.c: 451:spdk_nvme_print_completion: NOTICE: INVALID OPCODE (00/01) qid:0 c1 [2021-03-25 12:33:54.559826] nvme_qpair.c: 248:nvme_admin_qpair_print_command: NOTICE: ASYNC EVENT REQUEST (0c) q [2021-03-25 12:33:54.560356] nvme_qpair.c: 451:spdk_nvme_print_completion: NOTICE: INVALID OPCODE (00/01) qid:0 c1 [2021-03-25 12:33:54.560929] nvme_qpair.c: 248:nvme_admin_qpair_print_command: NOTICE: ASYNC EVENT REQUEST (0c) q [2021-03-25 12:33:54.561458] nvme_qpair.c: 451:spdk_nvme_print_completion: NOTICE: INVALID OPCODE (00/01) qid:0 c1 [2021-03-25 12:33:54.562018] nvme_qpair.c: 248:nvme_admin_qpair_print_command: NOTICE: ASYNC EVENT REQUEST (0c) q [2021-03-25 12:33:54.562548] nvme_qpair.c: 451:spdk_nvme_print_completion: NOTICE: INVALID OPCODE (00/01) qid:0 c1 [2021-03-25 12:33:54.563114] nvme_qpair.c: 248:nvme_admin_qpair_print_command: NOTICE: ASYNC EVENT REQUEST (0c) q [2021-03-25 12:33:54.563662] nvme_qpair.c: 451:spdk_nvme_print_completion: NOTICE: INVALID OPCODE (00/01) qid:0 c1 [2021-03-25 12:33:54.564237] nvme_qpair.c: 248:nvme_admin_qpair_print_command: NOTICE: ASYNC EVENT REQUEST (0c) q [2021-03-25 12:33:54.564777] nvme_qpair.c: 451:spdk_nvme_print_completion: NOTICE: INVALID OPCODE (00/01) qid:0 c1 [2021-03-25 12:33:54.566232] nvme_qpair.c: 248:nvme_admin_qpair_print_command: NOTICE: ASYNC EVENT REQUEST (0c) q [2021-03-25 12:33:54.566452] nvme_qpair.c: 451:spdk_nvme_print_completion: NOTICE: INVALID OPCODE (00/01) qid:0 c1 [2021-03-25 12:33:54.567013] nvme_qpair.c: 248:nvme_admin_qpair_print_command: NOTICE: ASYNC EVENT REQUEST (0c) q [2021-03-25 12:33:54.567580] nvme_qpair.c: 451:spdk_nvme_print_completion: NOTICE: INVALID OPCODE (00/01) qid:0 c1 [2021-03-25 12:33:54.568115] nvme_qpair.c: 248:nvme_admin_qpair_print_command: NOTICE: ASYNC EVENT REQUEST (0c) q [2021-03-25 12:33:54.568646] nvme_qpair.c: 451:spdk_nvme_print_completion: NOTICE: INVALID OPCODE (00/01) qid:0 c1 [2021-03-25 12:33:54.569262] nvme_qpair.c: 248:nvme_admin_qpair_print_command: NOTICE: ASYNC EVENT REQUEST (0c) q [2021-03-25 12:33:54.569775] nvme_qpair.c: 451:spdk_nvme_print_completion: NOTICE: INVALID OPCODE (00/01) qid:0 c1 [2021-03-25 12:33:54.570339] nvme_qpair.c: 248:nvme_admin_qpair_print_command: NOTICE: ASYNC EVENT REQUEST (0c) q [2021-03-25 12:33:54.570882] nvme_qpair.c: 451:spdk_nvme_print_completion: NOTICE: INVALID OPCODE (00/01) qid:0 c1 [2021-03-25 12:33:54.571464] nvme_qpair.c: 248:nvme_admin_qpair_print_command: NOTICE: ASYNC EVENT REQUEST (0c) q [2021-03-25 12:33:54.572023] nvme_qpair.c: 451:spdk_nvme_print_completion: NOTICE: INVALID OPCODE (00/01) qid:0 c1 [2021-03-25 12:33:54.572611] nvme_qpair.c: 248:nvme_admin_qpair_print_command: NOTICE: ASYNC EVENT REQUEST (0c) q [2021-03-25 12:33:54.573141] nvme_qpair.c: 451:spdk_nvme_print_completion: NOTICE: INVALID OPCODE (00/01) qid:0 c1 Attached to 0000:00:05.0 Using controller FEMU OpenChannel-SSD (vOCSSD0 ) with 1 namespaces. Namespace ID: 1 size: 4GB Initialization complete. [2021-03-25 12:33:54.660258] nvme_qpair.c: 248:nvme_admin_qpair_print_command: NOTICE: ASYNC EVENT REQUEST (0c) q [2021-03-25 12:33:54.660603] nvme_qpair.c: 451:spdk_nvme_print_completion: NOTICE: INVALID OPCODE (00/01) qid:0 c1 [2021-03-25 12:33:54.661115] nvme_qpair.c: 248:nvme_admin_qpair_print_command: NOTICE: ASYNC EVENT REQUEST (0c) q [2021-03-25 12:33:54.661657] nvme_qpair.c: 451:spdk_nvme_print_completion: NOTICE: INVALID OPCODE (00/01) qid:0 c1 [2021-03-25 12:33:54.662186] nvme_qpair.c: 248:nvme_admin_qpair_print_command: NOTICE: ASYNC EVENT REQUEST (0c) q [2021-03-25 12:33:54.662699] nvme_qpair.c: 451:spdk_nvme_print_completion: NOTICE: INVALID OPCODE (00/01) qid:0 c1 [2021-03-25 12:33:54.663223] nvme_qpair.c: 248:nvme_admin_qpair_print_command: NOTICE: ASYNC EVENT REQUEST (0c) q [2021-03-25 12:33:54.663785] nvme_qpair.c: 451:spdk_nvme_print_completion: NOTICE: INVALID OPCODE (00/01) qid:0 c1 [2021-03-25 12:33:54.664321] nvme_qpair.c: 248:nvme_admin_qpair_print_command: NOTICE: ASYNC EVENT REQUEST (0c) q [2021-03-25 12:33:54.664829] nvme_qpair.c: 451:spdk_nvme_print_completion: NOTICE: INVALID OPCODE (00/01) qid:0 c1 [2021-03-25 12:33:54.665445] nvme_qpair.c: 248:nvme_admin_qpair_print_command: NOTICE: ASYNC EVENT REQUEST (0c) q [2021-03-25 12:33:54.665943] nvme_qpair.c: 451:spdk_nvme_print_completion: NOTICE: INVALID OPCODE (00/01) qid:0 c1 [2021-03-25 12:33:54.666482] nvme_qpair.c: 248:nvme_admin_qpair_print_command: NOTICE: ASYNC EVENT REQUEST (0c) q [2021-03-25 12:33:54.667027] nvme_qpair.c: 451:spdk_nvme_print_completion: NOTICE: INVALID OPCODE (00/01) qid:0 c1 [2021-03-25 12:33:54.667893] nvme_qpair.c: 248:nvme_admin_qpair_print_command: NOTICE: ASYNC EVENT REQUEST (0c) q [2021-03-25 12:33:54.668459] nvme_qpair.c: 451:spdk_nvme_print_completion: NOTICE: INVALID OPCODE (00/01) qid:0 c1 [2021-03-25 12:33:54.669074] nvme_qpair.c: 248:nvme_admin_qpair_print_command: NOTICE: ASYNC EVENT REQUEST (0c) q [2021-03-25 12:33:54.669723] nvme_qpair.c: 451:spdk_nvme_print_completion: NOTICE: INVALID OPCODE (00/01) qid:0 c1 [2021-03-25 12:33:54.670350] nvme_qpair.c: 248:nvme_admin_qpair_print_command: NOTICE: ASYNC EVENT REQUEST (0c) q [2021-03-25 12:33:54.670993] nvme_qpair.c: 451:spdk_nvme_print_completion: NOTICE: INVALID OPCODE (00/01) qid:0 c1 [2021-03-25 12:33:54.671630] nvme_qpair.c: 248:nvme_admin_qpair_print_command: NOTICE: ASYNC EVENT REQUEST (0c) q [2021-03-25 12:33:54.672258] nvme_qpair.c: 451:spdk_nvme_print_completion: NOTICE: INVALID OPCODE (00/01) qid:0 c1 [2021-03-25 12:33:54.672908] nvme_qpair.c: 248:nvme_admin_qpair_print_command: NOTICE: ASYNC EVENT REQUEST (0c) q [2021-03-25 12:33:54.673509] nvme_qpair.c: 451:spdk_nvme_print_completion: NOTICE: INVALID OPCODE (00/01) qid:0 c1 [2021-03-25 12:33:54.674129] nvme_qpair.c: 248:nvme_admin_qpair_print_command: NOTICE: ASYNC EVENT REQUEST (0c) q [2021-03-25 12:33:54.674706] nvme_qpair.c: 451:spdk_nvme_print_completion: NOTICE: INVALID OPCODE (00/01) qid:0 c1 [2021-03-25 12:33:54.675340] nvme_qpair.c: 248:nvme_admin_qpair_print_command: NOTICE: ASYNC EVENT REQUEST (0c) q [2021-03-25 12:33:54.800299] nvme_qpair.c: 451:spdk_nvme_print_completion: NOTICE: INVALID OPCODE (00/01) qid:0 c1 INFO: using host memory buffer for IO [2021-03-25 12:33:54.801837] nvme_qpair.c: 280:nvme_io_qpair_print_command: NOTICE: OCSSD / VECTOR RESET (90) sqi1 [2021-03-25 12:33:54.802126] nvme_qpair.c: 451:spdk_nvme_print_completion: NOTICE: INVALID RESET (02/c1) qid:1 ci1 [2021-03-25 12:33:54.802562] nvme_qpair.c: 248:nvme_admin_qpair_print_command: NOTICE: ASYNC EVENT REQUEST (0c) q [2021-03-25 12:33:54.802858] nvme_qpair.c: 451:spdk_nvme_print_completion: NOTICE: INVALID OPCODE (00/01) qid:0 c1 [2021-03-25 12:33:54.803197] nvme_qpair.c: 248:nvme_admin_qpair_print_command: NOTICE: ASYNC EVENT REQUEST (0c) q [2021-03-25 12:33:54.803499] nvme_qpair.c: 451:spdk_nvme_print_completion: NOTICE: INVALID OPCODE (00/01) qid:0 c1 [2021-03-25 12:33:54.803882] nvme_qpair.c: 248:nvme_admin_qpair_print_command: NOTICE: ASYNC EVENT REQUEST (0c) q [2021-03-25 12:33:54.804171] nvme_qpair.c: 451:spdk_nvme_print_completion: NOTICE: INVALID OPCODE (00/01) qid:0 c1 [2021-03-25 12:33:54.804499] nvme_qpair.c: 248:nvme_admin_qpair_print_command: NOTICE: ASYNC EVENT REQUEST (0c) q [2021-03-25 12:33:54.804787] nvme_qpair.c: 451:spdk_nvme_print_completion: NOTICE: INVALID OPCODE (00/01) qid:0 c1 [2021-03-25 12:33:54.805123] nvme_qpair.c: 248:nvme_admin_qpair_print_command: NOTICE: ASYNC EVENT REQUEST (0c) q [2021-03-25 12:33:54.805726] nvme_qpair.c: 451:spdk_nvme_print_completion: NOTICE: INVALID OPCODE (00/01) qid:0 c1 [2021-03-25 12:33:54.806084] nvme_qpair.c: 248:nvme_admin_qpair_print_command: NOTICE: ASYNC EVENT REQUEST (0c) q [2021-03-25 12:33:54.806369] nvme_qpair.c: 451:spdk_nvme_print_completion: NOTICE: INVALID OPCODE (00/01) qid:0 c1 [2021-03-25 12:33:54.806703] nvme_qpair.c: 248:nvme_admin_qpair_print_command: NOTICE: ASYNC EVENT REQUEST (0c) q [2021-03-25 12:33:54.807000] nvme_qpair.c: 451:spdk_nvme_print_completion: NOTICE: INVALID OPCODE (00/01) qid:0 c1 [2021-03-25 12:33:54.807413] nvme_qpair.c: 248:nvme_admin_qpair_print_command: NOTICE: ASYNC EVENT REQUEST (0c) q [2021-03-25 12:33:54.807641] nvme_qpair.c: 451:spdk_nvme_print_completion: NOTICE: INVALID OPCODE (00/01) qid:0 c1 [2021-03-25 12:33:54.807978] nvme_qpair.c: 248:nvme_admin_qpair_print_command: NOTICE: ASYNC EVENT REQUEST (0c) q [2021-03-25 12:33:54.808281] nvme_qpair.c: 451:spdk_nvme_print_completion: NOTICE: INVALID OPCODE (00/01) qid:0 c1 [2021-03-25 12:33:54.808613] nvme_qpair.c: 248:nvme_admin_qpair_print_command: NOTICE: ASYNC EVENT REQUEST (0c) q [2021-03-25 12:33:54.808908] nvme_qpair.c: 451:spdk_nvme_print_completion: NOTICE: INVALID OPCODE (00/01) qid:0 c1 [2021-03-25 12:33:54.809603] nvme_qpair.c: 248:nvme_admin_qpair_print_command: NOTICE: ASYNC EVENT REQUEST (0c) q [2021-03-25 12:33:54.809849] nvme_qpair.c: 451:spdk_nvme_print_completion: NOTICE: INVALID OPCODE (00/01) qid:0 c1 [2021-03-25 12:33:54.810192] nvme_qpair.c: 248:nvme_admin_qpair_print_command: NOTICE: ASYNC EVENT REQUEST (0c) q [2021-03-25 12:33:54.810473] nvme_qpair.c: 451:spdk_nvme_print_completion: NOTICE: INVALID OPCODE (00/01) qid:0 c1 [2021-03-25 12:33:54.886881] nvme_qpair.c: 248:nvme_admin_qpair_print_command: NOTICE: ASYNC EVENT REQUEST (0c) q [2021-03-25 12:33:54.887098] nvme_qpair.c: 451:spdk_nvme_print_completion: NOTICE: INVALID OPCODE (00/01) qid:0 c1 [2021-03-25 12:33:54.887430] nvme_qpair.c: 248:nvme_admin_qpair_print_command: NOTICE: ASYNC EVENT REQUEST (0c) q [2021-03-25 12:33:54.887730] nvme_qpair.c: 451:spdk_nvme_print_completion: NOTICE: INVALID OPCODE (00/01) qid:0 c1 [2021-03-25 12:33:54.891282] nvme_qpair.c: 248:nvme_admin_qpair_print_command: NOTICE: ASYNC EVENT REQUEST (0c) q [2021-03-25 12:33:54.891500] nvme_qpair.c: 451:spdk_nvme_print_completion: NOTICE: INVALID OPCODE (00/01) qid:0 c1 [2021-03-25 12:33:54.891812] nvme_qpair.c: 248:nvme_admin_qpair_print_command: NOTICE: ASYNC EVENT REQUEST (0c) q [2021-03-25 12:33:54.892125] nvme_qpair.c: 451:spdk_nvme_print_completion: NOTICE: INVALID OPCODE (00/01) qid:0 c1 [2021-03-25 12:33:54.892433] nvme_qpair.c: 248:nvme_admin_qpair_print_command: NOTICE: ASYNC EVENT REQUEST (0c) q [2021-03-25 12:33:54.892749] nvme_qpair.c: 451:spdk_nvme_print_completion: NOTICE: INVALID OPCODE (00/01) qid:0 c1 [2021-03-25 12:33:54.893064] nvme_qpair.c: 248:nvme_admin_qpair_print_command: NOTICE: ASYNC EVENT REQUEST (0c) q [2021-03-25 12:33:54.893695] nvme_qpair.c: 451:spdk_nvme_print_completion: NOTICE: INVALID OPCODE (00/01) qid:0 c1

- ZNS Behavior

femu@fvm:~/MyFile/spdk$ sudo ./scripts/setup.sh [sudo] password for femu: 0000:00:04.0 (1af4 1004): Active mountpoints on /dev/sda2, so not binding PCI dev 0000:00:05.0 (8086 5845): nvme -> uio_pci_generic femu@fvm:~/MyFile/spdk$ sudo build/examples/hello_world [2021-03-25 13:04:08.826105] Starting SPDK v20.10 git sha1 e5d26ecc2 / DPDK 20.08.0 initialization... [2021-03-25 13:04:08.827183] [ DPDK EAL parameters: [2021-03-25 13:04:08.827416] hello_world [2021-03-25 13:04:08.827691] -c 0x1 [2021-03-25 13:04:08.827986] --log-level=lib.eal:6 [2021-03-25 13:04:08.828292] --log-level=lib.cryptodev:5 [2021-03-25 13:04:08.] EAL: No available hugepages reported in hugepages-1048576kB EAL: No legacy callbacks, legacy socket not created Initializing NVMe Controllers Attaching to 0000:00:05.0 [2021-03-25 13:04:08.958469] nvme_qpair.c: 248:nvme_admin_qpair_print_command: NOTICE: ASYNC EVENT REQUEST (0c) qid:0 cid:22 nsid:0 cdw10:00000000 cdw11:00000000 [2021-03-25 13:04:08.959193] nvme_qpair.c: 451:spdk_nvme_print_completion: NOTICE: INVALID OPCODE (00/01) qid:0 cid:22 cdw0:0 sqhd:0006 p:1 m:0 dnr:1 [2021-03-25 13:04:08.959488] nvme_qpair.c: 248:nvme_admin_qpair_print_command: NOTICE: ASYNC EVENT REQUEST (0c) qid:0 cid:21 nsid:0 cdw10:00000000 cdw11:00000000 [2021-03-25 13:04:08.959740] nvme_qpair.c: 451:spdk_nvme_print_completion: NOTICE: INVALID OPCODE (00/01) qid:0 cid:21 cdw0:0 sqhd:0007 p:1 m:0 dnr:1 [2021-03-25 13:04:08.960059] nvme_qpair.c: 248:nvme_admin_qpair_print_command: NOTICE: ASYNC EVENT REQUEST (0c) qid:0 cid:20 nsid:0 cdw10:00000000 cdw11:00000000 [2021-03-25 13:04:08.960297] nvme_qpair.c: 451:spdk_nvme_print_completion: NOTICE: INVALID OPCODE (00/01) qid:0 cid:20 cdw0:0 sqhd:0008 p:1 m:0 dnr:1 [2021-03-25 13:04:08.960602] nvme_qpair.c: 248:nvme_admin_qpair_print_command: NOTICE: ASYNC EVENT REQUEST (0c) qid:0 cid:19 nsid:0 cdw10:00000000 cdw11:00000000 [2021-03-25 13:04:08.960876] nvme_qpair.c: 451:spdk_nvme_print_completion: NOTICE: INVALID OPCODE (00/01) qid:0 cid:19 cdw0:0 sqhd:0009 p:1 m:0 dnr:1 [2021-03-25 13:04:08.961181] nvme_qpair.c: 248:nvme_admin_qpair_print_command: NOTICE: ASYNC EVENT REQUEST (0c) qid:0 cid:23 nsid:0 cdw10:00000000 cdw11:00000000 [2021-03-25 13:04:08.961462] nvme_qpair.c: 451:spdk_nvme_print_completion: NOTICE: INVALID OPCODE (00/01) qid:0 cid:23 cdw0:0 sqhd:000a p:1 m:0 dnr:1 [2021-03-25 13:04:08.961773] nvme_qpair.c: 248:nvme_admin_qpair_print_command: NOTICE: ASYNC EVENT REQUEST (0c) qid:0 cid:22 nsid:0 cdw10:00000000 cdw11:00000000 [2021-03-25 13:04:08.962036] nvme_qpair.c: 451:spdk_nvme_print_completion: NOTICE: INVALID OPCODE (00/01) qid:0 cid:22 cdw0:0 sqhd:000b p:1 m:0 dnr:1 [2021-03-25 13:04:08.962364] nvme_qpair.c: 248:nvme_admin_qpair_print_command: NOTICE: ASYNC EVENT REQUEST (0c) qid:0 cid:21 nsid:0 cdw10:00000000 cdw11:00000000 [2021-03-25 13:04:08.962621] nvme_qpair.c: 451:spdk_nvme_print_completion: NOTICE: INVALID OPCODE (00/01) qid:0 cid:21 cdw0:0 sqhd:000c p:1 m:0 dnr:1 [2021-03-25 13:04:08.963387] nvme_qpair.c: 248:nvme_admin_qpair_print_command: NOTICE: ASYNC EVENT REQUEST (0c) qid:0 cid:20 nsid:0 cdw10:00000000 cdw11:00000000 [2021-03-25 13:04:08.963480] nvme_qpair.c: 451:spdk_nvme_print_completion: NOTICE: INVALID OPCODE (00/01) qid:0 cid:20 cdw0:0 sqhd:000d p:1 m:0 dnr:1 [2021-03-25 13:04:08.963872] nvme_qpair.c: 248:nvme_admin_qpair_print_command: NOTICE: ASYNC EVENT REQUEST (0c) qid:0 cid:19 nsid:0 cdw10:00000000 cdw11:00000000 [2021-03-25 13:04:08.964049] nvme_qpair.c: 451:spdk_nvme_print_completion: NOTICE: INVALID OPCODE (00/01) qid:0 cid:19 cdw0:0 sqhd:000e p:1 m:0 dnr:1 [2021-03-25 13:04:08.964361] nvme_qpair.c: 248:nvme_admin_qpair_print_command: NOTICE: ASYNC EVENT REQUEST (0c) qid:0 cid:23 nsid:0 cdw10:00000000 cdw11:00000000 [2021-03-25 13:04:08.964591] nvme_qpair.c: 451:spdk_nvme_print_completion: NOTICE: INVALID OPCODE (00/01) qid:0 cid:23 cdw0:0 sqhd:000f p:1 m:0 dnr:1 [2021-03-25 13:04:08.964918] nvme_qpair.c: 248:nvme_admin_qpair_print_command: NOTICE: ASYNC EVENT REQUEST (0c) qid:0 cid:22 nsid:0 cdw10:00000000 cdw11:00000000 [2021-03-25 13:04:08.965141] nvme_qpair.c: 451:spdk_nvme_print_completion: NOTICE: INVALID OPCODE (00/01) qid:0 cid:22 cdw0:0 sqhd:0010 p:1 m:0 dnr:1 [2021-03-25 13:04:08.965458] nvme_qpair.c: 248:nvme_admin_qpair_print_command: NOTICE: ASYNC EVENT REQUEST (0c) qid:0 cid:18 nsid:0 cdw10:00000000 cdw11:00000000 [2021-03-25 13:04:08.965706] nvme_qpair.c: 451:spdk_nvme_print_completion: NOTICE: INVALID OPCODE (00/01) qid:0 cid:18 cdw0:0 sqhd:0012 p:1 m:0 dnr:1 [2021-03-25 13:04:08.966013] nvme_qpair.c: 248:nvme_admin_qpair_print_command: NOTICE: ASYNC EVENT REQUEST (0c) qid:0 cid:20 nsid:0 cdw10:00000000 cdw11:00000000 [2021-03-25 13:04:08.966281] nvme_qpair.c: 451:spdk_nvme_print_completion: NOTICE: INVALID OPCODE (00/01) qid:0 cid:20 cdw0:0 sqhd:0013 p:1 m:0 dnr:1 [2021-03-25 13:04:08.966588] nvme_qpair.c: 248:nvme_admin_qpair_print_command: NOTICE: ASYNC EVENT REQUEST (0c) qid:0 cid:19 nsid:0 cdw10:00000000 cdw11:00000000 [2021-03-25 13:04:08.967170] nvme_qpair.c: 451:spdk_nvme_print_completion: NOTICE: INVALID OPCODE (00/01) qid:0 cid:19 cdw0:0 sqhd:0014 p:1 m:0 dnr:1 Attached to 0000:00:05.0 Using controller FEMU ZNS-SSD Control (vZNSSD0 ) with 1 namespaces. Namespace ID: 1 size: 4GB Initialization complete. [2021-03-25 13:04:09.039279] nvme_qpair.c: 248:nvme_admin_qpair_print_command: NOTICE: ASYNC EVENT REQUEST (0c) qid:0 cid:23 nsid:0 cdw10:00000000 cdw11:00000000 [2021-03-25 13:04:09.039390] nvme_qpair.c: 451:spdk_nvme_print_completion: NOTICE: INVALID OPCODE (00/01) qid:0 cid:23 cdw0:0 sqhd:0015 p:1 m:0 dnr:1 [2021-03-25 13:04:09.039687] nvme_qpair.c: 248:nvme_admin_qpair_print_command: NOTICE: ASYNC EVENT REQUEST (0c) qid:0 cid:21 nsid:0 cdw10:00000000 cdw11:00000000 [2021-03-25 13:04:09.039951] nvme_qpair.c: 451:spdk_nvme_print_completion: NOTICE: INVALID OPCODE (00/01) qid:0 cid:21 cdw0:0 sqhd:0016 p:1 m:0 dnr:1 [2021-03-25 13:04:09.040300] nvme_qpair.c: 248:nvme_admin_qpair_print_command: NOTICE: ASYNC EVENT REQUEST (0c) qid:0 cid:18 nsid:0 cdw10:00000000 cdw11:00000000 [2021-03-25 13:04:09.040562] nvme_qpair.c: 451:spdk_nvme_print_completion: NOTICE: INVALID OPCODE (00/01) qid:0 cid:18 cdw0:0 sqhd:0017 p:1 m:0 dnr:1 [2021-03-25 13:04:09.040886] nvme_qpair.c: 248:nvme_admin_qpair_print_command: NOTICE: ASYNC EVENT REQUEST (0c) qid:0 cid:20 nsid:0 cdw10:00000000 cdw11:00000000 [2021-03-25 13:04:09.041133] nvme_qpair.c: 451:spdk_nvme_print_completion: NOTICE: INVALID OPCODE (00/01) qid:0 cid:20 cdw0:0 sqhd:0018 p:1 m:0 dnr:1 [2021-03-25 13:04:09.041479] nvme_qpair.c: 248:nvme_admin_qpair_print_command: NOTICE: ASYNC EVENT REQUEST (0c) qid:0 cid:22 nsid:0 cdw10:00000000 cdw11:00000000 [2021-03-25 13:04:09.041717] nvme_qpair.c: 451:spdk_nvme_print_completion: NOTICE: INVALID OPCODE (00/01) qid:0 cid:22 cdw0:0 sqhd:001a p:1 m:0 dnr:1 [2021-03-25 13:04:09.042041] nvme_qpair.c: 248:nvme_admin_qpair_print_command: NOTICE: ASYNC EVENT REQUEST (0c) qid:0 cid:23 nsid:0 cdw10:00000000 cdw11:00000000 [2021-03-25 13:04:09.042308] nvme_qpair.c: 451:spdk_nvme_print_completion: NOTICE: INVALID OPCODE (00/01) qid:0 cid:23 cdw0:0 sqhd:001b p:1 m:0 dnr:1 [2021-03-25 13:04:09.042615] nvme_qpair.c: 248:nvme_admin_qpair_print_command: NOTICE: ASYNC EVENT REQUEST (0c) qid:0 cid:21 nsid:0 cdw10:00000000 cdw11:00000000 [2021-03-25 13:04:09.043190] nvme_qpair.c: 451:spdk_nvme_print_completion: NOTICE: INVALID OPCODE (00/01) qid:0 cid:21 cdw0:0 sqhd:001c p:1 m:0 dnr:1 [2021-03-25 13:04:09.043674] nvme_qpair.c: 248:nvme_admin_qpair_print_command: NOTICE: ASYNC EVENT REQUEST (0c) qid:0 cid:18 nsid:0 cdw10:00000000 cdw11:00000000 [2021-03-25 13:04:09.044009] nvme_qpair.c: 451:spdk_nvme_print_completion: NOTICE: INVALID OPCODE (00/01) qid:0 cid:18 cdw0:0 sqhd:001d p:1 m:0 dnr:1 [2021-03-25 13:04:09.044447] nvme_qpair.c: 248:nvme_admin_qpair_print_command: NOTICE: ASYNC EVENT REQUEST (0c) qid:0 cid:19 nsid:0 cdw10:00000000 cdw11:00000000 [2021-03-25 13:04:09.044707] nvme_qpair.c: 451:spdk_nvme_print_completion: NOTICE: INVALID OPCODE (00/01) qid:0 cid:19 cdw0:0 sqhd:001e p:1 m:0 dnr:1 [2021-03-25 13:04:09.045101] nvme_qpair.c: 248:nvme_admin_qpair_print_command: NOTICE: ASYNC EVENT REQUEST (0c) qid:0 cid:22 nsid:0 cdw10:00000000 cdw11:00000000 [2021-03-25 13:04:09.045459] nvme_qpair.c: 451:spdk_nvme_print_completion: NOTICE: INVALID OPCODE (00/01) qid:0 cid:22 cdw0:0 sqhd:001f p:1 m:0 dnr:1 [2021-03-25 13:04:09.045824] nvme_qpair.c: 248:nvme_admin_qpair_print_command: NOTICE: ASYNC EVENT REQUEST (0c) qid:0 cid:23 nsid:0 cdw10:00000000 cdw11:00000000 [2021-03-25 13:04:09.046152] nvme_qpair.c: 451:spdk_nvme_print_completion: NOTICE: INVALID OPCODE (00/01) qid:0 cid:23 cdw0:0 sqhd:0000 p:1 m:0 dnr:1 [2021-03-25 13:04:09.046521] nvme_qpair.c: 248:nvme_admin_qpair_print_command: NOTICE: ASYNC EVENT REQUEST (0c) qid:0 cid:20 nsid:0 cdw10:00000000 cdw11:00000000 [2021-03-25 13:04:09.047216] nvme_qpair.c: 451:spdk_nvme_print_completion: NOTICE: INVALID OPCODE (00/01) qid:0 cid:20 cdw0:0 sqhd:0002 p:0 m:0 dnr:1 [2021-03-25 13:04:09.047605] nvme_qpair.c: 248:nvme_admin_qpair_print_command: NOTICE: ASYNC EVENT REQUEST (0c) qid:0 cid:18 nsid:0 cdw10:00000000 cdw11:00000000 [2021-03-25 13:04:09.047971] nvme_qpair.c: 451:spdk_nvme_print_completion: NOTICE: INVALID OPCODE (00/01) qid:0 cid:18 cdw0:0 sqhd:0003 p:0 m:0 dnr:1 [2021-03-25 13:04:09.048353] nvme_qpair.c: 248:nvme_admin_qpair_print_command: NOTICE: ASYNC EVENT REQUEST (0c) qid:0 cid:19 nsid:0 cdw10:00000000 cdw11:00000000 [2021-03-25 13:04:09.048675] nvme_qpair.c: 451:spdk_nvme_print_completion: NOTICE: INVALID OPCODE (00/01) qid:0 cid:19 cdw0:0 sqhd:0004 p:0 m:0 dnr:1 INFO: using host memory buffer for IO Hello world! [2021-03-25 13:04:09.140279] nvme_qpair.c: 248:nvme_admin_qpair_print_command: NOTICE: ASYNC EVENT REQUEST (0c) qid:0 cid:22 nsid:0 cdw10:00000000 cdw11:00000000 [2021-03-25 13:04:09.140730] nvme_qpair.c: 451:spdk_nvme_print_completion: NOTICE: INVALID OPCODE (00/01) qid:0 cid:22 cdw0:0 sqhd:0005 p:0 m:0 dnr:1 [2021-03-25 13:04:09.141184] nvme_qpair.c: 248:nvme_admin_qpair_print_command: NOTICE: ASYNC EVENT REQUEST (0c) qid:0 cid:21 nsid:0 cdw10:00000000 cdw11:00000000 [2021-03-25 13:04:09.141581] nvme_qpair.c: 451:spdk_nvme_print_completion: NOTICE: INVALID OPCODE (00/01) qid:0 cid:21 cdw0:0 sqhd:0006 p:0 m:0 dnr:1 [2021-03-25 13:04:09.142053] nvme_qpair.c: 248:nvme_admin_qpair_print_command: NOTICE: ASYNC EVENT REQUEST (0c) qid:0 cid:20 nsid:0 cdw10:00000000 cdw11:00000000 [2021-03-25 13:04:09.142457] nvme_qpair.c: 451:spdk_nvme_print_completion: NOTICE: INVALID OPCODE (00/01) qid:0 cid:20 cdw0:0 sqhd:0007 p:0 m:0 dnr:1 [2021-03-25 13:04:09.143351] nvme_qpair.c: 248:nvme_admin_qpair_print_command: NOTICE: ASYNC EVENT REQUEST (0c) qid:0 cid:18 nsid:0 cdw10:00000000 cdw11:00000000 [2021-03-25 13:04:09.143745] nvme_qpair.c: 451:spdk_nvme_print_completion: NOTICE: INVALID OPCODE (00/01) qid:0 cid:18 cdw0:0 sqhd:0008 p:0 m:0 dnr:1 [2021-03-25 13:04:09.144233] nvme_qpair.c: 248:nvme_admin_qpair_print_command: NOTICE: ASYNC EVENT REQUEST (0c) qid:0 cid:23 nsid:0 cdw10:00000000 cdw11:00000000 [2021-03-25 13:04:09.144606] nvme_qpair.c: 451:spdk_nvme_print_completion: NOTICE: INVALID OPCODE (00/01) qid:0 cid:23 cdw0:0 sqhd:000a p:0 m:0 dnr:1 [2021-03-25 13:04:09.145091] nvme_qpair.c: 248:nvme_admin_qpair_print_command: NOTICE: ASYNC EVENT REQUEST (0c) qid:0 cid:22 nsid:0 cdw10:00000000 cdw11:00000000 [2021-03-25 13:04:09.145448] nvme_qpair.c: 451:spdk_nvme_print_completion: NOTICE: INVALID OPCODE (00/01) qid:0 cid:22 cdw0:0 sqhd:000b p:0 m:0 dnr:1 [2021-03-25 13:04:09.145962] nvme_qpair.c: 248:nvme_admin_qpair_print_command: NOTICE: ASYNC EVENT REQUEST (0c) qid:0 cid:21 nsid:0 cdw10:00000000 cdw11:00000000 [2021-03-25 13:04:09.147189] nvme_qpair.c: 451:spdk_nvme_print_completion: NOTICE: INVALID OPCODE (00/01) qid:0 cid:21 cdw0:0 sqhd:000c p:0 m:0 dnr:1 [2021-03-25 13:04:09.147732] nvme_qpair.c: 248:nvme_admin_qpair_print_command: NOTICE: ASYNC EVENT REQUEST (0c) qid:0 cid:20 nsid:0 cdw10:00000000 cdw11:00000000 [2021-03-25 13:04:09.148072] nvme_qpair.c: 451:spdk_nvme_print_completion: NOTICE: INVALID OPCODE (00/01) qid:0 cid:20 cdw0:0 sqhd:000d p:0 m:0 dnr:1 [2021-03-25 13:04:09.148575] nvme_qpair.c: 248:nvme_admin_qpair_print_command: NOTICE: ASYNC EVENT REQUEST (0c) qid:0 cid:19 nsid:0 cdw10:00000000 cdw11:00000000 [2021-03-25 13:04:09.148903] nvme_qpair.c: 451:spdk_nvme_print_completion: NOTICE: INVALID OPCODE (00/01) qid:0 cid:19 cdw0:0 sqhd:000e p:0 m:0 dnr:1 [2021-03-25 13:04:09.149409] nvme_qpair.c: 248:nvme_admin_qpair_print_command: NOTICE: ASYNC EVENT REQUEST (0c) qid:0 cid:23 nsid:0 cdw10:00000000 cdw11:00000000 [2021-03-25 13:04:09.149807] nvme_qpair.c: 451:spdk_nvme_print_completion: NOTICE: INVALID OPCODE (00/01) qid:0 cid:23 cdw0:0 sqhd:000f p:0 m:0 dnr:1 [2021-03-25 13:04:09.150298] nvme_qpair.c: 248:nvme_admin_qpair_print_command: NOTICE: ASYNC EVENT REQUEST (0c) qid:0 cid:22 nsid:0 cdw10:00000000 cdw11:00000000 [2021-03-25 13:04:09.150657] nvme_qpair.c: 451:spdk_nvme_print_completion: NOTICE: INVALID OPCODE (00/01) qid:0 cid:22 cdw0:0 sqhd:0010 p:0 m:0 dnr:1 [2021-03-25 13:04:09.151147] nvme_qpair.c: 248:nvme_admin_qpair_print_command: NOTICE: ASYNC EVENT REQUEST (0c) qid:0 cid:18 nsid:0 cdw10:00000000 cdw11:00000000 [2021-03-25 13:04:09.151890] nvme_qpair.c: 451:spdk_nvme_print_completion: NOTICE: INVALID OPCODE (00/01) qid:0 cid:18 cdw0:0 sqhd:0012 p:0 m:0 dnr:1 [2021-03-25 13:04:09.152355] nvme_qpair.c: 248:nvme_admin_qpair_print_command: NOTICE: ASYNC EVENT REQUEST (0c) qid:0 cid:20 nsid:0 cdw10:00000000 cdw11:00000000 [2021-03-25 13:04:09.152751] nvme_qpair.c: 451:spdk_nvme_print_completion: NOTICE: INVALID OPCODE (00/01) qid:0 cid:20 cdw0:0 sqhd:0013 p:0 m:0 dnr:1 [2021-03-25 13:04:09.153223] nvme_qpair.c: 248:nvme_admin_qpair_print_command: NOTICE: ASYNC EVENT REQUEST (0c) qid:0 cid:19 nsid:0 cdw10:00000000 cdw11:00000000 [2021-03-25 13:04:09.234478] nvme_qpair.c: 451:spdk_nvme_print_completion: NOTICE: INVALID OPCODE (00/01) qid:0 cid:19 cdw0:0 sqhd:0014 p:0 m:0 dnr:1 [2021-03-25 13:04:09.235283] nvme_qpair.c: 248:nvme_admin_qpair_print_command: NOTICE: ASYNC EVENT REQUEST (0c) qid:0 cid:23 nsid:0 cdw10:00000000 cdw11:00000000 [2021-03-25 13:04:09.235321] nvme_qpair.c: 451:spdk_nvme_print_completion: NOTICE: INVALID OPCODE (00/01) qid:0 cid:23 cdw0:0 sqhd:0015 p:0 m:0 dnr:1 [2021-03-25 13:04:09.235345] nvme_qpair.c: 248:nvme_admin_qpair_print_command: NOTICE: ASYNC EVENT REQUEST (0c) qid:0 cid:21 nsid:0 cdw10:00000000 cdw11:00000000 [2021-03-25 13:04:09.235366] nvme_qpair.c: 451:spdk_nvme_print_completion: NOTICE: INVALID OPCODE (00/01) qid:0 cid:21 cdw0:0 sqhd:0016 p:0 m:0 dnr:1 [2021-03-25 13:04:09.235388] nvme_qpair.c: 248:nvme_admin_qpair_print_command: NOTICE: ASYNC EVENT REQUEST (0c) qid:0 cid:18 nsid:0 cdw10:00000000 cdw11:00000000 [2021-03-25 13:04:09.235410] nvme_qpair.c: 451:spdk_nvme_print_completion: NOTICE: INVALID OPCODE (00/01) qid:0 cid:18 cdw0:0 sqhd:0017 p:0 m:0 dnr:1 [2021-03-25 13:04:09.235432] nvme_qpair.c: 248:nvme_admin_qpair_print_command: NOTICE: ASYNC EVENT REQUEST (0c) qid:0 cid:20 nsid:0 cdw10:00000000 cdw11:00000000 [2021-03-25 13:04:09.235452] nvme_qpair.c: 451:spdk_nvme_print_completion: NOTICE: INVALID OPCODE (00/01) qid:0 cid:20 cdw0:0 sqhd:0018 p:0 m:0 dnr:1

-QEMU Behavior

qemu@qemu:~/MyFile/spdk$ sudo build/examples/hello_world [2021-03-25 21:10:53.785401] Starting SPDK v21.01-pre git sha1 7f6afb7bc / DPDK 20.08.0 initialization... [2021-03-25 21:10:53.785582] [ DPDK EAL parameters: [2021-03-25 21:10:53.785626] hello_world [2021-03-25 21:10:53.785654] -c 0x1 [2021-03-25 21:10:53.785687] --log-level=lib.eal:6 [2021-03-25 21:10:53.785708] --log-level=lib.cryptodev:5 [2021-03-25 21:10:53.] EAL: No available hugepages reported in hugepages-1048576kB EAL: No legacy callbacks, legacy socket not created Initializing NVMe Controllers Attaching to 0000:00:03.0 Attached to 0000:00:03.0 Using controller QEMU NVMe Ctrl (deadbeef ) with 1 namespaces. Namespace ID: 1 size: 51GB Initialization complete. INFO: using host memory buffer for IO Hello world!

lovers-under-moon avatar Mar 25 '21 13:03 lovers-under-moon

Thanks for reporting this. I will take a look later (need a while).

huaicheng avatar Mar 27 '21 00:03 huaicheng

Thanks for reporting this. I will take a look later (need a while).

Thank you

lovers-under-moon avatar Mar 27 '21 14:03 lovers-under-moon

The qemu I used is https://github.com/spdk/qemu/tree/spdk-5.0.0

lovers-under-moon avatar Mar 29 '21 02:03 lovers-under-moon

Thanks for reporting this. I will take a look later (need a while).

Installing, using and debugging FEMU can cause side effects including headache, nausea, agitation, and depression. If your research condition does not improve after using FEMU for a week, please talk to your advisor or us right away.

Haha,I think I have headache symptoms now.I don’t know where is the bug .

lovers-under-moon avatar Apr 09 '21 07:04 lovers-under-moon

SPDK use NVMe read/write (0x01, 0x02) Commands for OCSSD read and write,but FEMU OCSSD2.0 do not support NVMe read/write (0x01, 0x02) Commands.FEMU support Vector Chunk Write/Read( (0x91, 0x92)).You can modify SPDK's read and write interface or add NVMe read/write (0x01, 0x02) function to FEMU.

lovers-under-moon avatar Apr 20 '21 04:04 lovers-under-moon

Thanks for the updates.

The OCSSD standard specifies its special physical read/write/erase commands for interacting with the device, not sure why SPDK deviates from this spec. Nice finding though. It should be straightforward to support the spdk quirks. Unfortunately, I won't have time to work on this soon. Interested in submitting a patch? I would be happy to merge it.

Thanks.

On Mon, Apr 19, 2021 at 11:04 PM Denghui Wu @.***> wrote:

SPDK use NVMe read/write (0x01, 0x02) Commands for OCSSD read and write,but FEMU OCSSD2.0 do not support NVMe read/write (0x01, 0x02) Commands.FEMU support Vector Chunk Write/Read( (0x91, 0x92)).You can modify SPDK's read and write interface or add vector chunk Write/Read function to FEMU.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/ucare-uchicago/FEMU/issues/48#issuecomment-822957134, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAXGVAZPFJ57EUX6PZ6QWADTJT4ORANCNFSM4ZZKUZHA .

huaicheng avatar Apr 20 '21 16:04 huaicheng

With the above patch, the SPDK FTL "hello world" test runs fine on FEMU.

huaicheng avatar May 12 '21 03:05 huaicheng

With the above patch, the SPDK FTL "hello world" test runs fine on FEMU.

Thank you very much Now the "hello world" test can run successfully.But I found another problem when I mounted SPDK FTL.FEMU does not support "Asynchronous Event Request command" (NVMe admin command opcode=0x0C) and "Log Page Identifier Chunk Notification"(lid=0xD0). SPDK needs the support of these two admin commands to successfully mount user-mode FTL.

NVMe 1.3 specification. image

OCSSD2.0 specification image

Example of spdk mounting FTL 1.Run FEMU OCSSD mode: ./run-whitebox.sh

2.Build SPDK git clone https://github.com/spdk/spdk -b v20.10 cd spdk git submodule update --init sudo scripts/pkgdep.sh ./configure make

3.Mount FTL sudo HUGEMEM=5120 scripts/setup.sh sudo ./build/bin/nvmf_tgt

Then open another terminal window scripts/rpc.py bdev_nvme_attach_controller -b nvme0 -a 00:05.0 -t pcie scripts/rpc.py bdev_ocssd_create -c nvme0 -b nvme0n1 (Failed to run this) scripts/rpc.py bdev_ftl_create -b ftl0 -d nvme0n1

image

lovers-under-moon avatar May 15 '21 14:05 lovers-under-moon

Thanks for testing it! Will take a look later.

huaicheng avatar May 17 '21 20:05 huaicheng