runtime icon indicating copy to clipboard operation
runtime copied to clipboard
verified publisher icon tensorflow

ResNet50 test does not verify the result

Open hidefromkgb opened this issue 5 years ago • 10 comments

Seems like the only check the ResNet50 test has is this one: // CHECK: int1 = 1 How should I amend it so that it could actually verify TFRT for RN50 conformance? Am I missing something?

hidefromkgb avatar Sep 30 '20 19:09 hidefromkgb

In my build I extended DenseHostTensor value dump limit from 32 to SSIZE_MAX, but what the network returns does not look normal:

$ bazel-bin/tools/tfrt_translate --mlir-to-bef integrationtest/resnet/resnet50_graph_inference.mlir -o=integrationtest/resnet/resnet50_graph_inference.bef
$ bazel-bin/tools/bef_executor --tracing_level=verbose --work_queue_type=s integrationtest/resnet/resnet50_graph_inference.bef
Choosing memory leak check allocator.
Choosing single-threaded work queue.
--- Not running '' because it has arguments.
--- Running 'resnet50':
DenseHostTensor dtype = F32, shape = [1, 1001], values = [5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16, 5.899087e+16]
int1 = 1
BM:resnet50_benchmark:Duration(ns): 544777172
BM:resnet50_benchmark:Count: 1
BM:resnet50_benchmark:Time Min(ns): 544777172
BM:resnet50_benchmark:Time 50%(ns): 544777172
BM:resnet50_benchmark:Time 95%(ns): 544777172
BM:resnet50_benchmark:Time 99%(ns): 544777172
BM:resnet50_benchmark:CPU Min(ns): 543831000
BM:resnet50_benchmark:CPU 50%(ns): 543831000
BM:resnet50_benchmark:CPU 95%(ns): 543831000
BM:resnet50_benchmark:CPU 99%(ns): 543831000
BM:resnet50_benchmark:CPU utilization(percent): 9.982632e+01
$ _

hidefromkgb avatar Sep 30 '20 20:09 hidefromkgb

Hi @hidefromkgb, the ResNet50 test runs ResNet50 inference on CPU for one step and checks that its last layer (i.e. softmax layer) generates the expected numerical output as the corresponding TF model. So it does verify the result to some extend, right?

Could you explain what verification you are looking for?

Also note that that test is not expected to be run as a benchmark. We might write it differently if it is going to be run as a benchmark, for example, making sure that some variable initialization is done only once and it only does the necessary operation will be done as part of the production ResNet50 inference job.

lindong28 avatar Oct 02 '20 22:10 lindong28

@lindong28 In an ideal scenario I`d like to be able to upload my own 224×224×BGR images and get back the recognition percentages from SoftMax. Right now (on some implicit default image?) all its 1001 values expected to be in [0; 1) range equal 58990869004419072.f, or 0x5b5193e2 in hex, on a default CPU backend.
Is there a problem with my testing rig? What are the actual values it has to output?

Off-topic: I`m interested because I`m writing a custom non-CPU backend for TFRT, excited that I don`t need to rewrite TF anymore to accomplish that. Got many questions regarding the process, and it would be way cool if, as I progress, I could ask these questions somewhere.

[UPD:] Based on how SoftMax works it seems like the error I see comes either directly from the implementation of SoftMax in TFRT or from the dumper — for no matter how large the results are prior to SoftMax, at the exit they should be normalized to [0; 1) by definition.

hidefromkgb avatar Oct 03 '20 05:10 hidefromkgb

@hidefromkgb Yeah your observation is correct. The resnet50_graph_inference.mlir indeed does not include the softmax layer, whereas the ResNet50 inference in production will include the softmax layer as the last layer.

The softmax layer is intentionally removed from this test to make this test "stricter". The rational is that, softmax layer might map different numerical input values to the same numerical output values. For example, softmax layer will map both (2, 2, 2, 2) and (3, 3, 3, 3) to (0.25, 0.25, 0.25, 0.25). Also, for an input vector where one value is much larger than other values, softmax tends to map most values to 0.0 and map the largest value to 1.0 in its output. Since the test simply compares the numerical output value between TF and TFRT, it seems a bit "stricter" to compare the input of the softmax layer rather than its output.

Now that this test is exported to OSS TFRT and users might expect to see that the test really implements all layers of the ResNet50 inference job, I think it is better to include the softmax layer in this test. I will update this test later.

lindong28 avatar Oct 05 '20 15:10 lindong28

@lindong28 So, regardless of the presence of SoftMax, is this output value of 58990869004419072.f across all 1001 recognition classes the expected result? What is the input image I could use to reproduce this behaviour on RN50 in e.g. PyTorch or OpenVINO?

Or, more importantly, how can I substitute the input image to make the result more meaningful?

hidefromkgb avatar Oct 05 '20 15:10 hidefromkgb

@hidefromkgb I might have missed some knowledge here. Can you explain where does that 58990869004419072.f come from?

The MLIR test takes an input tensor, applies the ResNet50 inference pipeline (excluding the SotMax layer) to this input, and verifies that the output has the same numerical value as the TF's output. I suppose you have got that 58990869004419072.f after giving this pipeline a different input. Whether or not this is expected really depends on what is the TF's output for this input.

Note that we are still actively developing TFRT infra and there are a lot of feature development under progress. TFRT might not be ready to address the particular use-case you are looking for.

lindong28 avatar Oct 05 '20 16:10 lindong28

@lindong28 I did everything according to this instruction, with one notable exception: I don`t execute the test via Bazel but run both the conversion app and the inference app directly — so that later I could execute them under CGDB. I did run bazel test <…rest_of_the_command…> the first time though, to check the test result, and it passed.
My first comment in this ticket includes the exact commands I use and the output they produce.

This ungodly value is repeated 1001 times in the resulting tensor that I get printed after executing the newly converted .BEF file containing RN50 — instead of the recognition probabilities RN50 should output for each of the 1001 classes it is able to recognize. Again, it`s important to note that I don`t give it any input images, and maybe that`s why it complains about some anonymous BEF function requiring arguments it cannot find.

Am I doing something wrong? Seems like I must be — but, unfortunately, I don`t yet understand what exactly.

[UPD:] Provided that the prerequisites from the instruction I linked are met, are the commands that I use expected to produce correct results, or should I amend them somehow to see meaningful recognition probabilities?

hidefromkgb avatar Oct 05 '20 16:10 hidefromkgb

@hidefromkgb Sorry, I didn't realize you were talking about the 5.899087e+16 as shown in your first comment.

I just re-run the corresponding TF model without the softmax layer and it does produce the output where that value is repeated 1001 times. And if I add the softmax layer to the model, then the 286th value will be 1.0 and all other values would be 0.0, which would look reasonable.

You are running the commands correctly and the test is also verifying that the result is the same as TF. The output looks unintuitive because the softmax layer is removed from the model.

In order to see meaningful recognition probabilities, we would need to add the SoftMax layer to the model. TFRT repo currently does not have a good SoftMax kernel that can be used in this test. I will try to add one and see if it address the issue. I will let you know later.

lindong28 avatar Oct 05 '20 17:10 lindong28

@lindong28 Thanks, will be looking forward to that!

hidefromkgb avatar Oct 05 '20 17:10 hidefromkgb

@lindong28 And, since I`m developing a custom backend, could you please share the version of RN50 with SoftMax enabled in MLIR?
It`ll allow me both to see how TFRT reacts to a missing primitive and to try and implement it using the fallback mechanism (my backend doing SoftMax whereas the CPU serves as a fallback; according to my current understanding of the TFRT arch this should work).

Checks failure is not an issue for me, for I now know the source image contains a cat, maybe even the same one we and everyone else use for smoke testing — so I`ll be looking at the printed output only.

hidefromkgb avatar Oct 05 '20 17:10 hidefromkgb