The yolo series model does not get weights information via onnx.model.graph.input

[wangxudong-cq]

Bug Report

Which model does this pertain to?

https://github.com/onnx/models/blob/master/vision/object_detection_segmentation/yolov3/model/yolov3-10.onnx https://github.com/onnx/models/blob/master/vision/object_detection_segmentation/yolov4/model/yolov4.onnx

Describe the bug

Describe the problem with the model and the results you are seeing.

Reproduction instructions

OS Platform and Distribution (e.g. Linux Ubuntu 16.04):
Linux VM-1-159-ubuntu 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

ONNX version (e.g. 1.6):
Name: onnx Version: 1.10.1 Summary: Open Neural Network Exchange Home-page: https://github.com/onnx/onnx Author: ONNX Author-email: [email protected] License: Apache License v2.0 Location: /usr/local/anaconda3/envs/ubuntu/lib/python3.8/site-packages Requires: numpy, typing-extensions, protobuf, six Required-by:

Provide a code snippet to reproduce your errors.

import onnx
model = onnx.load('yolov3-10.onnx')
print(model.graph.input)

Output

[name: "input_1" type { tensor_type { elem_type: 1 shape { dim { dim_param: "unk__576" } dim { dim_value: 3 } dim { dim_param: "unk__577" } dim { dim_param: "unk__578" } } } } , name: "image_shape" type { tensor_type { elem_type: 1 shape { dim { dim_param: "unk__579" } dim { dim_value: 2 } } } } ]

Notes

The input array in GraphProto contains not only the node that we generally understand as the image input, but also all the weights in the model

[jcwchen]

Hi @wangxudong-cq, Have you checked whether the weights are stored in model.graph.initializer? Do you bump into any issue because they are not in the graph's input?

[wangxudong-cq]

Hi @wangxudong-cq, Have you checked whether the weights are stored in model.graph.initializer? Do you bump into any issue because they are not in the graph's input?

Maybe I'm not clear enough. I know that weights are stored in model.graph.initializer, but i meanweight-related information not in the graph's input, such as name and type.

[jcwchen]

Thank you for the clarification. Actually only models whose IR_VERSION < 4 requires graph.initializer be included in graph.input. For models whose IR_VERSION >=4 (like these two models), their initializer can not be included in graph.input. Therefore I was wondering what error did you bump into if the initializer does not exist in graph.input.

[wangxudong-cq]

Thank you for the clarification. Actually only models whose IR_VERSION < 4 requires graph.initializer be included in graph.input. For models whose IR_VERSION >=4 (like these two models), their initializer can not be included in graph.input. Therefore I was wondering what error did you bump into if the initializer does not exist in graph.input.

Thanks for your quick response. I have a scenario where i need to split the model into operator-level sub-models, so i need the input and output and weight information of each operator, thus I would like to get the relevant content through graph.input information.

[jcwchen]

Thank you for providing the details. I would say there are a lot of ONNX models (IR >= 4) missing input as initializer since IR does allow it. They are valid ONNX models so ideally you should make your code can obtain those information from initializer as well. Still, as a quick workaround, perhaps you can try something like:

def add_input_from_initializer(model : onnx.ModelProto):
    """
    Currently onnx.shape_inference doesn't use the shape of initializers, so add
    that info explicitly as ValueInfoProtos.
    Mutates the model.
    Args:
        model: The ModelProto to update.
    """
    # All (top-level) constants will have ValueInfos before IRv4 as they are all inputs
    if model.ir_version < 4:
        return

    def add_const_value_infos_to_graph(graph : onnx.GraphProto):
        inputs = {i.name for i in graph.input}
        existing_info = {vi.name: vi for vi in graph.input}
        for init in graph.initializer:
            # Check it really is a constant, not an input
            if init.name in inputs:
                continue

            # The details we want to add
            elem_type = init.data_type
            shape = init.dims

            # Get existing or create new value info for this constant
            vi = existing_info.get(init.name)
            if vi is None:
                vi = graph.input.add()
                vi.name = init.name

            # Even though it would be weird, we will not overwrite info even if it doesn't match
            tt = vi.type.tensor_type
            if tt.elem_type == onnx.TensorProto.UNDEFINED:
                tt.elem_type = elem_type
            if not tt.HasField("shape"):
                # Ensure we set an empty list if the const is scalar (zero dims)
                tt.shape.dim.extend([])
                for dim in shape:
                    tt.shape.dim.add().dim_value = dim

        # Handle subgraphs
        for node in graph.node:
            for attr in node.attribute:
                # Ref attrs refer to other attrs, so we don't need to do anything
                if attr.ref_attr_name != "":
                    continue

                if attr.type == onnx.AttributeProto.GRAPH:
                    add_const_value_infos_to_graph(attr.g)
                if attr.type == onnx.AttributeProto.GRAPHS:
                    for g in attr.graphs:
                        add_const_value_infos_to_graph(g)


    return add_const_value_infos_to_graph(model.graph)
add_input_from_initializer(original_model)

This script add_input_from_initializer can help the model to add inputs from all initializers.

[wangxudong-cq]

for node in graph.node:
            for attr in node.attribute:
                # Ref attrs refer to other attrs, so we don't need to do anything
                if attr.ref_attr_name != "":
                    continue

                if attr.type == onnx.AttributeProto.GRAPH:
                    add_const_value_infos_to_graph(attr.g)
                if attr.type == onnx.AttributeProto.GRAPHS:
                    for g in attr.graphs:
                        add_const_value_infos_to_graph(g)

Refer to this function about subgraph, i wonder how to deal with loop operator which contains more attributes

[jcwchen]

Could you please elaborate more on the problem? Can't the traverse of each attribute and recursive here handle that case? Thanks.