IOU and vision_for_prior issues
Hey mate:) It seems like nms_ios is not working very well. I tried on small objects. I mean, when I predict something, even there are two predictions with the same class close, it is not "concatenate" like the usual IOU way. It displays like before nms ios(link)
One more question. What is the purpose of vision_for_prior?

This is an example of my dataset.My images are 200200. I know that will be a lot of loss when it scales to 300300

Your effect doesn't look good. You need to train to 100 generations according to the default parameters first. I see that the confidence in the pictures you sent is all 0.2. I think the network seems to have no ability to distinguish. What is the situation and quantity of your data set, and whether it is useful for pre training weight? I used NMS. You should not be a problem of NMS, but a problem of no good training.
I trained the model for 210 generations and still has low accuracy and ~2.8 loss. I changed the confidence value at prediction to 0.2. In training, I use the defaults. So I have 7200 images.
Sorry, I think maybe you need to try other models, because this effect seems that the network has no discrimination ability. Yolo network has stronger discrimination ability, and the training effect may be better,Because I don't know your dataset, I can't make a good judgment in the next step