question about model

[EcustBoy]

Hi~ author, it's a great job~ and I have three questions~ (1) I haven't found the detailed network structure python script in your code, I wanna ask have you open source this file? I'll thanks for your reply~

(2) I notice interfuser also learn traffic light state, does it only consider the one traffic light appeared in front-view image? If so, is it possible that the model output error results if two traffic lights appear in front-view? (e.g. when ego_car's yaw is not parallel to lane when running close to junction area, it may see not only one traffic light, right?)

(3) How much will the traffic light detection impact on the final planning metric? In my opinion, transfuser first need to imitate expert trajectory by valid imitation training(i.e. equip implicitly understanding ability to traffic light, see green -> continue running, see red -> tends to slow down), then through explicit traffic light detection result, the safety will be further improved, right? Have you ever done an ablation experiment on traffic light detection? (i.e. only rely on implicitly imitation training, while remove the "detect red light --> brake" logical behavior)

[deepcs233]

Hi! (1) It can be found in https://github.com/opendilab/InterFuser/blob/main/interfuser/timm/models/interfuser.py (2) No, the traffic light predictor can utilize the feature from all sensors in our framework. (3) The traffic light detection has a very big impact on the final result. It will multiply 0.6 into the final score if the agent run the red light. Our framework is not based on imitation learning, so we haven't done similiar ablation experiments.

[EcustBoy]

thanks for your reply ~ about question (3), the "imitation learning" in my word means the model need expert traj as label~ and I think interfuser is done like this(the additional safety controller can be seen as a post-process step I think)

And I just found the "is_junction" flag is also learned in your model, I think "is_junction" flag and traffic light state info must be used together, right?

Beside, imaging such a scenario case (although with low probability)：ego car is runing close to junction and the traffic light is green, but the model outputs the wrong behavior to slow down and brake, can interfuser model give ego car the throttle to continue accelarating? (I seem to only find the slow down behavior given by safety controller for collision avoidance in the code, but I'm not sure about such scenario case)

[deepcs233]

Hi (1) Interfuser doesn't imitate the expert traj, it only predictes the driving route. (2) Yes, if "Is_junction" is True, we will ignore the output of the traffic light predictor. (3) According to response(1), our method only output the driving route (may be dangerous). So if the light is green there are no other dangerous situations,the ego car will not slow down.

[EcustBoy]

Hi (1) Interfuser doesn't imitate the expert traj, it only predictes the driving route. (2) Yes, if "Is_junction" is True, we will ignore the output of the traffic light predictor. (3) According to response(1), our method only output the driving route (may be dangerous). So if the light is green there are no other dangerous situations,the ego car will not slow down.

Thanks for your reply~ About answer (2), according to code, I think the output of the traffic light predictor is considered only when "Is_junction" is True, right?

[deepcs233]

In our released code, we ignore the output of the "junction" (because junction is alwayas greater than 0 (after softmax)). The reaction to the traffic light is depended on the definiation of the collected dataset.

For example, if you define that only when the traffic light is red and it influences the ego car, the collected light label is red. in this situtation, you can ignore the output of the "junction". But if you define that when the traffic light is red, the collected light label is red. You need to take consideration of the "junction" output, because if the ego car is in the junction, the red light needs to be ignored.

The output of "junction" can also be used when the ego car encounters the stop sign, please refer to https://github.com/opendilab/InterFuser/blob/e0682c350892a243cf40bf448622743f4b26d0f3/leaderboard/team_code/interfuser_controller.py#L126