Instance segmentation misses skeletal meshes inside Blueprint (e.g. BP_CrowdCharacter)

[ssaigarimella]

When using Cosys-AirSim’s instance segmentation in UE 5.2.1 on Ubuntu 22.04, any skeletal mesh nested inside a CapsuleComponent in a Blueprint (such as BP_CrowdCharacter shows in the screenshots) does not show up in the segmentation render target. Even after manually assigning segmentation IDs via the Python script below post-spawn, those actors are not rendered anywhere in the instance segmentation frame.

Code I used to set object segmentation IDs after spawn:

#!/usr/bin/env python3

import setup_path
import cosysairsim as airsim
import numpy as np         
import cv2                 

client = airsim.VehicleClient()
client.confirmConnection()

# 1 – see exactly what Unreal called the characters
crowd = client.simListSceneObjects("BP_CrowdCharacter.*")
print("Found:", crowd)

# 2 – (only if set InitialInstanceSegmentation = false)
for obj in crowd:
    client.simSetSegmentationObjectID(obj, 200, True)

# 3 – grab one segmentation frame (PNG bytes)
seg_png = client.simGetImage("front_center", airsim.ImageType.Segmentation)
assert seg_png, "Seg frame came back empty -> something above failed"

# 4 – decode and show it
img = cv2.imdecode(np.frombuffer(seg_png, np.uint8), cv2.IMREAD_UNCHANGED)
cv2.imshow("Instance Segmentation", cv2.cvtColor(img, cv2.COLOR_RGB2BGR))
cv2.waitKey(0)
cv2.destroyAllWindows()

I looked at issue #43 but for me it does not work even in Blocks. How do I get the correct colored instance segmentation frame with the BP_CrowdCharacters rendered with unique colors?

[ssaigarimella]

Related to issue #43, how do I force update the instance segmentation to recognize the newly added skeletal meshes after the characters have spawned?

[WouterJansen]

Hi, are these skeletal meshes created after begin play? In that case new actors and components need to be added through the SimModeBase API (blueprint or C++) so that the annotation system knows about it. This also counts for instance segmentation:

So as in this example blueprint code, get the SimModeBase object (the simulation API), and add a new actor to the instance segmentation system (reference to self is the actor to be added) using AddNewActorToInstanceSegmentation.

Another thing that is required is 'update annotation'. The is required to fully update all annotation cameras and LiDAR sensors (including instance segmentation) to get a forced update of all new objects to render. As this is a bit slow you can toggle this off like I did in the second execution of the AddNewActorToInstanceSegmentation function. This is handy when spawning many actors in blueprint in a loop and then only once all are spawned at the end run the ForceUpdateInstanceSegmentation function once.

https://blueprintue.com/blueprint/66djim33/

[ssaigarimella]

@WouterJansen Thanks for your reply, is this supposed to work even with the SkeletalMesh nested inside a CapsuleComponent in a Blueprint? The Blueprint instances which consist of the skeletal mesh are placed in the environment prior to hitting play -- so they are not created after begin play. The same issue persists. Just to confirm, how do I use the blueprint exactly?

[Kokika]

@ssaigarimella As I understand right does your character contains multiple mesh? The segmentation c++ code not fully implemented for multiple mesh for skeletal actor. If yes, I know the problem and I can make fix for it.

[ssaigarimella]

@Kokika It's a skeletal mesh with multiple meshes, yes. Thank you