Sim - Support Visualization of Photon's World

[gerth2]

Currently, simulated photon cameras have a lot of configurable options, and are fairly hard to debug. This issue serves as a working design document for how to properly visualize a simulated vision system.

Regardless of the underlying display technology, the display should indicate enough info for users to visually interpret what their simulation is doing. Here's an idea of what that could look like:

The top portion simulates what a camera with a very large FOV could see (along with the actual simulated camera FOV). In-view targets are shown with centroid markers to indicate where they were being seen at.

The bottom portion shows a "top down" view of where the camera is at on the field, relative to each simulated target, and what it's FOV can see.

The simulation currently does not account for target type or shape, it assumes the user simply inputs the centroid. This could be added, or the visualization could be modified to not imply some specific method of determining target coordinates.

In all cases, the visualization should update in real time with the simulation.

Looking for any and all feedback!

[gerth2]

Options for how to make Photonlib display something like that GUI above:

1. Simulated camera hosts a website (localhost:1234 or whatever) a) Cross platform viewing, but still need language-specific implementation for serving the webpage, or move to JNI
2. ImGUI or some other cross-platform setup. a) JNI possibly required?
3. Use wpilib constructs a) Top down view could potentially be done on the Field2D widget, if enough functionality is added to specify the size/shape/color of things which appear on it. b) Camera-perspective view could be done by strongly abusing a Mechanism2D widget in Glass. And by strongly abusing, I mean doing all the math in photonsim to project a 3d world into 2d, then transmitting that 2d visualization into a "mechanism" of disjointed ligaments.
4. Java Swing (built-in GUI) b) C++ = ???

[mcm001]

Option 3 seems doable assuming Mechanism2d is implemented If not, drawing on a webpage canvas seems reasonable in terms of cross-platform-ness

[mcm001]

Is this issue still applicable now that SimVisionSystem is a thing?

[gerth2]

I think so still, even with the vision system it's hard to debug which pose or transform is incorrect.

On Sat, Oct 2, 2021, 9:06 AM Matt @.***> wrote:

Is this issue still applicable now that SimVisionSystem is a thing?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/PhotonVision/photonvision/issues/275#issuecomment-932757787, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABC7B3SZ7Y4IRLXFEHUY4B3UE4GWRANCNFSM5BU24IDQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

[modelmat]

Earlier in January I worked on this as a demo project - https://github.com/modelmat/cameraviewdemoproject

It uses a slightly modified version of Mechanism2D inside of glass to draw the 2d projections (using the specific camera matrices) of 3d shapes inside of glass. You will need to compile the linked wpilib branch for it to work (I don't know why I called it sysid-generic-plot); it has few changes to the underlying Mechanism2D behaviour, but was intended to allow for a better API (that never happened)

The cameraview project contains code which does the 3D point projection, this would ideally be moved into a library, but again, demo project that I never completed.

https://user-images.githubusercontent.com/25718989/159191967-522a0a7e-864c-4d2c-a5e4-16bce793edb1.mp4

current issues:

• when points exist at the corners of the render view, if they go out of the rendered camera cv::projectPoints returns (0,0) and so you get lines going across the screen (possible resolution: detecting if this is the case and not rendering the line (could look weird if the line should be rendered)
• switch glass rendered to a list of [x, y, x, y, x, y] instead of the awful nested tree it is. Optional allowed fillable areas.
• If a line is very long (e.g. a field border) it doesn't form a curve, it just gets rendered as a straight line. solution: break up large distances into smaller lines internally
• no integration with photon actual sim code

[mdurrani808]

@gerth2 is this relevant with the new apriltag visualization updates?

[amquake]

I'm going to close this as (at least largely) completed with the recent simulation overhaul and visualization options.

Features described in this post that differ from the current implementation:

• GUI
  • The simulated camera view is currently drawn to a Mat using OpenCV, which allows it to be published through cscore and seen from shuffleboard/browser similarly to an actual camera stream from a coprocessor.
  • Instead of a custom top-down view, simply a Field2d is used to draw the robot, camera, and target poses. This does not allow a top-down visualization of camera field-of-vision.
• Camera view
  • The current camera view simulation is not comprehensive, and cannot simulate targets partially outside of the field-of-vision. It also struggles with barrel distortion and distorted tag images. Something like described here, where the extended camera view pictures the simulated world and is cropped down to the actual view, may be possible to address this.