Edit mode only suspends app `useFrame`/`useUpdate` calls

[krispya]

Edit mode is meant to have the app without any loops executing so that objects can be interacted with. The current implementation of this replaces useFrame and useUpdate with an editor version that lets us suspend the callback, but only for calls that originate in the user app -- not from libraries. Consequently, any library interaction or even pointer events from R3F still occur in edit mode.

This can be seen in the ClickAndHover R3F example.

Proposed solution

Instead, I think we should suspend the R3F loop by setting it to never and then running our own loop for rendering while in edit mode. When we exit edit mode, we suspend our loop and let the app execute as it normally would. The API for doing this would be the same between v8 and v9 but v9 would allow more options, for example suspending only certain stages for the purpose of debug.

[itsdouges]

So setting the top level Canvas used during edit mode to frameloop="never" and then implementing an alternate useFrame that would run during edit for some things? What about use cases such as wanting to preview an animation, or working on a shader where you'd want some code executing every frame so you can effectively work on it without needing to jump into play mode, it'd be a bit more nuanced as to what the best solution is. Third party modules make it harder but we can always transform that code as well so I wouldn't say it's off the table.

[krispya]

Maybe it'll help to do a breakdown of the bigger idea. I am modeling how I approach the editing system off of the Unreal and Unity editor, but mostly Unreal. There are three distinct actions that are used to control the editor states: play, stop and pause. The editor begins as if stopped so I'll start there.

• Stop: When stopped, the editor is in a typical edit mode. The app has no systems running at all. It is as if it is completely static and never initialized. This is where you fly around and place the static or start state of objects, volumes, triggers, lights, etc. We run our own editor loop for rendering and anything that needs to be active while editing goes into this loop too such as any camera controls.
• Play: The app starts running, We let the R3F loop do its thing and pause the editor loop. Here you can test whatever you want in the game. Doing a stop action from here will reset back to the initial app state and clear any effect of the app logic running, time is reset.
• Pause: When the pause action occurs, we suspend the R3F loop and go back to the edit mode, but the app state is not reset and time is paused. This way you can play, pause, then edit based on what the app state reveals. Resuming keeps running the R3F loop with whatever changes HMRed in code.

And then I am experimenting with modifying R3F core to allow for frame stepping. That is you pause the app and then choose a delta to advance by allowing for fine grained analysis as you edit.

Anyway, the way I see it no app systems should ever run while we are editing. They can only interfere and have unintended side effects that would not appear in production. To do this, my strategy is to suspend the R3F loop and put back just enough into an editor loop for whatever editing tasks we need. This isn't so bad because in R3F v9 the Stage class is a module and I might do the same with the whole scheduler so it can be easily reused inside the editor.

Overwriting the useFrame/useUpdate calls is then just for being able to enable and disable specific callbacks. It was previously for pausing the app logic but of course this is flawed and managing the entire R3F scheduler handles this at a more core level.

[krispya]

The more I thought about this, the more I am assured it is a good idea to use our own loop and not just be a root inside of R3F's. The main reason is the clock.

The R3F app's time should be completely independent from the editor. For example, if the position of an object is depended on clock.elapsedTime (usually with Math.sin), when you play and pause the app, it will work as expected with the animation continuing from where it left off. If instead the editor and app shared a clock then the animation would jump forward in time as soon as play was resumed since time kept ticking during edit mode.

The main con here is that we would need to implement the event manager on the editor side too since it relies on the R3F loop to execute. But this is fine. If we want some kind of "live edit mode" where the app logic runs AND the editor logic runs at the same time we will run into a (solvable) problem but I honestly don't think it's a good idea anyway considering the number of uncontrollable side effects that could occur.

[itsdouges]

Great insight thanks Krispy. Definitely resonates with me! What's needed for step in r3f core, vs implementing ontop of frameloop demand? I implemented my own in user land that would take control of use frame, probably similar to the current implementation after hearing about it. I do remember that demand has a bug where the clock keeps ticking.

[krispya]

To be honest, I haven't tried demand. The way R3F v8 is right now, it should work just as well since invalidate steps one frame at a time, but I have some conceptual concerns. (The clock is something I'll fix in general as it is completely busted if it doesn't just constantly run. )

The difference between never and demand is a conceptual one that needs to be worked out on the R3F side. never was originally intended for testing where you advance by a target amount of time (though the current implementation isn't that useful IMO), while demand was intended for on-demand rendering, so you only render frames when you need them but otherwise the app logic keeps ticking. I want to harden these concepts in core, but to me what they mean is that never should enforce no app logic process until advance is called and then simulates advancing by the target delta time while demand keeps ticking but rendering is paused until invalidate is called.

If others agree with these concepts then I would want to use never for editor purposes as demand could conceptually allow leaking.

Also, I am realizing that the createRoot system might do what I want. It generates a new clock for each root, I believe it was originally intended for managing multiple canvas in one app. It might actually make sense to take advantage of this and have an editor root that actually just operates on the main root's scene.

[itsdouges]

Conceptually I understand what we're planning on doing, but when we consider an implementation I need some help getting there. Is this understanding correct? If not let me know the specifics I'm missing:

• We have a canvas that overrides the r3f export, so in user land when they import it they get this override instead of the original one
• This canvas would have two roots, one for play/pause, one for edit
• Each root owns its own clock to prevent unexpected jumps between frames
• Switching between modes would result in unmounting children from one canvas and placing it in the other

---

As a side do we have a concept of opening individual scenes (components)? Or it's just whatever the userland is rendering beneath their Canvas atm?

[krispya]

When Nikhil gets done with the refactor I can push what I got started to build off of.

And the idea I had for the root was that there would be only one canvas but that the editor root would be "headless" and work off the scene being edited. If that kind of setup works is what needs to be tested.

[nksaraf]

Yeah I would like to be able to not remount everything when going from edit to play mode (okay when coming back).. remounting is always slightly slow and if we can avoid it we should.

[krispya]

Okay here is a proof of concept: https://codesandbox.io/p/sandbox/react-three-editor-root-test-iqyx9m?file=%2Fsrc%2Ffeatures%2Feditor%2Feditor.tsx

I hacked around adding a headless root with custom provider. Any R3F hook called inside the EditorProvider will use the editor root and any called in the usual app Canvas will get their original root. You can suspend and modify the loops independently but they share the same rAF and EventManager. (Note, App is paused here, go and comment out the useEffect setting its frameloop to never to have its loop run).

This needs a clean up, but it proves the idea!

[itsdouges]

Interesting! So to summarize what's happening:

1. Using the context export from r3f we can override the store that is provided to child r3f hooks (is advance used internally for useFrame?)
2. App world component "resets" the provider back to the original store
3. Editor world calling state.gl.render(state.scene, state.camera); takes over rendering (why?)
4. _roots hold r3f roots (a root is just threejs instances and config from the looks of it? Is this public API? Do we need to redeclare everything?)
5. To enable play -> pause -> edit flows it's just a matter of conditionally setting the editor provider + taking over the render loop when appropriate, or is there more involved?

Trying to understand the entire end to end flow, and what's private vs. public API 😄. Lot of new exports I've seen from this! _roots, advance, events and context...

[krispya]

Yeah! I'm digging into some of these internal more myself. Cody is the real genius behind the reconciler and root system.

No you got it. I'll break it down as I realized even my example has an issue that can be worked out better.

R3F uses a concept roots to define independent instances of renderers that are managed by the same R3F instance. So typically this is for each Canvas is defined, so no matter how many Canvas you create, they share the same rAF loop, event manager, etc.

Roots were not intended to be a public API but _roots is being exported for historical reasons and I am taking advantage of this fact to do a little manipulation. I am creating a custom root for the editor that shares the same scene and gl renderer as the app which makes it so that the editor loop and app loop are now independent. We can start and stop them without effecting one another and they keep their own time, callback subscribers, events, etc.

To take full advantage of this I then do what you say, I override the Canvas component in the app with ours where we inject the editor components which looks like this:

<OverrideCanvas>
 <DomComponents />
 <DomSystems />
  <Canvas>
    <EditorProvider>
      <FiberComponents />
      <FiberSystems />
      <AppProvider>
         <App />
       </AppProvider>
    </EditorProvider>
  </Canvas>
</OverrideCanvas>

And this way any R3F hook calls inside of EditorProvider use the editor store and any inside AppProvider use the app store not knowing at all that its environment was tampered with.

The issue I was having was that I was trying to create the editor root before the app root, even though the editor root needs to use the app root's scene and gl renderer. I was being foolish and didn't realize that actually the app root gets created first anyway and I don't need to portal it from inside App upward -- it gets created as soon as Canvas is mounted.

[krispya]

Oh and to answer some other questions. advance is not used in useFrame and could be defined in the store. It just happens to be defined in the loop file which is why it gets imported. It probably doesn't need to be written that way. And we need to take over rendering on the editor loop because we pause the app loop so if we want to display objects being moved around, we need renderer draw calls to come from the editor.

[itsdouges]

Thanks Krispy it's all becoming clearer now! I really like this direction. I see the value in getting this thought through, similar to other core functionality (code/scene sync, editor/scene state). Get them right and it's a breeze building on top.

I'll need to look at docs for advance. What does it do? I imagine if I look in the current canvas docs I'll see _roots used too right? Is there any risk that these APIs will be deprecated and removed, should we bring Cody in for a chat?

Love it

[krispya]

Here is the branch implementing this strategy: https://github.com/pmndrs/react-three-editor/pull/36

So far it's working pretty well, but I'll need to dig deeper for events and also some of the tunneling.

advance is used when the frameloop is set to 'never'. It takes a timestamp and advances (heh) the frozen loop by the delta with the previous advance call. IMO it makes more sense for you to pass in a delta and advance by an exact amount, but that's a discussion for v9.

As for _roots, I do think for v9 we will want to make a public API. Part of what I want to do with this editor is see if there is a legitimate use case for a "headless" root and if there is, then look at creating a public API for it. There is a createRoot function already but it requires a canvas element currently.

[krispya]

Okay! The implementation is now nearly complete with events working. I'll share what I learned about events so we can work on the details together. I had assumed that initializing a root store got you all you needed for an event manager, but this is not true. In fact, because R3F supports native and web, the events manager is initialized separately and passed into the store. This is what the events export is. For web, it is createPointerEvents aliased to a really poorly named export because I thought it was an object but turns out it is actually a factory. Other than that, there were a few other properties I needed to copy over into my headless root from the app root such as size and viewport and then things were good to go.

[krispya]

Events should finally be working with https://github.com/pmndrs/react-three-editor/pull/39