Bevy Subworld RFC

[NathanSWard]

Rendered

This is based on the discord discussion starting here.

This is a proposal for a Subworlds concept: disjoint sets of entities & their components.

[alice-i-cecile]

After a first skim, I have three other use cases in mind for this:

1. Prefab staging grounds, as raised in https://github.com/bevyengine/bevy/issues/1446. The idea is to have prefab entities ready to go (rather than always needing to load them from file), without having them affected by random systems.
2. Parallel worlds simulation for predictive AI. Let your AI imagine and evaluate hypotheticals without affecting the main world.
3. Undo-redo functionality. Cache old states in subworlds, then revert back to them quickly.

Do those sound feasible to you with this design?

[hymm]

How will subworlds interact with the scheduler? Specifically stages, states, and exclusive systems.

[jamescarterbell]

I'm wondering if this needs to be an engine level thing or if it can just be its own library with the subworlds as resources. Having it as a library pretty much clears up all the questions about how the resources are shared and also allows for the consumers to set it up as needed for their use case.

[alice-i-cecile]

I would be very interested to see if we could use this idea to temporarily store removed entities / components / relations in a purgatory world. I think that would clean up both the API and the impl in a satisfying way.

[alice-i-cecile]

`@sircarter had an excellent comment that I don't want to get lost:

If subworlds are going to be a engine feature it'd be nice if we could have queries look like this Query<Q, F, W> where W: DerefMut<target = World>

Or something so you could easily query subworlds. This could also be added to relations so you could have cross world relations

[cart]

This has some overlap with my pipelined rendering experiments:

• I have a Game World and a Render World
• I have a Game Schedule and a Render Schedule
• These need to be able to run independently in parallel (at times) and be synchronized (at times)

My current experimental solution is SubApps. Note that this is a hack-ey impl that only exists to get the ball rolling. It largely punts the synchronization (and parallel execution) problem, but it requires almost no changes to the current apis or app models, which is nice.

pub struct App {
    pub world: World,
    pub runner: Box<dyn Fn(App)>,
    pub schedule: Schedule,
    sub_apps: Vec<SubApp>,
}

struct SubApp {
    app: App,
    runner: Box<dyn Fn(&mut World, &mut App)>,
}

impl Plugin for PipelinedRenderPlugin {
    fn build(&self, app: &mut App) {
        /* more render stuff here */
       
        let mut render_app = App::empty();
        let mut extract_stage = SystemStage::parallel();
        // don't apply buffers when the stage finishes running
        // extract stage runs on the app world, but the buffers are applied to the render world
        extract_stage.set_apply_buffers(false);
        render_app
            .add_stage(RenderStage::Extract, extract_stage)
            .add_stage(RenderStage::Prepare, SystemStage::parallel())
            .add_stage(RenderStage::Render, SystemStage::parallel());
        
        app.add_sub_app(render_app, |app_world, render_app| {
            // extract
            extract(app_world, render_app);

            // prepare
            let prepare = render_app
                .schedule
                .get_stage_mut::<SystemStage>(&RenderStage::Prepare)
                .unwrap();
            prepare.run(&mut render_app.world);

            // render
            let render = render_app
                .schedule
                .get_stage_mut::<SystemStage>(&RenderStage::Render)
                .unwrap();
            render.run(&mut render_app.world);
        });
}

impl Plugin for PipelinedSpritePlugin {
    fn build(&self, app: &mut App) {
        // 0 is currently implicitly the "render app". A more complete solution would use typed labels for each app.
        app.sub_app_mut(0)
            .add_system_to_stage(RenderStage::Extract, extract_sprites.system())
            .add_system_to_stage(RenderStage::Prepare, prepare_sprites.system()));
        
        /* more render stuff here */
    }
}

[cart]

I'm definitely not advocating for SubApps as the solution at this point. Just sharing where my head is at / what my requirements currently are.

[alice-i-cecile]

From Discord: we could use this as a tool to get closer to "Instant" command processing.

Have three default worlds (or more likely two shadow worlds for "normal" world):

1. The main world, which works as we have now.
2. The Commands world, where components are immediately added / entities are instantly spawned before being reconciled.
3. The purgatory world, where removed components and entities live for two frames under a double buffering scheme before being permanently deleted.

This would be fantastic for prototyping and ergonomics, and has a nice clear mental model.

[tower120]

I would like to say that having separate worlds alone may be not enough.

Imagine that you have some simulation - not only you want to isolate it entities and systems/queries, it is also desirable to tick/update it at specific rate. You may want to have that simulation real-time synced, or may want to jump forward in time, by simulating hundreds of world frames at once (probably while not locking main world). In other words, tick world manually may be necessary, fixed step is not always an option.

So I would vote for something close to separate apps... That would also make development of certain game aspects (like mini-games, smart 3D HUD/interface, etc...) more isolated/modular. You can make your mini game as separate project, then just attach it to main one. (This could be achieved with plugins as well - but you can touch something in main world you want/should not)

[jamescarterbell]

@tower120 Subworlds may not be enough for ever use case, but they are still a useful additional and one that can be added independent of multiple executors/synced apps. Although for your examples I can imagine some good solutions that don't require multiple apps.

[cart]

Agreed. I think we'll want "multiple ECS schedules" and "multiple worlds" with the ability to pass an arbitrary number of worlds into a given schedule. I think "multiple apps" currently implies a tight [Schedule, World] coupling, which I think isn't what we want. Instead, I think we want to allow "arbitrary orchestrator logic" to determine how Schedules execute on Worlds. That way you don't need to create a bunch of empty Schedules when all you need is "staging worlds" for your app logic.

[tower120]

@cart Ok, that sounds even nicer. But what about Time resource? Doesn't schedulers coupled with time?

And I think I want to have some kind of "manual" scheduler, where you can do scheduler.update(delta_time). Like for case of some sort of turn-based simulation, or offline-simulation.

[alice-i-cecile]

And I think I want to have some kind of "manual" scheduler, where you can do scheduler.update(delta_time). Like for case of some sort of turn-based simulation, or offline-simulation.

For this, the current custom runner abstraction works pretty well; I think that we could integrate that nicely into what Cart's proposing to get a really lovely abstraction.

[cart]

Lots of things rely on specific System/World pairs. As long as the Time used by app logic is stored in the "app world" and it is ticked using the "app schedule", things will work as expected.

[tower120]

@alice-i-cecile Looks like that "runner" blocking.... It would be much nicer to update app/scheduler directly. Like to "move" world when we're want that.

BTW, why do you want "subworlds" (like hierarchical?), not just "worlds"(plain) ?

[alice-i-cecile]

BTW, why do you want "subworlds" (like hierarchical?), not just "worlds"(plain) ?

The initial implementation was easier if you stored them in Resources :P From my perspective, I'd enjoy a bit of hierarchy to have nice "purgatory" or "command staging" worlds associated with each "proper" world.

[cart]

We can always have freestanding Worlds that live inside of components / resources. But imo if we're integrating with the scheduler it should only care about "global top level worlds"

[tower120]

BTW, Unity ECS has interesting feature of world merging. In Unity ECS archetype's storage consist of 16Kb chunk linked list. So merge is close to just changing pointers in list. That is useful for async loading / world cells streaming. You load everything what you need in stage/temporary world, then when load is done - you just merge that world into active/main one..

Maybe bevy could utilize something like that with new multi-world system...

[cart]

Ooh thats very clever. We'd need to do that at the table level, not the archetype level (because archetypes are just metadata in Bevy), but its definitely possible.

I'm starting to think we need both:

Sub Worlds

Shared entity id space, shared (or easily mergable) metadata ids like ComponentId/ArchetypeId/BundleId, etc. Separate (but easily mergable) Component/Resource storages. These would be extremely useful for a number of scenarios:

• Replacing Commands: Commands are allocation heavy right now and can bottleneck op-heavy things. SubWorlds would allow us to move more work into systems, re-use Component allocations across runs (and/or directly move the allocated data into the main world), handle "archetype moves" within a system, run queries on entities that were just spawned, etc
• "Baked Scenes": Prepare entities in a "subworld" and "merge" it when its ready. However this is slightly different than Commands in that Scenes might be instantiated multiple times. Ids shouldn't be reused, Component values might need to be remapped (ex: for unique Asset or Physics handles), etc.

The major missing pieces here:

• Apis for creating and interacting with subworlds (should largely mirror the World api)
• Ability to share / sync archetype, component, and bundle information with source World. Direct sharing would likely involve locks. Indirect sharing would be memory-hungry. This will require some careful thought. It might require "reconciliation" logic for Archetypes created in SubWorlds that aren't yet in the main World, or that do exist in the main World, but were created elsewhere.
• Entity Id space management: ideally we don't need to "remap" entities created in subworlds (because users will want to store+use EntityIds created for them and those entities should remain valid when merged with the main world). allocating entities in parallel is already possible. The missing piece is handling metadata tracking (which is currently stored in a dense array). How should subworlds store this metadata, which won't be densely packed at zero? How will the main world account for entity ids allocated, but not yet merged into the main world? I'm guessing it will likely be something like "Make EntityMeta an enum { Live(Meta), Allocated } .... when entities.flush() is called, set subworld ids to Allocated". This will add an additional branch to entity location lookup, but its probably fine, given the benefits we're getting.
• Ability to efficiently merge storages. Maybe it uses pages like @tower120 mentioned to allow complete reuse of allocations. Maybe it just does the most efficient copy possible.
• Ensure SubWorlds cannot access data they shouldn't be able to in the context of a schedule (ideally they just don't have any references to this data). Id also like to avoid doing expensive "access control" on subworlds (ex: what legion does for a similar api). SubWorlds shouldn't have access to the main World data / they shouldn't be a "filter" on the main world data.
  • Edit: its worth discussing this a bit. We could use a system's Access<ArchetypeComponentId> to expose filtered access to the main World's data inside the system's SubWorld (much like legion does). But this would also mean doing more expensive runtime access control checks for every op.

Multiple Worlds

Completely separate Worlds without any shared state. Useful for running parallel tasks/contexts (ex: AppWorld, RenderWorld, NetworkWorld, OtherAppWorld, EditorWorld, etc).

The major missing pieces here:

• High level api for constructing these worlds
• Scheduler integration to support running logic on multiple worlds at the same time
• System integration to enable accessing Components and Resources from multiple worlds at the same time.
• User-configurable orchestration logic to determine when "multi-world schedules" run.

Summary

I think "sub worlds" could yield some pretty significant performance and functionality wins that would impact all Bevy apps, but they are a more complicated endeavor.

"Multiple worlds" are only useful in a few specific contexts, but the implementation scope is much smaller.

I don't think theres a good way to unify these two scenarios (and I don't think theres much value in doing so). We should probably start two separate efforts to design them.

[alice-i-cecile]

I completely agree with your analysis above @cart. Both are very useful, but have entirely distinct use cases and design challenges. Getting the high-level API right is about the only bit that I think should be done in unison.

[NathanSWard]

I'm starting to think we need both:

I also completely agree with this. And if I'm being honest, when I initially created the RFC, I was thinking more Multiple Worlds than what has turned into Sub Worlds I guess it was just a bad naming choice on my half.....

[tower120]

@cart What about "Universe" concept? Universe is basically EntitySpace (virtually, EntityId generator). Each world constructed_within/constructed_with universe and cannot be moved between universes. According to what you described: subworlds - is worlds within the same "universe", multiworlds - worlds in different "universes". Worlds within the same Universe can be merged and traversed by the same system(/scheduler?).

P.S. What kind of ECS storage bevy use? (I thought it is archetype-centic...)

[bjorn3]

Bevy supports both the table layout, which is often called archetype in other ECS'es and the sparse layout. The default is table, but you can override it to sparse on a per-component basis.

[Waridley]

Just being able to run the same SystemStage on multiple worlds would be extremely helpful for my use case: killcams with rollback netcode. Everything else in this discussion would just be bonus convenience features for me, but not being able to run the same SystemStage on two separate worlds makes killcams so awkward to implement that I probably won't bother until this feature is added to Bevy. A separate world is how Overwatch implements its killcams as well. It lets the game keep ticking in the background while the replay is being rendered.

Or heck, even being able to clone a SystemStage or Schedule before it is run for the first time would help a lot.

[alice-i-cecile]

My current gut feeling is that the basic design should be:

• each app stores a Vec<World>
• each app also stores a Vec<(Schedule, WorldId)>
• by default, there is only one of each
• schedules run on the world they are pointing to each tick (if any), and all schedules must complete before the next tick can begin (allowing for cross-world synchronization at the end of the tick)
• we add a Commands-like tool to transfer ECS data and ordinary Commands between worlds, create and delete worlds, and enable / disable schedules
• Schedules become clone-able
• use https://github.com/bevyengine/bevy/pull/2736 to allow a app.add_system(my_system).to_schedule(MySceduleLabels::Staging) API for better ergonomics

[TheBlckbird]

Any updates on this? Is it still planned to implement this?

[alice-i-cecile]

A solution for something in this space is still desired, but the ECS crew largely have their hands full pushing towards relations at the current time.