🔨 How should we load, store, and modify static entity data?

[zach2good]

I affirm:

• [x] I understand that if I do not agree to the following points by completing the checkboxes my issue will be ignored.
• [x] I have read and understood the Contributing Guide and the Code of Conduct.
• [x] I have searched existing issues to see if the issue has already been opened, and I have checked the commit log to see if the issue has been resolved since my server was last updated.

What problem are we trying to solve?

Capturing points from a discussion we're having:

• It's hard to modify the static mob/npc data that's loaded on startup to build Mob/NPC objects.

What caveats do we need to keep in mind about a possible solution?

• We don't want the solution to be too verbose
• We don't want to bloat process startup time
• We don't want the maintenance burden of additional complex tooling
• We want submitted changes to mob/npc data to be human-readable in PRs
• We don't want to introduce more languages into the repo alongside existing: C++, Lua, SQL, Python
• We want quick and convenient editor support
• We want validation at point of writing, and in CI

What options do we have?

• Eden's approach (I haven't looked too closely, but I think it works like this): Keep an intermediary language like YAML, and have tooling that converts that back into the SQL we expect, and load that as normal. This YAML is human readable, easy to review, and fast to load since it's converted back into the relevant SQL, or it's converted and inserted back into the db.
• My old proposed approach: We keep everything as it is, but we introduce a database editor written in Python/tkinter that changes your actual database and you dump it back to files with dbtool and submit changes that way.

Open to all suggestions (if you have a valid opinion. Not interested in non-developer-types weighing in here)

[zach2good]

One option that's being floated is to store a lot, if not all, of this static data in Lua files.

Benefits:

• Uses existing language
• With proper use of tables/enums, and dumping long enums into locals at the top of a file, data becomes extremely human readable - but still verbose

Drawbacks:

• Big scary for me: If we're loading lots of Lua at runtime, Lua is ironclad-ly single-threaded. We also can't make multiple Lua states and then send objects between them. I really really really don't want to roll back to the heavy startup things being singlethreaded again.

[KnowOne134]

Edens yaml approach also allows hand editing the database and export with dbtool to fill in the yaml data changes. Makes editing many blocks with a statement very easy instead of searching and changing 1 by 1

[jmcmorris]

Lua can actually be made multithreaded. The main challenge with Lua multithreading is that each Lua state is sandboxed so you have to synchronize data between the states in C++. However, since we're really just wanting it to be one way (Lua data to C++) we don't need to worry about this.

It should be entirely possible to spin up a bunch of threads with each one given a Lua state and list of mob files and they can each load up their files and store that data in C++. I probably would store the data in C++ and then join all of the thread data together at the end to avoid constantly checking a mutex for safely writing to the final data container.

[cocosolos]

Recently had a discussion about this and one thing I do like about Eden's yaml approach (not directly related to entities) is that they seem to maintain some enums like mods and effects as yaml then have tooling to generate the C++ and Lua tables, whereas we maintain each of those separately which can cause issues.

I do also like the idea of a GUI mob editor, which could exist alongside of and export to whatever system is decided on. Not really a fan of moving exclusively to Lua though as I think that while it may be easy to read and edit, it may be harder to search in ways that SQL allows. Plus having stuff available in the DB makes it very easy to integrate with basically any external/custom tooling.

[claywar]

I think one of the suggestions I'd like to float is us rethinking how mobs and other static data is structured in the first place. Mobs are a convoluted mess to navigate and implement at this time, and if we're going to move to another interpreted format, its time to take a step back and evaluate how this can better be done.

[zach2good]

it may be harder to search in ways that SQL allows

This is actually a very good point, thanks @cocosolos!

I think startup (and obviously our entity classes etc.) should stay as they are, where we load from SQL. I know I haven't measured anything yet, but I can't imagine loading large Lua tables from files and then building entities our of them could be anything but slower than loading from SQL.

So then the other data layout (Lua or otherwise) gets pooped back into the database, or back into SQL files for consumption as part of dbtool's regular running.

[zach2good]

Also yes @claywar, we should be inspecting how we build and lay out mobs before we start making changes, they're a nightmare