Legion ECS Discussion

(Théo Degioanni) #21

Excellent idea!

Scripting would work in specs without too much work. What Frizi is describing is how once your components are statically declared in specs, you cannot access them as flexibly as you could in their development-time form. This is not an issue for scripting itself as dynamic dispatcher rebuild and conditional dependencies are a development-time thing. I would need to know more about Frizi’s engine use case to give an opinion on how feasible specs is for their specific use case, but it is not related to scripting.

However I do agree that legion would give more advantages on those aspects, especially on that first point which to me is extremely important and extremely difficult to achieve without locking all the time in specs. But this is outside the scope of scripting.

TL; DR: scripting can work in specs if we want it to.

(OvermindDL1) #22

I just took a look at legion and I have a question:

This sounds exceptionally bad for an ECS library. I add and remove components very rapidly, all the time, all over the place, and this seems backwards from how it is traditionally done… Just how much ‘significantly slower’ is it and why? If it is baking component ‘sets’ into arrays then that sounds more like a dataflow system than ECS, which is fine, but it’s not ECS.

1 Like
(Zicklag) #23

From the Discord conversation:


It is optimised for entity creation, deletion and iteration speed. Dynamically adding and removing components from existing entities is supported but slow. You can generally design game code around avoiding the need to do component addition/removal outside of entity creation, but you cannot avoid creating, deleting or iterating through entities; those are the core functions of an ECS.

I do not think adding/removing components to existing entities is nearly as high a priority as creation and iteration speed

As far as I understand it, it is slower because of the chunked design of the entity storage, which is part of where it gets its speed everywhere else.

If you needed to add and remove components on an entity I would imagine you could add a boolean enabled field to the component and ignore any non-enabled components in the systems that use it.

(OvermindDL1) #24

That wouldn’t really work. In my old engine I add/remove components rapidly for determining things such as status effects, query information, etc… etc… and tend to number in the hundreds of components. Having all of those enabled on every possible entity that might have them, which tend to be in the hundreds of thousands of entities or far more), sounds like an exceptionally large waste of memory and processing.

(OvermindDL1) #25

The ‘chunked’ design that keeps being referenced looks like a classic dataflow pattern rather than ECS, and dataflow is exceptionally useful and fast, but it is not ‘dynamic’ in the realm that ECS is (ECS is quite literally a rapidly dynamic dataflow pattern).

Perhaps there should be two types of components, those that get chunked, ala the current dataflow design, and the other types that are stored out-of-band, ala normal ECS style (specs style for example). Generally the components that are rapidly added and removed tend to be fairly few in number (though not always). The few cases that tend to be large in number tend to change less often as well in the ‘bulk’ cases with a few (few hundred that is) entities changing fairly rapidly (boundary components are common here). This seems like it would satisfy both worlds, and it is possible they could work via the same interface as well, or at the very least the same query interface.

(Kae) #26

That comment about changing entity structure being significantly slower is misleading IMO. The existing legion implementation may be because it does heap allocations for a number of things, but all that is required is to swap remove the components from its current chunk and add it to a new chunk.

I don’t know what you mean with chunked component storage being somehow “not ECS” or “not dynamic”. Is there a true Scotsman argument being made?

(Kel) #27

All an ECS is, is a way to relate an index of items to the data for those items, while decoupling procedures operating on that data. Legion satisfies these properties, and is an ECS.

1 Like
(OvermindDL1) #28

Not at all, I think this is just a lack of definitions of terms, I use definitions that I’ve learned ~30 years ago with ECS having come out ~20-25 years ago, so essentially the definitions I’ve been taught (just to make things unambiguous):

  • Dataflow is a set of structs in one or more arrays where an array holds a single ‘type’ of struct. It is easy to add and remove entities to the system, but changing the data they hold involves removing them from one array and putting them onto another, if made well (POD in other words) that’s as simple as a memcpy of the different struct elements from one layout to another.
  • ECS Is Dataflow except instead of the components of an entity being held in the same allocation, instead each component is held in its own array with the same index across them being the same entity.

Most games tend to use dataflow in a large variety of areas, particle systems are the most common, many games actually use dataflow for their game entities in full, take Factorio as an example, where each ‘type’ of entity is a different dataflow array, to change components involves destroying the old and recreating it in another with a different set (how legion works it seems).

They are both accessed via indices, what differs is the memory layout and access patterns. The Dataflow pattern existed before the ECS pattern by a good couple of decades and ECS is considered a subpattern of Dataflow. In other words, all ECS’s are Dataflow, but not all Dataflow’s are ECS.

This is not at all saying that Dataflow is bad, it is absolutely more common than ECS (quite literally near every engine has parts that are dataflow, especially particle systems), just that the access patterns are different since Dataflow focuses more on static speed of small collections by combining the components into singular arrayed structs and ECS focuses more on rapid adding and removal of components.

Non-ECS Dataflow isn’t always faster than ECS either, when there are many components and the sizes become too large than it causes too large of jumps between objects when you are trying to access only small amounts of data.

This is also why some ECS engines combine both patterns into one engine, where certain components are marked as batched and get combined based on their batching tag, for example:

  • Component Transformation has batch tag 0
  • Etc PhysicsLink has batch tag 0
  • Renderable has batch tag 0
  • Inventory has batch tag 1
  • A has batch tag 1 (I’m running out of ideas for names, I just woke up…)
  • B has batch tag 1
  • C has batch tag 1
  • D has no batch tag
  • E has no batch tag
  • Jump has no batch tag
  • etc…

So component Transformation, if it exists for a given entity, will be combined into dataflow batches with others also tagged 0 (PhysicsLink and Renderable in this example). Inventory will be combined with A, B, and C anytime any of them exist. D and E and Jump are untagged and will never be batched with other components whatsoever, this is similar to how specs works now, these are optimal for components that are added and removed rapidly, where batching is most optimal for when they are less often used or only often used together.

This is generally considered the most optimal ECS pattern where it appears dynamic via the API but there are optimized paths using non-ecs style dataflow, though I never got around to adding batching in my old engine (other than manually in a few cases), it is what I’ve always wanted to experiment with.

I’ve not actually heard ECS being defined with such a definition in over 20 years that I’ve been using it?
That style could describe even the old Unity system, which is exceedingly and extremely inefficient and is not dataflow in any way whatsoever (with significant performance detriments because of it), and it is not generally considered an ECS even though it has entities, components, and calls to operate over that data. It’s always best to define things very precisely.

(Kae) #29

Thanks for the clarification! Then in these terms I think legion's storage design is a sort of combination?

  • Component data is partitioned into fixed size (16kB) allocations called “chunks” based on each entity’s set of component types. Each partition is called an Archetype.
  • Components within a chunk are stored in separate arrays for each component type.
  • An array of entity IDs is stored within a chunk. An index across component and entity ID arrays refer to data belonging to the same entity.
  • A hashmap is maintained in the World with the location of each entity for point lookups.

So adding/removing components from an entity would require the following steps,

  • Find/allocate a new chunk for the entity’s new Archetype
  • Copy component data and entity ID into the new chunk
  • Remove components and entity ID from current chunk, probably by replacing it with the last entity in the chunk
  • Update entity location entries

The runtime of this will increase relative to the total size of the entity’s component data, and does include 1-2 hashmap lookups, but there’s nothing inherently slow about it, at least by my definition. It’s all O(1) relative to the number of entities in the World.

(OvermindDL1) #30

That’s how it was seeming to me, I’d only seen that in a closed source library in the past, not an open source, so it was a nice surprise. ^.^

It might be fine as it is, benchmarks will tell for sure! :slight_smile:


@kabergstrom - curious, if I’m building a scene graph renderer thing for wasm (single-threaded), should I use your fork or the official repo?

Or generally, just kinda wondering what the roadmap is - I see your repo is 16 commits ahead… are you planning to diverge completely or just working towards a milestone before making a PR?

(Kae) #32

IMO you should use my fork, as I believe future work will be based off it. I should probably make a PR or discuss the maintainer situation with @Ayfid


Another ECS to consider: shipyard

FYI @kabergstrom - I made a tiny PR on the original repo to allow passing a name in. It’s actually required for using legion in wasm environments since the randomized name thing breaks there. It hasn’t been merged yet, so not sure what the right approach here is… maybe I should revoke the PR and make it against your fork instead?

(Joël Lupien) #35

It seems like legion didn’t have any updates in the past 5 months :confused:

(Kel) #36

This one has had updates in the past 10 days : D https://github.com/kabergstrom/legion/commits/master

(Joël Lupien) #37

Yeah, Frizi updated me on that but then I forgot to update my forum comment here. Thanks :wink:

1 Like