In all of the discussions/documents I’ve seen so far, the concept of a ‘script’ is discussed as a unique ‘thing’, with no regard for the concept of a runtime. When I see ‘dependent on eachother’, I become wary because this is conceptually introducing interdependence between working units (scripts) which co-exist already in a given runtime (Driver). The concept of types and schema should really be interdependent, as one defines the other - so I don’t really see the need to handle overlap/dependency, as the ‘types’ should be considered a shared state between all.
Whew, ok. Now with that out of the way. I am very wary of pre-defined component definitions schemas (the concept of having to pre-define all types in a file and edit them there) unless the process is completed automated. That is to say, if my rust AND all other language definitions are generated from that schema, its fine. However - this adds a huge layer of complexity and limitations on the rust data itself, and on what is described for the engine.
Personally, I am partial to Flatbuffers reflection for the schema, and possibly flatbuffers itself for some/all type interference and marshaling. Rather than discussing the merits of it, I am going to discuss the benefits of it abstractly, as I think thats part of the discussion. So without further rambling, here are the reasons its my fairly uneducated first go to.
- It has a native rust implementation, requiring no bindings
- It has a self describing reflection: (Ex: https://github.com/google/flatbuffers/blob/master/reflection/reflection.fbs) which can be extended and modified for our purposes if so required (although it already has an attributes feature, which can be used for the metadata such as iterators)
- It is zero-copy and unserialized, so data is read as-is with zero-cost abstractions
- It is expressive in both binary and text formats, allowing for FFI usage and user-editing
- It allows codegen to many languages, keeping type safety possible in rust and others where applicable
It supports flex building (building arbitrary messages) as well as strongly typed (defined via a fbs definition), for further flex(ability).
Now, how did I mentally envision this possibly tying in with FFI? A few ways:
- Schemas could exist in text and/or binary format for various build types (debug, develop and release packaging). These could also be statically baked for release builds
- Schema transfer to different drivers could be done in a binary fashion with simple FFI functions, returning the Reflection struct via FFI being simple
- This abstracts away the concept of a ‘schema’, and allows us to treat it as just another data type
- In regards to specs, this allows the storage of arbitrary component types (flatbuffers buffers) in a Vec variant, while providing us with strong typing auto-generation features. Additionally, it allows rust to still access this components, so they arn’t opaque (Flexbuffer access /w the reflection can be used).
Off the top of my head, I’m not aware of any other data schema language which provides both the description of data and the read/write of data in such an opaque format. If there is one, without hte overhead, I’d love to see how it compares.