Scripting: What Do We Need to Get There

(Zicklag) #33

It looks like somebody already asked the same thing. The response was:

Yes, these are pretty much equivalent. However, transmute can also change the types of non-pointer types and it can also change lifetimes. You can’t do that with as . Since transmute has such power (and therefore potential for misuse), its use is generally not recommend.

I looks like using as can do less damage when used incorrectly which is probably a good thing.

(Théo Degioanni) #34

Right, but here we want to transmute actual data, not references. So we do need to use transmute. Unless you want to do it differently?

(Zicklag) #35

Nope, this is the first time that I’ve done most of this so if you think transmute is good then lets go with it.

(Zicklag) #36

You think that nom would be overkill for a parser, too. I haven’t written a from scratch parser and the only one that I’ve read through was the Haxe XML parser. If we wanted to do it from scratch would that be a suitible design to model it after?

(Théo Degioanni) #37

No maybe nom would be suitable, I’ve just never used it. But the usual method to write a parser is to make a state machine: in our case, we would have the initial state, when we find the “struct” string we go to some state where we gather the name of the struct and wait for { to go to some other state where we now wait for a field, etc…

But honestly you can do whatever you want here, this is just what I would do because I am used to doing it like this.

1 Like
(Joël Lupien) #38

If you save your data in one endianness, and then load it on a computer with another endianness, you’ll have a lot of issues. Be careful to use a crate to change the endianness to the right one when saving or loading.

(Théo Degioanni) #39

Intermediate component representation should not be stored in the first place in my opinion. It could create desync issues.

(Zicklag) #40

If you want to send the data over the network for multiplayer or you want to save replays that might be a concern, though.

(Théo Degioanni) #41

If the data needs to be sent over network, it will go through the networking team’s stuff. They have that covered I’m sure.

(Zicklag) #42

What if we used Cap’nProto? It says on their site:

  • Inter-process communication: Multiple processes running on the same machine can share a Cap’n Proto message via shared memory. No need to pipe data through the kernel. Calling another process can be just as fast and easy as calling another thread.

It would be a universal, platform-independent, zero-copy memory representation. If it can share memory across processes on the same machine you could even skip the C API which could make it super-easy to integrate with other languages like Python or Lua possibly without writing any code in another language like Rust or C.

I’m not sure if there are other disadvantages yet.

(Théo Degioanni) #43

Does it support generics?

(Zicklag) #44

Looks like it:

https://capnproto.org/language.html#generic-types

(Théo Degioanni) #45

Can its memory layout be easily used by FFI?

(Zicklag) #46

I don’t know. I would think if there is a way to use messages sent over the network or IPC that there would be an easy enough way to send it over FFI, but that is just a guess.

(Théo Degioanni) #47

I mean beyond sending it, FFI is about using C-like memory layouts so that compilers can work with the memory directly without needing to translate it to their own way of managing memory. If languages cannot easily understand the format like that, it defeats the point. But if it can yes that would be a great solution.

I’ve looked over the internet and have seen some stuff but haven’t dug too deep into it. However one thing to keep in mind is that what I’ve seen was for python, for which most overheads are quite negligible next to the overall slowness of the interpreter, so we need to be sure of the very details.

(Zicklag) #48

The whole point of the memory format that it uses is that it can be used without having to deserialize it into a different memory layout. The Cap’n Proto schema format is compiled to the C, Rust, Python, or other language source code that is required to interact with the memory layout so that you don’t have to copy it to another memory structure to use it.

We will have to get deeper into it and test it out, maybe benchmark it too, but it looks like it does pretty much what we want it to. I’ll probably see if I can find some examples to try out and do some experimentation with it.

(Théo Degioanni) #49

Okay. My main concern is regarding LuaJIT. As Cap’n’Proto is not standard, the JIT probably won’t be able to make it work as fast as it makes standard FFI work.

(Zicklag) #50

If it was a problem you could just use a C or Rust wrapper around Lua. Instead of talking directly to Cap’nProto with Lua, you could use the Cap’nProto integration with C/Rust and then bind that directly to Lua through an FFI.

(Théo Degioanni) #51

I don’t understand how that would work without copying?

(Zicklag) #52

If Cap’nProto can avoid copying by sharing memory, there would be no copying from Amethyst->RustWrapper. I was thinking that there would be some way to present those values from the RustWrapper->Lua without copying, but I guess that wouldn’t be possible if Lua has to have the memory in a certain format for it to understand it.

Maybe there isn’t a way to do that. There is a WIP Cap’nProto Lua project that we might be able to look at how it works, but it might just use copying. We’d have to take a deeper look into it.

1 Like
Scripting: How Can Scripts Interact With Non-Scripted Components?