tinygo-org
Can't compile WASM with big buffers
I managed to use TinyGo as a drop-in replacement for Golang, but that was done by relying on syscall/js. I am trying to understand how to use TinyGo in WASI and use it without any dependency on syscall/js.
To give a bit of context, this is for an image conversion project. Currently, this is the code for both Golang and TinyGo https://github.com/brunoluiz/imagewand/blob/main/cmd/wasm/main.go
From my understanding, I am not able to use []byte as an input parameter of an exported WASI function, neither return one, correct? It seems I should pre-allocate a buffer and set it through JavaScript shared memory model.
As it is dealing with images, I need big pre-allocated buffers (at least 5Mb). To test my code, I tried something as:
package main
import "fmt"
const bufferSize = 1024 * 1024
type image [bufferSize]byte
var buffer image;
//export getWasmMemoryBufferPtr
func getWasmMemoryBufferPtr() *image {
return &buffer
}
//export printBuffer
func printBuffer() {
for _, v := range buffer {
fmt.Println(v)
}
}
func main() {}
Depending on the value bufferSize it doesn't compile. I assume the compiler/LLVM has some limitations to big pre-allocated memory spaces. How do I work around this?
Besides, when bufferSIze contains a high value (such as the example above), tinygo build time grows with it as well. Is it normal?
- System: Mac OS Monterey 12.5
- TinyGo version: 0.24
- Build command:
tinygo build -o app/wasm/main-wasi.wasm -target wasm ./cmd/wasi
fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0x7000061d54a8 pc=0x6fa8e50]
runtime stack:
runtime.throw({0x8327547?, 0x0?})
/Users/runner/hostedtoolcache/go/1.18.1/x64/src/runtime/panic.go:992 +0x71
runtime.sigpanic()
/Users/runner/hostedtoolcache/go/1.18.1/x64/src/runtime/signal_unix.go:802 +0x3a9
goroutine 65 [syscall]:
runtime.cgocall(0x4372410, 0xc000044e08)
/Users/runner/hostedtoolcache/go/1.18.1/x64/src/runtime/cgocall.go:157 +0x5c fp=0xc000044de0 sp=0xc000044da8 pc=0x4007bfc
tinygo.org/x/go-llvm._Cfunc_LLVMTargetMachineEmitToMemoryBuffer(0x32808200, 0x33911530, 0x1, 0xc005b8e090, 0xc0059441a0)
_cgo_gotypes.go:9444 +0x4c fp=0xc000044e08 sp=0xc000044de0 pc=0x422298c
tinygo.org/x/go-llvm.TargetMachine.EmitToMemoryBuffer.func1({0x3?}, {0xc?}, 0x2?, 0xc0000285bf?, 0xc0059441a0)
/Users/runner/go/pkg/mod/tinygo.org/x/[email protected]/target.go:273 +0xe5 fp=0xc000044e68 sp=0xc000044e08 pc=0x423fb45
tinygo.org/x/go-llvm.TargetMachine.EmitToMemoryBuffer({0x6ca64c1c?}, {0xc00004a900?}, 0x44f20?)
/Users/runner/go/pkg/mod/tinygo.org/x/[email protected]/target.go:273 +0x65 fp=0xc000044ee0 sp=0xc000044e68 pc=0x423f925
github.com/tinygo-org/tinygo/builder.Build.func6(0xc001265aa0)
/Users/runner/work/tinygo/tinygo/builder/build.go:626 +0x94 fp=0xc000044f70 sp=0xc000044ee0 pc=0x431aa74
github.com/tinygo-org/tinygo/builder.runJob(0xc001265da0, 0x0?)
/Users/runner/work/tinygo/tinygo/builder/jobs.go:222 +0x4f fp=0xc000044fc0 sp=0xc000044f70 pc=0x432802f
github.com/tinygo-org/tinygo/builder.runJobs.func2()
/Users/runner/work/tinygo/tinygo/builder/jobs.go:123 +0x2a fp=0xc000044fe0 sp=0xc000044fc0 pc=0x432796a
runtime.goexit()
/Users/runner/hostedtoolcache/go/1.18.1/x64/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc000044fe8 sp=0xc000044fe0 pc=0x4067061
created by github.com/tinygo-org/tinygo/builder.runJobs
/Users/runner/work/tinygo/tinygo/builder/jobs.go:123 +0x5db
goroutine 1 [chan receive]:
github.com/tinygo-org/tinygo/builder.runJobs(0x829b780?, 0xc002534570?)
/Users/runner/work/tinygo/tinygo/builder/jobs.go:132 +0x605
github.com/tinygo-org/tinygo/builder.Build({0x7ff7bfeff6cd, 0xa}, {0x7ff7bfeff6a8, 0x17}, 0xc0001de2d0, 0xc005ad17b0)
/Users/runner/work/tinygo/tinygo/builder/build.go:883 +0x32cd
main.Build({0x7ff7bfeff6cd, 0xa}, {0x7ff7bfeff6a8, 0x17}, 0xc00014c2c0)
/Users/runner/work/tinygo/tinygo/main.go:156 +0x18c
main.main()
/Users/runner/work/tinygo/tinygo/main.go:1443 +0x3b95
I've ended up managing to make a working example which uses other exported functions to access the memory model (different from what I've found around). Is this the correct way of doing or am I missing something? https://github.com/brunoluiz/imagewand/pull/4
It works, but it doesn't seem the best way of doing this. The advantage here is that appendToBuffer grows the size of the buffer dynamically, but there must be a way to do so in the JavaScript runtime, right?
Besides, I am not sure if the reset() really frees resources correctly.
https://github.com/brunoluiz/imagewand/pull/4/files#diff-919e53cf74ef9971dcfcca1fa3a15e332fc8176aea0a645505f4722ade400988R15-R18
I would recommend against exporting functions that deal with one byte at a time, as that may be the reason why things are getting overloaded. There's an example here of passing an image back to the host (go), though similar can be done in node if you need to use that. https://github.com/codefromthecrypt/ebiten-wasm-graphics/tree/update-wazero
Also, I see in this issue mention of "wasi", despite the description showing the target being "wasm". You might want to rephrase the description based on latest, or close it out if you've found another way since.
Another thing that might help is when you are dealing with memory, the memory is also exported to the host as the name "memory". When you deal with bytes at a time, there are times working with memory directly is better. here are some more hints https://wazero.io/languages/tinygo/#memory
ps the underlying issue seems related to this (is this a dupe @aykevl @dgryski? https://github.com/tinygo-org/tinygo/issues/3019)
I would recommend against exporting functions that deal with one byte at a time, as that may be the reason why things are getting overloaded. There's an example here of passing an image back to the host (go), though similar can be done in node if you need to use that. https://github.com/codefromthecrypt/ebiten-wasm-graphics/tree/update-wazero
I know that is not ideal. I wanted to have a working implementation of the project using WASI, hence I was trying to hack something around 😅 I took it from this website but, in this example, the buffer is quite small.
I couldn't find an official example on how to make this work with WASI, which is where manipulating the memory seems almost a requirement, mostly due to the limitations around input and output types.