wit-bindgen-rust: remove "standalone" mode in the future

[peterhuene]

PR #193 merges in an Opts::standalone flag intended to assist in generating code into standalone source files, with the ultimate intention being to precompile those files into external crates referenced by a user's component project.

One of the motivators for doing so is to give a close-as-possible experience to cargo today for consuming dependencies from a registry, but instead of referencing Rust crates, one references components and the bindings are generated into external crates. This leads to a more natural way of consuming component dependencies for Rust developers.

For example:

use backend::Backend;

struct Component;

impl Backend for Component {
    fn fetch(url: String) -> Vec<u8> {
        if let Some(data) = cache::get(&url) {
            return data;
        }

        let data = origin::fetch(&url);
        cache::put(&url, &data);
        data
    }
}

backend::export!(Component);

With the above, the backend, cache, and origin crates are all from generated bindings based on references in Cargo.toml.

See this gist for the currently generated code for the above example in "standalone" mode.

The generated backend::export! macro is used to declare that the Component type exports the backend interface. This is also needed to allow for the possibility of having a singular component type which exports multiple component interfaces.

However, we don't want to have two different "modes" for code generation for Rust. To make the above example work for the existing proc-macros and for generating code into standalone crates, we would need to remove all dependencies on super from the generated code.

This presents a problem for supporting resources however:

• HandleType is implemented for an expected type name in super.
• LocalHandle is implemented for an expected type name in super.
• canonical_abi_drop_<resource> implementation and the generated trait's drop_<resource> method are implemented for an expected type name in super.
• Calling static or resource methods from the exported functions is implemented for an expected type name in super.

The problem also presents itself for any types that contain a handle to a resource, as the type's definition needs to know the user's implementation type for the resource, which is not present in the bindings crate (it's the same reason the generated export! macro exists in the above example to provide the backend crate with the name of the type implementing the Backend trait to call).

If and when we redesign resource code generation to accommodate for standalone bindings crates, we should remove the "standalone" option and have a single output for Rust bindings.

[Michael-F-Bryan]

You might find our story interesting.

For some background, we've currently got 20+ implementations for a particular set of WIT files, and not being able to put wit_bindgen_export!() in a shared crate was becoming a real pain (can't share code for trait implementations, can't use ?, no convenience methods, code is super verbose, etc.). Wasmer also haven't updated their fork of wit-bindgen in quite a while so we couldn't use #193 without switching our WebAssembly VM from Wasmer to Wasmtime.

To deal with this, we took inspiration from how Rust's #[global_allocator] works...

First, we call wit_bindgen_rust::export!() in some shared crate.

// support/src/guest/bindings.rs

wit_bindgen_rust::import!("../wit-files/rune/runtime-v2.wit");
wit_bindgen_rust::export!("../wit-files/rune/proc-block-v2.wit");

Then we created a trait which represents our resource and would be implemented downstream.

// support/src/guest/proc_block.rs

pub trait ProcBlock {
    fn tensor_constraints(&self) -> TensorConstraints;
    fn run(&self, inputs: Vec<Tensor>) -> Result<Vec<Tensor>, RunError>;
}

Next comes extern "Rust" definitions for our ProcBlock's constructor and a free function the host uses to find out more about its functionality (name, description, input tensors, etc.).

// support/src/guest/proc_block.rs

extern "Rust" {
    fn __proc_block_metadata() -> Metadata;
    fn __proc_block_new(
        args: Vec<Argument>,
    ) -> Result<Box<dyn ProcBlock>, CreateError>;
}

From there, we can write a shim implementation in our support crate for the ProcBlockV2 trait that wit-bindgen generates.

// support/src/guest/proc_block.rs

struct ProcBlockV2;

impl proc_block_v2::ProcBlockV2 for ProcBlockV2 {
    fn metadata() -> Metadata {
        logging::initialize_logger();
        unsafe { __proc_block_metadata() }
    }

    fn create_node(
        args: Vec<Argument>,
    ) -> Result<wit_bindgen_rust::Handle<self::Node>, CreateError> {
        logging::initialize_logger();
        let proc_block = unsafe { __proc_block_new(args)? };
        Ok(Handle::new(Node(Box::new(proc_block))))
    }
}

pub struct Node(Box<dyn ProcBlock>);

impl proc_block_v2::Node for Node {
    fn tensor_constraints(&self) -> TensorConstraints {
        self.0.tensor_constraints()
    }

    fn run(&self, inputs: Vec<Tensor>) -> Result<Vec<Tensor>, RunError> {
        self.0.run(inputs)
    }
}

And to wrap it all up, we have a macro which end users can use to generate __proc_block_metadata() and friends.

// support/src/guest/proc_block.rs

/// Tell the runtime that a WebAssembly module contains a proc-block.
#[macro_export]
macro_rules! export_proc_block {
    (metadata: $metadata_func:expr, proc_block: $proc_block:ty $(,)?) => {
        #[doc(hidden)]
        #[no_mangle]
        pub fn __proc_block_metadata() -> $crate::guest::Metadata { $metadata_func() }

        #[doc(hidden)]
        #[no_mangle]
        pub fn __proc_block_new(
            args: Vec<$crate::guest::Argument>,
        ) -> Result<Box<dyn $crate::guest::ProcBlock>, $crate::guest::CreateError> {
            fn assert_impl_proc_block(_: &impl $crate::guest::ProcBlock) {}

            let proc_block = <$proc_block>::try_from(args)?;
            assert_impl_proc_block(&proc_block);

            Ok(Box::new(proc_block) as Box<dyn $crate::guest::ProcBlock>)
        }
    };
}

I really like this approach because, besides the export_proc_block!() macro, it all looks like normal Rust code and you don't need to know about wit-bindgen types (see here for an example). It also solves the awkward super issue where downstream crates need to somehow inject their types and free function implementations into a crate higher up the dependency tree.

[alexcrichton]

I think this is done now, so closing.