Unified coroutines a.k.a. Generator resume arguments

[semtexzv]

//Edit after a long time: For everyone, I suggest you read https://github.com/rust-lang/lang-team/issues/49, it summarizes state of things, and explains ideas explored in this RFC and subsequent discussion.

This RFC outlines a way to unify existing implementation of the Generator feature with the Fn* family of traits and closures in general. Integrating these 2 concepts allows us to simplify the language, making the generators 'just pinned closures', and allows implementation of new patterns due to addtional functionality of generators accepting resume arguments.

Generator resume arguments are a sought after feature, since in their absence the implementation of async-await was forced to utilize thread-local storage, making it no_std.

Rendered

The RFC builds upon original coroutines eRFC

Main contention points:

• Syntax of arguments changing between yields (explicit vs implicit) and the interaction with lifetimes.
• Use of tuples and connection to closures, and the interaction with yield being an expression which resolves to arguments passed to resume.

Examples of new patterns designed with proposed design of the generator trait & Feature:

• Re-implementation of Sink in terms of generators by @Nemo157 : https://play.rust-lang.org/?version=beta&mode=debug&edition=2018&gist=c1750a50fbeff78200537d6f133ecb6e

[RustyYato]

Another view of generators is that they are a more general form of Iterator. With this in mind, under future possibilities could we add combinators that can pass the output of one generator/iterator to he input of another generator? Basically baking the input arguments.

[RustyYato]

@semtexzv I was thinking something along the lines of, this

[Nemo157]

Similar to core::future::Future experimenting with combinators outside the standard library as part of futures I believe it would make sense to leave Generator combinators up to an external library to start (and by the time Generator might be approaching stabilisation we'll hopefully have some examples/experience with moving combinators into the standard library with Future).

[SimonSapin]

I feel that the RFC as currently written spends a lot of “weirdness budget” in a way that is not necessary. In “normal” closures, the bindings introduced for arguments (between the two | pipes) exist from the very start of the closure’s body, which matches their location in source code. This RFC proposes the same location for introducing names that are not available until the first yield, and then can change value at each yield. (This leads to questions like what if the previous value was borrowed?)

How about this alternative?

• Like in the current RFC, the Generator trait gains a new type parameter. Except I’ll call it ResumeArg (singular)

• ResumeArg can be any type, not only a tuple. (Although it might default to (), and using a tuple would be an idiomatic way to pass multiple values.)

• yield is an expression of type ResumeArg. That’s it. It is not concerned at all with name bindings. You can use that expression in a let $pattern = $expr;, but that’s not necessary. It could be passed as a function argument, or anything that accepts an expression.

• There is no way to specify the ResumeArg type as part of generator literal syntax. (Nothing is allowed between the pipes when yield is used.) This RFC and its proposed alternatives seem to try hard to find such a way, but that may not be necessary: it can be a type variable left for inference to resolve (based both on what the generator’s body does with its yield expression, and how the generator is used).

  There is nearby precedent for this: generator literals don’t have syntax to specify the Generator::Yield associated type either. And the -> $type syntax that specifies Generator::Return is optional.

[semtexzv]

@SimonSapin You raise an important point. But, like closures, the arguments are available from the start of the generator, since generator is created suspended at the start, like closures, and the first resume starts the generator from the start. The only difference compared to FnMut is that generator has multiple return points, and before each return, it stores its state.

Instead of a generator as it stands today, think about a pinned FnMut closure.

[SimonSapin]

Hmm I may have been mislead by this part of the RFC:

Notice that the argument to first resume call was unused,

I took this to mean that the proposed language semantics are that the argument to the first resume call is always dropped, because that call doesn’t have a corresponding yield expression. But maybe instead you only meant that this particular example generator does nothing with the argument of the first resume call?

If we consider that this first value passed to resume doesn’t need to be dropped, then indeed the “closure argument” syntax would be the appropriate way to make them available to the generator’s body.

So I think preference is closest to what is currently listed as alternative # 2. I’ll quote it below and comment inline:

2. Creating a new binding upon each yield

I’d still phrase this as: yield is an expression, whose value may or may not be bound to a name with let.

let gen = |name :&'static str| {

    let (name, ) = yield "hello";
    let (name, ) = yield name;
}

We are creating a new binding upon each yield point, and therefore are shadowing earlier bindings.

Yes about shadowing. But again, creating a binding doesn’t have to be mandatory.

This would mean that by default, the generator stores all of the arguments passed into it through the resumes.

Not necessarily, even with a binding. The value won’t be stored in the generator if it has been moved or dropped before the next yield. Like any other value manipulated on the stack of a generator.

Another issue with these approaches is, that they require progmmer, to write additional code to get the default behavior. In other words: What happens when user does not perform the required assignment ? Simply said, this code is permitted, but nonsensical:

let gen = |name: &static str| {
    yield "hello";
    let (name, ) = yield name;
}

This is not nonsensical at all. Like any $expr; statement, the first yield simply drops its resume value.

Another issue is, how does this work with loops ? What is the value assigned to third in following example ?

let gen = |a| {
    loop {
        println!("a : {:?}", a);
        let (a,) = yield 1;
        println!("b : {:?}", a);
        let (a,) = yield 2;
    }        
}
let first = gen.resume(("0"));
let sec = gen.resume(("1"));
let third = gen.resume(("2"));

third is the integer 1. But I think the intended question was: what line is printed third? The answer to that is a : "0". The let bindings shadowed a for the rest of the lexical scope (which is the rest of the loop). When the loop does its next iteration, control goes back out of the scope of shadowing a’s and the initial a is visible again. The same would happen with a similar loop outside of a generator.

[SimonSapin]

In short, I feel strongly that yield should be a stand-alone expression, not necessarily tied to name bindings.

---

Separately from the above, I feel there is a choice between:

• (As in the current RFC), the value passed to resume has to be a tuple. For the first resume call, this maps very nicely to closure argument syntax with any number of argument. However for subsequent calls, in the case of a 1-tuple we’d need somewhat-awkward unpacking like let (name,) = yield; foo(name).

• Or, the value passed to resume can be any type, and yield expressions have that type. let name = yield; foo(name) or even foo(yield) looks much nicer without tuple unpacking. However this forces the “closure argument” syntax to only have one argument. || {…} (zero argument) could be syntactic sugar for |_: ()| {…} (one argument of type unit-tuple). (Only for generators of course, not normal closures.) It’s for the initial bindings of multiple values as one tuple argument that this becomes awkward: |(foo, bar): (u32, &str)| {…} instead of |foo: u32, bar: &str| {…}

• Trying to reconcile both nice-to-haves creates inconsistencies or discontinuities that are undesirable IMO. For example, does |foo: u32| { yield } implement Generator<u32> or Generator<(u32,)>? Why?

[semtexzv]

Well, the argument HAS to be a tuple, and it HAS to be unpacked inside the argument list of the generator, since it is the behavior of closures, and deviating from this behavior would most certainly be a mistake. I again, point out the Fn* family of traits.

As for the third point, the argument would most certainly have to be a tuple. (Interaction of Fn traits and closures).

But yes, yield being an expression is one approach. The issue I have with it, is that it does not provide unified way to accept arguments upon the generator start, and it's resume. You have argument list at the start, and then a tuple at the resume. If we had Tuple unpacking/packing, we could resolve this, and I think that solution would be one of the best.

But there is something to be said about the default choice. Is it a good default to drop the values passed into resume ?

I accept syntactic inconvenience, but I think, the deviation from concepts introduced in closures is a huge mistage.

[Nemo157]

Not necessarily, even with a binding. The value won’t be stored in the generator if it has been moved or dropped before the next yield. Like any other value manipulated on the stack of a generator.

This would imply you need to scope every yield call (even moving the value out is not enough, see https://github.com/rust-lang/rust/issues/57478), generators are like normal code and drop their values at the end of scope. (Although there are optimizations applied to drop values early if that is not observable).

[SimonSapin]

@semtexzv

HAS to be

Yes, the goal of making generators and closures as close to each other as possible leads to arguments being a tuple. But I’m personally not very attached to that goal in the first place.

Closures have a whole family of traits with Fn, FnMut, and FnOnce. They are useful for reasons that doesn’t apply to generators. Fn taking &self doesn’t work for generators since resume always mutates (at least to track initial state v.s. each yield point v.s. returned). FnOnce defeats the point of having a generator in the first place.

@Nemo157 https://github.com/rust-lang/rust/issues/57478 is an implementation bug to be fixed, right?

[Nemo157]

Even if https://github.com/rust-lang/rust/issues/57478 is fixed I would expect there to be a lot of generators that just shadow their existing bindings without moving out of them, having those all be kept alive to be dropped at the end would not be good, e.g.

|arg: String| {
    let arg = yield;
    let arg = yield;
    let arg = yield;
}

would have to keep all 4 strings live until dropped when the generator completes.

[semtexzv]

@SimonSapin Yes, this zealotry for theoretical purity has drawbacks. But I believe this is the right choice in this case. The connection to FnOnce provides a way we could extend current compiler mechanics which apply to closures. And the RFC mentions possible FnPin trait which would extend the function trait hierarchy to connect it fully to generator trait.

This has to do with the future plans part of the RFC, the ultimate goal being the generator callable as a closure is. To ensure this is possible in the future, we have to design the generator trait in this way, and it introduces friction between :

1. yield being and expression and returning a tuple of the same type as the argument list written out between the closure syntax ( the 'pipes' )
2. yield being a statement, and the arguments changing values between yield points.

Simply, the 'sound', while not pragmatically correct choice now, will have implications down the road, that will allow us to make more pragmatic choices.

[SimonSapin]

@Nemo157 Yes, and? That preserving destruction order can cause unexpected generator size increase is not at all unique to values that yield evaluates to. For example:

|| {
    let foo = String::from(…);
    yield;
    let foo = String::from(…);
    yield;
    let foo = String::from(…);
    yield;
}

So giving yield weird semantics is not the way to fix this IMO. For this particular issue I feel that there is no substitute for education and reminding people that shadowing does not drop. Your example is easy enough to fix:

|mut arg: String| {
    arg = yield;
    arg = yield;
    arg = yield;
}

@semtexzv

The connection to FnOnce provides a way we could extend current compiler mechanics which apply to closures.

Sorry, I don’t understand what that means.

[semtexzv]

I meant the interaction between Fn* traits, and how are closures invoked. There have to be some mechanics for packing/unpacking the arguments passed into the closure.

// Edit: i meant the extern "rust-call" specified in the trait method and compiler desugaring fun(arg1,arg2) into fun.call((arg1,arg2)).

Also, the approach of assignment would be preferable. But it breaks down with multiple arguments.

[Boscop]

@SimonSapin I think it'd be much more preferable if one wasn't forced to make things mutable (or to wrap the yield in braces), just to prevent size increase of the generator.

[CAD97]

I came into this expecting I wasn't going to like the argument rebinding. But I think that the parallel with regular FnMut is the correct direction to take, at least if generator syntax continues to look like closures.

Sure, in the case of one argument, you can use mut and rebind, but what about with two or more?

|mut one: String, mut two: String| {
    (one, two) = yield; // ???
}

I think a view of generator syntax as "just" sugar over "FnPin" is the correct place to aim for. Consider the following:

|one: String, two: String| {
    let both = one.clone() + &two;
    loop {
        dbg!((&both, &one, &two));
        yield;
    }
}

"Desugared" to a "FnPin"-like syntax:

pin |one: String, two: String| -> GeneratorState<Return=!, Yield=()> {
    match self.state {
        0 => {
            self.both = one.clone() + &two;
            dbg!(&self.both, &one, &two);
            self.state = 1;
            Yield(())
        }
        1 => {
            dbg!(&self.both, &one, &two);
            self.state = 1;
            Yield(())
        }
    }
}

In a yield-value syntax:

|mut one: String, mut two: String| {
    let both = one.clone() + &two;
    loop {
        dbg!(&both, &one, &two);
        let resume = yield;
        one = resume.0;
        two = resume.1;
    }
}

Rebinding the arguments' names on yield may seem foreign at first, but it makes sense if you think of a generator as a resumable coroutine; the arguments are whatever they were to that specific resumption call. (Though to be fair, it only really makes sense if we eventually get resume spelled as gen(one, two), not gen.resume((one, two)). Mimicking extern "rust-call"'s API here makes sense to me.)

I think it's easier to teach that if you want a previous resumption's arguments you need to bind them locally than that you need to explicitly drop this resumption's arguments if you don't want them increasing your generator size. (Explicitly store them, implicitly drop, rather than implicitly store them, explicitly drop.)

Re-issuing the arguments solves the problems of how to handle multiple arguments with drop glue nicely. It also gives us a sensible way to handle passing a borrow to resume, I believe:

Add a new specially treated lifetime, 'yield. A lifetime passed to a generator literal has '_ = 'yield; 'yield doesn't need to exist in the surface language, just in MIR. To the body of the generator, &'yield is always valid, but copying it out of the argument bindings makes the other binding not live over yield points. Lifetime checking is done by the MIR borrowck phase on the CFG of the generator, which gets a new &'yield binding by the same name at every yield point.

I can write up a more formal RFC-style of this interpretation if people think it might be useful.

[CAD97]

Falling from this is an obvious question: what about generator free fn, not closures? And I think the answer should be that we don't have them.

In free function position, how would you interact with a generator?

fn foo(one: String, two: String) -> ! yield () {
    let both = one.clone() + &two;
    loop {
        dbg!(&both, &one, &two);
        yield;
    }
}

fn main() {
    foo("1".to_string(), "2".to_string()); // ???
    foo("3".to_string(), "4".to_string()); // ??????
}

The pinning and state requirement makes me think that always using closure syntax makes more sense:

fn foo() -> impl Generator<Return=!, Yield=()> {
    |one: String, two: String| {
        // blah blah blah yield
    }
}

(Honestly, an extension of this train of logic gets me thinking that maybe not having async fn wouldn't be so bad, and just have async closures. But that doesn't matter anymore.)

While the transform for async fn makes sense (async fn(Args) -> Ret => fn(Args) -> impl Future<Item=Ret> + '_), the transform for "yield fn" would be a lot more questionable (yield fn(Args) -> Ret yield Yield => fn() -> impl Generator<Return=Ret, Yield=Yield>? const impl Generator<Return=Ret, Yield=Yield>?).

Because of this, we don't have to worry about what rebinding acts like in free fn in a rebinding world. It's just the closure syntax that exists, and my argument is that creating new 'yield bindings accessed with the same name declared in the arguments list at each yield point, while foreign, is easier to teach on its own merit than yield returning a value on resume and generator size ballooning unless you know this One Weird Trick To Optimize Your Generators.

[semtexzv]

@CAD97 Thank you, I'd be thrilled if you explained how could the type checking work, since i'm not that qualified in this area. I'm going to reformat the RFC in a way that presents the proposed approach in full before examining other approaches, and more information about how would the type checking work would be invaluable.

[rpjohnst]

One thing to keep in mind is that the generator argument used by async will be a reference- this means it has none of problems of drop order or generator size. (Though of course its use by the async desugaring wouldn't be problematic either way because we could simply define away the problem in the desguaring.)

Someone directly writing an equivalent generator won't even be able to accidentally keep one around, because its lifetime is restricted to a single resume.

[rpjohnst]

On another note, I think it is much more important for generators to unify with Rust's various "effects," than it is to unify with closures.

First, like SimonSapin mentions, we gain very little by sharing the Fn traits- only FnMut really matches, and writing that mapping explicitly doesn't cost much.

Second, like the earlier question of whether to unify generators with async, unification prevents us from treating the two as orthogonal, combinable language features. Specifically, what if we want generator functions (and closures), similar to async functions (and closures)?

That is, we should track generator-ness like we track async-ness, and forbid programs from accidentally using generators as non-generators. This is a more useful consistency that facilitates composition with other "effects," and potentially leaves the door open for effect polymorphism.

[arcnmx]

FWIW I'm also in favour of yield simply being an expression, and the points regarding shadowing being easy to misuse don't bother me for reasons already discussed; the rest of the language consistently works in the same way, and moved/non-Drop values should presumably be optimized out of the generated state machine storage even if they aren't today. I do feel like the larger concern here is how to handle lifetimes that do not cross yield points, and a way to express that lifetime may guide which syntax is most appropriate? It does match what you'd expect from a for<'a> FnPin(&'a Context), just feels awkward to explicitly convey this somehow at the generator definition... for<'a> |&'a Context| { ... } I guess?

---

Also as a side note, what if generators are just closures that produce a generator, and thus can have execution before the first resume? The argument list would just be a way to move initial state into the generator I guess, and I'm not sure if this is a more or less weird of an approach to avoid unifying argument list and resume arg type:

fn x(start: usize) -> impl Generator<&'static str, Yield=bool, Return=u32> {
    // Possibly `FnOnce` instead if it captures something non-Copy?
    let x: impl Fn(usize) -> impl Generator = |start| {
        println!("running before first yield with {}", start);
        let first = yield;
        println!("second resume passed me {}", yield start == first.len());
        5u32
    };
    x(start)
}

let y = x(5);
// first println happens above!
let GeneratorState::Yield(true) = y.resume("hello");
let GeneratorState::Complete(5) = y.resume("hi");
// etc...

Or in a sugared form...

    yield fn x(start: usize) yield(&'static str) bool -> u32 {
        println!("running before first yield with {}", start);
        let first = yield;
        println!("second resume passed me {}", yield start == first.len());
        5u32
    }

    /* alternatively, sugared items could simply be
       || generators without early execution more in line with the RFC's shadowing
       alternative where the arg list unifies with the resume args, which then becomes inconsistent
    */

I suppose the need for having an explicit yield at the beginning makes it too weird though, and I even confused myself trying to reason about it! So I guess my personal leaning is toward the shadowing alternative rather than the "magic" one.

[Matthias247]

I agree on the comments regarding changing bindings being too much magic.

Isn't yield returning resume args already used in other languages and therefore a somewhat reasonable solution?

Another idea:

In order to mark the temporary nature of resume args, why not explicitly pass them through accessor functions, which would return most recent passed resume arg to the generator:

let gen = |name: &dyn Fn() -> 'static str| {
    yield "Hello";                
    yield name();
    return name();
}

That would be in-line with any other Rust code.

Non Fn parameters could be either rejected, or those would be the parameters that are only passed when the generator is started (are moved once inside the generator), and not passed on every resume.

[RustyYato]

Isn't yield returning resume args already used in other languages and therefore a somewhat reasonable solution?

In python, but given that python is dynamically typed it may work better for them than it would in Rust.

[Matthias247]

I was also thinking about the same issue that already came up in the discussion between @Nemo157 and @SimonSapin: The scoping/lifetimes with yield as expression is quite "uncommon".

Going back to the example and modifying it slightly the following questions arise:

let gen = |name :&'static str| {
    let (name1, ) = yield "hello";
    // Is name here still valid?
    let (name2, ) = yield name1;
    // Is name2 here still valid?
}

I think the answer is no. Otherwise the requirements regarding lifetimes for callers of resume() would be extremely hard to describe. Everything which is passed to resume() at one point of time would need to be marked as borrowed for the lifetime of the generator. Which is most likely not what we want. Rather we want things to be only borrowed for the duration of the resume call.

In order to make that work, one would need to explain to the borrow-checker that lifetimes of bindings which are produced by a yield end at the next yield.

Or it could be made a bit more obvious by enforcing scoping in the generator. E.g. by making yield taking a closure which gets passed the resume arg:

let gen = |name :&'static str| {
    yield("hello", |name1: &'static str| {
        yield(name1, |name2: &'static str| {})
    });
}

Now this isn't super nice either - and most of all it also doesn't solve the issue that name1 could be utilized after the next yield returned. Maybe it can work out if an extra rule is introduced that yield is only allowed as the last statement inside a yield closure, so that the parameter can't be utilized anymore after it.

Whether that's a great solution -> Don't know.

[SimonSapin]

Going back to the example and modifying it slightly the following questions arise:

let gen = |name :&'static str| {
    let (name1, ) = yield "hello";
    // Is name here still valid?
    let (name2, ) = yield name1;
    // Is name2 here still valid?
}

I think the answer is no. Otherwise the requirements regarding lifetimes for callers of resume() would be extremely hard to describe. Everything which is passed to resume() at one point of time would need to be marked as borrowed for the lifetime of the generator. Which is most likely not what we want. Rather we want things to be only borrowed for the duration of the resume call.

Yes, in this example all variables should be available until the end of the scope. Why wouldn’t they be? And yes, this means that the generator type needs to have a lifetime parameter and borrow the resume arguments. (If they have lifetimes other than 'static, unlike this example.) A way to enforce that would be to make ResumeArg an associated type rather than a type parameter of the trait:

impl Generator for MyGenerator {
    type ResumeArg = &'a str;
    // …
}

The above cannot use 'a because it is not in scope, so it needs to be added to the impl block:

impl<'a> Generator for MyGenerator {
    type ResumeArg = &'a str;
    // …
}

This is also now allowed because the 'a has an unused parameter. If the Generator trait doesn’t accept parameters, the only way to make that parameter be used is therefore to parameterize the type:

enum MyGenerator<'a> {
    Initial,
    Yield1 { name: &'a str },
    Yield2 { name: &'a str, name1: &'a str },
    Returned,
}

impl<'a> Generator for MyGenerator<'a> {
    type ResumeArg = &'a str;
    // …
}

[semtexzv]

Edit: Added an alternative design found on rust forums, please re-read the RFC.

For multiple different yield poins you'd have to add multiple lifetime arguments, so I don't think this is the right choice. Previous attempts went this way, and met the same issues, ultimately resolving to using the same hacks to solve the lack of GATs which could be used to solve lifetime issues.

[Matthias247]

Edit: What is here is wrong. See clarifications from @rpjohnst below.

Yes, in this example all variables should be available until the end of the scope. Why wouldn’t they be?

~~I think this doesn't work great for a variety of reasons:~~

~~1. I don't see how this would look the caller, and how it would interact with the borrow-checker.~~

~~All arguments that are passed to resume() need to be permanently consumed - which is not something that is a known concept. And things like the following code wouldn't be valid:~~

let mut data: String = "hello".to_owned();
gen.resume(&data); // Now gen internally captures a reference to the original string
data += " world"; // This invalidates the reference
gen.resume(&data); // Now gen would have an invalidated and a new reference

~~2. It's unclear how this would interact with control flow in generators.~~

~~E.g. something along~~

let gen = |name :&'static str| {
    let (name, done) = yield "hello";
    while !done  {
        let (name, done) = yield name;
    }
}

~~If this follows the rule to capture everything, then the generators size would be infinite.~~

~~If the binding is the same for all iterations of the loop so that only 2 variants are captured, then the behavior on the resumer side is unclear. For some resumes the argument is only borrowed for the duration of the resume, for others for a few resume calls until it is released again, and yet for others it is the whole lifetime of the generator.~~

~~This also makes any change inside the generator non API compatible anymore, since introducing any control flow would change the lifetimes.~~

3. ~~It increases generator size, even if applications don't require parameters across yield points~~

~~Lots of applications for resume args, e.g. passing Context for Futures, don't require any parameters across yield points. Passing them again is totally fine - or even expected, since things might have changed. Storing more - for the sake that some applications might require it - seems to be the wrong default. If some generators need data again they could just pass it again into the next resume call.~~

~~We could argue that if things are not used across yield points they are optimized away by the compiler. However in practice that seems to be tricky. For async fn there were already lots of questions when parameters are allowed to be dropped, in order to be semantically compatible with non async code. Here the latest changes were to keep the parameters as long as possible, which increases the Future sizes.~~

[rpjohnst]

The "yield evaluates to the resume argument" approach doesn't have any of those problems. And it doesn't have them precisely because it matches the behavior of normal Rust code more closely!

And things like the following code wouldn't be valid:

let mut data: String = "hello".to_owned();
gen.resume(&data); // Now gen internally captures a reference to the original string
data += " world"; // This invalidates the reference
gen.resume(&data); // Now gen would have an invalidated and a new reference

You can already write this in stable, non-generator Rust, and it handle both the cases of the generator keeping the argument and the generator dropping the argument. This simply needs to be encoded into the type of resume:

• If the type of resume is fn resume<'a>(&mut Self<'a>, &'a str) then the the borrow checker forbids the program above: https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=5287660952409cf4b44386edf0a55ffa
• If the type of resume is fn resume<'a>(&mut Self<'a>, &str) then the borrow checker allows the program above: https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=0f6660c3450cfcb0b3b9c3f48a646b08

let gen = |name :&'static str| {
    let (name, done) = yield "hello";
    while !done  {
        let (name, done) = yield name;
    }
}

If this follows the rule to capture everything, then the generators size would be infinite.

No, the bindings created in the loop body are dropped at the end of the iteration, and thus don't live across any yield point. (This also doesn't do what you want, for the same reason.)

If the binding is the same for all iterations of the loop so that only 2 variants are captured, then the behavior on the resumer side is unclear. ...

This also makes any change inside the generator non API compatible anymore, since introducing any control flow would change the lifetimes.

This is already the case for async/await, and is why Pin exists. This is just a fundamental part of how structs-with-references work in Rust today, you can't get around that. Even with the "magic mutating arguments" approach, the generator author has the same decision to make, because they can move the value into a local that does live across yield points.

Fortunately, as above, the resumer-side behavior can be determined by looking at the type of resume in the particular generator impl.

It increases generator size, even if applications don't require parameters across yield points

It does not, please stop making this claim. The counterexamples you bring up all have to do with Drop, which simply does not apply to reference arguments like Future's Context. And when a generator argument does implement Drop, matching sync code is good and a large part of why this design makes sense. If size is a problem here the author can simply drop the argument manually, just like they do in sync code.

[Matthias247]

Thanks for the clarifications. I wasn't aware of this kind of borrowing, and that it's intended to configure the borrowing mode via explicit type annotations.

I think more complete examples how that would look like could be helpful. I still can't fully follow where those things should go to (e.g. the example of @SimonSapin had a MyGenerator struct, but I imagined the actual struct to be compiler-generated).

[Nemo157]

• If the type of resume is fn resume<'a>(&mut Self<'a>, &'a str) then the the borrow checker forbids the program above: https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=5287660952409cf4b44386edf0a55ffa
• If the type of resume is fn resume<'a>(&mut Self<'a>, &str) then the borrow checker allows the program above: https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=0f6660c3450cfcb0b3b9c3f48a646b08

More accurately these signatures wouldn't be on the resume function as that's specified by the trait, but instead on the impl; impl<'a> Generator<(&'a str,)> for Self<'a> and impl<'a, 'b> Generator<(&'b str,)> for Self<'a>.

Problematically the former is saying that the argument must live for the same lifetime as the generator, there is no way I know of to say "the argument must be valid from now until the generator is dropped". For example this fails even though it could be valid, there's just (AFAIK) no way to write a set of signatures that satisfies the borrow checker.

This is all very straightforward from the caller side (it's identical to how the Fn* traits are used today), and works the same with all suggested syntaxes as they use the same trait definition. One issue with all the syntaxes is they don't have any way to actually indicate these lifetimes, if we start with the running example:

let gen = |_: &'static str| {
    let (name,) = yield "hello";
    let (_,) = yield name;
};

this is easily inferring the type impl Generator<(&'static str,), Yield = &'static str, Return = ()>.

We want to also be able to write a generator matching impl for<'a> Generator<(&'a str,), Yield = &'a str, Return = ()> (i.e. a generator that does not capture its input value) along with for<'a> impl Generator<(&'a str,), Yield = &'a str, Return = ()> + 'a (i.e. a generator that captures its input value, and so cannot outlive it). The syntax with an implicit lifetime could easily refer to either of these:

let gen = |_: &str| {
    let (name,) = yield "hello";
    let (_,) = yield name;
};

I think this could be valid under either interpretation, depending on whether you can propagate a lifetime between the previous yield-return and the current yield-argument. But, trying to store a value across a yield:

let gen = |name: &str| {
    let (_,) = yield "hello";
    let (_,) = yield name;
};

This could only be valid under the second interpretation, where implicit lifetimes are captured into the generator, rather than being only valid for the single resume call.