Make the computation of coroutine_captures_by_ref_ty more sophisticated by compiler-errors · Pull Request #123660 · rust-lang/rust (original) (raw)

compiler-errors

@rustbot rustbot added S-waiting-on-review

Status: Awaiting review from the assignee but also interested parties.

T-compiler

Relevant to the compiler team, which will review and decide on the PR/issue.

labels

Apr 9, 2024

compiler-errors

@compiler-errors

@compiler-errors

@bors bors added S-waiting-on-bors

Status: Waiting on bors to run and complete tests. Bors will change the label on completion.

and removed S-waiting-on-review

Status: Awaiting review from the assignee but also interested parties.

labels

Apr 11, 2024

bors added a commit to rust-lang-ci/rust that referenced this pull request

Apr 11, 2024

@bors

…iaskrgr

Rollup of 4 pull requests

Successful merges:

r? @ghost @rustbot modify labels: rollup

rust-timer added a commit to rust-lang-ci/rust that referenced this pull request

Apr 11, 2024

@rust-timer

Rollup merge of rust-lang#123660 - compiler-errors:coroutine-closure-env, r=oli-obk

Make the computation of coroutine_captures_by_ref_ty more sophisticated

Currently, we treat all the by-(mut/)ref borrows of a coroutine-closure as having a "closure env" borrowed lifetime.

When we have the given code:

let x: &'a i32 = ...;
let c = async || {
    let _x = *x;
};

Then when we call:

c()
// which, because `AsyncFn` takes a `&self`, we insert an autoref:
(&c /* &'env {coroutine-closure} */)()

We will return a future whose captures contain &'env i32 instead of &'a i32, which is way more restrictive than necessary. We should be able to drop c while the future is alive since it's not actually borrowing any data originating from within the closure's captures, but since the capture has that 'env lifetime, this is not possible.

This wouldn't be true, for example, if the closure captured i32 instead of &'a i32, because the 'env lifetime is actually necessary since the data (i32) is owned by the closure.

This PR identifies two criteria where we need to take the borrow with the closure env lifetime:

  1. If the closure borrows data from inside the closure's captures. This is not true if the parent capture is by-ref, OR if the parent capture is by-move and the child capture begins with a deref projection. This is the example described above.
  2. If we're dealing with mutable references, since we cannot reborrow &'env mut &'a mut i32 into &'a mut i32, only &'env mut i32.

See the documentation on should_reborrow_from_env_of_parent_coroutine_closure for more info.

important: As disclaimer states on that function, luckily, if this heuristic is not correct, then the program is not unsound, since we still borrowck and validate the choices made from this function -- the only side-effect is that the user may receive unnecessary borrowck errors.

Fixes rust-lang#123241

This was referenced

May 18, 2024

matthiaskrgr added a commit to matthiaskrgr/rust that referenced this pull request

May 22, 2024

@matthiaskrgr

fmease added a commit to fmease/rust that referenced this pull request

May 22, 2024

@fmease

rust-timer added a commit to rust-lang-ci/rust that referenced this pull request

May 22, 2024

@rust-timer

bors added a commit to rust-lang-ci/rust that referenced this pull request

Dec 13, 2024

@bors

…i-obk

Stabilize async closures (RFC 3668)

Async Closures Stabilization Report

This report proposes the stabilization of #![feature(async_closure)] (RFC 3668). This is a long-awaited feature that increases the expressiveness of the Rust language and fills a pressing gap in the async ecosystem.

Stabilization summary

async fn takes_an_async_fn(f: impl AsyncFn(&str)) {
    futures::join(f("hello"), f("world")).await;
}

takes_an_async_fn(async |s| { other_fn(s).await }).await;

Motivation

Without this feature, users hit two major obstacles when writing async code that uses closures and Fn trait bounds:

That is, for the first, we cannot write:

// We cannot express higher-ranked async function signatures.
async fn f<Fut>(_: impl for<'a> Fn(&'a u8) -> Fut)
where
    Fut: Future<Output = ()>,
{ todo!() }

async fn main() {
    async fn g(_: &u8) { todo!() }
    f(g).await;
    //~^ ERROR mismatched types
    //~| ERROR one type is more general than the other
}

And for the second, we cannot write:

// Closures cannot return futures that borrow closure captures.
async fn f<Fut: Future<Output = ()>>(_: impl FnMut() -> Fut)
{ todo!() }

async fn main() {
    let mut xs = vec![];
    f(|| async {
        async fn g() -> u8 { todo!() }
        xs.push(g().await);
    });
    //~^ ERROR captured variable cannot escape `FnMut` closure body
}

Async closures provide a first-class solution to these problems.

For further background, please refer to the motivation section of the RFC.

Major design decisions since RFC

The RFC had left open the question of whether we would spell the bounds syntax for async closures...

// ...as this...
fn f() -> impl AsyncFn() -> u8 { todo!() }
// ...or as this:
fn f() -> impl async Fn() -> u8 { todo!() }

We've decided to spell this as AsyncFn{,Mut,Once}.

The Fn family of traits is special in many ways. We had originally argued that, due to this specialness, that perhaps the async Fn syntax could be adopted without having to decide whether a general async Trait mechanism would ever be adopted. However, concerns have been raised that we may not want to use async Fn syntax unless we would pursue more general trait modifiers. Since there remain substantial open questions on those -- and we don't want to rush any design work there -- it makes sense to ship this needed feature using the AsyncFn-style bounds syntax.

Since we would, in no case, be shipping a generalized trait modifier system anytime soon, we'll be continuing to see AsyncFoo traits appear across the ecosystem regardless. If we were to ever later ship some general mechanism, we could at that time manage the migration from AsyncFn to async Fn, just as we'd be enabling and managing the migration of many other traits.

Note that, as specified in RFC 3668, the details of the AsyncFn* traits are not exposed and they can only be named via the "parentheses sugar". That is, we can write T: AsyncFn() -> u8 but not T: AsyncFn<Output = u8>.

Unlike the Fn traits, we cannot project to the Output associated type of the AsyncFn traits. That is, while we can write...

fn f<F: Fn() -> u8>(_: F::Output) {}

...we cannot write:

fn f<F: AsyncFn() -> u8>(_: F::Output) {}
//~^ ERROR

The choice of AsyncFn{,Mut,Once} bounds syntax obviates, for our purposes here, another question decided after that RFC, which was how to order bound modifiers such as for<'a> async Fn().

Other than answering the open question in the RFC on syntax, nothing has changed about the design of this feature between RFC 3668 and this stabilization.

What is stabilized

For those interested in the technical details, please see the dev guide section I authored.

Async closures

Other than in how they solve the problems described above, async closures act similarly to closures that return async blocks, and can have parts of their signatures specified:

// They can have arguments annotated with types:
let _ = async |_: u8| { todo!() };

// They can have their return types annotated:
let _ = async || -> u8 { todo!() };

// They can be higher-ranked:
let _ = async |_: &str| { todo!() };

// They can capture values by move:
let x = String::from("hello, world");
let _ = async move || do_something(&x).await };

When called, they return an anonymous future type corresponding to the (not-yet-executed) body of the closure. These can be awaited like any other future.

What distinguishes async closures is that, unlike closures that return async blocks, the futures returned from the async closure can capture state from the async closure. For example:

let vec: Vec<String> = vec![];

let closure = async || {
    vec.push(ready(String::from("")).await);
};

The async closure captures vec with some &'closure mut Vec<String> which lives until the closure is dropped. Every call to closure() returns a future which reborrows that mutable reference &'call mut Vec<String> which lives until the future is dropped (e.g. it is awaited).

As another example:

let string: String = "Hello, world".into();

let closure = async move || {
    ready(&string).await;
};

The closure is marked with move, which means it takes ownership of the string by value. The future that is returned by calling closure() returns a future which borrows a reference &'call String which lives until the future is dropped (e.g. it is awaited).

Async fn trait family

To support the lending capability of async closures, and to provide a first-class way to express higher-ranked async closures, we introduce the AsyncFn* family of traits. See the corresponding section of the RFC.

We stabilize naming AsyncFn* via the "parenthesized sugar" syntax that normal Fn* traits can be named. The AsyncFn* trait can be used anywhere a Fn* trait bound is allowed, such as:

/// In return-position impl trait:
fn closure() -> impl AsyncFn() { async || {} }

/// In trait bounds:
trait Foo<F>: Sized
where
    F: AsyncFn()
{
    fn new(f: F) -> Self;
}

/// in GATs:
trait Gat {
    type AsyncHasher<T>: AsyncFn(T) -> i32;
}

Other than using them in trait bounds, the definitions of these traits are not directly observable, but certain aspects of their behavior can be indirectly observed such as the fact that:

fn by_ref_call(c: impl AsyncFn()) {
    let fut = c();
    drop(c);
    //   ^ Cannot drop `c` since it is borrowed by `fut`.
}
fn by_ref_call(c: impl AsyncFnOnce()) {
    let fut = c();
    let _ = c();
    //      ^ Cannot call `c` since calling it takes ownership the callee.
}
fn is_async_fn(_: impl AsyncFn(&str)) {}

async fn async_fn_item(s: &str) { todo!() }
is_async_fn(s);
// ^^^ This works.

fn generic(f: impl Fn() -> impl Future<Output = ()>) {
    is_async_fn(f);
    // ^^^ This does not work (yet).
}

The by-move future

When async closures are called with AsyncFn/AsyncFnMut, they return a coroutine that borrows from the closure. However, when they are called via AsyncFnOnce, we consume that closure, and cannot return a coroutine that borrows from data that is now dropped.

To work around around this limitation, we synthesize a separate future type for calling the async closure via AsyncFnOnce.

This future executes identically to the by-ref future returned from calling the async closure, except for the fact that it has a different set of captures, since we must move the captures from the parent async into the child future.

Interactions between async closures and the Fn* family of traits

Async closures always implement FnOnce, since they always can be called once. They may also implement Fn or FnMut if their body is compatible with the calling mode (i.e. if they do not mutate their captures, or they do not capture their captures, respectively) and if the future returned by the async closure is not lending.

let id = String::new();

let mapped: Vec</* impl Future */> =
    [/* elements */]
    .into_iter()
    // `Iterator::map` takes an `impl FnMut`
    .map(async |element| {
        do_something(&id, element).await;
    })
    .collect();

See the dev guide for a detailed explanation for the situations where this may not be possible due to the lending nature of async closures.

Other notable features of async closures shared with synchronous closures

Lints

This PR also stabilizes the CLOSURE_RETURNING_ASYNC_BLOCK lint as an allow lint. This lints on "old-style" async closures:

#![warn(closure_returning_async_block)]
let c = |x: &str| async {};

We should encourage users to use async || {} where possible. This lint remains allow and may be refined in the future because it has a few false positives (namely, see: "Where do we expect rewriting || async {} into async || {} to fail?")

An alternative that could be made at the time of stabilization is to put this lint behind another gate, so we can decide to stabilize it later.

What isn't stabilized (aka, potential future work)

async Fn*() bound syntax

We decided to stabilize async closures without the async Fn*() bound modifier syntax. The general direction of this syntax and how it fits is still being considered by T-lang (e.g. in RFC 3710).

Naming the futures returned by async closures

This stabilization PR does not provide a way of naming the futures returned by calling AsyncFn*.

Exposing a stable way to refer to these futures is important for building async-closure-aware combinators, and will be an important future step.

Return type notation-style bounds for async closures

The RFC described an RTN-like syntax for putting bounds on the future returned by an async closure:

async fn foo(x: F) -> Result<()>
where
    F: AsyncFn(&str) -> Result<()>,
    // The future from calling `F` is `Send` and `'static`.
    F(..): Send + 'static,
{}

This stabilization PR does not stabilize that syntax yet, which remains unimplemented (though will be soon).

dyn AsyncFn*()

AsyncFn* are not dyn-compatible yet. This will likely be implemented in the future along with the dyn-compatibility of async fn in trait, since the same issue (dealing with the future returned by a call) applies there.

Tests

Tests exist for this feature in tests/ui/async-await/async-closures.

A selected set of tests:

Remaining bugs and open issues

Where do we expect rewriting || async {} into async || {} to fail?

let x: fn() -> _ = async || {};
fn needs_send_future(_: impl Fn(NotSendArg) -> Fut)
where
    Fut: Future<Output = ()>,
{}

needs_send_future(async |_| {});

History

Important feature history

Acknowledgements

Thanks to @oli-obk for reviewing the bulk of the work for this feature. Thanks to @nikomatsakis for his design blog posts which generated interest for this feature, @traviscross for feedback and additions to this stabilization report. All errors are my own.

r? @ghost

github-actions bot pushed a commit to rust-lang/miri that referenced this pull request

Dec 13, 2024

@bors

Stabilize async closures (RFC 3668)

Async Closures Stabilization Report

This report proposes the stabilization of #![feature(async_closure)] (RFC 3668). This is a long-awaited feature that increases the expressiveness of the Rust language and fills a pressing gap in the async ecosystem.

Stabilization summary

async fn takes_an_async_fn(f: impl AsyncFn(&str)) {
    futures::join(f("hello"), f("world")).await;
}

takes_an_async_fn(async |s| { other_fn(s).await }).await;

Motivation

Without this feature, users hit two major obstacles when writing async code that uses closures and Fn trait bounds:

That is, for the first, we cannot write:

// We cannot express higher-ranked async function signatures.
async fn f<Fut>(_: impl for<'a> Fn(&'a u8) -> Fut)
where
    Fut: Future<Output = ()>,
{ todo!() }

async fn main() {
    async fn g(_: &u8) { todo!() }
    f(g).await;
    //~^ ERROR mismatched types
    //~| ERROR one type is more general than the other
}

And for the second, we cannot write:

// Closures cannot return futures that borrow closure captures.
async fn f<Fut: Future<Output = ()>>(_: impl FnMut() -> Fut)
{ todo!() }

async fn main() {
    let mut xs = vec![];
    f(|| async {
        async fn g() -> u8 { todo!() }
        xs.push(g().await);
    });
    //~^ ERROR captured variable cannot escape `FnMut` closure body
}

Async closures provide a first-class solution to these problems.

For further background, please refer to the motivation section of the RFC.

Major design decisions since RFC

The RFC had left open the question of whether we would spell the bounds syntax for async closures...

// ...as this...
fn f() -> impl AsyncFn() -> u8 { todo!() }
// ...or as this:
fn f() -> impl async Fn() -> u8 { todo!() }

We've decided to spell this as AsyncFn{,Mut,Once}.

The Fn family of traits is special in many ways. We had originally argued that, due to this specialness, that perhaps the async Fn syntax could be adopted without having to decide whether a general async Trait mechanism would ever be adopted. However, concerns have been raised that we may not want to use async Fn syntax unless we would pursue more general trait modifiers. Since there remain substantial open questions on those -- and we don't want to rush any design work there -- it makes sense to ship this needed feature using the AsyncFn-style bounds syntax.

Since we would, in no case, be shipping a generalized trait modifier system anytime soon, we'll be continuing to see AsyncFoo traits appear across the ecosystem regardless. If we were to ever later ship some general mechanism, we could at that time manage the migration from AsyncFn to async Fn, just as we'd be enabling and managing the migration of many other traits.

Note that, as specified in RFC 3668, the details of the AsyncFn* traits are not exposed and they can only be named via the "parentheses sugar". That is, we can write T: AsyncFn() -> u8 but not T: AsyncFn<Output = u8>.

Unlike the Fn traits, we cannot project to the Output associated type of the AsyncFn traits. That is, while we can write...

fn f<F: Fn() -> u8>(_: F::Output) {}

...we cannot write:

fn f<F: AsyncFn() -> u8>(_: F::Output) {}
//~^ ERROR

The choice of AsyncFn{,Mut,Once} bounds syntax obviates, for our purposes here, another question decided after that RFC, which was how to order bound modifiers such as for<'a> async Fn().

Other than answering the open question in the RFC on syntax, nothing has changed about the design of this feature between RFC 3668 and this stabilization.

What is stabilized

For those interested in the technical details, please see the dev guide section I authored.

Async closures

Other than in how they solve the problems described above, async closures act similarly to closures that return async blocks, and can have parts of their signatures specified:

// They can have arguments annotated with types:
let _ = async |_: u8| { todo!() };

// They can have their return types annotated:
let _ = async || -> u8 { todo!() };

// They can be higher-ranked:
let _ = async |_: &str| { todo!() };

// They can capture values by move:
let x = String::from("hello, world");
let _ = async move || do_something(&x).await };

When called, they return an anonymous future type corresponding to the (not-yet-executed) body of the closure. These can be awaited like any other future.

What distinguishes async closures is that, unlike closures that return async blocks, the futures returned from the async closure can capture state from the async closure. For example:

let vec: Vec<String> = vec![];

let closure = async || {
    vec.push(ready(String::from("")).await);
};

The async closure captures vec with some &'closure mut Vec<String> which lives until the closure is dropped. Every call to closure() returns a future which reborrows that mutable reference &'call mut Vec<String> which lives until the future is dropped (e.g. it is awaited).

As another example:

let string: String = "Hello, world".into();

let closure = async move || {
    ready(&string).await;
};

The closure is marked with move, which means it takes ownership of the string by value. The future that is returned by calling closure() returns a future which borrows a reference &'call String which lives until the future is dropped (e.g. it is awaited).

Async fn trait family

To support the lending capability of async closures, and to provide a first-class way to express higher-ranked async closures, we introduce the AsyncFn* family of traits. See the corresponding section of the RFC.

We stabilize naming AsyncFn* via the "parenthesized sugar" syntax that normal Fn* traits can be named. The AsyncFn* trait can be used anywhere a Fn* trait bound is allowed, such as:

/// In return-position impl trait:
fn closure() -> impl AsyncFn() { async || {} }

/// In trait bounds:
trait Foo<F>: Sized
where
    F: AsyncFn()
{
    fn new(f: F) -> Self;
}

/// in GATs:
trait Gat {
    type AsyncHasher<T>: AsyncFn(T) -> i32;
}

Other than using them in trait bounds, the definitions of these traits are not directly observable, but certain aspects of their behavior can be indirectly observed such as the fact that:

fn by_ref_call(c: impl AsyncFn()) {
    let fut = c();
    drop(c);
    //   ^ Cannot drop `c` since it is borrowed by `fut`.
}
fn by_ref_call(c: impl AsyncFnOnce()) {
    let fut = c();
    let _ = c();
    //      ^ Cannot call `c` since calling it takes ownership the callee.
}
fn is_async_fn(_: impl AsyncFn(&str)) {}

async fn async_fn_item(s: &str) { todo!() }
is_async_fn(s);
// ^^^ This works.

fn generic(f: impl Fn() -> impl Future<Output = ()>) {
    is_async_fn(f);
    // ^^^ This does not work (yet).
}

The by-move future

When async closures are called with AsyncFn/AsyncFnMut, they return a coroutine that borrows from the closure. However, when they are called via AsyncFnOnce, we consume that closure, and cannot return a coroutine that borrows from data that is now dropped.

To work around around this limitation, we synthesize a separate future type for calling the async closure via AsyncFnOnce.

This future executes identically to the by-ref future returned from calling the async closure, except for the fact that it has a different set of captures, since we must move the captures from the parent async into the child future.

Interactions between async closures and the Fn* family of traits

Async closures always implement FnOnce, since they always can be called once. They may also implement Fn or FnMut if their body is compatible with the calling mode (i.e. if they do not mutate their captures, or they do not capture their captures, respectively) and if the future returned by the async closure is not lending.

let id = String::new();

let mapped: Vec</* impl Future */> =
    [/* elements */]
    .into_iter()
    // `Iterator::map` takes an `impl FnMut`
    .map(async |element| {
        do_something(&id, element).await;
    })
    .collect();

See the dev guide for a detailed explanation for the situations where this may not be possible due to the lending nature of async closures.

Other notable features of async closures shared with synchronous closures

Lints

This PR also stabilizes the CLOSURE_RETURNING_ASYNC_BLOCK lint as an allow lint. This lints on "old-style" async closures:

#![warn(closure_returning_async_block)]
let c = |x: &str| async {};

We should encourage users to use async || {} where possible. This lint remains allow and may be refined in the future because it has a few false positives (namely, see: "Where do we expect rewriting || async {} into async || {} to fail?")

An alternative that could be made at the time of stabilization is to put this lint behind another gate, so we can decide to stabilize it later.

What isn't stabilized (aka, potential future work)

async Fn*() bound syntax

We decided to stabilize async closures without the async Fn*() bound modifier syntax. The general direction of this syntax and how it fits is still being considered by T-lang (e.g. in RFC 3710).

Naming the futures returned by async closures

This stabilization PR does not provide a way of naming the futures returned by calling AsyncFn*.

Exposing a stable way to refer to these futures is important for building async-closure-aware combinators, and will be an important future step.

Return type notation-style bounds for async closures

The RFC described an RTN-like syntax for putting bounds on the future returned by an async closure:

async fn foo(x: F) -> Result<()>
where
    F: AsyncFn(&str) -> Result<()>,
    // The future from calling `F` is `Send` and `'static`.
    F(..): Send + 'static,
{}

This stabilization PR does not stabilize that syntax yet, which remains unimplemented (though will be soon).

dyn AsyncFn*()

AsyncFn* are not dyn-compatible yet. This will likely be implemented in the future along with the dyn-compatibility of async fn in trait, since the same issue (dealing with the future returned by a call) applies there.

Tests

Tests exist for this feature in tests/ui/async-await/async-closures.

A selected set of tests:

Remaining bugs and open issues

Where do we expect rewriting || async {} into async || {} to fail?

let x: fn() -> _ = async || {};
fn needs_send_future(_: impl Fn(NotSendArg) -> Fut)
where
    Fut: Future<Output = ()>,
{}

needs_send_future(async |_| {});

History

Important feature history

Acknowledgements

Thanks to @oli-obk for reviewing the bulk of the work for this feature. Thanks to @nikomatsakis for his design blog posts which generated interest for this feature, @traviscross for feedback and additions to this stabilization report. All errors are my own.

r? @ghost

jhpratt added a commit to jhpratt/rust that referenced this pull request

Mar 17, 2025

@jhpratt

…re, r=oli-obk

Improve upvar analysis for deref of child capture

Two fixes to the heuristic I implemented in rust-lang#123660. As I noted in the code:

Luckily, if this function is not correct, then the program is not unsound, since we still borrowck and validate the choices made from this function -- the only side-effect is that the user may receive unnecessary borrowck errors.

This indeed fixes unnecessary borrowck errors.

r? oli-obk

The heuristic is only valid if we deref a &T, not a &mut T or Box<T>, so make sure to check the type. This fixes:

struct Foo { precise: i32 }

fn mut_ref_inside_mut(f: &mut Foo) {
    let x: impl AsyncFn() = async move || {
        let y = &f.precise;
    };
}

Since the capture from f to &f.precise needs to be treated as a lending borrow from the parent coroutine-closure to the child coroutine.

The heuristic is also valid if any deref projection in the child capture's projections is a &T, but we were only looking at the last one. This ensures that this function is considered not to be lending:

struct Foo { precise: i32 }

fn ref_inside_mut(f: &mut &Foo) {
    let x: impl Fn() -> _ = async move || {
        let y = &f.precise;
    };
}

(Specifically, checking that impl Fn() -> _ is satisfied is exercising that the coroutine is not considered to be lending.)

rust-timer added a commit to rust-lang-ci/rust that referenced this pull request

Mar 17, 2025

@rust-timer

Rollup merge of rust-lang#138517 - compiler-errors:better-child-capture, r=oli-obk

Improve upvar analysis for deref of child capture

Two fixes to the heuristic I implemented in rust-lang#123660. As I noted in the code:

Luckily, if this function is not correct, then the program is not unsound, since we still borrowck and validate the choices made from this function -- the only side-effect is that the user may receive unnecessary borrowck errors.

This indeed fixes unnecessary borrowck errors.

r? oli-obk

The heuristic is only valid if we deref a &T, not a &mut T or Box<T>, so make sure to check the type. This fixes:

struct Foo { precise: i32 }

fn mut_ref_inside_mut(f: &mut Foo) {
    let x: impl AsyncFn() = async move || {
        let y = &f.precise;
    };
}

Since the capture from f to &f.precise needs to be treated as a lending borrow from the parent coroutine-closure to the child coroutine.

The heuristic is also valid if any deref projection in the child capture's projections is a &T, but we were only looking at the last one. This ensures that this function is considered not to be lending:

struct Foo { precise: i32 }

fn ref_inside_mut(f: &mut &Foo) {
    let x: impl Fn() -> _ = async move || {
        let y = &f.precise;
    };
}

(Specifically, checking that impl Fn() -> _ is satisfied is exercising that the coroutine is not considered to be lending.)