lang: Remove 3 lifetime definitions from Context

[acheroncrypto]

Problem

Context struct definition includes 4 lifetimes:

https://github.com/coral-xyz/anchor/blob/ebbad72fc431fb726004d68c56270e88c869428d/lang/src/context.rs#L24

While this is technically correct, and it's also what Rust does by default, it results in a poor developer experience because the lifetimes leak to the Anchor users in various cases. For example, remaining accounts usage requires annotating the instruction handler with lifetimes, which is quite difficult to figure out for people who're less experienced with lifetimes (not to mention this is completely unnecessary).

Summary of changes

Remove 3 (out of 4) lifetimes from the Context struct. In other words, make all references have the same lifetime. This makes it so much easier for Anchor users to to handle places that Context is used. For example, remaining accounts usage:

https://github.com/coral-xyz/anchor/blob/ebbad72fc431fb726004d68c56270e88c869428d/tests/misc/programs/remaining-accounts/src/lib.rs#L25-L27

simply becomes:

pub fn test_remaining_accounts(ctx: Context<TestRemainingAccounts>) -> Result<()> {

[vercel]

@acheroncrypto is attempting to deploy a commit to the coral-xyz Team on Vercel.

A member of the Team first needs to authorize it.

[juchiast]

How does this interact with the issues in #2770 and #3341
Basically, the type &'info AccountInfo<'info> is not very usable, AccountInfo<'info> is invariant because it contains RefCell, meaning we cannot cast the lifetime AccountInfo<'info> into a struct with shorter lifetime AccountInfo<'a>.

[acheroncrypto]

How does this interact with the issues in #2770 and #3341

It basically fixes the problem. It still requires a lifetime annotation, but we should be able to handle this automically in the program macro.

Basically, the type &'info AccountInfo<'info> is not very usable

Yeah, but the alternative is worse (declaring another lifetime as explained in the "Details" section of #2770).

AccountInfo<'info> is invariant because it contains RefCell, meaning we cannot cast the lifetime AccountInfo<'info> into a struct with shorter lifetime AccountInfo<'a>.

Does that even matter here? I don't think the Rust compiler would allow you to overwrite the data with a reference declared inside the instruction handler.

[juchiast]

Does that even matter here? I don't think the Rust compiler would allow you to overwrite the data with a reference declared inside the instruction handler.

With a normal struct, it is possible to change the lifetime to a smaller one by re-borrowing

struct X<'a> {
    s: &'a str,
}

fn cast<'a, 'b>(x: X<'a>) -> X<'b> where 'a: 'b {
    X { s: &*x.s }
   // or just `return x;`, rust can cast automatically
}

but we can't do this with AccountInfo<'a>. It's impossible to make &'a AccountInfo<'a> reference, if you need to use &'a AccountInfo<'a> reference, you are stuck with that lifetime forever.

spl-candy-guard case: they use UncheckedAccount<'info>, do some check manually then call Account::try_from. Fortunately we can still recover the original &'info AccountInfo<'info> inside UncheckedAccount, if the lifetime is lost (by cloning, for example), it is impossible to create &'info AccountInfo<'info> again.

[acheroncrypto]

Does that even matter here? I don't think the Rust compiler would allow you to overwrite the data with a reference declared inside the instruction handler.

With a normal struct, it is possible to change the lifetime to a smaller one by re-borrowing

struct X<'a> {
    s: &'a str,
}

fn cast<'a, 'b>(x: X<'a>) -> X<'b> where 'a: 'b {
    X { s: &*x.s }
   // or just `return x;`, rust can cast automatically
}

but we can't do this with AccountInfo<'a>. It's impossible to make &'a AccountInfo<'a> reference, if you need to use &'a AccountInfo<'a> reference, you are stuck with that lifetime forever.

Yeah, I'm aware of that, but I'm not sure how that answers my initial comment about the Rust compiler not allowing you to overwrite the data with a reference declared inside the instruction handler.

spl-candy-guard case: they use UncheckedAccount<'info>, do some check manually then call Account::try_from. Fortunately we can still recover the original &'info AccountInfo<'info> inside UncheckedAccount, if the lifetime is lost (by cloning, for example), it is impossible to create &'info AccountInfo<'info> again.

Since you're using "they" when referring to the Metaplex team, I'm assuming you're not from the Metaplex team, and you just want to interact with the mpl-candy-guard program. I'm also assuming you're using Anchor v0.29 and Solana v1.18, since that's what you used in metaplex-foundation/mpl-candy-machine#76.

If my assumptions are correct, then trying to fully upgrade that program is completely unnecessary for your use case, because you don't need the program's internal logic to be able to interact with a program (you only need implementation signatures). If you want to use the program's crate as a CPI client or an off-chain client, you can safely remove all its instruction handler logic, including the parts where it uses Account::try_from.

As a side note, the mpl-candy-guard's usage of Account::try_from in order to implement some sort of an optional account check also seems redundant in newer Anchor versions because Anchor now supports optional accounts in accounts structs with Option<T>.

Furthermore, you don't even need to upgrade anything to interact with older programs if you're using the latest version (v0.30.1). In fact, you don't even need to add programs as a dependency. Here are some useful links:

• https://www.anchor-lang.com/release-notes/0.30.0#dependency-free-program-declaration
• https://docs.rs/anchor-lang/0.30.1/anchor_lang/macro.declare_program.html
• lang: Add declare_program! macro #2857
• https://en.wikipedia.org/wiki/XY_problem

[viandwi24]

I am new to learning Anchor. How do you currently handle this issue? I am encountering same problem when trying to use Account::<MyAccount>::try_from(&ctx.accounts.my_account.to_account_info())?;. any solutions for now ? thanks

[BretasArthur1]

This is a fantastic proposal, and a huge win for the developer experience in Anchor.

I was initially cautious about the introduction of unsafe to handle the lifetime changes, but after a thorough review, I'm completely convinced. The SAFETY comment is clear, and the trade-off is absolutely worth it. Abstracting this lifetime complexity away at the framework level is the right move.

Removing the need for developers to manually juggle multiple lifetimes in the Context struct will significantly lower the barrier to entry and make day-to-day development much more pleasant. The simplification in function signatures, as shown in the remaining-accounts test, speaks for itself.

This is a high-impact, positive change. Excellent work, and I fully support merging this.

[BretasArthur1]

Just some considerations:
The core safety of this change hinges on the Context and its references not "escaping" the scope of the instruction handler. The primary edge case I can see is a developer attempting to store a reference from ctx in a static variable or sending it to another thread.

However, it seems the use of transmute to an anonymous lifetime '_ provides a strong mitigation here. The borrow checker should, in most safe code, correctly identify the shorter lifetime and prevent it from being assigned to a longer-lived variable. A developer would likely need to use another unsafe block to force such a leak, at which point they are already bypassing safety guarantees.

Are there any other scenarios, perhaps involving complex CPIs, where this assumption could be violated in a less obvious way? From my perspective, this looks robust for the vast majority of use cases.

[jacobcreech]

@deanmlittle for second look

[febo]

Looks good to me. Tried different ways to not use the unsafe block, but I could not find another way to do it – it is technically sound. I suggested to update the comment to be more clear about the transmute.

[acheroncrypto]

The lifetime assumption during the transmute causes compile errors if the instruction doesn't have any accounts:

error[E0107]: struct takes 0 lifetime arguments but 1 lifetime argument was supplied
   --> programs/misc/src/lib.rs:41:40
    |
15  |   #[program]
    |  ___________-
16  | | pub mod misc {
17  | |     use super::*;
...   |
41  | |     pub fn test_simulate(_ctx: Context<TestSimulate>, data: u32) -> Result<()> {
    | |                                        ^^^^^^^^^^^-
    | |________________________________________|__________|
    |                                          |          help: remove the unnecessary generics
    |                                          expected 0 lifetime arguments
    |
note: struct defined here, with 0 lifetime parameters
   --> programs/misc/src/context.rs:206:12
    |
206 | pub struct TestSimulate {}
    |            ^^^^^^^^^^^^

I see two potential solutions here:

1. Remove the type annotations from the transmute and let the compiler infer them:

   From:

   ::core::mem::transmute::<
       &mut #accounts_struct_name<'info>,
       &mut #accounts_struct_name<'_>
   >(&mut __accounts)

   To:

   ::core::mem::transmute(&mut __accounts)

2. Require instructions with no accounts to be annoted (e.g #[no_accounts] or #[instruction(no_accounts)] above the instruction handler)

   The reason for this is because the instruction handler doesn't know whether the instruction's accounts struct has a lifetime or not. Even if we add a constant that tells whether the struct has accounts afaik we still can't use it while generating the code because Rust's type system has precedence over conditional logic e.g.

   struct NoAccounts {}
   
   if false {
       core::mem::transmute::<NoAccounts<'info>, NoAccounts<'_>>(NoAccounts {})
   }

   Doesn't compile:

   error[E0107]: struct takes 0 lifetime arguments but 1 lifetime argument was supplied
   

I'd be happy to know if I'm missing something and there is a better way to get around this problem.

I think the latter option makes more sense because I don't feel comfortable letting the compiler do a blind transmute. The vast majority of instructions have accounts anyway.

[jamie-osec]

We can just introduce a lifetime-only transmute helper:

unsafe fn transmute_lifetime<'to, T>(value: &mut T) -> &'to mut T {
    ::core::mem::transmute(value)
}

in order to work generically. This is safe as T will not change, but could also be implemented as

unsafe {
    &mut *::core::ptr::from_mut(value)
}

To extend the lifetime via pointer round tripping

[acheroncrypto]

We can just introduce a lifetime-only transmute helper:

unsafe fn transmute_lifetime<'to, T>(value: &mut T) -> &'to mut T {
    ::core::mem::transmute(value)
}

in order to work generically. This is safe as T will not change

Wouldn't this just extend the lifetime of the mutable borrow (to 'info) rather than shortening the lifetime of the accounts struct?

but could also be implemented as

unsafe {
    &mut *::core::ptr::from_mut(value)
}

To extend the lifetime via pointer round tripping

Isn't this essentially the same as the blind transmute option?

[jamie-osec]

Wouldn't this just extend the lifetime of the mutable borrow (to 'info) rather than shortening the lifetime of the accounts struct?

Looking into this again - this actually works by laundering the lifetimes such that __accounts is no longer recognised as mutably borrowed.

Isn't this essentially the same as the blind transmute option?

No, there's no casting of the pointer. This just reborrows the same pointer as the same type but selects a new lifetime, disconnected from the original (so the same effect)

[jacobcreech]

@acheroncrypto we've decided to hold off on this change until 2.0.

[acheroncrypto]

Isn't this essentially the same as the blind transmute option?

No, there's no casting of the pointer. This just reborrows the same pointer as the same type but selects a new lifetime, disconnected from the original (so the same effect)

@jamie-osec That still sounds like they're essentially doing the same thing to me. I wanted to confirm this by looking at the generated binary and confirmed this PR (with both variants) has no effect on the binary (at least in the programs I tested). I don't see a reason why it would anyway, considering lifetimes literally don't exist in runtime.

Did you mention "casting of the pointer" because you thought the compiler wouldn't be able to tell that this is actually the same value and same type and consequently optimize by omitting all pointer logic? If so, Rust seems to have grown up and not be as dumb anymore.

Looking into this again - this actually works by laundering the lifetimes such that __accounts is no longer recognised as mutably borrowed.

Not sure I follow. The return type &'to mut T in

unsafe fn transmute_lifetime<'to, T>(value: &mut T) -> &'to mut T {
    ::core::mem::transmute(value)
}

looks like it would still make it get recognized as mutably borrowed.

@acheroncrypto we've decided to hold off on this change until 2.0.

@jacobcreech looking at real world usage (e.g. light-protocol/account-compression), I think this change would be extremely helpful for v1.

This is a massive devex improvement with seemingly no impact on program binaries/runtime. If we're not comfortable including this because of the unsafe usage, we can put this behind a feature flag.

[jamie-osec]

That still sounds like they're essentially doing the same thing to me. I wanted to confirm this by looking at the generated binary and confirmed this PR (with both variants) has no effect on the binary (at least in the programs I tested). I don't see a reason why it would anyway, considering lifetimes literally don't exist in runtime.

Yeah, the difference here is that it's a compile-time guard that ensures that we can't change the type accidentally, only the lifetime. The current method is probably fine, but I would personally be more comfortable constraining the types as much as possible when transmuting in macro-generated code.

looks like it would still make it get recognized as mutably borrowed.

(Highly UB example but just illustrative)
https://play.rust-lang.org/?version=stable&mode=debug&edition=2024&gist=835c508f2089f19af8cd8f48f9c1652e

The transmute helper returns a reference with a lifetime that is disjoint of the input, meaning we can mutate val despite a 'static borrow to that variable.

[jamie-osec]

I added the lifetime-specific helper as an extra-paranoid guard, and added a small note on variance to the safety comment to make it clear why this shrinking isn't done automatically; we'll merge this for V1

[acheroncrypto]

I added the lifetime-specific helper as an extra-paranoid guard, and added a small note on variance to the safety comment to make it clear why this shrinking isn't done automatically

@jamie-osec I think the helper function clearly extends the mutable borrow lifetime instead of shortening the inner lifetime. The impl and its documentation is misleading currently.

What we want instead is something like this:

unsafe fn __shorten_invariant_lifetime<'a, 'info: 'a>(
    value: &'a mut #accounts_struct_name<'info>,
) -> &'a mut #accounts_struct_name<'a> {
    unsafe { ::core::mem::transmute(value) }
}