Proposal for Adapt Stack Protector for Rust

[SparrowLii]

Proposal for Adapt Stack Protector for Rust

Stack smash protection(ssp) is a requirement for many products in actual production environments.

Although Rust is known for its memory safety, Rust's unsafe code may still cause stack smash risks. In the current industry, many products use Rust/C/C++ interop, which leads to the frequent use of unsafe code. This is the main reason why products have a great demand for Rust's stack smash protection.

There are three modes of stack protection: basic, strong and all. (Tracking issue here). These modes originate from Clang (here is the doc). For each mode, rustc only adds a flag to each function's attributes, after which LLVM handles the specific implementation details.

We have conducted tests on the impact of these models on binary and runtime performance, and here are the results. None of these modes will have an observable impact on runtime performance; the primary impact lies in the size of the generated binary.

The all mode will add stack protection instructions to all functions, which will result in an increase in the size of the final generated binary. (>1.5% in average, up tp >10%)

The basic mode detects the use of arrays within functions, which is not applicable to Rust in principle. Because in Rust, it is not possible to directly modify data on the function stack through out-of-bounds error.

For strong mode's coverage, we can categorize it into the following three scenarios:

• Using arrays in functions or data structures, which does not apply to Rust either.
• Calling stack memory allocation function (alloc, eg), which is applicable to Rust.
• Obtaining variable addresses in a function. This rule partially applies to Rust, as variable addresses in Rust are represented in two forms: references and pointers. Pointers can be used to modify data on the function stack, which is something we need to protect. References cannot be directly used to modify data on the function stack, but they may be converted into pointers, thereby posing a risk.

For the actual stack smash risks that may be encountered in Rust, Clang's strong mode is already sufficient.

From the above, it can be seen that the rules from Clang are not suitable for Rust. This MCP proposes implementing a stack protection scheme applicable to Rust, which ensures that the security is not inferior to Clang's strong mode while significantly reducing the impact on the size of the generated binary.

The proposed plan for this MCP is divided into short-term and long-term options. Short-term: Detect whether there is any behavior of obtaining references or generating pointers within functions, including the compiler's behavior of passing function parameters by reference in callcov. Then add the sspstrong flag to these functions' attributes to enable LLVM to generate ssp instructions. Using sspstrong instead of sspreq can prevent adding extra ssp when function are inlined.

Long-term: Determine whether the references obtained will ultimately be converted into raw pointers. We currently do not have a clear solution, but we can attempt to achieve this by recursively inspecting the data flow of each function in the function call chain.

Problems that require further work: Regarding the impact on LLVM's function debuginfo from not adding any ssp flags in rustc, we can refer to the implementation of __attribute__((no_stack_protector)) in Clang+LLVM. The path is clear, we just need to refine a solution suitable for rustc.

prototype PR: https://github.com/rust-lang/rust/pull/144879

Mentors or Reviewers

@rcvalle

Thanks for your help!

Process

The main points of the Major Change Process are as follows:

• [x] File an issue describing the proposal.
• [ ] A compiler team member or contributor who is knowledgeable in the area can second by writing @rustbot second.
  • Finding a "second" suffices for internal changes. If however, you are proposing a new public-facing feature, such as a -C flag, then full team check-off is required.
  • Compiler team members can initiate a check-off via @rfcbot fcp merge on either the MCP or the PR.
• [ ] Once an MCP is seconded, the Final Comment Period begins. If no objections are raised after 10 days, the MCP is considered approved.

You can read more about Major Change Proposals on forge.

Comments

This issue is not meant to be used for technical discussion. There is a Zulip stream for that. Use this issue to leave procedural comments, such as volunteering to review, indicating that you second the proposal (or third, etc), or raising a concern that you would like to be addressed.

[!CAUTION]

Concerns (3 active)

• lose-debuginfo-data
• impl-at-mir-level
• inhibit-opts

Managed by @rustbot—see help for details.

[rustbot]

[!IMPORTANT] This issue is not meant to be used for technical discussion. There is a Zulip stream for that. Use this issue to leave procedural comments, such as volunteering to review, indicating that you second the proposal (or third, etc), or raising a concern that you would like to be addressed.

Concerns or objections to the proposal should be discussed on Zulip and formally registered here by adding a comment with the following syntax:

@rustbot concern reason-for-concern 
<description of the concern> 

Concerns can be lifted with:

@rustbot resolve reason-for-concern 

See documentation at https://forge.rust-lang.org

cc @rust-lang/compiler

[rcvalle]

@rustbot second

[wesleywiser]

Recording some objections that should be resolved prior to accepting this MCP:

@rustbot concern impl-at-mir-level

Zulip: doing this on MIR [...] will end up up protecting a lot more functions than necessary, because it happens pre-(LLVM-)inlining, so there will be a lot of addresses taken that later get inlined out.

@rustbot concern inhibit-opts

Zulip: By inserting the stack protectors prior to optimization, we're most likely going to inhibit optimizations

@rustbot concern lose-debuginfo-data

Zulip: Some targets (eg, msvc) record in debuginfo whether stack protectors were enabled or disabled on a per-function basis and this data is used in automated compliance tooling to ensure standard deployment practices are being followed. By inserting stack protectors earlier than LLVM is aware, we lose out on this data being captured correctly.

EDIT: reformatted comment for machine parsing

[SparrowLii]

@wesleywiser Based on the previous discussion, I think these problems have solutions.

• The heuristics need to be reduced. Instead of adding stack protection based on whether a variable reference is obtained, the following behaviors are detected: （1）Converts a local variable reference to a pointer type （2）Passing reference types(ot types contains reference) to other functions: recursively check whether the function has the behavior of converting the reference to a pointer

• This does not affect llvm's optimizations, because we only add a LLVM attribute flag to the function, which is the same as the flag added in current stack-protection=all mode.

• If the current stack-protector=all mode does not have this problem, then this MCP does not have either. If it have, then we can add another flag to inform the debuginfo-data tool.