actix
Memory Leak
Memory Leak / Memory "not recycled" in Actix 3.3 #1943
The program is active_ On the web, my program has a startup memory of 14MB and requests the interface to perform some operations with a memory of 800MB+. However, the 800MB+in subsequent programs has not been released. What is the reason for this? Some of the language's setting features, or are there any useful tools such as PPROF that can detect memory overflow
Actix_ When web requests return big data, the memory is not released and will continue to be occupied. Is the original intention of this issue and will it be improved in the future
use actix_web::{get, HttpResponse, Responder}; use crate::controller::link_inspection::check::LinkInspection; use std::collections::HashMap; #[get("/check")] async fn check() -> impl Responder {
// 启动堆分析器
// let res = LinkInspection::new().doing().await;
// match res {
// Ok(response) => HttpResponse::Ok().json(response),
// Err(error) => HttpResponse::InternalServerError().body(error.to_string()),
// }
let mut h: HashMap<i64,i64> = HashMap::with_capacity(100000);
test(&mut h).await;
HttpResponse::InternalServerError().body(format!("{:?}", h))
}
async fn test (h : &mut HashMap<i64,i64>){
// tokio::time::sleep(tokio::time::Duration::from_secs(25)).await;
println!("start");
for v in 0..1000000{
h.insert(v,v);
}
}
What makes you think this is an issue with actix-web? Could you please provide a minimal reproducible example of this?
code
use actix_web::{App, HttpServer};
#[tokio::main]
async fn main() {
web().await;
}
async fn web() {
HttpServer::new(|| {
println!(
"WEB LISTENING TO {}:{}",
"127.0.0.1",
18080
);
// 在这里传入定义的服务
App::new().service(check)
})
.bind(("127.0.0.1", 18080))
.unwrap()
.run()
.await
.unwrap();
}
use actix_web::{get, HttpResponse, Responder};
use std::collections::HashMap;
#[get("/check")]
async fn check() -> impl Responder {
let mut h: HashMap<i64,i64> = HashMap::with_capacity(100000);
test(&mut h).await;
HttpResponse::InternalServerError().body(format!("{:?}", h))
}
async fn test (h : &mut HashMap<i64,i64>){
// tokio::time::sleep(tokio::time::Duration::from_secs(25)).await;
println!("start");
for v in 0..1000000{
h.insert(v,v);
}
}
start memory
request and response
Memory not released
cargo.toml
actix-web = "4.0.0"
tokio = { version = "1", features = ["full","tracing"] }
I also can reproduce increased memory usage both with debug and release mode.
Project - actix.zip
Maybe this is problem with memory allocator, that not have enough pressure to clear memory and do this only when necessary(for me, when I got ~300).
https://github.com/actix/actix-web/assets/41945903/4dd6f4ef-6cc3-470f-a07d-01c3211f66f5
Valgrind not show any memory leaks
==190166== HEAP SUMMARY:
==190166== in use at exit: 32,452 bytes in 84 blocks
==190166== total heap usage: 3,632 allocs, 3,548 frees, 237,465,785 bytes allocated
==190166==
==190166== LEAK SUMMARY:
==190166== definitely lost: 0 bytes in 0 blocks
==190166== indirectly lost: 0 bytes in 0 blocks
==190166== possibly lost: 1,988 bytes in 7 blocks
==190166== still reachable: 30,464 bytes in 77 blocks
==190166== suppressed: 0 bytes in 0 blocks
==190166== Rerun with --leak-check=full to see details of leaked memory
==190166==
==190166== For lists of detected and suppressed errors, rerun with: -s
==190166== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
It is possible that data passed to body function HttpResponse::InternalServerError().body(body) is never freed and
I also can reproduce increased memory usage both with debug and release mode.
Project - actix.zip
Maybe this is problem with memory allocator, that not have enough pressure to clear memory and do this only when necessary(for me, when I got ~300).
simplescreenrecorder-2023-11-23_12.35.54.mp4 Valgrind not show any memory leaks
==190166== HEAP SUMMARY: ==190166== in use at exit: 32,452 bytes in 84 blocks ==190166== total heap usage: 3,632 allocs, 3,548 frees, 237,465,785 bytes allocated ==190166== ==190166== LEAK SUMMARY: ==190166== definitely lost: 0 bytes in 0 blocks ==190166== indirectly lost: 0 bytes in 0 blocks ==190166== possibly lost: 1,988 bytes in 7 blocks ==190166== still reachable: 30,464 bytes in 77 blocks ==190166== suppressed: 0 bytes in 0 blocks ==190166== Rerun with --leak-check=full to see details of leaked memory ==190166== ==190166== For lists of detected and suppressed errors, rerun with: -s ==190166== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)It is possible that data passed to body function
HttpResponse::InternalServerError().body(body)is never freed and
I don't think that's the problem HttpResponse:: InternalServerError(). body (body)is never freed and
Take a look at the example below me
code
use std::io::prelude::*;
use std::net::{TcpListener, ToSocketAddrs};
use std::net::TcpStream;
use serde_json;
use serde::{Serialize, Deserialize};
use std::collections::HashMap;
use serde::ser::SerializeMap;
fn main() {
webs()
}
fn webs() {
let addr = "127.0.0.1:18080";
let socket_address = addr.to_socket_addrs().unwrap().next().unwrap();
let listener = TcpListener::bind(socket_address).unwrap();
let pool = ThreadPool::new(20);
for stream in listener.incoming().take(2) {
let stream = stream.unwrap();
pool.execute(|| {
handle_connection(stream);
});
}
println!("Shutting down.");
}
#[derive(Serialize, Deserialize)]
struct Data1 {
#[serde(serialize_with = "serialize_hashmap")]
data: HashMap<i64, i64>,
}
fn serialize_hashmap<S>(map: &HashMap<i64, i64>, serializer: S) -> Result<S::Ok, S::Error>
where
S: serde::Serializer,
{
let mut seq = serializer.serialize_map(Some(map.len()))?;
for (key, value) in map {
seq.serialize_entry(key, value)?;
}
seq.end()
}
fn handle_connection(mut stream: TcpStream) {
let mut buffer = [0; 1024];
stream.read(&mut buffer).unwrap();
println!("{:?}",String::from_utf8(Vec::from(buffer)));
let get = b"GET / HTTP/1.1\r\n";
let ch = b"GET /check HTTP/1.1\r\n";
let (status_line, res) = if buffer.starts_with(get) {
("HTTP/1.1 200 OK", "OK".to_string())
} else if buffer.starts_with(ch) {
let mut h: HashMap<i64,i64> = HashMap::with_capacity(100000);
test(&mut h);
let res = serde_json::to_string(&Data1{data:h}).unwrap();
("HTTP/1.1 200 OK", res.to_string())
} else {
("HTTP/1.1 404 NOT FOUND", "404".to_string())
};
let response = format!(
"{}\r\nContent-Length: {}\r\n\r\n{}",
status_line,
res.len(),
res
);
stream.write_all(response.as_bytes()).unwrap();
stream.flush().unwrap();
}
fn test (h : &mut HashMap<i64,i64>){
// tokio::time::sleep(tokio::time::Duration::from_secs(25)).await;
println!("start");
for v in 0..1000000{
h.insert(v,v);
}
}
// src/lib.rs
use std::{
sync::{mpsc, Arc, Mutex},
thread,
};
pub struct ThreadPool {
workers: Vec<Worker>,
sender: Option<mpsc::Sender<Job>>,
}
type Job = Box<dyn FnOnce() + Send + 'static>;
impl ThreadPool {
/// Create a new ThreadPool.
///
/// The size is the number of threads in the pool.
///
/// # Panics
///
/// The `new` function will panic if the size is zero.
pub fn new(size: usize) -> ThreadPool {
assert!(size > 0);
let (sender, receiver) = mpsc::channel();
let receiver = Arc::new(Mutex::new(receiver));
let mut workers = Vec::with_capacity(size);
for id in 0..size {
workers.push(Worker::new(id, Arc::clone(&receiver)));
}
ThreadPool {
workers,
sender: Some(sender),
}
}
pub fn execute<F>(&self, f: F)
where
F: FnOnce() + Send + 'static,
{
let job = Box::new(f);
self.sender.as_ref().unwrap().send(job).unwrap();
}
}
impl Drop for ThreadPool {
fn drop(&mut self) {
drop(self.sender.take());
for worker in &mut self.workers {
println!("Shutting down worker {}", worker.id);
if let Some(thread) = worker.thread.take() {
thread.join().unwrap();
}
}
}
}
struct Worker {
id: usize,
thread: Option<thread::JoinHandle<()>>,
}
impl Worker {
fn new(id: usize, receiver: Arc<Mutex<mpsc::Receiver<Job>>>) -> Worker {
let thread = thread::spawn(move || loop {
let message = receiver.lock().unwrap().recv();
match message {
Ok(job) => {
println!("Worker {id} got a job; executing.");
job();
}
Err(_) => {
println!("Worker {id} disconnected; shutting down.");
break;
}
}
});
Worker {
id,
thread: Some(thread),
}
}
}
curl --location --request GET '127.0.0.1:18080/check'
Still not released, but if data is not returned, it can be automatically released
I also can reproduce increased memory usage both with debug and release mode.
Project - actix.zip
Maybe this is problem with memory allocator, that not have enough pressure to clear memory and do this only when necessary(for me, when I got ~300).
simplescreenrecorder-2023-11-23_12.35.54.mp4 Valgrind not show any memory leaks
==190166== HEAP SUMMARY: ==190166== in use at exit: 32,452 bytes in 84 blocks ==190166== total heap usage: 3,632 allocs, 3,548 frees, 237,465,785 bytes allocated ==190166== ==190166== LEAK SUMMARY: ==190166== definitely lost: 0 bytes in 0 blocks ==190166== indirectly lost: 0 bytes in 0 blocks ==190166== possibly lost: 1,988 bytes in 7 blocks ==190166== still reachable: 30,464 bytes in 77 blocks ==190166== suppressed: 0 bytes in 0 blocks ==190166== Rerun with --leak-check=full to see details of leaked memory ==190166== ==190166== For lists of detected and suppressed errors, rerun with: -s ==190166== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)It is possible that data passed to body function
HttpResponse::InternalServerError().body(body)is never freed and
use actix_web::{App, HttpServer};
#[tokio::main]
async fn main() {
web().await;
}
async fn web() {
HttpServer::new(|| {
println!(
"WEB LISTENING TO {}:{}",
"127.0.0.1",
18080
);
// 在这里传入定义的服务
App::new().service(check)
})
.bind(("127.0.0.1", 18080))
.unwrap()
.run()
.await
.unwrap();
}
use actix_web::{get, HttpResponse, Responder};
use std::collections::HashMap;
#[get("/check")]
async fn check() -> impl Responder {
let mut h: HashMap<i64,i64> = HashMap::with_capacity(100000);
test(&mut h).await;
HttpResponse::InternalServerError().body(format!("{:?}", "OK"))
}
async fn test (h : &mut HashMap<i64,i64>){
// tokio::time::sleep(tokio::time::Duration::from_secs(25)).await;
println!("start");
for v in 0..1000000{
h.insert(v,v);
}
}
Still not released, but if data is not returned, it can be automatically released
I also can reproduce increased memory usage both with debug and release mode.
you say I also can reproduce increased memory usage both with debug and release mode.
I also can reproduce increased memory usage both with debug and release mode.
Project - actix.zip
Maybe this is problem with memory allocator, that not have enough pressure to clear memory and do this only when necessary(for me, when I got ~300).
simplescreenrecorder-2023-11-23_12.35.54.mp4 Valgrind not show any memory leaks
==190166== HEAP SUMMARY: ==190166== in use at exit: 32,452 bytes in 84 blocks ==190166== total heap usage: 3,632 allocs, 3,548 frees, 237,465,785 bytes allocated ==190166== ==190166== LEAK SUMMARY: ==190166== definitely lost: 0 bytes in 0 blocks ==190166== indirectly lost: 0 bytes in 0 blocks ==190166== possibly lost: 1,988 bytes in 7 blocks ==190166== still reachable: 30,464 bytes in 77 blocks ==190166== suppressed: 0 bytes in 0 blocks ==190166== Rerun with --leak-check=full to see details of leaked memory ==190166== ==190166== For lists of detected and suppressed errors, rerun with: -s ==190166== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)It is possible that data passed to body function
HttpResponse::InternalServerError().body(body)is never freed and
you say Maybe this is problem with memory allocator, that not have enough pressure to clear memory and do this only when necessary(for me, when I got ~300). Do you have any information on this? I'll study it. Thank you
I am using actix as well as a backend web server. And it seems like it is leaking memory. Even though the server is not receiving incoming request, the memory used never goes down and keep increasing upon new incoming request. I am seeing the same behavior (actix-web = "4.4.0"). I am using a tool for memory leak called HeapTrack, and here are some figures below. The code is built and tested in --release mode with cargo build --release and cargo run --release
It seems like there is no deallocation of the used memory even after hours of idle period. According to HeapTrack, that memory peak allocation is coming from Actix Http Dispatcher. See the figures below for more details:
BackTrace of the calls that keeps growing memory usage
Does anyone explain? Does it mean that Rust is not used by anyone in the production environment? It looks like it's too weak and cannot be truly used
I would not state about the language. i would state about the specific library we're using. We should understand the problem and have feedbacks about why this is happening. there will be an explanation for sure about why Actix Heap memory keeps growing. No programming language is safe from Memory leak if you have a long running thread that keeps a memory reference alive. Rust can't decide to clear a variable reference if it is being used. This issue here has nothing to do with rust. Rust is used in many prod environments. I have seen it myself in action in one of the organizations i was working for. Let's focus instead on the problem instead of having let go with opinionated view about a programming language.
You overthink it. I'm definitely using it because I trust and value it. Of course, I hope it's better, not because I have any bias against Rust
Rust is just a tool, if you used it not properly it will yield undesirable effects. it's just what needed to be understood here. Hope we have an answer from the main devs of this framework. Actix is great.
By the way, I also use the Same Actix.zip file, and i am able to reproduce the memory leak with a rust load testing tool called drill.
cargo install drill
drill --benchmark benchmark.yml
Here is a heaptrack memory dump you can use to analyze with heaptrack along with the project. actix.zip
I see the memory is growing at the same component location as mine which is Actix Http Dispatcher. It clearly shows that Actix Web is leaking 5GB of memory when launching drill upon "/check" endpoint in this example. The memory leak is really relevant and should be adressed with a Pull Request.
I also can reproduce increased memory usage both with debug and release mode.
you say I also can reproduce increased memory usage both with debug and release mode.
I also can reproduce increased memory usage both with debug and release mode. Project - actix.zip Maybe this is problem with memory allocator, that not have enough pressure to clear memory and do this only when necessary(for me, when I got ~300). simplescreenrecorder-2023-11-23_12.35.54.mp4 Valgrind not show any memory leaks
==190166== HEAP SUMMARY: ==190166== in use at exit: 32,452 bytes in 84 blocks ==190166== total heap usage: 3,632 allocs, 3,548 frees, 237,465,785 bytes allocated ==190166== ==190166== LEAK SUMMARY: ==190166== definitely lost: 0 bytes in 0 blocks ==190166== indirectly lost: 0 bytes in 0 blocks ==190166== possibly lost: 1,988 bytes in 7 blocks ==190166== still reachable: 30,464 bytes in 77 blocks ==190166== suppressed: 0 bytes in 0 blocks ==190166== Rerun with --leak-check=full to see details of leaked memory ==190166== ==190166== For lists of detected and suppressed errors, rerun with: -s ==190166== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)It is possible that data passed to body function
HttpResponse::InternalServerError().body(body)is never freed andyou say Maybe this is problem with memory allocator, that not have enough pressure to clear memory and do this only when necessary(for me, when I got ~300). Do you have any information on this? I'll study it. Thank you
I al
By the way, I also use the Same Actix.zip file, and i am able to reproduce the memory leak with a rust load testing tool called drill.
cargo install drill drill --benchmark benchmark.ymlHere is a heaptrack memory dump you can use to analyze with heaptrack along with the project. actix.zip
I see the memory is growing at the same location as mine which is active http dispatcher. It clearly shows that Actix Web is leaking 5GB of memory when launching drill upon "/check" endpoint in this example. The memory leak is really relevant and should be adressed with a Pull Request.
I also can reproduce increased memory usage both with debug and release mode.
you say I also can reproduce increased memory usage both with debug and release mode.
I also can reproduce increased memory usage both with debug and release mode. Project - actix.zip Maybe this is problem with memory allocator, that not have enough pressure to clear memory and do this only when necessary(for me, when I got ~300). simplescreenrecorder-2023-11-23_12.35.54.mp4 Valgrind not show any memory leaks
==190166== HEAP SUMMARY: ==190166== in use at exit: 32,452 bytes in 84 blocks ==190166== total heap usage: 3,632 allocs, 3,548 frees, 237,465,785 bytes allocated ==190166== ==190166== LEAK SUMMARY: ==190166== definitely lost: 0 bytes in 0 blocks ==190166== indirectly lost: 0 bytes in 0 blocks ==190166== possibly lost: 1,988 bytes in 7 blocks ==190166== still reachable: 30,464 bytes in 77 blocks ==190166== suppressed: 0 bytes in 0 blocks ==190166== Rerun with --leak-check=full to see details of leaked memory ==190166== ==190166== For lists of detected and suppressed errors, rerun with: -s ==190166== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)It is possible that data passed to body function
HttpResponse::InternalServerError().body(body)is never freed andyou say Maybe this is problem with memory allocator, that not have enough pressure to clear memory and do this only when necessary(for me, when I got ~300). Do you have any information on this? I'll study it. Thank you
I al
Yes, what you said makes sense. I have also posted on the forum before and asked about it. Most people mean that Rust releases memory, but the system memory collector did not trigger the collection or did not reach the pressure point to release this part of the memory. At the same time, this part of the memory is expected to be used in the future to avoid duplicate applications and releases
By the way, I also use the Same Actix.zip file, and i am able to reproduce the memory leak with a rust load testing tool called drill.
cargo install drill drill --benchmark benchmark.ymlHere is a heaptrack memory dump you can use to analyze with heaptrack along with the project. actix.zip I see the memory is growing at the same location as mine which is active http dispatcher. It clearly shows that Actix Web is leaking 5GB of memory when launching drill upon "/check" endpoint in this example. The memory leak is really relevant and should be adressed with a Pull Request.
I also can reproduce increased memory usage both with debug and release mode.
you say I also can reproduce increased memory usage both with debug and release mode.
I also can reproduce increased memory usage both with debug and release mode. Project - actix.zip Maybe this is problem with memory allocator, that not have enough pressure to clear memory and do this only when necessary(for me, when I got ~300). simplescreenrecorder-2023-11-23_12.35.54.mp4 Valgrind not show any memory leaks
==190166== HEAP SUMMARY: ==190166== in use at exit: 32,452 bytes in 84 blocks ==190166== total heap usage: 3,632 allocs, 3,548 frees, 237,465,785 bytes allocated ==190166== ==190166== LEAK SUMMARY: ==190166== definitely lost: 0 bytes in 0 blocks ==190166== indirectly lost: 0 bytes in 0 blocks ==190166== possibly lost: 1,988 bytes in 7 blocks ==190166== still reachable: 30,464 bytes in 77 blocks ==190166== suppressed: 0 bytes in 0 blocks ==190166== Rerun with --leak-check=full to see details of leaked memory ==190166== ==190166== For lists of detected and suppressed errors, rerun with: -s ==190166== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)It is possible that data passed to body function
HttpResponse::InternalServerError().body(body)is never freed andyou say Maybe this is problem with memory allocator, that not have enough pressure to clear memory and do this only when necessary(for me, when I got ~300). Do you have any information on this? I'll study it. Thank you
I al
Yes, what you said makes sense. I have also posted on the forum before and asked about it. Most people mean that Rust releases memory, but the system memory collector did not trigger the collection or did not reach the pressure point to release this part of the memory. At the same time, this part of the memory is expected to be used in the future to avoid duplicate applications and releases
Yes, your observation is right. and I also try to use "actix::main" wrapper instead of "tokio::main" , and it yields the same. I think, there may be leaked references. Rust ensures from compilation that active memory references are still valid. So, from a developer perspective, you may think a variable is not used. but, in the underlying system, there may be an active reference pointing to it. and being used by a sleeping thread. Actix is running on daemon mod, the variable lifecycle is bound to the threads maintaining alive by Tokio. On runtime, if there is any active reference pointing to a heap memory location, there can be memory Leaks. and rust official documentation states that as well with the use of RefCell smart pointers there can be memory leak if not being used properly. There is no where in rust documentation that they mention there can't be memory leak. There should be a closer look between Tokio and the way Actix Web uses variables that serializes data as response to the Client. in your case, you return a Hashmap with 100 000 values, and it seems that Actix Responder after serializing that data to the client still uses reference that avoids those values to be dropped. I think that can be the root cause of that memory leaks that makes memory usage grows everytime without going down.
This issue may be related to this problem - https://github.com/rust-lang/rust/issues/73307
This issue may be related to this problem - rust-lang/rust#73307
@qarmin Thanks for pointing us to one similar issue opened at Rust lang. It seems like the issue is closed without code change. or any sort of solution on how to fix it.
This issue may be related to this problem - rust-lang/rust#73307
@qarmin Thanks for pointing us to one similar issue opened at Rust lang. It seems like the issue is closed without code change. or any sort of solution on how to fix it.
I was thinking that this should be a very, very common question, but there is very little relevant information
By the way, I also use the Same Actix.zip file, and i am able to reproduce the memory leak with a rust load testing tool called drill.
cargo install drill drill --benchmark benchmark.ymlHere is a heaptrack memory dump you can use to analyze with heaptrack along with the project. actix.zip I see the memory is growing at the same location as mine which is active http dispatcher. It clearly shows that Actix Web is leaking 5GB of memory when launching drill upon "/check" endpoint in this example. The memory leak is really relevant and should be adressed with a Pull Request.
I also can reproduce increased memory usage both with debug and release mode.
you say I also can reproduce increased memory usage both with debug and release mode.
I also can reproduce increased memory usage both with debug and release mode. Project - actix.zip Maybe this is problem with memory allocator, that not have enough pressure to clear memory and do this only when necessary(for me, when I got ~300). simplescreenrecorder-2023-11-23_12.35.54.mp4 Valgrind not show any memory leaks
==190166== HEAP SUMMARY: ==190166== in use at exit: 32,452 bytes in 84 blocks ==190166== total heap usage: 3,632 allocs, 3,548 frees, 237,465,785 bytes allocated ==190166== ==190166== LEAK SUMMARY: ==190166== definitely lost: 0 bytes in 0 blocks ==190166== indirectly lost: 0 bytes in 0 blocks ==190166== possibly lost: 1,988 bytes in 7 blocks ==190166== still reachable: 30,464 bytes in 77 blocks ==190166== suppressed: 0 bytes in 0 blocks ==190166== Rerun with --leak-check=full to see details of leaked memory ==190166== ==190166== For lists of detected and suppressed errors, rerun with: -s ==190166== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)It is possible that data passed to body function is never freed and
HttpResponse::InternalServerError().body(body)you say Maybe this is problem with memory allocator, that not have enough pressure to clear memory and do this only when necessary(for me, when I got ~300). Do you have any information on this? I'll study it. Thank you
I al
Yes, what you said makes sense. I have also posted on the forum before and asked about it. Most people mean that Rust releases memory, but the system memory collector did not trigger the collection or did not reach the pressure point to release this part of the memory. At the same time, this part of the memory is expected to be used in the future to avoid duplicate applications and releases
I agree. On Mac OS, my memory seems to grow between 29M-30M and may return to 29M after a few attempts
my rustc version is rustc 1.76.0-nightly (3f28fe133 2023-12-18)
By the way, I also use the Same Actix.zip file, and i am able to reproduce the memory leak with a rust load testing tool called drill.
cargo install drill drill --benchmark benchmark.ymlHere is a heaptrack memory dump you can use to analyze with heaptrack along with the project. actix.zip I see the memory is growing at the same location as mine which is active http dispatcher. It clearly shows that Actix Web is leaking 5GB of memory when launching drill upon "/check" endpoint in this example. The memory leak is really relevant and should be adressed with a Pull Request.
I also can reproduce increased memory usage both with debug and release mode.
you say I also can reproduce increased memory usage both with debug and release mode.
I also can reproduce increased memory usage both with debug and release mode. Project - actix.zip Maybe this is problem with memory allocator, that not have enough pressure to clear memory and do this only when necessary(for me, when I got ~300). simplescreenrecorder-2023-11-23_12.35.54.mp4 Valgrind not show any memory leaks
==190166== HEAP SUMMARY: ==190166== in use at exit: 32,452 bytes in 84 blocks ==190166== total heap usage: 3,632 allocs, 3,548 frees, 237,465,785 bytes allocated ==190166== ==190166== LEAK SUMMARY: ==190166== definitely lost: 0 bytes in 0 blocks ==190166== indirectly lost: 0 bytes in 0 blocks ==190166== possibly lost: 1,988 bytes in 7 blocks ==190166== still reachable: 30,464 bytes in 77 blocks ==190166== suppressed: 0 bytes in 0 blocks ==190166== Rerun with --leak-check=full to see details of leaked memory ==190166== ==190166== For lists of detected and suppressed errors, rerun with: -s ==190166== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)It is possible that data passed to body function is never freed and
HttpResponse::InternalServerError().body(body)you say Maybe this is problem with memory allocator, that not have enough pressure to clear memory and do this only when necessary(for me, when I got ~300). Do you have any information on this? I'll study it. Thank you
I al
Yes, what you said makes sense. I have also posted on the forum before and asked about it. Most people mean that Rust releases memory, but the system memory collector did not trigger the collection or did not reach the pressure point to release this part of the memory. At the same time, this part of the memory is expected to be used in the future to avoid duplicate applications and releases
I agree. On Mac OS, my memory seems to grow between 29M-30M and may return to 29M after a few attempts my rustc version is
rustc 1.76.0-nightly (3f28fe133 2023-12-18)
use actix_web::{App, HttpServer,HttpRequest, http::header::ContentType};
use actix_web::body::BoxBody;
use serde::Serialize;
#[tokio::main]
async fn main() {
web().await;
}
#[derive(Serialize)]
struct Res{
h : Vec<i64>
}
impl Responder for Res{
type Body = BoxBody;
fn respond_to(self, req: &HttpRequest) -> HttpResponse<Self::Body> {
let body = serde_json::to_string(&self).unwrap();
HttpResponse::Ok()
.content_type(ContentType::json())
.body(body)
}
}
impl Drop for Res{
fn drop(&mut self) {
println!("drop res")
}
}
async fn web() {
HttpServer::new(|| {
println!(
"WEB LISTENING TO {}:{}",
"127.0.0.1",
18080
);
// 在这里传入定义的服务
App::new().service(check)
})
.bind(("127.0.0.1", 18080))
.unwrap()
.run()
.await
.unwrap();
}
WEB LISTENING TO 127.0.0.1:18080
WEB LISTENING TO 127.0.0.1:18080
WEB LISTENING TO 127.0.0.1:18080
start
drop res
start
drop res
start
drop res
This request output shows that res was dropped after use
By the way, I also use the Same Actix.zip file, and i am able to reproduce the memory leak with a rust load testing tool called drill.
cargo install drill drill --benchmark benchmark.ymlHere is a heaptrack memory dump you can use to analyze with heaptrack along with the project. actix.zip I see the memory is growing at the same location as mine which is active http dispatcher. It clearly shows that Actix Web is leaking 5GB of memory when launching drill upon "/check" endpoint in this example. The memory leak is really relevant and should be adressed with a Pull Request.
I also can reproduce increased memory usage both with debug and release mode.
you say I also can reproduce increased memory usage both with debug and release mode.
I also can reproduce increased memory usage both with debug and release mode. Project - actix.zip Maybe this is problem with memory allocator, that not have enough pressure to clear memory and do this only when necessary(for me, when I got ~300). simplescreenrecorder-2023-11-23_12.35.54.mp4 Valgrind not show any memory leaks
==190166== HEAP SUMMARY: ==190166== in use at exit: 32,452 bytes in 84 blocks ==190166== total heap usage: 3,632 allocs, 3,548 frees, 237,465,785 bytes allocated ==190166== ==190166== LEAK SUMMARY: ==190166== definitely lost: 0 bytes in 0 blocks ==190166== indirectly lost: 0 bytes in 0 blocks ==190166== possibly lost: 1,988 bytes in 7 blocks ==190166== still reachable: 30,464 bytes in 77 blocks ==190166== suppressed: 0 bytes in 0 blocks ==190166== Rerun with --leak-check=full to see details of leaked memory ==190166== ==190166== For lists of detected and suppressed errors, rerun with: -s ==190166== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)It is possible that data passed to body function is never freed and
HttpResponse::InternalServerError().body(body)you say Maybe this is problem with memory allocator, that not have enough pressure to clear memory and do this only when necessary(for me, when I got ~300). Do you have any information on this? I'll study it. Thank you
I al
Yes, what you said makes sense. I have also posted on the forum before and asked about it. Most people mean that Rust releases memory, but the system memory collector did not trigger the collection or did not reach the pressure point to release this part of the memory. At the same time, this part of the memory is expected to be used in the future to avoid duplicate applications and releases
I agree. On Mac OS, my memory seems to grow between 29M-30M and may return to 29M after a few attempts my rustc version is
rustc 1.76.0-nightly (3f28fe133 2023-12-18)use actix_web::{App, HttpServer,HttpRequest, http::header::ContentType}; use actix_web::body::BoxBody; use serde::Serialize; #[tokio::main] async fn main() { web().await; } #[derive(Serialize)] struct Res{ h : Vec<i64> } impl Responder for Res{ type Body = BoxBody; fn respond_to(self, req: &HttpRequest) -> HttpResponse<Self::Body> { let body = serde_json::to_string(&self).unwrap(); HttpResponse::Ok() .content_type(ContentType::json()) .body(body) } } impl Drop for Res{ fn drop(&mut self) { println!("drop res") } } async fn web() { HttpServer::new(|| { println!( "WEB LISTENING TO {}:{}", "127.0.0.1", 18080 ); // 在这里传入定义的服务 App::new().service(check) }) .bind(("127.0.0.1", 18080)) .unwrap() .run() .await .unwrap(); }WEB LISTENING TO 127.0.0.1:18080 WEB LISTENING TO 127.0.0.1:18080 WEB LISTENING TO 127.0.0.1:18080 start drop res start drop res start drop resThis request output shows that res was dropped after use
It seems it has to do with the underlying system allocator as well as argued in the issue closed by one of the rust maintainer. One in the thread suggested that he has seen improvement by using a custom global memory allocator which is "jemalloc". I tried it. the memory usage gets improved and may go up and down. but it doesn't completely free away the memory that's being deallocated. There is a thin line between the operating system allocator and the one used by rust (malloc subsystem calls) to manage memory allocation and deallocation.
Memory issue is being seen with some object types like Hashmap or vectors because they reside on the Heap and not on the stack. And actix web as well is doing some heavy lifting using those types as wells.
This issue may be related to this problem - rust-lang/rust#73307
@zqlpaopao Thanks once again for this link above about the issue which is opened to Rust lang itself. I found out with jemalloc library preloaded before running my rust application gives a little improvement to memory consumption. Memory may go up and down. And the memory usage is reasonable. But, it doesn't solve the leak effect. I used it myself on my linux system (require to install jemalloc package for your distribution or container image), here is how i run my app with "jemalloc". Make sure you install it before running this command
LD_PRELOAD=/usr/lib/libjemalloc.so.2 cargo run --release
Without having to use memory allocator crate that will grow my executable after compilation, it really performs well and my actix web application responds even better under heavy load testing with 8000 concurrent connections with 200 000 http requests that makes access to database to return info.
I even use it as well under alpine linux. require to set the environment variable for your container with the one above such that:
LD_PRELOAD=/usr/lib/libjemalloc.so.2 cargo
That should help a bit those who are struggling with their heap memory that grows(especially under linux/unix like system). I have not tried to compile jemalloc for windows and tried to use it. That can be an alternative or pain reliever for those who are struggling with this on a production environment.
This issue may be related to this problem - rust-lang/rust#73307
@zqlpaopao Thanks once again for this link above about the issue which is opened to Rust lang itself. I found out with jemalloc library preloaded before running my rust application gives a little improvement to memory consumption. Memory may go up and down. And the memory usage is reasonable. But, it doesn't solve the leak effect. I used it myself on my linux system (require to install jemalloc package for your distribution or container image), here is how i run my app with "jemalloc". Make sure you install it before running this command
LD_PRELOAD=/usr/lib/libjemalloc.so.2 cargo run --releaseWithout having to use memory allocator crate that will grow my executable after compilation, it really performs well and my actix web application responds even better under heavy load testing with 8000 concurrent connections with 200 000 http requests that makes access to database to return info.
I even use it as well under alpine linux. require to set the environment variable for your container with the one above such that:
LD_PRELOAD=/usr/lib/libjemalloc.so.2 cargoThat should help a bit those who are struggling with their heap memory that grows(especially under linux/unix like system). I have not tried to compile jemalloc for windows and tried to use it. That can be an alternative or pain reliever for those who are struggling with this on a production environment.
Thank you for the experiment and answers. Below are my test results
If I don't specify the size of the memory allocator startup to be around 7.1m
./test
If I specify
LD_PRELOAD=/usr/local/lib/libjemalloc.so.2 ./test
I have made 100 concurrent requests and my memory is growing
After the request ends, the memory returns to a result that is not much different from the initial value
This issue may be related to this problem - rust-lang/rust#73307
@zqlpaopao Thanks once again for this link above about the issue which is opened to Rust lang itself. I found out with jemalloc library preloaded before running my rust application gives a little improvement to memory consumption. Memory may go up and down. And the memory usage is reasonable. But, it doesn't solve the leak effect. I used it myself on my linux system (require to install jemalloc package for your distribution or container image), here is how i run my app with "jemalloc". Make sure you install it before running this command
LD_PRELOAD=/usr/lib/libjemalloc.so.2 cargo run --releaseWithout having to use memory allocator crate that will grow my executable after compilation, it really performs well and my actix web application responds even better under heavy load testing with 8000 concurrent connections with 200 000 http requests that makes access to database to return info. I even use it as well under alpine linux. require to set the environment variable for your container with the one above such that:
LD_PRELOAD=/usr/lib/libjemalloc.so.2 cargoThat should help a bit those who are struggling with their heap memory that grows(especially under linux/unix like system). I have not tried to compile jemalloc for windows and tried to use it. That can be an alternative or pain reliever for those who are struggling with this on a production environment.
Thank you for the experiment and answers. Below are my test results
If I don't specify the size of the memory allocator startup to be around 7.1m
./test
If I specify
LD_PRELOAD=/usr/local/lib/libjemalloc.so.2 ./test
I have made 100 concurrent requests and my memory is growing
After the request ends, the memory returns to a result that is not much different from the initial value
@zqlpaopao Jemalloc doesn't actually fix totally the memory allocation issue, it does a bit reduce its usage. It can grow but at a later time, reduce it though it's not complete to our expectation. I deep dive myself into the topic and does a lot of reading. This may have not to do with rust. This may have to do with the system default memory allocator. For linux, we're using GLibc malloc. And GLibc Malloc is the default memory allocator provided to all processes if not overriden. Hence, our rust process is using GLibc malloc. GLibC malloc can only return memory freed by processes to the Operating System by the order they were allocated. If a chunk of memory is freed before a previous allocated one. It won't be returned by GlibC malloc to the Operating system. So, memory allocations and time usage depends on applications and it's beyond Rust itself.
And it seems like all system memory allocators are trying to solve one common problem which is "Memory Fragmentation".
And i stumble on upon a blog From CloudFlare team that was dealing with a similar issue we're dealing with. And they really explain it very clear. Google has implemented a memory allocator called TCMalloc. And it seems that it solves their problem.
Facebook had also given it a try. All the big giants as well has been dealing with the issue we're dealing with now. we're just new to it . I am going to give TCMalloc a try.
https://blog.cloudflare.com/the-effect-of-switching-to-tcmalloc-on-rocksdb-memory-use
I did a test with libtcmalloc.so as LD_PRELOAD. it doesn't perform better than libjemalloc.so in my case. For my scenario with actix jemalloc really performs very well. This is a view of my process and memory consumption and deallocation in real time seen with atop linux utility. To have this view on atop, you have to press "m" to toggle the memory usage panel view.
With Jemalloc it really decreases my memory by looking at the VGROW and RGROW field with minus values "-0.2G" and "-0.1G" i.e reduces the virtual memory to 200M and the RAM to 100M. Jemalloc perform better for my case by launching my app with
LD_PRELOAD=/usr/lib/libjemalloc.so ./my_application
Feel free to use the malloc implementation that performs better in your case either the default one provided by your Operating Systems(Linux Glibc Malloc), or a custom one like TCMalloc(from Google), Jemalloc and so on.
Jemalloc is being maintained by Facebook. Here is a list of other custom system allocators you may test that may suite your scenarios: https://github.com/daanx/mimalloc-bench
我用libtcmalloc.so作为 LD_PRELOAD进行了测试。在我的例子中，它的性能并不比libjemalloc.so更好。对于我使用 actix jemalloc 的场景来说，它确实表现得非常好。这是使用atop linux 实用程序实时查看我的进程、内存消耗和释放的视图。要将此视图置于顶部，您必须按“m”来切换内存使用面板视图。
使用 Jemalloc，它通过查看VGROW和RGROW字段的负值“-0.2G”和“-0.1G”确实减少了我的内存，即将虚拟内存减少到 200M，将 RAM 减少到 100M。Jemalloc通过启动我的应用程序来更好地处理我的情况
LD_PRELOAD=/usr/lib/libjemalloc.so ./my_application
请随意使用在您的情况下表现更好的 malloc 实现，可以是操作系统提供的默认实现 ( Linux Glibc Malloc )，也可以是自定义实现，例如TCMalloc（来自 Google）、Jemalloc等。
Jemalloc 由 Facebook 维护。 以下是您可以测试的其他自定义系统分配器的列表，这些分配器可能适合您的场景： https: //github.com/daanx/mimalloc-bench
I have a requirement to run an agent monitoring program on Linux, but the requirement is that the memory cannot exceed 100mb, otherwise it will be killed. Therefore, I have encountered two problems: if libjemalloc is used, the memory will be a bit high, and for single threads, it still needs to reach 200mb+. TCmalloc has not been tested yet, but from your test screenshot, it is not low. If glibc is used, the initialization memory is not high, but it continues to increase, and it will also be killed. Is there a method for initializing the memory size under the control of libjemalloc, which would be perfect
@zqlpaopao I am wondering by curiosity what's your target hardware. is it an embedded system or a bare metal , Virtual machine, a Container deployed under Kubernetes with such limitation about memory. Or is this program will be running under a LXC cgroup which limits the maximum memory. In either scenario, i don't think a memory allocator is gonna solve your problem. I am not seeing any option from Jemalloc to set Maximum memory size. Jemalloc is implementing the kernel interface for memory allocation for "malloc" calls. I don't know if you set that with an environment Variables.
With that requirement, it means that you really need to have control over the memory usage. Allocators behavior won't solve that issue. No matter which one you choose. For example, i ran my app within a Alpine Docker container image of 17MB of size. And Alpine is using Musl LibC which is another implementation of Glibc and embeds as well it's own memory allocator. Memory usage is 50% lower than if i were running the app on my Linux distrib which based on GLibc. Musl seems to perform at a similar rate as Jemalloc.
@zqlpaopao我好奇地想知道你的目标硬件是什么。它是嵌入式系统还是裸机、虚拟机、部署在 Kubernetes 下的容器，对内存有这样的限制。或者这个程序将在限制最大内存的 LXC cgroup 下运行。在任何一种情况下，我认为内存分配器都不能解决你的问题。 我没有看到 Jemalloc 提供任何设置最大内存大小的选项。Jemalloc 正在实现“malloc”调用内存分配的内核接口。我不知道你是否使用环境变量设置了它。
有了这个要求，这意味着您确实需要控制内存的使用。分配者的行为并不能解决这个问题。无论您选择哪一个。例如，我在大小为 17MB 的Alpine Docker 容器映像中运行我的应用程序。 Alpine 使用 Musl LibC，它是 Glibc 的另一种实现，并且嵌入了它自己的内存分配器。 与我在基于 GLibc 的 Linux 发行版上运行该应用程序相比，内存使用量减少了 50%。 Musl 的表现似乎与****Jemalloc相似。 Yes, I actually run it on a physical machine, which is a daemon in Linux. However, it requires a lot of memory usage, so it is currently not possible to use Rust. However, with Golang, the CPU of the program will consume and fluctuate due to the need for GC
if you revert back to using glibc , don't do library preload with LD_PRELOAD. Set this variable and run your program with it.
MALLOC_ARENA_MAX=2 run_my_program
https://devcenter.heroku.com/articles/tuning-glibc-memory-behavior