baedert
Corebird leaks memory when removing tweets from timeline view
Of course you know that when you scroll down in Corebird, it scrolls infinitely e.g. it loads new tweets for as long as you scroll down. When you scroll back up to the top of corebird, after a couple seconds it removes all of the tweets it loaded and only keeps a few in the timeline view. The problem here is that after it removes these tweets from the view, it doesn't actually seem to free any of the memory being used for them. I've been able to trigger huge memory leaks by just scrolling down, scrolling back up, and letting the old tweets disappear (I've gotten over 200 MB of memory usage a couple of times).
Yeah, but once they get removed and you scroll back down, you don't "allocate" more memory, but rather reuse the memory that was freed when the tweets got removed. This is the expected behavior of GSlice, and apparently malloc (i.e. using G_SLICE=always-malloc) works the same. I've seen only 2 small memory leaks when running corebird under valgrind (even though just for a short period of time) -- and I made sure that both TweetListEntrys and Tweets get destroyed when tweets get removed from the timeline (using gobject-list).
I didn't notice that, it does seem to be reusing said memory. The problem is though corebird will still use way more ram then it should be using. Right now I'm seeing corebird using 301.2 MB. It should at least clean the slices it's allocated after we haven't scrolled down very much after a certain period of time.
I don't have any power about the slices (or buckets/bins) unfortunately. I don't really have a solution for this at the moment other than looking at the other leak I just found. Let's see how that turns out :)
Alright, so after the latest fixes to the memory leaks you added, the memory usage has definitely gone down drastically. Even loading huge amounts of tweets barely brings corebird above 50MiB of memory usage usually. So I think this is safe mark as closed, good work!
Unfortunately I think this needs to be reopened. I leave corebird open from time to time when I'm away from the computer for a little bit and occasionally leave it scrolled down, and when I leave the memory usage still rises dramatically and never goes back down :(. Still definitely not as bad as it was before though, but I still can get corebird to around 200 MB of memory usage just by forgetting to scroll back up.
Okay, so I've been thinking about this and I think I came up with an idea to keep the memory usage under control, and this is probably something that should be done anyway. How about limiting the number of tweets displayed at a time with corebird. Right now, we can scroll down our timeline and technically, load new tweets infinitely. This causes a problem because when we leave corebird scrolled down and new tweets come in, they just keep allocating new memory each time. So, if we limited how many tweets can be displayed and loaded at once, we wouldn't have to worry about new tweets coming in and taking more ram after we reach a certain point. And when we scroll down to the bottom/top of the currently displayed tweets, we just get rid of the tweets at the opposite end of the timeline and load new ones.
My favourite fix for it would be (of course) a model-based listbox implementation (model-based is done in the tweetmodel branch, but that doesn't solve the too-many-widgets problem right now).
Generally, I don't really feel comfortable (or the need, to be frankly) to complicate the handling tweet(-widget) handling here so much; When I scroll down and back up I expect the scrolling up to be instant and not require me to load tweets again that I just downlaoded a few seconds ago (and oh god just imagine what people who hit #255 think about that).
Also turns out that a binding problem caused us to leak the image data for inline media and avatars. Not sure if this was already present when this issue was opened though.
I am noticing that after leaving corebird running for a few hours I am getting multiple GB of memory usage. I haven't noticed a specific usage pattern that triggers it. Does anyone have any suggestions to help me debug this?
Have you tried using the debugging options glib's slice allocator has? IIRC you will need to force it to use plain malloc (it has an option for this) for valgrind to work On Jul 7, 2016 12:00 PM, "baedert" [email protected] wrote:
The usual way is: use valgrind, but that doesn't show anything.
— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/baedert/corebird/issues/314#issuecomment-231124806, or mute the thread https://github.com/notifications/unsubscribe/ABdNEQKC-vQqfOkb4FycEaTCbFWXorqXks5qTSKIgaJpZM4Dkey0 .
Horrible memory usage here as well on Arch Linux x86_64. Would like to help debug.
A naive run of valgrind with some basic timeline scrolling and switching between several accounts, yielded this summary:
==15290==
==15290== HEAP SUMMARY:
==15290== in use at exit: 38,436,547 bytes in 266,064 blocks
==15290== total heap usage: 4,866,886 allocs, 4,600,822 frees, 800,550,799 bytes allocated
==15290==
==15290== LEAK SUMMARY:
==15290== definitely lost: 99,124 bytes in 75 blocks
==15290== indirectly lost: 120,091 bytes in 4,540 blocks
==15290== possibly lost: 1,143,212 bytes in 1,154 blocks
==15290== still reachable: 32,958,640 bytes in 232,812 blocks
==15290== of which reachable via heuristic:
==15290== length64 : 16,760 bytes in 242 blocks
==15290== newarray : 2,512 bytes in 77 blocks
==15290== multipleinheritance: 3,656 bytes in 1 blocks
==15290== suppressed: 0 bytes in 0 blocks
==15290== Rerun with --leak-check=full to see details of leaked memory
==15290==
==15290== For counts of detected and suppressed errors, rerun with: -v
==15290== Use --track-origins=yes to see where uninitialised values come from
==15290== ERROR SUMMARY: 766 errors from 7 contexts (suppressed: 0 from 0)
Logs for a longer run with valgrind --log-file=valgrind.log --show-leak-kinds=all --leak-check=full corebird are attached: valgrind.log.gz
@baedert are these results expected? Does this fall under "doesn't show us anything"? Genuinely asking because I really don't know much about this.
More or less. Most of the stuff in the full log is just gobject 1-off allocation noise; the rest doesn't contain debugging symbols or is uninteresting. I was hoping for something "definitely lost" to show up but those 99kb aren't the memory leak I'm looking for... Basically, this should be a refcounting bug.
Edit: Now that I try it again, I have a hard time reproducing this with master.
@baedert if there any further tests you'd like me to run I'd be happy to do that. Corebird memory usage really is unreasonable, on order of GBs of memory after a few days of uptime.
if there any further tests you'd like me to run I'd be happy to do that.
I don't have specific instructions or tests but I wanted to run it with systemtap since that supposedly lets you inspect GObject leaks.
valgrind doesn't show a (definite) leak anywhere (apart from a few bytes in the avatar cache but that's because.... vala?) and using the GTK+ inspector I can only see that gtk+ leaks a few pixbufs when recoloring images, but that's it. The memory increase when scrolling down in a timeline (>40MB most of the time) can basically just come from the images loaded and as far as I can tell we are not leaking CbMedia, cairo_surface_t or GdkPixbufAnimation instances anywhere.
Just a small update, I noticed I was running a very old master version of Corebird and just built from the latest commit, will report back on how things look now.
I'm using a git snapshot from 20161010, memory usage is 4.5GiB after 6 days of uptime when I left the client running. I don't have sample numbers of how many tweets are there, so the observation I can make: the scrollbar is "in the middle", ie there are tons of read tweets but there's nothing like garbage collecting (similar to the original problem description).
There are some error messages in the output (multiple occurences):
(corebird:5542): Gtk-CRITICAL **: gtk_label_set_label: assertion 'GTK_IS_LABEL (label)' failed
(process:20179): GLib-CRITICAL **: g_slice_set_config: assertion 'sys_page_size == 0' failed
(corebird:5542): Gtk-WARNING **: gtk_widget_size_allocate(): attempt to underallocate MultiMediaWidget's child MediaButton 0x810f420. Allocation is 826x1, but minimum required size is 40x530.
** (corebird:5542): WARNING **: UserStream.vala:216: buf(NAME) == NULL. Starting timeout...
I'm aware there are no recommended debugging steps so far, please let me know if I can try something. I'll update from git and build with debugging and capture full log in the meantime.
the scrollbar is "in the middle", ie there are tons of read tweets but there's nothing like garbage collecting (similar to the original problem description).
The original problem description only applies once you scroll up and we remove tweets from the bottom. We never remove them from the top so if you just scroll to the middle and go on vacation for 2 weeks, it will most likely eat all your RAM.
