postalserver
V3: Storage is no longer emptied properly.
Describe the bug
Storage is no longer emptied properly.
See the screenshot. The limit is set to 15.5GB of storage and the server is currently using 18.8GB. I don't know if it's an issue with reporting or if there's an issue emptying the storage.
To Reproduce
- Go to Mail Server
- Set storage policy limit
- The limit is not respected
Expected behaviour
Just like in V2 I expected the storage to be emptied nightly.
Environment details
- OS: Ubuntu
It may be that the code is biased towards the days but I can not remember off the top of my head.
Have you tried connecting to the database to see how many raw tables you have?
FWIW Postal was not intended to store your messages long term, you should be doing that in whatever application you are using to create the emails and then you could also listen to webhooks and store the information for as long as you need.
It used to work just fine. It would (at least I understood that) delete all the messages older than, and then continue until it would meet the storage limit. This seems to no longer be the case:
MariaDB [postal-server-1]> SHOW TABLES;
+---------------------------+
| Tables_in_postal-server-1 |
+---------------------------+
| clicks |
| deliveries |
| links |
| live_stats |
| loads |
| messages |
| migrations |
| raw-2023-11-03 |
| raw-2023-11-04 |
| raw-2023-11-05 |
| raw-2023-11-06 |
| raw-2023-11-07 |
| raw-2023-11-08 |
| raw-2023-11-09 |
| raw-2023-11-10 |
| raw-2023-11-11 |
| raw-2023-11-12 |
| raw-2023-11-13 |
| raw-2023-11-14 |
| raw-2023-11-15 |
| raw-2023-11-16 |
| raw-2023-11-17 |
| raw-2023-11-18 |
| raw-2023-11-19 |
| raw-2023-11-20 |
| raw-2023-11-21 |
| raw-2023-11-22 |
| raw-2023-11-23 |
| raw-2023-11-24 |
| raw-2023-11-25 |
| raw-2023-11-26 |
| raw-2023-11-27 |
| raw-2023-11-28 |
| raw-2023-11-29 |
| raw-2023-11-30 |
| raw-2023-12-01 |
| raw-2023-12-02 |
| raw-2023-12-03 |
| raw-2023-12-04 |
| raw-2023-12-05 |
| raw-2023-12-06 |
| raw-2023-12-07 |
| raw-2023-12-08 |
| raw-2023-12-09 |
| raw-2023-12-10 |
| raw-2023-12-11 |
| raw-2023-12-12 |
| raw-2023-12-13 |
| raw-2023-12-14 |
| raw-2023-12-15 |
| raw-2023-12-16 |
| raw-2023-12-17 |
| raw-2023-12-18 |
| raw-2023-12-19 |
| raw-2023-12-20 |
| raw-2023-12-21 |
| raw-2023-12-22 |
| raw-2023-12-23 |
| raw-2023-12-24 |
| raw-2023-12-25 |
| raw-2023-12-26 |
| raw-2023-12-27 |
| raw-2023-12-28 |
| raw-2023-12-29 |
| raw-2023-12-30 |
| raw-2023-12-31 |
| raw-2024-01-01 |
| raw-2024-01-02 |
| raw-2024-01-03 |
| raw-2024-01-04 |
| raw-2024-01-05 |
| raw-2024-01-06 |
| raw-2024-01-07 |
| raw-2024-01-08 |
| raw-2024-01-09 |
| raw-2024-01-10 |
| raw-2024-01-11 |
| raw-2024-01-12 |
| raw-2024-01-13 |
| raw-2024-01-14 |
| raw-2024-01-15 |
| raw-2024-01-16 |
| raw-2024-01-17 |
| raw-2024-01-18 |
| raw-2024-01-19 |
| raw-2024-01-20 |
| raw-2024-01-21 |
| raw-2024-01-22 |
| raw-2024-01-23 |
| raw-2024-01-24 |
| raw-2024-01-25 |
| raw-2024-01-26 |
| raw-2024-01-27 |
| raw-2024-01-28 |
| raw-2024-01-29 |
| raw-2024-01-30 |
| raw-2024-01-31 |
| raw-2024-02-01 |
| raw-2024-02-02 |
| raw-2024-02-03 |
| raw-2024-02-04 |
| raw-2024-02-05 |
| raw-2024-02-06 |
| raw-2024-02-07 |
| raw-2024-02-08 |
| raw-2024-02-09 |
| raw-2024-02-10 |
| raw-2024-02-11 |
| raw-2024-02-12 |
| raw-2024-02-13 |
| raw-2024-02-14 |
| raw-2024-02-15 |
| raw-2024-02-16 |
| raw-2024-02-17 |
| raw-2024-02-18 |
| raw-2024-02-19 |
| raw-2024-02-20 |
| raw-2024-02-21 |
| raw-2024-02-22 |
| raw-2024-02-23 |
| raw-2024-02-24 |
| raw-2024-02-25 |
| raw-2024-02-26 |
| raw-2024-02-27 |
| raw-2024-02-28 |
| raw-2024-02-29 |
| raw-2024-03-01 |
| raw-2024-03-02 |
| raw-2024-03-03 |
| raw-2024-03-04 |
| raw-2024-03-05 |
| raw-2024-03-06 |
| raw-2024-03-07 |
| raw-2024-03-08 |
| raw-2024-03-09 |
| raw-2024-03-10 |
| raw-2024-03-11 |
| raw-2024-03-12 |
| raw-2024-03-13 |
| raw-2024-03-14 |
| raw-2024-03-15 |
| raw-2024-03-16 |
| raw-2024-03-17 |
| raw-2024-03-18 |
| raw-2024-03-19 |
| raw-2024-03-20 |
| raw-2024-03-21 |
| raw-2024-03-22 |
| raw-2024-03-23 |
| raw-2024-03-24 |
| raw-2024-03-25 |
| raw-2024-03-26 |
| raw-2024-03-27 |
| raw-2024-03-28 |
| raw-2024-03-29 |
| raw-2024-03-30 |
| raw-2024-03-31 |
| raw-2024-04-01 |
| raw-2024-04-02 |
| raw-2024-04-03 |
| raw-2024-04-04 |
| raw-2024-04-05 |
| raw-2024-04-06 |
| raw-2024-04-07 |
| raw-2024-04-08 |
| raw-2024-04-09 |
| raw-2024-04-10 |
| raw-2024-04-11 |
| raw-2024-04-12 |
| raw-2024-04-13 |
| raw-2024-04-14 |
| raw-2024-04-15 |
| raw-2024-04-16 |
| raw-2024-04-17 |
| raw-2024-04-18 |
| raw-2024-04-19 |
| raw-2024-04-20 |
| raw-2024-04-21 |
| raw-2024-04-22 |
| raw-2024-04-23 |
| raw-2024-04-24 |
| raw-2024-04-25 |
| raw-2024-04-26 |
| raw-2024-04-27 |
| raw-2024-04-28 |
| raw-2024-04-29 |
| raw-2024-04-30 |
| raw-2024-05-01 |
| raw-2024-05-02 |
| raw-2024-05-03 |
| raw-2024-05-04 |
| raw-2024-05-05 |
| raw-2024-05-06 |
| raw-2024-05-07 |
| raw-2024-05-08 |
| raw-2024-05-09 |
| raw-2024-05-10 |
| raw-2024-05-11 |
| raw-2024-05-12 |
| raw-2024-05-13 |
| raw-2024-05-14 |
| raw-2024-05-15 |
| raw-2024-05-16 |
| raw_message_sizes |
| spam_checks |
| stats_daily |
| stats_hourly |
| stats_monthly |
| stats_yearly |
| suppressions |
| webhook_requests |
+---------------------------+
211 rows in set (0.001 sec)
I've been doing my best to reduce the amount of storage time, since we've had issues with it being slow in the UI. But it's really useful to have a good backlog since users sometimes won't report that they're not receiving emails for several months.
Regardless, the storage cap feature is not working any more.
With nearly 200 days of raw messages, none of the settings seem to be working for you.
Version 3 changed the cron to use scheduled tasks instead, I think there should be a scheduled_tasks table in your main postal database which may or may not shed more light on the issue?
Looks fine to me:
MariaDB [postal]> SELECT * FROM scheduled_tasks;
+----+--------------------------------------+---------------------+
| id | name | next_run_after |
+----+--------------------------------------+---------------------+
| 1 | ActionDeletionsScheduledTask | 2024-05-21 12:15:00 |
| 2 | CheckAllDNSScheduledTask | 2024-05-21 12:15:00 |
| 3 | CleanupAuthieSessionsScheduledTask | 2024-05-21 12:15:00 |
| 4 | ExpireHeldMessagesScheduledTask | 2024-05-21 12:15:00 |
| 5 | ProcessMessageRetentionScheduledTask | 2024-05-22 03:00:00 |
| 6 | PruneSuppressionListsScheduledTask | 2024-05-22 03:00:00 |
| 7 | PruneWebhookRequestsScheduledTask | 2024-05-21 12:45:00 |
| 8 | SendNotificationsScheduledTask | 2024-05-21 11:47:21 |
| 9 | TidyQueuedMessagesTask | 2024-05-21 12:45:00 |
+----+--------------------------------------+---------------------+
9 rows in set (0.000 sec)
@willpower232 could you share how can we manually delete old emails like the task is doing? We are running out of space until a new release.
You should be able to safely remove the raw-* tables yourself until you're retaining the amount of raw message contents you desire
I'm also having the same problem. From my logs running "postal logs worker | grep ProcessMessageRetentionScheduledTask, I get the following:
worker_1 | 2024-05-27 03:00:28 +0000 INFO running task component=worker thread=tasks task=ProcessMessageRetentionScheduledTask worker_1 | 2024-05-27 03:00:28 +0000 INFO scheduling task to next run at 2024-05-28 03:00:00 UTC component=worker thread=tasks task=ProcessMessageRetentionScheduledTask worker_2 | 2024-05-28 03:00:48 +0000 INFO running task component=worker thread=tasks task=ProcessMessageRetentionScheduledTask worker_2 | 2024-05-28 03:00:48 +0000 INFO scheduling task to next run at 2024-05-29 03:00:00 UTC component=worker thread=tasks task=ProcessMessageRetentionScheduledTask
This looks like the task is completed within a second. I couldn't find any logged errors associated with this task.
I’m having the same issue. I’ve been manually clearing out tables but it’s really inconvenient and requires regular attention.
Ditto here as mentioned in another discussion.
Can we get @adamcooke's attention on this? It's not ideal and should be fixed fairly urgently IMO.
I unexpectedly ran out of space to such a point where of course mails stopped sending but even worse I couldn't actually get MySQL to run any sort of operation, even deleting tables.
I had to double our running costs by increasing the disk space before manually deleting raw messages.
Yes, I have the same out of space problem. Please @adamcooke you should handle it
