Every scrap redownloads Messages

Open trsmith78 opened this issue 2 years ago • 0 comments

Hi,

Just installed the latest and ran using a copy of my database from the 7.6.1 scraper. The intent was so that I didn't have to redownload thousands of posts and messages for this model again. Basically I want to pick up where 7.6.1 scraper left off during it's last scrape. It appears to work with one exception, it only downloaded new post, however, it redownloads all of the messages. Not only that, it redownloads ALL of the messages every time it runs.

Here is my config: { "info": { "version": 8.0 }, "settings": { "auto_site_choice": "onlyfans", "export_type": "json", "max_threads": -1, "min_drive_space": 0, "helpers": { "renamer": true, "reformat_media": true, "downloader": true, "delete_empty_directories": true }, "webhooks": { "global_webhooks": [], "global_status": true, "auth_webhook": { "succeeded": { "webhooks": [], "status": null, "hide_sensitive_info": true }, "failed": { "webhooks": [], "status": null, "hide_sensitive_info": true } }, "download_webhook": { "succeeded": { "webhooks": [], "status": null, "hide_sensitive_info": true }, "failed": { "webhooks": [], "status": null, "hide_sensitive_info": true } } }, "exit_on_completion": false, "infinite_loop": true, "loop_timeout": "3100", "dynamic_rules_link": "https://raw.githubusercontent.com/DIGITALCRIMINALS/dynamic-rules/main/onlyfans.json", "proxies": [], "cert": "", "random_string": "d1cb7924780e11eeac81ccb0daf1c48c", "tui": { "active": false, "host": "localhost", "port": 2112, "api_key": "d1cb7925780e11eeb847ccb0daf1c48c" } }, "supported": { "onlyfans": { "settings": { "auto_profile_choice": ["1"], "auto_model_choice": [ "frandullon86" ], "auto_api_choice": true, "auto_media_choice": "0", "browser": { "auth": true }, "jobs": { "scrape": { "subscriptions": true, "messages": true, "paid_contents": true }, "metadata": { "posts": true, "comments": true } }, "download_directories": [ "user_data/sites" ], "file_directory_format": "{site_name}/{model_username}/{api_type}/{value}/{media_type}", "filename_format": "{filename}.{ext}", "metadata_directories": [ "user_data/sites" ], "metadata_directory_format": "{site_name}/{model_username}/Metadata", "delete_legacy_metadata": false, "text_length": 255, "video_quality": "source", "overwrite_files": false, "date_format": "%d-%m-%Y", "ignored_keywords": [], "ignore_type": "free", "blacklists": [], "webhook": true } }, "fansly": { "settings": { "auto_profile_choice": [], "auto_model_choice": false, "auto_api_choice": true, "auto_media_choice": "", "browser": { "auth": true }, "jobs": { "scrape": { "subscriptions": true, "messages": true, "paid_contents": true }, "metadata": { "posts": true, "comments": true } }, "download_directories": [ "user_data/sites" ], "file_directory_format": "{site_name}/{model_username}/{api_type}/{value}/{media_type}", "filename_format": "{filename}.{ext}", "metadata_directories": [ "user_data/sites" ], "metadata_directory_format": "{site_name}/{model_username}/Metadata", "delete_legacy_metadata": false, "text_length": 255, "video_quality": "source", "overwrite_files": false, "date_format": "%d-%m-%Y", "ignored_keywords": [], "ignore_type": "", "blacklists": [], "webhook": true } } } }

[2023-11-01 23:13:35] Archive Completed in 2.53 Minutes [2023-11-01 23:13:35] Pausing scraper for 3100 seconds [2023-11-02 00:05:19] Assigning Jobs Processing Scraped Stories 100%|████████████████████████████████████| 33/33 [00:00<00:00, 2116.78it/s] frandullon86: Renaming files. Processing Scraped Posts 100%|████████████████████████████████| 2401/2401 [00:01<00:00, 2062.82it/s] frandullon86: Renaming files. Processing Scraped Messages 100%|██████████████████████████████████| 101/101 [00:00<00:00, 1011.98it/s] frandullon86: Renaming files. Processing Download: Name: frandullon86 | Type: Posts | Downloading: 1 | Total: 1 | Duplicates: 0 | Directory: user_data\sites\OnlyFans\frandullon86

[2023-11-02 00:08:11] Archive Completed in 2.91 Minutes [2023-11-02 00:08:11] Pausing scraper for 3100 seconds [2023-11-02 00:59:53] Assigning Jobs Processing Scraped Stories 100%|████████████████████████████████████| 33/33 [00:00<00:00, 2174.82it/s] frandullon86: Renaming files. Processing Scraped Posts 100%|████████████████████████████████| 2401/2401 [00:01<00:00, 1916.28it/s] frandullon86: Renaming files. Processing Scraped Messages 100%|██████████████████████████████████| 101/101 [00:00<00:00, 2588.02it/s] frandullon86: Renaming files. Processing Download: Name: frandullon86 | Type: Posts | Downloading: 1 | Total: 1 | Duplicates: 0 | Directory: user_data\sites\OnlyFans\frandullon86

[2023-11-02 01:02:56] Archive Completed in 3.06 Minutes [2023-11-02 01:02:56] Pausing scraper for 3100 seconds

I deleted the messages on the HDD just to confirm that it is indeed redownloading messages every scrape. It is.

Nov 02 '23 06:11 trsmith78