maps Allowed memory size exhausted running occ maps:scan-photos

⚠️ This issue respects the following points: ⚠️

[X] This is a bug, not a question or a configuration/webserver/proxy issue.
[X] This issue is not already reported on Github (I've searched it).
[X] Nextcloud Server is up to date. See Maintenance and Release Schedule for supported versions.
[X] Maps app is up to date. See Maintenance and Release Schedule for supported versions.
[X] I agree to follow Nextcloud's Code of Conduct.

Bug description

Have around 800k photos and running map version 1.0.2 on Nextcloud 25.0.5. I realize not all the photos are showing in the map so I ran the occ cmd and I get.

PHP Fatal error: Allowed memory size of 2147483648 bytes exhausted (tried to allocate 20480 bytes) in /var/www/html/nextcloud/3rdparty/doctrine/dbal/src/Driver/PDO/Result.php on line 121 {"reqId":"tEjayR76q1BY31SxzE1J","level":3,"time":"March 26, 2023 15:32:39","remoteAddr":"","user":"--","app":"PHP","method":"","url":"--","message":"Allowed memory size of 2147483648 bytes exhausted (tried to allocate 20480 bytes) at /var/www/html/nextcloud/3rdparty/doctrine/dbal/src/Driver/PDO/Result.php#121","userAgent":"--","version":"25.0.5.1","data":{"app":"PHP"}}

I am guessing the number of photos lead to this?

Steps to reproduce

run the cmd occ maps:scan-photos

Expected behavior

All photos being scanned

Installation method

None

Operating system

None

PHP engine version

PHP 8.1

Web server

Apache (supported)

Database engine version

MariaDB

Is this bug present after an update or on a fresh install of the server?

None

Is this bug present after an update or on a fresh install of the app?

None

Are you using the Nextcloud Server Encryption module?

None

What user-backends are you using?

[ ] Default user-backend (database)
[ ] LDAP/ Active Directory
[ ] SSO - SAML
[ ] Other

Configuration report

No response

List of activated Apps

- activity: 2.17.0
  - admin_audit: 1.15.0
  - audioplayer: 3.3.1
  - bruteforcesettings: 2.5.0
  - calendar: 4.3.1
  - camerarawpreviews: 0.8.1
  - cloud_federation_api: 1.8.0
  - comments: 1.15.0
  - contacts: 5.2.0
  - contactsinteraction: 1.6.0
  - dav: 1.24.0
  - federatedfilesharing: 1.15.0
  - files: 1.20.1
  - files_accesscontrol: 1.15.1
  - files_downloadactivity: 1.16.0
  - files_external: 1.17.0
  - files_pdfviewer: 2.6.0
  - files_rightclick: 1.4.0
  - files_sharing: 1.17.0
  - files_trashbin: 1.15.0
  - files_versions: 1.18.0
  - firstrunwizard: 2.14.0
  - groupfolders: 13.1.1
  - groupquota: 0.1.10
  - logreader: 2.10.0
  - lookup_server_connector: 1.13.0
  - maps: 1.0.2
  - memories: 4.12.4
  - metadata: 0.17.0
  - music: 1.8.1
  - nextcloud_announcements: 1.14.0
  - notifications: 2.13.1
  - oauth2: 1.13.0
  - password_policy: 1.15.0
  - photos: 2.0.1
  - previewgenerator: 5.2.1
  - privacy: 1.9.0
  - provisioning_api: 1.15.0
  - ransomware_protection: 1.14.0
  - related_resources: 1.0.4
  - richdocuments: 7.1.2
  - serverinfo: 1.15.0
  - settings: 1.7.0
  - spreed: 15.0.4
  - support: 1.8.0
  - survey_client: 1.13.0
  - suspicious_login: 4.3.0
  - systemtags: 1.15.0
  - tasks: 0.14.5
  - text: 3.6.0
  - theming: 2.0.1
  - twofactor_backupcodes: 1.14.0
  - twofactor_totp: 7.0.0
  - twofactor_webauthn: 1.1.2
  - updatenotification: 1.15.0
  - user_status: 1.5.0
  - viewer: 1.9.0
  - workflowengine: 2.7.0

Nextcloud Signing status

No response

Nextcloud Logs

No response

Browser Console

No response

Additional info

No response

Mar 26 '23 15:03 rarealphacat

Probably, you can try to limit the scan to certain users. And did you use the --now option?

Mar 26 '23 16:03 tacruc

I installed this app today on 26.0.0.11, it appears to be scanning all my photos (something like 40k) and believe I'm facing a similar/the same issue although it's referencing a different file: Allowed memory size of 536870912 bytes exhausted (tried to allocate 11332768 bytes) at /var/www/html/custom_apps/maps/lib/Helper/ExifGeoData.php#120

I did NOT run the occ command, this is just from installing the app and it's initial scan.

Mar 26 '23 21:03 vanantwerp

The issues are only related if @rarealphacat used the --now option.

Your issue is not bad. The exif extraction is leaking memory. After many photos php is killed by this error. Anyhow, the process continues with the next backgroundjob started by Cron.

Mar 26 '23 21:03 tacruc

Probably, you can try to limit the scan to certain users. And did you use the --now option?

No, I did not use the --now option. I just tried to limit scan to 1 user which has about 200k photos and seeing same error. I also tried the --now option and also same result.

Mar 27 '23 00:03 rarealphacat

I have the same problem. Mine won't go past 56k photos (with well over double that number).

Jun 21 '23 13:06 kylew01

The previous issue #126 about a memory leak is closed as fixed by #241 but I can report that I also got these same errors yesterday. The memory limit is set to 1GiB on my instance so I would really suspect a memory leak.

Allowed memory size of 1073741824 bytes exhausted (tried to allocate 9887592 bytes) at /var/www/html/nextcloud/apps/maps/lib/Helper/ExifGeoData.php#120 (multiple errors; all seem to be the same thing at row 120).

Feb 04 '24 23:02 MrRinkana

Moved my nextcloud from one cloud provider to another and while doing it moved my pictures from local path (/Photos) to external path (S3 bucket). Didn't use nextcloud to perform the move, but did it on CLI. I have done files:scan. Wondering why images don't show up on maps, I found this item. Executed occ maps:scan-photos which resulted to same issue as above, ie. running out of memory. There are some 40 000 images in my photo library.

Couple of questions:

Would it be possible to scan only a specified path? That could offer a workaround to scan the library in smaller bits
How many images one can expect to be able to scan with 500Mb of memory (the default)?
does occ maps:scan-photos go trough all of my files, both local and external?
As I did some manual moving of files, I wonder if my database is full of information about the old photos and if that data will be automatically "expired" or if it needs manual cleanup?

Jul 01 '24 15:07 nenatap

Replying to myself:

For each of the configured external storages, there is option "Allow previews". If that is unselected, then that external storage is not scanned with php occ maps:scan-photos
Setting PHP_MEMORY_LIMIT to 2Gigs got me trough 52k images. Nextcloud currently reports 1.3Gb memory in use.

Oh, going to map application states that a background job was created for adding those 52k images. Guess it will take some time. memory usage still on the same level.

Jul 01 '24 15:07 nenatap