notes
notes copied to clipboard
nextcloud
Unusable performance
Steps to reproduce
- Move a large tree of markdown files into the Notes tree using the filesystem (I did this on a desktop that uses the Linux nextcloud-desktop app for syncing). In my case, I have ~1200 markdown files.
- Visit the Notes app via the browser.
Expected behaviour
I expect the first time accessing the Notes app to take a bit of time while it indexes notes, and then subsequent access to take seconds at best (similar time frames to accessing the Files app).
Actual behaviour
While I expected this might take a bit of time initially, the fact is that it takes many, many minutes every time I visit the Notes app in the browser (currently at 21 minutes and still waiting).
Additionally, the long response times are not limited to the app, but also the API; I cannot use any mobile clients at all.
Screenshots
Not applicable.
Server
- Notes app version: 4.4.0
- Nextcloud version: 24.0.1
- OS: Ubuntu 20.04
- Web server: Apache 2.4
- PHP version: PHP 8.0.20
- Database: MySQL 8.0.29
Nextcloud configuration:
```
{
"system": {
"apps_paths": [
{
"path": "\/snap\/nextcloud\/current\/htdocs\/apps",
"url": "\/apps",
"writable": false
},
{
"path": "\/var\/snap\/nextcloud\/current\/nextcloud\/extra-apps",
"url": "\/extra-apps",
"writable": true
}
],
"supportedDatabases": [
"mysql"
],
"memcache.locking": "\\OC\\Memcache\\Redis",
"memcache.local": "\\OC\\Memcache\\Redis",
"redis": {
"host": "***REMOVED SENSITIVE VALUE***",
"port": 0
},
"passwordsalt": "***REMOVED SENSITIVE VALUE***",
"secret": "***REMOVED SENSITIVE VALUE***",
"trusted_domains": [
"localhost",
"cloud.mwop.net"
],
"trusted_proxies": "***REMOVED SENSITIVE VALUE***",
"datadirectory": "***REMOVED SENSITIVE VALUE***",
"dbtype": "mysql",
"version": "24.0.1.1",
"overwrite.cli.url": "http:\/\/localhost",
"dbname": "***REMOVED SENSITIVE VALUE***",
"dbhost": "***REMOVED SENSITIVE VALUE***",
"dbport": "",
"dbtableprefix": "oc_",
"mysql.utf8mb4": true,
"dbuser": "***REMOVED SENSITIVE VALUE***",
"dbpassword": "***REMOVED SENSITIVE VALUE***",
"installed": true,
"instanceid": "***REMOVED SENSITIVE VALUE***",
"objectstore": {
"class": "\\OC\\Files\\ObjectStore\\S3",
"arguments": {
"bucket": "cloud-mwop-net",
"key": "***REMOVED SENSITIVE VALUE***",
"secret": "***REMOVED SENSITIVE VALUE***",
"hostname": "cloud-mwop-net.nyc3.digitaloceanspaces.com",
"region": "nyc3",
"port": 443,
"use_ssl": true,
"use_path_style": true
}
},
"maintenance": false,
"theme": "",
"loglevel": 0,
"mail_from_address": "***REMOVED SENSITIVE VALUE***",
"mail_smtpmode": "smtp",
"mail_smtpsecure": "tls",
"mail_sendmailmode": "smtp",
"mail_domain": "***REMOVED SENSITIVE VALUE***",
"mail_smtpauthtype": "LOGIN",
"mail_smtpauth": 1,
"mail_smtphost": "***REMOVED SENSITIVE VALUE***",
"mail_smtpport": "587",
"mail_smtpname": "***REMOVED SENSITIVE VALUE***",
"mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
"default_phone_region": "us"
}
}
```
Client
Please complete the following information.
- Browser (incl. version): Brave 1.41.96
- OS: Ubuntu 22.04
Log files
I'm including the last 25 lines of the nextcloud.log file that mention the notes app; anything more is too big.
{"reqId":"hV7MzXuDsZieIDTiZiaB","level":0,"time":"2022-07-15T15:20:49+00:00","remoteAddr":"208.107.9.34","user":"matthew","app":"notes","method":"GET","url":"/index.php/apps/notes/notes","message":"Could not insert meta data for note 165298","userAgent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.5060.114 Safari/537.36","version":"24.0.1.1","exception":{"Exception":"OC\\DB\\Exceptions\\DbalException","Message":"An exception occurred while executing a query: SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry '165298-matthew' for key 'oc_notes_meta.notes_meta_file_user_index'","Code":1062,"Trace":[{"file":"/snap/nextcloud/31033/htdocs/lib/private/DB/QueryBuilder/QueryBuilder.php","line":329,"function":"wrap","class":"OC\\DB\\Exceptions\\DbalException","type":"::"},{"file":"/snap/nextcloud/31033/htdocs/lib/public/AppFramework/Db/QBMapper.php","line":139,"function":"executeStatement","class":"OC\\DB\\QueryBuilder\\QueryBuilder","type":"->"},{"file":"/var/snap/nextcloud/31033/nextcloud/extra-apps/notes/lib/Service/MetaService.php","line":141,"function":"insert","class":"OCP\\AppFramework\\Db\\QBMapper","type":"->"},{"file":"/var/snap/nextcloud/31033/nextcloud/extra-apps/notes/lib/Service/MetaService.php","line":85,"function":"createMeta","class":"OCA\\Notes\\Service\\MetaService","type":"->"},{"file":"/var/snap/nextcloud/31033/nextcloud/extra-apps/notes/lib/Controller/Helper.php","line":78,"function":"getAll","class":"OCA\\Notes\\Service\\MetaService","type":"->"},{"file":"/var/snap/nextcloud/31033/nextcloud/extra-apps/notes/lib/Controller/NotesController.php","line":63,"function":"getNotesAndCategories","class":"OCA\\Notes\\Controller\\Helper","type":"->"},{"file":"/var/snap/nextcloud/31033/nextcloud/extra-apps/notes/lib/Service/Util.php","line":26,"function":"OCA\\Notes\\Controller\\{closure}","class":"OCA\\Notes\\Controller\\NotesController","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/snap/nextcloud/31033/nextcloud/extra-apps/notes/lib/Controller/Helper.php","line":140,"function":"retryIfLocked","class":"OCA\\Notes\\Service\\Util","type":"::"},{"file":"/var/snap/nextcloud/31033/nextcloud/extra-apps/notes/lib/Controller/NotesController.php","line":89,"function":"handleErrorResponse","class":"OCA\\Notes\\Controller\\Helper","type":"->"},{"file":"/snap/nextcloud/31033/htdocs/lib/private/AppFramework/Http/Dispatcher.php","line":225,"function":"index","class":"OCA\\Notes\\Controller\\NotesController","type":"->"},{"file":"/snap/nextcloud/31033/htdocs/lib/private/AppFramework/Http/Dispatcher.php","line":133,"function":"executeController","class":"OC\\AppFramework\\Http\\Dispatcher","type":"->"},{"file":"/snap/nextcloud/31033/htdocs/lib/private/AppFramework/App.php","line":172,"function":"dispatch","class":"OC\\AppFramework\\Http\\Dispatcher","type":"->"},{"file":"/snap/nextcloud/31033/htdocs/lib/private/Route/Router.php","line":298,"function":"main","class":"OC\\AppFramework\\App","type":"::"},{"file":"/snap/nextcloud/31033/htdocs/lib/base.php","line":1023,"function":"match","class":"OC\\Route\\Router","type":"->"},{"file":"/snap/nextcloud/31033/htdocs/index.php","line":36,"function":"handleRequest","class":"OC","type":"::"}],"File":"/snap/nextcloud/31033/htdocs/lib/private/DB/Exceptions/DbalException.php","Line":71,"Previous":{"Exception":"Doctrine\\DBAL\\Exception\\UniqueConstraintViolationException","Message":"An exception occurred while executing a query: SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry '165298-matthew' for key 'oc_notes_meta.notes_meta_file_user_index'","Code":1062,"Trace":[{"file":"/snap/nextcloud/31033/htdocs/3rdparty/doctrine/dbal/src/Connection.php","line":1780,"function":"convert","class":"Doctrine\\DBAL\\Driver\\API\\MySQL\\ExceptionConverter","type":"->"},{"file":"/snap/nextcloud/31033/htdocs/3rdparty/doctrine/dbal/src/Connection.php","line":1719,"function":"handleDriverException","class":"Doctrine\\DBAL\\Connection","type":"->"},{"file":"/snap/nextcloud/31033/htdocs/3rdparty/doctrine/dbal/src/Connection.php","line":1173,"function":"convertExceptionDuringQuery","class":"Doctrine\\DBAL\\Connection","type":"->"},{"file":"/snap/nextcloud/31033/htdocs/lib/private/DB/Connection.php","line":294,"function":"executeStatement","class":"Doctrine\\DBAL\\Connection","type":"->"},{"file":"/snap/nextcloud/31033/htdocs/3rdparty/doctrine/dbal/src/Query/QueryBuilder.php","line":354,"function":"executeStatement","class":"OC\\DB\\Connection","type":"->"},{"file":"/snap/nextcloud/31033/htdocs/lib/private/DB/QueryBuilder/QueryBuilder.php","line":281,"function":"execute","class":"Doctrine\\DBAL\\Query\\QueryBuilder","type":"->"},{"file":"/snap/nextcloud/31033/htdocs/lib/private/DB/QueryBuilder/QueryBuilder.php","line":327,"function":"execute","class":"OC\\DB\\QueryBuilder\\QueryBuilder","type":"->"},{"file":"/snap/nextcloud/31033/htdocs/lib/public/AppFramework/Db/QBMapper.php","line":139,"function":"executeStatement","class":"OC\\DB\\QueryBuilder\\QueryBuilder","type":"->"},{"file":"/var/snap/nextcloud/31033/nextcloud/extra-apps/notes/lib/Service/MetaService.php","line":141,"function":"insert","class":"OCP\\AppFramework\\Db\\QBMapper","type":"->"},{"file":"/var/snap/nextcloud/31033/nextcloud/extra-apps/notes/lib/Service/MetaService.php","line":85,"function":"createMeta","class":"OCA\\Notes\\Service\\MetaService","type":"->"},{"file":"/var/snap/nextcloud/31033/nextcloud/extra-apps/notes/lib/Controller/Helper.php","line":78,"function":"getAll","class":"OCA\\Notes\\Service\\MetaService","type":"->"},{"file":"/var/snap/nextcloud/31033/nextcloud/extra-apps/notes/lib/Controller/NotesController.php","line":63,"function":"getNotesAndCategories","class":"OCA\\Notes\\Controller\\Helper","type":"->"},{"file":"/var/snap/nextcloud/31033/nextcloud/extra-apps/notes/lib/Service/Util.php","line":26,"function":"OCA\\Notes\\Controller\\{closure}","class":"OCA\\Notes\\Controller\\NotesController","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/snap/nextcloud/31033/nextcloud/extra-apps/notes/lib/Controller/Helper.php","line":140,"function":"retryIfLocked","class":"OCA\\Notes\\Service\\Util","type":"::"},{"file":"/var/snap/nextcloud/31033/nextcloud/extra-apps/notes/lib/Controller/NotesController.php","line":89,"function":"handleErrorResponse","class":"OCA\\Notes\\Controller\\Helper","type":"->"},{"file":"/snap/nextcloud/31033/htdocs/lib/private/AppFramework/Http/Dispatcher.php","line":225,"function":"index","class":"OCA\\Notes\\Controller\\NotesController","type":"->"},{"file":"/snap/nextcloud/31033/htdocs/lib/private/AppFramework/Http/Dispatcher.php","line":133,"function":"executeController","class":"OC\\AppFramework\\Http\\Dispatcher","type":"->"},{"file":"/snap/nextcloud/31033/htdocs/lib/private/AppFramework/App.php","line":172,"function":"dispatch","class":"OC\\AppFramework\\Http\\Dispatcher","type":"->"},{"file":"/snap/nextcloud/31033/htdocs/lib/private/Route/Router.php","line":298,"function":"main","class":"OC\\AppFramework\\App","type":"::"},{"file":"/snap/nextcloud/31033/htdocs/lib/base.php","line":1023,"function":"match","class":"OC\\Route\\Router","type":"->"},{"file":"/snap/nextcloud/31033/htdocs/index.php","line":36,"function":"handleRequest","class":"OC","type":"::"}],"File":"/snap/nextcloud/31033/htdocs/3rdparty/doctrine/dbal/src/Driver/API/MySQL/ExceptionConverter.php","Line":60,"Previous":{"Exception":"Doctrine\\DBAL\\Driver\\PDO\\Exception","Message":"SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry '165298-matthew' for key 'oc_notes_meta.notes_meta_file_user_index'","Code":1062,"Trace":[{"file":"/snap/nextcloud/31033/htdocs/3rdparty/doctrine/dbal/src/Driver/PDO/Statement.php","line":94,"function":"new","class":"Doctrine\\DBAL\\Driver\\PDO\\Exception","type":"::"},{"file":"/snap/nextcloud/31033/htdocs/3rdparty/doctrine/dbal/src/Connection.php","line":1163,"function":"execute","class":"Doctrine\\DBAL\\Driver\\PDO\\Statement","type":"->"},{"file":"/snap/nextcloud/31033/htdocs/lib/private/DB/Connection.php","line":294,"function":"executeStatement","class":"Doctrine\\DBAL\\Connection","type":"->"},{"file":"/snap/nextcloud/31033/htdocs/3rdparty/doctrine/dbal/src/Query/QueryBuilder.php","line":354,"function":"executeStatement","class":"OC\\DB\\Connection","type":"->"},{"file":"/snap/nextcloud/31033/htdocs/lib/private/DB/QueryBuilder/QueryBuilder.php","line":281,"function":"execute","class":"Doctrine\\DBAL\\Query\\QueryBuilder","type":"->"},{"file":"/snap/nextcloud/31033/htdocs/lib/private/DB/QueryBuilder/QueryBuilder.php","line":327,"function":"execute","class":"OC\\DB\\QueryBuilder\\QueryBuilder","type":"->"},{"file":"/snap/nextcloud/31033/htdocs/lib/public/AppFramework/Db/QBMapper.php","line":139,"function":"executeStatement","class":"OC\\DB\\QueryBuilder\\QueryBuilder","type":"->"},{"file":"/var/snap/nextcloud/31033/nextcloud/extra-apps/notes/lib/Service/MetaService.php","line":141,"function":"insert","class":"OCP\\AppFramework\\Db\\QBMapper","type":"->"},{"file":"/var/snap/nextcloud/31033/nextcloud/extra-apps/notes/lib/Service/MetaService.php","line":85,"function":"createMeta","class":"OCA\\Notes\\Service\\MetaService","type":"->"},{"file":"/var/snap/nextcloud/31033/nextcloud/extra-apps/notes/lib/Controller/Helper.php","line":78,"function":"getAll","class":"OCA\\Notes\\Service\\MetaService","type":"->"},{"file":"/var/snap/nextcloud/31033/nextcloud/extra-apps/notes/lib/Controller/NotesController.php","line":63,"function":"getNotesAndCategories","class":"OCA\\Notes\\Controller\\Helper","type":"->"},{"file":"/var/snap/nextcloud/31033/nextcloud/extra-apps/notes/lib/Service/Util.php","line":26,"function":"OCA\\Notes\\Controller\\{closure}","class":"OCA\\Notes\\Controller\\NotesController","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/snap/nextcloud/31033/nextcloud/extra-apps/notes/lib/Controller/Helper.php","line":140,"function":"retryIfLocked","class":"OCA\\Notes\\Service\\Util","type":"::"},{"file":"/var/snap/nextcloud/31033/nextcloud/extra-apps/notes/lib/Controller/NotesController.php","line":89,"function":"handleErrorResponse","class":"OCA\\Notes\\Controller\\Helper","type":"->"},{"file":"/snap/nextcloud/31033/htdocs/lib/private/AppFramework/Http/Dispatcher.php","line":225,"function":"index","class":"OCA\\Notes\\Controller\\NotesController","type":"->"},{"file":"/snap/nextcloud/31033/htdocs/lib/private/AppFramework/Http/Dispatcher.php","line":133,"function":"executeController","class":"OC\\AppFramework\\Http\\Dispatcher","type":"->"},{"file":"/snap/nextcloud/31033/htdocs/lib/private/AppFramework/App.php","line":172,"function":"dispatch","class":"OC\\AppFramework\\Http\\Dispatcher","type":"->"},{"file":"/snap/nextcloud/31033/htdocs/lib/private/Route/Router.php","line":298,"function":"main","class":"OC\\AppFramework\\App","type":"::"},{"file":"/snap/nextcloud/31033/htdocs/lib/base.php","line":1023,"function":"match","class":"OC\\Route\\Router","type":"->"},{"file":"/snap/nextcloud/31033/htdocs/index.php","line":36,"function":"handleRequest","class":"OC","type":"::"}],"File":"/snap/nextcloud/31033/htdocs/3rdparty/doctrine/dbal/src/Driver/PDO/Exception.php","Line":26,"Previous":{"Exception":"PDOException","Message":"SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry '165298-matthew' for key 'oc_notes_meta.notes_meta_file_user_index'","Code":"23000","Trace":[{"file":"/snap/nextcloud/31033/htdocs/3rdparty/doctrine/dbal/src/Driver/PDO/Statement.php","line":92,"function":"execute","class":"PDOStatement","type":"->"},{"file":"/snap/nextcloud/31033/htdocs/3rdparty/doctrine/dbal/src/Connection.php","line":1163,"function":"execute","class":"Doctrine\\DBAL\\Driver\\PDO\\Statement","type":"->"},{"file":"/snap/nextcloud/31033/htdocs/lib/private/DB/Connection.php","line":294,"function":"executeStatement","class":"Doctrine\\DBAL\\Connection","type":"->"},{"file":"/snap/nextcloud/31033/htdocs/3rdparty/doctrine/dbal/src/Query/QueryBuilder.php","line":354,"function":"executeStatement","class":"OC\\DB\\Connection","type":"->"},{"file":"/snap/nextcloud/31033/htdocs/lib/private/DB/QueryBuilder/QueryBuilder.php","line":281,"function":"execute","class":"Doctrine\\DBAL\\Query\\QueryBuilder","type":"->"},{"file":"/snap/nextcloud/31033/htdocs/lib/private/DB/QueryBuilder/QueryBuilder.php","line":327,"function":"execute","class":"OC\\DB\\QueryBuilder\\QueryBuilder","type":"->"},{"file":"/snap/nextcloud/31033/htdocs/lib/public/AppFramework/Db/QBMapper.php","line":139,"function":"executeStatement","class":"OC\\DB\\QueryBuilder\\QueryBuilder","type":"->"},{"file":"/var/snap/nextcloud/31033/nextcloud/extra-apps/notes/lib/Service/MetaService.php","line":141,"function":"insert","class":"OCP\\AppFramework\\Db\\QBMapper","type":"->"},{"file":"/var/snap/nextcloud/31033/nextcloud/extra-apps/notes/lib/Service/MetaService.php","line":85,"function":"createMeta","class":"OCA\\Notes\\Service\\MetaService","type":"->"},{"file":"/var/snap/nextcloud/31033/nextcloud/extra-apps/notes/lib/Controller/Helper.php","line":78,"function":"getAll","class":"OCA\\Notes\\Service\\MetaService","type":"->"},{"file":"/var/snap/nextcloud/31033/nextcloud/extra-apps/notes/lib/Controller/NotesController.php","line":63,"function":"getNotesAndCategories","class":"OCA\\Notes\\Controller\\Helper","type":"->"},{"file":"/var/snap/nextcloud/31033/nextcloud/extra-apps/notes/lib/Service/Util.php","line":26,"function":"OCA\\Notes\\Controller\\{closure}","class":"OCA\\Notes\\Controller\\NotesController","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/snap/nextcloud/31033/nextcloud/extra-apps/notes/lib/Controller/Helper.php","line":140,"function":"retryIfLocked","class":"OCA\\Notes\\Service\\Util","type":"::"},{"file":"/var/snap/nextcloud/31033/nextcloud/extra-apps/notes/lib/Controller/NotesController.php","line":89,"function":"handleErrorResponse","class":"OCA\\Notes\\Controller\\Helper","type":"->"},{"file":"/snap/nextcloud/31033/htdocs/lib/private/AppFramework/Http/Dispatcher.php","line":225,"function":"index","class":"OCA\\Notes\\Controller\\NotesController","type":"->"},{"file":"/snap/nextcloud/31033/htdocs/lib/private/AppFramework/Http/Dispatcher.php","line":133,"function":"executeController","class":"OC\\AppFramework\\Http\\Dispatcher","type":"->"},{"file":"/snap/nextcloud/31033/htdocs/lib/private/AppFramework/App.php","line":172,"function":"dispatch","class":"OC\\AppFramework\\Http\\Dispatcher","type":"->"},{"file":"/snap/nextcloud/31033/htdocs/lib/private/Route/Router.php","line":298,"function":"main","class":"OC\\AppFramework\\App","type":"::"},{"file":"/snap/nextcloud/31033/htdocs/lib/base.php","line":1023,"function":"match","class":"OC\\Route\\Router","type":"->"},{"file":"/snap/nextcloud/31033/htdocs/index.php","line":36,"function":"handleRequest","class":"OC","type":"::"}],"File":"/snap/nextcloud/31033/htdocs/3rdparty/doctrine/dbal/src/Driver/PDO/Statement.php","Line":92}}},"CustomMessage":"Could not insert meta data for note 165298"}}
As a follow-up...
I rebooted this particular server, and then re-accessed the Notes app. This particular run took over 20 minutes... but subsequent access has taken seconds, and I'm actually able to start syncing with the Nextcloud Notes android app. I expect as a result this may not be reproducible.
Another finding: any time I restart this server, I get the initial run of 20+ minutes, followed by reasonable subsequent loads.
Thanks for reporting this issue. Could you please provide more information about your data? You're writing about ~1200 markdown files. How large are they (in sum, megabytes)?
To be honest, the notes app is not designed to handle such an amount of note files. But on the other hand, it should not be that slow. But this also depends on the hardware you are using. So, what's about CPU and storage (speed) of your server?
