server Filenames with special characters (German "Umlaute") not found by "files:scan"

Steps to reproduce

Create a file containing special characters (e.g. test-aöü-test)
Sync file to Nextcloud instance (by web ui, Linux client, Gnome file manager and integrated Nextcloud client)
occ files:scan will say "stat: der Aufruf von statx für '/xxxx/test--test' ist fehlgeschlagen: Datei oder Verzeichnis nicht gefunden". (call of statx has failed, file not found).

Expected behaviour

File should be found whatever characters the filename contains.

Actual behaviour

The file is not found by the file scan. Note that the filename as shown by the command output is missing all non-ASCII(?) characters and therefore obviously wrong. Also: The file seems to be synced correctly (at least I hope so)! All clients and the web ui handle the file completely normally. So the error is not really obvious as long as you are not running the scan manually. occ files:cleanup does not remove or notice this "missing" file.

Server configuration

Operating system: Raspian OS, current version (32bit with 64bit kernel). Locale set to UTF8.de-DE

Web server: Apache 2.4 with PHP-FPM

Database: MariaDB 10.3 Locale of the tables set to utf8mb4_bin

PHP version: 8.0 Locale seems IMHO to be set correctly both for cli and fpm.

Nextcloud version: (see Nextcloud admin page) 22.2.3

Updated from an older Nextcloud/ownCloud or fresh install: Long-running instance, migrated from ownCloud years ago.

Where did you install Nextcloud from: tar package

Signing status:

Signing status

No errors have been found.

List of activated apps:

App list

Enabled:
  - accessibility: 1.8.0
  - activity: 2.15.0
  - bookmarks: 10.0.3
  - bruteforcesettings: 2.3.0
  - calendar: 3.0.4
  - circles: 22.1.1
  - cloud_federation_api: 1.5.0
  - comments: 1.12.0
  - contacts: 4.0.7
  - dav: 1.19.0
  - federatedfilesharing: 1.12.0
  - files: 1.17.0
  - files_external: 1.13.0
  - files_fulltextsearch: 22.0.1
  - files_pdfviewer: 2.3.1
  - files_retention: 1.11.1
  - files_rightclick: 1.1.0
  - files_sharing: 1.14.0
  - files_trashbin: 1.12.0
  - files_versions: 1.15.0
  - firstrunwizard: 2.11.0
  - fulltextsearch: 22.0.1
  - fulltextsearch_elasticsearch: 22.0.1
  - gpxpod: 4.3.0
  - impersonate: 1.9.0
  - logreader: 2.7.0
  - lookup_server_connector: 1.10.0
  - maps: 0.1.10
  - metadata: 0.15.0
  - nextcloud_announcements: 1.11.0
  - notes: 4.3.0
  - notifications: 2.10.1
  - oauth2: 1.10.0
  - password_policy: 1.12.0
  - passwords: 2021.12.20
  - privacy: 1.6.0
  - provisioning_api: 1.12.0
  - qownnotesapi: 21.12.0
  - richdocuments: 4.2.3
  - serverinfo: 1.12.0
  - settings: 1.4.0
  - sharebymail: 1.12.0
  - systemtags: 1.12.0
  - tasks: 0.14.2
  - text: 3.3.0
  - theming: 1.13.0
  - twofactor_backupcodes: 1.11.0
  - twofactor_nextcloud_notification: 3.3.1
  - twofactor_totp: 6.2.0
  - unroundedcorners: 1.0.3
  - updatenotification: 1.12.0
  - user_ldap: 1.12.1
  - viewer: 1.6.0
  - workflowengine: 2.4.0
Disabled:
  - admin_audit
  - collectives
  - contactsinteraction
  - dashboard
  - encryption
  - federation
  - files_markdown
  - files_texteditor
  - files_videoplayer
  - photos
  - recommendations
  - support
  - survey_client
  - user_status
  - weather_status

Nextcloud configuration:

Config report

{
    "system": {
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "localhost",
            "mydomain",
            "mydomain2"
        ],
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "mysql",
        "version": "22.2.3.0",
        "overwrite.cli.url": "https:\/\/mydomain\/nextcloud",
        "overwritehost": "mydomain:8443",
        "overwritewebroot": "\/nextcloud",
        "htaccess.RewriteBase": "\/nextcloud",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "memcache.local": "\\OC\\Memcache\\APCu",
        "ldapIgnoreNamingRules": false,
        "ldapProviderFactory": "OCA\\User_LDAP\\LDAPProviderFactory",
        "default_language": "de",
        "default_locale": "de_DE",
        "default_phone_region": "DE",
        "defaultapp": "files",
        "mysql.utf8mb4": true,
        "maintenance": false,
        "theme": "",
        "loglevel": 3,
        "mail_smtpmode": "sendmail",
        "mail_smtpsecure": "tls",
        "mail_sendmailmode": "smtp",
        "mail_smtpauth": 1,
        "mail_smtpauthtype": "LOGIN",
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
        "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
        "updater.release.channel": "stable",
        "has_rebuilt_cache": true,
        "trashbin_retention_obligation": "auto, 30",
        "versions_retention_obligation": "auto, 60",
        "session_keepalive": false,
        "auto_logout": false,
        "session_lifetime": 604800,
        "remember_login_cookie_lifetime": 63072000,
        "app_install_overwrite": [
            "ocdownloader",
            "twofactor_admin",
            "bruteforcesettings",
            "mindmap_app"
        ],
        "data-fingerprint": "830f24f7d337cd03c421a52196fcd5e8"
    }
}

Are you using external storage, if yes which one: local/smb/sftp/... local, webdav Are you using encryption: yes/no no Are you using an external user-backend, if yes which one: LDAP/ActiveDirectory/Webdav/... ldap

LDAP configuration (delete this part if not used)

LDAP config

+-------------------------------+---------------------------------------------+
| Configuration                 | s01                                         |
+-------------------------------+---------------------------------------------+
| hasMemberOfFilterSupport      |                                             |
| homeFolderNamingRule          |                                             |
| lastJpegPhotoLookup           | 0                                           |
| ldapAgentName                 |                                             |
| ldapAgentPassword             | ***                                         |
| ldapAttributesForGroupSearch  |                                             |
| ldapAttributesForUserSearch   |                                             |
| ldapBackupHost                |                                             |
| ldapBackupPort                |                                             |
| ldapBase                      | dc=domain,dc=de                             |
| ldapBaseGroups                | ou=groups,dc=domain,dc=de                   |
| ldapBaseUsers                 | ou=people,dc=domain,dc=de                   |
| ldapCacheTTL                  | 7200                                        |
| ldapConfigurationActive       | 1                                           |
| ldapDefaultPPolicyDN          |                                             |
| ldapDynamicGroupMemberURL     |                                             |
| ldapEmailAttribute            | mail                                        |
| ldapExperiencedAdmin          | 0                                           |
| ldapExpertUUIDGroupAttr       |                                             |
| ldapExpertUUIDUserAttr        |                                             |
| ldapExpertUsernameAttr        |                                             |
| ldapExtStorageHomeAttribute   |                                             |
| ldapGidNumber                 | gidNumber                                   |
| ldapGroupDisplayName          | cn                                          |
| ldapGroupFilter               | (&(|(objectclass=groupOfNames)))            |
| ldapGroupFilterGroups         |                                             |
| ldapGroupFilterMode           | 0                                           |
| ldapGroupFilterObjectclass    | groupOfNames                                |
| ldapGroupMemberAssocAttr      | member                                      |
| ldapHost                      | localhost                                   |
| ldapIgnoreNamingRules         |                                             |
| ldapLoginFilter               | (&(|(objectclass=inetOrgPerson))(uid=%uid)) |
| ldapLoginFilterAttributes     |                                             |
| ldapLoginFilterEmail          | 0                                           |
| ldapLoginFilterMode           | 0                                           |
| ldapLoginFilterUsername       | 1                                           |
| ldapMatchingRuleInChainState  | unknown                                     |
| ldapNestedGroups              | 0                                           |
| ldapOverrideMainServer        |                                             |
| ldapPagingSize                | 500                                         |
| ldapPort                      | 389                                         |
| ldapQuotaAttribute            |                                             |
| ldapQuotaDefault              |                                             |
| ldapTLS                       | 0                                           |
| ldapUserAvatarRule            | default                                     |
| ldapUserDisplayName           | displayname                                 |
| ldapUserDisplayName2          |                                             |
| ldapUserFilter                | (|(objectclass=inetOrgPerson))              |
| ldapUserFilterGroups          |                                             |
| ldapUserFilterMode            | 0                                           |
| ldapUserFilterObjectclass     | inetOrgPerson                               |
| ldapUuidGroupAttribute        | auto                                        |
| ldapUuidUserAttribute         | auto                                        |
| turnOffCertCheck              | 0                                           |
| turnOnPasswordChange          | 0                                           |
| useMemberOfToDetectMembership | 1                                           |
+-------------------------------+---------------------------------------------+

Client configuration

Browser: Brave, Firefox in current versions Nextcloud client 3.4.1 and 3.3.6 on Linux Current client on Android ...

Operating system: Linux, Android

Logs

Web server error log

No errors

Nextcloud log (data/nextcloud.log)

Nextcloud log

No errors in the log during occ files:scan

Browser log

IMHO not relevant

Jan 11 '22 04:01 Ra72xx

Here a SQL dump of oc_filecache for some missing entry:

`-- phpMyAdmin SQL Dump -- version 5.0.4deb2 -- https://www.phpmyadmin.net/

-- Host: localhost:3306 -- Erstellungszeit: 11. Jan 2022 um 05:54 -- Server-Version: 10.3.31-MariaDB-0+deb10u1 -- PHP-Version: 8.0.14

SET SQL_MODE = "NO_AUTO_VALUE_ON_ZERO"; START TRANSACTION; SET time_zone = "+00:00";

/*!40101 SET @OLD_CHARACTER_SET_CLIENT=@@CHARACTER_SET_CLIENT /; /!40101 SET @OLD_CHARACTER_SET_RESULTS=@@CHARACTER_SET_RESULTS /; /!40101 SET @OLD_COLLATION_CONNECTION=@@COLLATION_CONNECTION /; /!40101 SET NAMES utf8mb4 */;

-- -- Datenbank: `nextcloud`

-- -- Tabellenstruktur fÃ¼r Tabelle `oc_filecache`

CREATE TABLE oc_filecache ( fileid bigint(20) NOT NULL, storage bigint(20) NOT NULL DEFAULT 0, path varchar(4000) COLLATE utf8mb4_bin DEFAULT NULL, path_hash varchar(32) COLLATE utf8mb4_bin NOT NULL DEFAULT '', parent bigint(20) NOT NULL DEFAULT 0, name varchar(250) COLLATE utf8mb4_bin DEFAULT NULL, mimetype bigint(20) NOT NULL DEFAULT 0, mimepart bigint(20) NOT NULL DEFAULT 0, size bigint(20) NOT NULL DEFAULT 0, mtime bigint(20) NOT NULL DEFAULT 0, storage_mtime bigint(20) NOT NULL DEFAULT 0, encrypted int(11) NOT NULL DEFAULT 0, unencrypted_size bigint(20) NOT NULL DEFAULT 0, etag varchar(40) COLLATE utf8mb4_bin DEFAULT NULL, permissions int(11) DEFAULT 0, checksum varchar(255) COLLATE utf8mb4_bin DEFAULT NULL ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin ROW_FORMAT=COMPRESSED;

-- -- Daten fÃ¼r Tabelle `oc_filecache`

INSERT INTO oc_filecache (fileid, storage, path, path_hash, parent, name, mimetype, mimepart, size, mtime, storage_mtime, encrypted, unencrypted_size, etag, permissions, checksum) VALUES (10206292, 3, 'files/Dropbox/Ã¶Ã¶dÃ¶dÃ¶.xlsx', '5e0bcdcefc394d5795cde22225178885', 10200978, 'Ã¶Ã¶dÃ¶dÃ¶.xlsx', 97, 6, 5439, 1620699860, 1620699860, 0, 0, '9f92dd020e09025036873de12a916b61', 27, 'SHA1:dc76494488f2593e1d99bb52f0b7f573f665d715');

-- -- Indizes der exportierten Tabellen

-- -- Indizes fÃ¼r die Tabelle `oc_filecache`

ALTER TABLE oc_filecache ADD PRIMARY KEY (fileid), ADD UNIQUE KEY fs_storage_path_hash (storage,path_hash), ADD KEY fs_parent_name_hash (parent,name), ADD KEY fs_storage_mimetype (storage,mimetype), ADD KEY fs_storage_mimepart (storage,mimepart), ADD KEY fs_storage_size (storage,size,fileid), ADD KEY fs_mtime (mtime), ADD KEY fs_size (size);

-- -- AUTO_INCREMENT fÃ¼r exportierte Tabellen

-- -- AUTO_INCREMENT fÃ¼r Tabelle `oc_filecache`

ALTER TABLE oc_filecache MODIFY fileid bigint(20) NOT NULL AUTO_INCREMENT, AUTO_INCREMENT=10221848; COMMIT;

/*!40101 SET CHARACTER_SET_CLIENT=@OLD_CHARACTER_SET_CLIENT /; /!40101 SET CHARACTER_SET_RESULTS=@OLD_CHARACTER_SET_RESULTS /; /!40101 SET COLLATION_CONNECTION=@OLD_COLLATION_CONNECTION */;`

Jan 11 '22 04:01 Ra72xx

Similar to https://github.com/nextcloud/server/issues/7762

Jan 11 '22 07:01 solracsf

Hi, please update to 24.0.9 or better 25.0.3 and report back if it fixes the issue. Thank you!

My goal is to add a label like e.g. 25-feedback to this ticket of an up-to-date major Nextcloud version where the bug could be reproduced. However this is not going to work without your help. So thanks for all your effort!

If you don't manage to reproduce the issue in time and the issue gets closed but you can reproduce the issue afterwards, feel free to create a new bug report with up-to-date information by following this link: https://github.com/nextcloud/server/issues/new?assignees=&labels=bug%2C0.+Needs+triage&template=BUG_REPORT.yml&title=%5BBug%5D%3A+

Jan 23 '23 12:01 szaimen

Same problem with german Umlaute here in NC 25.0.4 (up-to-date stable):

Using nextcloudcmd to sync folders locally by script. One folder is named "Büro" (german for "Office"), but nextcloudcmd is not able to find that folder. The Windows & Linux UI version can select that folder and is syncing without problems; only nextcloudcmd is struggeling:

call: nextcloudcmd -u buero -p *** --non-interactive --path "\Büro" c:\destpath https://x.y.de

No change, if the path is given with out without quotation marks

error:

02-25 11:23:38:883 [ info nextcloud.sync.networkjob ]: OCC::LsColJob created for "https://x.y.de" + "/B├╝ro/" "OCC::DiscoverySingleDirectoryJob" 02-25 11:23:38:883 [ info nextcloud.sync.engine ]: shouldDiscoverLocaly "" true 02-25 11:23:38:914 [ warning nextcloud.sync.networkjob ]: QNetworkReply::ContentNotFoundError "Server replied \"404 Not Found\" to \"PROPFIND https://x.y.de/remote.php/dav/files/buero/B├╝ro/\"" QVariant(int, 404) 02-25 11:23:38:914 [ info nextcloud.sync.networkjob.lscol ]: LSCOL of QUrl("https://x.y.de/remote.php/dav/files/buero/B├╝ro/") FINISHED WITH STATUS "ContentNotFoundError Server replied \"404 Not Found\" to \"PROPFIND https://x.y.de/remote.php/dav/files/buero/B├╝ro/\"" 02-25 11:23:38:914 [ warning nextcloud.sync.discovery ]: LSCOL job error "Error transferring https://x.y.de/remote.php/dav/files/buero/B├╝ro/ - server replied: Not Found" 404 QNetworkReply::ContentNotFoundError 02-25 11:23:38:914 [ warning nextcloud.sync.discovery ]: Server error in directory "" 404 02-25 11:23:38:914 [ warning default ]: Sync error: "Server replied with an error while reading directory \"\" : Error transferring https://x.y.de/remote.php/dav/files/buero/B├╝ro/ - server replied: Not Found" 02-25 11:23:38:914 [ info nextcloud.sync.engine ]: Sync run took 32 ms

FYI: User named "buero" - folder within that user "buero" is named "Büro"

Feb 25 '23 10:02 kimjohans

any update on this ? Upload through browser does work and scanning some folders work too but files won't be scanned

Jun 15 '23 07:06 vegaz337

I haven't had any issue like this any more for months. However, much has changed in the meantime (NC updates, switch from MariaDB to PostgreSQL, updates of the underlying OS), so I cannot really say what has helped in my case.

Jun 15 '23 12:06 Ra72xx

I have a similar issue with Czech characters. Neither folders nor files containing Czech characters are visible/syncs from Windows nextcloud client. Such characters include (ě,č,ž,...). Folder names and file names with these characters can be created in web UI without a problem. And again are not synced with windows client. I was trying to move from owncloud to next cloud but if this issue is not resolved I would have to consider going back :( Using containerized version nextcloud:27-apache, postgres:15-alpine, redis:6-alpine

Update (upgraded from 25 over 26 to 27 and problem persists)

Aug 07 '23 15:08 jandusek4

Filenames with special characters (German "Umlaute") not found by "files:scan"

Steps to reproduce

Expected behaviour

Actual behaviour

Server configuration

LDAP configuration (delete this part if not used)

Client configuration

Logs

Web server error log

Nextcloud log (data/nextcloud.log)

Browser log

`-- phpMyAdmin SQL Dump -- version 5.0.4deb2 -- https://www.phpmyadmin.net/

-- -- Datenbank: nextcloud

-- -- Tabellenstruktur fÃ¼r Tabelle oc_filecache

-- -- Daten fÃ¼r Tabelle oc_filecache

-- -- Indizes der exportierten Tabellen

-- -- Indizes fÃ¼r die Tabelle oc_filecache

-- -- AUTO_INCREMENT fÃ¼r exportierte Tabellen

-- -- AUTO_INCREMENT fÃ¼r Tabelle oc_filecache

-- -- Datenbank: `nextcloud`

-- -- Tabellenstruktur fÃ¼r Tabelle `oc_filecache`

-- -- Daten fÃ¼r Tabelle `oc_filecache`

-- -- Indizes fÃ¼r die Tabelle `oc_filecache`

-- -- AUTO_INCREMENT fÃ¼r Tabelle `oc_filecache`