Failed to get movies list

[nasserbinlaboun]

Hello I am facing an issue when I try to view old videos

I am running Motioneye in a Docker container

Version: edge sha256:770d9868836f21ce75e095ec47a7ee90b5172c6ea3f37eee205890ce5a8f597a

   ERROR: timeout waiting for the media listing process to finish

    INFO: An interrupt signal received, closing …

   ERROR: timeout waiting for the media listing process to finish

    INFO: An interrupt signal received, closing …

   ERROR: timeout waiting for the media listing process to finish

    INFO: An interrupt signal received, closing …

   ERROR: timeout waiting for the media listing process to finish

    INFO: An interrupt signal received, closing …

[nasserbinlaboun]

#3141 I think this issue is related to this one, I tried this version ghcr.io/motioneye-project/motioneye@sha256:3a38c03592203b6039873be026238a625da3f7bab099b93faf3024542dfae14c and it is working as it should be.

[Semmu]

I have been facing the same issue and my solution was to delete or move old recordings to a different place, out of motionEye's reach.

The reason is that the current API endpoint to get the list of past recordings is not really optimized IIRC and simply runs into a HTTP timeout issue, which is 30 secs by default. (When I was debugging this issue I dug into the source code, but it was a long time ago, so I don't know where the corresponding code exactly is).

I observed what my server was doing during these requests via atop and it showed 100% disk I/O for the duration of the request.

Moving old recordings and only keeping the last couple of months solved the issue for me. You could also use the built-in movie lifetime limit feature to automatically delete old recordings.

[MichaIng]

#3141 seems to occur with v0.42.y only, while 0.43.y solves it. This issue however happens with v0.43.y, i.e. the images from here as well, right?

Might not even be Docker-specific. Here is the code which loops through the media directory: https://github.com/motioneye-project/motioneye/blob/dev/motioneye/mediafiles.py#L108-L123

Maybe it has some issues with too long lists. You say there is the 30 seconds timeout. So if you move away only a certain number of move files, it can work, but take like 20 seconds or more? Just to be sure the issue is not some sort of limit or step after which it becomes slow, but that instead the loop itself is too slow.

Aside of looping recursively to every file in the directory list, it stats them each to know whether its a dir and in case recursively loops through that as well.

Also do you use prefixes/groups? Actually this is the default with the filename template and these %x time/date conversion identifiers. In that case there is another loop: https://github.com/motioneye-project/motioneye/blob/dev/motioneye/mediafiles.py#L139-L157 However, it does essentially the same, just does not loop into directories recursively.

Maybe there is a more efficient way to differentiate between files and dirs instead of os.listdir + os.stat + S_ISREG.

This article seems to cover it, and has a more efficient way, using an iterator instead of a list that is fully loaded into memory. And is_file() instead of stat() + S_ISREG: https://www.pythontutorials.net/blog/best-way-to-get-files-list-of-big-directory-on-python/#python-tools-for-iterative-directory-listing

Hah, awesome task to test Copilot with. Though not even sure whether my extended free plan covers Copilot creating PRs. At least I am offered to ping Copilot in an existing PR to make changes. Let's see ...

Just to make that clear, since there is so much controversial around AI for coding: I will never merge any PR that I do not 100% fully understand. I will never accept any cryptic or nonsense PR, no matter whether from AI or human, but every line needs to be fully understood, match coding standards, and of course be functional and solve the issue it intends to fix. For this particular example, I know pretty exactly how I expect the loop to be adjusted, using the method from the linked article. So this is for testing whether Copilot can identify my intention from this comment, follow and extract the information/method I linked above, and create a PR which implements this exact method without any overhead/additional changes. Syntactically it should be able to code Python extremely well, and we have CI checks in place for that anyway, so it is mostly about understanding my/our intention precisely.

EDIT: So far so good. Let's see what it is doing. As "prompt" it quoted OP comment only, hence not sure whether it considers this my comment here at all. If not, will be interesting whether it is still able to find the related part in the code from OP context, and even a better performing file loop 👀.

EDIT2: I am impressed. But it requires some tuning.

[MichaIng]

Okay, the PR is ready, but while the change is generally reasonable, I am not sure whether it performs that much better, given that a stat() call is still needed for actual files, just the one from the iterator object os.DirEntry.stat() instead of os.stat().

@nasserbinlaboun @Semmu since you use Docker, it is a bit more difficult to apply, since container images are build only after merging the PR, while I do not want to merge it until it shows significant performance benefits. Do you know how to access the container image via

docker exec -it <container_id/name> /bin/bash

and replace the /usr/local/lib/python3*/dist-packages/motioneye/mediafiles.py there with the one from the PR? https://github.com/motioneye-project/motioneye/pull/3215/files

The other two files are irrelevant. Once done, restart either the meyectl process in the container, or the whole container.

[nasserbinlaboun]

Thank you for your hard work, i tried the new file mediafiles.py from this PR https://github.com/motioneye-project/motioneye/pull/3215/files and it is not working showing me same error

   ERROR: timeout waiting for the media listing process to finish

    INFO: An interrupt signal received, closing …

Session terminated, killing shell...    INFO: An interrupt signal received, closing …

    INFO: server stopped

    INFO: tasks stopped

 ...killed.

[zagrim]

@MichaIng slow media file list generation has been a long standing nag when there are many files (it's even mentioned in MEOS wiki: https://github.com/motioneye-project/motioneyeos/wiki/FAQ#what-should-i-do-to-have-pictures-attached-to-motion-notification-emails), so this issue should also be possible to replicate on any slow platform when there are a ton of stored media files (perhaps more easily if the storage media is also slow, like an SD card).

I have an SSD attached to my ME RPi setup and I don't recall having run into this, but I'm mentioning this here in case it helps finding a way to test the changes. Also, my RPi setup is the "production" environment so I'd rather not to start messing with it. But with a vacant RPi (an old model, like RPi 2, perhaps an RPi Zero would also work), large enough SD card and a suitably large set of media files to simulate stored recordings, maybe the issue could be replicated without too much effort 🤷

[Semmu]

I think the issue comes from an architectural problem.

When loading past recordings, motionEye will load ALL past recordings with detailed information about them, then when selecting a specific day, it will load a subset of that data again. (which, in this case, is actually justified to display details about that recording sizes and whatnot.)

I have recently set up a second camera and it has currently 12710 recorded files, totaling around 52GBs of storage. Loading the media files for this camera takes almost 12 seconds and the response is 2.4MBs.

You can see that the response payload starts with details about recordings from 2025-11-04 (from when the camera was installed), but that data is not used nor needed on the current screen. All that network traffic and backend computation is basically wasted for no reason, since the data is not used on the frontend in any way.

Then after selecting a specific day, it will load redundant data again (now limited to that day), but this response is much quicker and loads a reasonable amount information.

So all in all I think the solution is to return less data on the list/ endpoint without a prefix parameter. We only need to know the days which have recordings, and the number of the recordings in each day - but no need for additional data, like the full list of them, file sizes, etc. All that is loaded anyways on the same endpoint when it has a prefix parameter.

[Semmu]

Also more optimization ideas could be to aggressively cache backend computation results about things that won't change anymore. This includes recording count for past days, video file sizes, etc. This would significantly reduce the I/O operations when requesting the same data from the frontend now and then.

[zagrim]

Good point @Semmu on the excess data for the first request (with no date filtering). But there still needs to be all the recursive iteration over directory contents and getting the timestamps, so plenty of file system I/O will still remain. But plenty of that is (or at least is supposed to be) optimised in the PR linked to this issue. Or well, it doesn't do anything about the response data from the unfiltered request but unless getting the MIME type is expensive (and looking at mimetypes.guess_type docs, it shouldn't be), there might not be much savings to be gained, other than from data serialisation and transfer.

Until the performance boost from the PR has been observed, I'd refrain from pondering about any aggressive caching strategies. The list of stored files DOES change: new files are being saved, users can move/delete files themselves, ME also has the clean-up feature which deletes older files etc. So cache invalidation and refresh would also need to be implemented (and even executed quite frequently) if file list caching was implemented.

[MichaIng]

First of all, the code of the PR works, but it does not seem to enhance performance significantly. I recorded one true video, added 1000 empty *.mkv files and 1,000,000 empty *.txt files.

Without the change:

Nov 22 15:41:26 VM-Trixie meyectl[6062]:  WARNING: Starting mediafiles loop without prefix: 0 files
Nov 22 15:41:26 VM-Trixie meyectl[6062]:  WARNING: Starting mediafiles loop without prefix: 0 files
Nov 22 15:41:58 VM-Trixie meyectl[6062]:  WARNING: Fiinished mediafiles loop without prefix: 1001002 files
Nov 22 15:41:58 VM-Trixie meyectl[6062]:  WARNING: Fiinished mediafiles loop without prefix: 1001002 files
Nov 22 15:42:01 VM-Trixie meyectl[6063]:  WARNING: Starting mediafiles loop with prefix "2025-11-22": 0 files
Nov 22 15:42:30 VM-Trixie meyectl[6063]:  WARNING: Finished mediafiles loop with prefix "2025-11-22": 1001 files

• It loops through the overall 1,001,002 files (includes the 2025-11-22 directory itself)
• The non-prefix function calls itself for the subdir, hence the doubled logs: starts loop => starts loop through 2025-11-22 dir => finishes sub dir loop with 1,001,002 list entries => finishes parent loop with does not add any more entries
• The one and only 2025-11-22 directory is selected OOTB in the UI, hence is starts another prefix loop for that one, finishing with the 1001 *.mkv files in the list. It calls stat() as well on all files, but compared to non-prefix loop returns only those with a known video file extension.

Both loops take around 30 seconds, which does not cause a timeout in my case. Makes sense that it is very similar, since the overall number of files and stat() calls is the same.

Hmm, on a subsequent call it became much faster 🤔:

Nov 22 16:00:48 VM-Trixie meyectl[6069]:  WARNING: Starting mediafiles loop without prefix: 0 files
Nov 22 16:00:48 VM-Trixie meyectl[6069]:  WARNING: Starting mediafiles loop without prefix: 0 files
Nov 22 16:01:04 VM-Trixie meyectl[6069]:  WARNING: Fiinished mediafiles loop without prefix: 1001002 files
Nov 22 16:01:04 VM-Trixie meyectl[6069]:  WARNING: Fiinished mediafiles loop without prefix: 1001002 files
Nov 22 16:01:07 VM-Trixie meyectl[6070]:  WARNING: Starting mediafiles loop with prefix "2025-11-22": 0 files
Nov 22 16:01:19 VM-Trixie meyectl[6070]:  WARNING: Finished mediafiles loop with prefix "2025-11-22": 1001 files

Makes it difficult to compare. But now with the change, two subsequent calls:

Nov 22 16:05:06 VM-Trixie meyectl[6118]:  WARNING: Starting mediafiles loop without prefix
Nov 22 16:05:06 VM-Trixie meyectl[6118]:  WARNING: Starting mediafiles loop without prefix
Nov 22 16:05:21 VM-Trixie meyectl[6118]:  WARNING: Finished mediafiles loop without prefix: 1001002 files
Nov 22 16:05:21 VM-Trixie meyectl[6118]:  WARNING: Finished mediafiles loop without prefix: 1001002 files
Nov 22 16:05:24 VM-Trixie meyectl[6119]:  WARNING: Starting mediafiles loop with prefix "2025-11-22"
Nov 22 16:05:35 VM-Trixie meyectl[6119]:  WARNING: Finished mediafiles loop with prefix "2025-11-22": 1001 files
Nov 22 16:05:47 VM-Trixie meyectl[6121]:  WARNING: Starting mediafiles loop without prefix
Nov 22 16:05:47 VM-Trixie meyectl[6121]:  WARNING: Starting mediafiles loop without prefix
Nov 22 16:06:02 VM-Trixie meyectl[6121]:  WARNING: Finished mediafiles loop without prefix: 1001002 files
Nov 22 16:06:02 VM-Trixie meyectl[6121]:  WARNING: Finished mediafiles loop without prefix: 1001002 files
Nov 22 16:06:05 VM-Trixie meyectl[6122]:  WARNING: Starting mediafiles loop with prefix "2025-11-22"
Nov 22 16:06:17 VM-Trixie meyectl[6122]:  WARNING: Finished mediafiles loop with prefix "2025-11-22": 1001 files

Means at least in my case in a VM with fast NVMe SSD, after files landed in Linux memory cache as well, both methods perform pretty similar.

For completeness, after a reboot with clean memory cache, new method:

Nov 22 16:09:33 VM-Trixie meyectl[845]:  WARNING: Starting mediafiles loop without prefix
Nov 22 16:09:33 VM-Trixie meyectl[845]:  WARNING: Starting mediafiles loop without prefix
Nov 22 16:10:12 VM-Trixie meyectl[845]:  WARNING: Finished mediafiles loop without prefix: 1001002 files
Nov 22 16:10:12 VM-Trixie meyectl[845]:  WARNING: Finished mediafiles loop without prefix: 1001002 files
Nov 22 16:10:14 VM-Trixie meyectl[846]:  WARNING: Starting mediafiles loop with prefix "2025-11-22"
Nov 22 16:10:23 VM-Trixie meyectl[846]:  WARNING: Finished mediafiles loop with prefix "2025-11-22": 1001 files

Old method after reboot:

Nov 22 16:14:23 VM-Trixie meyectl[529]:  WARNING: Starting mediafiles loop without prefix: 0 files
Nov 22 16:14:23 VM-Trixie meyectl[529]:  WARNING: Starting mediafiles loop without prefix: 0 files
Nov 22 16:15:01 VM-Trixie meyectl[529]:  WARNING: Fiinished mediafiles loop without prefix: 1001002 files
Nov 22 16:15:01 VM-Trixie meyectl[529]:  WARNING: Fiinished mediafiles loop without prefix: 1001002 files
Nov 22 16:15:03 VM-Trixie meyectl[530]:  WARNING: Starting mediafiles loop with prefix "2025-11-22": 0 files
Nov 22 16:15:13 VM-Trixie meyectl[530]:  WARNING: Finished mediafiles loop with prefix "2025-11-22": 1001 files

Really not much difference. Also memory usage in both cases is the same: it goes up to ~1 GiB while scanning the files. I guess it is caused by the list, not so much by the iteration.

There is actually some room for enhancement. Why do we stat() files which are not added to the list anyway? With the old method, that was needed to even find out whether it is a directory or a file, but with the new method, this is not needed anymore.

Uhhh, see this massive improvement, also memory usage does not go up anymore:

Nov 22 17:05:59 VM-Trixie meyectl[1310]:  WARNING: Starting mediafiles loop without prefix
Nov 22 17:05:59 VM-Trixie meyectl[1310]:  WARNING: Starting mediafiles loop without prefix
Nov 22 17:06:01 VM-Trixie meyectl[1310]:  WARNING: Finished mediafiles loop without prefix: 1001 files
Nov 22 17:06:01 VM-Trixie meyectl[1310]:  WARNING: Finished mediafiles loop without prefix: 1001 files
Nov 22 17:06:02 VM-Trixie meyectl[1311]:  WARNING: Starting mediafiles loop with prefix "2025-11-22"
Nov 22 17:06:04 VM-Trixie meyectl[1311]:  WARNING: Finished mediafiles loop with prefix "2025-11-22": 1001 files

However, this is only such an enhancement since 99.9% of files in my case have a *.txt extension. If it is all video files with respective extension, they would all be stat()ed.

I have those changes locally, let's see whether Copilot can figure out the same.

[MichaIng]

Hehe, Copilot can code Python and identify hints correctly. Not bad. For comparison what I had and tested locally in the end, bevor giving it instructions:

def findfiles(path: str, exts: typing.List[str]) -> typing.List[tuple]:
    files = []

    logging.warning(f'Starting mediafiles loop without prefix')

    for entry in os.scandir(path):
        # ignore hidden files/dirs and other unwanted files
        if entry.name.startswith('.') or entry.name == 'lastsnap.jpg':
            continue

        if entry.is_dir(follow_symlinks=False):
            files.extend(findfiles(entry.path, exts))
            continue

        if not entry.is_file(follow_symlinks=False):  # not a regular file
            continue

        if not [e for e in exts if entry.path.lower().endswith(e)]:
            continue

        try:
            st = entry.stat(follow_symlinks=False)

        except Exception as e:
            logging.error(f'stat failed: {e}')
            continue

        files.append((entry.path, st))

    logging.warning(f'Finished mediafiles loop without prefix: {len(files)} files')

    return files


def _list_media_files(
    directory: str, exts: typing.List[str], prefix: str = None
) -> typing.List[tuple]:

    if prefix is not None:
        if prefix == 'ungrouped':
            prefix = ''

        root = os.path.join(directory, prefix)
        if not os.path.exists(root):
            return []

        logging.warning(f'Starting mediafiles loop with prefix "{prefix}"')

        media_files = []
        for entry in os.scandir(root):
            # ignore hidden files/dirs and other unwanted files
            if entry.name.startswith('.') or entry.name == 'lastsnap.jpg':
                continue

            if not entry.is_file(follow_symlinks=False):  # not a regular file
                continue

            if not [e for e in exts if entry.path.lower().endswith(e)]:
                continue

            try:
                st = entry.stat(follow_symlinks=False)

            except Exception as e:
                logging.error(f'stat failed: {e}')
                continue

            media_files.append((entry.path, st))

        logging.warning(f'Finished mediafiles loop with prefix "{prefix}": {len(media_files)} files')

        return media_files

    else:
        return findfiles(directory, exts)

[MichaIng]

Does anyone know a reason why we would not want to follow symlinks? It also was a bit inconsistent before:

• With given prefix/subdir, symlinks were ignored, also symlinks to dirs not recursively iterated.
• Without prefix given, symlinks to files were treated as media files.
• In the current state of the PR, symlinks are ignored in both functions/cases.

Maybe for security reasons, to prevent file creation/deletion elsewhere on the system if some attacker manages to create a symlink in the media dir. UNIX permissions should prevent it, but e.g. it has access to its own config dir, and that way theoretically could make configs directly accessible or alter them from the media files UI. Yeah, until someone asks for symlink support, seems to be best to ignore them for now. If we implement it, we need to check all function calls and possible implications carefully, in case adding some sanity checks. But while it adds some theoretical flexibility to split media across different drives and e.g. network shares, I see not much practical use case: one can replace the whole camera media dir with a symlink already, if wanted.

That way, #3215 is ready my end. There are some more theoretical enhancements possible, like:

• skipping is_dir() check if is_file() is true. Currently it is the other way around, but there will be a lot more files than dirs, hence exchanging that will reduce overall calls, but also causes a more nested code strucutre.
• merging the os.scandir() loops code-wise: call findfiles() with a new bool recurse argument as condition whether to recurse into dirs or not, and call it in any case from _list_media_files(), just with recurse = True as 3rd argument if no prefix is given.

But leaving it as is for now, I don't think it will add significant benefit, and we may instead want to skip the stat() call for recursive scanning entirely if the detailed info is never actually needed in that case. But better as separate PR, to keep it reviewable.

@Semmu @nasserbinlaboun can you test again the latest version of the script, to see whether this reduces the scan duration and in case solves the error?

https://raw.githubusercontent.com/motioneye-project/motioneye/refs/heads/copilot/fix-failed-to-get-movies-list/motioneye/mediafiles.py

[nasserbinlaboun]

same issue with this one https://raw.githubusercontent.com/motioneye-project/motioneye/refs/heads/copilot/fix-failed-to-get-movies-list/motioneye/mediafiles.py

I tested it on an HDD containing 329,557 videos and images, which total 1.2 TB.

[MichaIng]

You could btw raise list_media_timeout in /etc/motioneye/motioneye.conf and check how long it actually takes, so we get an idea how much is missing, and how much the change at least reduced the duration.

With movies and pictures mixed, e.g. half/half, it should cut half as well the duration, roughly. But obviously that is not enough yet in your case.

---

I intend to merge it without further major changes, but think about more changes in separate PRs. Smaller chunks are easier to review and understand.

1. The two involved functions can be pretty easily merged. Now that the code has been restructured and the iterative function does the extension checks and skips stat() for non-movie resp. non-picture files, returning the same tuple as the parent function, they do almost exactly the same. Using the existence of the prefix argument as condition for iterating through subdirs, the parent function can call itself to achieve the very same thing. The prefix check and handling at the top can be slightly restructured to minimize executed code on iterative calls, and the dir check could be done after the file check, to do one check less for files. Marginal performance benefit, but it should not be worse most importantly, while cleaning up code redundancy and the need to understand why there are two functions in the first place.

2. As my tests indicate, the stat() call is indeed the major time consumer, where skipping it for (almost all) files reduced the duration from ~15 seconds to ~2 seconds. Means, if we either add a new argument to indicate whether those are needed or not, or split functions to have one with stat() object and one without, we should be able to enhance performance a lot more. What makes it difficult is that indeed recursive and non-recursive (prefix) calls can both sometimes require the stats:

   • When opening movies or pictures from the UI, there is a full recursive file list requests which does not require the stats. This is the critical one which times out in your case. The full list is used to create the list of all subdirectories with number of respective media files inside. Directory entries are generated for every in case nested dir separately, so e.g. there would be separate entries for Camera1/2025/11/23, Camera1/2025/11, and Camera1/2025, when nesting media files that way via % tokens in the video/picture filename entry. The fixed base is the respective camera dir. This is also the reason why, if any subdir=prefix is given, it must not iterate through subdirs, otherwise the same media files would appear in multiple subdirs in the list. Since it is also possible to not use subdirs, by avoiding any / in the filename setting, the dummy ungrouped is then passed, which appears as well as entry in the GUI directory list, which is set to empty string in the list function (to not cause a non-existing dir name), but has still the effect to not iterate through subdirs. However, so after first full file list call, used to generate the directory list only, a second call is done for the selected directory. This time the stats are actually used, to show time size, timestamps etc. Another call is done for every next directory when selected in the UI. The processed information are stored, hence these per-dir calls are done once only for each dir, until the whole media window is closed. There would be 2 ways to enhance the situation:
     1. Since the first full recursive file list contains that stats already, information for all subdirs could be processed already, hence no further network call for file lists is needed when browsing through directories. Easy to implement, affecting only the respective sections with the /list/ API calls in the frontend main.js. However, it would not have any impact in this timeout issue, since the timeout is done for each individual API call in the backend.
     2. Implement an argument or dedicated API method to get a simple file list without stats. Use this for the first API call to generate the directory list, then obtain the stats only for the individual subdir calls. In the backend, we could split the list function into one with and one without stats, which is why I would want to first merge the iterative loop function, to not have 4 functions in the end. An alternative would be to use a single function, but set the 2nd tuple entry None if stats are not needed. Not so beautiful, but we should not allow the function to have different return types.
   • When automatic cleanup of old files (max file age) is enabled, the full iterative list is requested and stats required for the timestamps. Not really something we can enhance there, aside of raising the timeout (e.g. with a dedicated setting), since this happens in the background (and backend) only.
   • Email and Telegram notifications can as well request a full recursive picture file list, and use timestamps to attach pictures with timestamps around the motion event. They do have their own much shorter default 10 seconds timeouts already, so this is the most problematic case. However, if the default %Y-%m-%d/ subdir prefix is used for picture filenames, it requests a list for this subdir only. It is strange that it is so hardcoded. Would make more sense to obtain any subdir name from the picture filename setting. But the conversion specifiers (% tokens) would need to be replaced, of course. Currently it uses strftime() which does not support all conversion specifiers. Probably best we could do is run strftime() on the obtained subdir name, check whether the dir exist, in case use it, else do a full recursive scan.
   • All other file list calls are for particular subdirs (also called "groups" or "prefix"), and often do not require stats either. So they could benefit from a no-stats argument/function, but should not be affected by timeout issues anyway.

[zagrim]

Uh, I notice I had thought that stats were needed also for the full recursive file list to group the files by date, but if the file listing actually uses the subdir names (probably a good idea, allows e.g. splitting the subdirs on hourly level on a busy setup by using %Y-%m-%d-%H as the subdir name), then surely that call could be made more efficient. I should've read the code more carefully 😞
Since timeouts is the issue, making the currently slowest call (the recursive listing) faster would at least allow generating the subdir list more reliably.

[nasserbinlaboun]

I raised list_media_timeout and it took around 3 minute to load the list.

[MichaIng]

Uff crazy. Would be interesting to compare to the old script/function, but with this amount of data the duration can vary a lot, also based on Linux page cache.

#3219 will do some code simplification so the core function is cleaner and easier to understand, but without significant performance impact. Afterwards I'll start working on a way to make the stat() calls optional. Need to think about whether to create a dedicated function for this, have null/None values in the returned tuples, or whether there is a transparent/clean/common way in Python to have one function returning different types based on input arguments.