Feature: download to custom path, but also copy to local temp cache folder

[tonywagner]

I love being able to set a custom download path to my SMB network share, so I can share files between multiple instances of Kodi and have a lot more storage, but this causes two problems:

1. Emulators like MAME, which don't support VFS, can't load games directly from the network share: https://forum.kodi.tv/showthread.php?tid=340239

2. RetroArch can't load games over SMB either: https://forums.libretro.com/t/accessing-a-shared-smb-folder/18462

But since IAGL already has a temp cache folder on local storage, it would be fairly straightforward to have it copy the file from the SMB share to local storage before processing it.

As a proof of concept, I defined a new copy function in resources/lib/main.py as follows:

def copy_file_to_directory(file_in,directory_to):
	overall_success = True
	file_out = None
	new_filepath = os.path.join(directory_to,os.path.split(file_in)[-1])
	success = xbmcvfs.copy(file_in,new_filepath)
	if not success:
		xbmc.log(msg='IAGL:  The file could not be copied: %(file_from)s, to: %(file_to)s' % {'file_from': file_in, 'file_to': new_filepath}, level=xbmc.LOGDEBUG)
	else:
		xbmc.log(msg='IAGL:  File copied from: %(file_from)s, to: %(file_to)s' % {'file_from': file_in, 'file_to': new_filepath}, level=xbmc.LOGDEBUG)
		file_out = new_filepath

	return file_out

Then in that same file, I added a block of code between the check_file_validity() call and the post_process calls, which calls our copy function if the downloaded file path is different than the default temp folder:

		# Copy file to IAGL temp directory, if different, and update location
		if os.path.split(self.current_saved_files[ii])[0] != os.path.join(self.IAGL.get_temp_cache_path(),''):
			xbmc.log(msg='IAGL:  Copying file %(original_file)s to temp folder %(temp_folder)s' % {'original_file': self.current_saved_files[ii], 'temp_folder': self.IAGL.get_temp_cache_path()}, level=xbmc.LOGDEBUG)
			self.current_saved_files[ii] = copy_file_to_directory(self.current_saved_files[ii],self.IAGL.get_temp_cache_path())
			xbmc.log(msg='IAGL:  Updated file location to %(temp_file)s' % {'temp_file': self.current_saved_files[ii]}, level=xbmc.LOGDEBUG)

This works, although we probably want to make it configurable as a global / per-emu setting. If you have any tips on that, let me know and I can submit this as a pull request, if you want.

[zach-morris]

I think i understand the request. I think the ultimate goal would be to support 'sources' other than http, so instead of a game residing at: http://archive.org/mario.nes You'd want it residing at smb://path_to_my_games/mario.nes

IAGL would recognize the source URL as non-http and then 'copy the file to the temp directory and launch' instead of 'download to the temp directory and launch'. Is that the goal?

I've recently updated all the file functions to be xbmcvfs compatible (in the latest master git), so this might potentially be doable without too much issue.

[tonywagner]

That might be interesting too, but the way I am using it is in addition to the http download.

Select a game in IAGL, and it downloads from archive.org to the specified download path (in this case, a SMB share) if it doesn't already exist there, just like normal.

But then instead of processing/launching the file direct from the SMB share, it first copies it to the temp_iagl folder on the local file system. So MAME, RetroArch, etc. can launch it.

[tonywagner]

So in other words, you get to download to permanent shared storage, but also leverage the existing temp_iagl folder to satisfy the local filesystem requirements of MAME and RetroArch.

[pejobo]

I did it the other way round: replaced the storage folder with a (soft) link to my download folder (which can be any on mounted drive/location). You have to protect the folder so that IAGL cannot delete it.

[tonywagner]

I did it the other way round: replaced the storage folder with a (soft) link to my download folder (which can be any on mounted drive/location).

If you can mount a network share to a local folder in your operating system, then you shouldn't really need to use the temp_iagl folder at all, unless you simply want to avoid adding a custom download path for each system/list.

But I'm using Fire TV clients, which don't allow mounting network shares in the local filesystem (at least not without root / custom firmware?). Sorry, I probably should have mentioned that!

So Kodi/IAGL being able to download to the SMB share for long-term storage, while also copying the current game to the temp_iagl folder to load it in MAME/RetroArch, is really useful on Fire TV.

[pejobo]

Ok, now I understand. But then I would recommend another solution (in the addon): Options to set a "cache dir" and a "launch dir". If given, the launch dir should be used for launching (obvious) and always be cleaned. The "cache dir" should only be cleaned according to the actual cache size setting (btw is there an option for "no cleaning" yet?).

[tonywagner]

Options to set a "cache dir" and a "launch dir"

That could work, although I'd probably skip the "cache dir" because I'm already using the "Update Download Path" option. (And it might be a bad idea to allow users to manually set any directory that is going to be cleaned -- select the wrong directory, and you might lose some files!)

The way I saw it, there is already a custom "Update Download Path" option on a per-system basis. And there's already a local cache folder (temp_iagl) whose default setting is to always be cleaned (current game only). Those are great options and could remain exactly the same.

So my idea was to add a single extra setting, a per-system toggle to always copy to/launch from the local cache folder, even when the "Download Path" is different.

[zach-morris]

I've started work on the alternate (non http) source part of this, but I'm not sure I fully understand the situation. Lets see if I get this right:

1. Source of the game is http and destination is local share (the 'normal' way)

• This already works.

2. Source is http, you want to store it to a network drive

• IAGL can save it to a network drive (I've tested an OS mounted SMB share and a 'kodi mounted' SMB share, and the difference between the two is not trivial). Saving doesn't seem to be the issue.
• Now you want to launch it from that share location. If it's an OS mounted SMB share, then depending on the core it will work. If it's a Kodi mounted SMB share (a network source), then it will work in Retroplayer depending on the core, but it wont work in RA because RA doesn't know anything about the Kodi mounted share. So the goal in this situation is to save it to the network share, then post process, copy to the local path and launch it from there. Then the next time you try and 'download' the game, it will see the file is available in the network location and just copy it from there post process.

3. Source is a local SMB / network share

• Same as above, but skip the download part. Copy it to the local file location to launch

Did I get that right? This is complicated. I can't forsee off the top of my head how I'd automatically detect what core can do what. The easiest thing I can think of is some addon option that you check that says "launch network saved files by copying them to the local machine", and it will just be a blanket setting. If the addon detects a network save location, then if this is enabled, then after download, it will post process, then post post process and copy the files to the local game_cache path for launching.

[tonywagner]

Did I get that right?

Zach, I think you've got it! A blanket setting like that sounds perfect.

[zach-morris]

It's been a while, but I'm getting back into it. I think I have this address with the last commit, just need to test it on different filetypes. There were a few bugs to sort out, but I believe IAGL can save to a 'Kodi Network Source' now reliably (tested with smb and nfs sources) There's now an advanced setting, which when enabled will look to see if the download location is a 'Kodi Network Source', it will download the game to that source, and then copy the file(s) to the temporary cache folder (local filesystem) and launch it from there instead - useful for those addons that can't load games from the network (some retroplayer addons) and for external launching (since retroarch doesn't have any access to Kodi network sources). This is a blanket setting for now.

If the game had been previously downloaded, IAGL will just copy the file(s) from the SMB source. Pretty handy and a neat way to give Kodi a hybrid online/NAS launching way to play retro games.

[zach-morris]

I believe this is mostly implemented in the latest release, may not work perfectly for some game lists that need wonky post processing to 'massage' the files before launch (that needs some work). Needs some testing in general though. See the new advanced setting:

.