configure .cache via $XDG_CACHE_DIR

[bukzor]

Forked from #1029

---

a) A very convenient, and standard, way to configure tools' cache location is to set $XDG_CACHE_DIR. Many of my project's fixtures do this for other reasons already. Similarly, the most convenient, and standard, way to set "basetmp" would be $TMPDIR.

b) "something a CI system can easily pick up" is exactly these environment variables. Many systems will already support this, and those that don't will support injecting environment variables.

c) "non-containered ci" will either support ~/.cache correctly, or set $XDG_CACHE_DIR, because this is necessary for other tools that use the standard. Again, this is already handled in many of my projects because of other tools.

[RonnyPfannschmidt]

is there a good package that helps finding those paths and helps clearing unused caches

after all if the cache folder is outside of the working directory, its lifetime is unrelated to the working directory

imagine for example a unaware ci system that makes one folder per build could easily DOS the ci server due to filling up ~/.cache of the ci user

with the current mode of operation, cache and working directory are simply co-located and share a lifetime, so data management is easy, the cache is gone when one deletes the working directory

im not per se opposed to following XDG, but it makes lifetimes of file-system objects more tricky on our case and seems not really available on windows either (its my personal impression that xdg works only well on posix boxes with a typical linux/bsd setup)

[bukzor]

Those are good points...

I don't have a good answer. The applications I've seen don't worry about cleaning up like that. It would work for me if we use the current behavior when XDG_CACHE_DIR doesn't exist.

@asottile

[bukzor]

This one at least helps find a good platform-specific cache dir: https://pypi.python.org/pypi/appdirs/1.4.0

[bukzor]

This one helps manage expiry for a file-based cache. Making one of the keys be the working directory would make it work.

http://pythonhosted.org/pyfscache/

[RonnyPfannschmidt]

i shorty investigated appdirs and pyfscache, unfortunately both are unsuitable

appdirs seems bug-ridden pyfscache is for something entirely different not fitting the use-case

[RonnyPfannschmidt]

how about introducing a ini value/env variable to select the cache backend, which defaults to worktree, but one can choose xdg/disabled?

@hpk42 oppinions?

[asottile]

Personally I'd rather default to xdg instead of needing to add .cache to the gitignore in every one of my projects

[RonnyPfannschmidt]

while i agree that it is a nice default, we cant do a change like that befor 3.0

also before we can do it propperly we need to come up with a solid way to clean things up

[asottile]

Maybe introducing the cache plugin defaulted to on should have been in 3.0 too :/

[RonnyPfannschmidt]

since its a normal feature addition, it doesnt warrant a release that large

[s-trooper]

i hate ".cache" folder for aesthetic reason. my projects folders are already populated by ide, git, python and others "support folders". sometimes i have more folders than actually python program files! please make ".cache" location configurable!

[nicoddemus]

how about introducing a ini value/env variable to select the cache backend

Anyone is against this? While it may not be the best option, it is simple to implement and backward compatible.

It should support variable expansion and default to .cache for backward compatibility. This would allow users to change it like this:

[pytest]
cache_dir=$XDG_CACHE_DIR

Or use an environment variable:

PYTEST_CACHE_DIR=/some/path

[RonnyPfannschmidt]

I'm against the most easy variant, I want a simple solution

[bukzor]

@RonnyPfannschmidt I've found that the function you'd need exists in pip.utils:

https://github.com/pypa/pip/blob/develop/pip/utils/appdirs.py#L13

[nicoddemus]

How about we add a configuration option that decides between the two systems? cache_location, which can be local (<rootdir>/.cache, the default) or system (using the heuristics suggested by @bukzor)? This could be added in 2.9.

If we make a poll and people wants to change system as the default, we can announce that the default will change in 2.10, giving plenty of time for people to change to their desired setting while in 2.9 and not get caught off guard.

BTW, I'm just throwing some ideas so we can reach some consensus, I don't mind having the .cache directory in pytest's rootdir: I configured my git-global-ignore file to always ignore it.

[RonnyPfannschmidt]

:+1:

[bukzor]

For the implementation of system, I'd probably just import and use pip.utils. Would you all be opposed to a dependency on pip from pytest? I imagine ~100% of python installations have pip installed already.

On Wed, Dec 23, 2015 at 10:29 PM Ronny Pfannschmidt < [email protected]> wrote:

[image: :+1:]

— Reply to this email directly or view it on GitHub https://github.com/pytest-dev/pytest/issues/1089#issuecomment-167052256.

[bukzor]

Actually, the pip function seems to be closely related to the appdirs package implementation. I'm not sure which is derived from which though.

https://github.com/ActiveState/appdirs/blob/master/appdirs.py#L257

[nicoddemus]

IMHO I would just copy the function over, licensing permitting... depend on the entire pip library just for the sake of a small function like that is not worth it, I think, as I'm not sure how stable is that API.

[RonnyPfannschmidt]

I'd go for publishing a minimalistic lib that implements it

[antoche]

Hi, Is the recommended solution to add .cache to every project's .gitignore? Or is there a consensus on a solution for this issue? Thanks, A.

[RonnyPfannschmidt]

@antoche for now yes

[nicoddemus]

There's a cache_dir ini option which is now available in 3.2.

Should we change the default of that option to $XDG_CONFIG_DIR (if defined) in future pytest versions?

[RonnyPfannschmidt]

the config dir is under all circumstances always the wrong folder for the cache

we should use the cache home as according to the spec,

and we should take a look at using the appdirs lib to facilitate those details

[bukzor]

@nicoddemus: No, the default should be $XDG_CACHE_DIR, which in turn has a default of $HOME/.cache.

$XDG_CACHE_HOME defines the base directory relative to which user specific non-essential data files should be stored. If $XDG_CACHE_HOME is either not set or empty, a default equal to $HOME/.cache should be used.

If you want to support macos and windows in a natural way, you'll want a library for this. I think this is the right solution: https://github.com/ActiveState/appdirs#the-problem

[RonnyPfannschmidt]

@bukzor thanks for the clarification and correcting my mistake

[nicoddemus]

Thanks @bukzor.

Hmm just realized that with this change the cache will no longer be per-repository, but will be global per-user. This might be a problem because plugins assume a per-repository cache, for example pytest's --lf option.

[RonnyPfannschmidt]

@nicoddemus borgbackup for example uses the repository path as extra path component into the per user caches in order to keep things appart

[bukzor]

I've used that strategy myself with success, but I only care about Unixen. You'll want a proof of concept on Windows, but I don't see why it wouldn't work.

On Sun, Aug 6, 2017, 09:09 Ronny Pfannschmidt [email protected] wrote:

@nicoddemus https://github.com/nicoddemus borgbackup for example uses the repository path as extra path component into the per user caches in order to keep things appart

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/pytest-dev/pytest/issues/1089#issuecomment-320516073, or mute the thread https://github.com/notifications/unsubscribe-auth/AAnFSIZ0E-r_YIMrnHaUZRiQBLR5Bl4Oks5sVeUagaJpZM4GF3LC .

[blueyed]

What should it default then to? $XDG_CACHE_DIR/pytest/$pytest_rootdir ?