Fetching dependencies from git

[danlamanna]

I have a fairly simple project that I would like to set up that requires a dependency that isn't hosted on a package manager. It seems this is becoming more common as people use straight and package-vc-install to install directly. Is it possible to declare such a dependency in eldev given that it's not on a package manager or on a local path?

[doublep]

No, it's currently not possible, but I long have thought about adding a plugin (i.e. optional extension) to Eldev that would do just this, probably using straight. If you could specify in more details what you'd need, I could have a look and maybe implement this in 1.5, the next version.

[danlamanna]

I'd like the functionality and bootstrapping that's available to eldev-use-local-dependency but with an external git repo that can be pulled from a host. Whether it uses the spec format from package-vc.el or straight.el doesn't make a big difference to me.

[doublep]

A problem here is that I have never used such functionality. I basically understand what you need and why, and I see how it would be a useful addition to Eldev too. But to design a good and usable interface I'd rather first see some real-world examples. If possible (i.e. not a private project), please provide a link to the outer project and the dependency. If not, at least show some code how you would use it, especially how you autofetch it from Git it in your normal Emacs — I assume you already do that.

[suhail-singh]

... especially how you autofetch it from Git it in your normal Emacs

FWIW, for the package doctest this is what my init.el configuration looks like in Emacs 29.1:

(use-package doctest :ensure nil
  :vc (doctest :url "https://github.com/ag91/doctest.git"
               :branch "master"
               :rev :newest))

The :ensure nil prevents use-package from trying to fetch it from the package-archives.

[doublep]

Hm, I just looked at the docstring of use-package and it doesn't mention :vc, at least in the version I have (2.4.5). Is it new? Is it built-in in use-package, or does it need something else?

[suhail-singh]

My bad! I'm using a not-yet-available-via-upstream-use-package extension called vc-use-package. Earlier in my init.el I have:

(unless (package-installed-p 'vc-use-package)
  (package-vc-install "https://github.com/slotThe/vc-use-package" :newest))

Of course, instead of first installing vc-use-package and then using that extension via the :vc keyword in use-package, you may want to directly use package-vc-install to install.

[doublep]

I see. So, basically, if Eldev was to support this, in project's file Eldev you would somehow specify where (and maybe how) to get certain dependencies. Something like:

(eldev-fetch-package-from 'abc                             ; dependency package name
                          "https://github.com/abc.el/abc"  ; the URL
                          ; possible additional options, e.g. :fetcher 'git
                          )

And then when it needs to install dependency X, it first checks if it has a URL for it. If it does, fetch (via Git or whatever) from that URL, else fall back to the usual PA lookup. (Internally this could be implemented via some existing tool or not, don't know yet.)

Is that the desired mechanism? This means you'd have to enumerate all dependencies not available in package archives (MELPA or something) together with their URL. On the other hand, there is probably no way to look up "canonical repository URL for Elisp package named X", so you do have to specify it manually...

[suhail-singh]

in project's file Eldev you would somehow specify where (and maybe how) to get certain dependencies

The simplest implementation I can imagine would be one where in the Eldev file one would note for each dependency the arguments that would be needed to install said dependency via package-vc-install. And then eldev would take care of invoking package-vc-install for each of those.

A more generalized approach would be one which parameterizes over the installation mechanism (e.g. package-vc-install vs straight.el etc). Having never used straight.el, I'm not sure what advantage having the option to use it would confer over being restricted to package-vc-install.

... you'd have to enumerate all dependencies not available in package archives (MELPA or something) together with their URL.

Yes, I believe that that's unavoidable (and that's okay).

[suhail-singh]

@doublep I encountered this issue today when trying to manage my fork of git-email via eldev (since one of the "soft" dependencies isn't on any archive).

If there were some way to specify git URLs for some of the dependencies (perhaps via a specification syntax as the one used by the :vc keyword in use-package in the upcoming Emacs 30 release), it would help matters.

[doublep]

Let's suppose your project-x declares that dependency libfoo must be fetched from certain Git repository. If libfoo itself depends on libbar, how should Eldev find that package?

Probably an option would be that libbar is to be looked up in archives that project-x itself declares. Or do you have a better idea?

[suhail-singh]

Let's suppose your project-x declares that dependency libfoo must be fetched from certain Git repository. If libfoo itself depends on libbar, how should Eldev find that package?

If libfoo doesn't use Eldev, then looking for ~~libfoo~~ libbar in the archives and the git URLs that project-x declares is the only sensible thing in my opinion.

If libfoo uses Eldev itself, perhaps it may help to use that dependency information. However, even in that case it would help for project-x configuration to have the ability to override things (if so desired).

Probably an option would be that libbar is to be looked up in archives that project-x itself declares.

I believe if that's the only thing that is implemented, that it would be adequate.

[d12frosted]

If libfoo itself depends on libbar, how should Eldev find that package?

I think Eldev should not treat libbar in any special way. It's just a transitive dependency. Try to fetch it based on Eldev file of the project-x and if this transient dependency can not be satisfied - let the user explicitly either add a proper archive or declare from which repository to fetch.

If libfoo uses Eldev itself, perhaps it may help to use that dependency information.

If it's just to get dependency information, I am against this idea as it would complicate things both implementation and usage-wise. Since Eldev is a build tool, the only meaningful solution here is to build differently depending on existence of Eldev file - but do it in an 'isolated' way, i.e. without polluting and messing around with project-x configurations.

I believe if that's the only thing that is implemented, that it would be

I agree, just an ability to pull dependencies from Git would be a HUGE improvement. It would unblock me from using Eldev in all scenarios that I have.

[suhail-singh]

I agree, just an ability to pull dependencies from Git would be a HUGE improvement.

Agreed. Hopefully, we won't have to wait too much longer.

@doublep please let us know in case there are additional points that you would like some discussion / clarification on.

[doublep]

Sorry, don't have time for it now. I made some design decisions and research, decided that I wouldn't try to use any external tool, because it would be likely impossible to merge with existing dependency management. Instead, I would pretty much reuse local dependency code, but with directory being hidden, checked out from Git and fixed to "package" loading mode. But I haven't written anything yet...

[doublep]

Finally committed some basic implementation. There are likely still many bugs, so I don't plan to release in the next days yes. Testing is certainly welcome (use eldev --unstable upgrade-self).

Basic usage: add sth. like

(eldev-use-vc-repository 'mylib :git "CLONE-FROM")

or

(eldev-use-vc-repository 'mylib :github "USER/REPO")

to file Eldev. Everything else should work as with regular dependencies. Note that like regular dependencies, VC/Git-based ones are not upgraded automatically. You need to run eldev upgrade manually.

Git dependencies are currently not inherited, so if your library uses one, the project needs to redeclare it. May add inheritance before 1.11.

[suhail-singh]

Seems that there are some issues. After upgrading to eldev 20241208.1709, in a project with the following Eldev file:

;; -*- mode: emacs-lisp; lexical-binding: t -*-

;; Uncomment some calls below as needed for your project.
(eldev-use-package-archive 'gnu-elpa)
(eldev-use-package-archive 'nongnu-elpa)
(eldev-use-package-archive 'melpa)
(eldev-use-vc-repository 'piem :git "https://git.kyleam.com/piem")

(eldev-add-extra-dependencies 'build 'magit)
(eldev-add-extra-dependencies 'build 'notmuch)
(eldev-add-extra-dependencies 'build 'piem)

(eldev-use-plugin 'maintainer)

(setq eldev-project-main-file "git-email.el")
(add-to-list 'eldev-update-copyright-fileset "!fdl-1.3.texi")

I observe failure when running any Eldev commands:

$> eldev version
Symbol’s function definition is void: eldev-use-vc-repository
Failed setup step: loading file ‘Eldev’
Run with ‘--debug’ (‘-d’) option to see error backtrace

Backtrace for eldev -d version:

Debugger entered--Lisp error: (void-function eldev-use-vc-repository)
  (eldev-use-vc-repository 'piem :git "https://git.kyleam.com/piem")
  load-with-code-conversion("/path/to/project/Eldev" "/path/to/project/Eldev" nil t)
  load("/path/to/project/Eldev" nil t t)
  eldev--set-up()
  eldev-cli(("-d" "version"))
  (kill-emacs (eldev-cli (append (cdr (member "--" command-line-args)) nil)))
  command-line-1(("--execute" "(let ((eldev--emacs-version (format \"%s.%s\" emacs-major-version emacs-minor-version))\n      (eldev--dir           (getenv \"ELDEV_DIR\"))\n      ;; This is intentional.  First, this is in case ELDEV_LOCAL is\n      ;; defined, second, this is just Eldev default for packages.\n      (load-prefer-newer    t))\n  ;; Setting `debug-on-error' would be useful, but it can break many\n  ;; `package-*' functions, since those use `with-demoted-errors' and\n  ;; so `condition-case-unless-debug'.\n  (unless (and (fboundp 'version<=) (version<= \"24.4\" eldev--emacs-version))\n    (error \"Eldev requires Emacs 24.4 or newer\"))\n  (setf package-user-dir\n        (expand-file-name \"bootstrap\"\n                          (expand-file-name eldev--emacs-version\n                                            (if (> (length eldev--dir) 0)\n                                                eldev--dir\n                                              (if (file-directory-p \"~/.eldev\")\n                                                  \"~/.eldev\"\n                                                ;; Duplicating not-yet-available code from `eldev-xdg-cache-home'.\n                                                (expand-file-name \"eldev\"\n                                                                  (let ((eldev--xdg-cache-dir (getenv \"XDG_CACHE_HOME\")))\n                                                                    (if (and eldev--xdg-cache-dir (file-name-absolute-p eldev--xdg-cache-dir))\n                                                                        eldev--xdg-cache-dir\n                                                                      \"~/.cache\")))))))\n        package-directory-list nil\n        package-archives       nil)\n  (require 'package)\n  (package-initialize t)\n  (let ((package-archives '((\"melpa-stable\" . \"http://stable.melpa.org/packages/\")))\n        (archive-name      \"MELPA Stable\")\n        (inhibit-message  t)\n        (eldev-local      (getenv \"ELDEV_LOCAL\"))\n        eldev-pkg\n        requirements)\n    (unless (= (length eldev-local) 0)\n      (if (string-prefix-p \":pa:\" eldev-local)\n          (setf package-archives `((\"bootstrap-pa\" . ,(file-name-as-directory (substring eldev-local (length \":pa:\")))))\n                archive-name     \"a local package archive\")\n        (with-temp-buffer\n          (insert-file-contents (expand-file-name \"eldev.el\" eldev-local))\n          (setf eldev-pkg                    (package-buffer-info)\n                (package-desc-dir eldev-pkg) (expand-file-name eldev-local))\n          ;; Currently Eldev has no external dependencies, but let's be generic.\n          (dolist (requirement (package-desc-reqs eldev-pkg))\n            (unless (package-activate (car requirement))\n              (push requirement requirements))))))\n    (when (if eldev-pkg\n              requirements\n            (not (package-activate 'eldev)))\n      (let ((inhibit-message nil))\n        (message \"Bootstrapping Eldev for Emacs %s from %s...\\n\" eldev--emacs-version archive-name)\n        (when eldev-pkg\n          (message \"Eldev package itself will be used from `%s'\\n\" eldev-local)))\n      ;; See `eldev-retrying-for-robustness'; since Eldev is not bootstrapped yet, we have\n      ;; to inline everything.  No control from command line here.\n      (let* ((all-retry-delays (when (equal (getenv \"CI\") \"true\") '(30 60 120 180 300)))\n             (remaining-delays all-retry-delays))\n        (catch 'obtained-result\n          (while t\n            (condition-case error\n                (throw 'obtained-result (let ((debug-on-error (and debug-on-error (null remaining-delays))))\n                                          ;; See similar workarounds for `package-refresh-contents' in `eldev.el'.\n                                          (let* (failure\n                                                 (failure-catcher (lambda (original archive &rest arguments)\n                                                                    (unless failure\n                                                                      (condition-case-unless-debug error\n                                                                          (apply original archive arguments)\n                                                                        (error (setf failure (cons error (if (consp archive) (car archive) archive)))))))))\n                                            (advice-add 'package--download-one-archive :around failure-catcher)\n                                            (unwind-protect\n                                                (package-refresh-contents)\n                                              (advice-remove 'package--download-one-archive failure-catcher))\n                                            (when failure\n                                              (error \"%s (when updating contents of package archive `%s')\" (error-message-string (car failure)) (cdr failure))))))\n              (error (let ((inhibit-message nil)\n                           (delay           (pop remaining-delays)))\n                       (unless delay\n                         (when all-retry-delays\n                           (message \"Giving up: too many retries already\"))\n                         (signal (car error) (cdr error)))\n                       (message \"%s\" (error-message-string error))\n                       (message \"Assuming this is an intermittent problem, waiting %s before retrying...\\n\"\n                                (if (< delay 60) (format \"%s s\" delay) (format \"%s m\" (/ delay 60))))\n                       (sleep-for delay)\n                       (let ((n (- 5 (length remaining-delays))))\n                         (message \"Retry #%d%s...\" n (if (= n 5) \", the last\" \" of maximum 5\")))))))))\n      (if eldev-pkg\n          (package-download-transaction (package-compute-transaction nil requirements))\n        (package-install 'eldev)))\n    (when eldev-pkg\n      (push `(eldev . (,eldev-pkg)) package-alist)\n      ;; `package--autoloads-file-name' is package-private.\n      (let* ((autoloads-file     (expand-file-name (format \"%s-autoloads\" (package-desc-name eldev-pkg))\n                                                   (package-desc-dir eldev-pkg)))\n             (autoloads-disabler (lambda (do-load file &rest args) (unless (equal file autoloads-file) (apply do-load file args)))))\n        ;; Otherwise old Emacs versions print an ugly error having not found the autoloads file.\n        (advice-add #'load :around autoloads-disabler)\n        (package-activate-1 eldev-pkg)\n        ;; As of commit 1d5b164109b in Emacs repository, `package-activate-1' no longer modifies `load-path',\n        ;; leaving this to the autoloads file.  As we don't have such a file, we have to do that ourselves.\n        (add-to-list 'load-path (package-desc-dir eldev-pkg))\n        (advice-remove #'load autoloads-disabler))))\n  (require 'eldev)\n  (eldev-start-up))" "--execute" "(kill-emacs (eldev-cli (append (cdr (member \"--\" command-line-args)) nil)))" "--" "-d" "version"))
  command-line()
  normal-top-level()

[doublep]

Sorry, it's eldev-use-vc-dependency, not ...-vc-repository.

[suhail-singh]

Does the eldev-use-vc-dependency form obviate the need for a eldev-add-extra-dependencies form?

For instance if we have:

(eldev-use-vc-dependency 'piem :git "https://git.kyleam.com/piem")

Do we still need:

(eldev-add-extra-dependencies 'build 'piem)

[doublep]

They are independent.

eldev-use-vc-dependecy is comparable to eldev-use-package-archive in that it tells Eldev how to find packages (former: one specific package in the given Git repository; latter: in a package archive, whatever its contents is). They both don't tell anything about what you need the packages for.

For that, packages must be declared as dependencies in project headers (in your main .el file or *-pkg.el). Or they can be listed in eldev-add-extra-dependencies. For this aspect there are no changes.

[doublep]

Actually, now that I think of it, ...-use-vc-repository would be a better name (and more similar to ...-use-package-archive), since it doesn't itself declare that it will necessary be a dependency. It's certainly still possible to rename, since there is no stable release with this functionality. What do you think?

[suhail-singh]

In that case, eldev-use-vc-repository is a more appropriate name, IMO.

BTW, if it helps (it certainly would help me) please consider testing out the feature on: https://codeberg.org/suhail/git-email/commits/branch/dev. The last commit adds piem as a dependency via eldev-use-vc-.... I was not able to invoke eldev compile successfully after that. The first invocation seemed to get stuck, but perhaps I was too eager. After C-c'ing out of the seemingly stuck eldev compile invocation I encountered the following:

$> eldev compile
Output of the ‘git’ process:
fatal: destination path '/path/to/project/git-email/.eldev/git/piem' already exists and is not an empty directory.

‘git’ process exited with error code 128

[doublep]

That's why I don't want to release yet: fixed three different bugs as a result.

After the fixes, in git-email I added these two lines:

(eldev-use-vc-repository 'piem :git "https://git.kyleam.com/piem")
(eldev-add-extra-dependencies '(build eval emacs) 'piem)

This still results in an error, but this time the problem is in piem (or rather its infrastructure): Eldev doesn't know how to build a package from it, because the project doesn't tell it:

$ eldev eval 1
[1/0] Installing package ‘piem’ (0.5.0.20241204.112) from ‘https://git.kyleam.com/piem’...
Dependency ‘transient’ is not available
Required by package ‘piem’
There is no file ‘Eldev’ in the project; consider running ‘eldev init’
Child Eldev process for VC-fetched package ‘piem’ exited with error code 1
Failed step: installing dependency package ‘piem’

I could resolve that by effectively providing Eldev settings for piem from git-email like this:

(eldev-use-vc-repository 'piem :git "https://git.kyleam.com/piem"
                         :setup '(eldev-use-package-archive 'gnu-elpa))

You can instead make piem buildable with Eldev directly, then git-email won't have to provide the setup form for it.

[suhail-singh]

You can instead make piem buildable with Eldev directly, then git-email won't have to provide the setup form for it.

In this particular instance, since piem is a project maintained by a third-party, having the option to specify :setup in eldev-use-vc-repository is valuable.

Having resolved the issue with piem, what would be the appropriate eldev-use-vc-repository incantation for mu4e whose package-vc-install invocation looks like below?

(package-vc-install '(mu4e :url "https://github.com/djcb/mu"
                           :lisp-dir "./mu4e")
                    "v1.12.7")

[suhail-singh]

The latest commit on git-email dev branch adds the following eldev-use-vc-repository invocation for mu4e (inspired from vc-use-package extension's syntax):

(eldev-use-vc-repository 'mu4e :git "https://github.com/djcb/mu"
                         :lisp-dir "./mu4e"
                         :rev "v1.12.7")

It currently fails with:

$> eldev compile
No .el files with package headers in ‘/path/to/project/git-email/.eldev/git/mu4e/’

Perhaps some :setup needs to be provided?

[doublep]

I don't plan to support its syntax.

:lisp-dir "./mu4e"

If it means what I think it means, you can include (setf eldev-project-source-dirs "mu4e") into the :setup form. If you need to combine several setup forms, just use progn.

:rev "v1.12.7"

What does that mean?

Currently ...-use-vc-repository uses HEAD and treats it like a snapshot build by appending commit timestamp to the version. I plan to add optional handling of tagged (like what MELPA uses) stable builds.

[suhail-singh]

:rev "v1.12.7"

What does that mean?

A revision could be a git commit or a git tag. This in addition to the :branch (which defaults to the HEAD of said branch) allows one to check out a specific version of the dependency.

Hopefully something like that would be supported.

[suhail-singh]

you can include (setf eldev-project-source-dirs "mu4e") into the :setup form.

Hm I'm unable to get this to work. I've pushed the related change to git-email's dev branch.

[doublep]

A revision could be a git commit or a git tag. This in addition to the :branch (which defaults to the HEAD of said branch) allows one to check out a specific version of the dependency.

Hopefully something like that would be supported.

What do you need this for? It's probably not difficult to support, but I don't understand the need for it. Normal users of your project will not be limited by your decision to use a specific version of a dependency, they will use the latest overall (snapshot) or the latest released (stable) version.

Elisp package archives, for example, don't give you access to old releases at all, so Eldev cannot support that in that case in principle. With Git repositories it could, but why should it be different?

Hm I'm unable to get this to work. I've pushed the related change to git-email's dev branch.

I have a branch named future-doc and I have some documentation for this ...-use-vc-repository there already. Check it out if you can. For example, here is what paragraph about :setup says:

This form will be given to child Eldev processes whenever a package needs to be created out of the VC-fetched project. Most likely you will need to tell child Eldev how to locate dependencies of that project. The best way to figure it out is to check out the repository and create file Eldev in it with required contents (eldev init might help). Instead of committing the created file, convert its contents into an Elisp form (use progn if needed) and pass it as a value of :setup in your main project.

I suggest you try "the best way" outlined above yourself first. Basically, make mu4e buildable (eldev package) with Eldev. You cannot commit your work, but you can directly convert produced Eldev file into a setup form and include said form into your project's Eldev.

[suhail-singh]

What do you need this for?

For having a more deterministic build process.

Normal users of your project will not be limited by your decision to use a specific version of a dependency, they will use the latest overall (snapshot) or the latest released (stable) version.

Some users may also use specific releases. I am one such user.

Basically, make mu4e buildable (eldev package) with Eldev. You cannot commit your work, but you can directly convert produced Eldev file into a setup form and include said form into your project's Eldev.

Noted. That's a sensible approach.

[doublep]

Some users may also use specific releases. I am one such user.

OK, I added this feature. Pass :commit "..." to eldev-use-vc-repository (can be a tag name or a full Git commit hash).

Noted. That's a sensible approach.

Any success with that?