org-basb-code

org-basb-code copied to clipboard

#+TITLE: Emacs CODE implementation #+AUTHOR: Renat Galimov #+EXPORT_FILE_NAME: index

[[https://github.com/renatgalimov/org-basb-code/actions/workflows/ci.yml/badge.svg]]

• CODE implementation :PROPERTIES: :DIR: ~/projects/org-basb-code/attachments/ :header-args: :tangle no :END:

https://renatgalimov.github.io/org-basb-code/

** How to work with this guide?

• Copy-paste the [[#emacs-config][config]] source block at the end of this document into your =.emacs= file or into a file in your emacs directory.
• In case of putting it into a separate file, add the following code to your emacs config file =(load-file "~/path/to/basb.el")=
• If you want to try this setup on a fresh Emacs do the following:
  • Create a new directory inside of your HOME: =mkdir ~/.emacs.code.d=
  • Put there a new =init.el=: #+caption: Contents of ~/.emacs.code.d/CODE.el #+begin_src emacs-lisp (setq user-init-file (or load-file-name (buffer-file-name))) (setq user-emacs-directory (file-name-directory user-init-file)) (load-file "~/.emacs.code.d/CODE.el") #+end_src
  • Copy-paste the [[*Emacs config][config]] into a new ="~/.emacs.code.d/CODE.el"=
  • Load a fresh Emacs with =emacs -q --load ~/.emacs.code.d/init.el=
  • Emacs will download and install all the packages defined in this guide. You might need depelopment tools installed to compile some packages. File a bug with errors you get and I will extend this guide.
• I will be using =C-c n= prefix for keybindings in this package.

** Prerequisites *** Pandoc

#+caption: Install Pandoc on Ubuntu #+begin_src bash :exports code :eval never-export sudo apt install pandoc #+end_src

#+caption: Install Pandoc on Mac OS with homebrew #+begin_src bash :exports code :eval never-export brew install pandoc #+end_src

Ensure that Pandoc is in your path.

#+caption: Pandoc version #+begin_src bash :exports both :eval never-export :results verbatim pandoc --version #+end_src

#+RESULTS: : pandoc 2.14.0.2 : Compiled with pandoc-types 1.22, texmath 0.12.3, skylighting 0.10.5.1, : citeproc 0.4.0.1, ipynb 0.1.0.1 : User data directory: /Users/renat.galimov/.local/share/pandoc : Copyright (C) 2006-2021 John MacFarlane. Web: https://pandoc.org : This is free software; see the source for copying conditions. There is no : warranty, not even for merchantability or fitness for a particular purpose.

*** melpa and melpa-stable We will be downloading packages from these repos

#+caption: Set up MELPA #+name: set-up-melpa #+begin_src emacs-lisp :exports code :eval never-export :results none (unless (boundp 'package-archives) (setq package-archives '()))

(let* ((no-ssl (and (memq system-type '(windows-nt ms-dos)) (not (gnutls-available-p)))) (proto (if no-ssl "http" "https"))) (add-to-list 'package-archives (cons "gnu" (concat proto "://elpa.gnu.org/packages/")) t) ;; (add-to-list 'package-archives (cons "nongnu" (concat proto "://elpa.nongnu.org/packages/")) t) (add-to-list 'package-archives (cons "melpa" (concat proto "://melpa.org/packages/")) t) (add-to-list 'package-archives (cons "melpa-stable" (concat proto "://stable.melpa.org/packages/")) t))

(package-refresh-contents)

(unless (bound-and-true-p package--initialized) (package-initialize)) #+end_src

*** straight.el

For this guide, we will be using [[https://github.com/raxod502/straight.el][straight.el]] as your package manager.

#+caption: Install straight.el #+name: install-straight-el #+begin_src emacs-lisp :exports code :eval never-export :results none (defvar bootstrap-version) (if (package-installed-p 'use-package) (eval-when-compile (require 'use-package)) (package-refresh-contents) (package-install 'use-package))

(unless (boundp 'straight-use-package)
  (message "Bootstrapping straight.el")
  (let ((bootstrap-file
         (expand-file-name "straight/repos/straight.el/bootstrap.el" user-emacs-directory))
        (bootstrap-version 5))
    (unless (file-exists-p bootstrap-file)
      (with-current-buffer
          (url-retrieve-synchronously
           "https://raw.githubusercontent.com/raxod502/straight.el/develop/install.el"
           'silent 'inhibit-cookies)
        (goto-char (point-max))
        (eval-print-last-sexp)))
    (load bootstrap-file nil 'nomessage))
  (message "Bootstrapping straight.el done"))

#+end_src

*** Org-roam Obviously, we need [[https://github.com/org-roam/org-roam][org-roam]] itself #+caption: Install org-roam #+name: install-org-roam #+begin_src emacs-lisp :exports code :eval never-export :results none (setq org-roam-v2-ack t) (use-package org-roam :after org :ensure t :pin "melpa-stable" :bind (("C-c n l" . #'org-roam-buffer-toggle) ("C-c n f" . #'org-roam-node-find) ("C-c n v" . #'org-roam-node-visit) ("C-c n i" . #'org-roam-node-insert) (("C-c q" . #'org-roam-tag-add)))

:config
;; We want org-id to work with our roam directory
(setq org-roam-file-extensions '("org" "org_archive")
      org-id-extra-files (org-roam-list-files)))

#+end_src

** Capture

#+begin_quote Emacs is a text editor. #+end_quote

*** Capture documents

• Capture targets:

  • E-books

  • Documents

  • Videos

  • Audios

  • Images

    To capture we use Pandoc and [[https://github.com/tecosaur/org-pandoc-import][org-pandoc-import]]

    #+name: install-org-pandoc-import #+begin_src emacs-lisp (straight-use-package '(org-pandoc-import :host github :repo "tecosaur/org-pandoc-import" :files ("*.el" "filters" "preprocessors")))

    (global-set-key (kbd "C-c n o") #'org-pandoc-import-as-org) #+end_src

    Pandoc can convert almost any text format to org-mode representation. One of the current drawbacks - it cannot import online web pages.

*** Capture web-pages

**** org-web-tools [[https://github.com/alphapapa/org-web-tools][org-web-tools]] offers =org-web-tools-read-url-as-org= function, which can download an URL to an org buffer.

#+name: install-org-web-tools #+begin_src emacs-lisp (use-package org-web-tools :ensure t :pin "melpa-stable" :bind (("C-c n u" . org-web-tools-read-url-as-org))) #+end_src

Often, downloaded files need manual cleanup.

**** Firefox web-clipper and Pandoc

An alternative approach is to use a web-clipper and Pandoc.

• Open the web article in your browser and activate a web-clipper. I use Firefoxes built-in one.

• Save entire HTML into a file.

• Convert the HTML with Pandoc #+caption: Convert HTML to Org with Pandoc #+begin_src bash pandoc -f html -t org .html -o .org #+end_src

  The resulting file might require some cleanup, but the quality of the output is the best among other methods.

*** Capture templates <<Please, contribute your capture templates>>

*** Finding your own records Crawling over your own notes is a key part of the project workflow.

Below I summarized information about all full-text search engines I found for org-mode.

Depending on your own need you might select one or multiple of them.

Unfortunately, I didn't find any ideal solution for a full-text search yet. Packages that make better sorting are slower than packages that give results fast but in a random order.

***** Org full-text search :table: :PROPERTIES: :ID: 47985238-3e66-4201-969c-16d1858b797e :COLUMNS: %25ITEM %FRONTEND %SPEED %SORT :END:

• Requested features:

  • helm or counsel integration.

  • live search.

  • result previewing support.

  • a key-binding to capture results into currently clocked org file.

  • results ordering support

    • Headlines
    • Summary
    • Hightlight

  • large file-sets support

    #+CAPTION: Full text search engines features #+BEGIN: columnview :hlines 1 :id local :maxlevel 2 :indent t | ITEM | FRONTEND | SPEED | SORT | |---------------------------+----------+-------+------| | Org-roam Full-text search | | | | | _ ripgrep (helm-rg) | helm ivy | fast | nil | | _ helm-org-rifle | helm | slow | t | | _ deft | helm | fast | nil | | _ org-ql | helm | slow | | | _ recoll | helm ivy | fast | nil | | _ org-fts | ivy | fast | | | _ org-agenda search | | | | #+END:

******* ripgrep (helm-rg) :PROPERTIES: :frontend: helm ivy :full_text_search: t :speed: fast :sort: nil :END:

[[https://github.com/cosmicexplorer/helm-rg][GitHub - cosmicexplorer/helm-rg: ripgrep is nice]] Now I use it as a default text search engine.

#+caption: Install ripgrep in MacOS #+begin_src bash brew install ripgrep #+end_src

#+caption: Install helm-rg #+name: install-helm-rg #+begin_src emacs-lisp (use-package helm-rg :ensure t :after org-roam :pin "melpa-stable" :config (defun helm-rg-roam-directory (&optional query) "Search with rg in your roam directory, QUERY." (interactive) (let ((helm-rg-default-directory org-roam-directory) (helm-rg--current-dir org-roam-directory)) (helm-rg query nil))) :bind (("C-c n R" . helm-rg-roam-directory))) #+end_src

******* helm-org-rifle :PROPERTIES: :frontend: helm :speed: slow :sort: t :END:

[[https://github.com/alphapapa/org-rifle][GitHub - alphapapa/org-rifle: Rifle through your Org-mode buffers and acquire your target]]

This one is good. It gives you an idea about the context. But it's not ordering the data by the highlights.

I find org-rifle too slow at the moment. But its output is exacly what I want.

#+caption: Install org-rifle #+name: install-org-rifle #+begin_src emacs-lisp :comments both (use-package helm-org-rifle :ensure t :after org-roam :pin "melpa-stable" :config (defun org-rifle-roam-directory () (interactive) (helm-org-rifle-directories org-roam-directory)) :bind (("C-c n s" . org-rifle-roam-directory))) #+end_src

******* deft :PROPERTIES: :frontend: helm :speed: fast :sort: nil :END: [[https://github.com/dfeich/helm-deft][GitHub - dfeich/helm-deft: A helm based Emacs module to help search in a predetermined list of directories. Inspired by the deft module.]]

Helm implementation didn't work for me. So I set a default version here.

One of the drawbacks here is that you can't see the text you matched. From my point of view - helm-rg gives more precise information about the context.

#+name: install-deft #+begin_src emacs-lisp (use-package deft :ensure t :after org-roam :config (setq deft-directory org-roam-directory deft-recursive t) :bind (("C-c n d" . deft))) ;; (use-package helm-deft ;; :ensure t ;; :straight (:host github ;; :repo "dfeich/helm-deft" ;; :files ("*.el")) ;; :config ;; (setq helm-deft-dir-list `(,org-roam-directory) ;; helm-deft-extension '("org")) ;; :bind (("C-c n d" . helm-deft)))

#+end_src

******* org-ql :PROPERTIES: :frontend: helm :speed: slow :END: [[https://github.com/alphapapa/org-ql][GitHub - alphapapa/org-ql: An Org-mode query language, including search commands and saved views]]

Doesn't look suitable for large filesets, but helm implementation is good for medium-sized collections.

#+name: install-org-ql #+begin_src emacs-lisp (use-package org-ql :ensure t :after org :config (progn (message "Loading org-ql") (setq org-ql-search-directories-files-recursive t org-ql-search-directories-files-regexp ".org\(_archive\)?$")) )

(use-package helm-org-ql :ensure t :after org-ql :config (setq helm-org-ql-recursive-paths t) :bind (("C-c n q" . helm-org-ql-org-directory))) #+end_src ******* recoll :PROPERTIES: :frontend: helm ivy :speed: fast :sort: nil :END: [[https://github.com/emacs-helm/helm-recoll][GitHub - emacs-helm/helm-recoll: helm interface for the recoll desktop search tool.]] I found recoll being to hard to set up. I wasn't able to get it working on MacOS.

******* org-fts :PROPERTIES: :speed: fast :frontend: ivy :END: [[https://github.com/zot/microfts/tree/main/elisp][microfts/elisp at main · zot/microfts · GitHub]]

• It didn't work on MacOS from scratch.
• I tried to compile its binary manually but that didn't work either.
• This one looks promising. Let's keep an eye on it.

******* org-agenda search

• Not trying this for now because agenda wants to open all its files for search.

** Organize *** Org-mode tags

This is a default method. You just tag your projects with a =project= tag, areas with an =area= tag and so on.

You can search projects with helm or with org-ql:

#+caption: Finding projects with helm [[file:Emacs_CODE_implementation/2021-07-04_07-52-17_screenshot.png]]

#+caption: Finding projects with org-ql [[file:Emacs_CODE_implementation/2021-07-04_07-57-55_screenshot.png]]

*** Org-roam links

An alternative to use Org-roam references as tags. You create notebooks called: =projects=, =areas=, =resources= and insert a link to the target record according to where it belongs.

#+caption: Links to an area within notes [[file:Emacs_CODE_implementation/2021-07-04_08-05-12_screenshot.png]]

If you go to the =areas= note and call =org-roam-buffer-toggle=, you will see a backlink to you note in the side buffer.

#+caption: Backlinks to all areas in the side buffer [[file:Emacs_CODE_implementation/2021-07-04_08-07-11_screenshot.png]]

*** Org-roam-ui

A nice second-brain visualizator: [[https://github.com/org-roam/org-roam-ui][org-roam-ui]].

#+caption: org-roam-ui design #+attr_org: :width 400px [[file:Emacs_CODE_implementation/2021-11-03_21-44-11_screenshot.png]]

#+caption: Install org-roam-ui #+name: install-org-roam-ui #+begin_src elisp :exports code :eval never-export :results none (use-package org-roam-ui :ensure t :after org-roam ;; normally we'd recommend hooking orui after org-roam, but since org-roam does not have ;; a hookable mode anymore, you're advised to pick something yourself ;; if you don't care about startup time, use :hook (after-init . org-roam-ui-mode)

:config
(setq org-roam-ui-sync-theme t
      org-roam-ui-follow t
      org-roam-ui-update-on-save t
      org-roam-ui-open-on-start t))

#+end_src

** Distill *** Progressive summarization **** Make org-emphasize multi-linear

To bypass the limit of two lines for org-emphasize marks enable the code below.

#+name: increase-org-emphasis-limit #+begin_src emacs-lisp ;; Make org-emphasis to work on up to 10 lines selection. (setcar (nthcdr 4 org-emphasis-regexp-components) 10) (org-set-emph-re 'org-emphasis-regexp-components org-emphasis-regexp-components) #+end_src

**** Highlighting :LOGBOOK: CLOCK: [2021-06-13 Вс 07:16]--[2021-06-13 Sun 07:31] => 0:15 CLOCK: [2021-06-13 Вс 06:49]--[2021-06-13 Вс 07:15] => 0:26 :END:

Highlighting is a key part of progressive summarization. Here I will be highlighting with =highlight.el= and enriched mode. If you don't want to put your text file into the [[https://www.gnu.org/software/emacs/manual/html_node/emacs/Enriched-Mode.html][enriched text mode]], you can use =org-emphasize= instead of functions provided here.

***** highlight.el

When working with plain text buffers, like org-mode or markdown, you can use [[https://www.gnu.org/software/emacs/manual/html_node/emacs/Enriched-Mode.html][enriched text mode]] with the =highlight= library to mark the text.

#+caption: Set up highlight.el #+name: setup-highlight-el #+begin_src emacs-lisp ;; If you get errors saying somethign about facemenu, try ;; uncommenting this. ;; (setq facemenu-menu nil)

(use-package highlight :ensure t :config (defun hlt-general() (interactive) (unless (bound-and-true-p enriched-mode) (enriched-mode t)) (hlt-highlight-region (region-beginning) (region-end) 'highlight))

;; Without this setq highlights won't be saved even in enriched mode
(setq hlt-use-overlays-flag nil)

:bind (("C-c n h" . hlt-general)
       ("C-c n H" . hlt-unhighlight-region)))

;; If you cannot save your enriched files because of the :inherit ;; error, try uncommenting this function.

;; (defun enriched-face-ans (face) ;; "Return annotations specifying FACE. ;; FACE may be a list of faces instead of a single face; ;; it can also be anything allowed as an element of a list ;; which can be the value of the `face' text property." ;; (cond ((and (consp face) (eq (car face) 'foreground-color)) ;; (list (list "x-color" (cdr face)))) ;; ((and (consp face) (eq (car face) 'background-color)) ;; (list (list "x-bg-color" (cdr face)))) ;; ((and (listp face) (eq (car face) :foreground)) ;; (list (list "x-color" (cadr face)))) ;; ((and (listp face) (eq (car face) :background)) ;; (list (list "x-bg-color" (cadr face)))) ;; ((and (listp face) (eq (car face) :inherit)) ;; (enriched-face-ans (cdr face))) ;; ((listp face) ;; (apply 'append (mapcar 'enriched-face-ans face))) ;; ((let* ((fg (face-attribute face :foreground)) ;; (bg (face-attribute face :background)) ;; (props (face-font face t)) ;; (ans (cdr (format-annotate-single-property-change ;; 'face nil props enriched-translations)))) ;; (unless (eq fg 'unspecified) ;; (setq ans (cons (list "x-color" fg) ans))) ;; (unless (eq bg 'unspecified) ;; (setq ans (cons (list "x-bg-color" bg) ans))) ;; ans)))) #+end_src

***** org-capture a region :ATTACH: :PROPERTIES: :ID: 1FD14535-77E8-4296-9BC9-A72DB1CB7E0F :DIR: ~/projects/org-basb-code/attachments :END:

To keep track of highlighted notes we will use =org-capture=.

#+caption: A capture template #+name: highlight-capture #+begin_src elisp ;; Use =org-capture f= to put a link to the text you selected.into an ;; org entry with the current timer enabled.

(defun r/org-capture-get-selected-text () (with-current-buffer (org-capture-get :original-buffer) (string-trim (replace-regexp-in-string "\n" " " (cond ((eq major-mode 'pdf-view-mode) (pdf-info-gettext (pdf-view-current-page) (car (pdf-view-active-region)))) (t (buffer-substring-no-properties (region-beginning) (region-end)))))))) (defun r/org-capture-get-link (path) (with-current-buffer (org-capture-get :original-buffer) (cond ((eq major-mode 'pdf-view-mode) (switch-to-buffer (org-capture-get :original-buffer)) (org-pdftools-get-link)) (t (concat path "::" (r/org-capture-get-selected-text))))))

(with-eval-after-load "org-capture" (add-to-list 'org-capture-templates '("f" "Curently watched" item (clock) "%(r/org-capture-get-selected-text) [[%(r/org-capture-get-link "%F")][↗]]%?" :unnarrowed t)))

;; The code below automatically highlights the region we captured (defun do-highlight-on-capture () "Highlight selected region of the buffer you were in at capture." (save-excursion (with-current-buffer (plist-get org-capture-plist :original-buffer) (cond ((eq major-mode 'pdf-view-mode) (switch-to-buffer (org-capture-get :original-buffer)) (pdf-annot-add-highlight-markup-annotation (car (pdf-view-active-region)))) (t (hlt-general)))))) (defun highlight-on-capture () (when (equal (plist-get org-capture-plist :key) "f") (do-highlight-on-capture)))

(add-hook 'org-capture-after-finalize-hook #'highlight-on-capture) #+end_src

This is my basic marking mechanism. Whenever I'm reading an article in Emacs (transformed to an org-mode or markdown file), I click =C-c f= to insert an entry to the notebook I'm currently on.

#+caption: Highlighting on capture demo #+attr_org: :width 400px [[file:attachments/highlighting-with-org-capture.gif]] *** Working with PDF files :ATTACH: :PROPERTIES: :ID: B5A87A1B-E0FE-40D6-AC63-AD52DB283C31 :CUSTOM_ID: working-with-pdf-files :END:

Don't forget to install pdf-tools dependencies.

#+name: install glib on MacOS #+begin_src bash brew install glib #+end_src

#+name: install-pdf-tools #+begin_src emacs-lisp (use-package pdf-tools :ensure t :mode ("\.pdf\'" . pdf-view-mode) :init (require 'pdf-info) (condition-case nil (pdf-info-check-epdfinfo) (error (let ((default-directory (file-name-directory pdf-info-epdfinfo-program))) (pdf-tools-install t t)))))

(use-package org-pdftools :ensure t :hook (org-mode . org-pdftools-setup-link)) #+end_src ** Express This area is too large to cover it in this guide. If you're new to org-mode consult

[[https://orgmode.org/manual/Exporting.html][Exporting]] and [[https://orgmode.org/manual/Publishing.html][publishing]] sections of the org manual.

• Emacs config :PROPERTIES: :CUSTOM_ID: emacs-config :END: #+caption: Config source block #+begin_src emacs-lisp :noweb yes :comments noweb :tangle (if (boundp 'org-basb-code-file) org-basb-code-file "~/emacs/CODE.el") <> <> <> <> <> <> <> <> <> <> <> <> <> <> #+end_src

• Useful links

• [[https://gist.github.com/mwfogleman/267b6bc7e512826a2c36cb57f0e3d854][GTD/BASB Templates for Emacs and Org-Mode · GitHub]]
• [[yt:AyhPmypHDEw]]
• [[https://github.com/weirdNox/org-noter][GitHub - weirdNox/org-noter: Emacs document annotator, using Org-mode]]

• Improving this guide

• [ ] per-org-document style

• [ ] org-pdf-import videos for

  • [ ] epub
  • [ ] doc files

• [ ] org web read url

• [ ] fix invalid face error.

• [ ] Faces not saving in enriched mode in vanilla emacs setup.

• [ ] Links made by capture are broken.

• [X] Use normal emacs

• [ ] Org pdftools link

• [ ] Capturing emails

• [ ] Multi-line emphasis are not working on MacOS Emacs

• [ ] Org pdftools: fix links for Linux and MacOS

• [ ] Videos:

  Be slow when doing videos. Give more details?

  • [ ] Capturing different formats:
    • [ ] HTML
    • [ ] PDF
  • [ ] Organize: