autoloading functions defined by yourself

[Luis-Henriquez-Perez]

Hi Noctuid! I'm a big fan of your work. I think that evil-lispyville and general are excellent packages. And your dotfiles are a great example to look at. I actually agree a lot with your dotfiles readme. It should definitely be in awesome emacs configs. Actually I'll probably pull request it soon.

In your awaken.org file you mention that you aggressively autoload functions to reduce startup time. With functions from other packages, I see how that happens using the :commands keywords. However, for some packages I have a lot of functions that are defined by me in my big org file. If it's a just a few, using with-eval-after-load on them (or the :config keyword in use-package) is fine. But what if there are a lot, like hundreds of them? If we try to use with-eval-after-load it will likely produce a small lag when the feature is loaded because many functions are being defined. In your awaken.org file it doesn't look like you've run into such a case yet, you only define a relatively small number of functions for each package.

The basic problem is that emacs's autoload facility assumes you're going to use separate files for your autoloads.

There are two ways I thought of to deal with this.

1. is to tangle to multiple autoload .el files and place calls to autoload in the org file. (This could be automated say with a :autoload switch in code blocks so you wouldn't actually have to create your own autoload calls).)

The problem is that tangling time (with a custom tangle function) increases significantly when tangling to multiple files. This potentially could be resolved if you could somehow only tangle to a file only if source blocks that tangle to it have changed, but I'm not sure how to check that.

2. Use a custom autoload method powered by with-eval-after-load

I hacked together my own autoload method. The gist is to have a list alist of (FEATURE . LOAD-FEATURE-FN). When you have a function you want to autoload (in this example it's an autoload adding evil integration to eshell) function:

(with-eval-after-load '+eshell-evil-autoloads
  (defun my-autoload-fun ()
    ...))
(fset 'my-autoload-fun (alist-get '+eshell-evil=autoloads my-autoload-alist))

The call to the my-autoload-alist will return this lambda:

(lambda (&rest args)
  (interactive)
  (provide 'eshell-evil-autoloads)
  (funcall this-command args))

So all of the evil eshell autoloads point to the same lambda.

Of course this needs to be tidied up a bit perhaps one lambda for all interactive functions and one lambda for all non-interactive. Also if we only create one lambda the neither docstring will nor the arguments not be available in advance.

The big qualm I have is I'm not sure how efficient this is and whether it's worth it. I know autoload is coded in c which I assume was to avoid the overhead of creating these so many functions in lisp.

I'd appreciate any ideas you'd have on this.

[noctuid]

Have you actually tested that using autoloads is faster? Or just (with-eval-after-load 'package (require 'my-functions))? I'd be surprised if there was a big difference.

If there isn't a big difference and the pause is a big annoyance you might be better off loading these functions during initialization or with an idle timer if you don't normally use them right away.

[Luis-Henriquez-Perez]

Have you actually tested that using autoloads is faster? Or just (with-eval-after-load 'package (require 'my-functions))?

I haven't tested. I should profile because a lot of questions come down to that. For instance how costly is it exactly to define a function? How many functions need to be defined for the pause to be noticeable? And does the cost defining a function increase with the size of it's body? I'll look into these questions and get back to you.

[Luis-Henriquez-Perez]

This is the macro for autoloading functions I've settled with for now (note that progn is aliased to do and after! is a wrapper around with-eval-after-load). I've also put two benchmarks I've done. The time it takes to define a function is so small I don't this even defining hundreds will have a significant impact on initialization, but I use these autoloads anyway.

Also on as a side issue, you might want to checkout what I did with the macro elisp-block! in my init.el. It was influenced by your tangling strategy but I added a way to get source block where the error comes from as well as a way to wrap an arbitrary number of forms around source blocks based on their switches.

defautoload macro

(defmacro defautoload! (name args docstring &rest body)
  "Autoload function or macro with feature."
  (declare (indent defun) (doc-string 3))

  (-let* (((&plist :feature :macro :fn)
           (cl-loop while (member (car body) '(:feature :macro :fn))
                    append (list (pop body) (pop body))))
          (interactivep
           (and (listp (car body)) (eq 'interactive (caar body))))
          (caller
           (if macro
               (if (booleanp macro) 'defmacro macro)
             (if (or (null fn) (booleanp fn)) 'defun fn))))
    `(do
       (after! ,feature (,caller ,name ,args ,docstring ,@body))
       (fset ',name
             (lambda ()
               ,@(when interactivep '((interactive)))
               (fmakunbound ',name)
               (provide ',feature)
               (condition-case nil
                   (,name ,@args)
                 (void-function
                  (message "Function %S was not provided by feature %s"
                           ',name ',feature))))))))

Example Autoload

(defautoload! void//kill-emacs-brutally ()
  "Have an external process kill emacs."
  :feature +os
  (interactive)
  (when (yes-or-no-p "Do you want to BRUTALLY kill emacs?")
    (call-process "kill" nil nil nil "-9" (number-to-string (emacs-pid)))))

Creating a function with a defun

(-let [names (--map (gensym "my-fun") (make-list 1000 nil))]
  (do1
      (benchmark 1
                 `(do
                    ,@(--map `(defun ,it (a b c d e f) (+ 1 2 3 4)) names)))
    ;; unbind these functions afterwards.
    (-each names #'fmakunbound)))

"Elapsed time: 0.009856s"

"Elapsed time: 0.010263s"

"Elapsed time: 0.010260s"

"Elapsed time: 0.003379s"

"Elapsed time: 0.009283s"

"Elapsed time: 0.009172s"

"Elapsed time: 0.008890s"

"Elapsed time: 0.009109s"

"Elapsed time: 0.009411s"

"Elapsed time: 0.010047s"

Fsetting a symbol to a lambda

(-let ((names (--map (gensym "my-fun") (make-list 1000 nil))))
  (do1 (benchmark 1
                  `(progn
                     ,@(--map `(fset ',it (lambda (a b c d e f) (+ 1 2 3 4)))
                              names)))
    (-each names #'fmakunbound)))

"Elapsed time: 0.003937s"

"Elapsed time: 0.003949s"

"Elapsed time: 0.003958s"

"Elapsed time: 0.003966s"

"Elapsed time: 0.004083s"

"Elapsed time: 0.001289s"

"Elapsed time: 0.002047s"

"Elapsed time: 0.002029s"

"Elapsed time: 0.001925s"

"Elapsed time: 0.004017s"

[noctuid]

The time it takes to define a function is so small I don't this even defining hundreds will have a significant impact on initialization, but I use these autoloads anyway.

Well if you are making hundreds, it does add up. For the most part it's probably a matter of preference whether to define them right away or later. defautoload! seems like an interesting way to still use with-eval-after-load but without the nesting.

Also on as a side issue, you might want to checkout what I did with the macro elisp-block! in my init.el.

I saw your comment on reddit. I like the idea and will probably add something similar to general.el.

[Luis-Henriquez-Perez]

I updated defautoload. I'll post it here later today in case you were going to do something similar.

I saw your comment on reddit. I like the idea and will probably add something similar to general.el.

I'm glad you liked it. I saw your reply. I'll probably post a follow-up with a few more ideas, once I flesh them out a bit.

The most interesting is exploiting the elisp-block! to create namespaces. It would be similar to the names package but it would be much more accurate. I was thinking along the lines of declaring a a namespace like this:

This is for the s package.

(define-namespace 's- trim trim-left trim-right ...)

And then elisp-block! could search through body and replace all cases of NAMESPACED-SYMBOL with ORIGINAL-SYMBOL. To get the implementation right is tricky though. Because you need to deal with cases like being able to use helpful/describe-variable on namespaced symbols symbols. As well as proper syntax highlighting.