ORGANIZING CODE

Packages are defined with defpackage.

(defpackage #:my-package
  (:use #:cl))

my-package inherits or "uses" all of the external symbols of cl, a nickname for the common-lisp package, which contains all of the standard Common Lisp symbols.

(describe 'cl:defun)

ResultCOMMON-LISP:DEFUN
  [symbol]
DEFUN names a macro:
  Lambda-list: (NAME LAMBDA-LIST &BODY BODY)
  Documentation:
    Define a function at top level.
  Source file: SYS:SRC;CODE;MACROS.LISP
 => ; No value

If you don't include this line, you will need to add a package-qualifier to all of the symbols built into Common Lisp. If you evaluate this code, you'll get a strange error:

(defpackage #:messy-package)
(in-package #:messy-package)

(defun hello-world ()
  (print "Hello world"))

ResultThe variable HELLO-WORLD is unbound.
   [Condition of type COMMON-LISP:UNBOUND-VARIABLE]

Why? Because the defun special operator is actually a macro whose first argument is a name for the function you are defining. Because it's a macro, the order of operations is to first expand the macro and then execute the code. During the macro expansion, the hello-world symbol will be interned and its function name value will be bound to hello-world.

However, because we didn't :use #:cl, defun is interpreted as the name of a function, which requires first that all inner forms be evaluated and their values returned. And since hello-world isn't bound to any value yet, we get the above error.

All that is to say, unless you're a Lisp wizard concocting a sexp brew, you should be adding (:use #:cl) to all of your package definitions.

Notice also the call to in-package. Forms following a call to in-package will have access to, and be accessible in, that package. That's because symbols are "interned" in packages If you have a defpackage but no in-package, then none of your symbols will be interned in that package, and none of the forms will have access to the symbols in that package (without a package-qualifier prefix).

The most common and intuitive way they are interned is when they are defined as the name of a function, a variable, etc.

Which package a symbol is interned to depends on a couple of factors. If you are using the REPL and you execute (defparameter *name* 'micah) in the REPL, the symbol *name* will be interned into whichever package the REPL is synced to. For example:

CL-USER> (defparameter *name* 'micah)

Here, *name* is interned into the default CL-USER package.

If you are in a file buffer and you sly-eval or sly-compile the defparameter form, *name* will be interned into the package named in in-package.

(defpackage #:my-package
  (:use #:cl))
(in-package #:my-package) 

To use symbols from one package in another, all you need to do is:

1. Make sure both packages are compiled/loaded.
2. Add a package-qualifer to the beginning of the symbol.

Here, for example, the package-qualifier for *global* is my-package:.

(defpackage #:my-package
  (:use #:cl)
  (:export #:*global*))

(in-package #:my-package)

(defparameter *global* "This is a global variable interned in MY-PACKAGE.")

(defpackage #:your-package
  (:use #:cl))
(in-package #:your-package)

(print my-package:*global*)

ResultThis is a global variable interned in MY-PACKAGE.

(symbol-package 'my-package:*global*)

Result#<PACKAGE "MY-PACKAGE">

We can add our own *global* in your-package:

(defparameter *global* "This is a global variable interned in YOUR-PACKAGE.")

And then call both globals together:

(list my-package:*global* *global*)

Result("This is a global variable interned in MY-PACKAGE." "Global variable interned in YOUR-PACKAGE.")

You can modify the values of package-qualified variables:

(let ((my-package:*global* "Symbol from MY-PACKAGE lexically rebound in YOUR-PACKAGE"))
  my-package:*global*)

ResultSymbol from MY-PACKAGE lexically rebound in YOUR-PACKAGE

You may also want to import the symbol, allowing you to use it without a package-name qualifier.

(defpackage #:your-package
  (:use #:cl)
  (:import-from #:my-package #:*global*))

If you start this way, then you won't need a package name qualifier when you use it.

This is not the case if you use my-package. Change the definition of your-package. If you compile this form you will get an error:

(defpackage #:your-package
  (:use #:cl #:my-package))

ResultUSE-PACKAGE (MY-PACKAGE:*GLOBAL*) causes name-conflicts in
#<PACKAGE "YOUR-PACKAGE"> between the following symbols:
  YOUR-PACKAGE::*GLOBAL*, MY-PACKAGE:*GLOBAL*
   [Condition of type SB-EXT:NAME-CONFLICT]
Restarts:
 0: [KEEP-OLD] Keep YOUR-PACKAGE::*GLOBAL* accessible in YOUR-PACKAGE (shadowing MY-PACKAGE:*GLOBAL*).
 1: [TAKE-NEW] Make MY-PACKAGE:*GLOBAL* accessible in YOUR-PACKAGE (uninterning YOUR-PACKAGE::*GLOBAL*).
 2: [RESOLVE-CONFLICT] Resolve conflict.
 3: [ABORT] Abort compilation.
 4: [*ABORT] Return to SLY's top level.
 5: [ABORT] abort thread (#<THREAD tid=20391 "slynk-worker" RUNNING {7005097833}>)

Now you'll have several different restarts to choose from, including TAKE-NEW, KEEP-OLD, etc. and a bit of a mess to clean up. My recommendation is to only use :use sparingly in situations you know you aren't going to have conflicts later (like in your testing packages).

If you choose KEEP-OLD, you will shadow the my-package:*global* with your-package:*global*. Shadowing means that the imported symbol is hidden, allowing the same symbol in the current package.

My experience has been that doing anything short of aborting, unintern ing the symbol or unuse-package ing the package I'm using leads to frustration. You're better off not using :use in the first place.

Even if you do as I recommend you might still have this name conflict problem. Consider this situation:

(defpackage #:my-package
  (:use #:cl)
  (:export #:my-function))
(in-package #:my-package)

(defun my-function () (print "my-function"))

(defpackage #:my-other-package
  (:use #:cl)
  (:import-from #:my-package))

(in-package #:my-other-package)

(defun my-other-function ()
  (my-package:my-function))

Right now, we have a couple of problems.

1. I forgot to export the symbol from my-package.
2. I don't have a package-qualifier attached to my-function when I try to use it.

When I try to compile my-other-function, I'll get a style-warning about an undefined function. At this point, there are two options:

1. Add a package name qualifier, as in (my-package:my-function); or
2. Add the function symbol in the imports as below

If you add the package name qualifier, you'll get an error:

(defpackage #:my-other-package
  (:use #:cl)
  (:import-from #:my-package
                #:my-function))
(in-package #:my-other-package)

Result,*Org Src essentials.org[ lisp ]*:14:25:
  read-error: 
    READ error during COMPILE-FILE:
      The symbol "MY-FUNCTION" is not external in the MY-PACKAGE package.
        Line: 2, Column: 25, File-Position: 52
        Stream: #<SB-INT:FORM-TRACKING-STREAM for "file /var/tmp/slime1YpOBd"
 {7008A37DA3}>
Compilation failed.

Oh bother, forgot to export it.

(defpackage #:my-package
  (:use #:cl)
  (:export #:my-function))

Recompile and continue.

If you fix the problem while it is just a style-warning, you'll be okay. But if you happen to execute code that uses my-function, then you'll get an error message:

The function MY-OTHER-PACKAGE::MY-FUNCTION is undefined.
   [Condition of type UNDEFINED-FUNCTION]
Restarts:
 0: [CONTINUE] Retry calling MY-FUNCTION.
 1: [USE-VALUE] Call specified function.
 2: [RETURN-VALUE] Return specified values.
 3: [RETURN-NOTHING] Return zero values.
 4: [*ABORT] Return to SLY's top level.
 5: [ABORT] abort thread (#<THREAD tid=15507 "slynk-worker" RUNNING
 {7005A4AA33}>)

Whoops, now let's add it to imports and recompile.

IMPORT (MY-PACKAGE::MY-FUNCTION) causes name-conflicts in
#<PACKAGE "MY-OTHER-PACKAGE"> between the following symbols:
  MY-OTHER-PACKAGE::MY-FUNCTION, MY-PACKAGE::MY-FUNCTION
   [Condition of type SB-EXT:NAME-CONFLICT]
Restarts:
 0: [SHADOWING-IMPORT-IT] Shadowing-import MY-PACKAGE::MY-FUNCTION, uninterning MY-OTHER-PACKAGE::MY-FUNCTION.
 1: [DONT-IMPORT-IT] Don't import MY-PACKAGE::MY-FUNCTION, keeping MY-OTHER-PACKAGE::MY-FUNCTION.
 2: [RESOLVE-CONFLICT] Resolve conflict.
 3: [ABORT] Abort compilation.
 4: [*ABORT] Return to SLY's top level.
 5: [ABORT] abort thread (#<THREAD tid=20827 "slynk-worker" RUNNING {700670E963}>)

If you choose SHADOWING-IMPORT-IT and try to recompile your code again, you'll get a similar error:

,*Org Src essentials.org[ lisp ]*:7:1:
  warning: 
    MY-OTHER-PACKAGE also shadows the following symbols:
      (MY-PACKAGE::MY-FUNCTION)
    ==>
      (SB-IMPL::%DEFPACKAGE "MY-OTHER-PACKAGE" 'NIL 'NIL 'NIL 'NIL '("CL")
                            '(("MY-PACKAGE" "MY-FUNCTION")) 'NIL 'NIL
                            '("MY-OTHER-PACKAGE") 'NIL ...)
Compilation failed.

If all this has your head spinning now, you're not alone.

To reiterate, you have two options:

Add a package name qualifier:

(defpackage #:my-package
  (:use #:cl)
  (:export #:my-function))              ; <- Don't forget to export.
(in-package #:my-package)

(defun my-function () (print "my-function"))

(defpackage #:my-other-package
  (:use #:cl)
  (:import-from #:my-package))

(in-package #:my-other-package)

(defun my-other-function ()
  (my-package:my-function))             ; <- Added package name qualifier

(my-other-function)

Or add my-function to your imports:

(defpackage #:my-package
  (:use #:cl)
  (:export #:my-function))
(in-package #:my-package)

(defun my-function () (print "my-function"))

(defpackage #:my-other-package
  (:use #:cl)
  (:import-from #:my-package
                #:my-function)) ; <- Added import

(in-package #:my-other-package)

(defun my-other-function ()
  (my-function))

Then while in my-other-package, unintern my-function.

It'll take some getting used to, but eventually you'll know how to clean up any messes you make.

It's possible for packages to define nicknames. Nicknames are useful for calling individual symbols from packages that have long names.

(defpackage #:a-package-with-a-long/and-annoying/name
  (:use #:cl)
  (:nicknames #:smol-name)
  (:export #:smol-hello))

(in-package #:a-package-with-a-long/and-annoying/name)

(defun smol-hello ()
  :hi)

(defpackage #:my-package
  (:use #:cl)
  (:import-from #:some-package))

(in-package #:my-package)

Now we can call the function using the nickname:

(smol-name:smol-hello)

Result:HI

If a package doesn't define its own nickname, you can define a nickname local to your package:

(defpackage #:a-package-with-a-long/and-annoying/name
  (:use #:cl)
  (:export #:smol-hello))

(in-package #:a-package-with-a-long/and-annoying/name)

(defun smol-hello ()
  :hi)

(defpackage #:my-package
  (:use #:cl)
  (:local-nicknames (#:annoying #:a-package-with-a-long/and-annoying/name)))

(in-package #:my-package)

The local nickname works the same way:

(in-package #:my-package)
(annoying:smol-hello)

Result:HI

Unfortunately, you can't define nicknames for individual symbols.

(defpackage #:a-package-with-a-long/and-annoying/name
  (:use #:cl)
  (:export
   #:a-very-long-and-annoying-function-or-macro-name #:smol-hello))

(in-package #:a-package-with-a-long/and-annoying/name)

(defun smol-hello () :hi)

(defun a-very-long-and-annoying-function-or-macro-name () :oof)

(defpackage #:my-package (:use #:cl) (:local-nicknames (#:annoying
  #:a-package-with-a-long/and-annoying/name)))

(in-package #:my-package)

No shortcut for long symbol names:

;; Deal with it.
(annoying:a-very-long-and-annoying-function-or-macro-name)

Result:OOF

The good news is that symbols with long names like that aren't usually in public APIs.

UIOP is a cross-implementation compatibility library that comes with ASDF (more on that in a second). Its define-package macro provides some extra capabilities that the standard defpackage doesn't. We'll see them in use later when discussing different factoring styles.

ASDF is the defacto-standard library (available out of the box with SBCL and other implementations) that adds convenient code loading orchestration capabilities to Lisp. You do so by defining a defsystem in a .asd file in your project's root directory and then loading the system. When you load the system, ASDF will load your other files depending on your system definition.

Because defsystem is a third-party extension to Lisp and was not previously available, there are different styles of organizing packages and systems in older projects that are less common in more modern projects. We'll be taking a look at different styles of factoring later, but for now let's focus on defsystem alone.

Let's say we have a project that looks like this:

% tree
.
├── almighty-system.asd
└── src
    └── core.lisp

This is our defsystem:

(defsystem "almighty-system"
  :author "Micah Killian"
  :version "0.0.1"
  :description "A system for demonstrating how to define systems."
  :depends-on ("local-time")
  :components (:module "src"
               :pathname "src"
               (:file "core")))

If we load this system (more on that later), ASDF will look for a local copy of the local-time library, downloading it with Quicklisp (more on that later) if a local copy doesn't exist. After loading the defsystem defined by local-time, ASDF will read the core.lisp file in the src directory and load it into the Lisp image.

Loading a system requires first that the .asd file is loaded.

(asdf:load-asd (merge-pathnames *default-pathname-defaults* "almighty-system.asd"))

This requires some explanation. asdf:load-system takes a pathname to the .asd file. If you have the absolute path available, you can use that:

(asdf:load-asd "

/Users/micah/lisp/almighty/content/essentials/code/projects/almighty-system/almighty-system.asd")

Or, in the REPL, you can type , (sly-mrepl-shortcut) and select set-directory and ensure that the current working directory is set to the project root (where the .asd file lives), and then run the other line above. set-directory will set the value of *default-pathname-defaults*, and merge-pathnames does what it says. Thus, it's equivalent to the following:

(asdf:load-asd (merge-pathnames "
/Users/micah/lisp/almighty/content/essentials/code/projects/almighty-system/" "
almighty-system.asd"))

After you load the asd file, you can load the system with , and load system or (asdf:load-system "almighty-system") in the REPL.

A good representation of the Mother Of All package strategy of Common Lisp code organization is hunchentoot–a popular Lisp web server and web dev framework. If we take a look at the file tree it looks like this:

% tree
.
├── acceptor.lisp
├── CHANGELOG
├── CHANGELOGTBNL
├── compat.lisp
├── conditions.lisp
├── cookie.lisp
├── docs
│   ├── hunchentoot.gif
│   ├── index.html
│   └── LICENSE.txt
├── easy-handlers.lisp
├── headers.lisp
├── hunchentoot.asd
├── lispworks.lisp
├── log.lisp
├── make-docstrings.lisp
├── mime-types.lisp
├── misc.lisp
├── packages.lisp
├── README.md
├── release-checklist.txt
├── reply.lisp
├── request.lisp
├── run-test.lisp
├── session.lisp
├── set-timeouts.lisp
├── specials.lisp
├── ssl.lisp
├── taskmaster.lisp
├── test
│   ├── ca-built-via-xca.xdb
│   ├── ca.crt
│   ├── client.crt
│   ├── favicon.ico
│   ├── fz.jpg
│   ├── packages.lisp
│   ├── script-engine.lisp
│   ├── script.lisp
│   ├── server.crt
│   ├── server+ca.crt
│   ├── test-handlers.lisp
│   ├── test-key-no-password.key
│   └── UTF-8-demo.html
├── url-rewrite
│   ├── packages.lisp
│   ├── primitives.lisp
│   ├── specials.lisp
│   ├── url-rewrite.lisp
│   └── util.lisp
└── util.lisp

The organization is flat, lots of separate files, mostly in the project root directory.

Hunchentoot has one package definition (well, two), found in the packages.lisp file.

(in-package :cl-user)

(defpackage #:hunchentoot
  (:nicknames #:tbnl)
  (:use :cl :cl-ppcre :chunga :flexi-streams :url-rewrite :alexandria)
  (:shadow #:defconstant
           #:url-encode)
  (:export #:*acceptor*
           #:*catch-errors-p*
           #+:lispworks
           #:*cleanup-function*
           #+:lispworks
           #:*cleanup-interval*
           #:*content-types-for-url-rewrite*
           #:*default-connection-timeout*
           #:*default-content-type*
           #:*dispatch-table*
           #:*file-upload-hook*
                                        ; ... and many more
           ))                           

:nicknames defines what other package name qualifiers you can use besides hunchentoot. That means that Hunchentoot itself provides you the ability to use (tbnl:*acceptor*) if you want. Useful for allowing users to use a smaller package name qualifier on their symbols.

Hunchentoot uses :use for several utility libraries. Again, I don't recommend it, but it is not uncommon.

:shadow here creates an symbol or symbols that are initially unbound in the package. Clearly, there was a naming conflict somewhere between Common Lisp's built-in defconstant and the url-encode symbol intended to have two different values (utils.lisp and url-rewrite/url-rewrite.lisp both define url-encode functions). What this means is that in the hunchentoot package, if you write url-encode while you are in the hunchentoot package, it has a different value than if you are in the url-rewrite package.

It's a wonky way of preventing a naming conflict error because the hunchentoot package uses :url-rewrite instead of importing it. However, you might sometimes need to use :shadow to get around naming conflicts caused by other systems/libraries that use Common Lisp symbol names, so it's something to keep in mind.

The important point is this: All of the files in the project root begin with (in-package :hunchentoot). Each file contributes their defined symbols to the :hunchentoot package.

If all this package talk has the ol' nogging working overtime, you might be thinking, "Hey Micah, doesn't that assume that the packages.lisp file is loaded before all the others?"

Yes, yes it does. Let's take a look at the hunchentoot.asd file:

(defsystem :hunchentoot
  :serial t
  :version "1.3.1"
  :description "Hunchentoot is a HTTP server based on USOCKET and
  BORDEAUX-THREADS. It supports HTTP 1.1, serves static files, has a simple
  framework for user-defined handlers and can be extended through subclassing."
  :license "BSD-2-Clause"
  :depends-on (:chunga
               :cl-base64
               :cl-fad
               :cl-ppcre
               :flexi-streams
               (:feature (:not (:or :lispworks :hunchentoot-no-ssl))
                         :cl+ssl)
               :md5
               :alexandria
               :rfc2388
               :trivial-backtrace
               (:feature (:not :lispworks) :usocket)
               (:feature (:not :lispworks) :bordeaux-threads))
  :components ((:module "url-rewrite"
                :serial t
                :components ((:file "packages")
                             (:file "specials")
                             (:file "primitives")
                             (:file "util")
                             (:file "url-rewrite")))
               (:file "packages")
               (:file "lispworks" :if-feature :lispworks)
               (:file "compat" :if-feature (:not :lispworks))
               (:file "specials")
               (:file "conditions")
               (:file "mime-types")
               (:file "util")
               (:file "log")
               (:file "cookie")
               (:file "reply")
               (:file "request")
               (:file "session")
               (:file "misc")
               (:file "headers")
               (:file "set-timeouts")
               (:file "taskmaster")
               (:file "acceptor")
               (:file "ssl" :if-feature (:not :hunchentoot-no-ssl))
               (:file "easy-handlers"))
  :perform (test-op (o c) (load (merge-pathnames "run-test.lisp" (system-source-directory
 c)))))

Notice :serial t up at the top. That means that all of the :components will be loaded in the order they are listed. So, after all of the :depends-on dependency systems are loaded, the url-rewrite module will be loaded. Its contents will also be loaded in order, and so first the url-rewrite/packages.lisp file will be loaded, then specials.lisp, primitives.lisp, etc. After that module is loaded, the first file in the project root that will be loaded is its packages.lisp file that we look at before. The Mother Of All packages.lisp is where all the symbols for the majority of the project live and are namespaced.

There are only two other systems in the Hunchentoot project: one for development and the other for testing. They don't provide any further insight into this strategy, so we will skip them.

In summary, the Mother Of All Package Strategy is this: you have one package (defpackage :hunchentoot) and one system (defsystem :hunchentoot), and that one package :uses or imports from several other packages. Other code files begin with (in-package :hunchentoot), giving them access to the symbols defined in other files and contributing the symbols they define to that package. The packages.lisp file's exports define the public interface for users. This style is "retro", but still perfectly usable especially on smaller projects, and even larger projects like Hunchentoot. The vend library is a modern Lisp library that use this Mother of All Package Strategy.

It's also possible that a single project might have several systems, primarily used as optional extensions to the library's core functionality. transducers is one such library, as is mito. However, transducers still uses just one package for the core functionality. It's defsystem definition also uses the serial t approach of loading source files in order. This Multi-System Strategy is modular, yet does not exclude the possibility of using the Mother Of All Package Strategy.

mito, on the other hand, in addition to having multiple systems–one for core functionality and others for auxiliary functionality–also uses the One Package Per File Strategy. This is a more "modern" approach and considered by some to be the preferred strategy generally.

Let's take a look at the mito system:

(defsystem "mito"
  :version "0.2.0"
  :author "Eitaro Fukamachi"
  :license "BSD 3-Clause"
  :depends-on ("mito-core"
               "mito-migration"
               "lack-middleware-mito"
               (:feature :sb-package-locks "cl-package-locks"))
  :components ((:file "src/mito"))
  :description "Abstraction layer for DB schema"
  :in-order-to ((test-op (test-op "mito-test"))))

Here, we notice that the :depends-on list is three other subsystems in the mito project. Let's look at mito-core.asd:

(defsystem "mito-core"
  :version "0.2.0"
  :author "Eitaro Fukamachi"
  :license "BSD 3-Clause"
  :depends-on ((:version "dbi" "0.11.1")
               "sxql"
               "cl-ppcre"
               "closer-mop"
               "dissect"
               "trivia"
               "local-time"
               "uuid"
               "alexandria")
  :components ((:file "src/core" :depends-on ("core-components"))
               (:module "core-components"
                :pathname "src/core"
                :components
                ((:file "dao" :depends-on ("dao-components"))
                 (:module "dao-components"
                  :pathname "dao"
                  :depends-on ("connection" "class" "db" "conversion" "logger" "
util")
                  :components
                  ((:file "table" :depends-on ("column" "mixin" "view"))
                   (:file "view" :depends-on ("column"))
                   (:file "mixin" :depends-on ("column"))
                   (:file "column")))
                 (:file "class" :depends-on ("class-components"))
                 (:module "class-components"
                  :pathname "class"
                  :depends-on ("error" "util")
                  :components
                  ((:file "table" :depends-on ("column"))
                   (:file "column")))
                 (:file "connection" :depends-on ("error"))
                 (:file "type" :depends-on ("db"))
                 (:file "db" :depends-on ("db-drivers" "connection" "class" "
util"))
                 (:module "db-drivers"
                  :pathname "db"
                  :depends-on ("logger" "util")
                  :components
                  ((:file "mysql")
                   (:file "postgres")
                   (:file "sqlite3")))
                 (:file "conversion")
                 (:file "logger")
                 (:file "error")
                 (:file "util")))))

Now we see some third-party systems in the :depends-on list. What's more interesting here is the definition in :components: Many of the files use :depends-on as well, pointing to other files within the system. We don't see :serial t here, either. Instead of a linear dependency tree, we have a graph. In the hunchentoot system, we saw that the more minor, prerequisite files like packages.lisp came first. Here in the mito-core system, the major files are listed first, with their prerequisites coming later in the list. The order is inverted.

Let's take a look inside src/mito.lisp (the only component in the mito.asd file above):

(uiop:define-package #:mito
  (:use #:cl)
  (:use-reexport #:mito.core
                 #:mito.migration))
(in-package #:mito)

Instead of defpackage, Mito uses UIOP's define-package macro. It uses it because it provides a new option, :use-reexport. It's like :use, but with two important differences:

1. It will import all of the exported symbols of the packages that are passed as arguments, and then reexport them.
2. If there is a naming conflict between symbols in the packages passed in the form, the first one takes precedence, shadowing versions that come later. That means that if there is a mito.core:cool-symbol and a mito.migration:cool-symbol, because mito.core comes first in the above definition, the mito package with inherit mito.core:cool-symbol and then that version will be reexported.

In systems like Mito, the reexport options (:reexport, :use-reexport, and :mix-reexport) are used to help easily "bubble up" exports from peripheral packages into one final public API.

However, although Mito bubbles up dependencies with :depends-on, using the One Package Per File Strategy doesn't preclude using :serial t, either.

There is one other strategy worth noting. It's perhaps best represented by the utopian web framework. Let's take a look at its system definition:

(defsystem "utopian"
  :class :package-inferred-system
  :version "0.9.1"
  :author "Eitaro Fukamachi"
  :license "LLGPL"
  :description "Web application framework"
  :pathname "src"
  :depends-on ("utopian/main")
  :in-order-to ((test-op (test-op "utopian-tests"))))

(register-system-packages "lack-component" '(#:lack.component))
(register-system-packages "lack-request" '(#:lack.request))
(register-system-packages "lack-response" '(#:lack.response))
(register-system-packages "mystic" '(#:mystic.util))
(register-system-packages "mystic-file-mixin" '(#:mystic.template.file))

The first thing you'll notice is that the definition is short. The dependencies only include…another file in the system? And there are no :components! To investigate, let's look in the "utopian/main"… file? Package? System?

It's found in src/main.lisp:

(uiop:define-package #:utopian
  (:nicknames #:utopian/main)
  (:use #:cl)
  (:mix-reexport #:utopian/routes
                 #:utopian/views
                 #:utopian/context
                 #:utopian/app
                 #:utopian/config
                 #:utopian/exceptions))
(in-package #:utopian)

We already saw :nicknames in the Hunchentoot packages.lisp file. :mix-reexport is like :use-reexport that we saw in Mito, except that it will inherit all symbols, exported and not, from the packages that follow into the current package. Every single symbol from those six packages above are going to be inherited into the #:utopian package.

But now you may be wondering, "How do they get loaded if the system file doesn't list them in :components?"

That's not all, friends. The mystery deepens when we look in the utopian/routes package file:

(defpackage #:utopian/routes
  (:use #:cl)
  (:import-from #:utopian/context
                #:*request*
                #:*response*)
  (:import-from #:utopian/file-loader
                #:intern-rule)
  (:import-from #:myway
                #:make-mapper
                #:connect
                #:next-route)
  (:import-from #:lack.request
                #:request-parameters)
  (:export #:defroutes
           #:routes
           #:routes-mapper
           #:routes-controllers-directory
           #:route

           ;; from MyWay
           #:next-route))
;; ...

Wait, there is a third-party library--myway–imported in this package! That wasn't in the system definition!

Actually, it is. Look again. See this line?

:class :package-inferred-system

That one line drastically changes how packages and systems work.

All packages in a :package-inferred-system are treated as entire systems themselves.

Package inferred systems will all have an entry-point into the system, like :depends-on ("utopian/main"). Remember: up until now, dependencies were systems. We were confused because :utopian is a package, not a system. But because of :package-inferred-system, :utopian and all of its dependent packages are treated like their own systems by ASDF! That's why the files for the rest of the project aren't listed in :depends-on: utopian/main is treated like a system, where its prerequisite packages are as well. ASDF will do the work of resolving the dependencies, hence why you don't need to list them manually. It will load prerequisite packages/systems both in the utopian project, including third-party libraries, downloading them if necessary.

Package inferred systems are the Package Per File taken to its limit: System Per File.

Using package inferred systems requires following a few rules which I'll demonstrate.

(defsystem "almighty"
  :class :package-inferred-system
  :author "Micah Killian"
  :version "0.0.1"
  :description "A system for demonstrating how to define systems."
  :pathname "src"
  :depends-on ("almighty/main"))

Notice here that the system is named almighty, and that the package/system in :depends-on begins with almighty. The entry-point and all subsequent packages in your package-inferred-system must begin with almighty.

Notice also that :pathname is set. The tree view of this project looks like this:

.
├── almighty.asd
└── src
    ├── config.lisp
    ├── db.lisp
    ├── main.lisp
    └── models
        └── person.lisp

And the files look like this:

;; src/main.lisp
(uiop:define-package #:almighty
  (:use #:cl)
  (:nicknames #:almighty/main)
  (:use-reexport #:almighty/config
                 #:almighty/db
                 #:almighty/models/person))
(in-package #:almighty)

;; src/db.lisp
(defpackage #:almighty/db
  (:use #:cl)
  (:export #:hello-from-db))
(in-package #:almighty/db)

(defun hello-from-db ()
  :hello-from-deebee)

;; src/config.lisp
(defpackage #:almighty/config
  (:use #:cl)
  (:export #:hello-from-config))
(in-package #:almighty/config)

(defun hello-from-config ()
  :hello-from-config)

;; src/models/person.lisp
(defpackage #:almighty/models/person
  (:use #:cl)
  (:export
   #:hello-person))
(in-package #:almighty/models/person)

(defun hello-person ()
  :i-am-a-person)

In a PIS, The package name of a system begins with the system name, almighty. The part that follows is the file name, such as almighty/db for src/db.lisp. If a package's file is in a subdirectory, the name of the directory comes before the file name, as in almighty/models/person for src/models/person.lisp.

If you don't follow these rules, ASDF won't be able to find the files. For example, if you have a file src/wrong-name.lisp that defines a package almighty/correct-name and try to :import-from, :use-reexport, etc. the package almighty/correct-name in another package like almighty/db, you will receive an error stating:

Unknown location:
  error: 
    Component "almighty/correct-name" not found, required by
    #<PACKAGE-INFERRED-SYSTEM "almighty/db">
Compilation failed.

With the exceptions of the different system definition and file and package naming conventions imposed by them, PIS are the same as the One Package Per File Strategy above.

The architect of ASDF wrote a document precisely to answer this question. Unfortunately, it leaves more questions than answers. First, it explains problems with the other strategies we've covered:

When you start writing large enough systems, putting everything in one big package leads to a big mess: it's hard to find what function is defined where, or should be defined where; you invent your own symbol prefixing system to avoid name clashes; totally unrelated things end up in the same mother-of-all package; you divide your mother-of-all package into a few subpackages, but as the software keeps growing each of these packages in turn becomes too big.

Meanwhile, as you grow large enough libraries, you find that you loading a big library just to use a small part of it becomes a big hassle, leading to code bloat, too much recompilation, too much re-testing, and not enough understanding of what's going on.

So the problems with the Mother Of All Package Strategy are:

1. Naming conflicts.
2. Hard to understand.
3. A hassle to load and recompile.

We saw with Hunchentoot that "shadowing" of symbols was a little hard to understand, yet appears to be the solution to naming conflicts in a project factored as it is. If you have files using symbol names found in other files in project, it can be hard to understand which symbol's value is the actual value used in the overall package. Thus, you namespace using some sort of naming convention.

Additionally, you might try dividing into some sub-packages, but it's somewhat arbitrary, meaning developers have to come up with their own coding conventions regarding package management. Whether you use one package or multiple, if your system is large enough, those strategies will ultimately cause trouble.

After explaining how Package Inferred Systems work, he explains how they solve the above problems:

This allows for large modular libraries, wherein you may use one file, and only that file and its transitive dependencies will be loaded, rather than the entire humongous library.

This also helps you enforce a discipline wherein it is always clear in which file each symbol is defined, which files have symbols used by any other file, etc.

In a discussion I had with the author on X, our friend François-René Rideau (or fare on Github), I asked about the rationale behind PIS, why it is the best practice, etc. He said that the motivation to create PIS was to help large teams, but he also said that PIS can be useful for individuals with large or ambitious projects. He said that large .asd files (remember Mito's system?) tend to be error-prone. Further, he says,

[In large .asd files] you often leave a lot of obsolete dependencies that slow the build and the understanding of the code by newcomers.

So large system definitions are scary to mess with, making it safer and therefore more likely for teams to leave unused dependencies in the system. That can make understanding the code more difficult, and make builds unnecessarily slower.

While ultimately he didn't make a conclusive recommendation to use PIS, he appears to prefer using PIS from the beginning to improve modularity and understandability of code, and to make coordination with other developers easier.

So the argument in favor of Package Inferred Systems boils down to this:

1. It helps reduce problems of coordination with other developers.
2. It helps make the code more understandable.
3. It helps reduce the amount of unnecessary code that remains in the project.

The question is: Does it really do all those things? And are PIS more effective than the alternatives?

Understanding packages and systems is one of the more challenging things about learning Common Lisp for newcomers. That's why I've gone into more detail here on the subject than I have with other language features.

Before I make my counter-argument to PIS, I must emphasize that I have no professional experience with Common Lisp, nor have I written any large projects with it. Fare, on the other hand, is an accomplished Lisper with experience writing Lisp in large teams. The odds that I am not understanding the benefits of PIS because of a lack of experience are astronomically high. I make my counter-arguments with fear and trepidation in my heart. All I have to guide me are my small brain and an intuition–as limited as it may be–about how simple programs and programming should be.

My experience of package inferred systems is that they actually led to more confusion, not less. Perhaps you felt it too as we took our journey through these different strategies.

Hunchentoot listed all third-party dependencies in the system definition–thus you had a sense of what code you needed to download, and what your potential exposure to bugs or security problems was. The defsystem used the :serial t option to let you, the newcomer, know that all of the :component files were loaded in the order they were listed. The Mother Of All packages.lisp gave us an easy to read overview of the public API. Without any sophisticated knowledge or tools, the code was understandable (with perhaps the exception of the use of :shadow for a couple of symbols).

Was it always that way? How many challenges did the project have with coordination or understandability before it reached the point where we saw it? How much unseen discipline was practiced in the naming and organization of symbols in the project? For example, perhaps it was difficult to organize and line up the files in the defsystem definition so that all of the right code was loaded in the correct order? Is the project bloated? We see only a snapshot of the project as it is now. We can't know the challenges of coordination, dependency management, and code factoring that had to be overcome just from looking at the source.

However, when we look at the source, the design of the system is undeniably simple and easy to understand, even for a dummy newcomer like me.

The same is true of the vend library, which uses a similar strategy.

mito, on the other hand, has a somewhat more difficult to follow defsystem: the liberal use of :depends-on and the inversion of the dependency tree makes the system definition much more difficult to follow. The system has several subsystems, and some of the packages use options like :use-reexport and so-on. You need to have a deeper understanding of how systems work, how shadowing works, and how those package options will effect the final shape of the system. While Hunchentoot was fairly transparent, Mito's system architecture introduced a fair bit of noisy abstractions that make reasoning about the system difficult at a glance.

But perhaps with experience I would see the value in the abstractions? Perhaps the alternative would have been noisier, more difficult to extend with other developers, and more bloated?

And then there's utopian and :package-inferred-system. The .asd file is small, but that just left us more questions than answers. Chief among them is: how many dependencies does it have, and where do they come from? What does the final public API actually look like after all subsystem symbols are bubbled up to the top system? What code is loaded (and potentially executed), and when?

About API discoverability, Fare said the following in our discussion on X:

The SLIME REPL, not the unaided source code, is how you discover the code. Plus the generated documentation.

This is an important point: Great tools can make a developer's job easier. That's why one of the goals of this book is to help you understand all of the tools available to you in Emacs. If you code Common Lisp in Emacs without knowing about window management, REPL shortcuts, SLY debugger restarts/keybindings, evil-mode VIM keybindings, etc. you are missing out on a great number of very powerful tools for efficient and pleasant coding.

However, I disagree with Fare about how code is discovered. Yes, we can type SPC m g d or g d to use sly-edit-definition to go to the definition of some symbol. Yes, we can look at the documentation. But none of those provide a clear overview of third-party dependencies of the project nor a clear view of the shape of the system as a whole. Tools like PIS blur our vision of the whole system. When I look at projects that use those tools, every bone in my body screams, "Complexity spirit demon."

Unfortunately, I am not the only one that feels this way. Do a search for "package-inferred-system" on X and you'll find a lot of confusion or even distaste for it.

Still, there are some notable people that really enjoy it. The most well-known users are probably the Eitaro Fukamachi (https://github.com/fukamachi) (author of mito and many other libraries) and Alexander Artemenko (https://github.com/svetlyak40wt)–the man behind 40ants (https://github.com/40ants) and Ultralisp–both of whom are prolific giants in the Common Lisp community (I think I'm starting to sweat here). Alexander has explained why he likes PIS on X:

For me this more important consequence is that I have not to figure out dependencies between files anymore and to hardcode them in the *.asd files. System definition can be just: (defsystem "foo" :class :package-inferred-system :depends-on ("foo/main"))

This echos what Fare says about system understandability and ease of code factoring. So maybe there is something to the arguments about the challenges of factoring that standard defsystem definitions impose.

My brain may not be big enough to understand the ease-of-use and ease-of-understanding arguments. There's a good chance it's a skill issue.

However, there is one feature of PIS that unfortunately I can't forgive, no matter how small my brain.

The fact that third-party dependencies are not defined in the .asd file, but instead are spread across the codebase, is simply not acceptable. As an Almighty Lisp programmer, dependency management needs to be crystal clear both for understanding and for containment. Package inferred systems simply make it too easy to introduce dependencies in unknown places. My goal is the keep my use of dependencies limited, decreasing my dependencies over time. If it's easy to add and forget about them, then the temptation to increase my dependencies will be too great.

The community doesn't appear to view PIS as the way forward. Community adoption of PIS is very low. Package inferred systems have been around since 2014, more than 10 years ago. At present, while there are nearly 5000 systems available on the Quicklisp repository, only around 60 unique systems use :package-inferred-system.

LEM–an Emacs clone written from scratch in Common Lisp–and Coalton–a new Lisp language implementing a Haskell-like type system–are among the largest and most ambitious publicly available projects. Neither use PIS. Both use the :serial t option for the :components, manually listing dependencies and files to load from the project. Both make heavy use of the One Package Per File Strategy, but LEM also uses a hybrid Mother Of All Package. Coalton provides a public API in a package.lisp file! In fact, Coalton uses a strategy similar to Fare's LIL library–providing a package.lisp file for smaller "modules" in the system (modules by convention) and reexporting the symbols from the module files, allowing the system to bubble up symbols to the top level packages.

So to answer the question, "Is it essential to learn and use package inferred systems?", the answer is No. For small projects, a Mother Of All package can work fine, but the most common approach on modern projects by far is One Package Per File with some strategy for bubbling up and potentially listing exported symbols into a final API.