Make the LMFDB pip installable

- Add pyproject.toml (setuptools) with metadata, dependencies, package data
  (templates, static files, yaml) and entry points: after
  `sage -pip install -e .` (which is what requirements.txt now does),
  `import lmfdb` works without PYTHONPATH manipulation and the website can
  be started from anywhere with `lmfdb` or `sage -python -m lmfdb`.
  The version number is single-sourced in lmfdb/version.py.  Since
  psycodict is not yet released on PyPI, it is installed from its git
  repository; that must change before the lmfdb can be released on PyPI.

- Importing lmfdb no longer connects to postgres: lmfdb.db is now a lazy
  proxy that connects on first use (or explicitly via db.connect()), so
  `import lmfdb` is cheap and has no side effects (no config.ini,
  flasklog, slow_queries.log or secret_key created).

- Move the configuration code to lmfdb/config.py so that using the
  database does not pull in lmfdb.utils (and thus flask and sage);
  lmfdb/utils/config.py remains as a backwards-compatible shim, still
  runnable as a script as the CI does.  When running from a git checkout
  everything is located as before (config.ini and secret_key at the root);
  when installed as a package they live in ~/.lmfdb (overridable via
  LMFDB_HOME/LMFDB_CONFIG or --config-file), with a config.ini in the
  current directory taking precedence.

- Log files no longer pollute the current directory: flasklog,
  slow_queries.log and verification logs now default to logs/ under the
  checkout root (or ~/.lmfdb/logs when installed).  Existing config files
  with the old default values are redirected transparently; explicitly
  configured paths are respected.

- Move CONTRIBUTORS.yaml into the lmfdb package so the acknowledgments
  page works when installed, and run the knowl code-reference git greps
  in the checkout root rather than the process working directory (they
  now degrade gracefully when lmfdb is not run from a checkout).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Author: roed314

.github/copilot-instructions.md

@@ -169,7 +169,7 @@ pyflakes start-lmfdb.py user-manager.py lmfdb/
 - **40+ test files** across the codebase
 - **71,000+ lines** of Python code
 - **Complex mathematical application** with deep domain expertise required
-- **Large contributor base** (see CONTRIBUTORS.yaml)
+- **Large contributor base** (see lmfdb/CONTRIBUTORS.yaml)
 
 ## Working Effectively
 

.gitignore

@@ -14,6 +14,12 @@ config.ini
 
 flasklog
 slow_queries.log
+/logs/
+
+# Packaging artifacts
+*.egg-info/
+build/
+dist/
 
 # Testing files
 htmlcov

GettingStarted.md

@@ -57,6 +57,15 @@ Installation
    sage -pip install -r requirements.txt
    ```
 
+   This installs the LMFDB itself (in editable mode, i.e. pointing at your
+   checkout) together with its dependencies, so that `import lmfdb` works
+   from `sage` regardless of the current directory; there is no need to add
+   the checkout to your `PYTHONPATH`.  Equivalently, you can run
+   `sage -pip install -e .` (add `[dev]` to also get the testing tools).
+   Importing `lmfdb` does not connect to the database; the connection is
+   established the first time `lmfdb.db` is used, or explicitly via
+   `db.connect()`.
+
    ### Troubleshooting with packages.
 
    - If you have not run the site for a while you might get an error
@@ -129,6 +138,10 @@ Running
    sage -python start-lmfdb.py --debug
    ```
 
+   Since the LMFDB is installed by the step above, you can equivalently run
+   `lmfdb --debug` (a script installed by pip) or
+   `sage -python -m lmfdb --debug`, from any directory.
+
  * The effect of the (optional) --debug is that you will be running
    with the beta flag switched on as at dev.lmfdb.org, and also that
    if code fails your browser will show useful debugging information.
@@ -163,6 +176,17 @@ Running
    on using a different database instance, you can do so by changing
    config.ini in the root of the lmfdb directory.
 
+ * Locations of configuration and log files: when running from a git
+   checkout, the configuration file is `config.ini` at the root of the
+   checkout (as it always was) and log files (`flasklog`,
+   `slow_queries.log`, verification logs) go to the `logs/` subdirectory.
+   When the LMFDB is installed as a package rather than run from a checkout,
+   these files live in `~/.lmfdb` instead (unless there is a `config.ini` in
+   the current directory).  The environment variables `LMFDB_HOME` (the
+   directory for all of these files) and `LMFDB_CONFIG` (the path of the
+   configuration file) override these defaults, as does the `--config-file`
+   command-line option.
+
 
 CoCalc
 ======

MANIFEST.in

@@ -0,0 +1,9 @@
+# Non-python files that ship with the lmfdb package (templates, static
+# assets, yaml data, code snippet test fixtures); together with
+# `include-package-data = true` in pyproject.toml this also determines
+# which data files are installed
+recursive-include lmfdb *.yaml *.html *.css *.js *.png *.jpg *.gif *.svg *.svgz *.ico *.txt *.md *.rst *.log *.sage
+include lmfdb/number_fields/Database-info
+include lmfdb/galois_groups/Database-info
+global-exclude *.pyc
+global-exclude __pycache__

README.md

@@ -14,7 +14,11 @@ Wiki
 Development
 -----------
 
-* [Getting Started](https://github.com/LMFDB/lmfdb/blob/main/GettingStarted.md) -- cheat sheet for setting up lmfdb
+* [Getting Started](https://github.com/LMFDB/lmfdb/blob/main/GettingStarted.md) -- cheat sheet for setting up lmfdb.
+  The LMFDB is pip installable: `sage -pip install -e .` from a checkout
+  (this is what `sage -pip install -r requirements.txt` does), after which
+  `import lmfdb` works from `sage` anywhere and the website can be started
+  with `lmfdb` or `sage -python -m lmfdb`.
 * [Development Guide](https://github.com/LMFDB/lmfdb/blob/main/Development.md) -- organizing development
 * [Style Guide](https://github.com/LMFDB/lmfdb/blob/main/StyleGuide.md) -- how things on LMFDB pages should be styled to give the web site a coherent look
 * [Command-line search](https://github.com/LMFDB/lmfdb/blob/main/CommandLineSearch.md) -- searching the LMFDB from the command line with `lmfdb_search`

CONTRIBUTORS.yaml lmfdb/CONTRIBUTORS.yamlCONTRIBUTORS.yaml renamed to lmfdb/CONTRIBUTORS.yaml

@@ -1,11 +1,30 @@
+"""
+LMFDB: the database of L-functions, modular forms, and related objects.
 
-# psycodict was added as a dependency on May 10, 2024; we provide a useful error message for people upgrading
-try:
-    import psycodict
-    assert psycodict
-except ImportError:
-    print('Missing dependency; try running "sage -pip install -r requirements.txt" in the LMFDB home folder.')
-    raise
+Importing this package is lightweight; in particular it does not connect
+to the database.  The connection is established the first time ``lmfdb.db``
+is used, for example::
 
-from .lmfdb_database import db
-assert db
+    from lmfdb import db
+    db.ec_curvedata.lookup("11.a1")
+
+or explicitly via ``db.connect()``.  To run the website, use the ``lmfdb``
+command (or ``python -m lmfdb``, or ``start-lmfdb.py`` from a git checkout).
+"""
+
+
+def __getattr__(name):
+    # Lazy attributes (PEP 562), so that `import lmfdb` stays cheap
+    if name == "db":
+        try:
+            from .lmfdb_database import db
+        except ImportError:
+            print('Missing dependency; try running "sage -pip install -e ." in the LMFDB home folder.')
+            raise
+        globals()["db"] = db
+        return db
+    raise AttributeError("module 'lmfdb' has no attribute %r" % (name,))
+
+
+def __dir__():
+    return sorted(set(globals()) | {"db"})

lmfdb/__init__.py

@@ -0,0 +1,7 @@
+"""
+Start the LMFDB website via ``python -m lmfdb`` (or ``sage -python -m lmfdb``).
+"""
+from lmfdb.website import main
+
+if __name__ == "__main__":
+    main()

lmfdb/__main__.py

@@ -1,4 +1,4 @@
-from .utils.config import get_secret_key
+from .config import get_secret_key
 import os
 from socket import gethostname
 import time
@@ -18,12 +18,13 @@
 from markupsafe import escape
 from sage.env import SAGE_VERSION
 from sage.all import cached_function
-# acknowledgment page, reads info from CONTRIBUTORS.yaml
+# acknowledgment page, reads info from lmfdb/CONTRIBUTORS.yaml
 
 from .logger import critical
 from .homepage import load_boxes, contribs
 
-LMFDB_VERSION = "LMFDB Release 1.2.1"
+from .version import version as _lmfdb_version
+LMFDB_VERSION = "LMFDB Release " + _lmfdb_version
 
 ############################
 #         Main app         #
@@ -575,7 +576,7 @@ def add_colors():
         if color not in all_color_schemes:
             color = None
         if color is None:
-            from .utils.config import Configuration
+            from .config import Configuration
             color = Configuration().get_color()
     return {"color": all_color_schemes[color].dict()}