Merge pull request #8 from notesofcliff/codex/update-documentation-for-new-project-structure

docs: update paths for new project layout

Author: iLoveTux

README.md

@@ -17,6 +17,7 @@ Delve is a powerful, extensible platform for ingesting, transforming, and search
 - `bootstrap.py` for automated build, packaging, and asset management
 - `frontend/` for JavaScript and SCSS assets
 - `doc/` for user and admin documentation
+- `utilities/cli/` for ingestion utilities such as `tail-files.py` and `syslog-receiver.py`
 
 ## Quick Start
 
@@ -66,6 +67,18 @@ python manage.py createsuperuser
 python manage.py runserver
 ```
 
+### 9. (Optional) Start additional services
+```bash
+# Task scheduler
+python manage.py qcluster
+
+# Syslog server
+python utilities/cli/syslog-receiver.py
+
+# Tail log files
+python utilities/cli/tail-files.py /var/log/*.log
+```
+
 > **Defaults:** Delve ships with Whitenoise + CherryPy by default to keep air-gapped/offline use simple. Swap components as desired.
 ## Dependency Management
 

doc/admin/Backup_and_Restore.md

@@ -9,21 +9,21 @@ Use the `dumpdata` command to export data from the database to a JSON file. This
 To back up the entire database, run the following command:
 
 ```bash
-fl dumpdata > backup.json
+python manage.py dumpdata > backup.json
 ```
 
 ### Backing Up Specific Apps
 To back up specific apps (e.g., `users`, `auth`), run the following command:
 
 ```bash
-fl dumpdata users auth > backup_users_auth.json
+python manage.py dumpdata users auth > backup_users_auth.json
 ```
 
 ### Scheduling Backups
 You can schedule regular backups using a task scheduler like `cron` on Linux or Task Scheduler on Windows. Here is an example of a `cron` job that runs a backup every day at midnight:
 
 ```cron
-0 0 * * * /path/to/delve/venv/bin/fl dumpdata > /path/to/backups/backup_$(date +\%F).json
+0 0 * * * cd /path/to/delve && /path/to/delve/venv/bin/python manage.py dumpdata > /path/to/backups/backup_$(date +\%F).json
 ```
 
 ## Restoring Data
@@ -33,7 +33,7 @@ Use the `loaddata` command to import data from a JSON file into the database. Th
 To restore data from a backup file, run the following command:
 
 ```bash
-fl loaddata backup.json
+python manage.py loaddata backup.json
 ```
 
 The `loaddata` command will load the data into the proper app automatically.
@@ -58,7 +58,8 @@ Here is an example backup script for Linux:
 DELVE_DIR="/path/to/delve"
 
 # Run the backup command
-$DELVE_DIR/fl dumpdata > "$DELVE_DIR/backups/backup_$(date +\%F).json"
+cd "$DELVE_DIR"
+python manage.py dumpdata > "$DELVE_DIR/backups/backup_$(date +\%F).json"
 ```
 
 ### Automating Restores
@@ -75,7 +76,8 @@ Here is an example restore script for Linux:
 DELVE_DIR="/path/to/delve"
 
 # Run the restore command
-$DELVE_DIR/fl loaddata "$DELVE_DIR/backups/backup.json"
+cd "$DELVE_DIR"
+python manage.py loaddata "$DELVE_DIR/backups/backup.json"
 ```
 
 By following these steps, you can effectively back up and restore your Delve instance, ensuring that your data is protected and can be quickly recovered in case of an issue.

doc/admin/Configuration.md

@@ -84,7 +84,7 @@ Delve has several specific configuration options that you can set in `settings.p
 
 ## Example Configuration
 
-Please see the provided `example-settings.py` which can be found in the Delve installation directory at `./delve/example-settings.py` for an example.
+Use the existing `delve/settings.py` file in the project root as a reference for available options and environment variable support.
 
 
 [Previous: Bootstrap Guide](/doc/admin/Bootstrap_Guide.md) | [Next: Ingesting Data](/doc/admin/Ingesting_Data.md)

doc/admin/Ingesting_Data.md

@@ -89,7 +89,7 @@ python utilities/cli/tail-files.py --sourcetype "my_sourcetype" "/path/to/logfil
 python utilities/cli/tail-files.py --no-verify --server "https://delve.example.com" "/path/to/logfile.log"
 
 # Provide username and password for authentication to prevent being prompted
-python utilities/cli/tail-files.py--username "myuser" --password "mypassword" "/path/to/logfile.log"
+python utilities/cli/tail-files.py --username "myuser" --password "mypassword" "/path/to/logfile.log"
 
 # Slightly decrease verbosity of log output
 python utilities/cli/tail-files.py -v "/path/to/logfile.log"
@@ -98,7 +98,7 @@ python utilities/cli/tail-files.py -v "/path/to/logfile.log"
 python utilities/cli/tail-files.py -vvvvvv "/path/to/logfile.log"
 
 # Monitor multiple log files using glob patterns
-python utilities/cli/tail-files.py"/var/log/*.log"
+python utilities/cli/tail-files.py "/var/log/*.log"
 ```
 
 ### Additional Information

doc/admin/Installation_and_Setup.md

@@ -16,12 +16,7 @@ Follow these steps to install Delve:
 
 1. **Download Delve**: Download the latest release from the [releases page](https://github.com/notesofcliff/delve/releases).
 2. **Extract Files**: Unzip the downloaded file to your desired location.
-3. **Configure Settings**: Copy the example settings and URL files.
-
-   ```
-   cp ./delve/example-settings.py ./delve/settings.py
-   cp ./delve/example-urls.py ./delve/urls.py
-   ```
+3. **Configure Settings**: Adjust `delve/settings.py` and `delve/urls.py` for your environment. These files are provided in the repository and can be customized directly.
 
 **Important**: It is very important to change your `SECRET_KEY` setting. The default setting will invalidate all sessions and more on every restart of the server. `SECRET_KEY` should be set to a randomly generated string that is kept secret and safe. The `python manage.py gen-secret-key` command will print such a string that can be copied and pasted into your `settings.py`.
 
@@ -47,10 +42,10 @@ Follow these steps to install Delve:
 
    ```
    # Start the web server (Explore UI, Admin UI and REST API)
-   ./dlv serve
+   python manage.py serve
 
    # Start the task scheduler
-   ./dlv qcluster
+   python manage.py qcluster
 
    # Start the syslog server
    python utilities/cli/syslog-receiver.py
@@ -59,7 +54,7 @@ Follow these steps to install Delve:
    python utilities/cli/tail-files.py /var/log/*.log
    ```
 
-**NOTE**: Utilities launched with `dlv` are generally configured in settings.py, while utilities in `utilities/cli/` are generally configured via command line arguments.
+**NOTE**: Utilities launched with `python manage.py` are configured in `settings.py`, while utilities in `utilities/cli/` are generally configured via command line arguments.
 
 For details on automating these steps into a repeatable build, see the [Bootstrap Guide](/doc/admin/Bootstrap_Guide.md).
 
@@ -68,7 +63,7 @@ For details on automating these steps into a repeatable build, see the [Bootstra
 Delve uses CherryPy to host the Django web app. The `serve` management command starts the CherryPy server to serve the Delve web UI.
 
 ```bash
-./dlv serve
+python manage.py serve
 ```
 
 The following settings in `settings.py` control the behavior of the CherryPy web server:
@@ -86,7 +81,7 @@ The following settings in `settings.py` control the behavior of the CherryPy web
 - **DELVE_ACCEPTED_QUEUE_TIMEOUT**: How long to wait for an HTTP request to be accepted before timing out.
 - **DELVE_SERVER_MAX_THREADS**: The max number of threads to spawn to handle web requests.
 
-**NOTE**: The provided example-settings.py will check environment variables of the same name for all of these server-specific configurations as well as other settings. 
+**NOTE**: The provided `settings.py` will check environment variables of the same name for all of these server-specific configurations as well as other settings.
 
 ## Initial Configuration
 After installation, perform the initial configuration to tailor Delve to your needs. Configuration settings are found in the `settings.py` file.
@@ -95,7 +90,7 @@ After installation, perform the initial configuration to tailor Delve to your ne
 
 Delve Supervisor is a very simple service (currently only available on Windows) that is configured via the `DELVE_SERVICE_COMMANDS` value in `settings.py`. 
 
-`DELVE_SERVICE_COMMANDS` should be a list of commands to run when the Delve Supervisor service is started. These commands are supposed to run forever (like `./dlv serve` and `./dlv qcluster`). Each command will be run and if any of the processes die, that process will be restarted.
+`DELVE_SERVICE_COMMANDS` should be a list of commands to run when the Delve Supervisor service is started. These commands are supposed to run forever (like `python manage.py serve` and `python manage.py qcluster`). Each command will be run and if any of the processes die, that process will be restarted.
 
 ### Installing Delve Supervisor Service on Windows
 If you are on Windows, you can use the following command from the Delve directory to install the Delve supervisor service. Run the command from an Administrator Command Prompt:

doc/admin/Troubleshooting.md

@@ -52,7 +52,7 @@ If you encounter issues that you cannot resolve, consider accessing support reso
 - **Django Documentation**: [https://docs.djangoproject.com/](https://docs.djangoproject.com/)
 - **Django Rest Framework Documentation**: [https://www.django-rest-framework.org/](https://www.django-rest-framework.org/)
 - **Django Extensions Documentation**: [https://django-extensions.readthedocs.io/](https://django-extensions.readthedocs.io/)
-- **Delve Documentation**: Included with every Delve documentation in the `doc` directory, available in the Delve Web UI (default http://127.0.0.1:8000/docs/) and in the [Github Repository](https://github.com/notesofcliff/delve/blob/main/src/home/doc/index)
+- **Delve Documentation**: Included with every Delve documentation in the `doc` directory, available in the Delve Web UI (default http://127.0.0.1:8000/docs/) and in the [Github Repository](https://github.com/notesofcliff/delve/blob/main/doc/index.md)
 - **Delve Community**: Refer our [Github Repository](https://github.com/notesofcliff/delve) for reporting issues, submitting feature requests, and contacting the project maintainers.
 
 By following these troubleshooting techniques and utilizing available resources, you can effectively diagnose and resolve issues with your Delve instance.

doc/user/App_Developer_Guide.md

@@ -291,7 +291,7 @@ search --last-15-minutes text__icontains=fail index=logs
 ```
 
 2. Schedule the saved query using Django Q:
-   1. Ensure the Q cluster is running (`./dlv qcluster` or via Windows Service)
+   1. Ensure the Q cluster is running (`python manage.py qcluster` or via Windows Service)
    2. Access the Django admin interface at `/admin/`
    3. Navigate to `Django Q > Scheduled tasks`
    4. Click "Add Scheduled task"

doc/user/Ingesting_Data.md

@@ -75,28 +75,28 @@ Here are some example commands to use the file-tail utility:
 
 ```bash
 # Basic usage with default settings
-python tail-files.py "/path/to/logfile.log"
+python utilities/cli/tail-files.py "/path/to/logfile.log"
 
 # Specify a custom index
-python tail-files.py --index "my_index" "/path/to/logfile.log"
+python utilities/cli/tail-files.py --index "my_index" "/path/to/logfile.log"
 
 # Specify a custom sourcetype
-python tail-files.py --sourcetype "my_sourcetype" "/path/to/logfile.log"
+python utilities/cli/tail-files.py --sourcetype "my_sourcetype" "/path/to/logfile.log"
 
 # Disable TLS hostname verification for the Delve server
-python tail-files.py --no-verify --server "https://delve.example.com" "/path/to/logfile.log"
+python utilities/cli/tail-files.py --no-verify --server "https://delve.example.com" "/path/to/logfile.log"
 
 # Provide username and password for authentication to prevent being prompted
-python tail-files.py --username "myuser" --password "mypassword" "/path/to/logfile.log"
+python utilities/cli/tail-files.py --username "myuser" --password "mypassword" "/path/to/logfile.log"
 
 # Slightly decrease verbosity of log output
-python tail-files.py -v "/path/to/logfile.log"
+python utilities/cli/tail-files.py -v "/path/to/logfile.log"
 
 # Most verbose log output
-python tail-files.py -vvvvvv "/path/to/logfile.log"
+python utilities/cli/tail-files.py -vvvvvv "/path/to/logfile.log"
 
 # Monitor multiple log files using glob patterns
-python tail-files.py "/var/log/*.log"
+python utilities/cli/tail-files.py "/var/log/*.log"
 ```
 
 ### Additional Information
@@ -145,37 +145,37 @@ Here are some example commands to use the syslog-receiver utility:
 
 ```bash
 # Basic usage with default settings for UDP
-python syslog-receiver.py --udp
+python utilities/cli/syslog-receiver.py --udp
 
 # Basic usage with default settings for TCP
-python syslog-receiver.py --tcp
+python utilities/cli/syslog-receiver.py --tcp
 
 # Basic usage with default settings for TCP and UDP
-python syslog-receiver.py --tcp --udp
+python utilities/cli/syslog-receiver.py --tcp --udp
 
 # Listen on a custom port for TCP
-python syslog-receiver.py --tcp --tcp-port 9514
+python utilities/cli/syslog-receiver.py --tcp --tcp-port 9514
 
 # Specify a custom index and sourcetype for UDP
-python syslog-receiver.py --udp --index "my_index" --sourcetype "my_sourcetype"
+python utilities/cli/syslog-receiver.py --udp --index "my_index" --sourcetype "my_sourcetype"
 
 # Specify a custom index and sourcetype for TCP
-python syslog-receiver.py --tcp --index "my_index" --sourcetype "my_sourcetype"
+python utilities/cli/syslog-receiver.py --tcp --index "my_index" --sourcetype "my_sourcetype"
 
 # Use TLS for TCP with specified certificate and key
-python syslog-receiver.py --tcp --tcp-cert "/path/to/cert.pem" --tcp-key "/path/to/key.pem"
+python utilities/cli/syslog-receiver.py --tcp --tcp-cert "/path/to/cert.pem" --tcp-key "/path/to/key.pem"
 
 # Disable TLS hostname verification for the Delve server
-python syslog-receiver.py --no-verify --server "https://delve.example.com" --tcp
+python utilities/cli/syslog-receiver.py --no-verify --server "https://delve.example.com" --tcp
 
 # Provide username and password for authentication to prevent being prompted
-python syslog-receiver.py --username "myuser" --password "mypassword" --udp
+python utilities/cli/syslog-receiver.py --username "myuser" --password "mypassword" --udp
 
 # Lowest verbosity of log output (less than default)
-python syslog-receiver.py -v --udp
+python utilities/cli/syslog-receiver.py -v --udp
 
 # Most verbose log output (more than default)
-python syslog-receiver.py -vvvvvv --udp
+python utilities/cli/syslog-receiver.py -vvvvvv --udp
 ```
 
 ### Additional Information