Skip to content

Commit 250c2ab

Browse files
committed
Add IO and registry documentation; update guides
Added new documentation for lzl.io.file, lzl.io.persistence, and lzl.io.ser modules. Updated and expanded guides for lzl.load, lzl.logging, lzl.pool, lzl.proxied, lzo.registry, lzo.types, and lzo.utils. Improved the main index page with flash examples and feature highlights. Updated mkdocs.yml navigation to include new IO subpages.
1 parent 058381e commit 250c2ab

12 files changed

Lines changed: 691 additions & 681 deletions

File tree

docs/api/lzl/io/file.md

Lines changed: 124 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,124 @@
1+
# lzl.io.file - Unified File Operations
2+
3+
The `lzl.io.file` module provides a powerful, unified abstraction for file system operations, supporting both local and cloud storage (S3, MinIO, R2) with synchronous and asynchronous APIs. It automatically selects the appropriate backend based on the file path scheme.
4+
5+
## Overview
6+
7+
The `File` class is the main entry point. It acts as a factory that instantiates the correct concrete path object (e.g., `Path` for local files, `FileS3Path` for S3).
8+
9+
::: lzl.io.file
10+
options:
11+
members:
12+
- File
13+
14+
## Supported Schemes
15+
16+
- **Local Files**: `/path/to/file`, `relative/path`
17+
- **AWS S3**: `s3://bucket/key`
18+
- **MinIO**: `minio://bucket/key`
19+
- **Cloudflare R2**: `r2://bucket/key`
20+
21+
## Usage Examples
22+
23+
### Basic File I/O
24+
25+
```python
26+
from lzl.io import File
27+
28+
# Write text (sync)
29+
File("data.txt").write_text("Hello World")
30+
31+
# Read text (async)
32+
content = await File("data.txt").async_read_text()
33+
34+
# Check existence
35+
if await File("data.txt").async_exists():
36+
print("File exists!")
37+
```
38+
39+
### Cloud Storage (S3)
40+
41+
```python
42+
from lzl.io import File
43+
44+
# Working with S3 paths
45+
s3_file = File("s3://my-bucket/data.csv")
46+
47+
# Read bytes
48+
data = await s3_file.read_bytes()
49+
50+
# Get metadata
51+
size = s3_file.size
52+
last_modified = s3_file.stat().st_mtime
53+
```
54+
55+
### Pydantic Integration
56+
57+
`File` is fully compatible with Pydantic v1 and v2, making it ideal for configuration models.
58+
59+
```python
60+
from pydantic import BaseModel
61+
from lzl.io import File
62+
63+
class Config(BaseModel):
64+
dataset_path: File
65+
output_dir: File
66+
67+
# Validates and converts strings to File objects
68+
config = Config(
69+
dataset_path="s3://data/sets/train.parquet",
70+
output_dir="/tmp/output"
71+
)
72+
73+
print(config.dataset_path.scheme) # 's3'
74+
```
75+
76+
### Custom Loaders
77+
78+
You can register custom loaders for specific file extensions.
79+
80+
```python
81+
from lzl.io import File
82+
import json
83+
84+
def load_json(file: File):
85+
return json.loads(file.read_text())
86+
87+
# Register the loader
88+
File.register_loader(".json", load_json)
89+
90+
# Now you can load directly (implementation dependent on registered hooks)
91+
# data = File("config.json").load()
92+
```
93+
94+
## Advanced Features
95+
96+
### Directory Management
97+
98+
```python
99+
# Get the parent directory
100+
parent = File.get_dir("path/to/file.txt")
101+
102+
# Check object size
103+
size = File.get_object_size("some data")
104+
print(f"Size: {size.human_readable}")
105+
```
106+
107+
## Spec and Path Types
108+
109+
Deep dive into the underlying path implementations and specifications.
110+
111+
::: lzl.io.file.spec.main
112+
::: lzl.io.file.path
113+
114+
## Configuration
115+
116+
Configure storage backends and behavior.
117+
118+
::: lzl.io.file.configs
119+
120+
## Utilities
121+
122+
Helper functions for file operations.
123+
124+
::: lzl.io.file.utils

docs/api/lzl/io/persistence.md

Lines changed: 116 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,116 @@
1+
# lzl.io.persistence - Data Persistence
2+
3+
The `lzl.io.persistence` module offers robust persistence mechanisms, providing a dictionary-like interface backed by various storage engines (SQLite, Redis, Object Storage). It supports caching, asynchronous access, and data serialization.
4+
5+
## Persistent Dictionary
6+
7+
The `PersistentDict` is the core class. It mimics a standard Python dictionary but persists its contents.
8+
9+
::: lzl.io.persistence.main
10+
options:
11+
members:
12+
- PersistentDict
13+
14+
## Initialization
15+
16+
```python
17+
from lzl.io.persistence import PersistentDict
18+
19+
# Local SQLite backend (default if no scheme provided)
20+
cache = PersistentDict("my_app_cache", serializer="json")
21+
22+
# Redis backend
23+
redis_cache = PersistentDict(
24+
"my_redis_cache",
25+
backend="redis",
26+
base_key="app:v1",
27+
expiration=3600
28+
)
29+
30+
# Object Storage backend (S3)
31+
s3_cache = PersistentDict(
32+
"s3_cache",
33+
base_key="s3://my-bucket/cache/prefix",
34+
serializer="pickle"
35+
)
36+
```
37+
38+
## Features
39+
40+
### Async Support
41+
42+
Most methods have an `async` equivalent prefixed with `a` (e.g., `aget`, `aset`, `adelete`).
43+
44+
```python
45+
await cache.aset("key", "value")
46+
value = await cache.aget("key")
47+
```
48+
49+
### Context Managers & Locking
50+
51+
Ensure data consistency with context managers that handle locking.
52+
53+
```python
54+
# Sync context
55+
with cache.acquire_context():
56+
cache["key"] = "new_value"
57+
# Changes are flushed on exit
58+
59+
# Async context
60+
async with cache.acquire_acontext():
61+
await cache.aset("key", "async_value")
62+
```
63+
64+
### Mutation Tracking
65+
66+
`PersistentDict` tracks changes to mutable objects (like lists or dicts) retrieved from the cache and saves them back if they are modified within a tracking context.
67+
68+
```python
69+
with cache.track_changes("user:123", "get") as user_data:
70+
user_data["login_count"] += 1
71+
# user_data is automatically saved back to the backend if it changed
72+
```
73+
74+
### Math & Set Operations
75+
76+
Native support for atomic increments and set operations (especially useful with Redis).
77+
78+
```python
79+
# Increment
80+
cache.incr("counter", 1)
81+
82+
# Set operations
83+
cache.sadd("users", "alice", "bob")
84+
members = cache.smembers("users")
85+
```
86+
87+
## Backends
88+
89+
Supported backends implementations.
90+
91+
- **Local**: Stores data in local files.
92+
- **SQLite**: High-performance, single-file database (Recommended for local persistence).
93+
- **Redis**: Distributed in-memory store.
94+
- **Object Storage**: S3, MinIO, R2 for cloud persistence.
95+
96+
::: lzl.io.persistence.backends
97+
98+
## Serialization
99+
100+
Data is serialized before storage. Supported formats:
101+
- `json`: Human-readable, widely supported.
102+
- `pickle`: Python-specific, supports complex objects.
103+
- `msgpack`: Binary, efficient.
104+
105+
You can configure compression (gzip, zstd) alongside serialization.
106+
107+
## Metrics
108+
109+
Attach metrics to track usage or values within the dictionary.
110+
111+
```python
112+
from lzl.io.persistence.addons import CountMetric
113+
114+
cache.configure_metric("hits", kind="count")
115+
cache.metrics["hits"].incr()
116+
```

docs/api/lzl/io/ser.md

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
# lzl.io.ser - Serialization
2+
3+
High-performance serialization utilities supporting JSON, Pickle, MsgPack, and compression.
4+
5+
## Main Interface
6+
7+
::: lzl.io.ser.base
8+
9+
## Formatters
10+
11+
::: lzl.io.ser._json
12+
::: lzl.io.ser._pickle
13+
::: lzl.io.ser._msgpack
14+
15+
## Usage
16+
17+
```python
18+
from lzl.io.ser import serialize, deserialize
19+
20+
data = {"complex": "object"}
21+
22+
# Auto-detect format based on context or configuration
23+
s = serialize(data, format="json")
24+
d = deserialize(s, format="json")
25+
```

docs/api/lzl/load.md

Lines changed: 43 additions & 36 deletions
Original file line numberDiff line numberDiff line change
@@ -1,63 +1,70 @@
11
# lzl.load - Lazy Loading
22

3-
The `lzl.load` module provides utilities for lazy loading of modules and dependencies, enabling deferred imports and reducing startup time.
3+
The `lzl.load` module provides the `LazyLoad` proxy and other utilities for deferred imports. This pattern drastically reduces application startup time by only loading heavy dependencies when they are first accessed.
44

5-
## Module Reference
5+
## Key Components
66

7-
::: lzl.load
8-
options:
9-
show_root_heading: true
10-
show_source: true
7+
### LazyLoad
118

12-
## Overview
9+
The core proxy class. It intercepts attribute access to trigger the import.
1310

14-
Lazy loading defers the import of modules until they are actually needed, which can significantly improve application startup time and reduce memory footprint.
11+
::: lzl.load.main
12+
options:
13+
members:
14+
- LazyLoad
15+
- lazy_load
16+
- load
17+
- reload
1518

16-
## Usage Examples
19+
## Usage Guide
1720

1821
### Basic Lazy Loading
1922

23+
Instead of top-level imports, define a proxy.
24+
2025
```python
2126
from lzl.load import LazyLoad
2227

23-
# Create a lazy reference to a module
24-
numpy = LazyLoad('numpy')
28+
# 'numpy' is NOT imported yet
29+
np = LazyLoad("numpy")
2530

26-
# The module is only imported when accessed
27-
array = numpy.array([1, 2, 3]) # Import happens here
31+
def process_data(data):
32+
# 'numpy' is imported here, on the first attribute access
33+
return np.array(data)
2834
```
2935

30-
### Lazy Loading with Aliases
36+
### Handling Optional Dependencies
3137

32-
```python
33-
from lzl.load import LazyLoad
34-
35-
# Load with an alias
36-
pd = LazyLoad('pandas', 'pd')
38+
You can configure `LazyLoad` to automatically install missing packages (though use with caution in production).
3739

38-
# Use as normal
39-
df = pd.DataFrame({'a': [1, 2, 3]})
40+
```python
41+
# If 'pandas' is missing, it will attempt to pip install it
42+
pd = LazyLoad("pandas", install_missing=True)
4043
```
4144

42-
### Conditional Imports
45+
### Dependency Chains
4346

44-
```python
45-
from lzl.load import LazyLoad
47+
If a module depends on another lazy module being loaded first (e.g., for side effects), you can declare dependencies.
4648

47-
# Only import if actually used
48-
optional_module = LazyLoad('some.optional.module')
49-
50-
if needs_feature:
51-
optional_module.do_something()
49+
```python
50+
# specific_setup must be loaded before my_module
51+
setup = LazyLoad("my_app.specific_setup")
52+
mod = LazyLoad("my_app.my_module", dependencies=setup)
5253
```
5354

54-
## Benefits
55+
### Type Checking
5556

56-
- **Faster Startup**: Modules are only imported when needed
57-
- **Reduced Memory**: Unused modules don't consume memory
58-
- **Simplified Dependencies**: Optional dependencies can be handled gracefully
59-
- **Better Testing**: Mock imports more easily in tests
57+
For static analysis (mypy/pyright), you can use `TYPE_CHECKING` blocks to keep type hints working while using lazy loading at runtime.
58+
59+
```python
60+
from typing import TYPE_CHECKING
61+
from lzl.load import LazyLoad
6062

61-
## Implementation Details
63+
if TYPE_CHECKING:
64+
import pandas as pd
65+
else:
66+
pd = LazyLoad("pandas")
6267

63-
The `LazyLoad` class uses Python's import system to defer module loading. When you access an attribute on a lazy-loaded module, the actual import is triggered transparently.
68+
def get_df() -> "pd.DataFrame":
69+
return pd.DataFrame()
70+
```

0 commit comments

Comments
 (0)