Skip to content

readme.md

Latest commit

 

History

History
207 lines (162 loc) · 7.21 KB

readme.md

File metadata and controls

207 lines (162 loc) · 7.21 KB
Raw

fsync

CREATE TABLE file_metadata (
    id SERIAL PRIMARY KEY,
    file_path TEXT NOT NULL,
    file_name TEXT NOT NULL,
    file_size BIGINT,
    creation_time TIMESTAMP,
    modification_time TIMESTAMP,
    access_time TIMESTAMP
);

import os
import psycopg2
from datetime import datetime

# Database connection details
DB_HOST = "localhost"
DB_NAME = "your_database_name"
DB_USER = "your_username"
DB_PASSWORD = "your_password"

# Directory to scan
SCAN_DIRECTORY = "/path/to/your/directory"

def get_file_metadata(filepath):
    """Collects essential metadata for a given file."""
    try:
        stats = os.stat(filepath)
        return {
            "file_path": filepath,
            "file_name": os.path.basename(filepath),
            "file_size": stats.st_size,
            "creation_time": datetime.fromtimestamp(stats.st_ctime),
            "modification_time": datetime.fromtimestamp(stats.st_mtime),
            "access_time": datetime.fromtimestamp(stats.st_atime),
        }
    except FileNotFoundError:
        print(f"File not found: {filepath}")
        return None
    except Exception as e:
        print(f"Error collecting metadata for {filepath}: {e}")
        return None

def insert_metadata_into_db(metadata):
    """Inserts file metadata into the PostgreSQL database."""
    conn = None
    try:
        conn = psycopg2.connect(host=DB_HOST, database=DB_NAME, user=DB_USER, password=DB_PASSWORD)
        cur = conn.cursor()
        
        insert_query = """
        INSERT INTO file_metadata (file_path, file_name, file_size, creation_time, modification_time, access_time)
        VALUES (%s, %s, %s, %s, %s, %s)
        ON CONFLICT (file_path) DO UPDATE SET
            file_name = EXCLUDED.file_name,
            file_size = EXCLUDED.file_size,
            creation_time = EXCLUDED.creation_time,
            modification_time = EXCLUDED.modification_time,
            access_time = EXCLUDED.access_time;
        """
        
        cur.execute(insert_query, (
            metadata["file_path"],
            metadata["file_name"],
            metadata["file_size"],
            metadata["creation_time"],
            metadata["modification_time"],
            metadata["access_time"],
        ))
        conn.commit()
        cur.close()
    except (Exception, psycopg2.Error) as error:
        print(f"Error while connecting to PostgreSQL or inserting data: {error}")
    finally:
        if conn:
            conn.close()

def scan_and_store_metadata(directory):
    """Scans a directory for files, collects metadata, and stores it in the database."""
    for root, _, files in os.walk(directory):
        for file in files:
            filepath = os.path.join(root, file)
            metadata = get_file_metadata(filepath)
            if metadata:
                insert_metadata_into_db(metadata)
                print(f"Metadata for {file} stored.")

if __name__ == "__main__":
    scan_and_store_metadata(SCAN_DIRECTORY)
    print("File scanning and metadata collection complete.")

CLI usage

The package provides a small CLI entrypoint you can run with Python's -m flag.

Examples:

Index a directory and print JSON metadata:

python -m fsync.cli index /path/to/dir --recursive --hash sha256

Compare two directories and print a JSON report:

python -m fsync.cli compare /path/to/dirA /path/to/dirB --recursive --hash sha256

The output is JSON written to stdout and can be redirected to a file for further processing.

Additional options

  • --workers N: run N hashing workers (threads) to compute file checksums in parallel.
  • --format jsonl: (index) write one JSON object per line for streaming/large outputs.
  • --format pretty: (compare) print a concise human summary; pair with --show N to show sample entries.
  • --show N: show up to N sample matches in pretty compare output.
  • --progress: show a progress bar during indexing (requires tqdm package).
  • --verbose: increase CLI verbosity (repeat for more verbose logging).

Hashing note

You can pass --hash b3sum to use the external b3sum utility (Blake3) for faster hashing when available. If b3sum is not installed, the CLI will fall back to the default sha256.

Benchmark

You can run a simple benchmark which creates files and measures hashing throughput:

fsync benchmark /tmp/fsync-bench --count 100 --size 4096 --hash b3sum --workers 4

If your b3sum binary is not on PATH, pass its path via --b3sum-path /path/to/b3sum.

You can also install the package locally and get a fsync console script:

pip install -e .
fsync index /path/to/dir --workers 4 --format jsonl > out.jsonl

Sync planning (fsync sync-plan)

compare tells you what differs; sync-plan projects that hash-based diff onto rsync inputs. fsync never copies bytes itself — it writes --files-from lists plus a guarded run.sh you review and run.

# Bidirectional union (default, dry-run): each side's unique files flow to the
# other; same-path/different-content files are listed as conflicts, NOT synced.
fsync sync-plan /home/developer developer@10.55.0.2:/home/developer --out-dir plan
# (or scan locally and point rsync elsewhere)
fsync sync-plan /home/developer /mnt/other \
  --src /home/developer --dest developer@10.55.0.2:/home/developer --out-dir plan
bash plan/run.sh        # review first; re-run sync-plan with --execute to drop -n

Generated files: plan.a_to_b.lst, plan.b_to_a.lst (rsync --files-from lists), plan.conflicts.txt, plan.renames.sh (content-identical files under a different name — a rename, not a re-copy), and run.sh.

Key options:

  • --conflict review|newer|a-wins|b-wins — how to route same-path/different-content files (default review: left for a human, never auto-clobbered).
  • --mirror a-to-b|b-to-a — one-way mirror (the only mode that proposes deletions; they are written to a list and commented out in run.sh). Ignores --conflict.
  • --from-report report.json — consume a saved compare --output report (requires --src/--dest).
  • --execute — emit live rsync commands (default is a dry run with -n).
  • --src / --dest — rsync endpoints (default: the resolved scan dirs).
  • --rsync-flags — override the defaults (-aHAXS --numeric-ids --ignore-times --info=progress2 --partial).

Safety: copies never --delete; --ignore-times makes rsync transfer exactly the hash-chosen files (so a same-size/same-mtime-but-different-content file is never silently skipped); deletions only ever appear in mirror mode and are emitted commented-out. After running, re-run fsync compare to verify.

Fast link: Thunderbolt/USB4 host-to-host

A single TB4/USB4 cable between the two machines gives a ~10–20 Gbit/s point-to-point network (kernel thunderbolt_net) — much faster than GbE for a full /home sync. One-time setup on each box:

sudo scripts/tb-net.sh 1   # on the scanning machine -> 10.55.0.1/30
sudo scripts/tb-net.sh 2   # on the peer             -> 10.55.0.2/30

Plug the cable (direct port-to-port is the reliable option; a dock's downstream TB port usually works too), wait for thunderbolt0 to appear, then use the developer@10.55.0.2:... endpoints exactly as in the examples above. The 10.55.0.x addresses in this doc are that link.