|
| 1 | +# Replit + Google Drive backup recipe |
| 2 | + |
| 3 | +Run [slackdump](https://github.com/rusq/slackdump) inside a Replit |
| 4 | +workspace and stream the resulting `archive/` directory to Google Drive |
| 5 | +without ever shipping a 10+ GB tarball through your laptop. |
| 6 | + |
| 7 | +This is useful when: |
| 8 | + |
| 9 | +- Your Slack workspace is large enough that a full archive would not |
| 10 | + fit on the machine you usually run slackdump on. |
| 11 | +- You want the archive to live on Drive (cheap cold storage, easy to |
| 12 | + share with teammates) rather than local disk. |
| 13 | +- You want the run to be unattended: kick it off in Replit, walk away, |
| 14 | + come back to a directory on Drive that mirrors `archive/` exactly. |
| 15 | + |
| 16 | +The recipe consists of one Node.js script |
| 17 | +([`drive-upload-folder.mjs`](drive-upload-folder.mjs)) that walks the |
| 18 | +local archive directory, recreates the same folder tree on Drive, and |
| 19 | +uploads every file resumably. A JSON manifest next to the script |
| 20 | +tracks per-file progress so you can stop and restart freely. |
| 21 | + |
| 22 | +## Prerequisites |
| 23 | + |
| 24 | +- A Replit workspace with the **Google Drive integration** enabled. |
| 25 | + The script uses Replit's connector proxy |
| 26 | + (`@replit/connectors-sdk`) so you do not need to manage OAuth |
| 27 | + client credentials yourself. |
| 28 | +- slackdump has already produced an archive directory in the |
| 29 | + workspace, for example via `slackdump archive -o ./archive`. |
| 30 | +- Node.js 18+ (already provided by Replit; the script uses global |
| 31 | + `fetch`). |
| 32 | +- A Google Drive folder where the `archive/` subfolder should be |
| 33 | + created. Its folder ID is the part after `/folders/` in the |
| 34 | + Drive URL. |
| 35 | + |
| 36 | +## Setup |
| 37 | + |
| 38 | +```bash |
| 39 | +# From this directory, install the one runtime dependency. |
| 40 | +npm install @replit/connectors-sdk |
| 41 | +``` |
| 42 | + |
| 43 | +## Usage |
| 44 | + |
| 45 | +```bash |
| 46 | +DRIVE_PARENT_ID="<your-drive-folder-id>" \ |
| 47 | +ARCHIVE_DIR="/path/to/archive" \ |
| 48 | +node drive-upload-folder.mjs |
| 49 | +``` |
| 50 | + |
| 51 | +The script prints progress every 5 seconds (file count, bytes, |
| 52 | +throughput, ETA) and writes a final reconciliation report comparing |
| 53 | +local file count and total bytes to what Drive reports. |
| 54 | + |
| 55 | +If the run is interrupted (you stopped it, the workspace got |
| 56 | +suspended, the network blipped), just run the same command again. |
| 57 | +The manifest tells the script which files are already done and which |
| 58 | +were partially uploaded; partial uploads resume from the byte offset |
| 59 | +the Drive session reports, not from zero. |
| 60 | + |
| 61 | +## Configuration |
| 62 | + |
| 63 | +All paths and tunables are env vars; defaults are friendly for the |
| 64 | +"`archive/` lives next to the script" case: |
| 65 | + |
| 66 | +| Variable | Default | Notes | |
| 67 | +| --- | --- | --- | |
| 68 | +| `DRIVE_PARENT_ID` | (required) | Google Drive folder ID that will contain the `archive/` subfolder. | |
| 69 | +| `ARCHIVE_DIR` | `./archive` | Local archive root to mirror to Drive. | |
| 70 | +| `MANIFEST_PATH` | `./archive-upload-manifest.json` | Per-file resume state. Safe to keep across runs. | |
| 71 | +| `REPORT_PATH` | `./archive-upload-report.txt` | Final reconciliation report. | |
| 72 | +| `PROGRESS_INTERVAL_MS` | `5000` | How often to log progress. | |
| 73 | +| `CHUNK_SIZE_MB` | `64` | Resumable upload chunk size. Larger is faster on a clean line, smaller resumes more cheaply on a bad one. | |
| 74 | + |
| 75 | +## How it works |
| 76 | + |
| 77 | +The Replit connector proxy rejects request bodies larger than ~1 MB, |
| 78 | +so multipart uploads are not viable for real archive files. Every |
| 79 | +file therefore uses Google Drive's resumable upload protocol: |
| 80 | + |
| 81 | +1. **Initiate session via the proxy.** The proxy forwards a tiny |
| 82 | + JSON metadata body to Drive and returns a session URL signed by |
| 83 | + Google. |
| 84 | +2. **PUT file bytes directly to the session URL.** This bypasses |
| 85 | + the proxy entirely, so chunk size is bounded only by what your |
| 86 | + network and Drive can sustain. |
| 87 | +3. **Save offset to the manifest after every chunk.** On restart, |
| 88 | + the script asks Drive for the canonical server offset before |
| 89 | + continuing — the manifest is a hint, not the source of truth. |
| 90 | +4. **Re-initiate expired sessions transparently.** Drive expires |
| 91 | + resumable sessions after ~7 days; the script catches `404`/`410` |
| 92 | + and starts a fresh session for the affected file. |
| 93 | +5. **Reconcile at the end.** After the upload loop, the script |
| 94 | + walks the destination on Drive and reports both file count and |
| 95 | + total bytes, so you can verify byte-exact parity before deleting |
| 96 | + the local archive. |
| 97 | + |
| 98 | +## Caveats |
| 99 | + |
| 100 | +- Drive Shared Drives have a 400,000-file limit per drive. Very |
| 101 | + large slackdump archives can approach this; consider tarring the |
| 102 | + whole `archive/` directory and uploading that single file if you |
| 103 | + hit the cap. |
| 104 | +- Files larger than ~5 TB cannot be uploaded to Drive at all (Google |
| 105 | + limit). This is well above any realistic slackdump archive. |
| 106 | +- The script does not delete files from Drive that no longer exist |
| 107 | + locally; it is additive only. |
0 commit comments