feat: add docker pipeline and preview fixes
This commit is contained in:
@@ -0,0 +1,142 @@
|
||||
# Zero-Bandwidth Duplicati API
|
||||
|
||||
Thin `Node.js + Express` metadata service for Duplicati restores. The backend owns only SQLite metadata and enhancement jobs; file bytes stay on WebDAV and are never proxied through the runtime restore APIs.
|
||||
|
||||
## What changed
|
||||
|
||||
- Multiple uploaded Duplicati task databases can now coexist at the same time.
|
||||
- Every uploaded task library becomes its own `sourceId`.
|
||||
- A single uploaded `Duplicati-server.sqlite` can be stored separately and used to map friendly task names onto task database files.
|
||||
- Directory browsing and file metadata APIs are now `sourceId`-scoped.
|
||||
- The frontend is now split into three pages:
|
||||
- `/` for source management
|
||||
- `/source.html?sourceId=...` for a source workbench
|
||||
- `/file.html?sourceId=...&id=...` for video preview and browser-direct downloads
|
||||
- Video preview and downloads now pull encrypted dblocks directly from WebDAV in the browser, decrypt them locally, and never proxy the file stream through the backend.
|
||||
- The backend can build an enhanced SQLite copy per source by fetching remote `*.dblock.zip.aes` files, decrypting them, and writing:
|
||||
- `archive_entry_index`
|
||||
- `volume_crypto_cache`
|
||||
- `volume_scan_inventory`
|
||||
- `enhancement_meta`
|
||||
|
||||
## Endpoints
|
||||
|
||||
- `GET /api/sources`
|
||||
- `GET /api/sources/:sourceId`
|
||||
- `POST /api/sources/upload`
|
||||
- `GET /api/server-db`
|
||||
- `POST /api/server-db/upload`
|
||||
- `PUT /api/sources/:sourceId/secrets`
|
||||
- `POST /api/sources/:sourceId/enhance`
|
||||
- `GET /api/sources/:sourceId/enhance/status`
|
||||
- `GET /api/ls?sourceId=...&path=/&snapshot=latest`
|
||||
- `GET /api/file-info?sourceId=...&id=...`
|
||||
- `GET /healthz`
|
||||
|
||||
Compatibility routes:
|
||||
|
||||
- `POST /api/source/upload`
|
||||
- `GET /api/source/current`
|
||||
|
||||
`/api/source/current` only works when the system currently contains exactly one source. If there are multiple sources, it returns `409 SOURCE_ID_REQUIRED`.
|
||||
|
||||
## Quick start
|
||||
|
||||
```bash
|
||||
npm install
|
||||
copy .env.example .env
|
||||
npm start
|
||||
```
|
||||
|
||||
Environment variables:
|
||||
|
||||
- `PORT`: HTTP port, default `3000`
|
||||
- `APP_DB_PATH`: service-owned SQLite path, default `./data/app.sqlite`
|
||||
- `UPLOAD_DIR`: source storage root, default `./data/sources`
|
||||
- `PREVIEW_MAX_BYTES`: max file size flagged as safe for in-memory preview, default `16777216`
|
||||
|
||||
## Multi-source model
|
||||
|
||||
Each upload is stored under its own directory:
|
||||
|
||||
- `data/sources/<sourceId>/raw.sqlite`
|
||||
- `data/sources/<sourceId>/enhanced.sqlite`
|
||||
- `data/sources/<sourceId>/enhanced.sqlite.tmp`
|
||||
- `data/sources/<sourceId>/work/*`
|
||||
|
||||
Duplicate uploads are deduplicated by SHA-256. If the same task database is uploaded again, the service returns the existing `sourceId` instead of creating a second copy.
|
||||
|
||||
The optional `Duplicati-server.sqlite` upload is stored separately under the upload root and never appears in `/api/sources`. It is only used to resolve friendly task names by matching `path.basename(Backup.DBPath)` against each uploaded task database filename.
|
||||
|
||||
When a server DB is available:
|
||||
|
||||
- source cards prefer `Backup.Name` as `displayName`
|
||||
- `displayNameSource` becomes `server-db`
|
||||
- `matchedBackupName` returns the matched task name
|
||||
|
||||
If no match exists, the source keeps using its original filename as `displayName`.
|
||||
|
||||
## Upload model
|
||||
|
||||
There are now two upload paths:
|
||||
|
||||
1. Task DB upload: `POST /api/sources/upload`
|
||||
2. Server DB upload: `POST /api/server-db/upload`
|
||||
|
||||
Use the task DB upload for random-name Duplicati job databases such as `TMQRJYNADS.sqlite`.
|
||||
|
||||
Use the server DB upload only for `Duplicati-server.sqlite`. Uploading that file to the task DB endpoint is rejected with `422 INVALID_SOURCE_SCHEMA`.
|
||||
|
||||
## Enhancement flow
|
||||
|
||||
Raw Duplicati task databases can browse files immediately, but `/api/file-info` stays gated until archive indexes exist.
|
||||
|
||||
To enhance a source:
|
||||
|
||||
1. Upload a raw task database.
|
||||
2. Save WebDAV base URL, auth mode, username/password, and backup passphrase.
|
||||
3. Start enhancement for that `sourceId`.
|
||||
4. The backend serially downloads each `*.dblock.zip.aes`, decrypts the AES Crypt v2 container, scans the ZIP central directory, and writes supplemental tables into an enhanced SQLite copy.
|
||||
5. After success, the source flips to `ready` and `/api/file-info` starts returning ordered segments and dblock mappings.
|
||||
|
||||
Current runtime assumptions:
|
||||
|
||||
- WebDAV auth supports `basic` and `anonymous`
|
||||
- AES enhancement currently supports AES Crypt v2 volumes
|
||||
- enhancement jobs run serially in-process
|
||||
- credentials are stored in `app.sqlite` and are not returned by the API
|
||||
|
||||
If enhancement fails, source browsing still works, and `/api/file-info` returns `409 ENHANCEMENT_FAILED`.
|
||||
|
||||
## Frontend restore flow
|
||||
|
||||
The runtime restore path stays "thick frontend, thin backend":
|
||||
|
||||
1. `source.html` browses the selected source with `/api/ls`
|
||||
2. `file.html` fetches `/api/file-info`
|
||||
3. the browser asks WebDAV directly for the required `*.dblock.zip.aes`
|
||||
4. the browser decrypts AES Crypt v2 locally
|
||||
5. the browser caches plain ZIP volumes in OPFS
|
||||
6. a root-scope `Service Worker` serves pseudo-Range responses for `<video>`
|
||||
7. downloads write restored bytes directly to disk with `showSaveFilePicker()`
|
||||
|
||||
Important constraints:
|
||||
|
||||
- the backend still does not proxy file bytes
|
||||
- browser-side WebDAV credentials are entered separately on `file.html`
|
||||
- the current v1 range model fetches each needed dblock as a whole encrypted file, then decrypts it in the browser
|
||||
- remote random access is therefore "logical range over cached dblocks", not true O(1) random access inside `.aes`
|
||||
|
||||
Supported browser target:
|
||||
|
||||
- desktop Chromium with `Service Worker`, `OPFS`, `IndexedDB`, `showSaveFilePicker()`, `Web Crypto`, and `DecompressionStream`
|
||||
|
||||
## Supplemental schema
|
||||
|
||||
The enhancement job writes the SQL shape described in [sql/supplemental-schema.sql](C:/Users/nanxun/Documents/jiami/sql/supplemental-schema.sql).
|
||||
|
||||
If you already have a pre-enhanced SQLite file, uploading it is also supported. In that case the source is immediately marked `ready`.
|
||||
|
||||
## Runtime note
|
||||
|
||||
This implementation uses Node 24's built-in `node:sqlite` module. In this Windows environment, native `sqlite3` compilation is not reliable enough, so the service uses the built-in adapter instead.
|
||||
Reference in New Issue
Block a user