# Zero-Bandwidth Duplicati API Thin `Node.js + Express` metadata service for Duplicati restores. The backend owns only SQLite metadata and enhancement jobs; file bytes stay on WebDAV and are never proxied through the runtime restore APIs. ## What changed - Multiple uploaded Duplicati task databases can now coexist at the same time. - Every uploaded task library becomes its own `sourceId`. - A single uploaded `Duplicati-server.sqlite` can be stored separately and used to map friendly task names onto task database files. - Directory browsing and file metadata APIs are now `sourceId`-scoped. - The frontend is now split into three pages: - `/login.html` for first-run setup and sign-in - `/` for source management - `/source.html?sourceId=...` for a source workbench - `/file.html?sourceId=...&id=...` for video preview and browser-direct downloads - Video preview and downloads now pull encrypted dblocks directly from WebDAV in the browser, decrypt them locally, and never proxy the file stream through the backend. - The backend can build an enhanced SQLite copy per source by fetching remote `*.dblock.zip.aes` files, decrypting them, and writing: - `archive_entry_index` - `volume_crypto_cache` - `volume_scan_inventory` - `enhancement_meta` ## Endpoints - `GET /api/sources` - `GET /api/sources/:sourceId` - `POST /api/sources/upload` - `GET /api/auth/state` - `POST /api/auth/setup` - `POST /api/auth/login` - `POST /api/auth/logout` - `GET /api/system/tls` - `PUT /api/system/tls` - `POST /api/system/tls/upload` - `DELETE /api/system/tls/custom-certificate` - `GET /api/server-db` - `POST /api/server-db/upload` - `PUT /api/sources/:sourceId/secrets` - `POST /api/sources/:sourceId/enhance` - `GET /api/sources/:sourceId/enhance/status` - `GET /api/ls?sourceId=...&path=/&snapshot=latest` - `GET /api/file-info?sourceId=...&id=...` - `GET /healthz` Compatibility routes: - `POST /api/source/upload` - `GET /api/source/current` `/api/source/current` only works when the system currently contains exactly one source. If there are multiple sources, it returns `409 SOURCE_ID_REQUIRED`. ## Quick start ```bash npm install copy .env.example .env npm start ``` Environment variables: - `PORT`: HTTP port, default `3000` - `HTTPS_PORT`: HTTPS port, default `3443` - `APP_DB_PATH`: service-owned SQLite path, default `./data/app.sqlite` - `UPLOAD_DIR`: source storage root, default `./data/sources` - `TLS_DIR`: certificate storage root, default `./data/tls` - `PREVIEW_MAX_BYTES`: max file size flagged as safe for in-memory preview, default `16777216` - `AUTH_SESSION_TTL_DAYS`: login session lifetime in days, default `30` ## Authentication The system now requires username/password login before any source-management or restore API can be used. - On first run, open `/login.html` and create the initial administrator account. - After setup, the protected pages `/`, `/source.html`, and `/file.html` redirect unauthenticated users back to the login page. - API routes return: - `401 AUTH_SETUP_REQUIRED` when no administrator exists yet - `401 AUTH_REQUIRED` when there is no valid session - `401 SESSION_EXPIRED` when a saved session has expired Implementation notes: - credentials are stored in the service-owned `app.sqlite` - passwords are hashed with Node's built-in `scrypt` - sessions are stored server-side in SQLite and issued as `HttpOnly` cookies - the same cookie is intentionally usable over both HTTP and HTTPS, so it is not forced to `Secure` - `/healthz` stays public so Docker/Jenkins health checks do not need to log in ## HTTPS and certificates The service now starts both listeners at the same time: - HTTP on `PORT` - HTTPS on `HTTPS_PORT` There is no forced redirect. You can open either protocol directly. TLS behavior: - if no custom certificate is configured, the app automatically generates and keeps a self-signed certificate so HTTPS always works - the self-signed certificate is regenerated from the configured primary domain and SAN list when you save self-signed TLS settings - custom certificates can be configured from the homepage either by pasting PEM text or by uploading PEM files - deleting the custom certificate switches the service back to the generated self-signed certificate immediately Certificate files are stored under `TLS_DIR`, while certificate metadata and the active mode are stored in `app.sqlite`. ## Docker deployment note Expose both ports when running the container: - `3000` for HTTP - `3443` for HTTPS Example: ```bash docker run --rm -p 3000:3000 -p 3443:3443 duplicati-preview ``` ## Multi-source model Each upload is stored under its own directory: - `data/sources//raw.sqlite` - `data/sources//enhanced.sqlite` - `data/sources//enhanced.sqlite.tmp` - `data/sources//work/*` Duplicate uploads are deduplicated by SHA-256. If the same task database is uploaded again, the service returns the existing `sourceId` instead of creating a second copy. The optional `Duplicati-server.sqlite` upload is stored separately under the upload root and never appears in `/api/sources`. It is only used to resolve friendly task names by matching `path.basename(Backup.DBPath)` against each uploaded task database filename. When a server DB is available: - source cards prefer `Backup.Name` as `displayName` - `displayNameSource` becomes `server-db` - `matchedBackupName` returns the matched task name If no match exists, the source keeps using its original filename as `displayName`. ## Upload model There are now two upload paths: 1. Task DB upload: `POST /api/sources/upload` 2. Server DB upload: `POST /api/server-db/upload` Use the task DB upload for random-name Duplicati job databases such as `TMQRJYNADS.sqlite`. Use the server DB upload only for `Duplicati-server.sqlite`. Uploading that file to the task DB endpoint is rejected with `422 INVALID_SOURCE_SCHEMA`. ## Enhancement flow Raw Duplicati task databases can browse files immediately, but `/api/file-info` stays gated until archive indexes exist. To enhance a source: 1. Upload a raw task database. 2. Save WebDAV base URL, auth mode, username/password, and backup passphrase. 3. Start enhancement for that `sourceId`. 4. The backend serially downloads each `*.dblock.zip.aes`, decrypts the AES Crypt v2 container, scans the ZIP central directory, and writes supplemental tables into an enhanced SQLite copy. 5. After success, the source flips to `ready` and `/api/file-info` starts returning ordered segments and dblock mappings. Current runtime assumptions: - WebDAV auth supports `basic` and `anonymous` - AES enhancement currently supports AES Crypt v2 volumes - enhancement jobs run serially in-process - credentials are stored in `app.sqlite` and are not returned by the API If enhancement fails, source browsing still works, and `/api/file-info` returns `409 ENHANCEMENT_FAILED`. ## Frontend restore flow The runtime restore path stays "thick frontend, thin backend": 1. `source.html` browses the selected source with `/api/ls` 2. `file.html` fetches `/api/file-info` 3. the browser asks WebDAV directly for the required `*.dblock.zip.aes` 4. the browser decrypts AES Crypt v2 locally 5. the browser caches plain ZIP volumes in OPFS 6. a root-scope `Service Worker` serves pseudo-Range responses for `