duplicati_preview/README.md

# Zero-Bandwidth Duplicati API

Thin `Node.js + Express` metadata service for Duplicati restores. The backend owns only SQLite metadata and enhancement jobs; file bytes stay on WebDAV and are never proxied through the runtime restore APIs.

## What changed

- Multiple uploaded Duplicati task databases can now coexist at the same time.
- Every uploaded task library becomes its own `sourceId`.
- A single uploaded `Duplicati-server.sqlite` can be stored separately and used to map friendly task names onto task database files.
- Directory browsing and file metadata APIs are now `sourceId`-scoped.
- The frontend is now split into three pages:
  - `/login.html` for first-run setup and sign-in
  - `/` for source management
  - `/source.html?sourceId=...` for a source workbench
  - `/file.html?sourceId=...&id=...` for video preview and browser-direct downloads
- Video preview and downloads now pull encrypted dblocks directly from WebDAV in the browser, decrypt them locally, and never proxy the file stream through the backend.
- The backend can build an enhanced SQLite copy per source by fetching remote `*.dblock.zip.aes` files, decrypting them, and writing:
  - `archive_entry_index`
  - `volume_crypto_cache`
  - `volume_scan_inventory`
  - `enhancement_meta`

## Endpoints

- `GET /api/sources`
- `GET /api/sources/:sourceId`
- `POST /api/sources/upload`
- `GET /api/auth/state`
- `POST /api/auth/setup`
- `POST /api/auth/login`
- `POST /api/auth/logout`
- `GET /api/system/tls`
- `PUT /api/system/tls`
- `POST /api/system/tls/upload`
- `DELETE /api/system/tls/custom-certificate`
- `GET /api/server-db`
- `POST /api/server-db/upload`
- `PUT /api/sources/:sourceId/secrets`
- `POST /api/sources/:sourceId/enhance`
- `GET /api/sources/:sourceId/enhance/status`
- `GET /api/ls?sourceId=...&path=/&snapshot=latest`
- `GET /api/file-info?sourceId=...&id=...`
- `GET /healthz`

Compatibility routes:

- `POST /api/source/upload`
- `GET /api/source/current`

`/api/source/current` only works when the system currently contains exactly one source. If there are multiple sources, it returns `409 SOURCE_ID_REQUIRED`.

## Quick start

```bash
npm install
copy .env.example .env
npm start
```

Environment variables:

- `PORT`: HTTP port, default `3000`
- `HTTPS_PORT`: HTTPS port, default `3443`
- `APP_DB_PATH`: service-owned SQLite path, default `./data/app.sqlite`
- `UPLOAD_DIR`: source storage root, default `./data/sources`
- `TLS_DIR`: certificate storage root, default `./data/tls`
- `PREVIEW_MAX_BYTES`: max file size flagged as safe for in-memory preview, default `16777216`
- `AUTH_SESSION_TTL_DAYS`: login session lifetime in days, default `30`

## Authentication

The system now requires username/password login before any source-management or restore API can be used.

- On first run, open `/login.html` and create the initial administrator account.
- After setup, the protected pages `/`, `/source.html`, and `/file.html` redirect unauthenticated users back to the login page.
- API routes return:
  - `401 AUTH_SETUP_REQUIRED` when no administrator exists yet
  - `401 AUTH_REQUIRED` when there is no valid session
  - `401 SESSION_EXPIRED` when a saved session has expired

Implementation notes:

- credentials are stored in the service-owned `app.sqlite`
- passwords are hashed with Node's built-in `scrypt`
- sessions are stored server-side in SQLite and issued as `HttpOnly` cookies
- the same cookie is intentionally usable over both HTTP and HTTPS, so it is not forced to `Secure`
- `/healthz` stays public so Docker/Jenkins health checks do not need to log in

## HTTPS and certificates

The service now starts both listeners at the same time:

- HTTP on `PORT`
- HTTPS on `HTTPS_PORT`

There is no forced redirect. You can open either protocol directly.

TLS behavior:

- if no custom certificate is configured, the app automatically generates and keeps a self-signed certificate so HTTPS always works
- the self-signed certificate is regenerated from the configured primary domain and SAN list when you save self-signed TLS settings
- custom certificates can be configured from the homepage either by pasting PEM text or by uploading PEM files
- deleting the custom certificate switches the service back to the generated self-signed certificate immediately

Certificate files are stored under `TLS_DIR`, while certificate metadata and the active mode are stored in `app.sqlite`.

## Docker deployment note

Expose both ports when running the container:

- `3000` for HTTP
- `3443` for HTTPS

Example:

```bash
docker run --rm -p 3000:3000 -p 3443:3443 duplicati-preview
```

## Multi-source model

Each upload is stored under its own directory:

- `data/sources/<sourceId>/raw.sqlite`
- `data/sources/<sourceId>/enhanced.sqlite`
- `data/sources/<sourceId>/enhanced.sqlite.tmp`
- `data/sources/<sourceId>/work/*`

Duplicate uploads are deduplicated by SHA-256. If the same task database is uploaded again, the service returns the existing `sourceId` instead of creating a second copy.

The optional `Duplicati-server.sqlite` upload is stored separately under the upload root and never appears in `/api/sources`. It is only used to resolve friendly task names by matching `path.basename(Backup.DBPath)` against each uploaded task database filename.

When a server DB is available:

- source cards prefer `Backup.Name` as `displayName`
- `displayNameSource` becomes `server-db`
- `matchedBackupName` returns the matched task name

If no match exists, the source keeps using its original filename as `displayName`.

## Upload model

There are now two upload paths:

1. Task DB upload: `POST /api/sources/upload`
2. Server DB upload: `POST /api/server-db/upload`

Use the task DB upload for random-name Duplicati job databases such as `TMQRJYNADS.sqlite`.

Use the server DB upload only for `Duplicati-server.sqlite`. Uploading that file to the task DB endpoint is rejected with `422 INVALID_SOURCE_SCHEMA`.

## Enhancement flow

Raw Duplicati task databases can browse files immediately, but `/api/file-info` stays gated until archive indexes exist.

To enhance a source:

1. Upload a raw task database.
2. Save WebDAV base URL, auth mode, username/password, and backup passphrase.
3. Start enhancement for that `sourceId`.
4. The backend serially downloads each `*.dblock.zip.aes`, decrypts the AES Crypt v2 container, scans the ZIP central directory, and writes supplemental tables into an enhanced SQLite copy.
5. After success, the source flips to `ready` and `/api/file-info` starts returning ordered segments and dblock mappings.

Current runtime assumptions:

- WebDAV auth supports `basic` and `anonymous`
- AES enhancement currently supports AES Crypt v2 volumes
- enhancement jobs run serially in-process
- credentials are stored in `app.sqlite` and are not returned by the API

If enhancement fails, source browsing still works, and `/api/file-info` returns `409 ENHANCEMENT_FAILED`.

## Frontend restore flow

The runtime restore path stays "thick frontend, thin backend":

1. `source.html` browses the selected source with `/api/ls`
2. `file.html` fetches `/api/file-info`
3. the browser asks WebDAV directly for the required `*.dblock.zip.aes`
4. the browser decrypts AES Crypt v2 locally
5. the browser caches plain ZIP volumes in OPFS
6. a root-scope `Service Worker` serves pseudo-Range responses for `<video>`
7. downloads write restored bytes directly to disk with `showSaveFilePicker()`

Important constraints:

- the backend still does not proxy file bytes
- browser-side WebDAV credentials are entered separately on `file.html`
- the current v1 range model fetches each needed dblock as a whole encrypted file, then decrypts it in the browser
- remote random access is therefore "logical range over cached dblocks", not true O(1) random access inside `.aes`

Supported browser target:

- desktop Chromium with `Service Worker`, `OPFS`, `IndexedDB`, `showSaveFilePicker()`, `Web Crypto`, and `DecompressionStream`

## Supplemental schema

The enhancement job writes the SQL shape described in [sql/supplemental-schema.sql](C:/Users/nanxun/Documents/jiami/sql/supplemental-schema.sql).

If you already have a pre-enhanced SQLite file, uploading it is also supported. In that case the source is immediately marked `ready`.

## Runtime note

This implementation uses Node 24's built-in `node:sqlite` module. In this Windows environment, native `sqlite3` compilation is not reliable enough, so the service uses the built-in adapter instead.