duplicati_preview/README.md

Zero-Bandwidth Duplicati API

Thin Node.js + Express metadata service for Duplicati restores. The backend owns only SQLite metadata and enhancement jobs; file bytes stay on WebDAV and are never proxied through the runtime restore APIs.

What changed

• Multiple uploaded Duplicati task databases can now coexist at the same time.
• Every uploaded task library becomes its own sourceId.
• A single uploaded Duplicati-server.sqlite can be stored separately and used to map friendly task names onto task database files.
• Directory browsing and file metadata APIs are now sourceId-scoped.
• The frontend is now split into three pages:
  • /login.html for first-run setup and sign-in
  • / for source management
  • /source.html?sourceId=... for a source workbench
  • /file.html?sourceId=...&id=... for video preview and browser-direct downloads
• Video preview and downloads now pull encrypted dblocks directly from WebDAV in the browser, decrypt them locally, and never proxy the file stream through the backend.
• The backend can build an enhanced SQLite copy per source by fetching remote *.dblock.zip.aes files, decrypting them, and writing:
  • archive_entry_index
  • volume_crypto_cache
  • volume_scan_inventory
  • enhancement_meta

Endpoints

• GET /api/sources
• GET /api/sources/:sourceId
• POST /api/sources/upload
• GET /api/auth/state
• POST /api/auth/setup
• POST /api/auth/login
• POST /api/auth/logout
• GET /api/system/tls
• PUT /api/system/tls
• POST /api/system/tls/upload
• DELETE /api/system/tls/custom-certificate
• GET /api/server-db
• POST /api/server-db/upload
• PUT /api/sources/:sourceId/secrets
• POST /api/sources/:sourceId/enhance
• GET /api/sources/:sourceId/enhance/status
• GET /api/ls?sourceId=...&path=/&snapshot=latest
• GET /api/file-info?sourceId=...&id=...
• GET /healthz

Compatibility routes:

• POST /api/source/upload
• GET /api/source/current

/api/source/current only works when the system currently contains exactly one source. If there are multiple sources, it returns 409 SOURCE_ID_REQUIRED.

Quick start

npm install
copy .env.example .env
npm start

Environment variables:

• PORT: HTTP port, default 3000
• HTTPS_PORT: HTTPS port, default 3443
• APP_DB_PATH: service-owned SQLite path, default ./data/app.sqlite
• UPLOAD_DIR: source storage root, default ./data/sources
• TLS_DIR: certificate storage root, default ./data/tls
• PREVIEW_MAX_BYTES: max file size flagged as safe for in-memory preview, default 16777216
• AUTH_SESSION_TTL_DAYS: login session lifetime in days, default 30

Authentication

The system now requires username/password login before any source-management or restore API can be used.

• On first run, open /login.html and create the initial administrator account.
• After setup, the protected pages /, /source.html, and /file.html redirect unauthenticated users back to the login page.
• API routes return:
  • 401 AUTH_SETUP_REQUIRED when no administrator exists yet
  • 401 AUTH_REQUIRED when there is no valid session
  • 401 SESSION_EXPIRED when a saved session has expired

Implementation notes:

• credentials are stored in the service-owned app.sqlite
• passwords are hashed with Node's built-in scrypt
• sessions are stored server-side in SQLite and issued as HttpOnly cookies
• the same cookie is intentionally usable over both HTTP and HTTPS, so it is not forced to Secure
• /healthz stays public so Docker/Jenkins health checks do not need to log in

HTTPS and certificates

The service now starts both listeners at the same time:

• HTTP on PORT
• HTTPS on HTTPS_PORT

There is no forced redirect. You can open either protocol directly.

TLS behavior:

• if no custom certificate is configured, the app automatically generates and keeps a self-signed certificate so HTTPS always works
• the self-signed certificate is regenerated from the configured primary domain and SAN list when you save self-signed TLS settings
• custom certificates can be configured from the homepage either by pasting PEM text or by uploading PEM files
• deleting the custom certificate switches the service back to the generated self-signed certificate immediately

Certificate files are stored under TLS_DIR, while certificate metadata and the active mode are stored in app.sqlite.

Docker deployment note

Expose both ports when running the container:

• 3000 for HTTP
• 3443 for HTTPS

Example:

docker run --rm -p 3000:3000 -p 3443:3443 duplicati-preview

Multi-source model

Each upload is stored under its own directory:

• data/sources/<sourceId>/raw.sqlite
• data/sources/<sourceId>/enhanced.sqlite
• data/sources/<sourceId>/enhanced.sqlite.tmp
• data/sources/<sourceId>/work/*

Duplicate uploads are deduplicated by SHA-256. If the same task database is uploaded again, the service returns the existing sourceId instead of creating a second copy.

The optional Duplicati-server.sqlite upload is stored separately under the upload root and never appears in /api/sources. It is only used to resolve friendly task names by matching path.basename(Backup.DBPath) against each uploaded task database filename.

When a server DB is available:

• source cards prefer Backup.Name as displayName
• displayNameSource becomes server-db
• matchedBackupName returns the matched task name

If no match exists, the source keeps using its original filename as displayName.

Upload model

There are now two upload paths:

1. Task DB upload: POST /api/sources/upload
2. Server DB upload: POST /api/server-db/upload

Use the task DB upload for random-name Duplicati job databases such as TMQRJYNADS.sqlite.

Use the server DB upload only for Duplicati-server.sqlite. Uploading that file to the task DB endpoint is rejected with 422 INVALID_SOURCE_SCHEMA.

Enhancement flow

Raw Duplicati task databases can browse files immediately, but /api/file-info stays gated until archive indexes exist.

To enhance a source:

1. Upload a raw task database.
2. Save WebDAV base URL, auth mode, username/password, and backup passphrase.
3. Start enhancement for that sourceId.
4. The backend serially downloads each *.dblock.zip.aes, decrypts the AES Crypt v2 container, scans the ZIP central directory, and writes supplemental tables into an enhanced SQLite copy.
5. After success, the source flips to ready and /api/file-info starts returning ordered segments and dblock mappings.

Current runtime assumptions:

• WebDAV auth supports basic and anonymous
• AES enhancement currently supports AES Crypt v2 volumes
• enhancement jobs run serially in-process
• credentials are stored in app.sqlite and are not returned by the API

If enhancement fails, source browsing still works, and /api/file-info returns 409 ENHANCEMENT_FAILED.

Frontend restore flow

The runtime restore path stays "thick frontend, thin backend":

1. source.html browses the selected source with /api/ls
2. file.html fetches /api/file-info
3. the browser asks WebDAV directly for the required *.dblock.zip.aes
4. the browser decrypts AES Crypt v2 locally
5. the browser caches plain ZIP volumes in OPFS
6. a root-scope Service Worker serves pseudo-Range responses for <video>
7. downloads write restored bytes directly to disk with showSaveFilePicker()

Important constraints:

• the backend still does not proxy file bytes
• browser-side WebDAV credentials are entered separately on file.html
• the current v1 range model fetches each needed dblock as a whole encrypted file, then decrypts it in the browser
• remote random access is therefore "logical range over cached dblocks", not true O(1) random access inside .aes

Supported browser target:

• desktop Chromium with Service Worker, OPFS, IndexedDB, showSaveFilePicker(), Web Crypto, and DecompressionStream

Supplemental schema

The enhancement job writes the SQL shape described in sql/supplemental-schema.sql.

If you already have a pre-enhanced SQLite file, uploading it is also supported. In that case the source is immediately marked ready.

Runtime note

This implementation uses Node 24's built-in node:sqlite module. In this Windows environment, native sqlite3 compilation is not reliable enough, so the service uses the built-in adapter instead.