Zero-Bandwidth Duplicati API

Thin Node.js + Express metadata service for Duplicati restores. The backend owns only SQLite metadata and enhancement jobs; file bytes stay on WebDAV and are never proxied through the runtime restore APIs.

What changed

• Multiple uploaded Duplicati task databases can now coexist at the same time.
• Every uploaded task library becomes its own sourceId.
• A single uploaded Duplicati-server.sqlite can be stored separately and used to map friendly task names onto task database files.
• Directory browsing and file metadata APIs are now sourceId-scoped.
• The frontend is now split into three pages:
  • / for source management
  • /source.html?sourceId=... for a source workbench
  • /file.html?sourceId=...&id=... for video preview and browser-direct downloads
• Video preview and downloads now pull encrypted dblocks directly from WebDAV in the browser, decrypt them locally, and never proxy the file stream through the backend.
• The backend can build an enhanced SQLite copy per source by fetching remote *.dblock.zip.aes files, decrypting them, and writing:
  • archive_entry_index
  • volume_crypto_cache
  • volume_scan_inventory
  • enhancement_meta

Endpoints

• GET /api/sources
• GET /api/sources/:sourceId
• POST /api/sources/upload
• GET /api/server-db
• POST /api/server-db/upload
• PUT /api/sources/:sourceId/secrets
• POST /api/sources/:sourceId/enhance
• GET /api/sources/:sourceId/enhance/status
• GET /api/ls?sourceId=...&path=/&snapshot=latest
• GET /api/file-info?sourceId=...&id=...
• GET /healthz

Compatibility routes:

• POST /api/source/upload
• GET /api/source/current

/api/source/current only works when the system currently contains exactly one source. If there are multiple sources, it returns 409 SOURCE_ID_REQUIRED.

Quick start

npm install
copy .env.example .env
npm start

Environment variables:

• PORT: HTTP port, default 3000
• APP_DB_PATH: service-owned SQLite path, default ./data/app.sqlite
• UPLOAD_DIR: source storage root, default ./data/sources
• PREVIEW_MAX_BYTES: max file size flagged as safe for in-memory preview, default 16777216

Multi-source model

Each upload is stored under its own directory:

• data/sources/<sourceId>/raw.sqlite
• data/sources/<sourceId>/enhanced.sqlite
• data/sources/<sourceId>/enhanced.sqlite.tmp
• data/sources/<sourceId>/work/*

Duplicate uploads are deduplicated by SHA-256. If the same task database is uploaded again, the service returns the existing sourceId instead of creating a second copy.

The optional Duplicati-server.sqlite upload is stored separately under the upload root and never appears in /api/sources. It is only used to resolve friendly task names by matching path.basename(Backup.DBPath) against each uploaded task database filename.

When a server DB is available:

• source cards prefer Backup.Name as displayName
• displayNameSource becomes server-db
• matchedBackupName returns the matched task name

If no match exists, the source keeps using its original filename as displayName.

Upload model

There are now two upload paths:

1. Task DB upload: POST /api/sources/upload
2. Server DB upload: POST /api/server-db/upload

Use the task DB upload for random-name Duplicati job databases such as TMQRJYNADS.sqlite.

Use the server DB upload only for Duplicati-server.sqlite. Uploading that file to the task DB endpoint is rejected with 422 INVALID_SOURCE_SCHEMA.

Enhancement flow

Raw Duplicati task databases can browse files immediately, but /api/file-info stays gated until archive indexes exist.

To enhance a source:

1. Upload a raw task database.
2. Save WebDAV base URL, auth mode, username/password, and backup passphrase.
3. Start enhancement for that sourceId.
4. The backend serially downloads each *.dblock.zip.aes, decrypts the AES Crypt v2 container, scans the ZIP central directory, and writes supplemental tables into an enhanced SQLite copy.
5. After success, the source flips to ready and /api/file-info starts returning ordered segments and dblock mappings.

Current runtime assumptions:

• WebDAV auth supports basic and anonymous
• AES enhancement currently supports AES Crypt v2 volumes
• enhancement jobs run serially in-process
• credentials are stored in app.sqlite and are not returned by the API

If enhancement fails, source browsing still works, and /api/file-info returns 409 ENHANCEMENT_FAILED.

Frontend restore flow

The runtime restore path stays "thick frontend, thin backend":

1. source.html browses the selected source with /api/ls
2. file.html fetches /api/file-info
3. the browser asks WebDAV directly for the required *.dblock.zip.aes
4. the browser decrypts AES Crypt v2 locally
5. the browser caches plain ZIP volumes in OPFS
6. a root-scope Service Worker serves pseudo-Range responses for <video>
7. downloads write restored bytes directly to disk with showSaveFilePicker()

Important constraints:

• the backend still does not proxy file bytes
• browser-side WebDAV credentials are entered separately on file.html
• the current v1 range model fetches each needed dblock as a whole encrypted file, then decrypts it in the browser
• remote random access is therefore "logical range over cached dblocks", not true O(1) random access inside .aes

Supported browser target:

• desktop Chromium with Service Worker, OPFS, IndexedDB, showSaveFilePicker(), Web Crypto, and DecompressionStream

Supplemental schema

The enhancement job writes the SQL shape described in sql/supplemental-schema.sql.

If you already have a pre-enhanced SQLite file, uploading it is also supported. In that case the source is immediately marked ready.

Runtime note

This implementation uses Node 24's built-in node:sqlite module. In this Windows environment, native sqlite3 compilation is not reliable enough, so the service uses the built-in adapter instead.