Architecture
filex is a self-hosted file manager designed for the "single binary" deploy model: one Go executable that serves both the HTTP API and the embedded admin SPA, talking to pluggable storage / auth / DB drivers.
- Component diagram
- Request lifecycle
- Driver model
- DB schema
- Sync worker
- Realtime hub
- Operation queue
- Embed flow
- Plug & play external services
- Repository layout
Component diagram
┌──────────────────────────────────────────────────────────────────────────┐
│ Browser │
│ │
│ Admin UI (Vue 3 SPA, embedded) Embedded <filex-explorer> WC │
│ - login / config / users - any host page │
│ - storages / sync runs / audit - same backend, different shell │
│ │
│ │ /api/... │ /api/... │
└─────────┼───────────────────────────────────────┼────────────────────────┘
│ │
▼ ▼
┌──────────────────────────────────────────────────────────────────────────┐
│ filex (Go binary) │
│ │
│ ┌────────────────────────────────────────────────────────────────────┐ │
│ │ HTTP layer (chi router, middleware stack) │ │
│ │ logging · auth (JWT/cookie) · rate-limit · CORS · tracing │ │
│ └────────────────────────────────────────────────────────────────────┘ │
│ │ │
│ ┌────────────────────────┼─────────────────────────┐ │
│ ▼ ▼ ▼ │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────────┐ │
│ │ File API │ │ Admin API │ │ Auth API │ │
│ │ /api/files │ │ /api/admin │ │ /api/auth │ │
│ └──────┬──────┘ └──────┬──────┘ └────────┬────────┘ │
│ │ │ │ │
│ ▼ ▼ ▼ │
│ ┌───────────────────┐ ┌──────────────────┐ ┌──────────────────┐ │
│ │ Storage drivers │ │ Search (Bleve) │ │ Auth drivers │ │
│ │ local · s3 · sftp │ │ full-text + │ │ local · oidc · │ │
│ │ webdav · ftp· smb │ │ metadata │ │ ldap · proxy_hdr │ │
│ │ │ │ │ │ windows · pam │ │
│ └─────────┬─────────┘ └─────────┬────────┘ └─────────┬────────┘ │
│ │ │ │ │
│ ▼ ▼ ▼ │
│ ┌──────────────────────────────────────────────────────────────────┐ │
│ │ DB drivers (sqlite / mysql / postgres) │ │
│ │ files (cache) · users · sessions · shares · audit · sync_runs │ │
│ │ storages · uploads · operations · external_services │ │
│ └──────────────────────────────────────────────────────────────────┘ │
│ │ │
│ ┌────────────────────────┼──────────────────────────┐ │
│ ▼ ▼ ▼ │
│ ┌──────────────┐ ┌─────────────────┐ ┌─────────────┐ │
│ │ Sync worker │ │ Thumb pipeline │ │ Op runner │ │
│ │ etag / size+ │ │ image · video · │ │ copy/move/ │ │
│ │ mtime diff + │ │ pdf · office │ │ extract bg │ │
│ └──────┬───────┘ └────────┬────────┘ └─────────────┘ │
│ │ │ │
│ ▼ ▼ │
│ Storage backends ffmpeg / vips / gs / ONLYOFFICE │
│ (local FS, S3, SFTP, (full image only; absent in slim) │
│ WebDAV) │
│ │
└──────────────────────────────────────────────────────────────────────────┘
│
▼ on capability discovery
┌────────────────────────┐
│ External services │
│ (URL = enabled) │
│ - OnlyOffice │
│ - Drawio │
│ - Mermaid (CSR-only) │
└────────────────────────┘Request lifecycle
A typical "list directory" call:
1. GET /api/files/manager?path=/storage1/sub
└─ middleware: log → cors → auth → rate-limit
2. Handler resolves user, asserts read perm on /storage1/sub
3. Looks up cached listing in DB:
SELECT * FROM files
WHERE storage_id = $1 AND parent_path = '/sub'
ORDER BY name LIMIT 1000
(1-5 ms even on 50k entries - indexed on (storage_id, parent_path))
4. For each row, mints a signed thumb_url (HMAC) if applicable
5. Returns JSON { entries, total, storage }Read paths never hit the storage backend. Writes (mkdir, upload, move, delete) write through to both the storage driver and the DB cache so they stay consistent within the request.
If a backend file changed outside filex (e.g. someone uploaded via S3 console), the sync worker reconciles within a configurable interval (default 15 min, per storage).
A write that filex performed itself is not left to that: every write surface calls the shared post-write gate, which indexes the file, emits the webhook event, queues the antivirus scan and announces the change on the realtime hub. The sync exists to discover what filex did not do.
Driver model
Three driver families, each with a Go interface + a built-in registry. Add a new driver = implement the interface + register at init.
A storage driver no longer has to be compiled in: a plugin is a separate program filex speaks HTTP/JSON to, and it is registered in the same registry as
plugin:<name>- with the capabilities it declares probed before anything is registered at all.
Storage driver
type Storage interface {
List(ctx context.Context, path string) ([]Entry, error)
Stat(ctx context.Context, path string) (Entry, error)
Open(ctx context.Context, path string, off, n int64) (io.ReadCloser, error)
Put(ctx context.Context, path string, r io.Reader, size int64) (string, error) // returns etag
Delete(ctx context.Context, path string) error
Mkdir(ctx context.Context, path string) error
Move(ctx context.Context, from, to string) error
// Optional:
PresignPut(ctx context.Context, path string, parts int) ([]string, error)
Sync(ctx context.Context, since time.Time, cb SyncCallback) error
}Built-ins: local, s3, sftp, webdav, ftp, smb - plus any installed plugin, registered as plugin:<name>.
Auth driver
type Authenticator interface {
Login(ctx context.Context, creds Credentials) (User, error)
}
type Provisioner interface {
Provision(ctx context.Context, claims map[string]any) (User, error)
}Built-ins: local (bcrypt), oidc, ldap, proxy-header, windows and pam (the last two only from Admin → Identity providers). Multiple drivers can be enabled simultaneously. A driver that redirects (oidc) gets its own button; the four that read a password (local, ldap, windows, pam) share the one form and are chained - auth.LoginChain tries them in the configured order and the first to accept wins, so local first keeps admin@local answerable while the directory is down. The same chain feeds the file protocols through protocolauth.Directory.
DB driver
The DB layer is split:
- Migrations via goose, as one tree per dialect -
backend/db/migrations/{sqlite,postgres,mysql}/, each embedded by its own driver (Driver.MigrationsFS()). Same numbers, same table and column names, hand-written per engine: a reserved word, aDEFAULTan engine will not take on aTEXTcolumn, or a statement block MySQL's driver refuses are not things one guarded file can express. A schema comparison holds the three trees to the same names - see DATABASES.md. - Queries are hand-written behind the one
db.Storeinterface. SQLite and MySQL share an implementation - the placeholders and column names are identical, and the one construct they genuinely disagree on is rewritten at call time (ON CONFLICT … excluded.x→ON DUPLICATE KEY UPDATE … VALUES(x), a no-op on SQLite). PostgreSQL has its own, because$nplaceholders andRETURNINGinstead ofLastInsertIdare not a rewrite.
Built-ins:
sqlite-modernc.org/sqlite(pure Go, no CGO)mysql-go-sql-driver/mysqlpostgres-jackc/pgx/v5
DB schema
The tables an operator or an integrator is most likely to need. ⚠ This is a map, not the schema: the real one has roughly twice as many tables and 51 migrations at v0.43.0, and some names here are the older ones (files is nodes, operations is ops_queue, thumbs is thumbnails). Read backend/db/migrations/ for the truth. Names use singular_or_plural to match Laravel conventions of the sister projects.
| Table | Purpose |
|---|---|
users | account row; bcrypt hash for local; OIDC sub/iss |
sessions | session cookies + Bearer token cache |
storages | named storage instances + driver config (encrypted) |
files | DB-cached file tree, indexed by (storage_id, parent_path). Every row also says whose it is - owner_id (who put it here), last_actor_id (who touched it last) and external_upload (it arrived through an anonymous drop link). NULL on either id means system: nobody put it here through filex, and no user row is invented for it. A trashed row also says who put it in the trash (deleted_by, NULL when nobody is recorded) |
file_metadata | lightweight extended attrs (mime override, label, color tag) |
shares | public links: token, PIN, expiry, max_downloads, owner |
share_downloads | individual download events (audit) |
uploads | multipart upload state (parts, etags, expires) |
operations | long-running ops (copy, extract, archive); progress, status |
sync_runs | per-storage sync history with counts and errors |
audit_events | all auditable user actions |
external_services | OnlyOffice/Drawio config + last_check |
thumbs | thumbnail cache index (bytes live on disk; released when the node is purged, and a reconciler sweeps orphans at boot and every FILEX_THUMBS_SWEEP_INTERVAL; at boot it also re-fits pages cached at full size before 0.41.0) |
migration_lock | goose migration lock |
ER diagram (high-level):
users 1───1 sessions
users 1──* shares 1──* share_downloads
users 1──* audit_events
storages 1──* files (parent_path indexed)
storages 1──* sync_runs
files 1──1 thumbs
users 1──* uploads
users 1──* operations
external_services (config row per service)Sync worker
Background goroutine that reconciles each storage with the DB cache.
loop:
for each storage with sync_interval elapsed:
run := create_sync_run(storage_id)
seen := {}
# each directory is listed and applied with the storage's row gate held
# alone (#192); a folder its parent's listing showed but that is gone when
# the walk reaches it counts as a listing that failed, not an empty one
for entry in storage.Sync(since=last_run_started):
if entry.path is inside /.filex-trash/, /.versions/ or /.thumbs/:
skip # filex's own bookkeeping,
# -- not catalogue content
seen.add(entry.path)
if row is staged/failed: # a staged upload never flipped
if no staging session and entry has the committed size and is not
older than the commit: # the bytes did land
mark stored, queue antivirus scan
else:
touch seen_at only # keep the committed size/time
continue
upsert(files, storage_id, entry) # keeps the row's mime when the
# listing has none (object stores)
# anything LIVE inside .filex-trash/ is a defect, and both kinds are fixed
for f in db.files where storage_id=$id and path under .filex-trash/ and not deleted:
if f.storage_key points OUTSIDE the trash: # a deletion something revived
soft_delete(f) # -> back in the trash
else: # a row minted for trash bytes
hard_delete(f) # -> dropped, bytes untouched
# ANY row under .versions/ or .thumbs/ (live or trashed) was minted by an
# older walk: dropped, deepest first, with its search document
for f in db.files where storage_id=$id and path under .versions/ or .thumbs/:
hard_delete(f) # catalogue only, bytes untouched
# a row an older version soft-deleted WHERE IT STOOD (path outside the
# trash) holds no bytes in the trash: not a trash entry, dropped (#74)
drop_rows(db.files where storage_id=$id and deleted and path not in .filex-trash/)
# tombstone pass - a node not seen this run is a CANDIDATE, not a verdict
if seen < 0.7 * last_ok_run.seen: # the whole listing looks wrong
skip the pass entirely # (a failed/aborted run is no baseline)
# from here to the drops, the storage's row gate is held alone: no
# rename, move, delete or restore filex is making is half way (#192)
for f in db.files where storage_id=$id and seen_at < run_started:
if f.path is inside filex's own trees: # never, whatever else went wrong
keep
if f.path is below a folder whose listing failed, or below a folder
its parent's listing showed but that was gone when the walk got there:
keep # unseen says nothing there
if f moved or went since it was listed: # read again first
keep # judged where it is next pass
if f.transfer_state != "stored": # filex never put the bytes there
keep
elif storage.Stat(f.path) is found: # the listing missed it (folders
keep # are asked too)
elif Stat failed for any other reason: # we could not check
keep
else:
gone.add(f)
drop_rows(gone) # genuinely gone → out of the
# catalogue, NOT into the trash
drop_rows(rows): # deepest first
for f in rows + (rows deleted in place below each folder in rows):
if f is a folder and any row still names it as its parent:
keep # the cascade must not take it
elif f, read again, no longer stands where it was confirmed gone:
keep # a move re-homed it in between
else:
hard_delete(f) # quota released; shares, versions,
# comments, tags cascade; search
# doc + thumbnail released
log INFO "dropped the row of an object gone from the storage"
finish_sync_run(run) # on a context the run's own
# cancellation cannot reach: "ok",
# "failed", or "aborted" if cut short
# at worker start, before any run: a row still open belongs to a process
# that is gone
close every sync_run with no finished_at as "aborted"A folder rescan (POST /api/admin/storages/{id}/sync?path=) runs the same walk from one catalogued folder's row (Worker.RescanFolder): the one-pass listing asks for that subtree only, the tombstone candidates are the stale rows strictly below the folder (an exact, character-counted prefix - never LIKE), the whole-listing guard compares against the folder's own live row count, and a listing that failed part-way skips the tombstone pass. It takes the storage's one-run lock and writes no sync_runs row.
⚠ Absence from a listing is not proof of deletion, and answering it with "move to trash" turns any unrelated bug into lost data - which is exactly what happened in GitHub #16, where uploads that never reached S3 were trashed by the next run. So the pass has to be right about the deletion, and three things stand between a missing object and the trash:
- A whole-listing guard. If the run saw more than 30 % fewer entries than the previous one, no deletions happen at all - that shape is a backend glitch, not a mass deletion.
- Did filex ever store it? A node whose
transfer_stateis notstoredis in staging or is a failed upload. Its absence from the backend is the expected state and is never a deletion. This also closes the race where a run landing between publishing a node and finishing its transfer would trash a healthy upload. - A direct second opinion. For the remaining candidates the driver is asked with
Stat. Only a definite not-found counts; an object the listing missed butStatcan see is kept, and so is one that could not be checked at all (permissions, timeout, 503) - "I could not check" must never read as "it is gone".
A file genuinely deleted in the bucket still leaves filex, so deletions made outside filex are still reflected - but it leaves the catalogue, not into the Trash (issue #74). The Trash is where filex keeps the bytes of what was deleted in filex; an object deleted elsewhere has no bytes there, and up to 0.47 its row was listed in the Trash with a Restore that could bring nothing back. Because dropping a row cannot be undone, folders are confirmed by Stat too, nothing below a folder whose listing failed is a candidate, and a folder row goes only once no row names it as its parent. Stat runs only for candidates that survive step 1, so a healthy sync costs nothing extra. Every row dropped is said in the server log at INFO (the first 50 of a pass one by one, then a count).
The row gate
⚠⚠ A rename, a move, a delete and a restore change a storage in two steps, and the scan must not judge the catalogue between them (issue #192).
The bytes move first (Mover.Move, trash.Put) and the rows follow (protocolsync.MoveRows, SoftDeleteAndRetag, the restore). In between, the catalogue says one thing and the storage another, and nothing on either side can tell that apart from a change made outside filex. A scan running in that gap was wrong about rows that were fine. Measured in the 0.53 test run (e2e 159 and 172, WebKit, under load): a queued folder rename met the fsnotify scan two seconds after a write. The scan listed the folder's parent before the bytes moved and the folder itself after; "not found" was read as "listed, nothing in it", every row below the folder was confirmed gone by a Stat of the path it still named, and dropped. The renamed folder opened empty, and its files came back on the next scan as new rows, without their shares, versions or comments. Nothing was logged.
Three rules now stand there:
- A folder its parent's listing showed, gone when the walk lists it, is a folder the walk could not list - never an empty one. Nothing below it is a candidate this pass; the next pass sees the storage as it is.
- Each storage has one row gate (
internal/rowgate). A two-step change holds it shared from its first byte to its last row, on every surface that makes one (the table below). The scan holds it alone while it lists one directory and applies the listing, and while the tombstone pass confirms and drops; the lazy catalogue holds it alone for each folder it reconciles. A judgement therefore sees such a change either wholly before it or wholly after it. Changes do not wait for each other, and a judgement holds the gate for one directory, one batch of tombstone candidates (64) or one lazy folder at a time, so a rename never waits for a whole scan. - A candidate is read again before it is judged and again before it is dropped, and kept when it moved or went in between. This covers part of what the gate does not: a second filex process on the same database (a replica beside the main instance) does not see this process's gate.
Who holds the gate (since 0.55, issue #201; up to 0.54 only the queue and the explorer's rename and move did, and rule 3 alone stood in for the rest):
| Surface | Two-step changes | Where the gate is taken |
|---|---|---|
| Operations queue | rename, move, delete into the trash, restore, a cross-storage move's source delete | ops (runRename, runOne, runRestore, crossTransfer) |
| Explorer, inside the request | rename, move, delete (to the trash, or for good on a storage without one) | vfRename, vfMove, deleteItem (rowgate.ChangeCtx / rowgate.MoveCtx, waited for on the request) |
| WebDAV, SFTP, FTPS, NFS, S3 gateway | rename / MOVE, delete into the trash, delete for good | protocolsync.Syncer.Relocate, Discard, Purge, PurgeTree - one shared frame for every protocol server |
AI surface and MCP (file_move, file_delete) | move within a storage, delete, a cross-storage move's source delete | aiOps.Move, aiOps.Delete, aiOps.moveAcross |
| Drafts | saving a draft to its place, discarding it (into the trash, or for good on a storage without one) | SaveDraft, DiscardDraft (discardDraftGated) |
| Trash | a restore made inside the request (POST /api/files/manager/restore without queued=1, one entry or a batch) | Trash.restoreGated |
| Antivirus | moving an infected file into the trash | AntivirusScanner.quarantine |
A write is not a two-step change for the catalogue: the row is upserted by its path after the bytes land, so a scan in between catalogues the object at that same path and the write's upsert lands on the same row. Uploads, saves, copies, archive extraction, version restores and the vault's object writes therefore take no gate.
⚠ Take the gate at the outermost step of a change (the queue's job, the request handler, the protocol verb), never inside the shared helpers it calls, never twice for one change, never while holding a judgement, and never inside a database transaction (the package comment of internal/rowgate says why).
Changes come first; a judgement steps back (since 0.55, sec055). Up to the first 0.55 cut the gate was a sync.RWMutex, which favours the writer: one scan waiting behind one long change - a folder moved or deleted object by object over WebDAV or SFTP, which can take hours - kept every later change on that storage out, and the operations queue's single worker with them (every tenant's copies, moves, deletes and upload commits), until the long change let go. Now:
- A judgement never queues ahead of a change. It takes the gate only while no change holds it; while one does it steps back, without keeping any change out, for at most 30 seconds (
sync.GateWait,rowgate.JudgeWithin), and is then deferred: the walk stops and the run is recorded with the reason, the tombstone pass leaves its remaining candidates to the next pass, a lazy folder is reconciled at its next turn. A long change postpones its storage's scan, never anything else. (A storage under a steady stream of overlapping changes is not judged until the stream pauses: its scan is deferred, never wrong.) - A change waits only while a judgement holds the gate, and on its request's context (
rowgate.MoveCtx). A client that gives up while it waits takes nothing and changes nothing. Once the gate is held the change detaches from the client (storage.DetachMutation) and runs to its end. - What was checked before the wait is asked again under the gate. The destination found free (WebDAV
Overwrite: Fand the destination x/net/webdav removed forOverwrite: T, SFTP's v3 rename, the explorer's and the queue'sNameTaken, a move'sMoveDest, a draft's target) and the source as it was stat'ed (WebDAV MOVE and DELETE) are asked again once the change holds the gate (storage.StillFree,storage.StillAsSeen,protocolsync.Check). A name taken meanwhile is refused (409name_takenin the explorer, 409TARGET_TAKENfor a draft,os.ErrExiston SFTP and the AI surface), a source that changed meanwhile is refused too (storage.ErrChangedMeanwhile, 412 where an HTTP handler answers it); nothing is replaced and nothing new is trashed. x/net/webdav answers every refused MOVE or DELETE with its own status (403 for a MOVE). The explorer's and the queue's moves pick their de-collided name under the gate. - The operations queue never stands in a judged gate. Before a job that holds a gate (move, delete, rename, restore) the worker asks whether a judgement holds that storage's gate (
rowgate.Judged); if so the job goes back to the queue as it was, the storage's later jobs stay where they are for that round (one storage's jobs keep their order), and the other storages' jobs run (ops/gate_lane.go, both the main worker and the finishing lane). A move to another storage asks again before each item's copy: with its source storage being judged, the job goes back with the items it has not reached (their count stays on the row). Once an item's bytes are on the far side, the delete of its source waits for the gate on the job's own context, so Cancel ends the wait and keeps the source; it is not put back there, since a second run would copy the item again. The agent's move to another storage (aiOps.moveAcross) waits on the caller's context before its copy (rowgate.Await) and on the detached one after. - A folder deleted for good object by object holds the gate a batch at a time (
protocolsync.Syncer.PurgeTree, WebDAV on an object store without a trash; 100 objects per hold): each batch's bytes and rows go together, and a scan between two batches sees what is true. - A folder moved or trashed object by object fences its prefixes instead of holding the gate (since 0.55,
rowgate.FenceCtx). Between two objects a scan would find part of the folder at its new name with no rows and part of its rows at a path whose objects have left - exactly what the gate hides - but all of that lies under two prefixes: the folder's path and its destination's. So the change fences those (storage.FenceFor/FenceAtdecide: a folder, on a storage that moves one object at a time,storage.ObjectByObject), and the scan goes on everywhere else. Under the gate, each judgement reads the fence set once (rowgate.Fences: a lock-free load, then one map lookup for the directory's fenced children and one comparison per fenced prefix): the walk does not list a fenced directory and leaves a fenced child out of its parent's listing (both join the unlisted, so nothing below them is a tombstone candidate), the tombstone pass skips a row under a fence, and the lazy catalogue defers a fenced folder and counts a fenced child as seen. Once the rows have followed, the fence opens and counts as a finished change (rowgate.Moves), and the next pass sees the folder where the change put it. A fence is set only while no judgement holds the gate, so a judgement either ended before the first object moved or sees it. It is let go bydeferhowever the change ends - an error, a Cancel, a panic. A change holds ONE of the two, never both (rowgate.HoldCtx); a judgement never waits for a fence, so there is no order between them to get wrong. Who fences: the queue's rename, move, delete and restore (holdFor; a move fences its destination once it has picked the name,fenceToo; a restore its trash key and its original path,RestoreFencer) and a cross-storage move's source delete, the explorer's rename, move and delete and the trash's restore inside the request (restoreGated, through the sameTrash.RestoreFence), the protocol servers'RelocateandDiscard, and the agent's move, delete and cross-storage source delete. The protocol servers ask the catalogue whether the source is a folder and, for a source with no row yet (a lazily catalogued folder nobody has opened, objects another tool wrote), the storage (oneStat); aStatthat fails leaves the change on the gate. A file, and every change on a storage with real folders (local, SFTP, SMB, WebDAV, FTP: one rename call), holds the gate as before; so does a purge (one call, orPurgeTreea batch at a time). The antivirus quarantine moves one file and holds the gate, and leaves a file under a fenced prefix alone: its folder is being moved or trashed, the quarantine fails and the queue's retry finds the file where the change put it. - A long hold is visible.
filex_rowgate_oldest_change_seconds,filex_rowgate_changes,filex_rowgate_long_changes_totalandfilex_rowgate_deferred_judgements_totalper storage, and for fencesfilex_rowgate_fencesandfilex_rowgate_oldest_fence_seconds(METRICS.md), and a warning in the log when a change lets go after holding the gate for a minute or more.
The antivirus quarantine reads the infected file's row again once it holds the gate: a file renamed, moved, trashed or rewritten while it was scanned is not quarantined from where it was (the move of the old path used to answer "not found", which counted as quarantined, and the row was sent to the trash while the infected bytes stayed live at the new name); the job fails and the queue's retry scans the row as it is now.
An object store's one-pass listing (storage.TreeWalker) is a picture of the storage taken when the walk starts. The walk reads directories from it only as long as no two-step change has finished on the storage since before the picture was taken (rowgate.Moves, which a fence's release counts too); from the first one on, it asks the storage for each directory, so a folder renamed during the walk is not catalogued again at its old name.
The walk and the trash
⚠⚠ The walk does not enter .filex-trash/, and it never un-deletes a row.
Deleting a file in filex is a rename: the bytes move to .filex-trash/<unix>-<rand>__<name> and the node row is soft-deleted and retagged to that key (TRASH-VERSIONING.md). Quarantine is the same operation - the antivirus job produces an identical row (PROTECTION.md) - so nothing in the catalogue tells the two apart.
The walk used to descend into the trash bucket, find an object with no live row, find the soft-deleted one, and clear deleted_at on it. A file the user deleted therefore came back at the next pass, and an infected file left quarantine at the next pass: the security control expired on a timer nobody set. It fired on every pass, on every driver, with no condition attached - poll, fsnotify and driver-watch all funnel into the same full RunOnce walk, so there is no incremental mode in which it did not happen.
Two rules now stand in its place:
.filex-trash/is skipped. The rows for everything in there already exist, retagged to the very keys sitting on the storage, and they are maintained by the trash service - restore, retention purge - never by a listing. (A consequence:seenno longer counts trashed objects, so a storage whose trash held more than 30 % of its objects trips the whole-listing guard once on the first pass after upgrading.)A trashed row is never revived. When an object turns up at a path where a trashed row still sits - an out-of-band restore, or simply a new file with an old name - the trashed row stays in the trash and the object is catalogued as a new node, which is indexed and treated as new everywhere. Bytes that reappear at a path are not the file that was deleted there; reviving the row would hand them another file's identity, version history, comments and shares, and nothing downstream would ever look at them again. Migration
00032makes the(storage_id, path_hash)unique index live-only so the new row is possible - the same thing00018did for(storage_id, parent_id, name)."Treated as new" means catalogued, indexed and scanned. The walk hands every file it newly catalogues - and every file whose content drifted - to the antivirus queue, so a file that appears on a storage out of band is scanned whether it lands on a fresh path or on one a trashed row once held. See PROTECTION.md.
Anything found live inside .filex-trash/ is repaired on the spot, which is what heals an install that ran an earlier version: a revived deletion is soft-deleted again (keeping storage_key, so restore still knows where it came from), and a row the old walk minted for the trash's own bytes is dropped outright. Bytes are never touched by either.
.versions/ and .thumbs/ are filex's too, and the walk skips them the same way (versioning.IsInternalTree, anchored at the storage root - the one list the sync walk, a cross-storage copy and the public share pages all ask). The walk used to skip only the trash, so a full scan minted a system-owned row for every snapshot folder and file. Unseen once the walk stopped listing them, the folder rows would have gone to the trash in place - confirmGone has no object to Stat for a directory - and purging a trashed folder deletes its prefix on the backend: every version of every file. So every row under those two trees, live or trashed, is dropped from the catalogue before the tombstone pass runs (deepest first, search documents included, backend untouched), and the tombstone pass itself refuses any row inside an internal tree, so a cleanup that failed is a cleanup deferred, never a version history in the trash.
storage.Sync compares the backend's etag when the driver reports one (S3, WebDAV PROPFIND) and the object's size and modification time when it does not - which is local, SFTP, SMB and FTP. Either way the walk is a full re-list; the fingerprint decides which rows it updates. What each of those catches, and the two kinds of external change that slip past the second one, is in STORAGE.md → Drift detection.
Realtime hub
An open explorer is told what changed rather than asking. internal/realtime holds one hub per process; a browser mints a short-lived ticket at POST /api/files/ws-ticket, opens GET /api/ws, and subscribes to the folders it has on screen. It gets two kinds of frame: presence (who else is here, what they have focused) and change (something in this folder moved).
Two properties matter to everything else in this document:
- An explorer with a healthy socket does not poll. The 12 s re-listing in
useRealtime.tsis the fallback for a socket that failed. So a write that emits no frame is not "slow to appear" - it does not appear at all until the user navigates. - Bursts are coalesced on the leading edge. The first change in a quiet room goes out immediately and unmodified; everything after it is merged into one frame per window (200 ms, doubling to 1.5 s while the burst continues, reset when the folder goes quiet). A merged frame carries
count. Nothing is dropped - a burst always ends with a frame reflecting its final state.
Every write surface reaches the hub through the same post-write gate, and the five protocol servers reach it through internal/protocolsync, the one package they already shared. The contract, the frame shapes and the debounce advice for integrators are in REALTIME.md.
Operation queue
ops_queue is the durable work list behind antivirus scans, content extraction, thumbnails, and async copy/move/delete. Three drivers implement one interface - sqlite (in the app DB), postgres (SELECT … FOR UPDATE SKIP LOCKED) and redis - and a shared contract test runs the same suite against all three.
Three columns carry more meaning than their names suggest:
priority- claimedORDER BY priority DESC, enqueued_at ASCon all three drivers, which is what keeps an interactive scan ahead of a twenty-thousand file first import. ⚠ The redis driver ignored it entirely until v0.34.0; its pending set is now a sorted set whose score encodes priority and arrival, claimed by a single Lua script, converted from the old LIST at startup. Downgrading past that conversion is not supported.not_before- a delayed operation, used by the editor's debounced save-scan. Deliberately a row rather than a timer in the process, so a deploy does not take every pending scan with it.dedup_key- unique among pending rows only, so "one pending scan per file" holds while a save burst is arriving and is released the moment a worker claims the scan.
Embed flow
The Go binary embeds two front-end bundles via //go:embed:
backend/embed/
├── admin/ # web/dist (Vue 3 SPA)
│ ├── index.html
│ ├── assets/...
└── web/ # packages/webcomponent/dist
├── filex.js
└── filex.cssIn Go:
//go:embed embed/admin/*
var adminFS embed.FS
//go:embed embed/web/*
var webFS embed.FSThese are mounted at:
| Route | Source | Notes |
|---|---|---|
/admin/* | embed/admin/ | SPA - fallthrough to index.html for client routing |
/drive/* | embed/admin/ | The same SPA, end-user front door. vue-router reads its history base from the prefix that served the document, so a session that starts here stays on /drive/… |
/files/edit | embed/admin/ | The standalone editor, outside both prefixes (openPageBase) |
/embed.js | embed/web/filex.js | Web Component bundle |
/embed.css | embed/web/filex.css | Optional (the WC ships CSS-in-JS too) |
Build flow:
1. pnpm -r --filter='./packages/*' build # core, webcomponent, react
2. pnpm --filter='./web' build # admin SPA
3. node scripts/sync-embed.mjs # copy dist into backend/embed
4. cd backend && go build ./cmd/filex # //go:embed picks up the dirsThe same flow drives Docker (multi-stage), goreleaser (CI artifacts), and local development (pnpm run build:all).
Plug & play external services
OnlyOffice, Drawio and the converter are used only when they're configured; Mermaid renders client-side and needs no service at all. The capability is exposed via GET /api/capabilities and the front-end hides UI affordances when a capability is false.
⚠ The external_services row is the single runtime source of truth, read on every use behind a one-second cache that the admin PATCH invalidates. It used to be a snapshot of env/YAML taken once at boot, which is how an operator could configure OnlyOffice in the admin UI, watch Test answer 200, and still get 503 onlyoffice not configured from the editor forever (GitHub #17). Env and config.yaml are declarative configuration for that row: a service they name is re-asserted onto it at every boot and reported env_managed: true, so an admin-UI edit to one of those applies immediately and is reverted at the next start.
admin PATCH /api/admin/external/onlyoffice ──┐
▼
external_services
│
read on every use (1 s cache,
invalidated by the PATCH)
▼
editor descriptor, capabilities,
the converter URL handed to AI/MCPAdding a new external service is mechanical:
- Add row in
config.yamlschema +external_servicestable seed. - Implement health probe (
internal/external/<svc>/probe.go). - Surface it in
/api/capabilities. - Front-end conditional UI block.
Repository layout
filemanager/
├── backend/ # Go service
│ ├── cmd/filex/ # main.go (cobra CLI)
│ ├── internal/
│ │ ├── api/ # chi handlers
│ │ ├── auth/ # auth drivers + middleware
│ │ ├── capability/ # /api/capabilities aggregator
│ │ ├── config/ # YAML + env loader
│ │ ├── db/ # Store interface + one driver per dialect
│ │ ├── model/ # domain types
│ │ ├── search/ # Bleve wrapper
│ │ ├── server/ # HTTP wiring
│ │ ├── share/ # public-link handlers
│ │ ├── storage/ # storage drivers
│ │ ├── sync/ # background sync worker
│ │ └── thumb/ # thumbnail pipeline
│ ├── db/
│ │ ├── migrations/ # goose .sql - sqlite/ postgres/ mysql/
│ │ └── queries/ # sqlc definitions - not generated from;
│ │ # the drivers hold the real SQL
│ ├── embed/ # populated by sync-embed.mjs
│ ├── go.mod
│ └── sqlc.yaml
│
├── packages/
│ ├── core/ # @brftech/filex-core (Vue SFC)
│ ├── webcomponent/ # @brftech/filex (WC wrapper)
│ └── react/ # @brftech/filex-react (lit/react)
│
├── web/ # Vue 3 admin SPA (embedded)
├── demo/ # standalone HTML demos
├── docker/
│ ├── Dockerfile # the :latest / :full image, ~241 MB
│ ├── Dockerfile.slim # the :slim image, ~62 MB, binary only
│ └── Dockerfile.local # local hot-fix builds from a host dist
│
├── scripts/
│ └── sync-embed.mjs # web/dist + wc/dist → backend/embed
│
├── docs/ # this directory
│
├── .gitlab/ # CI helpers (placeholders)
├── .gitlab-ci.yml # private pipeline: the test chain (CONTRIBUTING)
├── .goreleaser.yml # release matrix
├── docker-compose.yml # full stack with profiles
├── package.json # workspace root
├── pnpm-workspace.yaml # pnpm workspaces
└── README.md