project aqua

Abandon all hope, ye who enter here.
This is a brand new project, most things are probably incomplete and/or terribly broken.

DISCLAIMER: do NOT use this software unless you understand how to maintain frequent backups of a PostgreSQL database as well as backups for a (potentially) large chunk of your filesystem. This software presently provides no built-in facilities for maintaining such backups. You must do them yourself.

P.S: you should be doing backups anyways.

---

aqua is a suite of programs to manage large media libraries using a combination of two ideas:

• Content addressable file storage: all media imported into aqua will be stored on disk according to its unique SHA-256 fingerprint. This allows for rapid (exact) duplicate detection, along with quick file lookups.

• Searchable tags: users can create any number of tags and apply them to any number of hashes. This allows for extremely flexible organization, along with powerful search & query capabilities.

The suite consists of a few major parts:

• aqua: the reference UI, built as a web application. It's very rough, mostly because I've been neglecting it and working on a native (C#/WPF) Windows GUI instead.

• aqua-watch: a small directory watcher which instantaneously imports media into the aqua database when it is written to a directory. This enables a very nice: "save it-then-tag it" workflow whereby you can simply open the app and browse untagged entries.

• aqua-thumbfix: any entries tagged as "THUMB" will be reprocessed by the same thumbnailing engine that aqua-watch uses. This is useful if you've somehow imported a file which aqua could not thumbnail at the time of import. It's also useful if your thumbnail storage has become lost or corrupted.

These two applications currently live in a separate repo, since they're written in C#:

• sister-agnes: simply marks entries in the database which do not exist on disk. This should be fast enough as it only cares about directory listings, but it does enumerate your entire content store. As such it's a fairly expensive operation, and frankly I'm not sure how it performs on non-solid state storage. Use this sparingly.

• aqua_ui_wpf: this lives in a separate repo, it's a C# / WPF application that provides a native frontend and file browser. I'd love to eventually write this in Rust, but from my (admittedly very brief) survey of Rust bindings to GUI toolkits -- it's just not ready.

These programs will share a common database. At the moment aqua requires that you have access to a working PostgreSQL database. (This database is only used for storing metadata about the repository, as such it does not necessarily need to be on the same machine which hosts the media files themselves.)

Getting Started

This is a Rust project, you will need the following pre-requisites:

• a working postgresql database (google instructions for your specific OS)
• rustc and cargo, which can be installed with rustup.
• diesel_cli for setting up the database: $ cargo install diesel_cli
  • See: BUILD-INSTR for information about setting up diesel on various operating systems.
• a .env file, copy sample.env to .env and edit it to suit your environment.

Once these are installed you can build the project as follows:

0. $ diesel setup -- this will initialize a brand new database, you only need to do this once
1. $ diesel migration run -- this will apply the most recent patches to your database schema
2. $ cargo build -- this will build the project
3. $ cargo run --bin aqua -- this will start the web server on port 3000.

At the moment a few routes that can be used include:

• GET /tags/{schema}/{name} lists all entries for a given tag (by name)
• GET /entries/{id} sends the file for a given entry (by id)
• GET /entries/{id} sends a thumbnail for a given entry (by id)
• GET /entries/{id}/tags sends a JSON encoded list of tags for a given entry.

Why?

Frankly: I think modern incarnations of filesystems are flawed by design. What is a filename? It's a human readable tag so you can quickly identify a document alongside its peers. What are its peers? Other listings in the same parent directory, of course. Tell me: does your manilla folder error out if you try to put two copies of the same document in? What about two different documents with the same cover page? Of course it doesn't, because "directories" can contain anything and everything. -- Not only are modern filesystems based upon a rather terrible "office metaphor", they don't even adhere to the fundamental principles of said physical metaphor!

If a filename only exists to aid the human, why must it be unique? Even modern OS shells have admitted that filenames are not in fact unique. This is evidenced in the fact that almost every save dialog, download manager, etc. will automatically re-sequence duplicate filenames.

Consider the digital camera: what meaning do its filenames have? They're usually either some sort of sequence number, e.g: a sequence tag, or they're a timestamp which has already been stored by the filesystem! The timestamp is only appended to work around the limitation that filenames must be unique, it serves no functional purpose!

To that end: why are you limited to only one human readable tag? Libraries have many catalogs which you can search through. Their contents are, effectively, pointers into their shelves. In this way: aqua is exactly like a library. Through schemas and tags you build up a number of indices which are easily searchable by a human. These point to truly unique file entries in a content- addressable filestore.

If aqua is like a library that means that you, my dear user, have become a librarian. Let me just take this opportunity to say: librarians are awesome.

To address these concerns, aqua separates the filesystem into two orthogonal concerns:

• Storing & accessing unique file entries on disk efficiently
• Storing metadata about those files such that it is easily queryable

I've written more about the motivation for aqua, along with the "user story" of a potential command line shell. You can view it in this gist.