GitHub - sigoden/dufs: A file server that supports static serving, uploading, searching, accessing control, webdav... (original) (raw)

Dufs is a distinctive utility file server that supports static serving, uploading, searching, accessing control, webdav...

Features

Serve static files
Download folder as zip file
Upload files and folders (Drag & Drop)
Create/Edit/Search files
Resumable/partial uploads/downloads
Access control
Support https
Support webdav
Easy to use with curl

Install

With cargo

With docker

docker run -v `pwd`:/data -p 5000:5000 --rm sigoden/dufs /data -A

With Homebrew

Binaries on macOS, Linux, Windows

Download from Github Releases, unzip and add dufs to your $PATH.

CLI

Dufs is a distinctive utility file server - https://github.com/sigoden/dufs

Usage: dufs [OPTIONS] [serve-path]

Arguments:
  [serve-path]  Specific path to serve [default: .]

Options:
  -c, --config <file>        Specify configuration file
  -b, --bind <addrs>         Specify bind address or unix socket
  -p, --port <port>          Specify port to listen on [default: 5000]
      --path-prefix <path>   Specify a path prefix
      --hidden <value>       Hide paths from directory listings, e.g. tmp,*.log,*.lock
  -a, --auth <rules>         Add auth roles, e.g. user:pass@/dir1:rw,/dir2
  -A, --allow-all            Allow all operations
      --allow-upload         Allow upload files/folders
      --allow-delete         Allow delete files/folders
      --allow-search         Allow search files/folders
      --allow-symlink        Allow symlink to files/folders outside root directory
      --allow-archive        Allow download folders as archive file
      --allow-hash           Allow ?hash query to get file sha256 hash
      --enable-cors          Enable CORS, sets `Access-Control-Allow-Origin: *`
      --render-index         Serve index.html when requesting a directory, returns 404 if not found index.html
      --render-try-index     Serve index.html when requesting a directory, returns directory listing if not found index.html
      --render-spa           Serve SPA(Single Page Application)
      --assets <path>        Set the path to the assets directory for overriding the built-in assets
      --log-format <format>  Customize http log format
      --log-file <file>      Specify the file to save logs to, other than stdout/stderr
      --compress <level>     Set zip compress level [default: low] [possible values: none, low, medium, high]
      --completions <shell>  Print shell completion script for <shell> [possible values: bash, elvish, fish, powershell, zsh]
      --tls-cert <path>      Path to an SSL/TLS certificate to serve with HTTPS
      --tls-key <path>       Path to the SSL/TLS certificate's private key
  -h, --help                 Print help
  -V, --version              Print version

Examples

Serve current working directory in read-only mode

Allow all operations like upload/delete/search/create/edit...

Only allow upload operation

Serve a specific directory

Serve a single file

Serve a single-page application like react/vue

Serve a static website with index.html

Require username/password

Listen on specific host:ip

Listen on unix socket

Use https

dufs --tls-cert my.crt --tls-key my.key

API

Upload a file

curl -T path-to-file http://127.0.0.1:5000/new-path/path-to-file

Download a file

curl http://127.0.0.1:5000/path-to-file # download the file curl http://127.0.0.1:5000/path-to-file?hash # retrieve the sha256 hash of the file

Download a folder as zip file

curl -o path-to-folder.zip http://127.0.0.1:5000/path-to-folder?zip

Delete a file/folder

curl -X DELETE http://127.0.0.1:5000/path-to-file-or-folder

Create a directory

curl -X MKCOL http://127.0.0.1:5000/path-to-folder

Move the file/folder to the new path

curl -X MOVE http://127.0.0.1:5000/path -H "Destination: http://127.0.0.1:5000/new-path"

List/search directory contents

curl http://127.0.0.1:5000?q=Dockerfile # search for files, similar to find -name Dockerfile curl http://127.0.0.1:5000?simple # output names only, similar to ls -1 curl http://127.0.0.1:5000?json # output paths in json format

With authorization (Both basic or digest auth works)

curl http://127.0.0.1:5000/file --user user:pass # basic auth curl http://127.0.0.1:5000/file --user user:pass --digest # digest auth

Resumable downloads

curl -C- -o file http://127.0.0.1:5000/file

Resumable uploads

upload_offset=$(curl -I -s http://127.0.0.1:5000/file | tr -d '\r' | sed -n 's/content-length: //p') dd skip=$upload_offset if=file status=none ibs=1 |
curl -X PATCH -H "X-Update-Range: append" --data-binary @- http://127.0.0.1:5000/file

Health checks

curl http://127.0.0.1:5000/__dufs__/health

Advanced Topics

Access Control

Dufs supports account based access control. You can control who can do what on which path with --auth/-a.

dufs -a admin:admin@/:rw -a guest:guest@/
dufs -a user:pass@/:rw,/dir1 -a @/

Use @ to separate the account and paths. No account means anonymous user.
Use : to separate the username and password of the account.
Use , to separate paths.
Use path suffix :rw/:ro set permissions: read-write/read-only. :ro can be omitted.

-a admin:admin@/:rw: admin has complete permissions for all paths.
-a guest:guest@/: guest has read-only permissions for all paths.
-a user:pass@/:rw,/dir1: user has read-write permissions for /*, has read-only permissions for /dir1/*.
-a @/: All paths is publicly accessible, everyone can view/download it.

Auth permissions are restricted by dufs global permissions. If dufs does not enable upload permissions via --allow-upload, then the account will not have upload permissions even if it is granted read-write(:rw) permissions.

Hashed Password

DUFS supports the use of sha-512 hashed password.

Create hashed password:

$ openssl passwd -6 123456 # or mkpasswd -m sha-512 123456 666tWMB51u6Kb2ui3wd$5gVHP92V9kZcMwQeKTjyTRgySsYJu471Jb1I6iHQ8iZ6s07GgCIO69KcPBRuwPE5tDq05xMAzye0NxVKuJdYs/

Use hashed password:

dufs -a 'admin:$6$tWMB51u6Kb2ui3wd$5gVHP92V9kZcMwQeKTjyTRgySsYJu471Jb1I6iHQ8iZ6s07GgCIO69KcPBRuwPE5tDq05xMAzye0NxVKuJdYs/@/:rw'

The hashed password contains $6, which can expand to a variable in some shells, so you have to use single quotes to wrap it.

Two important things for hashed passwords:

Dufs only supports sha-512 hashed passwords, so ensure that the password string always starts with $6$ .
Digest authentication does not function properly with hashed passwords.

Hide Paths

Dufs supports hiding paths from directory listings via option --hidden <glob>,....

dufs --hidden .git,.DS_Store,tmp

The glob used in --hidden only matches file and directory names, not paths. So --hidden dir1/file is invalid.

dufs --hidden '.' # hidden dotfiles dufs --hidden '/' # hidden all folders dufs --hidden '.log,.lock' # hidden by exts dufs --hidden '.log' --hidden '.lock'

Log Format

Dufs supports customize http log format with option --log-format.

The log format can use following variables.

variable	description
$remote_addr	client address
$remote_user	user name supplied with authentication
$request	full original request line
$status	response status
http_∣arbitraryrequestheaderfield.examples:http_	arbitrary request header field. examples: http_∣arbitraryrequestheaderfield.examples:http_user_agent, $http_referer

The default log format is '$time_iso8601 <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>l</mi><mi>o</mi><msub><mi>g</mi><mi>l</mi></msub><mi>e</mi><mi>v</mi><mi>e</mi><mi>l</mi><mo>−</mo></mrow><annotation encoding="application/x-tex">log_level - </annotation></semantics></math>loglevel−remote_addr "$request" $status.

2022-08-06T06:59:31+08:00 INFO - 127.0.0.1 "GET /" 200

A json log format is also supported.

dufs --log-format '{"time":"$time_local","addr":"$remote_addr","uri":"$request_uri", "method":"$request_method","status":$status}'

{"time":"2022-08-06T06:59:31+08:00","addr":"127.0.0.1","uri":"/", "method":"GET","status":200}

Disable http log

Log user-agent

dufs --log-format '$remote_addr "$request" <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>s</mi><mi>t</mi><mi>a</mi><mi>t</mi><mi>u</mi><mi>s</mi></mrow><annotation encoding="application/x-tex">status </annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6151em;"></span><span class="mord mathnormal">s</span><span class="mord mathnormal">t</span><span class="mord mathnormal">a</span><span class="mord mathnormal">t</span><span class="mord mathnormal">u</span><span class="mord mathnormal">s</span></span></span></span>http_user_agent'

2022-08-06T06:53:55+08:00 INFO - 127.0.0.1 "GET /" 200 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36

Log remote-user

dufs --log-format '$remote_addr <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>r</mi><mi>e</mi><mi>m</mi><mi>o</mi><mi>t</mi><msub><mi>e</mi><mi>u</mi></msub><mi>s</mi><mi>e</mi><mi>r</mi><mi mathvariant="normal">&quot;</mi></mrow><annotation encoding="application/x-tex">remote_user &quot;</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8444em;vertical-align:-0.15em;"></span><span class="mord mathnormal">re</span><span class="mord mathnormal">m</span><span class="mord mathnormal">o</span><span class="mord mathnormal">t</span><span class="mord"><span class="mord mathnormal">e</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.1514em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">u</span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span class="mord mathnormal" style="margin-right:0.02778em;">ser</span><span class="mord">&quot;</span></span></span></span>request" $status' -a /@admin:admin -a /folder1@user1:pass1

2022-08-06T07:04:37+08:00 INFO - 127.0.0.1 admin "GET /" 200

Environment variables

All options can be set using environment variables prefixed with DUFS_.

[serve-path]                DUFS_SERVE_PATH="."
    --config <file>         DUFS_CONFIG=config.yaml
-b, --bind <addrs>          DUFS_BIND=0.0.0.0
-p, --port <port>           DUFS_PORT=5000
    --path-prefix <path>    DUFS_PATH_PREFIX=/dufs
    --hidden <value>        DUFS_HIDDEN=tmp,*.log,*.lock
-a, --auth <rules>          DUFS_AUTH="admin:admin@/:rw|@/" 
-A, --allow-all             DUFS_ALLOW_ALL=true
    --allow-upload          DUFS_ALLOW_UPLOAD=true
    --allow-delete          DUFS_ALLOW_DELETE=true
    --allow-search          DUFS_ALLOW_SEARCH=true
    --allow-symlink         DUFS_ALLOW_SYMLINK=true
    --allow-archive         DUFS_ALLOW_ARCHIVE=true
    --allow-hash            DUFS_ALLOW_HASH=true
    --enable-cors           DUFS_ENABLE_CORS=true
    --render-index          DUFS_RENDER_INDEX=true
    --render-try-index      DUFS_RENDER_TRY_INDEX=true
    --render-spa            DUFS_RENDER_SPA=true
    --assets <path>         DUFS_ASSETS=./assets
    --log-format <format>   DUFS_LOG_FORMAT=""
    --log-file <file>       DUFS_LOG_FILE=./dufs.log
    --compress <compress>   DUFS_COMPRESS=low
    --tls-cert <path>       DUFS_TLS_CERT=cert.pem
    --tls-key <path>        DUFS_TLS_KEY=key.pem

Configuration File

You can specify and use the configuration file by selecting the option --config <path-to-config.yaml>.

The following are the configuration items:

serve-path: '.' bind: 0.0.0.0 port: 5000 path-prefix: /dufs hidden:

tmp
'*.log'
'*.lock' auth:
admin:admin@/:rw
user:pass@/src:rw,/share
'@/' # According to the YAML spec, quoting is required. allow-all: false allow-upload: true allow-delete: true allow-search: true allow-symlink: true allow-archive: true allow-hash: true enable-cors: true render-index: true render-try-index: true render-spa: true assets: ./assets/ log-format: '$remote_addr "$request" statusstatus statushttp_user_agent' log-file: ./dufs.log compress: low tls-cert: tests/data/cert.pem tls-key: tests/data/key_pkcs1.pem

Customize UI

Dufs allows users to customize the UI with your own assets.

dufs --assets my-assets-dir/

If you only need to make slight adjustments to the current UI, you copy dufs's assets directory and modify it accordingly. The current UI doesn't use any frameworks, just plain HTML/JS/CSS. As long as you have some basic knowledge of web development, it shouldn't be difficult to modify.

Your assets folder must contains a index.html file.

index.html can use the following placeholder variables to retrieve internal data.

__INDEX_DATA__: directory listing data
__ASSETS_PREFIX__: assets url prefix

A customized 404.html page is also supported.

License

dufs is made available under the terms of either the MIT License or the Apache License 2.0, at your option.

See the LICENSE-APACHE and LICENSE-MIT files for license details.