kahing /goofys
a high-performance, POSIX-ish Amazon S3 file system written in Go
★ 5,567⑂ 531👁 115⑃ 170● 644▤ 25
-1
Daily
5,567
stars
531
forks
115
watchers
170
prs
644
issues
25
releases
README
Goofys is a high-performance, POSIX-ish Amazon S3 file system written in Go
Overview
Goofys allows you to mount an S3 bucket as a filey system.
It's a Filey System instead of a File System because goofys strives
for performance first and POSIX second. Particularly things that are
difficult to support on S3 or would translate into more than one
round-trip would either fail (random writes) or faked (no per-file
permission). Goofys does not have an on disk data cache (checkout
catfs), and consistency model is
close-to-open.
Installation
On Linux, install via pre-built binaries.
You may also need to install fuse too if you want to mount it on startup.
On macOS, install via Homebrew:
ShellSession
$ brew cask install osxfuse
$ brew install goofys
Or build from source with Go 1.10 or later:
ShellSession
$ export GOPATH=$HOME/work
$ go get github.com/kahing/goofys
$ go install github.com/kahing/goofys
Usage
ShellSession
$ cat ~/.aws/credentials
[default]
aws_access_key_id = AKID1234567890
aws_secret_access_key = MY-SECRET-KEY
$ $GOPATH/bin/goofys
$ $GOPATH/bin/goofys if you only want to mount objects under a prefix
Users can also configure credentials via the
AWS CLI
or the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables.
To mount an S3 bucket on startup, make sure the credential is
configured for root, and can add this to /etc/fstab:
goofysbucket /mnt/mountpoint fuse _netdev,allow_other,--file-mode=0666,--dir-mode=0777 0 0
See also: Instruction for Azure Blob Storage, Azure Data Lake Gen1, and Azure Data Lake Gen2.
Got more questions? Check out questions other people asked
Benchmark
Using --stat-cache-ttl 1s --type-cache-ttl 1s for goofys
-ostat_cache_expire=1 for s3fs to simulate cold runs. Detail for the
benchmark can be found in
bench.sh. Raw data
is available as well. The test was run on an EC2 m5.4xlarge in us-west-2a
connected to a bucket in us-west-2. Units are seconds.
!Benchmark result
To run the benchmark, configure EC2's instance role to be able to write to $TESTBUCKET, and then do:
ShellSession
$ sudo docker run -e BUCKET=$TESTBUCKET -e CACHE=false --rm --privileged --net=host -v /tmp/cache:/tmp/cache kahing/goofys-bench
result will be written to $TESTBUCKET
See also: cached benchmark result and result on Azure.
License
Copyright (C) 2015 - 2019 Ka-Hing Cheung
Licensed under the Apache License, Version 2.0
Current Status
goofys has been tested under Linux and macOS.
List of non-POSIX behaviors/limitations:
only sequential writes supported
does not store file mode/owner/group
use --(dir|file)-mode or --(uid|gid) options
does not support symlink or hardlink
ctime, atime is always the same as mtime
cannot rename directories with more than 1000 children
unlink returns success even if file is not present
fsync is ignored, files are only flushed on close
Compatibility with non-AWS S3
goofys has been tested with the following nTop issues
#207Issue with auto-mounting on boot39 comments · 2017-07-25#206investigate small write buffer size performance issue17 comments · 2017-07-20#175Goofys needs support for mknod to work in NFS client environments..14 comments · 2017-04-26#527Cannot build from the source - go get fails with accountsRes.Value undefined in api/common/conf_azure.go line 27213 comments · 2020-06-16#377Constant goofys crash13 comments · 2018-11-09#178Will Symlinks ever be supported?12 comments · 2017-05-03#139Make MPU part size configurable12 comments · 2016-12-21#488file and director lost11 comments · 2020-01-15
Latest release
v0.24.0 - mo virus mo problems v0.24.0 2020-04-04A bunch of small fixes all over the place, xattr should work better, fixed writing to the wrong path if we mounted with a prefix and we read the directory where a new file was concurrently being created, fixed some inode caching behavior, also improved handling of azure blob in storage accounts with hierarchical namespace enabled. Also fixed mounting azure blob using only environment variables. Built with go 1.13.1 me on Twitter. !Github Release