kahing /goofys

a high-performance, POSIX-ish Amazon S3 file system written in Go

★ 5,567⑂ 531👁 115⑃ 170● 644▤ 25
GoApache-2.0Created 2015-09-14
Download: v0.24.0 - mo virus mo problemsv0.24.0 ↓0.24.02020-04-04
-1
Daily
5,567
stars
531
forks
115
watchers
170
prs
644
issues
25
releases

README

Goofys is a high-performance, POSIX-ish Amazon S3 file system written in Go






 Overview

Goofys allows you to mount an S3 bucket as a filey system.

It's a Filey System instead of a File System because goofys strives
for performance first and POSIX second. Particularly things that are
difficult to support on S3 or would translate into more than one
round-trip would either fail (random writes) or faked (no per-file
permission). Goofys does not have an on disk data cache (checkout
catfs), and consistency model is
close-to-open.

 Installation

 On Linux, install via pre-built binaries. 
You may also need to install fuse too if you want to mount it on startup.

 On macOS, install via Homebrew:

ShellSession
$ brew cask install osxfuse
$ brew install goofys


 Or build from source with Go 1.10 or later:

ShellSession
$ export GOPATH=$HOME/work
$ go get github.com/kahing/goofys
$ go install github.com/kahing/goofys


 Usage

ShellSession
$ cat ~/.aws/credentials
[default]
aws_access_key_id = AKID1234567890
aws_secret_access_key = MY-SECRET-KEY
$ $GOPATH/bin/goofys  
$ $GOPATH/bin/goofys    if you only want to mount objects under a prefix


Users can also configure credentials via the
AWS CLI
or the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables.

To mount an S3 bucket on startup, make sure the credential is
configured for root, and can add this to /etc/fstab:


goofysbucket   /mnt/mountpoint        fuse     _netdev,allow_other,--file-mode=0666,--dir-mode=0777    0       0


See also: Instruction for Azure Blob Storage, Azure Data Lake Gen1, and Azure Data Lake Gen2.

Got more questions? Check out questions other people asked

 Benchmark

Using --stat-cache-ttl 1s --type-cache-ttl 1s for goofys
-ostat_cache_expire=1 for s3fs to simulate cold runs. Detail for the
benchmark can be found in
bench.sh. Raw data
is available as well. The test was run on an EC2 m5.4xlarge in us-west-2a
connected to a bucket in us-west-2. Units are seconds.

!Benchmark result

To run the benchmark, configure EC2's instance role to be able to write to $TESTBUCKET, and then do:
ShellSession
$ sudo docker run -e BUCKET=$TESTBUCKET -e CACHE=false --rm --privileged --net=host -v /tmp/cache:/tmp/cache kahing/goofys-bench
 result will be written to $TESTBUCKET


See also: cached benchmark result and result on Azure.

 License

Copyright (C) 2015 - 2019 Ka-Hing Cheung

Licensed under the Apache License, Version 2.0

 Current Status

goofys has been tested under Linux and macOS.

List of non-POSIX behaviors/limitations:
   only sequential writes supported
   does not store file mode/owner/group
     use --(dir|file)-mode or --(uid|gid) options
   does not support symlink or hardlink
   ctime, atime is always the same as mtime
   cannot rename directories with more than 1000 children
   unlink returns success even if file is not present
   fsync is ignored, files are only flushed on close

 Compatibility with non-AWS S3

goofys has been tested with the following n

Top issues

Latest release

v0.24.0 - mo virus mo problems v0.24.0 2020-04-04
A bunch of small fixes all over the place, xattr should work better, fixed writing to the wrong path if we mounted with a prefix and we read the directory where a new file was concurrently being created, fixed some inode caching behavior, also improved handling of azure blob in storage accounts with hierarchical namespace enabled. Also fixed mounting azure blob using only environment variables.

Built with go 1.13.1

 me on Twitter.

!Github Release

Snapshot: 2026-11-16T09:23:14Z · Data license Apache-2.0 · JSON