JNR-FUSE library for using FUSE from Java

Hi!

In this article, I’ll tell you how to implement userspace file system using Java without a line of kernel space code. I’ll also show you how to connect Java and native code without writing C code and save maximum performance.

Originally, I posted this article on habrahabr (in Russian).

FUSE

First of all, it is important to understand what FUSE is. FUSE - FileSystem in Userspace - helps users without any privileges to create their file system without a necessity to write code in a kernel space.

This is possible because the filesystem code runs in userspace. And the FUSE module is just a bridge between the kernel API and your code. FUSE was officially included in the Linux source tree in 2.6.14.

So, you can easily create your own filesystem (here is the simplest example). There are a lot of areas where you can use it. For example, you can quickly write a FS where Github or DropBox would be the backend.

Or, let’s imagine, you have a business application where all user files are stored in a database. But your client wants to have access to them from a filesystem on the server. Of course to duplicate files in the filesystem and the database is the wrong decision. And here FUSE comes in; you just need a little FUSE program which handles all user requests to the directory and redirects them to the database.

Java and native code

So far so good. But implementation of FUSE starts from "include header <fuse.h>". But your business application is written in Java. Obviously, it is necessary to communicate with the native code.

JNI

The standard tool for a native communication in Java is JNI. But it brings a lot of complexity. Especially because for using FUSE we need a lot of callbacks from the native code to Java. And "write once" is suffering in this case (However, in the case of FUSE it is not so important). If you try to find projects that implement a FUSE wrapper using JNI you will find a few projects, but they have been obsolete for a long time. And their API is very inconvenient.

JNA

Another option is JNA library. JNA (Java Native Access) gives you the possibility to get access to the native code easily without using JNI only by using Java code. It is very easy; you just need to declare an interface which matches the native code and get an implementation through "Native.loadLibrary". And that’s all. Another advantage of JNA is very detailed documentation. The project is alive, and it is in active development.

Moreover, a good project implementing FUSE using JNA already exists. But JNA has a lot of problems with performance. JNA is reflection based, so jumping from native code to Java code with data conversion is very expensive. It is not so important if native calls are rare. But a FS has a lot of native calls. The only way to speed up fuse-jna is to read files using big chunks, but it doesn’t always work. For example, when you have no access to a client’s code or when all files are small. Obviously, we need a library that combines JNI performance and JNA convenience.

JNR

And here is where JNR (Java Native Runtime) comes in. JNR, like JNA, is based on libffi library, but it uses a bytecode generation instead of reflection. And as a result, JNR achieves excellent performance. The information about JNR is very limited. The most detailed piece of information is Charles Nutter’s talk on JVMLS 2013. But despite the lack of information, JNR is already a big ecosystem actively used by JRuby. A lot of jnr-based libraries such as unix-sockets and posix-api are actively used by different projects.

JNR is a library which became a basis for the development of JEP 191 - Foreign Function Interface, which is targeted for Java 10. In comparison to JNA, JNR has no proper documentation, and you need to look for answers in the source code. That is the main reason for this mini-guide.

Specialties of writing code using Java Native Runtime

Function binding

The simplest binding

import jnr.ffi.*;
import jnr.ffi.types.pid_t;

/**
 * Gets the process ID of the current process, and that of its parent.
 */
public class Getpid {
    public interface LibC  {
        public @pid_t long getpid();
        public @pid_t long getppid();
    }

    public static void main(String[] args) {
        LibC libc = LibraryLoader.create(LibC.class).load("c");

        System.out.println("pid=" + libc.getpid() + " parent pid=" + libc.getppid());
    }
}

Here we are the loading java library which matches the native interface by name.

In the case of FUSE, we need an interface with method fuse_main_real where FuseOperations structure with all callbacks passes as an argument.

public interface LibFuse {
    int fuse_main_real(int argc, String argv[], FuseOperations op, int op_size, Pointer user_data);
}

Implementation of structures

Often you need to work with structure which is located at a certain address, for example with fuse_bufvec structure:

struct fuse_bufvec {
    size_t count;
    size_t idx;
    size_t off;
    struct fuse_buf buf[1];
};

For its implementation, we need to make a successor of jnr.ffi.Struct.

import jnr.ffi.*;

public class FuseBufvec extends Struct {
    public FuseBufvec(jnr.ffi.Runtime runtime) {
        super(runtime);
    }
    public final size_t count = new size_t();
    public final size_t idx = new size_t();
    public final size_t off = new size_t();
    public final FuseBuf buf = inner(new FuseBuf(getRuntime()));
}

After that, you have to set proper callback implementation into the getattr field.

fuseOperations.getattr.set((path, stbuf) -> 0);

Enum

Enum implementation is not so obvious as other parts of the library. You need to inherit your enum from jnr.ffi.util.EnumMapper.IntegerEnum and implement method intValue

enum fuse_buf_flags {
    FUSE_BUF_IS_FD    = (1 << 1),
    FUSE_BUF_FD_SEEK    = (1 << 2),
    FUSE_BUF_FD_RETRY    = (1 << 3),
};

public enum FuseBufFlags implements EnumMapper.IntegerEnum {
    FUSE_BUF_IS_FD(1 << 1),
    FUSE_BUF_FD_SEEK(1 << 2),
    FUSE_BUF_FD_RETRY(1 << 3);

    private final int value;

    FuseBufFlags(int value) {
        this.value = value;
    }

    @Override
    public int intValue() {
        return value;
    }
}

Work with memory

• For working with direct memory wrapper jnr.ffi.Pointer exists.
• You can allocate memory using jnr.ffi.Memory
• The entry point of JNR API learning is jnr.ffi.Runtime

This knowledge is enough to implement a simple cross-platform wrapper for some native library.

JNR-FUSE

What I’ve implemented is a FUSE wrapper in my project jnr-fuse. Previously I used fuse-jna, but it was a bottleneck in the FS implementation. During the development, I tried to save compatibility with the native interface (<fuse.h>).

For implementing your own file system, you need to extend ru.serce.jnrfuse.FuseStubFS and implement several methods. Fuse_operations contain a lot of methods, but for getting your FS up and running, you just need to implement several methods. It is very easy, here are some examples.

Currently, only Linux is supported (x86 and x64).

Library exists in JCenter.

Gradle

repositories {
    jcenter()
}

dependencies {
    compile 'com.github.serceman:jnr-fuse:0.1'
}

Maven

<repositories>
    <repository>
        <id>central</id>
        <name>bintray</name>
        <url>http://jcenter.bintray.com</url>
    </repository>
</repositories>

<dependencies>
    <dependency>
        <groupId>com.github.serceman</groupId>
        <artifactId>jnr-fuse</artifactId>
        <version>0.1</version>
    </dependency>
</dependencies>

JNR-FUSE and FUSE-JNA performance comparison

In my case the FS was read-only, and I was interested in throughput. The performance will mostly depend on your FS implementation, so if you already use fuse-jna, you’ll be able to change it easily to jnr-fuse, write a test, and to see the performance difference on your workload. (It will be helpful anyway because we all love to achieve new levels of performance, right?)

In order to show performance the difference, I moved MemoryFS implementation from fuse-jna to jnr-fuse with minimal changes and ran a reading test. For the test, I used fio framework.

[readtest]
blocksize=4k
directory=/tmp/mnt/
rw=randread
direct=1
buffered=0
ioengine=libaio
time_based=60
size=16M
runtime=60

serce@SerCe-FastLinux:~/git/jnr-fuse/bench$ fio read.ini
readtest: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=1
fio-2.1.3
Starting 1 process
readtest: Laying out IO file(s) (1 file(s) / 16MB)
Jobs: 1 (f=1): [r] [100.0% done] [24492KB/0KB/0KB /s] [6123/0/0 iops] [eta 00m:00s]
readtest: (groupid=0, jobs=1): err= 0: pid=10442: Sun Jun 21 14:49:13 2015
read: io=1580.2MB, bw=26967KB/s, iops=6741, runt= 60000msec
slat (usec): min=46, max=29997, avg=146.55, stdev=327.68
clat (usec): min=0, max=69, avg= 0.47, stdev= 0.66
lat (usec): min=47, max=30002, avg=147.26, stdev=327.88
clat percentiles (usec):
| 1.00th=[ 0], 5.00th=[ 0], 10.00th=[ 0], 20.00th=[ 0],
| 30.00th=[ 0], 40.00th=[ 0], 50.00th=[ 0], 60.00th=[ 1],
| 70.00th=[ 1], 80.00th=[ 1], 90.00th=[ 1], 95.00th=[ 1],
| 99.00th=[ 2], 99.50th=[ 2], 99.90th=[ 3], 99.95th=[ 12],
| 99.99th=[ 14]
bw (KB /s): min=17680, max=32606, per=96.09%, avg=25913.26, stdev=3156.20
lat (usec): 2=97.95%, 4=1.96%, 10=0.02%, 20=0.06%, 50=0.01%
lat (usec): 100=0.01%
cpu: usr=1.98%, sys=5.94%, ctx=405302, majf=0, minf=28
IO depths: 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit: 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete: 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued: total=r=404511/w=0/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
READ: io=1580.2MB, aggrb=26967KB/s, minb=26967KB/s, maxb=26967KB/s, mint=60000msec, maxt=60000msec

serce@SerCe-FastLinux:~/git/jnr-fuse/bench$ fio read.ini
readtest: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=1
fio-2.1.3
Starting 1 process
readtest: Laying out IO file(s) (1 file(s) / 16MB)
Jobs: 1 (f=1): [r] [100.0% done] [208.5MB/0KB/0KB /s] [53.4K/0/0 iops] [eta 00m:00s]
readtest: (groupid=0, jobs=1): err= 0: pid=10153: Sun Jun 21 14:45:17 2015
read: io=13826MB, bw=235955KB/s, iops=58988, runt= 60002msec
slat (usec): min=6, max=23671, avg=15.80, stdev=19.97
clat (usec): min=0, max=1028, avg= 0.37, stdev= 0.78
lat (usec): min=7, max=23688, avg=16.29, stdev=20.03
clat percentiles (usec):
| 1.00th=[ 0], 5.00th=[ 0], 10.00th=[ 0], 20.00th=[ 0],
| 30.00th=[ 0], 40.00th=[ 0], 50.00th=[ 0], 60.00th=[ 0],
| 70.00th=[ 1], 80.00th=[ 1], 90.00th=[ 1], 95.00th=[ 1],
| 99.00th=[ 1], 99.50th=[ 1], 99.90th=[ 2], 99.95th=[ 2],
| 99.99th=[ 10]
lat (usec): 2=99.88%, 4=0.10%, 10=0.01%, 20=0.01%, 50=0.01%
lat (usec): 100=0.01%, 250=0.01%
lat (msec): 2=0.01%
cpu: usr=9.33%, sys=34.01%, ctx=3543137, majf=0, minf=28
IO depths: 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit: 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete: 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued: total=r=3539449/w=0/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
READ: io=13826MB, aggrb=235955KB/s, minb=235955KB/s, maxb=235955KB/s, mint=60002msec, maxt=60002msec

The only information this test shows us is the difference in reading a file using fuse-jna and jnr-fuse, but it gives us an understanding of the level of performance difference of JNA and JNR. If you are interested, you can write a more detailed test for native calls using the JMH tool.

The performance differences in throughput and latency are about ~10 times. Charles Nutter in his presentation gave us the same numbers.

References

• FUSE on GitHub
• JNR on GitHub
• Charles Nutter presentation about JNR
• JEP 191
• Java HelloFuse /CHelloFuse

The jnr-fuse project is located on GitHub. I’ll appreciate stars, pull-requests, and comments. I’ll be glad to answer any questions you have about JNR or jnr-fuse.