libgfapi and the Linux target driver (iSCSI)

This post describes modifications to the Linux Target driver to work with Gluster’s “gfapi” . It is a follow up to an earlier post on Gluster’s block IO performance over iSCSI. Those tests used FUSE, which incurred data copies and context switches. 

That “FUSE penalty” can be avoided using libgfapi. The libgfapi library can be inserted into the Linux target driver rather easily, as it has a nice extensible framework. In this manner IO commands may be forwarded directly to the storage server.

The diagram below depicts gluster’s datapath when FUSE is present and absent. Using gfapi, the red boxes are removed (a kernel trip and extra hop).

To make that happen, I’ve modified the Linux target driver to have a “gluster” module. The implementation was mostly a one–to-one replacement of POSIX APIs with “gluster” APIs. For example read and writes were translated into gf_read and gf_writes.  Some performance results are at the end of this post. 

Feel free to download it off the forge. I expect it will be open sourced after some cleanup, additional testing, etc.

Interestingly, Ceph has already done this. Their plug-in to the Linux target driver uses librados to access the Ceph OSD.  It is not enabled by default – you need to recompile the target driver to use it.

Notes:

• The Linux target driver has been replaced by LIO in RHEL7. LIO is a kernel entity similar to SCST. It got into version 2.6.38 (March 2011). The performance implications of using it are unclear. I’ll take a look at that in January.
• Performance seemed to get worse as the number of streams goes up. This may be due to the synchronous gfapi calls used. A future post will look into some of the options to increase parallelism, such as:
  • The target API has an option to set the number of worker threads.
  • gfapi has asynchronous APIs.

Performance:

1. Same 2 node configuration as in this earlier post.
2. The performance improvement of gfapi in lieu of FUSE varied. But generally, libgfapi bought a 10-20% speedup.
3. I’ll update this post with random read performance.