Around the Storage Block Blog
Find out about all things data storage from Around the Storage Block at HP Communities.

Demystifying the Linear Tape File System (LTFS)

MarkF_TapeJ.jpgBy Mark Fleischhauer, Tape Storage Solutions Engineering Manager

 

Since the introduction of the Linear Tape File System (LTFS), a number of articles, white paper and tech briefs have been written about it and its impact on tape. In this blog, I’m going to offer some context around what the excitement is all about and discuss how LTFS is being used and deployed. 

 

A file system for tape

To start, LTFS is just what is says it is – a file system for linear tape. In simple terms, LTFS defines two partitions on the tape:

  1. Partition 0 contains the list of files and their location on the tape
  2. Partition 1 contains the actual files

This allows the tape media to be “self-describing” in identifying each individual file on the tape with direct access to each file. It allows for the creation of a directory structure to make file management even easier. The specification for LTFS is open and is controlled and managed by the Storage Networking Industry Association (SNIA). A number of contributing members are working on the specification, including those from HP and IBM, who ensure that future releases are interoperable with previous versions and add new features to address the growing demand for LTFS. 

 

Config.jpg

 

Making tape as easy to access, manage and share as disk

As a file system for tape, various tape utilities have been developed allowing a tape drive to mount within an operating system (Windows, Linux, and MAC), thereby allowing for basic file system ScreenShot.jpgoperations such as directory creation and drag and drop to move files to and from the tape drive. The specification ensures that media written by one file system (i.e. Windows) can be read by another file system (i.e. Mac). Now a user can use a tape drive like a large thumb drive. Just like a thumb drive, LTO tapes can be used to share data between systems or can provide a place to store data for safekeeping. HP provides a set of utilities known as HP StoreOpen to simplify the use of tape with LTFS. HP StoreOpen Standalone provides a GUI to mount a standalone tape drive and format a tape with LTFS. HP StoreOpen Automation mounts an HP StoreEver tape library and presents each tape cartridge in the library as an individual file folder, automatically moving tapes in and out of the tape drives as you read/write to a given tape folder.

 

The value of LTFS for archive

Because LTFS is an open format, content stored on LTFS-formatted tapes can be read by any application that has support for LTFS. This makes LTO with LTFS an ideal solution for long-term archive. A number of advanced archiving solutions available today have integrated LTFS into their product offerings. These solutions provide many more features above and beyond basic tape access such as tape duplication, media tracking, meta data management and often employ some sort of disk cache/buffer to optimize tape access. One such solution is HP StoreEver Tape as NAS , which combines the access benefits of NAS with the cost, reliability and long-term retention benefits of tape. With these solutions, you can leverage the cost efficiencies of tape and avoid vendor lock-in to retrieve your data down the road. 

 

We at HP have partnered with a number of LTFS archive solution providers to ensure interoperability with our HP StoreEver products. HP StoreOpen is a great supplementary solution as it can be used to read tapes exported from these advanced solutions without having to install the solution software product. A complete listing of these solutions can be found at: www.hp.com/go/ebs

 

The growing ecosystem for LTFS solutions

LTFS has provided new opportunities to add tape to existing software solutions without the need to become tape experts. Using the open source code and HP StoreOpen solutions available from HP, developers can quickly and easily add tape and tape library support to their software products. This has generated a number of new product offerings that were not available just a few years ago.

 

Changing the game for data exchange and archive

The Linear Tape File System is a true game changer. It’s an open file system for tape, managed and supported by SNIA by leading hardware and software companies like HP giving it credibility and long-term support. From basic drag and drop access to full-featured solutions, users can benefit from LTFS as a multi-vendor data transport and/or long-term archive solution. Combined with the HP StoreEver LTO tape drives and tape libraries, you have a solution that is cost effective, reliable and built on open standards—and that is the perfect combination for long-term archive. 

 

Deeper info dive

The Value of HP StoreEver Tape for Archive White Paper

HP StoreOpen and LTFS Best practices

HP StoreEver LTFS customer case studies

 

And some videos to watch on the tape archive and LTFS topic

 

 

The Value of HP StoreEver Tape for Archive

 

 

 

The Value of HP StoreOpen with LTFS

 

 

Comments
dregister | ‎07-30-2014 01:03 AM

Mark

     I am an engineer at Technicolor On Location Services using many HP LTO5 and LTO6 desktop and Autoloader based drives to archive the dailies footage shot by Hollywood studios for features and TV.

     We have been working to extend our Archive of dailies files to LTO function in our Dailies system  to allow writing to LTFS on HP LTO6 drives on Mac 5,1 towers running Mavericks (OSX 10.9.2) for several months and have completed it, except for a reliability issue that causes a bad LTFS write to a single file on LTO out of ~1 million successful files written.   The exact same system can write tens of Terabytes using tar to the same HP LTO drives without ever experiencing a badly written file.   We would like HP to help us with this LTFS reliability issue.   Would you help us, or direct us to someone at HP who can help us?

 

     Technicolor On Location Services has archived Petabytes of data to dozens of HP LTO5 and LTO6 drives using Tar on our Centos6 computers over the last couple of years, and has been using LTFS on Centos 6 for about 6 months now, writing hundreds of Terabytes, without ever having a problem where one file was written badly to one of our primary or backup LTOs (which are written simultaneously), while the same file was written properly to the other.    However, once I got my LTFS Archive on Mavericks working, we now have several Mac tower, Mavericks systems that will do just that: if we write ~ 1 TB of data to a pair of LTOs, about 1 out of 5 times we write that 1 TB  of data, one file on either of our LTO tapes will fail its MD5 read back verification, while its mate, written at the same time, will pass.  The other 4 times we write that 1TB to both tapes, all files (which are each about 7 MB in size) on both tapes pass MD5 read back verification.  When I compare the good and bad file byte by byte, I find there are several 1K blocks that differ, while the rest of the 7 MB file matches.

                We’ve tried a variety of hardware changes, including changing entire computers (more than once) around the hard drive,  that did not fix this intermittent write problem, and we’ve got one Mavericks system (and only one) that had the problem for a couple of weeks, but no longer does.  We’ve also tried multiple SAS cards and HP LTO6 drives without effect.   Further, after I got Tar working on Mavericks I found I can write that 1TB of data ten times without any verification errors on any of our systems, including the ones that exhibit the LTFS write problem within five 1TB passes. This indicates the intermittent write problem is limited to the LTFS software (most likely OSXfuse or HP StoreOpen Standalone) and is not hardware related.   We also tried building new system drives from scratch, and the intermittent LTFS write error persists.

                We’ve made sure the Computer OS, Atto h680 driver and card firmware, HP Ltfs file system version, Fuse version and HP open store standalone version all match between the system that works and the ones that don’t, to no avail.    All of our computers (the one working and the two not working) are Macpro 5,1 towers running OSX 10.9.2.  The working system is running OSXfuse 2.6.2 and HP StoreOpen Standalone version 2.1.0.   We have tried combinations  of OSXfuse 2.6.2 and 2.6.4 and HP StoreOpen Standalone versions 2.1.0 and 2.2.1 on the other systems without fixing this problem.

                Has HP heard of any other customers writing many Terabytes of data via LTFS to HP LTO6 drives on Mavericks on 5,1 Mac towers?

                Are any of them experiencing intermittent write problems while writing many Terabytes of data via LTFS to HP LTO6 drives?

                What help can HP offer us to eliminate this problem?

Storage-Expert | ‎08-07-2014 02:18 PM

Thank you for getting in touch and posting your question. HP has many customers using LTFS without issue, and we have not heard of anyone experiencing that particular problem. 

 

Your write-up indicates that you have already done a thorough job of trying all the obvious steps to isolate the fault.  LTFS on Centos and on Mac are built from the same source tree so is less likely to be the culprit here.  Also the fact that tar has not exhibited the problem may be related to the different block transfer sizes used, which could mask the underlying problem. 

 

One thing to check would be the version of HBA driver in use; Atto’s release notes for their latest Mac driver (v2.25) indicate that they have corrected a couple of issues which may be relevant. If that does not resolve the problem ,then please contact us directly at storeopen.cpe@hp.com so that we can help investigate further.

Leave a Comment

We encourage you to share your comments on this post. Comments are moderated and will be reviewed
and posted as promptly as possible during regular business hours

To ensure your comment is published, be sure to follow the community guidelines.

Be sure to enter a unique name. You can't reuse a name that's already in use.
Be sure to enter a unique email address. You can't reuse an email address that's already in use.
Type the characters you see in the picture above.Type the words you hear.
Search
Showing results for 
Search instead for 
Do you mean 
About the Author
This profile is for team blog articles posted. See the Byline of the article to see who specifically wrote the article.


Follow Us
The opinions expressed above are the personal opinions of the authors, not of HP. By using this site, you accept the Terms of Use and Rules of Participation