Making sense of WAFL

[#1]

By Karl Dohm, HP Storage Architect

Extensible NetApp Blog (http://blogs.netapp.com/extensible_netapp) contains some posts describing WAFL.  It sums WAFL up as an internal component which...

...provides mechanisms for building file-system semantics, it manages the on-disk format, it manages the free and allocated space, and provides a logical and physical volume manager.

and further making the argument that it is not a file system, but rather an essential part of one.   Fair enough, calling it a file system might be splitting hairs and technically incorrect, but given the amount of confusion across vendors in the industry, it is likely that this common misunderstanding emanated from NetApp's own documentation.  

Whether the details of the FAS internals are technically part of WAFL, or part of OnTap, or part of something else, the main point is that none of that is particularly relevant to the Storage Administrator.    What matters most is performance, space efficiency, and ease of use. 

From this point on I'm not going to split hairs and discern between WAFL and the remainder of FAS internal software/firmware - because no one but NetApp architects really care.  For the benefit of simplicity lets call it all WAFL.

WAFL is a rather unique approach to organizing data on the spindles and controlling the flow of the data to the spindles.  No doubt WAFL can come across as impressive in sales presentations because it is very different than the approach used by EMC, IBM, and HP.  In this series of posts, we will explore the other side of WAFL, highlighting some of the problems that WAFL brings to the Storage Administrator, none of which we expect NetApp will fully acknowledge.

Today lets touch on fragmentation.  Some in the industry say, WAFL is "fragmentation by design".   I didn't make it up, but like WAFL being called a file system, its one of those things you tend to hear if the conversation is around NetApp.  This statement strikes me as accurate because WAFL tries to do full stripe writes whenever it can, meaning that it prioritizes writing non sequential blocks in the same stripe over read modify write operations associated with RAID-4 or RAID-DP parity calculation. 

Translating that to the world of the Storage Administrator, this means that gradually the throughput of the FAS degrades over time when the workload has a random component.  Most real world workloads have a random component.  Applications along the line of Microsoft Exchange present a nightmare situation for the FAS.  The throughput degradation can be significant, and throughput can be unpredictable because it varies depending on history of writes. 

For those that question this assertion as being somehow biased, try the following.  Take a FAS system and create a new volume with a new LUN.  Baseline the system by running a sequential read workload and measuring the result.  Notice that the number is already not very impressive.  Next run a few hours of random workload, say 8KB 50/50 R/W, which is similar to MS Exchange.  Now try the sequential read load again and observe the new throughput.  Chances are you will have some new questions for your NetApp sales rep.  

Next time we will discuss the benefit of reallocation, NetApp's answer to fragmentation, and explore how much this really helps the problem.

[#2]

Hi!

There are three ways to read this blog post.

1. FAS systems are not appropriate for Exchange.

Rather than argue my case, I'll let Avanade argue my case, that our systems are appropriate for Exchnage.

media.netapp.com/.../Avanade_Testing_Center_NetApp_Whitepaper_Exchange.pdf

2. FAS systems perform poorly over time.

We have an SPC-1 benchmark which measures a similar work load you describe. This particular SPC-1 benchmark measured our performance over a protracted period time.

www.storageperformance.org/.../a00062_NetApp_FAS3040-48hr-sustain_executive-summary.pdf

3. FAS systems perform poorly as capacity gets reduced.

Alex McDonald put together a response to that argument.

blogs.netapp.com/.../finding-a-pair.html

Look, at the end of the day, trying to make assertions about WAFL behavior by reversing engineering our architecture is a dangerous game. There are techniques and mechanisms for solving those problems that exist. And we have smart people working on solving them.

cheers,

kostadis

[#3]

This must be a joke.

[#4]

Thank You for this post, it is a clear example of misdirection (although you thought you had hit something).

Your statement "Translating that to the world of the Storage Administrator, this means that gradually the throughput of the FAS degrades over time when the workload has a random component.  Most real world workloads have a random component.  Applications along the line of Microsoft Exchange present a nightmare situation for the FAS." actually refers to an Ideal situation for FAS as it is optimized for random workloads.

No matter how powerfull your missile, it is useless if you aim wrong.

I will give you an exercise.

1.set up a realistic random IO test.

2.Take a timing.

3.Scramble your data

4. Take a timing

repeat steps 3 and 4 many times

I hope you learn something about how irrelevant fragmentation is for random IO.

[#5]

Instead of trying to address the comments here (which a couple had no substance), Karl posted a follow up that you can find at www.communities.hp.com/.../making-sense-of-wafl-part-2.aspx

[#6]

I worked at NetApp for 4 yrs.  I disagree with this behavior that has been outlined.  I ran numerous jetstress & iometer tests to simulate Exchange workloads and did not see the same pattern.  However, what I did see and what is simple to argue is that WAFL fragments over time due to multiple snapshots (such as a 10-day retention, cycling over many months) and deduplication.  It is well-known within the NetApp ranks that deduplication changes the pointers between indirect inodes and the actual blocks.  This redirection to another [common] block will cause fragmentation since duplicate 4K blocks are freed regardless of their location or proximity to sequences of blocks (or extents).  This affects sequential read performance by ~10 - 20%.  Similarly, taking snapshots, such as one might use for "backups" of a volume, keeps old blocks in the volume until the snapshot is deleted.  However, the snapshot is just a set of pointers to old blocks representing a pt in time.  Those blocks can be scattered all over the place.  Therefore, when the snap is deleted, those scattered blocks are freed-up for reuse, thus causing fragmentation.  Yes, "realloc" command can/should be used to defrag the volume occasionally.  But, most folks don't have the time and patience to do this activity.  DataONTAP should be running this as a background process...but it does not today. 

Let me emphasize - random write activity on NetApp does NOT cause degradation of subsequent reads.  You do not have a clear understanding of how WAFL works if you believe this.  However, the 2 other processes, outlined above, will lead to fragmentation, which causes degradation of sequential read performance as well as space efficiency. 

[#7]

@mobiuslink - thanks for stopping by.  I'm not the engineer that ran the testing so I'm far from an expert here (and since this was something we did three years ago, not sure if I can get him to respond).  We ran the ESRP benchmark as outlined and I don't believe snapshots were at all involved in what we did. So not sure I can buy into the "it's because of snapshots". 

A couple of other things - I had a current NetApp employee tweet me privately in the very recent past to say that performance of a FAS system absolutely degrades after burn in.  He then insisted that NetApp doesn't use new, prestine systems when running benchmarks.  This private conversation was the first time anyone associated with NetApp admitted to performance degradation.

Lastly, I'd suggest you look at the other posts Karl Dohm wrote.  You can find them here: Other WAFL labeled blog posts.  You'll see several - three more that were a follow up to this article and one he wrote titled Understanding FAS ESRP results.

Thanks again for stopping by -