Category Archives: Storage Networking

SNW 2011 Monday

Storage Networking No Comments

The usual Monday on a storage conference. It’s quite obvious all vendors are poaching cloud storage, seems to be the next big thing but what I couldn’t find out is what cloud storage actually is. Seems all vendors have a different mindset and they try to align their product set to the very cloudy definition of cloud storage.Second big thing is converged networking. Well, you know my thoughts around that. Mixing two totally different cultures is doomed to fail but we’ll see in the long term. The third big thing is solid state disks. Not new and I’m wondering what the fuss is all about. Just another layer of blocks which happen to be very fast but only for specific purposes. When the pricing come down significantly and when vendors are able to sort out the back-end performance we’ll see more adoption is guess.

I have a great day though. Met some of the most notorious bloggers like Seve Foskett, Stu Miniman, Marc Farley, Calvin Zito and others. Marc took us out for lunch at a sushi restaurant down town San Jose together with calvin, enrico and fabio.

From left to right Calvin Zito, Marc Farley, Fabio Raposseli and Enrico Signoretti.

The day for me started of with a session on Storage Security by Gordon Arnold from IBM. It’s very clear that the biggest problem is, again, not the technical side but more the side of legislation and how to adapt the technology to that legislation. Some countries are very restrictive in data locality, recoverability, authenticity and other labels you can hook this up to. There are quite some challenges in this particular arena.

David Dale from NetApp went on with a session on solid state with some very nice comparisons on price per IO and capacity. As always these things don’t go well together so again if the pricing on SSD’s is fixed we’ll see some more adoption in the marketplace.

Dennis Martin from Demartek had a session in which he outlined the benefits and current landscape of FCoE. I think he and I both agreed that the organisational issue as well as the post-sales support issues might have big ramifications on adoption from FCoE in the market. Generally speaking the storage guys don’t like the networking guys and vice versa. It’s a different mindset.

After that it was time for The Woz. (and that’s not an acronym for The Wizard of Oz :-)) Steve Wozniack, co-founder of Apple and these days tied to Fusion-IO. What can you say from the man who more or less invented the PC as we know it. He had some good jokes and still drives people to think about technology and encourages students to discover the capabilities of it.

After lunch had a catchup on Virtual Instruments but had nothing new for me unfortunately. The day concluded with a small cocktail party and had some chats with people around. It is still a very small business and everyone knows everyone but its good to see so many news things coming although it still think a lot of it have to be formalised and standardised before customers should pick up any of it. It’s still a lot of buzz but no real meat that will actually fix stuff.

The Apple iPad equivalent of storage has yet to be found and I still don’t see any vendor able to solved the puzzle. Maybe the next couple of days will bring more. At least we have #storagebeers coming up. 🙂

See ya, tomorrow.

Cheers,
E

SNWUSA – The tourist sunday

Storage Networking No Comments

Yesterday I met up with Mike Heffernan aka @virtualheff and had some diner in Palo Alto. (Ohh btw mike, thanks for picking up the bill.) and discussed the usual  things around technicalities which we could do better so it was a useful brainstorming session as well. 


Today I first went to the Computer History Museum in Mountain View. Very nice to see the legacy and evolution of computer science. After that I drove to San Fransisco and spent some time over there. Very nice city but you don’t want to ride a bike there. Man it’s hills are steep. So you do the usual tourist stuff so here are some pictures.

 Google’s First.

 Doesn’t need any intro. Unfortunately the tickets were sold out so I couldn’t visit this time.

Maybe the most famous bridge in the world.
More pictures are posted on Fotki  and here .
Tomorrow SNW is going to kick off. I’m very exited and I hope it going to be worth the trip. Curious how many people have registered for my presso on Thursday.
Cheers
Erwin van Londen

SNWUSA Pre-conference schedule

Storage Networking No Comments

As this week is going to fairly interesting at SNWUSA in Santa Clara I thought I’d might give some short daily updates. In the twittersphere there is already some great noise spread with all schedules around #storagebeers and other social interactions. Yesterday was travelday for me. Getting on a massive Airbus A380  is always quite an experience. Since I used my points on other gadgets (mainly for my kids)
I had to fly tourist class so 14 hours being cranked in a chair is not the best way to travel but on an A380 it’s still doable. At least they have some pretty good entertainment (or annoyance distraction) system in that aircraft so time “flew” by pretty fast. The flying Kangaroo brought me safely to Los Angeles.
After a short wait in LAX the next plane (Ambrear, what a difference) took me to San Jose and I stopped by the office on Central Express way to catch up with some emails and check out the HDS corporate HQ.

Tonight I’m going to have a catchup with buddy Heff (no not that Heff, Michael Heffernan or @virtualheff for insiders) and have some dinner.

I think it’s going to be a good and exiting week.

More to come.

Cheers
E

Maintenance

Storage Networking No Comments

Why do I keep wondering why companies don’t maintain their infrastructure. It looks to be more of an exception than a rule to come along software and firmware which is newer than 6 months old. True, I admit, it’s a beefy task to keep this all lined up but then again you know this in advance so why isn’t there any plan to get this sorted every 6 months or even more frequent.


In my day-to-day job I see a lot of information from many customers around the world. Sometimes during implementation phases there is a lot of focus on interoperability and software certification stacks. Does Server A with HBA B work with switch Y on storage platform Z? This is very often a rubber stamping methodology which goes out the door with the project team. The moment a project has been done and dusted this very important piece is very often neglected due to many reasons. Most of them are time constraints, risk aversion, change freezes, fear something goes wrong etc. However when you look at the risk businesses take by taking chances not properly maintaining their software is like walking on a tightrope over the Grand Canyon with wind-gusts over 10 Beaufort. You might get a long way but sooner or later you will fall.

Vendors do actually fix things although some people might think otherwise. Remember in a storage array are around 800.000 pieces of hardware and a couple of million lines of software which make this thing run and store data for you. If you compare that to a car and would run the same maintenance schedule you’re requiring the car to run for 120 years non-stop without changing oil, filters, tyres, exhaust etc.etc. So would you trust your children in such a car after 2 years or even after 6 months. I don’t, but still businesses do seem to take the chances.

So is it fair for vendors to ask (or demand) you to maintain your storage environment. I think it is. Back in the days when I had my feet wet in the data-centers (figuratively speaking that is) once I managed a storage environment of around 250 servers, 18 FC switches and 12 arrays so a pretty beefy environment in those days. I’d set myself a threshold for firmware and drivers not to exceed a 4 month lifetime. That meant that if newer code came out from a particular vendor it was impossible that code was not implemented before those 4 months were over.
I spent around two days every quarter to generate the paperwork with change requests, vendor engineers appointments etc and 2 two days to get it all implemented. Voilà done.The more experience you become in this process the better and faster it will be done.

Another problem is with storage service providers, or service providers in general. They also are depending on their customers to get all the approvals stamped, signed and delivered which is very often seen a huge burden so they just let it go and eat the risk of something going wrong. The problem is that during RFP/RFI processes the customers do not ask for this, the sales people are not interested since this might become a show-stopper for them and as such nothing around the ongoing infrastructure maintenance is documented or contractually written in delivery or sales documents.

As a storage or service provider I would turn this “obligation” in my advantage and say: Dear Mr, customer, this is how we maintain our infrastructure environment so we, and you, are assured of immediate and accurate vendor support, have up-to-date infrastructure to minimize the risk of anything going downhill at some point in time.”

I’ve seen it happen were high severity issues with massive outages were logged with vendors where those vendors came back and say “Oh yes Sir, we know of this problem and we have fixed that a year ago but you haven’t implemented this level of firmware, it is far outdated”.

How would that look if you’re a storage/service provider to your customers or a bank who’s online banking system had some coverage in the newspapers??

Anyway, the moral is “Keep it working by maintaining your environment”.

BTW, at SNWUSA Spring 2011 I wrote a SNIA tutorial “Get Ready for Support” which outlines some of the steps you need to take before contacting your vendors’ support organisation. Just log on to www.snia.org/tutorials. It will be posted there after the event.

Cheers,
Erwin van Londen

Why FCoE will die a silent death

Fibre Channel 3 Comments

I’ve said it before, storage is not simple. There are numerous things you have to take into account when designing and managing a storage network. The collaboration between applications, IO stacks and storage networks have to be very stable in order to get something useful out of it both in stability as well as performance. If something goes wrong its not just annoying but it might be disastrous for companies and people.


Now I’ve been involved in numerous positions in the storage business from storage administrator to SAN architect and from pre-sales to customer support and I know what administrators/users need to know in order to get things working and keep it this way. The complexity that comes to the administrators is increasing every year as does the workload. A decade ago I use to manage just a little over a terrabyte of data and that was pretty impressive in those days. Today some admins have to manage a petabyte of data (yes, a 1000 fold more). Now going from a 32GB diskdrive to a 1TB diskdrive might look like their life just simplified but nothing is further from the truth. The impact it has when something goes wrong is immense. Complexity of applications, host/storage based virtualisation etc etc have all added to an increase of skills required to operate these environments.

So what does this have to with FCoE. Think of it as this: you have two very complex environments (TCPIP/networking and FibreChannel Storage) who by definition have no clue what the other is about. Now try to merge these two together to be able to transport packets through the same cable. How we do that? We rip away the lower level of the ISO and FC layers, replace that with a new 10GbE CEE interface, create a new wrapper with new frameheaders, addressing and protocol definitions on those layers and away we go.

Now this might look very simple but believe me, this was the same with fibre channel 10 years ago. Look how the protocol evolved. Not only in speeds and feeds but also tremendously in functionality. Examples are VSAN’s, Virtual Fabrics, FibreChannel Routing to name a few. Next to that the density of the FC fabrics has increased as does the functionality on storage arrays. I already wrote in a previous article that networking people in general are not interested in application behaviour. They don’t care about IO profiles, responsetimes and some packet loss since TCPIP will solve that anyway. They just transport packets through a pipe and if the pipe isn’t big enough they replace it with a bigger pipe or re-route some of the flow to another pipe. That is what they have done for years and they are extremely good at it. Storage people on the other hand need to know exactly what it hitting their arrays and disks. They have a much more vertical approach because each application has a different behaviour on storage. If you mix a large sequential load with a very random one hitting the same arrayports and spindles you know you are in a bad position.

So here is were politics will collide. Who will manage the FCoE network. Will it be the networking people? (Hey, it’s Ethernet right? So it belongs to us!). Normally I have no problem with that but they have to prove that they know how FibreChannel behaves, what a Ficon SBC codes set looks like as well as an FCP SCSI CDB. (I see some question marks coming already)
Now FCoE doesn’t work on your day-to-day ethernet or fibrechannel switch. You have to have specialized equipment like CEE and FCF switches to get things going. Most of them are not backwards compatible so they act more as a bridging device between an CEE and FC network. This in turn add significantly to the cost you were trying to save by knocking off a couple of HBA’s and network cards.

FCoE looks great but the added complexity in addition to an entire mindshift of networking and storage management plus the need for extremely well trained personnel will make this technology sit in a closet for at least 5 years. There it will mature over time so true storage and networking convergence might me possible as a real business value add. At the time of this writing the standard is just a year old and will need some fixing up.

Businesses are looking of ways to save cost, reduce risk and simplify environments. FCoE currently gives neither of these.

Save money managing storage effectively

Storage Networking No Comments

How much tools do you use to manage your storage environment.

On average the storage admin uses 5 tools to manage a storage infrastructure.

1. Host tools (for getting host info like used capacity, volume configs etc.)
2. HBA tools (some OS’es don’t have a clue around that)
3. Fabric tools (extremely important)
4. Array tools (even more important)
5. Generic tools (for getting some sort of consolidated overview. Mainly Excel worksheets :-))


Sometime storage management is performed like below:

As you can see thing can become quite complicated when storage infrastructures grow and you’ll need a bigger whiteboard. At the point you have an enterprise storage infrastructure you’ll probably need a bigger building and a lot more whiteboards. 🙂

So what is the best way?

One word:

Integration, Integration, Integration, Integration.

The database boys know this for a long time. Don’t store the same information twice. This is called Database Normalization.
The same thing applies to storage management tools. Make sure that you use tools that have an integrated framework which leverages as much components as possible.

In case you’re using Hitachi kit is pretty easy. Their entire Hitachi Storage Command Suite works together and share single configuration repositories. The best thing is they do that even across their entire array product line from SMS to USP-V and even from two generations ago (so that includes the 9900 and 9500 series) This way other modules can make use of this. The other benefit is that you only have to deploy single host agents to obtain host info like volumes, filesystems, capacity usage etc and have that shared across all different products. Now be aware there is no silver bullet for managing all storage from a single pane of glass if you have a heterogenious environment. Every vendor has it own way of doing things and although the SNIA is making good progress with SMI-S it’s still lacking much of the nifty features storage vendors have released lately.

RTFM

Storage Networking No Comments

Yeah its been a while. A lot has happened in two years. One thing that really jumps out is I moved Down Under. Yep, now inhabitant of kangarooland and I’ve loved every day of it.

To storage:
You don’t want to know how many questions I get who’s answers have been perfectly described in all sorts of manuals. This almost leads to the point were my job becomes a manual reader and a walking storage encyclopedia. 🙂 Now that’s something to put on my CV.


The big problem is however with so many different (storage) products and related documentation I can understand the problem storage admins have these days. Storage infrastructures become more and more complex and an ever increasing level of knowledge is required to maintain all of this. Take into account all different updates these guys get from their vendors almost on a monthly basis then you can imagine what their workday looks like. My life is pretty easy. I only have to keep track of around 80 software products and approx 15 storage hardware platforms because I work for one of those vendors. Multiply that by an average of around 17 manuals per product between 10 and over 5000 (yes, five-thousand) pages and …… you do the maths. Take into account that I also need to know what happens on a OS level from an IO stack perspective including all the different virtualisation kit that is out there including Mainframe z/OS so this pretty much sums up my daily life.. 😉

No, I’m not pitying myself. I have a fantastic wife, wonderful kids and good job, so I’m quite happy with what’s going on in my life.

Going back to the storage admins. The big difference between them and myself is I have access to all the information I need plus some competitive information of my com-colleagues. The storage admins totally rely of what the vendors want them to have and that very often is extremely restricted. I can understand that a lot of this is market sensitive and belongs as company confidential behind locks, however I also think that we should give the right information/documentation (in any form you like) in a structured and easy to understand format without the nitty/gritty stuff that is totally irrelevant. This will easy the burden which a lot of you guys out there suffer and believe me I’ve been there.

A second way of sharing experiences and knowledge is user communities. The perfect example for me has always been Encompass or DECUS. The best user community ever, affiliated to Digital Equipment Corporation. (HP still picks the fruit from that). I think it’s extremely important that vendor should provide a platform were their users can share expierences (good or bad) and be able to leverage the knowledge of his/her peers.

One of my primary tasks, besides being a technical conscience to my sales reps, is to provide my customers (you storage admins) with all the information they need and to help them manage the kit I sold them so they can be heroes within their company.

TTY later.

Greetz,
Erwin