Tag Archives: Hitachi

Hitachi Va….. what???

VANTARA. Hitachi Vantara. Yes, took me a while to get used to it as well after almost 15 years at HDS but I must say reading all the internal and external communications and overall business and technical transformation gets me more excited by the day.

HDS, Pentaho and Hitachi Insight Group are combined into a new company focussing on IoT, Analytics and helping customers obtain the maximum use and benefit out of their operational technologies via bullet proof IT from HDS.

Two months ago I had solar panels installed. Living in the Australian Sunshine Coast Hinterland I thought I’d let the name and location do some work for me and as a positive side-effect bring my power bills down as well. In addition to that I’ll prevent a few tonnes of CO2 emissions by not having the power companies burn black rock. The challenge is however to make the best use out of the solar installation. As I did not opt for a battery installation (simply too much money and the ROI makes it currently not worth it) the panels deliver between 4 and 5KWh on average depending on a few factors like cloud overcast, angle of the sun, shade of trees etc. Continue reading

SCSI UNMAP and performance implications

When listening to Greg Knieriemens’ podcast on Nekkid Tech there was some debate on VMWare’s decisision to disable the SCSI UNMAP command on vSphere 5.something. Chris Evans (www.thestoragearchitect.com) had some questions why this has happened so I’ll try to give a short description.

Be aware that, although I work for Hitachi, I have no insight in the internal algorithms of any vendor but the T10 (INCITS) specifications are public and every vendor has to adhere to these specs so here we go.

With the introduction of thin provisioning in the SBC3 specs a whole new can of options, features and functions came out of the T10 (SCSI) committee which enabled applications and operating systems to do all sorts of nifty stuff on storage arrays. Basically it meant you could give a host a 2 TB volume whilst in the background you only had 1TB physically available. The assumption with thin provisioning (TP) is that a host or application wont use that 2 TB in one go anyway so why pre-allocate it.

So what happens is that the storage array will provide the host with a range of addressable LBA’s (Logical Block Addresses) which the host is able to use to store data. In the back-end on the array these LBA’s are then only allocated upon actual use. The array has one or more , so called, disk pools where it can physically store the data. The mapping from the “virtual addressable LBA” which the host sees and the back-end physical storage is done by mapping tables. Depending on the implementation between the different vendor certain “chunks” out of these pools are reserved as soon as one LBA is allocated. This prevents performance bottlenecks from a housekeeping perspective since it doesn’t need to manage each single LBA mapping. Each vendor has different page/chunks/segment sizes and different algorithms to manage these but the overall method of TP stay the same.

So lets say the segment size on an array is 42MB (:-)) and an application is writing to an LBA which falls into this chunk. The array updates the mapping tables, allocates cache-slots and all the other housekeeping stuff that is done when a write IO is coming in.  As of that moment the entire 42 MB is than allocated to that particular LUN which is presented to that host. Any subsequent write to any LBA which falls into this 42MB segment is just a regular IO from an array perspective. No additional overhead is needed or generated w.r.t. TP maintenance. As you can see this is a very effective way of maintaining an optimum capacity usage ratio but as with everything there are some things you have to consider as well like over provisioning and its ramifications when things go wrong.

Lets assume that is all under control and move on.

Now what happens if data is no longer needed or deleted. Lets assume a user deletes a file which is 200MB big (video for example) In theory this file had occupied at least 5 TP segments of 42MB. But since many filesystems are very IO savvy they do not scrub the entire 42MB back to zero but just delete the FS entry pointer and remove the inodes from the inode table. This means that only a couple of bytes effectively have been removed on the physical disk and array cache.
The array has no way of knowing that these couple of bytes, which have been returned to 0, represent an entire 200MB file and as such these bytes are still allocated in cache, on disk and the TP mapping table. This also means that these TP segments can never be re-mapped to other LUN’s for more effective use if needed. To overcome this there have been some solutions to overcome this like host-based scrubbing (putting all bits back to 0), de-fragmentation to re-align all used LBA’s and scrub the rest and some array base solutions to check if segments do contain on zero’s and if so remove them from the mapping table and therefore make the available for re-use.

As you can imagine this is not a very effective way of using TP. You can be busy clearing things up on a fairly regular basis so there had to be another solution.

So the T10 friends came up with two new things namely “write same” and “unmap”. Write same does exactly what it says. It issues a write command to a certain LBA and tells the array to also write this bit stream to a certain set of LBA’s. The array then executes this therefore offloading the host from keeping track of all the write commands so it can do more useful stuff than pushing bits back and forth between himself and the array. This can be very useful if you need to deploy a lot of VM’s which by definition have a very similar (if not exactly) the same pattern. The other way around it has a similar benefit that if you need to delete VM’s (or just one) the hypervisor can instruct the array to clear all LBA’s associated with that particular VM and if the UNMAP command is used in conjunction with the write same command you basically end up with the situation you want. The UNMAP command instructs the array that a certain LBA (LBA’s) are no longer in use by this host and therefore can be re-used in the free pool.

As you can imagine if you just use the UNMAP command this is very fast from a host perspective and the array can handle this very quickly but here comes the catch. If the host instructs the array to UNMAP the association between the LBA and the LUN it is basically only a pointer from the mapping table that is removed. the actual data does still exist either in cache or on disk. If that same segment is then re-allocated to another host in theory this particular host can issue a read command to any given LBA in that segment and retrieve the data that was previously written by the other system. Not only can this confuse the operating system but it also implies a huge security risk.

In order to prevent this the array has one or more background threads to clear out these segments before they are effectively returned to the pool for re-use. These tasks normally run on a pretty low priority to not interfere with normal host IO. (Remember that it still is (or are) the same CPU(s) who have to take care of this.) If CPU’s are fast and the background threads are smart enough under normal circumstances you hardly see any difference in performance.

As with all instruction based processing the work has to be done either way, being it the array or the host. So if there is a huge amount of demand where hypervisors move around a lot of VM’s between LUN’s and/or arrays, there will be a lot of deallocation (UNMAP), clearance (WRITE SAME) and re-allocation of these segments going on. It depends on the scheduling algorithm at what point the array will decide to reschedule the background and frontend processes so that the will be a delay in the status response to the host. On the host it looks like a performance issue but in essence what you have done is overloading the array with too many commands which normally (without thin provisioning) has to be done by the host itself.

You can debate if using a larger or smaller segment size will be beneficial but that doesn’t matter at all. If you use a smaller segment size the CPU has much more overhead in managing mapping tables whereas using bigger segment sizes the array needs to scrub more space on deallocation.

So this is the reason why VMWare had disabled the UNMAP command in this patch since a lot of “performance problems” were seen across the world when this feature was enabled. Given the fact that it was VMWare that disabled this you can imagine that multiple arrays from multiple vendors might be impacted in some sense otherwise they would have been more specific on array vendors and types which they haven’t done.

HP ends Hitachi relationship

Well, this maybe a bit premature and I don’t have any insights in Leo’ s agenda but when you apply some common sense and logic you cannot draw another conclusion than within the foreseeable future this will happen. “And why would that be?” you say, “They (HP) have a fairly solid XP installed base and they seem to do sell enough to make it profitable and they also have embarked on the P9500 train”.

Yes, indeed, however take a look at it from the other side. HP has currently 4 lines of storage products, the MSA inherited thru the Compaq merger which comes out of Houston and specifically targeted at the SMB market, the EVA, from the Digital/Compaq StorageWorks stable, which has been the only HP owned modular array which has done well in the SME space, the XP/P9500 obviously thru their Hitachi OEM relationship and, since last year, the 3-Par kit. When you compare these products they do have a lot of overlap in many areas especially in the open systems space. It is therefore that the R&D budgets for all the 4 products eat up a fair amount of dollars. Besides that, HP also has to set aside a huge amount of money for Sales, Pre-Sales, Services and Customer support in training, marketing etc to be able to provide a solution of which a customer will only choose the one which fits their needs. So just from a product perspective there is a 1:4 sales ratio. I don’t even mention the choices customers have from the competition. For the lower part of the pie (MSA & small EVA) HP heavily relies on their channel but from a support and marketing perspective this still requires a significant investment to keep those product lines alive. HP just has released their latest generation of the EVA but as far as I know has not commented on future generations. It is to be expected that as long as the EVA sells like it has always done the development of it will continue.

With the acquisition of 3-Par last year HP has dived very deep in their money pit and paid 2.3 billion dollars for them. You don’t make such an investment to just keep a certain product out of the hands of a competitor (Dell in this case). You do want this product to sell like hotcakes to be able to shorten your ROI as much as possible. Leo has quite some shareholders to answer to. It then depends where you get the most margins from and it is very clear that when you combine the ROI needs of 3-Par and the margins they will obviously make on that product HP will most likely prefer to sell 3-Par before XP/P9500 even if the latter would be a better fit for the solution needed by the customer. When you put it all together you’ll notice that even within the storage division of HP there is a fair amount of competition between the product lines and no R&D department for either of those want to loose. So who needs to give??

There are two reasons why HP would not end their relation ship with Hitachi, Mainframe and Customer demand. Neither of the native HP product have Mainframe support so if HP decides to end the Hitachi relationship they will certainly loose that piece as well as obtaining the risk that same customer chooses the competition for the rest of the stack as well. Also if XP/P9500 customers already have made significant investment investment in Hitachi based products, they most certainly will not like a decision like this. HP, however is also not reluctant to make these harsh decisions. History proves they’ve done it before. (Abruptly ending and OEM relationship with EMC as an example.)

So, if you are an HP customer who just invested in Hitachi technology, rest assure you will always have a fallback scenario and that of course is to deal with Hitachi itself. Just broaden your vision and give HDS a call to see what they have to offer. You’ll be very pleasantly surprised.


(post-note 18-05-2011) Some HP customers have already been told that 3-Par equipment is now indeed HP preferred solution they will offer unless Mainframe is involved.

 (post-note 10-07-2011) Again more and more proof is surfacing. See Chris Mellor’s post on El Reg over here

The Smarter Storage Admin (Work Smarter not Longer)

Lets start off with a question: Who is the best storage admin?
1. The one that starts at 07:00 AM and leaves at 18:00 PM
2. The one that starts at 09:00 AM and leaves at 16:00 PM

Two simple answers but they can make a world of difference to employers. Whenever an employer answers with no. 1 they often have the remark that this admin does a lot more work and is more loyal to the company. They might be right however the daily time spent at work is not a good qualifier for productivity so the amount of work done might be less than no.2. This means that an employer has to measure on other points and define clear milestones that have to be fulfilled.

Whenever I visit customers I often get the complaint that they spend too much time doing day to day administration like digging through log files, checking status messages, restoring files or emails etc. etc. These activities can occupy more than 60% of an administrators day which can be avoided.
To be more efficient one has to change the mindset from knowing all to knowing what doesn’t work. It’s a very simple principle however to get there you have to do a lot planning.
An example is when a server reboots do I want to know if the switch port goes offline? Maybe I do, maybe I don’t. It all depends on what the impact of that server is. Is it planned or not or maybe this server belongs to a test environment in which case I don’t want to get a phone-call in the middle of the night at all.

The software and hardware in a storage environment consists of many different components and they all have to work together. The primary goal of such an environment is to move bytes back and forth to disk, tape or another medium and they do that pretty well nowadays. The problem however is management of all these different components which require all different management tools, learning tracks and operation procedures. Even if we shift our mindset to “What doesn’t work”, we still have to spend a lot of time and effort in thing we often don’t want to know.

Currently there are no tools available who support the whole range of hardware and software so for specific tasks we still need the tools the vendors provide. However for day to day administration there are some good tools which might be very beneficial for administrators. These tools can save more than 40% of an administrators time so they can do more work in less time. It takes a simple calculation to determine the ROI and another pro is that the chances of making mistakes is drastically reduced.

Another thing to consider is if these tools fit into the business processes if these are defined within a company. Does the company have ITIL, Prince2 or any other method of IT service management in place. If so the storage management tool has to align to these processes since we don’t want to do things twice.

Last but not least is the support for open standards. The SNIA (Storage Networking Industry Association) is an non-profit organization which was founded by some storage vendors in the late 90’s. The SNIA works in conjunction with its members around the globe to make storage networking technologies understandable, simpler to implement, easier to manage, and recognized as a valued asset to business. One of the standards ,which was recently certified by ANSI, is SMI-S. This standard defines a very large subset of storage components which can be managed through a single common methodology. This means that you’ll get one common view of all your storage assets with the ability to manage it through a single interface independent of the vendor. If your storage management tool is based on this standard you do not have a vendor lock-in and day to day operations will be more efficient.
This implies however that the vendor also has to support the SMI-S standard so make sure you make the right choice if you are looking for a storage solution and ask the vendor if he supports the SMI-S standard and to what extend.