IDC says unstructured data will overtake structured in 2008, while Hitachi Data Systems says it is ready to help

A TRANSITION YEAR AHEAD

IDC says unstructured data will overtake structured in 2008, while Hitachi Data Systems says it is ready to help

Story by TONY WALTHAM

For data centres, 2008 will be a year of transition with IDC Corp analysts predicting that companies will be buying more storage to support file-based unstructured data than for traditional, structured data.

This shift comes as the volume of file-based, unstructured data is exploding, while compliance and privacy regulations have "changed the rules," IDC vice president for Storage Systems Research Richard Villars told journalists at a recent Hitachi Data Systems (HDS) Storage Summit here.

Information was proliferating, he said, noting that the volume of storage shipped to data centres in the Asia Pacific region was now growing at a compound rate of 85.1 percent a year.

Challenges stemming from this included "doing more with less" and making storage easier to use, as well as finding business-savvy IT professionals and better managing power and cooling, he said.

Improved operational efficiency was the way to cope with these challenges, Villars said, while noting that the intelligent controller was the "interesting addition" in the evolution of storage architecture.

Storage virtualization was now critical for information and data management and this should be applied to the mobility of both data and applications as well as for automated thin provisioning, he said, adding that thin provisioning, or capacity virtualization, allowed storage to be dynamically assigned across storage arrays and solved the problem of stranded storage and under-utilized capacity.

The surge in unstructured data meant that "secondary storage was the new primary" and this presented major new management issues for data centres, Villars said. He gave the example of Vodafone whose IT department was "looking at a massive consolidation effort in Europe," having told him that five years ago, unstructured data had been less than five percent of their storage capacity in the data centre. Today it was 30 percent and within two years it would be over 50 percent, he quoted Vodaphone as having said.

Most unstructured data was in a file format, so file-based information management would become a critical function in a storage solution, or "how well the storage systems support the challenges at a file and not the traditional block level," he said.

Looking ahead, the next interesting area would be the active archive concept, he added, noting that archived data dwarfed active data in the enterprise and this was a critical new challenge.

According to HDS chief operating officer (COO) Jack Domme, much of the company's strategy around storage had started a few years ago around this concept of unstructured storage - looking at "how the data centre would answer questions like full discovery, compliance and regulation?"

Up until now, most of that unstructured data had sat in departmental silos, in content management systems and in email systems, he said in an interview, noting how HDS's USP, USP V and USP VM platforms now provided services that allowed customers to place all their content and files on their current storage infrastructure.

Virtualization enabled customers to leverage everything they had done for their applications, being able to apply it to unstructured data and to move that data, he said.

"If I have a lot of unstructured data I can move it to the appropriate tier, I can now search all that unstructured data in the infrastructure," Domme said, contrasting this with the situation on other platforms today where "if you have to discover unstructured data for compliance reasons, you have to go in through every single application."

He said that by simply storing it on that common platform "we can search all the unstructured data in one place and provide the result. We're talking about turning months of discovery into a couple of days or hours of automated discovery and search," HDS's COO said.

Other major providers of enterprise storage, IBM and EMC, provided their customers with "separate islands of storage," he noted. Sun Microsystems and Hewlett Packard are both HDS partners, providing their customers with Hitachi storage under OEM agreements. According to Wachovia, the combined sales made Hitachi the leading provider of enterprise storage for Q3, 2007.

"A Centera product by EMC is a separate interface, it's a separate appliance. So you can't leverage the appliances on the same common platform. You cannot leverage the same services, you cannot leverage the same investment, the same governance, the same process," Domme said, noting that this was "a big change... something no one else has really brought forward."

He also said that HDS was creating the concept of no-spin disk for inactive data. "Say, for example, data is older than 90 days and it has not been accessed. Remember, we are now in the infrastructure setting the policies or executing the policies that the data centre, the company or the business wants to execute. So when we say that if a certain file is over 90 days, we should want to move that to a third or a fourth or fifth tier. We may say that we want to move it to a no-spin disk. We may say that it should be stored as a certain file type or data type.

"All of that policy execution now is being executed in the infrastructure. Today, it's executed individually, in every single application. So the kind of change and the kind of management that's going to take place to get that of that all in order is going to be tremendous.

"So when we see this inactive disk or this inactive tier, we see this as a way to save power, a way to save cost, but also a way to keep it discoverable, to keep it active, in one sense, so that people can access that data, at the least expensive cost," HDS's COO explained.

In addition to companies being able to meet compliance needs, being able to discover all the information, structured or unstruc- tured, is also being leveraged is for data mining.

Domme explained that with knowledge management the only way to find certain data was to go into the application or to go into that department. However, consolidating all information into one common infrastructure allowed people to search across all applications independently of their applications and data.

This would expand into data mining, he said, while noting that document management experts regard the biggest expense when holding data for long periods of time as being repurposing data to new data types that could can be read by the current applications.

"So, storing data for 100 years is really not a big issue. Reading it and constantly keeping it up to date to be read by the latest application is a huge undertaking.

"Doing that at an infrastructural level is the way to do it. You do not want the applications to own their own data, per se, because they do not have the capabilities to execute common policies.

"So that's where the common data centre and management takes over. When we own a piece of data like an X-Ray or an MRI scan, it may be 50 years old, who's going to make sure that the next application is going to be able to read that?" But if it was managed centrally, you could repurpose that data constantly to be read by the newest application, Domme explained.

HDS's intelligent controller-based virtualization was important to customers because it provided simplified management, the ability to move data back and forth, the ability to align the infrastructural cost to the business value of that data, Domme said.

"We can't store everything on tier one today. We'd have to be able to move that data... it's really lifecycle management, you remember that was a big buzzword a while ago? And really all that translated to was the capability to constantly move data to keep aligned with the business.

"And who's doing that? That's what this whole thing with virtualization is about - to dynamically move data in accordance to what is the business value of that data.

"So whether that was a data warehousing example, where you're moving tables from one tier to another as they needed to be accessed, or whether we're moving unstructured data or health records, based on the need at certain times to move it down multiple tiers and then move it back, you must have the infrastructure to dynamically move this data without the applications knowing.

That way the applications would always have access to their data and the infrastructure would be responsible for executing the right policy as to where it should be, he explained.

Noting that 40 percent of HDS's revenues were now derived from software and services, Domme said that this meant that HDS's whole business model had changed, adding that year-on-year growth of 7 percent and "tremendous growth" in the Asia Pacific region indicated the change that the company was going through, being a solutions provider.

HDS has also expanded its small modular systems, which it is marketing through a channel model, having recently appointed Acer (Thailand) as a co-branded supplier.

HDS Asia Pacific solution and products director Vivekanand Venugopal noted that its whole strategy in the region had been through the channel and that 80 percent of its business in the region came from channel partners.

In Thailand, resellers include Stream IT Consulting and DataOne Asia, while HDS's global resellers are Acer, Sun, Ingram Micro, HP and Lenovo.