Get Used to the Ever-Expanding Storage Hierarchy

The storage landscape is being transformed thanks to the introduction of technologies like NVM-Express, and newer types of non-volatile memory (NVM). But despite these advancements, the old-school technologies of tape and disk are still with us.  And that’s going to make life more interesting for both the people that build these systems and the ones who use them.

At last week’s Next I/O Platform event, we talked with Andy Walls, IBM Fellow and CTO of Flash Storage, about this development, and he had plenty of ideas about where all of these new technologies are taking us and how we can take advantage of them.

The complexity of this situation picks up on the theme we reported on earlier this week that talked about the problems of deepening storage hierarchies, where the traditional two tiers of tape and disk have expanded to a four thanks to the addition of flash drives and storage-class memory.

“In reality, it’s worse than that,” said Walls. “If you look at flash, it used to be simple: flash was a layer. Now there’s NVMe flash, which by and large is TLC, triple-level cell. Well now there’s QLC, four bits per cell, which has much less endurance, but is much cheaper.” As a result, flash devices have basically bifurcated with these two technologies, he said.  And that doesn’t count the more costly SLC flash that Samsung and others are offering for enterprise drives.

Storage-class memory (SCM) is something of a special case, since currently Intel has the inside track on this tier with its Optane Persistent Memory offering. Other memory makers are developing competing SCM technologies, including Samsung, SK Hynix and Western Digital, but have yet to offer a serious challenge in the market. Anderson believes the most promising technology for this storage layer is phase change memory (PCM), which by the way, is most likely the technology underlying Optane’s 3D XPoint memory.

“The problem here is economical, not technological,” said Walls. It takes about a billion dollars to develop and bring such a product to market. And diverting fab capacity and engineering resources for an R&D project of that size is a risky proposition.

Despite the promise of things like SCM and these newer NVM technologies, people are still buying tapes and disks. That’s because the cost per bit on both media has continued to drop, even as NVM technologies have advanced.

“These layers aren’t going to go away,” said Walls.  He remembers talking to customers in the 1990s who told him tape was dead. That kind of forecast was premature, to say the least. Today, some people, especially those in the flash business, are trying to make the case that spinning disks are now dead. But as Anderson noted, even though he is sympathetic to that sentiment, hard disks aren’t going away either. At a penny a gigabyte or less, disks capacity is several times less expensive than that of the cheapest flash today. . Likewise, tape storage is said to cost about one-sixth that of disks on a capacity basis.

“To me a tier is ideal if you have an order of magnitude difference in cost and an order of magnitude difference in access time,” Walls told us. But the data has to be in the optimal tier to take advantage of those performance differences.

As others have suggested, Walls believes such data management is not suited to mere mortals. The storage system itself will have to decide when to move the data and which tier to move it to. That means storage systems will have to be instilled with a lot more intelligence that they currently have. But according to Walls, that level of smarts is well with our reach, given the recent advancements in analytics and machine learning.

Plus, there is a lot of predictability with regard to how we access our data, which can be teased out with the right algorithms. Walls believes the key to this is to understand the metadata better, which will help us figure out the optimal way to manage our data.

“If we can figure out how to develop a self-driving car, I think we can do this,” he said.