Startup Claims Performance Leadership With New AI Edge Chip

Hailo, an AI startup based in Israel, has released its initial chip that the company claims is “the world’s top performing deep learning processor,” with the Hailo-8 chip claimed to deliver 26 tera-operations pers second (TOPS), while consuming only a few watts of power.

If true, that would certainly put it near or at the top of its class in performance for edge applications in areas like self-driving cars, drones, smart appliances, and virtual/augmented reality devices. The challenge in these edgey environments has always been to get AI processors with the requisite performance for these applications but consuming only the small amounts of power available in these settings.

In fact, Hailo is positioning its new offering as chip that “enables edge devices to run sophisticated deep learning applications that could previously run only on the cloud.” However, doesn’t mean Hailo-8 is as powerful as a top-of-the-line inference GPU for the datacenter. Nvidia’s latest T4 GPU clocks in a 130 TOPS (INT8), which is about five times speedier than Hailo-8. However, the T4 consumes more than 20 times the power of the Hailo-8 offering, which, as we said, is a critical factor in edge environments.

Despite the considerable challenges of delivering both performance and energy efficiency, there is a boatload of companies competing in this AI edge space. They include big players such as Nvidia, Intel, and Google, as well as a bunch of startups like AIStorm, Flex Logic, Efinix, Cornami, Mythic, Cambricon Technologies, and now Hailo. And that’s only a partial list.

The reason for all this enthusiasm is that AI processing at the edge looks like it’s going to be a multi-billion-dollar opportunity within just a few years. And since it’s still early going in this market, essentially all the revenue is up for grabs. AI analyst firm Tractica forecasts edge-based chipsets for artificial intelligence will be worth $51.6 billion by 2025, a number they predict will be 3.5 times larger than the amount spent for AI chips in the datacenter.

Tractica research director Aditya Kaul writes that “the current regime of cloud-based training and inference will give way to a more decentralized approach where training for larger models will occur in the cloud, but for the most part, AI models will be trained and inferred at the edge.”

Which brings us to back to Hailo. Like many companies going after this nascent market, Hailo is of recent vintage, having established itself in 2017. Its founders, Orr Danon, now the CEO, Avi Baum, now the CTO, and Hadar Zeitlin, now the Chief Business Development Officer, were all members of the Israel Defense Forces.

Over the past two years, the company has been infused with $26.5 million in venture capital, which it has used to bring the Halilo-8 chip to market. Truth be told, the product is not yet generally available. Hailo is currently sampling the chip for “select customers” across a number of industries, with a general release date still to be determined. Initially, Hailo’s major focus will be in the autonomous vehicle (AV) space, a market that has a convenient on-ramp in the form of products being developed for cars with advanced driver-assistance systems (ADAS).

At this point, Nvidia and Intel are leading the field in the AV and ADAS space, based on their respective AI technologies.  Nvidia latest offering in this area is based on its Jetson AGX Xavier chip, an AI processor that delivers up to 30 TOPS (INT8) of performance and draws 30 watts. Intel’s newest chip for this space is its fourth-generation Mobileye EyeQ SoC, which delivers 2.5 teraflops (FP16) in about 3 watts.

Hailo-8’s 26 TOPS (presumably INT8) with just a handful of watts looks pretty good when you look at the raw numbers. The company compared its Hailo-8 against Nvidia’s Xavier AGX on the ResNet-50 benchmark and reported that for a similar frame rate, its chip provided 2.8 TOPS/watt, while the Nvidia silicon delivered only 0.14 TOPS/watt. Of course, there are a lot of other criteria to consider, like cost, software maturity, and the completeness of the solution, but beating the competition in raw performance-per-watt by a factor of 20 is an impressive start.

Not every edge application will require the same level of performance and energy efficiency, or even the same level of integration with their host, so it’s not a zero-sum competition for these companies. In fact, given all the different types of devices and application requirements, as well as the size of the inference market, there’s apt to be a lot more diversity in the AI edge space than in the datacenter. Which means not only do we expect to see more such companies popping up on regular basis, but we also expect to see the big IT providers gobbling up the best of them as the AI edge market gains momentum.