Opinion

Big data, small footprint.

Jul 05, 2018

Indefinite hardware capacity is a dangerous – and costly – illusion.

The 21st century digital revolution owes a great deal to the dramatic drop in computational resource costs observed in recent decades. While Moore’s law and its variants churned out ever-more-powerful hardware at ever-less-expensive rates, the advent of cloud-based computing (and the pooled asset-sweating it offers) catapulted affordable super-processing power to the masses.

Presented with seemingly unlimited resources, software engineers have become lax about basic tenets of computational efficiency. As a result, they have embraced a dangerous unbounded mentality: “just throw more CPUs at a task and it will be solved”, many of them have come to think. And, indeed, several software projects are technically and economically feasible without demanding too much in the way of leaner computer science principles.

Times are quickly changing though. The ratio between hardware performance vs. unit costs is no longer increasing at the same pace from the 80s, 90s and 00s. Moore’s law, as originally conceived, is dead. For a multitude of new generation applications, a careless approach to resource consumption may then prove disastrous.  

One such application is network analytics: the sheer magnitude of processing required to go, often in real-time, from billions of unstructured data points to readily consumable insights becomes prohibitively costly when performed inefficiently. Add to that the in-premise infrastructure that network analytics typically requires (due to security reasons and the impossibility of carrying out high-scale data transmission with remote clouds) and it offers the perfect scenario in which ‘getting more done with less’ becomes a must.

“Moore’s law is dead. For a multitude of new-generation applications, a careless approach to resource consumption may thus prove disastrous.”

The academic research out of which Niometrics’ core technology sprung out is, for example, based on high-performance engineering principles. They have enabled the company’s Deep Network Analytics (DNA) platform to deliver big data experience out of significantly smaller footprints, demanding up to 80% less hardware than other alternatives would. When hardware investments can reach millions of dollars, and when impoverishing insights is not an acceptable alternative to minimising costs, that degree of efficiency makes a big difference.

Every component of Niometrics’ stack was designed with resource optimisation in mind. And, just like a sum of parts adding up to a stronger whole, those smaller bits and pieces come together to deliver a markedly high-performing suite of solutions.

Some examples of optimised components in Niometrics stack are:

  • NCORE, the company’s deep packet inspection engine, which extracts, sifts, and provides contextual data further upstream in the form of raw logs, in-memory counters and live analytics alerts. It achieves that with current processing speeds of up to 100 Gbps per node (against the 40 Gbps per node often achieved by other providers), more than halving the investment required for a DPI engine deployment;
  • NLIVE, Niometrics’ in-memory streaming analytics and aggregation software, which performs real-time analyses on massive flows of live events captured by NCORE. It processes up to 1 million events per second per node, handling streaming aggregation and state tracking for up to 10 million active users at once. Such performance levels often require much larger hardware capacity in order to be observed under less optimised frameworks;
  • NSTORE, a high-volume event record indexing system which enables the storage and on-demand querying of individual user information, performing up to 1 million record inserts per second (vs. 40 k inserts per second observed in standard NOSQL databases). Superior insertion rates allow for more user-level events, and therefore more user experience details, to be stored for subsequent consultation and analyses.
  • NOLAP, a bulk processor and embedded analytics platform that maintains complex multi-dimensional summaries over traffic data. It enables queries for extracting aggregate information or detailed examination of sessions of interest by processing up to 600 Mb in 1 pass, delivering OLAP capabilities at a fraction of usual hardware requirements.

When those and other parts come together within Niometrics’ stack, they are individually engineered for maximum performance. On top of that, their tight integration delivers an additional boost that cements a bottleneck-free data pipeline, capable of extracting substantially more output from the hardware dedicated to it. That fulfils a cornerstone of the company’s philosophy: the stanch pursuit of big data outputs with small footprint requirements.