Big Data & SQL: The Bipartisan Solution We Really Need

As our data assets continue to grow larger and more complex, the need for Big Data technologies has never been greater. Today, there are many tools are frameworks available to handle these needs, most built on the underlying Hadoop framework. Since Hadoop’s inception over a decade ago, many individuals and organizations have been struggling to leverage the framework due to its requirement for Java programming skills. There have even been articles and blogs published describing Hadoop as “the death of SQL”. This presented an immense challenge for organizations that derive value from traditional Data Warehousing and SQL skills.

Fortunately, the industry answered in a big way.  The Apache Software Foundation brought SQL into the big data space with a “SQL-to-MapReduce” platform called Hive, an interactive query language called Impala and a “SQL-to-Hadoop” ingestion engine called Sqoop.  Eventually, Data Warehousing was brought full-scale into the big data space with tools such as Teradata and Informatica, as well as cloud resources like Azure SQL Data Warehouse, Amazon Redshift and Google BigQuery, to name a few. There are even some tools emerging that leverage Hybrid SQL languages, such as Azure Data Lake Analytics’ U-SQL, to combine the flexibility of other programming languages with the familiarity of SQL.

Needless to say, Big Data will not be “the death of SQL”.  There are many organizations harmoniously leveraging SQL alongside Hadoop to create amazing Big Data Analytic Solutions that seamlessly leverage existing Data Warehousing and application development skillsets.  The same fundamentals are being applied to leverage SQL alongside use cases like Data Lakes, Streaming Data, Machine Learning and Internet of Things applications.

Contrary to early report predictions, the future of data analytics will not see the downfall of traditional Data Warehousing, but an integration of these techniques with Data Lakes, Streaming Data, Machine Learning and Internet of Things applications.  At Syntelli Solutions, we are experts in integrating the “tried and true” with the “fast and new”.