If you’ve been reading about Apache Spark, you might be worried about whether you have to relearn all of your skills for using it to interact with databases. With Apache Spark, whether you’re a DBA or a developer, you’ll be able to interact with Apache Spark in the way you’re used to—while solving real problems.
What Is SparkSQL?
SparkSQL, as the name suggests, is a way to use Apache Spark using the SQL language. Apache Spark makes it easy to run complex queries over lots of nodes, something that’s rather difficult with conventional RDBMSs like MySQL.
Unlike a NoSQL database, you don’t have to learn a new query language or database model. It offers the advantage of NoSQL in scalability, and ease of running over a cluster while using the familiar SQL query model. You can import a number of different data formats in SparkSQL, such as Parquet files, JSON data, as well as RDDs (the native data format of Apache Spark).
SparkSQL allows for both interactive and batch operations. You can take advantage of Spark’s speed, running queries in real time. Spark is so fast partly because of lazy evaluation, which means that queries won’t actually be computed until you need some kind of output.
By using a REPL (i.e. interactive shell), you can explore your data using SparkSQL in real time. You can choose either Spark’s native Scala or Python.
If you haven’t noticed, Spark draws on a lot of functional programming concepts from languages like Haskell and Lisp: lazy evaluation, immutable data structures, and an interactive REPL. These concepts aren’t exactly new, as Lisp data back to the late ‘50s.
