In this article, we will go through the basics of spark with scala. The topics covered are:

In this spark series, I will continue to add more details on the spark dataframe. This post will specifically cover various aspects of dataframe creation. I am going to include how a python datastructure can be converted into a spark dataframe and how to specify the data type/schema associated with the spark dataframe.These series will have more code snippets and less explaination as most of the things are self explanatory.

Spark is very popular among data engineering professionals and companies working with large scaled data. Spark enables us to process large volumes of data with ease in a clusted (Distributed) environment.