In this spark series, I will continue to add more details on the spark dataframe. This post will specifically cover various aspects of dataframe creation. I am going to include how a python datastructure can be converted into a spark dataframe and how to specify the data type/schema associated with the spark dataframe.These series will have more code snippets and less explaination as most of the things are self explanatory.

Docker is used widely among software development teams across the globe. It has become a norm for software developers to use docker for rapid application development. In this article I will be covering basics of docker, commonly used terms and concepts around docker. I will be doing the steps/commands on macos and python project. But the concepts can be applied to other projects as well.

Serverless computing has become very popular in recent years. Serverless computing allows teams/organizations to develop an application without worrying about the server. Official AWS documentation describes it as - Run Code Without Thinking About Servers. There are many use cases in which serverless computing can be beneficial over the traditional approach. If done in the right way, it can also help small teams save money on infrastructure.