Working with columns in pyspark
Selecting and renaming dataframes in pyspark
This article with cover various ways of selecting columns in spark dataframe. For the demo, we are going to use Auto-mpg dataset available from kaggle.
Pandas Datetime
Manipulating datetime columns in pandas
This post explains how to work with date and time in pandas. Date and time are very common for a dataset to have. Based on the use case, the column should be transformed.
Pyspark Data Sources
Pyspark supported data sources
Spark support various data sources. Spark has some core data sources built into it while the others are available and maintained by other developers from the community. In this post, I am going to explain the core data sources supported by pyspark.
