This post explains how to work with date and time in pandas. Date and time are very common for a dataset to have. Based on the use case, the column should be transformed.

Spark support various data sources. Spark has some core data sources built into it while the others are available and maintained by other developers from the community. In this post, I am going to explain the core data sources supported by pyspark.

I had some issue reading a csv directly on the databricks community edition. So after going through some articles, I finally found the workaround. Databricks has disabled to use csv directly for pandas as you may encounter FileNotFoundError: [Errno 2] No such file or directory:.