How to Ace the Exam
I recently cleared databricks associate spark developer certification. In this article, I am going to list down the resources I used in for the exam preparation. Here is the curriculum for the exam
Datetime manipulation methods in pyspark
This article with cover various datetime transformations in spark dataframe. For the demo, we are going to use Store Sales dataset available from kaggle.
Dropping columns in pyspark
Working with large datasets in PySpark can be challenging, especially when it comes to cleaning and preprocessing the data. One of the most common tasks in data preprocessing is dropping unnecessary columns from a DataFrame. In this blog post, we will explore how to drop columns from a PySpark DataFrame using the drop() method.
Data Engineer | Photographer | Traveller
I am a data engineer with more than 7 years of experience in the software industry. I started my career as a system analyst (QA automation) and eventually switched to software development. I am a wildlife photographer with keen interest in wildlife conservation and research.
Learn More