Workaround to read csv from DBFS using pandas
I had some issue reading a csv directly on the databricks community edition. So after going through some articles, I finally found the workaround. Databricks has disabled to use csv directly for pandas as you may encounter FileNotFoundError: [Errno 2] No such file or directory:.
Spark Architecture
Apache spark is a distributed compute engine used to process large volume/amount of data. In this article I am going to provide information on how it works behind the scenes.
Basics of spark with scala
In this article, we will go through the basics of spark with scala. The topics covered are:
Data Engineer | Photographer | Traveller
I am a data engineer with more than 7 years of experience in the software industry. I started my career as a system analyst (QA automation) and eventually switched to software development. I am a wildlife photographer with keen interest in wildlife conservation and research.
Learn More