Simple ETL Project for beginners
Setting up PostgreSQL using docker
Pandas
Pandas Helper Functions
Some functions which are useful in EDA or other DE efforts.
In this article, I am going to provide few helper functions which can be used for various data engineering tasks. I will keep appending new functions with time.
Combine csv files python
Combine multiple csv files in python
import os
import glob
import pandas as pd
path = os.getcwd()
extension = 'csv'
csv_files = glob.glob('*.{}'.format(extension))
df_list = []
for file in csv_files:
df = pd.read_csv(file)
df_list.append(df)
pd.concat(df_list).to_csv("combined_file.csv", index=False)
Note: In order to perform the same with excel, change the value of extension and use read_excel method instead read_csv.
