ETL workflow and data analysis. ETL-workflow using prefect and pygrametl (SCD, slow changing dimension). Product classification based on product name.

Introduction to the data pipeline management with Airflow. Airflow schedule and maintain numerous ETL processes running on a large scale Enterprise Data Warehouse

etl日志收集分析工具

Python | ETL | Google APIs

Amazon Reviews Metrics

This repository contains Data Engineering solution using ETL (Extract, Transform, Load) implementation for the sales data analysis of Apple products. The solution is designed to handle diverse data formats and is implemented on Databricks using PySpark, Python, and Databricks utilities.Factory Method Design Pattern has been implemented for reading.

Framework to write ETL Pipelines controlled by a central config store.

Repository for playing with spark

dataflow pipeline mlflow spider

A simple data processing framework for a quick, no-frills setup of a local data pipeline.

utility to enable flexible ETL scenarios, supports golang plug-in for built-in consumer|transformer|producer options

Data Tweak is a simplified, lightweight ETL framework based on Apache Spark.

Python package that enables customized loading of data from a CSV file into a MySQL database

The repository contains Structured Query Language (SQL) Scripts. The Multiple SQL scripts for various projects which includes data cleaning, data pre-processing, data processing, data transformation and insights gaining through Query Language.

Collection of pkgs to build pipelines in JS/TS

Apache Spark based 'Dist' utility to supplement Data Cooker ETL tool

ETL project for EIA alternative fuels

Effectively demonstrate the ETL process using data on Covid-19, Python, and a non-relational database (MongoDB). Created a web scraper to automate the pulling of data.

A repository containing a Power BI project leveraging the "global_superstore_2016" dataset, showcasing visualizations and insights derived from the data for global sales analysis and forecasting.