Scheduling Big Data Workloads and Data Pipelines in the Cloud with pyDag

Run Jupyter Notebooks (and store data) on Google Cloud Platform.

An educational project to build an end-to-end pipline for near real-time and batch processing of data further used for visualisation and a machine learning model.

GCP_Data_Enginner

Scalable Trust-Signal Detection: A Big Data pipeline using PySpark and GCP Dataproc to classify 8GB+ of Amazon reviews with high-precision Random Forest modeling. Engineered for horizontal scalability and verified data integrity

Data Workflows with GCP Dataproc, Apache Airflow and Apache Spark

Weekly-automated data engineering pipeline that tracks global sea turtle migration patterns by ingesting occurrence records from the OBIS REST API (1.5M+) across 4 endangered species, detecting anomalous sighting patterns using STL seasonal decomposition, and surfacing findings via Streamlit.

Yelp ETL Pipeline in Apache Spark on Google Cloud Dataproc

PySpark Job that runs in Dataproc cluster, loads data from Cloud Storage to BigQuery table.

A Java based project aims to extract news articles from large .sgm file, process them and load them into MongoDB Database. It includes an Apache Spark job for word frequency analysis directly from .sgm files, and a sentiment analysis implementation using a Bag-of-Words model in Java.

A Scala Spark based project to experiment with map-reduce algorithms on big data graph shaped

This repo contains details about how I leveraged mock data to build Pyspark real time streaming to Iceberg table on GCS Bucket and creating Big Query External Tables on GCS bucket, Thanks

Código fuente: Análisis de Vuelos basado en trabajo de Valliappa Lakshmanan.

This report contains details about how youtube trends are implemented with sample mock data generator and leveraging Pyspark using Dataproc cluster and scheduling the Pyspark scripts leveraging Airflow and creating tables in Apache Iceberg format in GCS bucket and also creating Big Query aggregated tables, Thanks

Determination of which words occur in a dataset of textbooks along with each word's occurrence count identification with the help of Google Cloud Platform based Dataproc cluster formation.

Projeto do Curso "Criando um Ecossistema Hadoop Totalmente Gerenciado com Google Cloud Dataproc" do Bootcamp Data Engineer da Digital Innovation One

Example terraform project using GCP to provision an Apache Spark Cluster with a Jupyter Notebook interface.

Training a classification model as a Dataproc Job and using Kafka/PubSub connector for real-time prediction using pre-trained models

Creating gcloud dataproc cluster with this github action