Creating a Big Data environment with Hadoop Pseudodistributed Mode with Apache Spark

Creation, assembly, installation, configuration and documentation of a Big Data environment with Hadoop Pseudodistributed Mode, containing Apache Spark, two of the most advanced frameworks for Big Data processing, and also highlighting how this environment can be useful for Big Data processing.

Python Data Engineer - Data Analysis and Graph Visualizations.

The ultimate goal is to show specific metrics in some chart views. A Dataset in CSV format contains data from candidates who participated in a company's selection processes (simulated data), and you will have to carry out some analyzes and manipulations based on this data.

SQL Analytics and Data Science

Analytical skills with SQL to extract statistics, perform calculations and aggregations, summarize data, generate reports and perform various types of analysis.

Data Ingestion and Analysis with Azure Databricks

Data ingestion and analysis project using AWS and Azure technologies, integrated with the powerful Databricks platform.

Serverless Data Engineering Project

This project aims to create an automated and scalable data pipeline to collect, process, and visualize weather information. The data source is an external API that provides real-time weather information, such as temperature and geographic location.

Azure Pipelines NYC Payroll Analytics

Data Integration Pipelines for NYC Payroll Data Analytics

Real-Time ETL Pipeline with Airflow, Kafka, Spark and Cassandra

Real-time ETL pipeline using Airflow, Kafka, Spark, and Cassandra to ingest, process, and store streaming data in a distributed, containerized architecture.

Databricks Spark Engineering

Distributed data engineering with Apache Spark, PySpark and Databricks. Hands-on examples covering Delta Lake, Unity Catalog, performance tuning and Structured Streaming - built for production pipelines and certification preparation.

Medallion Lakehouse with SCD Type 1 & 2 — Azure Databricks

End-to-end Lakehouse pipeline on Azure Databricks implementing the Medallion Architecture with SCD Type 1 and Type 2 via Delta Lake MERGE. Auto Loader ingests JSON, Parquet and CSV into external Delta tables on ADLS Gen2, with full Unity Catalog governance: RBAC, PII tags, column masking and row-lev

Medallion Lakehouse on AWS (Glue + Iceberg)

AWS-native Medallion Lakehouse (Bronze/Silver/Gold) built with AWS Glue, Apache Iceberg, Lake Formation and Step Functions. Implements SCD Type 1 and Type 2 for historical tracking of customer and product data, orchestrated end-to-end and validated via Amazon Athena.