Channel Title: Soumil Shah Channel Description: As a Data Engineer with expertise in Apache Hudi and Iceberg, I specialize in building scalable data ingestion pipelines in the AWS Big Data ecosystem and data lakes. I am the creator of the "LakeBoost" framework, which integrates Apache Hudi with AWS Glue ETL, significantly improving operational efficiency and reducing costs for large-scale data operations. With advanced skills in Apache Spark and modern data platforms, I design and implement high-performance systems that enable robust data workflows. Beyond engineering, I am passionate about educating others and share my knowledge through my YouTube channel, which has 43,000 subscribers and over 1,600 videos on big data technologies. My goal is to make complex data engineering concepts accessible to a global audience, fostering both technical innovation and learning. <video> Stay away from these financial mistakes  Description: </video> <video> How Customers and Companies Can Use Hudi 1.0.0 on EMR Serverless | Developer Guide Description: Code https://github.com/soumilshah1995/hudi-1-0-0-emr-serverless/blob/main/README.md </video> <video> How to use Custom Images in EMR Serverless | Custom Docker File Description: How to use Custom Images in EMR Serverless | Custom Docker File Reading Materials https://gallery.ecr.aws/emr-serverless/spark/emr-7.5.0 https://github.com/aws-samples/emr-serverless-samples/tree/main/examples/pyspark/custom-images </video> <video> Learn How to configure Spark Session to Join Managed (S3 Table Buckets) and Unmanaged Iceberg Table Description: Learn How to configure your Spark Session to Join Managed (S3 Table Buckets) and Unmanaged Iceberg Tables Blog https://www.linkedin.com/pulse/learn-how-configure-your-spark-session-join-managed-s3-soumil-shah-obhde/?trackingId=jr6W%2BpAhRAebJbMkdQuuqg%3D%3D Code https://soumilshah1995.blogspot.com/2025/01/learn-how-to-configure-your-spark.html Read Seamlessly Migrating to AWS Managed Iceberg Tables: Strategies and Best Practices https://www.linkedin.com/pulse/seamlessly-migrating-aws-managed-iceberg-tables-strategies-shah-b682e/?trackingId=ZcvtenMMTe%2BYGwYGNCRQkA%3D%3D </video> <video> Financial mistakes made at different stages of life #music Description: </video> <video> Thank you everyone for attending talk Description: </video> <video> Learn to Use S3 Table Buckets Locally: Ingest Data from Raw to Silver with MERGE INTO Commands Description: code https://github.com/soumilshah1995/s3tables-locally/blob/main/README.md </video> <video> How to Ingest Data Incrementally from S3 Using S3 Events & SQS into S3 table Bucket|Run it Locally Description: code https://github.com/soumilshah1995?tab=repositories Blog https://www.linkedin.com/pulse/learn-how-ingest-data-incrementally-from-s3-using-events-soumil-shah-qvuhe/?trackingId=cKY%2FQiGtQv2%2FAFKd1LogAA%3D%3D </video> <video> Q/A | Office Hours | Ask Questions | 12/28/2024 Description: </video> <video> Amazon S3 Tables Store Tabular Data in S3 Amazon Web Services Description: </video> <video> Creating a Ray Cluster on EMR on EC2 and Submitting Your First Ray Job #6 Description: Creating a Ray Cluster on EMR on EC2 and Submitting Your First Ray Job #6 code https://github.com/soumilshah1995/emr-ec2-labs/tree/main/labs/lab6 Read More https://aws.amazon.com/blogs/opensource/scaling-ai-and-machine-learning-workloads-with-ray-on-aws/ https://aws.amazon.com/blogs/big-data/introducing-aws-glue-for-ray-scaling-your-data-integration-workloads-using-python/ https://www.ray.io </video> <video> Learn about Apache Hudi 1.0.0 Expression Index | hands on Labs #2 Description: Learn about Apache Hudi 1.0.0 Expression Index | hands on Labs #2 Code https://github.com/soumilshah1995/hudi-1.0.0-labs/blob/main/Labs/lab2.sql Spark SQL https://github.com/soumilshah1995/hudi-1.0.0-labs/blob/main/notes Read More https://hudi.apache.org/docs/sql_queries/#query-using-column-stats-expression-index https://www.linkedin.com/pulse/understanding-expression-indexes-hudi-improving-query-soumil-shah-q4vye/?trackingId=Lqf1a1P4S4qZDvpYf6ASww%3D%3D Apache Hudi </video> <video> Learn How to Ship and Publish Your First Python Package to PyPI with UV Description: 🚀 Want to ship your first Python package to PyPI? 🚀 In my latest blog post, I walk you through the step-by-step process of creating and publishing your Python package using UV. From setting up your package to uploading it to PyPI, this guide simplifies the entire process, so you can share your code with the world easily! Check it out and get started today: How to Ship and Publish Your First Python Package to PyPI with UV https://www.linkedin.com/pulse/learn-how-ship-publish-your-first-python-package-pypi-soumil-shah-os1fe/?trackingId=nTmUwH%2F5SZWY4avbxuuvyA%3D%3D #Python #PyPI #UV #PackageManagement #DeveloperTools #OpenSource </video> <video> ✨ Happy New Year! 2025 🎉 ✨ Description: ✨ Happy New Year! 🎉 ✨ Wishing everyone a year full of growth, learning, and success! 🌟 In 2025, I’m committed to growing, learning, and improving myself every step of the way. I also promise to share my journey and insights through videos, blogs, and posts to help others in their own path to success. Thank you all for your support—it means the world to me. Let’s make this year amazing together! 💪 #HappyNewYear #GrowthMindset #SharingKnowledge </video> <video> How to Use the New Hudi Streamer with Hudi 1.0.0 on EMR Serverless 7.5.0 | hands on Labs Description: How to Use the Hudi Streamer with New Hudi 1.0.0 on EMR Serverless 7.5.0 | hands on Labs Code https://github.com/soumilshah1995/hudi-1.0.0-emr-serverless Steps https://github.com/soumilshah1995/hudi-1.0.0-emr-serverless/blob/main/notes </video>

Contacts and Links

Description

As a Data Engineer with expertise in Apache Hudi and Iceberg, I specialize in building scalable data ingestion pipelines in the AWS Big Data ecosystem and data lakes. I am the creator of the "LakeBoost" framework, which integrates Apache Hudi with AWS Glue ETL, significantly improving operational efficiency and reducing costs for large-scale data operations. With advanced skills in Apache Spark and modern data platforms, I design and implement high-performance systems that enable robust data workflows. Beyond engineering, I am passionate about educating others and share my knowledge through my YouTube channel, which has 43,000 subscribers and over 1,600 videos on big data technologies. My goal is to make complex data engineering concepts accessible to a global audience, fostering both technical innovation and learning.

Videos from channel