Channel Title: The Geek Narrator Channel Description: The GeekNarrator is here to make you curious, excited and inspired about Technology and Software Engineering. You will see in depth technical discussions with actionable insights from the industry experts. These insights will help you get better at Software Engineering, as it comes from the real world engineering experience. <video> Database Trends and More with Peter Zaitsev @percona Description: Deep Dive into Databases with Peter Zaitsev | The GeekNarrator Podcast Join host Kaivalya Apte and special guest Peter Zaitsev from Percona on this episode of the Geeknerder Podcast. They discuss Peter's fascinating journey into the world of databases, founding Percona, and the evolution of open source database solutions. Topics include the rise of PostgreSQL, the comparison between MySQL and PostgreSQL, database observability, the impact of cloud and Kubernetes on database management, licensing changes in popular databases like Redis, and career advice for database administrators and developers. Stay tuned for insights on the future of databases, observability strategies, and the role of AI in database management. 00:00 Introduction and Guest Welcome 00:14 Peter's Journey into Databases 04:15 The Rise of PostgreSQL vs MySQL 18:17 Challenges in Managing Database Clusters 24:36 Common Developer Mistakes with Databases 30:59 MongoDB's Success and Future 34:53 Redis and Licensing Changes 37:07 Elastic's License Change and Its Impact 38:25 Redis Fork and Industry Collaboration 40:27 Kubernetes and Cloud-Native Databases 47:47 Challenges in Database Upgrades and Migrations 54:58 Load Testing and Observability 01:09:02 Future of Database Administration and Development 01:15:13 Conclusion and Final Thoughts Become a member of The GeekNarrator to get access to member only videos, notes and monthly 1:1 with me. Like building stuff? Try out CodeCrafters and build amazing real world systems like Redis, Kafka, Sqlite. Use the link below to signup and get 40% off on paid subscription. https://app.codecrafters.io/join?via=geeknarrator If you like this episode, please hit the like button and share it with your network. Also please subscribe if you haven't yet. Database internals series: https://youtu.be/yV_Zp0Mi3xs Popular playlists: Realtime streaming systems: https://www.youtube.com/playlist?list=PLL7QpTxsA4se-mAKKoVOs3VcaP71X_LA- Software Engineering: https://www.youtube.com/playlist?list=PLL7QpTxsA4sf6By03bot5BhKoMgxDUU17 Distributed systems and databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4sfLDUnjBJXJGFhhz94jDd_d Modern databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4scSeZAsCUXijtnfW5ARlrsN Stay Curios! Keep Learning! </video> <video> World of Stream Processing with Apurva Mehta and Yingjun Wu Description: Understanding Stream Processing: Trends, Challenges, and Future Directions with Apoorva Mehta and Yingjun In this episode, host Kaivalya Apte discusses stream processing, its evolution, current trends and future directions, with experts Apoorva Mehta, co-founder and CEO of Responsive, and Yingjun, co-founder of Rising Wave Labs. Topics include the benefits and challenges of using Kafka Streams, the role of cloud-native object storage, and the importance of real-time processing. The conversation also covers different paradigms such as batch vs. stream processing, consistency issues, and exactly-once processing, providing insights and practical advice for businesses and developers. Chapters: 00:00 Introduction to the GeekNerdR Podcast 00:29 Meet the Experts: Apoorva and Yingjun 02:39 Stream Processing: Background and Evolution 05:41 The Rise of Cloud-Native Systems 07:11 Kafka's Role in Stream Processing 21:09 Understanding Real-Time and Freshness in Data 34:43 Choosing Between Stream and Batch Processing 35:53 Legacy Systems and the Move to Stream Processing 37:16 Real-Time Monitoring and Business Opportunities 38:59 Understanding Different Processing Paradigms 41:41 Kafka Streams and Application Development 44:00 User Personas and System Choices 50:45 Implementing Real-Time Data Limits 54:50 Hybrid Models and Streaming Databases 56:44 Consistency and Exactly Once Processing 01:03:26 Checkpointing and Cloud-Native State Implementation 01:10:00 Conclusion and Final Thoughts Links: RisingWave Labs: https://risingwave.com/ Responsive: https://www.responsive.dev/ Become a member of The GeekNarrator to get access to member only videos, notes and monthly 1:1 with me. Like building stuff? Try out CodeCrafters and build amazing real world systems like Redis, Kafka, Sqlite. Use the link below to signup and get 40% off on paid subscription. https://app.codecrafters.io/join?via=geeknarrator If you like this episode, please hit the like button and share it with your network. Also please subscribe if you haven't yet. Database internals series: https://youtu.be/yV_Zp0Mi3xs Popular playlists: Realtime streaming systems: https://www.youtube.com/playlist?list=PLL7QpTxsA4se-mAKKoVOs3VcaP71X_LA- Software Engineering: https://www.youtube.com/playlist?list=PLL7QpTxsA4sf6By03bot5BhKoMgxDUU17 Distributed systems and databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4sfLDUnjBJXJGFhhz94jDd_d Modern databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4scSeZAsCUXijtnfW5ARlrsN Stay Curios! Keep Learning! </video> <video> System Design the formal way with FizzBee Description: In this video I talk to Jayaprabhakar Kadarkarai aka JP who is the founder of FizzBee. FizzBee is a design specification language and model checker to help developers verify their design before writing even a single line of implementation code. We have discussed where it is applicable, what are the benefits, how does it work and many other interesting challenges with examples. Chapters: 00:00 Introduction 01:13 Challenges in Designing Distributed Systems 03:13 Understanding Design Specification Languages 04:00 The Value of Structured Design Documents 09:00 When to Use Design Specification Languages 21:27 Modeling a Travel Booking System 22:51 Ensuring Atomicity in Distributed Systems 26:09 Handling Failures and Consistency 34:45 Refinement in System Design 35:38 Balancing Abstraction and Implementation 37:53 Common Pitfalls in Modeling and Implementation 40:02 Challenges in System Design and Implementation 40:12 Two-Way Feedback in System Design 41:01 Performance Considerations in Implementation 41:36 Importance of Solid Design Blueprints 41:56 Model-Based Testing and Continuous Integration 43:27 Updating Design Documentation 44:38 Simulation Testing vs. Model Checking 45:32 Design Issues and Formal Verification 49:51 Applying Formal Verification to Existing Systems 55:35 Common Design Problems and Solutions 01:07:57 Future Enhancements in Design Specification Tools 01:12:50 Getting Started with FizzBee FizzBee : https://fizzbee.io/ Get in touch with JP: https://www.linkedin.com/in/jayaprabhakar Like building stuff? Try out CodeCrafters and build amazing real world systems like Redis, Kafka, Sqlite. Use the link below to signup and get 40% off on paid subscription. https://app.codecrafters.io/join?via=geeknarrator If you like this episode, please hit the like button and share it with your network. Also please subscribe if you haven't yet. Database internals series: https://youtu.be/yV_Zp0Mi3xs Popular playlists: Realtime streaming systems: https://www.youtube.com/playlist?list=PLL7QpTxsA4se-mAKKoVOs3VcaP71X_LA- Software Engineering: https://www.youtube.com/playlist?list=PLL7QpTxsA4sf6By03bot5BhKoMgxDUU17 Distributed systems and databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4sfLDUnjBJXJGFhhz94jDd_d Modern databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4scSeZAsCUXijtnfW5ARlrsN Stay Curios! Keep Learning! #distributedsystems #systemdesign #formalmethods </video> <video> WarpStream: A drop-in replacement for Kafka Description: In this episode of The GeekNarrator podcast, host Kaivalya Apte interviews Ryan and Richie, the founders of WarpStream. They discuss the architecture, benefits, and core functionalities of WarpStream, a drop-in replacement for Apache Kafka. The conversation covers their experience with Kafka, the design decisions behind WarpStream, and the operational challenges it addresses. They also delve into the seamless migration process, the scalability, and cost benefits, the integration with the Kafka ecosystem, and potential future features. This episode is a must-watch for developers and tech enthusiasts interested in modern, distributed data streaming solutions. Chapters: 00:00 Introduction 02:27 Introducing Warpstream: A Kafka Replacement 11:07 Deep Dive into Warpstream's Architecture 35:42 Exploring Kafka's Ordering Guarantees 36:52 Handling Buffering and Compaction 38:44 Efficient Data Reading and File Caching 44:06 WarpStream's Flexibility and Cost Efficiency 01:06:59 Future Features Links: WarpStream : https://www.warpstream.com/ Blog: https://www.warpstream.com/blog X: Ryan: https://x.com/ryanworl Richard Artoul: https://x.com/richardartoul Kaivalya Apte: https://x.com/thegeeknarrator Like building stuff? Try out CodeCrafters and build amazing real world systems like Redis, Kafka, Sqlite. Use the link below to signup and get 40% off on paid subscription. https://app.codecrafters.io/join?via=geeknarrator If you like this episode, please hit the like button and share it with your network. Also please subscribe if you haven't yet. Database internals series: https://youtu.be/yV_Zp0Mi3xs Popular playlists: Realtime streaming systems: https://www.youtube.com/playlist?list=PLL7QpTxsA4se-mAKKoVOs3VcaP71X_LA- Software Engineering: https://www.youtube.com/playlist?list=PLL7QpTxsA4sf6By03bot5BhKoMgxDUU17 Distributed systems and databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4sfLDUnjBJXJGFhhz94jDd_d Modern databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4scSeZAsCUXijtnfW5ARlrsN Stay Curios! Keep Learning! #distributedsystems #kafka #s3 #streaming </video> <video> How to Write Powerful Assertions with Antithesis in Your Code Description: Learn different ways to write powerful assertions in Antithesis, including using the SDK and accessing the analytic database. Discover how to find and convert error messages into programmatic assertions, ensuring better code quality and preventing future issues. #AssertionWriting #AntithesisSDK #CodeQuality #ErrorMessages #ProgrammaticAssertions #SDKIntegration #CodeTesting #BugPrevention #AnalyticDatabase #CodeOptimization </video> <video> DBOS internals - Build reliable backends 10x faster Description: The GeekNarrator memberships can be joined here: https://www.youtube.com/channel/UC_mGuY4g0mggeUGM6V1osdA/join Membership will get you access to member only videos, exclusive notes and monthly 1:1 with me. Here you can see all the member only videos: https://www.youtube.com/playlist?list=UUMO_mGuY4g0mggeUGM6V1osdA ------------------------------------------------------------------------------------------------------------------------------------------------------------------ About this episode: ------------------------------------------------------------------------------------------------------------------------------------------------------------------ In this episode we are talking to Peter and Qian, co-founders of DBOS. The conversation covers the challenges of creating fault-tolerant applications, the architecture of DBOS, and how it addresses reliability at multiple layers. Chapters: 00:00 Introduction to the Geeknerder Podcast 00:29 Meet the Co-Founders of DBOSS 01:25 The Core Problem: Building Reliable Systems 02:05 How DBOSS Solves Reliability Issues 04:29 Understanding DBOSS Architecture 06:09 Deep Dive into DBOSS Library 08:36 Postgres and State Management 18:31 Handling Parallel Steps and Performance Concerns 26:00 Observability and Version Control 30:18 Running Multiple Code Versions 30:58 Managing Workflow Versions 32:03 Surgery on Workflow States 33:15 Library Annotations and Durable Execution 34:24 Migrating to the Cloud Version 37:23 Handling Email Workflows 42:41 Transactional Guarantees with Postgres 48:44 Technical Challenges and Multi-Tenancy 54:12 Real-World Use Cases and Benefits 59:45 Conclusion and Final Thoughts Some important links: - Main website: https://www.dbos.dev/ - DBOS docs: https://docs.dbos.dev/ - Open-source DBOS Transact libraries: - Python: https://github.com/dbos-inc/dbos-transact-py - TypeScript: https://github.com/dbos-inc/dbos-transact-ts ------------------------------------------------------------------------------------------------------------------------------------------------------------------ Like building real stuff? ------------------------------------------------------------------------------------------------------------------------------------------------------------------ Try out CodeCrafters and build amazing real world systems like Redis, Kafka, Sqlite. Use the link below to signup and get 40% off on paid subscription. https://app.codecrafters.io/join?via=geeknarrator ------------------------------------------------------------------------------------------------------------------------------------------------------------------ Link to other playlists. LIKE, SHARE and SUBSCRIBE ------------------------------------------------------------------------------------------------------------------------------------------------------------------ If you like this episode, please hit the like button and share it with your network. Also please subscribe if you haven't yet. Database internals series: https://youtu.be/yV_Zp0Mi3xs Popular playlists: Realtime streaming systems: https://www.youtube.com/playlist?list=PLL7QpTxsA4se-mAKKoVOs3VcaP71X_LA- Software Engineering: https://www.youtube.com/playlist?list=PLL7QpTxsA4sf6By03bot5BhKoMgxDUU17 Distributed systems and databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4sfLDUnjBJXJGFhhz94jDd_d Modern databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4scSeZAsCUXijtnfW5ARlrsN Stay Curios! Keep Learning! </video> <video> Practical Systems Learning & Verification with Jack Vanlightly Description: Practical Systems Learning & Verification with Jack Vanlightly Welcome to The GeekNarrator podcast! In this episode, host Kaivalya Apte goes deeper into the practical applications of formal methods with Jack Vanlightly, a principal technologist at Confluent. With years of experience in distributed systems, Jack discusses his journey and how formal methods have been instrumental in system design verification and bug detection. The conversation covers Jack's background, his process of using formal methods, the significance of modelling, verification, documentation, and systems learning, as well as the future evolution of tooling and its applications. Tune in to understand the intricacies of how formal methods can transform your approach to distributed systems! Chapters: 00:00 Introduction to the episode 00:37 Meet Jack VanLightly: Principal Technologist at Confluent 02:17 Jack's Journey into Distributed Systems 04:29 Discovering the Power of Formal Methods 08:11 Modeling and Simulation in Formal Methods 13:43 Verification and Safety Properties 19:02 Documentation and Communication Challenges 20:43 Formal Methods as a Systems Learning Tool 24:26 Practical Applications and Case Studies 56:38 Future of Formal Verification and Closing Thoughts Jack's Blog: https://jack-vanlightly.com/ Become a member of The GeekNarrator to get access to member only videos, notes and monthly 1:1 with me. Like building stuff? Try out CodeCrafters and build amazing real world systems like Redis, Kafka, Sqlite. Use the link below to signup and get 40% off on paid subscription. https://app.codecrafters.io/join?via=geeknarrator If you like this episode, please hit the like button and share it with your network. Also please subscribe if you haven't yet. Database internals series: https://youtu.be/yV_Zp0Mi3xs Popular playlists: Realtime streaming systems: https://www.youtube.com/playlist?list=PLL7QpTxsA4se-mAKKoVOs3VcaP71X_LA- Software Engineering: https://www.youtube.com/playlist?list=PLL7QpTxsA4sf6By03bot5BhKoMgxDUU17 Distributed systems and databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4sfLDUnjBJXJGFhhz94jDd_d Modern databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4scSeZAsCUXijtnfW5ARlrsN Stay Curios! Keep Learning! </video> <video> Building a continuous profiler with Frederic from Polar Signals Description: Building a Continuous Profiler with Frederic from Polar Signals | Geek Narrator Podcast In this episode we chat with Frederic from Polar Signals. We dive deep into the intricacies of building a continuous profiler, the challenges faced, and the unique solutions developed by Polar Signals. Frederic shares insights from his background in observability and discusses the innovations in FrostDB, a custom columnar database designed for high-performance query and storage of profiling data. Chapters: 00:00 Introduction 00:29 Frederic's Background 03:40 What is Continuous Profiling? 06:56 Challenges in Data Collection 18:22 Profiling Data Ingestion and Storage Architecture 27:23 Querying Data 28:52 High Cardinality Data and Cost Optimization 23:39 Tenant Isolation and Load Management 41:24 Performance Optimizations 46:02 Testing & Deterministic Simulation 50:33 Technical and Organizational Learnings 54:32 Future of Polar Signals 56:21 Conclusion You can check more about Polar Signals here: https://www.polarsignals.com/ Become a member of The GeekNarrator to get access to member only videos, notes and monthly 1:1 with me. Like building stuff? Try out CodeCrafters and build amazing real world systems like Redis, Kafka, Sqlite. Use the link below to signup and get 40% off on paid subscription. https://app.codecrafters.io/join?via=geeknarrator If you like this episode, please hit the like button and share it with your network. Also please subscribe if you haven't yet. Database internals series: https://youtu.be/yV_Zp0Mi3xs Popular playlists: Realtime streaming systems: https://www.youtube.com/playlist?list=PLL7QpTxsA4se-mAKKoVOs3VcaP71X_LA- Software Engineering: https://www.youtube.com/playlist?list=PLL7QpTxsA4sf6By03bot5BhKoMgxDUU17 Distributed systems and databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4sfLDUnjBJXJGFhhz94jDd_d Modern databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4scSeZAsCUXijtnfW5ARlrsN Stay Curios! Keep Learning! #distributedsystems #systemdesign #formalmethods </video> <video> Power of #Duckdb with Postgres: pg_duckdb Description: The GeekNarrator memberships can be joined here: https://www.youtube.com/channel/UC_mGuY4g0mggeUGM6V1osdA/join Membership will get you access to member only videos, exclusive notes and monthly 1:1 with me. Here you can see all the member only videos: https://www.youtube.com/playlist?list=UUMO_mGuY4g0mggeUGM6V1osdA ------------------------------------------------------------------------------------------------------------------------------------------------------------------ About this episode: ------------------------------------------------------------------------------------------------------------------------------------------------------------------ Hey folks - In this episode we have Jelte with us, who is the main contributor to the pg_duckdb project, which is a postgres extension to add the #duckdb power to our beloved #postgresql. We will try to understand how it works? Why is it needed and what's the future of pg_duckdb? If you love #Postgres or #Duckdb or just understanding #database internals then this episode will give you pretty solid insights into Postgres query processing, Duckdb analytics, Postgres extension ecosystem and so on. Basics: pg_duckdb is a Postgres extension that embeds DuckDB's columnar-vectorized analytics engine and features into Postgres. We recommend using pg_duckdb to build high performance analytics and data-intensive applications. Chapters: 00:00 Introduction to PG-DuckDB 03:40 Understanding the Integration of DuckDB with Postgres 06:23 Architecture of PG-DuckDB: Query Processing Explained 10:02 Configuring DuckDB for Analytics Queries 15:37 Managing Workloads: Transactional vs. Analytical 21:02 Observability and Debugging in DuckDB 25:58 Data Deletion and GDPR Compliance 30:46 Schema Management and Migration Challenges 33:14 Managing Schema Changes in Databases 35:21 Upgrading Database Extensions 36:33 Enhancing Data Reading Methods 38:33 Future Features and Improvements 45:54 Use Cases for PGDuckDB 50:03 Challenges in Building the Extension 55:25 Getting Involved with PGDuckDB Important links: The duckdb discord server, which has a pg_duckdb channel inside it: https://discord.duckdb.org/ repo: https://github.com/duckdb/pg_duckdb good-first-issue issues: https://github.com/duckdb/pg_duckdb/issues?q=sort%3Aupdated-desc+is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22 ------------------------------------------------------------------------------------------------------------------------------------------------------------------ Like building real stuff? ------------------------------------------------------------------------------------------------------------------------------------------------------------------ Try out CodeCrafters and build amazing real world systems like Redis, Kafka, Sqlite. Use the link below to signup and get 40% off on paid subscription. https://app.codecrafters.io/join?via=geeknarrator ------------------------------------------------------------------------------------------------------------------------------------------------------------------ Link to other playlists. LIKE, SHARE and SUBSCRIBE ------------------------------------------------------------------------------------------------------------------------------------------------------------------ If you like this episode, please hit the like button and share it with your network. Also please subscribe if you haven't yet. Database internals series: https://youtu.be/yV_Zp0Mi3xs Popular playlists: Realtime streaming systems: https://www.youtube.com/playlist?list=PLL7QpTxsA4se-mAKKoVOs3VcaP71X_LA- Software Engineering: https://www.youtube.com/playlist?list=PLL7QpTxsA4sf6By03bot5BhKoMgxDUU17 Distributed systems and databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4sfLDUnjBJXJGFhhz94jDd_d Modern databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4scSeZAsCUXijtnfW5ARlrsN Stay Curios! Keep Learning! #sql #postgres #databasesystems </video> <video> How would you design a database on Object Storage? Description: Join Kaivalya Apte and Simon Hørup Eskildsen from Turbopuffer as they talk about the complexities of building a database on top of object storage. Discover the key challenges, the nuances of various storage formats, and the critical trade-offs involved. Learn from Simon's rich experience, from his time at Shopify to creating Turbopuffer. This episode covers everything—from approaches to write-ahead logs to multi-tenancy and object storage advancements. Perfect for database enthusiasts and those keen on first-principles thinking! 00:00 Introduction 00:17 Simon's Background and Journey to TurboBuffer 02:42 Challenges in Database Scalability 04:21 Experimenting with Vector Databases 05:02 Cost Implications of Vector Databases 05:52 Architectural Considerations for Search Workloads 07:39 Building a Database on Object Storage 16:14 Designing a Simple Database on Object Storage 26:01 Handling Multiple Writers and Consistency 31:26 Trade-offs in Write Operations 32:36 Optimizing MySQL Write Performance 34:03 Batching Writes in Object Storage 35:08 Time-Based vs Size-Based Batching 36:32 Understanding Amplification in Databases 42:26 Challenges with Cold Queries 44:02 Building and Persisting B-Trees 50:53 Separating Workloads in Databases 56:07 Multi-Tenancy Challenges 01:00:39 Choosing Storage Formats 01:06:10 Key Innovations in Object Storage Databases Important links: - https://github.com/sirupsen/napkin-math (numbers) - https://turbopuffer.com/ - https://turbopuffer.com/architecture - https://sirupsen.com/napkin/problem-10-mysql-transactions-per-second - https://sirupsen.com (my blog, napkin math) - https://sirupsen.com/subscribe (napkin math newsletter) - https://github.com/rkyv/rkyv rkyv rust Become a member of The GeekNarrator to get access to member only videos, notes and monthly 1:1 with me. Like building stuff? Try out CodeCrafters and build amazing real world systems like Redis, Kafka, Sqlite. Use the link below to signup and get 40% off on paid subscription. https://app.codecrafters.io/join?via=geeknarrator If you like this episode, please hit the like button and share it with your network. Also please subscribe if you haven't yet. Database internals series: https://youtu.be/yV_Zp0Mi3xs Popular playlists: Realtime streaming systems: https://www.youtube.com/playlist?list=PLL7QpTxsA4se-mAKKoVOs3VcaP71X_LA- Software Engineering: https://www.youtube.com/playlist?list=PLL7QpTxsA4sf6By03bot5BhKoMgxDUU17 Distributed systems and databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4sfLDUnjBJXJGFhhz94jDd_d Modern databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4scSeZAsCUXijtnfW5ARlrsN Stay Curios! Keep Learning! </video> <video> AutoMQ: A New Kafka Alternative on S3 That Addresses Your Costs and Pain Points Description: In this video, we dive into AutoMQ, an interesting streaming project that reinvent Apache Kafka’s storage layer to gain many attractive features, mainly including: - 10x cost-effective than Apache Kafka: AutoScaling、Support Spot Instance、Separate Storage to S3. All this make AutoMQ 10x cost-effective than Apache Kafka. - Easy to operate: No need to manage the cluster's capacity yourself. Stateless Broker that can autoscale in seconds. Forget data skew, hot and cold data competition. Self-blancing fixes them all automatically. - High performance: Single digit ms latency with high throughput as Apache Kafka, but with much better catch-up reads performance. - Easy to migrate: 100% Compatible with Apache Kafka, so you don't need to change anyting you already have. Access to the new bootstrap server endpoint and all things are done. Explore how AutoMQ integrates with object storage, providing a cost-effective, scalable, and efficient solution for modern cloud-based needs. Learn about the limitations of traditional Kafka storage and how AutoMQ addresses these with its seamless and innovative approach. AutoMQ, the cloud-native Apache Kafka alternative that is 100% compatible with Apache Kafka,10x cost-effective and can auto-scaling in seconds. ------------------------------------------------------------------------------------------------------------------------------------------ Chapters: 00:00 Introduction 00:40 Recap on Apache Kafka 01:33 Kafka’s Challenges with Storage 03:13 Operational Complexity of Kafka 04:35 Tiered Storage Approach 08:07 Introduction to AutoMQ 09:30 Key Features of AutoMQ 10:43 How AutoMQ Works 13:00 Benefits of Using AutoMQ 15:41 Conclusion ------------------------------------------------------------------------------------------------------------------------------------------ A real benchmark on AWS shows the advantages of AutoMQ with Apache Kafka: https://docs.automq.com/automq/benchmarks/benchmark-automq-vs-apache-kafka ------------------------------------------------------------------------------------------------------------------------------------------ To learn more about AutoMQ, the 100% Kafka compatible streaming data platform on S3 with 10x cost-effective and 100x elasticity, visit https://automq.com Join the Slack community! https://automq.slack.com/join/shared_invite/zt-29h17vye9-thf31ebIVL9oXuRdACnOIA#/shared-invite/email Contribute to the GitHub repository: https://github.com/AutoMQ/automq Follow AutoMQ on social media: Twitter - https://x.com/AutoMQ_Lab LinkedIn - https://www.linkedin.com/company/automq/ </video> <video> Learnings from building Open Source Distributed Systems with Kishore Gopalakrishna Description: In this episode of The Geek Narrator podcast, hosted by Kaivalya Apte, we welcome a special guest, Kishore Gopalakrishna from StarTree, co-author of Apache Pinot and other notable projects. Kishore shares his extensive experience in building real-time analytics and streaming systems, including Apache Pino, Espresso, Apache Helix, and Third Eye. The episode delves into the motivations and challenges behind creating these systems, the innovations they brought to distributed systems, and the impact of community on open-source projects. Kishore also discusses the evolution of testing methodologies, cost optimizations in transactional and analytical systems, and key considerations for companies evaluating real-time analytics solutions. Don't miss this in-depth conversation packed with valuable insights for both seasoned developers and tech enthusiasts! Chapters: 00:00 Introduction 03:13 Building Distributed Systems at LinkedIn 08:57 Testing and Challenges in Distributed Systems 30:50 Advantages of Columnar Storage 33:04 The Importance of Upserts 34:24 Building a Strong Open Source Community 41:10 Challenges and Lessons in System Design 51:35 Real-Time Analytics: Do You Need It? StarTree: https://startree.ai/ Apache Pinot: https://pinot.apache.org/ Like building stuff? Try out CodeCrafters and build amazing real world systems like Redis, Kafka, Sqlite. Use the link below to signup and get 40% off on paid subscription. https://app.codecrafters.io/join?via=geeknarrator If you like this episode, please hit the like button and share it with your network. Also please subscribe if you haven't yet. Database internals series: https://youtu.be/yV_Zp0Mi3xs Popular playlists: Realtime streaming systems: https://www.youtube.com/playlist?list=PLL7QpTxsA4se-mAKKoVOs3VcaP71X_LA- Software Engineering: https://www.youtube.com/playlist?list=PLL7QpTxsA4sf6By03bot5BhKoMgxDUU17 Distributed systems and databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4sfLDUnjBJXJGFhhz94jDd_d Modern databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4scSeZAsCUXijtnfW5ARlrsN Stay Curios! Keep Learning! #distributedsystems #kafka #s3 #streaming #realtimeanalytics #database #pinot #startree </video> <video> Database Internals - SlateDB with Chris Riccomini Description: Welcome back to another episode! Today, I have a special guest, Chris Riccomini, joining me to delve into the exciting world of databases. In this episode, we focus on SlateDB, a new and innovative database that's making waves in the tech community. We'll cover a wide range of topics, including the architecture of SlateDB, its internals, design decisions, and some fascinating use cases. Chris, a seasoned software engineer with a background at LinkedIn and WePay, shares his journey and the motivations behind creating SlateDB. 🎙️ Chatpers: 00:00 Introduction to the Topic and Guest 01:58 Chris Riccomini's Background and Experience 04:19 The Genesis of SlateDB 04:54 Understanding SlateDB's Architecture 10:22 The Rise of Object Storage in Databases 13:43 Exploring SlateDB's Features and Trade-offs 32:54 Understanding Latency Trade-offs 34:12 Exploring Storage Formats and Manifest Files 37:25 Caching Strategies and Optimizations in SlateDB 50:21 Consistency Guarantees and Transactionality 52:36 Integration and Resource Management in SlateDB 56:04 Future Prospects and Use Cases for SlateDB SlateDB: https://slatedb.io/ More about Chris: https://cnr.sh/ Like building stuff? Try out CodeCrafters and build amazing real world systems like Redis, Kafka, Sqlite. Use the link below to signup and get 40% off on paid subscription. https://app.codecrafters.io/join?via=geeknarrator If you like this episode, please hit the like button and share it with your network. Also please subscribe if you haven't yet. Database internals series: https://youtu.be/yV_Zp0Mi3xs Popular playlists: Realtime streaming systems: https://www.youtube.com/playlist?list=PLL7QpTxsA4se-mAKKoVOs3VcaP71X_LA- Software Engineering: https://www.youtube.com/playlist?list=PLL7QpTxsA4sf6By03bot5BhKoMgxDUU17 Distributed systems and databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4sfLDUnjBJXJGFhhz94jDd_d Modern databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4scSeZAsCUXijtnfW5ARlrsN Stay Curios! Keep Learning! #distributedsystems #systemdesign #formalmethods </video> <video> Database Internals - NileDB Postgres re-engineered for multitenant apps Description: Database Internals - NileDB: Postgres Re-engineered for Multitenant Apps with Gwen Shapira Join us in this episode as we dive deep into the intricacies of NileDB, a groundbreaking database re-engineered for multi-tenant applications. Our special guest, Gwen Shapira, co-founder of NileDB and a notable figure in the database field, shares her insights and technical know-how on solving the common challenges faced by multitenant SaaS applications. From the benefits of using Postgres as the underlying database to the unique tenant isolation features of NileDB, we cover it all. Don't miss out on learning about AI native capabilities, handling schema migrations, and ensuring zero downtime data migrations. Chapters: 00:00 Introduction 07:19 Challenges in Multi-Tenant Databases 11:09 Tenant Isolation and NILDB's Approach 34:16 Necessary Modifications for Tenant Data 37:47 Zero Downtime Data Migrations 44:32 Handling Schema Migrations 59:11 AI Use Cases and Vector Embedding Storage 59:51 Technical and Non-Technical Learnings from Building Nile 01:05:03 Future Plans and Upcoming Features NileDB: https://www.thenile.dev/ Blog: https://www.thenile.dev/blog Gwen's Linkedin: https://www.linkedin.com/in/gwenshapira Gwen's Twitter: https://twitter.com/gwenshap #postgres #sql #ai Become a member of The GeekNarrator to get access to member only videos, notes and monthly 1:1 with me. Like building stuff? Try out CodeCrafters and build amazing real world systems like Redis, Kafka, Sqlite. Use the link below to signup and get 40% off on paid subscription. https://app.codecrafters.io/join?via=geeknarrator If you like this episode, please hit the like button and share it with your network. Also please subscribe if you haven't yet. Database internals series: https://youtu.be/yV_Zp0Mi3xs Popular playlists: Realtime streaming systems: https://www.youtube.com/playlist?list=PLL7QpTxsA4se-mAKKoVOs3VcaP71X_LA- Software Engineering: https://www.youtube.com/playlist?list=PLL7QpTxsA4sf6By03bot5BhKoMgxDUU17 Distributed systems and databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4sfLDUnjBJXJGFhhz94jDd_d Modern databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4scSeZAsCUXijtnfW5ARlrsN Stay Curios! Keep Learning! </video> <video> The JVM Internals - Basics (Episode 1) Description: Join as a member to get access: https://www.youtube.com/channel/UC_mGuY4g0mggeUGM6V1osdA/join Understanding JVM Internals: Episode 1 - Basics and Key Concepts Hello everyone. In this first episode of the JVM internal series, we look into the fundamentals of the Java Virtual Machine (JVM). We'll explore what the JVM is, why it's powerful, and cover essential concepts such as platform independence, byte-code execution, memory management, concurrency, and runtime data areas. This episode sets the foundation for deeper discussion in future episodes, offering practical insights into tuning JVM parameters and understanding its implementation details. Join me as we go deeper into JVM internals! Chapters: 00:00 Introduction to JVM Internals Series 00:20 Understanding the Basics of JVM 01:25 Platform Independence and JVM Specification 04:13 Bytecode and Class Files 12:44 Execution and Just-In-Time Compilation 14:58 Memory Management and Garbage Collection 16:30 Concurrency and Program Execution 17:31 Overview of Runtime Data Areas 18:37 Conclusion and Next Steps Key references: Java languages specs: https://docs.oracle.com/javase/specs/ JVM specification: https://docs.oracle.com/javase/specs/jvms/se23/jvms23.pdf Stay tuned for upcoming episodes in this series. ============================================================================= Like building stuff? Try out CodeCrafters and build amazing real world systems like Redis, Kafka, Sqlite. Use the link below to signup and get 40% off on paid subscription. https://app.codecrafters.io/join?via=geeknarrator ============================================================================= Database internals series: https://youtu.be/yV_Zp0Mi3xs Popular playlists: Realtime streaming systems: https://www.youtube.com/playlist?list=PLL7QpTxsA4se-mAKKoVOs3VcaP71X_LA- Software Engineering: https://www.youtube.com/playlist?list=PLL7QpTxsA4sf6By03bot5BhKoMgxDUU17 Distributed systems and databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4sfLDUnjBJXJGFhhz94jDd_d Modern databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4scSeZAsCUXijtnfW5ARlrsN Stay Curios! Keep Learning! </video>

Contacts and Links

Description

The GeekNarrator is here to make you curious, excited and inspired about Technology and Software Engineering. You will see in depth technical discussions with actionable insights from the industry experts. These insights will help you get better at Software Engineering, as it comes from the real world engineering experience.

Videos from channel