Channel Title: Mutual Information Channel Description: Highly informative videos on machine learning and related topics <video> The Kelly Criterion Description: The machine learning consultancy: https://truetheta.io Join my email list to get educational and useful articles (and nothing else!): https://mailchi.mp/truetheta/true-theta-email-list Want to work together? See here: https://truetheta.io/about/#want-to-work-together The Kelly Criterion provides the optimal strategy when betting on random outcomes with known probabilities. SOCIAL MEDIA LinkedIn : https://www.linkedin.com/in/dj-rich-90b91753/ Twitter : https://twitter.com/DuaneJRich Github: https://github.com/Duane321 Enjoy learning this way? Want me to make more videos? Consider supporting me on Patreon: https://www.patreon.com/MutualInformation TIMESTAMPS 0:00 Intro/The Game 1:16 Growth Rates 3:43 People Play Terribly 4:46 The Best Strategy Possible 7:10 The Kelly Formula Is Dangerous CODE https://github.com/Duane321/mutual_information/tree/main/videos/kelly_criterion SOURCES I first came across the Kelly Criterion in [1], which features it as a powerful example of the relevance and usefulness of information theory. In chapter 6, you'll find an exact and more general discussion of this idea. Wikipedia, [2], was also useful, as it is frequently. That's where I learned about [3], which was an interesting case study about how bad we are at this game. As an aside, I actually ran the experiment on a friend and didn't find similar results - I guess she was pretty smart! She bet about 3-5 dollars on each flip, with the wagers moving to the larger side of this range as the wealth increased. [4] is the original paper. It's a nice explanation, but relies on some background knowledge of information theory. If you're curious about the connection to information theory, this is the place to look. -------------------------------- [1] T. M. Cover and J. A. Thomas. Elements of Information Theory. 2nd edition. John Wiley, 2006. [2] Kelly Criterion, Wikipedia, https://en.wikipedia.org/wiki/Kelly_criterion [3] V. Haghani, R Dewey, "Rational Decision-Making Under Uncertainty: Observed Betting Patterns on a Biased Coin", Journal of Portfolio Management, 2016 [4] J. L. Kelly Jr, "A New Interpretation of Information Rate", Bell System Technical Journal, 1956 </video> <video> Is the Future of Linear Algebra.. Random? Description: The machine learning consultancy: https://truetheta.io Join my email list to get educational and useful articles (and nothing else!): https://mailchi.mp/truetheta/true-theta-email-list Want to work together? See here: https://truetheta.io/about/#want-to-work-together "Randomization is arguably the most exciting and innovative idea to have hit linear algebra in a long time." - First line of the Blendenpik paper, H. Avron et al. Follow up post: https://truetheta.io/concepts/linear-algebra/randomized-numerical-linear-algebra/ SOCIAL MEDIA LinkedIn : https://www.linkedin.com/in/dj-rich-90b91753/ Twitter : https://twitter.com/DuaneJRich Github: https://github.com/Duane321 SUPPORT https://www.patreon.com/MutualInformation SOURCES Source [1] is the paper that caused me to create this video. [3], [7] and [8] provided a broad and technical view of randomization as a strategy for NLA. [9] and [12] informed me about the history of NLA. [2], [4], [5], [6], [10], [11], [13] and [14] provide concrete algorithms demonstrating the utility of randomization. [1] Murray et al. Randomized Numerical Linear Algebra. arXiv:2302.11474v2 2023 [2] Melnichenko et al. CholeskyQR with Randomization and Pivoting for Tall Matrices (CQRRPT). arXiv:2311.08316v1 2023 [3] P. Drineas and M. Mahoney. RandNLA: Randomized Numerical Linear Algebra. Communications of the ACM. 2016 [4] N. Halko, P. Martinsson, and J. Tropp. Finding Structure with Randomness: Probabilistic Algorithms for Constructing Approximate Matrix Decompositions. arXiv:0909.4061v2 2010 [5] Tropp et al. Fixed Rank Approximation of a Positive-Semidefinite Matrix from Streaming Data. NeurIPS Proceedings. 2017 [6] X. Meng, M. Saunders, and M. Mahoney. LSRN: A Parallel Iterative Solver for Strongly Over- Or Underdetermined Systems. SIAM 2014 [7] D. Woodruff. Sketching as a Tool for Numerical Linear Algebra. IBM Research Almaden. 2015 [8] M. Mahoney. Randomized Algorithms for Matrices and Data. arXiv:1104.5557v3. 2011 [9] G. Golub and H van der Vorst. Eigenvalue Computation in the 20th Century. Journal of Computational and Applied Mathematics. 2000 [10] J. Duersch and M. Gu. Randomized QR with Column Pivoting. arXiv:1509.06820v2 2017 [11] Erichson et al. Randomized Matrix Decompositions Using R. Journal of Statistical Software. 2019 [12] J. Gentle et al. Software for Numerical Linear Algebra. Springer. 2017 [13] H. Avron, P. Maymounkov, and S. Toledo. Blendenpik: Supercharging LAPACK's Least-Squares Solver. Siam. 2010 [14] M. Mahoney and P. Drineas. CUR Matrix Decompositions for Improved Data Analysis. Proceedings of the National Academy of Sciences. 2009 TIMESTAMPS 0:00 Significance of Numerical Linear Algebra (NLA) 1:35 The Paper 2:20 What is Linear Algebra? 5:57 What is Numerical Linear Algebra? 8:53 Some History 12:22 A Quick Tour of the Current Software Landscape 13:42 NLA Efficiency 16:06 Rand NLA's Efficiency 18:38 What is NLA doing (generally)? 20:11 Rand NLA Performance 26:24 What is NLA doing (a little less generally)? 31:30 A New Software Pillar 32:43 Why is Rand NLA Exceptional? 34:01 Follow Up Post and Thank You's </video> <video> Temporal Difference Learning (including Q-Learning) | Reinforcement Learning Part 4 Description: The machine learning consultancy: https://truetheta.io Join my email list to get educational and useful articles (and nothing else!): https://mailchi.mp/truetheta/true-theta-email-list Want to work together? See here: https://truetheta.io/about/#want-to-work-together Part four of a six part series on Reinforcement Learning. As the title says, it covers Temporal Difference Learning, Sarsa and Q-Learning, along with some examples. SOCIAL MEDIA LinkedIn : https://www.linkedin.com/in/dj-rich-90b91753/ Twitter : https://twitter.com/DuaneJRich Github: https://github.com/Duane321 Enjoy learning this way? Want me to make more videos? Consider supporting me on Patreon: https://www.patreon.com/MutualInformation SOURCES [1] R. Sutton and A. Barto. Reinforcement learning: An Introduction (2nd Ed). MIT Press, 2018. [2] H. Hasselt, et al. RL Lecture Series, Deepmind and UCL, 2021, https://www.youtube.com/playlist?list=PLqYmG7hTraZDVH599EItlEWsUOsJbAodm SOURCE NOTES The video covers topics from chapters 6 and 7 from [1]. The whole series teaches from [1]. [2] has been a useful secondary resource. TIMESTAMP 0:00 What We'll Learn 0:52 No Review 1:18 TD as an Adjusted Version of MC 2:49 TD Visualized with a Markov Reward Process 6:34 N-Step Temporal Difference Learning 8:08 MC vs TD on an Evaluation Example 11:50 TD's Trade-Off between N and Alpha 12:47 Why does TD Perform Better than MC? 15:29 N-Step Sarsa 17:15 Why have N above 1? 19:02 Q-Learning 20:50 Expected Sarsa 21:48 Cliff Walking 25:04 Windy GridWorld 28:12 Watch the Next Video! NOTES Code to compare TD vs MC on the evaluation task: https://github.com/Duane321/mutual_information/tree/main/videos/td_learning_sarsa_and_q_learning </video> <video> Monte Carlo And Off-Policy Methods | Reinforcement Learning Part 3 Description: The machine learning consultancy: https://truetheta.io Join my email list to get educational and useful articles (and nothing else!): https://mailchi.mp/truetheta/true-theta-email-list Want to work together? See here: https://truetheta.io/about/#want-to-work-together Part three of a six part series on Reinforcement Learning. It covers the Monte Carlo approach a Markov Decision Process with mere samples. At the end, we touch on off-policy methods, which enable RL when the data was generate with a different agent. SOCIAL MEDIA LinkedIn : https://www.linkedin.com/in/dj-rich-90b91753/ Twitter : https://twitter.com/DuaneJRich Github: https://github.com/Duane321 Enjoy learning this way? Want me to make more videos? Consider supporting me on Patreon: https://www.patreon.com/MutualInformation SOURCES [1] R. Sutton and A. Barto. Reinforcement learning: An Introduction (2nd Ed). MIT Press, 2018. [2] H. Hasselt, et al. RL Lecture Series, Deepmind and UCL, 2021, https://www.youtube.com/playlist?list=PLqYmG7hTraZDVH599EItlEWsUOsJbAodm SOURCE NOTES The video covers topics from chapters 5 and 7 from [1]. The whole series teaches from [1]. [2] has been a useful secondary resource. TIMESTAMP 0:00 What We'll Learn 0:33 Review of Previous Topics 2:50 Monte Carlo Methods 3:35 Model-Free vs Model-Based Methods 4:59 Monte Carlo Evaluation 9:30 MC Evaluation Example 11:48 MC Control 13:01 The Exploration-Exploitation Trade-Off 15:01 The Rules of Blackjack and its MDP 16:55 Constant-alpha MC Applied to Blackjack 21:55 Off-Policy Methods 24:32 Off-Policy Blackjack 26:43 Watch the next video! NOTES Link to Constant-alpha MC applied to Blackjack: https://github.com/Duane321/mutual_information/tree/main/videos/monte_carlo_for_RL_and_off_policy_methods The Off-Policy method you see at 25:00 is different from the rule you'll see in the textbook at eq 7.9 (which will be MC if n goes to inf). That's because they are showing re-weighted IS and I'm showing plain ( high variance) IS. </video> <video> Importance Sampling Description: The machine learning consultancy: https://truetheta.io Join my email list to get educational and useful articles (and nothing else!): https://mailchi.mp/truetheta/true-theta-email-list Want to work together? See here: https://truetheta.io/about/#want-to-work-together Calculating expectations is frequent task in Machine Learning. Monte Carlo methods are some of our most effective approaches to this problem, but they can suffer from high variance estimates. Importance Sampling is a clever technique to obtain lower variance estimates. SOCIAL MEDIA LinkedIn : https://www.linkedin.com/in/dj-rich-90b91753/ Twitter : https://twitter.com/DuaneJRich Github: https://github.com/Duane321 Enjoy learning this way? Want me to make more videos? Consider supporting me on Patreon: https://www.patreon.com/MutualInformation SOURCES [1] was my primary source. Chapter 17 of [2] and chapter 23 of [3] provided a useful discussion more directed at the use cases of Machine Learning. ----------------------------- [1] E. Anderson, "Monte Carlo Methods and Importance Sampling", https://ib.berkeley.edu/labs/slatkin/eriq/classes/guest_lect/mc_lecture_notes.pdf [2] I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning, MIT Press, 2016 [3] K. P. Murphy. Machine Learning: A Probabilistic Perspective, MIT Press, 2012 TIMESTAMP 0:00 Intro 0:16 Monte Carlo Methods 2:29 Monte Carlo Example 3:57 Distribution of Monte Carlo Estimate 6:06 Importance Sampling 9:00 Importance Sampling Example 11:40 When to use Importance Sampling </video> <video> Policy Gradient Methods | Reinforcement Learning Part 6 Description: The machine learning consultancy: https://truetheta.io Join my email list to get educational and useful articles (and nothing else!): https://mailchi.mp/truetheta/true-theta-email-list Want to work together? See here: https://truetheta.io/about/#want-to-work-together Policy Gradient Methods are among the most effective techniques in Reinforcement Learning. In this video, we'll motivate their design, observe their behavior and understand their background theory. SOCIAL MEDIA LinkedIn : https://www.linkedin.com/in/dj-rich-90b91753/ Twitter : https://twitter.com/DuaneJRich Github: https://github.com/Duane321 Enjoy learning this way? Want me to make more videos? Consider supporting me on Patreon: https://www.patreon.com/MutualInformation SOURCES FOR THE FULL SERIES [1] R. Sutton and A. Barto. Reinforcement learning: An Introduction (2nd Ed). MIT Press, 2018. [2] H. Hasselt, et al. RL Lecture Series, Deepmind and UCL, 2021, https://youtu.be/TCCjZe0y4Qc [3] J. Achiam. Spinning Up in Deep Reinforcement Learning, OpenAI, 2018 ADDITIONAL SOURCES FOR THIS VIDEO [4] J. Achiam, Spinning Up in Deep Reinforcement Learning: Intro to Policy Optimization, OpenAI, 2018, https://spinningup.openai.com/en/latest/spinningup/rl_intro3.html [5] D. Silver, Lecture 7: Policy Gradient Methods, Deepmind, 2015, https://youtu.be/KHZVXao4qXs TIMESTAMPS 0:00 Introduction 0:50 Basic Idea of Policy Gradient Methods 2:30 A Familiar Shape 4:23 Motivating the Update Rule 10:51 Fixing the Update Rule 12:55 Example: Windy Highway 16:47 A Problem with Naive PGMs 19:43 Reinforce with Baseline 21:42 The Policy Gradient Theorem 25:20 General Comments 28:02 Thanking The Sources LINKS Windy Highway: https://github.com/Duane321/mutual_information/tree/main/videos/policy_gradient_methods NOTES [1] When motivating the update rule with an animation protopoints and theta bars, I don't specify alpha. That's because the lengths of the gradient arrows can only be interpretted on a relative basis. Their absolute numeric values can't be deduced from the animation because there was some unmentioned scaling done to make the animation look natural. Mentioning alpha would have make this calculation possible to attempt, so I avoided it. </video> <video> Why Do Tree Based-Models Outperform Neural Nets on Tabular Data? Description: Paper: https://openreview.net/forum?id=Fp7__phQszn Twitter thread explaining key results: https://twitter.com/GaelVaroquaux/status/1549422403889106944?s=20&t=lcC0_2HbjlLKjLTl0LvqGw </video> <video> What happens at the Boundary of Computation? Description: The machine learning consultancy: https://truetheta.io Join my email list to get educational and useful articles (and nothing else!): https://mailchi.mp/truetheta/true-theta-email-list In this video, we look inside the bizarre busy beaver function. SOCIAL MEDIA LinkedIn : https://www.linkedin.com/in/dj-rich-90b91753/ Twitter : https://twitter.com/DuaneJRich Github: https://github.com/Duane321 Enjoy learning this way? Want me to make more videos? Consider supporting me on Patreon: https://www.patreon.com/MutualInformation REFERENCE NOTES As mentioned, [1] was the primary source. The first video and this one were originally inspired by reading [1], so many of the ideas can be traced to that point. Anyone still interested should continue their search there. Sources [3]-[4] were essential for understanding the exact nature of the busy beaver and champion machines. In fact, I should have also thanks Pascal Michel in the video. He has maintained a detailed recorded of which machines were champions at which points and I found myself referencing it regularly. Sources [5]-[7] plus some conversations with the author were essential for my understanding of the Collatz-like behavior. REFERENCES [1] S. Aaronson, "The Busy Beaver Frontier", https://www.scottaaronson.com/papers/bb.pdf [2] T. Radó. "On non-computable functions". Bell System Technical Journal. 41 (3): 877–884, 1962 [3] P. Michel. "The Busy Beaver competition: a historical survey". 2019. https://arxiv.org/abs/0906.3749 [4] P. Michel. "Problems in Number Theory from the Busy Beaver Competition". Logical Methods in Computer Science Vol. 11(4:10), pp. 1–35, 2015 [5] S. Ligocki, "BB(6, 2) is greater than 10^10...^10", 2022, https://www.sligocki.com/2022/06/21/bb-6-2-t15.html [6] S. Ligocki, "Collatz-like behavior of Busy Beavers", 2021, https://www.sligocki.com/2021/07/17/bb-collatz.html [7] S. Ligocki, "BBB Complex Collatz-like Behavior", 2021, https://www.sligocki.com/2022/02/22/collatz-complex.html TIME CODES 0:00 Introduction 0:42 Reviewing the Basics 2:11 How does it grow faster than anything computable? 2:58 Using Collatz for Absurd Growth 3:59 Collatz in the 5-state machine 7:31 Exponential Collatz in the 6-state machine 9:05 The Busy Beavers answer famous open problems 11:23 The Busy Beavers are unknowable by any mathematical system 14:10 The Conjectures 14:26 Thank You's </video> <video> The Boundary of Computation Description: The machine learning consultancy: https://truetheta.io Join my email list to get educational and useful articles (and nothing else!): https://mailchi.mp/truetheta/true-theta-email-list Want to work together? See here: https://truetheta.io/about/#want-to-work-together There is a limit to how much work algorithms can do. SOCIAL MEDIA LinkedIn : https://www.linkedin.com/in/dj-rich-90b91753/ Twitter : https://twitter.com/DuaneJRich Github: https://github.com/Duane321 Enjoy learning this way? Want me to make more videos? Consider supporting me on Patreon: https://www.patreon.com/MutualInformation THE BUSY BEAVER WORLD In my opinion, the best place to start is Aaronson's "The Busy Beaver Frontier" paper (ref [1]). It's accessible, well written and provides a comprehensive view of the BB-world. Also, Googology represents the community in their efforts/excitement well (e.g. see ref [8]). NOTES [1] When describing how Sigma(4) is computed, I say "whittle to a small group that halt and ran to produce the max count of 1's". It has been pointed out to me that this isn't an accurate description. It's more accurate to say we whittle down to a small group that *don't halt*. What's done is, many machines are run, some halt and some continue to run. The hard work is in proving that the machines still running will in fact run forever. Once that is proved, the busy beaver number is the max count of ones written by all machines that halted so far. REFERENCE NOTES As mentioned, I mainly read Scott Aaronson's work for this (ref [1], [3]) and his blog: https://scottaaronson.blog/, where he posts BB related updates. Second to this was the wikipedia page (ref [6]). Also, [2] and [7] were useful for understanding how people pursue bounds and values for Sigma(n). The original paper (ref [4]) was useful to understand why Sigma(n) was invented in the first place (to provide a well defined non-computable function) and to understand the original format of the BB competition. The bounds related to Graham's sequence were found in [6] and [8]. REFERENCES [1] S. Aaronson, "The Busy Beaver Frontier", https://www.scottaaronson.com/papers/bb.pdf [2] A. H. Brady. "The determination of the value of Rado’s noncomputable function Σ(k) for four-state Turing machines". Mathematics of Computation, 1983. [3] A. Yedidia and S. Aaronson. "A relatively small Turing machine whose behavior is independent of set theory." Complex Systems, (25):4, 2016, https://arxiv.org/abs/1605.04343 [4] T. Radó. "On non-computable functions". Bell System Technical Journal. 41 (3): 877–884, 1962 [5] Copeland, B. Jack, "The Church-Turing Thesis", The Stanford Encyclopedia of Philosophy (Summer 2020 Edition), Edward N. Zalta (ed.) https://plato.stanford.edu/archives/sum2020/entries/church-turing/ [6] Wikipedia contributors. "Busy beaver" Wikipedia, 2023, https://en.wikipedia.org/wiki/Busy_beaver [7] P. Michel. "The Busy Beaver competition: a historical survey". 2019. https://arxiv.org/abs/0906. 3749 [8] Wythagoras (Username), "The nineteenth Busy Beaver number is greater than Graham's Number!", googology.fandom.com, 2016, https://googology.fandom.com/wiki/User_blog:Wythagoras/The_nineteenth_Busy_Beaver_number_is_greater_than_Graham%27s_Number! [9] S. Ligocki, "BB(6, 2) is greater than 10^10...^10", 2022, https://www.sligocki.com/2022/06/21/bb-6-2-t15.html TIME CODES 0:00 Introduction 1:21 A Binary Turing Machine 2:42 Two Things to Know about Turing Machines 3:54 What is the Busy Beaver Function? 5:40 Why is it hard to calculate? 6:28 Computability 7:41 A Shot at the King 10:36 The Busy Beavers reference open problems 11:39 Its values cannot be proven in some systems 12:08 The Busy Beaver World </video> <video> Factor Analysis and Probabilistic PCA Description: The machine learning consultancy: https://truetheta.io Join my email list to get educational and useful articles (and nothing else!): https://mailchi.mp/truetheta/true-theta-email-list Want to work together? See here: https://truetheta.io/about/#want-to-work-together Factor Analysis and Probabilistic PCA are classic methods to capture how observations 'move together'. SOCIAL MEDIA LinkedIn : https://www.linkedin.com/in/dj-rich-90b91753/ Twitter : https://twitter.com/DuaneJRich Enjoy learning this way? Want me to make more videos? Consider supporting me on Patreon: https://www.patreon.com/MutualInformation SOURCES [1] was my primary source since it provides the algorithm used in the Scikit Learn's Factor Analysis software (which is what I use). Since it walks through the derivation of the fitting procedure, it is quite technical. Ultimately, that level of detail came in handy for this video. [2] and [4] were my go-to for Probabilistic PCA. A primary reason is Christopher Bishop is one of the originators of PPCA. That came with a lot of thoughtful motivation for the approach. The discussion there includes a lot advantages of PPCA over PCA. [3] was my refresher on this subject when I first decided to this video. Like many of us, I'm a fan of Andrew Ng, so I was curious how he'd explain the subject. He emphasized that this model is particularly useful in high dimension-low data environments - something I forward in this video. [5] is an excellent overview of FA and PPCA (as long as you're comfortable with linear algebra and probability). In fact, Kevin Murphy's entire book is like that for every subject and that's why it's my absolute favorite text. --------------------------- [1] D. Barber, Bayesian Reasoning and Machine Learning, Cambridge University Press, 2012 [2] C. Bishop, Pattern Recognition and Machine Learning, Springer, 2006 [3] A. Ng, "Lecture 15 - EM Algorithm & Factor Analysis | Stanford CS229: Machine Learning (Autumn 2018)", https://www.youtube.com/watch?v=tw6cmL5STuY&t=946s [4] M. Tipping and C. Bishop, "Mixtures of Probabilistic Principal Component Analysers", MIT Press, 1999 [5] K. P. Murphy. Probabilistic Machine Learning (Second Edition), MIT Press, 2021 CONTENTS 0:00 Intro 0:21 The Problem Factor Analysis Solves 2:27 Factor Analysis Visually 5:52 The Factor Analysis Model 10:56 Fitting a Factor Analysis Model 14:13 Probabilistic PCA 15:43 Why is it Probabilistic "PCA"? 16:59 The Optimal Noise Variance TOOLS If you'd like to apply Factor Analysis, I'd recommend Scikit-Learn: https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.FactorAnalysis.html Social Media Twitter : https://twitter.com/DuaneJRich Patreon : https://www.patreon.com/MutualInformation </video> <video> Reinforcement Learning, by the Book Description: The machine learning consultancy: https://truetheta.io Join my email list to get educational and useful articles (and nothing else!): https://mailchi.mp/truetheta/true-theta-email-list Want to work together? See here: https://truetheta.io/about/#want-to-work-together Part one of a six part series on Reinforcement Learning. If you want to understand the fundamentals in a short amount of time, you're in the right place. SOCIAL MEDIA LinkedIn : https://www.linkedin.com/in/dj-rich-90b91753/ Twitter : https://twitter.com/DuaneJRich Github: https://github.com/Duane321 Enjoy learning this way? Want me to make more videos? Consider supporting me on Patreon: https://www.patreon.com/MutualInformation SOURCES [1] R. Sutton and A. Barto. Reinforcement learning: An Introduction (2nd Ed). MIT Press, 2018. [2] H. Hasselt, et al. RL Lecture Series, Deepmind and UCL, 2021, https://www.youtube.com/playlist?list=PLqYmG7hTraZDVH599EItlEWsUOsJbAodm [3] D. Silver, Lecture 1: Introduction to Reinforcement Learning, Deepmind, 2015, https://www.youtube.com/watch?v=2pWv7GOvuf0&t=4139s [4] Y. Wang, Pricing at Lyft, Lyft, 2022, https://eng.lyft.com/pricing-at-lyft-8a4022065f8b [5] A. Irpan, Deep Reinforcement Learning Doesn't Work Yet, 2018, https://www.alexirpan.com/2018/02/14/rl-hard.html [6] D. Silver, et al., Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm, Deepmind, 2017. [7] J. Schrittwieser, et al. Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model, Deepmind, 2020. [8] R. Roy, et al. PrefixRL: Optimization of Parallel Prefix Circuits using Deep Reinforcement Learning, Nvidia, 2022 [9] J. Degrave, et al. Magnetic control of tokamak plasmas through deep reinforcement learning, Nature, 2021. SOURCE NOTES [1] is my primary source for this series. It largely determined the notation and the set of topics and motivated much of the commentary. This series could not be what it is without the comprehensive and consistent presentation of this vast subject provided by this text. If you're interested in learning more, I highly recommend this text. In preparation, I also took Deepmind's course [2], with the intend to understand a different perspective. The material is similar, though not entirely. Notably, Deepmind's problem statement involves agents receiving observations, which are summarized into an agent-specific state. This is a more application-ready problem statement. Early drafts of the series included this version of the problem, but it was cut, since it complicated following topics. Overall, I learned about RL substantially from this course. The Maze demonstration of the value function comes from David Silver's 2015 lecture [3]. [4] - [9] are sources for all papers and blogs referenced at the start of the video. It includes a blog [4] where Lyft describes how RL is used in their pricing algorithms. NOTES With regards to the statement "RL won't be the same revolution that Neural Networks were. That's OK - NNs are quite a high bar". I should elaborate. I anticipate a response of, "You can't separate RL and NN, since much of the most impactful applications of RL involve NNS". Yes, comparing these technologies is fraught. In effect, I'm presuming that if a widespread subscription to RL's problem statement is enabled by more performant NNs, that is part of the RL migration. That's not fair if we're to attribute successes to either RL or NN. In my view, this attribution isn't important. I'm merely making the claim that RL will become a primary component in many production systems. The comparison of RL vs NNs is an accidental symptom of how I pitched the RL trend. In retrospect, I would have phrased this differently. TIMESTAMP 0:00 The Trend of Reinforcement Learning 2:46 A Six Part Series 3:24 A Finite Markov Decision Process and Our Goal 9:02 An Example MDP 12:49 State and Action Value Functions 15:00 An Example of a State Value Function 16:28 The Assumptions 17:58 Watch the Next Video! CORRECTIONS 1) In the MASE, I have a -16 where there should be a -14 (Thank you rogiervdw). </video> <video> Why Do Neural Networks Love the Softmax? Description: The machine learning consultancy: https://truetheta.io Join my email list to get educational and useful articles (and nothing else!): https://mailchi.mp/truetheta/true-theta-email-list Want to work together? See here: https://truetheta.io/about/#want-to-work-together Neural Networks see something special in the softmax function. SOCIAL MEDIA LinkedIn : https://www.linkedin.com/in/dj-rich-90b91753/ Twitter : https://twitter.com/DuaneJRich Github: https://github.com/Duane321 Enjoy learning this way? Want me to make more videos? Consider supporting me on Patreon: https://www.patreon.com/MutualInformation SOURCE NOTES I decided to make this video when inspecting jacobians/gradients starting from the end of a small network. Right near the softmax, the jacobian looked simple enough that I suspected interesting math behind it. And there was. I came across several excellent blogs on the Softmax's jacobian and its interaction with the negative log likelihood. Source [1] was the primary source, since it was quite well explained and used condensed notation. [2] was useful for understanding the broader context and [3] was a separate, thorough perspective. SOURCES [1] M. Peterson, "Softmax with cross-entropy," https://mattpetersen.github.io/softmax-with-cross-entropy, 2017 [2] I. Goodfellow, Y. Bengio, A. Courville, Deep Learning, MIT Press, 2016, section 6.2.2.3 [3] M. Lester James, "Understanding softmax and the negative log-likelihood," https://ljvmiranda921.github.io/notebook/2017/08/13/softmax-and-the-negative-log-likelihood/, 2017 TIME CODES 0:00 Everyone uses the softmax 0:23 A Standard Explanation 3:20 But Why the Exponential Function? 3:57 The Broader Context 6:05 Two Choices Together 6:51 The Gradient 10:07 Other Reasons </video> <video> The Most Important (and Surprising) Result from Information Theory Description: The machine learning consultancy: https://truetheta.io Join my email list to get educational and useful articles (and nothing else!): https://mailchi.mp/truetheta/true-theta-email-list Want to work together? See here: https://truetheta.io/about/#want-to-work-together Information Theory contains one idea in particular that has had an incredibly impact on our society. David MacKay's lecture: https://www.youtube.com/playlist?list=PLruBu5BI5n4aFpG32iMbdWoRVAA-Vcso6 SOCIAL MEDIA LinkedIn : https://www.linkedin.com/in/dj-rich-90b91753/ Twitter : https://twitter.com/DuaneJRich Github: https://github.com/Duane321 Enjoy learning this way? Want me to make more videos? Consider supporting me on Patreon: https://www.patreon.com/MutualInformation SOURCES Chapters 1, 9 and 10 from [1] were the primary source. Chapter 7 of [3] was a secondary source. [2] is inclusive of Shannon's 1948 original paper and provided useful historical context. [1] D. MacKay. Information Theory, Inference and Learning Algorithms. Cambridge University Press. 2003. [2] C. E. Shannon and W. Weaver. The Mathematical Theory of Communication. University of Illinois Press. 1964 [3] T. M. Cover and J. A. Thomas. The Elements of Information Theory, 2nd Edition. Wiley-Interscience. 2006. TIMESTAMPS 0:00 Problem Statement and the R3 Coding Strategy 2:06 Bit Error Probability and Rate 3:30 The Trillion Dollar Question 4:12 Claude Shannon Proves Something Remarkable 6:11 Sidebar on other Educational Content 6:49 The Trick 7:38 Check out David Mackay's Textbook and Lectures, plus Thank You </video> <video> Function Approximation | Reinforcement Learning Part 5 Description: The machine learning consultancy: https://truetheta.io Join my email list to get educational and useful articles (and nothing else!): https://mailchi.mp/truetheta/true-theta-email-list Want to work together? See here: https://truetheta.io/about/#want-to-work-together Here, we learn about Function Approximation. This is a broad class of methods for learning within state spaces that are far too large for our previous methods to work. This is part five of a six part series on Reinforcement Learning. SOCIAL MEDIA LinkedIn : https://www.linkedin.com/in/dj-rich-90b91753/ Twitter : https://twitter.com/DuaneJRich Github: https://github.com/Duane321 Enjoy learning this way? Want me to make more videos? Consider supporting me on Patreon: https://www.patreon.com/MutualInformation SOURCES [1] R. Sutton and A. Barto. Reinforcement learning: An Introduction (2nd Ed). MIT Press, 2018. [2] H. Hasselt, et al. RL Lecture Series, Deepmind and UCL, 2021, https://www.youtube.com/playlist?list=PLqYmG7hTraZDVH599EItlEWsUOsJbAodm SOURCE NOTES This video covers topics from chapters 9, 10 and 11 from [1], with only a light covering of chapter 11. [2] includes a lecture on Function Approximation, which was a helpful secondary source. TIMESTAMP 0:00 Intro 0:25 Large State Spaces and Generalization 1:55 On Policy Evaluation 4:31 How do we select w? 6:46 How do we choose our target U? 9:27 A Linear Value Function 10:34 1000-State Random Walk 12:51 On Policy Control with FA 14:26 The Mountain Car Task 19:30 Off-Policy Methods with FA LINKS 1000-State Random Walk Problem: https://github.com/Duane321/mutual_information/blob/main/videos/function_approximation/large_random_walk.ipynb Mountain Car Task: https://github.com/Duane321/mutual_information/blob/main/videos/function_approximation/mountain_car.ipynb NOTES [1] In the Mountain Car Task, I left out a hyperparameter to tune: Lambda. This controls how far away the evenly spaced proto-points are from any given evaluation point. If lambda is very high, the prototypical points are considered very close together, and they won't do a good job discriminating different values over the state space. But if lambda is too low, then the prototypical points won't share any information beyond a tiny region surrounding each point. </video> <video> Bellman Equations, Dynamic Programming, Generalized Policy Iteration | Reinforcement Learning Part 2 Description: The machine learning consultancy: https://truetheta.io Join my email list to get educational and useful articles (and nothing else!): https://mailchi.mp/truetheta/true-theta-email-list Want to work together? See here: https://truetheta.io/about/#want-to-work-together Part two of a six part series on Reinforcement Learning. We discuss the Bellman Equations, Dynamic Programming and Generalized Policy Iteration. SOCIAL MEDIA LinkedIn : https://www.linkedin.com/in/dj-rich-90b91753/ Twitter : https://twitter.com/DuaneJRich Github: https://github.com/Duane321 Enjoy learning this way? Want me to make more videos? Consider supporting me on Patreon: https://www.patreon.com/MutualInformation SOURCES [1] R. Sutton and A. Barto. Reinforcement learning: An Introduction (2nd Ed). MIT Press, 2018. [2] H. Hasselt, et al. RL Lecture Series, Deepmind and UCL, 2021, https://www.youtube.com/playlist?list=PLqYmG7hTraZDVH599EItlEWsUOsJbAodm SOURCE NOTES The video covers the topics of Chapter 3 and 4 from [1]. The whole series teaches from [1]. [2] was a useful secondary resource. TIMESTAMP 0:00 What We'll Learn 1:09 Review of Previous Topics 2:46 Definition of Dynamic Programming 3:05 Discovering the Bellman Equation 7:13 Bellman Optimality 8:41 A Grid View of the Bellman Equations 11:24 Policy Evaluation 13:58 Policy Improvement 15:55 Generalized Policy Iteration 17:55 A Beautiful View of GPI 18:14 The Gambler's Problem 20:42 Watch the Next Video! </video>

Contacts and Links

Description

Highly informative videos on machine learning and related topics

Videos from channel