172: Transformers and Large Language Models

172: Transformers and Large Language ModelsIntro topic: Is WFH actually WFC?News/Links:Falsehoods Junior Developers Believe about Becoming Seniorhttps://vadimkravcenko.com/shorts/falsehoods-junior-developers-believe-about-becoming-senior/Pure PursuitTutorial with python code: https://wiki.purduesigbots.com/software/control-algorithms/basic-pure-pursuit Video example: https://www.youtube.com/watch?v=qYR7mmcwT2w PID without a PHDhttps://www.wescottdesign.com/articles/pid/pidWithoutAPhd.pdfGoogle releases Gemmahttps://blog.google/technology/developers/gemma-open-models/Book of the ShowPatrick: The Eye of the World by Robert Jordan (Wheel of Time)https://amzn.to/3uEhg6vJason: How to Make a Video Game All By Yourselfhttps://amzn.to/3UZtP7bPatreon Plug https://www.patreon.com/programmingthrowdown?ty=hTool of the ShowPatrick: Stadia Controller Wifi to Bluetooth Unlockhttps://stadia.google.com/controller/index_en_US.htmlJason: FUSE and SSHFShttps://www.digitalocean.com/community/tutorials/how-to-use-sshfs-to-mount-remote-file-systems-over-sshTopic: Transformers and Large Language ModelsHow neural networks store informationLatent variablesTransformersEncoders & DecodersAttention LayersHistoryRNNVanishing Gradient ProblemLSTMShort term (gradient explodes), Long term (gradient vanishes)Differentiable algebraKey-Query-ValueSelf AttentionSelf-Supervised Learning & Forward ModelsHuman FeedbackReinforcement Learning from Human FeedbackDirect Policy Optimization (Pairwise Ranking) ★ Support this podcast on Patreon ★

Intro topic: Monitor setupsNews/Links:BlueScuti, Willis, beats Tetrishttps://www.youtube.com/watch?v=GuJ5UuknsHUPalWorld accused of being an AI Producthttps://www.forbes.com/sites/paultassi/2024/01/22/palworld-accused-of-using-genai-with-no-evidence-so-far/?sh=26a9651b42394 Billion if-statements to determine if a number is even or oddhttps://andreasjhkarlsson.github.io/jekyll/update/2023/12/27/4-billion-if-statements.htmlSeamless M4Thttps://ai.meta.com/blog/seamless-m4t/Book of the ShowPatrick:Foundation by Isaac Asimovhttps://amzn.to/3SrmgnPJason: Propaganda by Edward Bernayshttps://amzn.to/47JUCXJPatreon Plug https://www.patreon.com/programmingthrowdown?ty=hTool of the ShowPatrick: The Room Gamehttps://www.fireproofgames.com/games/the-roomJason:Incredibuildhttps://www.incredibuild.com/Topic: Compilers and Interpreters (Request by Jessica W.)Machine CodeArchitecture SpecificAssemblySingle vs Two Pass CompilerHigh level LanguagesIntermediate RepresentationJVM ByteCode vs Machine Code for portabilityScripting/InterpretersJITProfile Guided OptimizationResourceshttps://www.craftinginterpreters.com/https://nandgame.com/Turing Complete ★ Support this podcast on Patreon ★

Predictions:Jason VR for WorkLowering AI training cost/ improved efficiencyRISC-V takeoffPatrickAi claim of AGIAi peer reviewerAi Video GeneratorMore space vehicles reaching orbitEarly career, finding role at FAANG, liaising vs shipping code. Startup?3 part. 1. How and when current hype for AI will end? 2. Shape of the show 3. Upcoming in techWhat are essential programmer knowledge items?CS Student, how to organize life and goals? What purpose life should serve?What kind of programmer were you in college?Happy Holidays! ★ Support this podcast on Patreon ★

Intro topic: Testing your car batteryNews/Links:Tech Layoffs still going onhttps://www.sfchronicle.com/tech/article/google-layoffs-california-companies-18465600.php Real-time dreamy Cloudscapes with Volumetric Raymarchinghttps://blog.maximeheckel.com/posts/real-time-cloudscapes-with-volumetric-raymarching/Robot Rascalshttps://en.wikipedia.org/wiki/Robot_Rascals Meta Quest 3 https://www.theverge.com/23906313/meta-quest-3-review-vr-mixed-reality-headsetBook of the ShowPatrick:HyperLogLog Paperhttps://static.googleusercontent.com/media/research.google.com/en//pubs/archive/40671.pdf Jason: Eureka! NVIDIA Research Breakthrough Puts New Spin on Robot Learning https://blogs.nvidia.com/blog/2023/10/20/eureka-robotics-research/ Patreon Plug https://www.patreon.com/programmingthrowdown?ty=hTool of the ShowPatrick: Techtonica: https://store.steampowered.com/app/1457320/Techtonica/ Jason:ESP32 development board: https://amzn.to/3Qpmb20 WEMOSTopic: HyperLogLogMotivationCardinality CountingLinearCountingHash + expectation of collision based on how fullBloom FilterLogLogUse first N bits as bucketUse max sequential 0s in each bucketAverageHyperLogLogHandle empty bucketsUse correction factor like linear counting for low counts (number of empty buckets) and high countsDistributingTransfer bucket counts ★ Support this podcast on Patreon ★

Intro topic: What are expectations on developers that have nothing to do with programmingNews/Links:Pushing for a lower dev estimate is like negotiating weather with a Meteorologisthttps://smartguess.is/blog/your-estimate-is-less-than-that/Announcing python in excelhttps://techcommunity.microsoft.com/t5/excel-blog/announcing-python-in-excel-combining-the-power-of-python-and-the/ba-p/38934397 Habits of Highly Effective Software Engineershttps://makingsmallercircles.com/articles/7-habits-of-highly-effective-software-engineers/Raspberry pi 5 begins shippinghttps://www.phoronix.com/news/Raspberry-Pi-5-ShippingBook of the ShowPatrick:Harry Potter and the Sorcerer's Stone Illustrated Editionhttps://amzn.to/3St3L35Jason: The Pete and Sebastian Showhttps://podcasts.apple.com/us/podcast/the-pete-and-sebastian-show/id570256898Patreon Plug https://www.patreon.com/programmingthrowdown?ty=hTool of the ShowPatrick: Obsidianobsidian.mdJason:Ink by Inklehttps://github.com/inkle/inkTopic: GodotWhat is a game engine?Graphics, animation, particle effectsTilemapsEntity component systemsPhysicsSoundI/OInput handling (touchscreen, joystick)GDScriptWhy use a game engine?PortabilityAllow for testing individual components of the gameLibraries for ads, in-app-purchasesGodotOpen sourceFocused on 2-D but now many 3-D featuresGreat development environmentAI HeroStarted in phaser (development rut)Moved to Godot, rapid prototyping ★ Support this podcast on Patreon ★

Intro topic: Jogging MetricsNews/Links:Unholy LLMhttps://huggingface.co/Undi95/Unholy-v1-12L-13B The reverse red herring  https://www.blameless.com/blog/the-reverse-red-herringThe "ens–tification" of TikTokhttps://www.wired.com/story/tiktok-platforms-cory-doctorow/ Response letter to Godot is not the new Unityhttps://sampruden.github.io/posts/godot-is-not-the-new-unity/https://gist.github.com/reduz/cb05fe96079e46785f08a79ec3b0ef21Book of the ShowPatrick:Math Games with Bad Drawings by Ben Orlinhttps://amzn.to/48qlg9AJason: The Invisible Hook: The Hidden Economics of Pirateshttps://amzn.to/3LChBffPatreon Plug https://www.patreon.com/programmingthrowdown?ty=hTool of the ShowPatrick: Factorio (Desktop Game)Jason:AI Hero (iOS and Android)Topic: Desktop user interfacesWhat is a user interface?Web and Mobile UI toolkitsDesktop optionsQtCross platform, custom UI elementsQt creator, code generatorWxWidgetsCross platform, uses native UI elements WxFormDesigner, code generationCan look different on different operating systemsElectronLocal nodejs webserverHtml/JavaScript technologyRequires interprocess communication to use other languages  Jupyter notebooksMathematica-like notebook Not for distribution StreamlitPython to web compilerGame EnginesUnity, Godot, UnrealTips for building desktop UIUI is slow (startup time, interaction time)Separate the UI from the engine & business logic ★ Support this podcast on Patreon ★

- pganalyze: https://pganalyze.com/- Weekly series "5mins of Postgres": https://www.youtube.com/channel/UCDV_1Dz2Ixgl1nT_3DUZVFw- How Postgres chooses which index to use: https://pganalyze.com/blog/how-postgres-chooses-index- CMU databases courses: https://db.cs.cmu.edu/courses/- Postgres community: https://www.postgresql.org/community/As well as social links:- Mastodon: https://hachyderm.io/@lukas- Twitter/X: @pganalyze, @LukasFittl- GitHub: @pganalyze, @lfittl- LinkedIn: https://www.linkedin.com/in/lfittl/ , https://www.linkedin.com/company/pganalyze/ ★ Support this podcast on Patreon ★

Intro topic: Revisiting the power of SpreadsheetsNews/Links:LK-99 Isn’t a Superconductorhttps://www.nature.com/articles/d41586-023-02585-7Normalizing Flowshttps://pyro.ai/examples/normalizing_flows_i.htmlHow is llama.cpp possible?https://finbarr.ca/how-is-llama-cpp-possible/Chat with open source large language modelshttps://chat.lmsys.org/Book of the ShowPatrick: Math with Bad Drawings by Ben Orlinhttps://amzn.to/44dsgDzJason: Overboard! https://play.google.com/store/apps/details?id=com.InkleLtd.Overboard Nhl=en_US&gl=US Patreon Plug https://www.patreon.com/programmingthrowdown?ty=hTool of the ShowPatrick: ffmprovisr https://amiaopensource.github.io/ffmprovisr/Jason:Pandas read_ods() read_excel()Topic: Differential EquationsWhy should programmers learn about DiffEqLaw of Large NumbersWhat are differential equations?When you know the rate of changeEigenVectors & EigenValuesWhat is Jacobian What is Jacobian? | The right way of thinking derivatives and integralsSpecial casesPartial Differential EquationsOrdinary Differential EquationsWhy solvers are importantNumerical Stability at larger step sizesExample: https://medium.com/@pukumarathe/eulers-method-and-runge-kutta-4th-order-method-in-python-b4a0068a8ebe Fun ExamplesPredator-Prey relationships in scipyhttps://scientific-python.readthedocs.io/en/latest/notebooks_rst/3_Ordinary_Differential_Equations/02_Examples/Lotka_Volterra_model.html Physics Engines for gameshttps://youtu.be/52n2qKgwW_Q PageRankhttps://arxiv.org/pdf/2001.08973.pdf  ★ Support this podcast on Patreon ★

Things to consider when choosing a databaseSpeed & LatencyConsistency, ACID ComplianceScalabilityLanguage support & Developer ExperienceRelational vs. Non-relational (SQL vs. NoSQL)Data typesSecurityDatabase environmentClient vs Server accessInfo on Kris & Harper:Website: harperdb.ioTwitter: @harperdbio, @kriszypGithub: @HarperDB, @kriszyp ★ Support this podcast on Patreon ★

Episode 163 - RecursionIntro topic: Electric CarsNews/Links:Snake Game in 101 Bytes in a QR Codehttps://www.reddit.com/r/programming/comments/15ab4ct/my_qr_code_snake_game_is_now_only_101_bytes/Superconductor Rumors aboundhttps://arstechnica.com/science/2023/08/whats-going-on-with-the-reports-of-a-room-temperature-superconductor/OpenWormhttps://github.com/openworm/OpenWormCreator of vim passes awayhttps://news.itsfoss.com/vim-creator-passed-away/Book of the ShowPatrick:Little Book of Common Sense Investing by Jack Bogle https://amzn.to/43YqANRJason: Mistborn Saga: https://amzn.to/3DJkUN8Patreon Plug https://www.patreon.com/programmingthrowdown?ty=hTool of the ShowJason:reMarkable https://remarkable.com/Patrick: Stellarium (iOS and Android)Topic: RecursionWhat is itDivide-And-ConquerFibonacci numbersHow to (not) teach recursionPractical ApplicationsGraph operationsTree retrieval, balancingGraph SearchSpatial partitioningPitfallsStack sizeHow to solve problems with recursion(1) Consider the base cases(2) Build the recursive step(3) Look for ways the recursion will not terminate and fix(4) (rest are optional) Remove global contexts(5) Add memoization(6) Build solutions incrementally ★ Support this podcast on Patreon ★