E05: Sessions - I built an Open-Source Secrets Tool Live (and OBS Died Halfway)

Own the knowledge, rent the thinking: building organizational memory for AI agentsYou can give an AI agent access to your company's wiki, Slack, CRM and project tracker, and it still won't know who owns a decision, which information is current, or what it's allowed to share. That isn't a model problem. It's a memory problem.In this episode, recorded live at AI DevCon in Brooklyn, Ran Aroussi shares what VarOps learned from a year of building organizational memory for AI agents. The findings were tested against three public benchmarks, and the failures and negative results are published too.Three architectural decisions that changed everythingStore claims, not chunks. Distill information into structured knowledge, with evidence, timestamps and provenance attached.Make time and absence explicit. Track what changed, keep the history, and teach agents when to say "I don't know."Enforce permissions before inference. Models should never receive information that the person asking isn't authorized to see.The results99.8% evidence retrieval on LongMemEval35 million tokens ingested for $8.24Fully local models reaching 84–96% of all-frontier performanceLocal memory paired with rented frontier reasoning reaching 96–99%45 of 45 adversarial isolation probes refused or containedThe bigger lessonAI agents don't need bigger context windows or endless conversation histories. They need a governed understanding of the organization they work for. The model is a seat we rent. The memory is the asset we own.LinksRead the full research paper, Own the Knowledge, Rent the ThinkingLearn more about VarOps and Resident AI00:00 Why Agents Get It Wrong01:03 The WhatsApp Contract Trap03:33 Three Trust Questions04:35 Stop Dumping Data05:53 Living Org Model08:27 Why RAG Breaks09:51 Claims With Provenance10:47 Distillation Works13:06 Benchmarks And Costs15:15 Permissions And History16:50 Saying I Don't Know19:51 Security Before Inference20:41 Role Based Answers Demo22:44 Attack Testing Results23:28 Local And Hybrid Models25:15 Key Takeaways And Wrap

The show returns after a month-long break to announce the launch of VarOps (varops.com) and introduces its premise via a short documentary about enterprises losing critical, undocumented decision knowledge when key employees are unavailable. Ran argues organizations adopt AI incorrectly by dumping documents into RAG systems, relying on prompts for security, and trying to mold long-established processes around new tools instead of building AI around the business. He describes VarOps’ approach – “Resident AI” owned by the company – supported by a published paper on a governed operational model and a “large organization model” with version-controlled decisions and deterministic access control. Benchmarks show local models plus frontier reasoning can reach ~96–99% of frontier performance at low cost, motivating model and data sovereignty. Ran also discusses new frontier releases and Jev, a fast classification-focused model, emphasizing models are not interchangeable and small accuracy differences matter in mission-critical enterprise settings.00:00 Welcome Back VarOps Launch01:17 Resident AI Mini Documentary04:22 Why AI Adoption Fails09:02 Acropolis Research Findings11:37 New Model Wave Pricing15:06 System Architecture Varys18:40 Local Models Sovereignty21:19 VarOps Site Bot Magazine23:27 Jev System One Classifier29:08 Model Fit Error Rates30:48 Wrap Up Newsletter Invite

Host Ran Aroussi (Old School / New Tech) shares a recent “war story” from his agency Automaze: a hard-to-reproduce mobile bug where calls were sometimes dropped only on a client’s devices, which the team couldn’t reproduce for 2–3 weeks. After two long sessions using a Droid Factory setup with agents and a “small council” debate between models (e.g., Fable and Sol), he got the team unstuck, found the issue, and then turned the session logs into an internal guide and public article on his methodology. Key practices include distrusting agent conclusions (treat “preexisting/flaky/unrelated” as hypotheses), demanding proof (including his Proof library requiring video evidence), using extensive unit and end-to-end tests, asking for a numeric confidence level before production, controlling environments, handling merge conflicts with full review/testing, switching models for independent review, and re-spec’ing from scratch to detect drift from the implementation plan.00:00 Show Begins00:58 The Sneaky Mobile Bug02:46 Agents Find The Fix05:23 Never Trust Preexisting08:29 Proof Over Promises10:14 Confidence Level Gate12:47 Model Debate Reviews17:08 Merge Conflicts Standards21:23 Spec From Scratch25:49 Ten Rules Summary31:30 Token Usage And Wrap Up-----Old School / New Tech is produced by VarOps.Subscribe to the newsletter – the written version of this show, weekly: varops.comRead the books – Company-Scale Agentic AI (for operators and the C-suite) and Production-Grade Agentic AI (for engineers): varops.com/booksWork with us – VarOps installs resident AI inside mid-market companies. It runs where your work already happens, sees the whole operation, and what it learns belongs to you: varops.com/transform

Why Software Factories Still Need Humans in the LoopRan Aroussi discusses “software factories” as automated systems that manage the full software development and deployment cycle, arguing the industry is moving from a single workstation to virtual, always-on cloud environments with agents running off-machine. Prompted by Dex (HumanLayer) describing a fully automated “lights-out” factory that produced “slop,” and Uncle Bob’s claim he doesn’t read agent-written code when surrounded by extreme constraints, Ran explains why he doesn’t believe in lights-out automation. He outlines Automaze’s internal factory, Cloop: tickets/PRDs are created and grounded in an indexed knowledge graph of the codebase; multiple models generate and critique PRDs; a human approves; agents implement in parallel, run unit/e2e/quality tests, simplify and profile code, then perform code and security review, looping up to five times before escalating to a human. Humans remain essential for alignment, enforcement, and final validation via PR review and temporary deployments, avoiding false positives, runaway costs, and untrustworthy results.-----Old School / New Tech is produced by VarOps.Subscribe to the newsletter – the written version of this show, weekly: varops.comRead the books – Company-Scale Agentic AI (for operators and the C-suite) and Production-Grade Agentic AI (for engineers): varops.com/booksWork with us – VarOps installs resident AI inside mid-market companies. It runs where your work already happens, sees the whole operation, and what it learns belongs to you: varops.com/transform

Company-Scale Agentic AI: Why AI Isn’t Stupid, It’s Blind (and How to Fix It)In this episode of Old School / New Tech, Ran Aroussi explains what “company-scale agentic AI” really requires and why most AI rollouts fail: the model isn’t stupid, it’s blind to how the business actually operates. He shares why he wrote a new executive-focused book, Company-Scale Agentic AI (a free follow-up to Production Grade Agentic AI), and outlines a five-part loop—observe, understand, build, run, compound—built around an always-on “company brain” that absorbs emails, chats, calls, and files to map processes, relationships, and bottlenecks. Ran emphasizes the difference between deterministic automations and true agents, argues for starting with human-gated execution to build trust, and highlights role-based access control via middleware as essential for organization-wide deployments. He also describes a browser “morning brief” workflow that keeps tasks from falling through the cracks and urges teams to adapt AI to existing tools instead of forcing employees to change how they work.00:00 Welcome and Topic00:29 Why I Wrote It02:20 AI Is Blind06:45 Building Company Brain07:28 The Five Step Loop11:15 Automation vs Agents13:53 Gated Readiness Dial17:01 Role Based Access20:58 Morning Brief Extension22:34 Meet People Where They Work24:57 Loop Recap and Wrap-----Old School / New Tech is produced by VarOps.Subscribe to the newsletter – the written version of this show, weekly: varops.comRead the books – Company-Scale Agentic AI (for operators and the C-suite) and Production-Grade Agentic AI (for engineers): varops.com/booksWork with us – VarOps installs resident AI inside mid-market companies. It runs where your work already happens, sees the whole operation, and what it learns belongs to you: varops.com/transform

How I Work: My Solo Dev Setup, Time Management, and Running two CompaniesA solo episode answering a question I keep getting asked: how do I get through it all?I walk through my full setup - the dedicated remote Mac that acts as my "local" environment, Droid as my coding harness, Paseo for exploratory work, Cloop for mature codebases, MUXI as my assistant running through Claude Desktop over MCP - plus why I self-host almost everything, and it has nothing to do with saving money.Then the harder part: how I manage time. My job stopped being doing the work and became deciding which bucket the work goes in. I make 20-30 decisions a day and almost all of them are classifications - rails or agents, automated or human, or time to kill the process entirely. Anything I've done more than a few times gets automated into one of three buckets. Most of my time goes into thinking rather than typing, stripping products back to first principles and deciding what not to build.I also cover why most of my automations are deliberately gated behind human approval, why I read every line the agents produce, how Automaze actually runs without me, where leads really come from after 15 years of open source, and why I embed myself as the FDE with every new client before handing off.All of it exists to protect the context in my head. Everything else is scaffolding.00:00 Welcome and Envapor00:50 Why My Workflow02:16 Cloud Local Setup04:03 Coding Tools Stack07:26 Personal Productivity Apps08:51 Self-Hosting Philosophy09:46 Multi-Agent Workflow10:35 Time Management Decisions11:54 Automation Buckets14:40 Research and Judgment17:25 Email and Gated AI19:45 Running Automaze VarOps20:58 Inbound Leads Flywheel22:08 FDE Founder Onboarding25:16 Wrap Up and Newsletter-----Old School / New Tech is produced by VarOps.Subscribe to the newsletter – the written version of this show, weekly: varops.comRead the books – Company-Scale Agentic AI (for operators and the C-suite) and Production-Grade Agentic AI (for engineers): varops.com/booksWork with us – VarOps installs resident AI inside mid-market companies. It runs where your work already happens, sees the whole operation, and what it learns belongs to you: varops.com/transform

Why can't we just commit our .env files?" A clip of that question showed up on my feed, and instead of tweeting about it, I sat down and built the answer - live, in one sitting, from an empty repo to a working open-source tool.This is that build. Envapor is a Git-native utility that encrypts the values in your .env files: you edit .env exactly like you do now, Git stores it encrypted on commit and hands back plaintext on checkout. No .env.enc, no wrapper commands, nothing to change about how your app loads config.Along the way: why deterministic encryption is the whole ballgame for keeping diffs readable, the difference between a tool that looks done and one you'd actually trust with production secrets, an unplanned OBS crash 20 minutes in that moved the whole thing to YouTube, and the judgment calls an AI agent gets subtly wrong that would've shipped a broken security tool.Two and a half hours, no script, no highlight reel. The finished tool is open source and linked below.🔗 Repo: https://github.com/automazeio/envapor🔗 Full unedited build (YouTube): https://www.youtube.com/watch?v=p4BYP9DFp9c&list=PLWwTwUPfg-UU-----Old School / New Tech is produced by VarOps.Subscribe to the newsletter – the written version of this show, weekly: varops.comRead the books – Company-Scale Agentic AI (for operators and the C-suite) and Production-Grade Agentic AI (for engineers): varops.com/booksWork with us – VarOps installs resident AI inside mid-market companies. It runs where your work already happens, sees the whole operation, and what it learns belongs to you: varops.com/transform

Why AI Adoption Fails at the Human Layer (and How to Fix It)Muximus and Ran discuss why AI adoption in organizations often fails due to human and organizational factors rather than the technology itself, with people—especially senior leaders—freezing from fear of seeming behind, FOMO, and pressure to master fast-changing tools. They argue companies are still in a “make me an AI” phase, shopping for tools instead of aligning AI with real workflows. Key recommendations include picking any one tool and committing for months to break paralysis, then implementing structured training based on how teams already work; fitting AI into existing processes rather than remolding the organization; measuring outcomes like speed and stress reduction instead of token usage; creating internal champions and casual knowledge-sharing sessions that also help leadership learn; expecting a short-term productivity dip; leveraging AI already embedded in existing SaaS tools before building custom; and avoiding constant switching to new models unless driven by capability, cost, or deprecation.00:00 AI Adoption Paradox00:40 Fear of Looking Behind01:37 Corporate Make Me AI03:01 Pick One Tool First05:04 Workflow First Training08:10 Measure Real Outcomes11:04 Champions And Meetups15:07 Expect The Productivity Dip17:22 Do The Groundwork17:45 Use Built In AI20:34 Stop Chasing New Models22:14 When To Switch Models25:22 Recap And Farewell-----Old School / New Tech is produced by VarOps.Subscribe to the newsletter – the written version of this show, weekly: varops.comRead the books – Company-Scale Agentic AI (for operators and the C-suite) and Production-Grade Agentic AI (for engineers): varops.com/booksWork with us – VarOps installs resident AI inside mid-market companies. It runs where your work already happens, sees the whole operation, and what it learns belongs to you: varops.com/transform

Why the Terminal Never Dies: CLI Power, AI Agents, and Practical GuardrailsIn episode three of Old School New Tech, the hosts, Ran Aroussi and Muximus, argue that the terminal is the “cockroach of interfaces” because it persists structurally, not nostalgically: it is the lowest-level, most direct, composable interface to the machine. They discuss how power users kept the CLI alive for speed, logs, and file control, and note AI tools followed a similar path from chat demos to APIs and CLIs before polished desktop GUIs. Pipes are explained as chaining command outputs into inputs to build modular workflows, with an example from algorithmic trading where shell pipelines beat heavier tooling for manipulating large CSV market datasets. They propose non-developers and C-suites should learn basic CLI steps (ls, cd, cat/less, grep, simple pipes) and use an AI assistant in-terminal as a tutor, while stressing risks like lack of guardrails and never running unknown commands (e.g., rm -rf).00:00 Episode Kickoff00:41 Terminal Never Dies01:16 CLI Origins and Comeback04:49 Why CLI Wins05:15 Pipes Explained06:05 Real World Speed Story08:11 AI Tools Under the Hood09:59 CLI for Everyone13:20 Beginner CLI Roadmap15:36 Power Without Guardrails17:32 CLI vs GUI Wrap20:56 Final Thoughts and Outro-----Old School / New Tech is produced by VarOps.Subscribe to the newsletter – the written version of this show, weekly: varops.comRead the books – Company-Scale Agentic AI (for operators and the C-suite) and Production-Grade Agentic AI (for engineers): varops.com/booksWork with us – VarOps installs resident AI inside mid-market companies. It runs where your work already happens, sees the whole operation, and what it learns belongs to you: varops.com/transform

From Org Charts to Pods: Builders, Sellers, Operators, and AI AgentsIn episode two of Old School New Tech, Ran Aroussi and co-host Muximus debate shifting from traditional org charts to a system-driven “pod” model where early-stage companies primarily need builders and sellers, with classic middle management deferred. Ran argues that under ~20 people startups should avoid coordination-heavy roles, adding that middle management becomes useful around 20–30 headcount, with a key early exception being an operator/chief-of-staff-style role that bridges build and sell. They discuss AI agents handling coordination and grunt work, while junior developers function as apprentices learning orchestration, specs, and production debugging rather than syntax, with “learned” experience shrinking faster than “gained” experience. On the sell side, a hybrid pipeline role manages AI-driven prospecting and follow-up while handling calls. Administrative functions should be outsourced early, later becoming shared resources at the firm level across multiple pods.00:00 Welcome Back00:29 Builders And Sellers02:45 When Management Returns03:11 Chief Of Staff Operator04:27 Junior Dev Apprentices08:08 Learned Vs Gained Experience10:26 Sales Pod Mirror13:32 Outsource And Shared Resources17:10 Is Middle Layer Relocated20:32 Wrap Up And Takeaways-----Old School / New Tech is produced by VarOps.Subscribe to the newsletter – the written version of this show, weekly: varops.comRead the books – Company-Scale Agentic AI (for operators and the C-suite) and Production-Grade Agentic AI (for engineers): varops.com/booksWork with us – VarOps installs resident AI inside mid-market companies. It runs where your work already happens, sees the whole operation, and what it learns belongs to you: varops.com/transform