Episode 7 - OpenAI cybersecurity grant program

In this episode, Natalie and Ivan, in a conference call with members of GenAI, discuss OpenAI's cybersecurity grant program, and go over the suggestions made to applicants by the company on its webpage.

In this episode, we go through OWASP's recently released list of vulnerability classes for LLMs. The list includes: Prompt injections Data leakage Inadequate sandboxing Unauthorized code execution SSRF vulnerabilities Overreliance on LLM-generated content Inadequate AI alignment Insufficient access controls Improper error handling Training data poisoning We note that while some of the items on this list are definitely worth mentioning, others feel a bit more disconnected from the topic of cybersecurity and likely can't be solved through technical means. Reference: Rich Harang's twitter thread on the subject.

This episode focuses on whether social engineering techniques, traditionally used against humans, could be leveraged against language models. Natalie and Ivan start by discussing the use of LLMs in capture the flag competitions, and mention the emergence of new types of challenges revolving around prompt hacking. This leads the discussion towards the various ways that can be used to circumvent a model's fine tuning. The hosts discuss the challenges and scoring systems in such competitions, as well as the potential ethical concerns in exploring the capabilities and limitations of AI language models. They draw parallels between AI prompt crafting and social engineering, speculating on how techniques used in human social engineering could potentially be adapted for AI prompt hacking. Overall, the conversation explores the evolving relationship between security, AI, and the ethical considerations surrounding the use of language models in hacking competitions.

AI can often make decisions that are difficult for humans to understand. I n this episode, we'll discuss the importance of explainability and transparency in AI systems, how they can be achieved, what's the problem with biased training data and how does explainability help us overcome bias. What is explainability and why is it significant in the context of AI? How to increase reliance on AI-based systems for security applications using explainability? How can we have AI transparency? How to use it to overcome bias? Given bias won’t be gone, how to take into account its impact? Examples of problematic biases Do we have sufficient data to train AIs to recognize and classify APT (Advanced Persistent Threat) activities? What are the risks of overfitting and underfitting? Links mentioned in the show: Acedemic paper evaluating the security of CoPilot from 12/2021 Can OS LLM detect bugs in code?

In this episode we're deep diving into the fascinating world of Reverse Engineering and how AI is transforming this field. Introduction to Reverse Engineering: A brief overview of the concept of RE, its significance in the fields of security, software development, and technology, and the state of the art. AI Techniques in Reverse Engineering: A discussion on the various AI techniques and algorithms used in Reverse Engineering. AI-driven RE Applications: Insight into the practical applications of AI in Reverse Engineering, including malware analysis, vulnerability detection, and software de-obfuscation. Ivan's AI-based Open Source Reverse Engineering tool Gepetto: It uses OpenAI's gpt-3.5-turbo and gpt-4 models to provide meaning to functions decompiled by IDA Pro. https://github.com/JusticeRage/Gepetto Challenges and Limitations: A discussion on the potential challenges and limitations of using AI in reverse engineering, such as the need for large datasets.

In this episode we're discussing five ways you can use AI to carry out attacks: AI-assisted Social Engineering Attacks AI-powered Malware AI-driven Password Cracking AI-assisted DDoS Attacks AI-based Vulnerability Discovery and Exploitation Needless to say, the purpose is to get acquainted with what can be, and learn how to be better prepared.

In this podcast, we'll discuss the cross section of AI and Security. Sometimes it will be more AI focused, sometimes more security focused, sometimes discussing what’s new. In this very first episode, we’re introducing ourselves: Ivan is a Senior Security Researcher at Kaspersky, and Natalie is an AI Consultant and an OpenAI Developer Ambassador. We then discuss four ways of attacking AI and how to protect against those: Attacks on the model  Attacks on the data used by the model Crafting the prompt to cause unwanted outputs (with time this got the name Prompt Injections) Adversarial Model-to-Model Interaction