JustForFun
  • Hello World
  • AI/ML/LLM Security
    • 🐣What is this AREA?
    • 📚Self-Study AI/ML/LLM Roadmap for Security Folks
    • 🌟AI/ML Dictionary
    • 🌰Generative AI in a Nutshell
    • 👹(WIP) AI/ML/LLM Application Security Testing
      • 💉(WIP) Offensive Approach for Prompt Injection Attacks
      • 👾Standard Input: Prompt Injection
      • ⚠️(WIP) Training Issues
      • 🎑(WIP) Multi-Modal LLM Application Security Testing
      • ✨(WIP) Resources
  • Random Research Area
    • What is this AREA?
    • Phishing with MS Office Docs
      • VSTO and Malicious Office Docs
    • Malware Analysis & Development
      • Malware Development
  • AppSecNotes
    • 3rd Party Recon with Javascript - Part1
    • DAV Methods and Old Features
    • API Security Notes
  • OSEP Preperation Notes
    • OSEP Journey Begin!
    • Basics and More
    • Payload Types (Staged vs. Non-Staged)
    • File Smuggling with HTML & JS
    • VBA Basics
    • Basic Phishing Macro Creation Tricks
  • Somethings and Past
    • HackTheBox Lab – Invite Code Write-Up
    • OSCP Yolculuğum
    • VulnHub – SkyTower CTF Walkthrough
    • Markdown Syntax
    • Web Uygulama Güvenliği Ve Güvenli Kod Geliştirme LYK-2014 Notlarım
    • Yalnızca Eğlenmek İçin
Powered by GitBook
On this page

Was this helpful?

  1. AI/ML/LLM Security

(WIP) AI/ML/LLM Application Security Testing

Any AI/ML/LLM-related application should undergo pen testing and red teaming activities according to size and needs. We should pay the same attention to traditional applications here. There is no difference in the risk perspective here. Due to the model's capabilities and improvision nature, high-profile risks are more likely.

In the application security testing, I prefer to examine under two headlines: - Predictive AI System Security Testing - Generative AI System Security Testing

In general, both have a model and data the models are trained on. However, the first-category systems involve using machine learning models to forecast future events or behaviours based on historical data. The second-category systems are working on generating entirely new results using the experience they got from the provided data. Both sides share some architectural parts and joint attack surfaces. For example, both have training data, models, and tokenization as their central parts. For the attack surface parts, both have similar tokenization issues, privacy attacks, and model bias risks on their attack surfaces.

All experienced and skilled security testers have a common approach: Know your target! So, whether you're Predictive or Generative, you should know your target. First questions to know your target:

  • What kind of model does it use?

  • How it's trained, and how the fine-tuning cycles will go, if there are any?

  • What is the data source? Could it be poisoned?

  • Where is the tokenization? What kind of tokens can you use for manipulating the model via inputs? Are there any special tokens that can trigger some behaviours like prediction? For example, MASK token.

  • What kind of data storage is used?

PreviousGenerative AI in a NutshellNext(WIP) Offensive Approach for Prompt Injection Attacks

Last updated 1 year ago

Was this helpful?

👹
Page cover image