# Hello World print("Hello World") ## Who am I? An alien like you! If you look for my social media, here the links: LinkedIn for the career summary: {% embed url="" %} Twitter for chit chats: {% embed url="" %} # What is this AREA? In 2023, I started to lean on the AI/ML/LLM security area, thanks to my self-study habits. I'll keep my notes here! So; These are the voyages of a security enthusiast. Its continuing mission is to explore strange new knowledge. To seek out new ideas and new techniques. To boldly go where no one has gone before! # Self-Study AI/ML/LLM Roadmap for Security Folks This roadmap is coming from a security enthusiast who wanted to improve herself in AI/ML/LLM area to discover its security side. Small tips before jumping in it: * You can have the Coursera courses in audit-only mode. You should only know that you can't get a course certificate or grades in audit mode. 1. 2. * You can jump to security playgrounds without waiting to finish the training list. No harm in poking around and jumping back. Sometimes, you may not know what you will have exploited. So, never give up, long run but worth it. * I strongly recommend building some small apps to understand how it works! * Check [MLU](https://www.youtube.com/@machinelearninguniversity1942/playlists), it's made by AWS <3 You can find the playlists here: ### Roadmap 1. The start point is a good detailed entrance training: Introduction to Machine Learning in Production course from DeepLearning.AI - 2. To get much more know-how about neural networks, supervised learning, and regression problems, this course can be a second step for this roadmap. 3. This course is developed by DeepLearning.AI. At the end of the course, you'll gain foundational knowledge, practical skills, and a functional understanding of how generative AI works. Hopefully, you'll be able to dive into the latest research on Gen AI to understand how companies are creating value with cutting-edge technology- Instruction from expert AWS AI practitioners who actively build and deploy AI in business use cases today. \ As another option, this course could be quite useful: 4. The prompt is everything! You need to learn how you can talk with an LLM! 5. First steps to prompt injection attacks: 6. Small Playgrounds to keep poking around: * Gandalf LLM Challenge - * * Mosscap LLM Challenge - * DoubleSpeak Chat - * 7. More know-how about neural networks and training cycles! \ Advanced Learning Algorithms - 8. This part may require a bit more math background than the others. But it's a gem to learn about Reinforcement Learning. Unsupervised Learning Recommenders Reinforcement Learning - 9. Deep Neural Network - 10. NLP Sequence Models - ### References for Security / Essential to Look 1. OWASP Machine Learning Security Top 10 - 2. OWASP Top 10 for LLM - 3. MITRE ATLAS - 4. HuggingFace.co is the ‘GitHub’ of LLMs and machine learning. This is a central point for ML models, datasets, and other content. The transformers library, maintained by HuggingFace, is the cornerstone of the foundation in open-source LLMs. 5. LLM Agency - LlamaIndex (formerly GPT Index) - LangChain, transformers, agents, llamaindex and other similar libraries or custom codes give LLMs agency. This concept is an agency in the truest meaning of the word. These libraries and the concept of agency expose tools and capabilities to an LLM via prompting and then give the model a task. The model is iteratively prompted by making a plan and then drilling down through each step until it gets a final answer. This is where the major new functionality sits and where the new vulnerabilities will be found.\ 6. LLM Agency - Langchain - 7. LLM Agency - Transformers - 8. Some repose that I got good info\ [https://github.com/jiep/offensive-ai-compilation]()\ \ \ \ \ \ ### YouTube Channels to Stay Updated I've got all those good channel advice from [Garrett](https://www.linkedin.com/in/garrettgalloway/)! (^^,) Thank you so much! [https://www.youtube.com/@matthew\_berman ]()\ \ \ \ \ \ \ \ \ \ \ \ \ \ # AI/ML Dictionary While learning all those new terms, I needed to look back occasionally. So, this dictionary is coming from those lookbacks. * **Data**: Machine learning algorithms rely on data to learn patterns and make predictions. The data can be structured (e.g., databases, spreadsheets) or unstructured (e.g., text, images). It is crucial to have a sufficient and representative dataset for effective learning. * **Training**: During the training phase, the machine learning model learns patterns and relationships in the data. The model is fed with labeled examples (known as the training set) consisting of input data and corresponding desired outputs or labels. Note that training heavily biases a model for some specific output. * **Fine-tuning**: Fine-tuning is an extension of training that is often referred to in the context of LLMs. This form of training typically takes a model after more generalized training and heavily biases it towards a specific type of completion like instruction following, summarization, translation, chat assistant, and other concepts. There are some advanced methods (LoRA + PEFT) that help to prevent or nearly eliminate catastrophic forgetting. * **Features**: Features are specific measurable properties or characteristics of the data that are used as inputs to the machine learning model. Choosing relevant and informative features is important for the model to learn effectively. * **Supervised Learning**: In supervised learning, the training data includes both input data and corresponding labels. The goal is to learn a mapping between inputs and outputs, enabling the model to make accurate predictions on new, unseen data. * **Unsupervised Learning**: In unsupervised learning, the training data consists only of input data without any corresponding labels. The goal is to discover patterns, relationships, or structures in the data, such as clustering similar data points or dimensionality reduction. * **Model**: The term model can refer to the software that defines/implements model architecture, the weights/biases, or both the architecture and the weights/biases combined. Model weights/biases can range in size from a single integer value of a single element up to multiple terabytes in size for the largest of LLMs. The software that defines/implements the model architecture is usually orders of magnitude smaller than the weights/biases themselves for modern LLMs. * **Foundation Model:** The term Foundation Models are large-scale machine learning models trained on a wide variety of internet text, serving as a base or "foundation" for numerous downstream tasks. These models, such as GPT-3 or GPT-4, exhibit general applicability across diverse tasks, even those unseen during training. They also demonstrate few-shot learning, or the ability to understand a task from minimal examples. However, they pose challenges and ethical concerns, including harmful biases and misuse potential, as they often reproduce the biases present in their training data. Efforts are underway to mitigate these issues and enhance their safety and usefulness. * **Model Evaluation**: After training, the model's performance is evaluated using a separate dataset called the validation or test set. Evaluation metrics, such as accuracy, precision, recall, or mean squared error, are used to assess how well the model generalizes to new, unseen data. * **Prediction/Inference**: Once trained, the machine learning model can be used to make predictions or decisions on new, unseen data. The model takes the input data and produces an output or prediction based on what it has learned during training. * **Overfitting and Underfitting**: Overfitting occurs when a model performs well on the training data but fails to generalize to new data. Underfitting happens when a model is too simple to capture the underlying patterns in the data. Balancing between these two is essential to build a well-performing model. * **Catastrophic Forgetting**: Catastrophic forgetting can be considered similar to overfitting, however it is typically referenced in the context of LLMs that lose entire subject areas of knowledge or behavioral capabilities when they are fine-tuned or retrained to do a specific task. * **Feature Engineering**: Feature engineering involves selecting, transforming, and creating features from the raw data to improve the model's performance. It requires domain knowledge and creativity to extract meaningful information that helps the model learn effectively. * **Deep Learning**: Deep learning is a subset of machine learning that focuses on training artificial neural networks with multiple layers (deep neural networks). Deep learning models have shown remarkable success in various domains, including computer vision, natural language processing, and speech recognition. * **Supervised Learning Algorithms**: Supervised learning algorithms include popular methods like linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), and naive Bayes. These algorithms learn from labeled training data to make predictions or classify new data. * **Unsupervised Learning Algorithms**: Unsupervised learning algorithms focus on finding patterns, structures, or relationships in the data without using labeled examples. Clustering algorithms, such as k-means clustering and hierarchical clustering, group similar data points together. Dimensionality reduction techniques, such as principal component analysis (PCA) and t-SNE, reduce the number of features while retaining important information. * **Reinforcement Learning**: Reinforcement learning is a type of machine learning where an agent learns to interact with an environment to maximize a reward signal. The agent takes actions in the environment and receives feedback in the form of rewards or penalties. It learns through trial and error, aiming to discover an optimal policy for decision-making. * **Neural Networks**: Neural networks are a class of machine learning models inspired by the structure and function of the human brain. They consist of interconnected nodes called artificial neurons or "neurons." Deep neural networks (DNNs) are neural networks with multiple hidden layers, enabling them to learn hierarchical representations of data. Convolutional Neural Networks (CNNs) are commonly used for image and video analysis, while Recurrent Neural Networks (RNNs) are well-suited for sequential data, such as text or time series. * **Feature Selection**: Feature selection aims to identify the most relevant and informative features from a dataset. It helps to reduce dimensionality, improve model performance, and prevent overfitting. Techniques like forward selection, backward elimination, and regularization methods (e.g., Lasso or Ridge regression) can be used for feature selection. * **Cross-Validation**: Cross-validation is a technique used to assess the performance of a machine learning model. It involves splitting the dataset into multiple subsets (folds). The model is trained on a portion of the data and evaluated on the remaining fold. This process is repeated several times, and the results are averaged to obtain a more reliable estimate of the model's performance. * **Hyperparameter Tuning**: Hyperparameters are parameters that define the behavior and performance of machine learning algorithms. Hyperparameter tuning involves selecting the optimal combination of hyperparameter values to improve the model's performance. Techniques like grid search, random search, and Bayesian optimization can be used for hyperparameter tuning. * **Transfer Learning**: Transfer learning is a technique where knowledge learned from one task or domain is applied to another related task or domain. Instead of training a model from scratch, pre-trained models (usually trained on large datasets) can be used as a starting point. This approach saves computational resources and helps in situations with limited training data. * **Ensemble Learning**: Ensemble learning combines multiple individual models to make predictions or decisions. It aims to improve the overall performance and robustness of the model. Techniques like bagging (e.g., random forests), boosting (e.g., AdaBoost, Gradient Boosting), and stacking are commonly used in ensemble learning. * **Bias**: Bias is an overloaded term within AI/ML and is not inherently negative in its use. When discussing model bias in the context of inputs and outputs, bias is exactly what you are trying to achieve when you are training a model if you want it to respond in a specific way. You are attempting to sway a model to a specific output based on a specific input, you are biasing the output. Understand that this term is used in the normal context of AI/ML but also in the context of fairness and toxicity; always ensure you have appropriate context to interpret the meaning. * **Bias and Fairness**: Machine learning models can inadvertently inherit biases from the training data, leading to unfair or discriminatory outcomes. Bias and fairness considerations are essential to ensure that machine learning models treat individuals fairly and equitably across different groups. Techniques like bias detection, bias mitigation, and fairness-aware learning are actively researched areas. * **Data Preprocessing**: Data preprocessing is a crucial step in machine learning. It involves cleaning, transforming, and normalizing the data to improve the quality and suitability for the learning algorithm. Common preprocessing techniques include handling missing values, dealing with outliers, scaling features, and encoding categorical variables. * **Imbalanced Data**: Imbalanced data refers to datasets where the number of examples in different classes is significantly imbalanced. For instance, in fraud detection or rare disease prediction, the positive class (e.g., fraud cases or rare diseases) may be underrepresented. Handling imbalanced data requires techniques such as oversampling, undersampling, or using specialized algorithms like SMOTE (Synthetic Minority Over-sampling Technique). * **Model Evaluation Metrics**: Model evaluation metrics assess the performance of machine learning models. The choice of metrics depends on the specific problem and the type of model. Common evaluation metrics include accuracy, precision, recall, F1 score, area under the ROC curve (AUC-ROC), mean squared error (MSE), mean absolute error (MAE), and many more. * **Time Series Analysis**: Time series analysis deals with data that is collected over time, such as stock prices, temperature records, or sensor data. Techniques like autoregressive integrated moving average (ARIMA), seasonal decomposition of time series (STL), and recurrent neural networks (RNNs) with Long Short-Term Memory (LSTM) are commonly used for time series forecasting and analysis. * **Model Deployment**: Once a machine learning model is trained, it needs to be deployed in a production environment for real-world use. This involves integrating the model into an application, creating APIs for model inference, and ensuring scalability, performance, and reliability. Deployment frameworks like TensorFlow Serving, Flask, or Docker containers are often used. * **Online Learning**: Online learning, also known as incremental learning or streaming learning, refers to a learning approach where the model is continuously updated and adapted to new incoming data. This is particularly useful when dealing with large-scale or streaming datasets that arrive sequentially and need real-time learning and prediction. * **Interpretability and Explainability**: As machine learning models become more complex, there is a growing need to understand and interpret their decisions. Interpretable machine learning techniques aim to provide insights into the model's decision-making process, making it easier to understand and trust the model's predictions. Techniques like feature importance, SHAP values, and surrogate models are used for model interpretability. * **AutoML**: AutoML (Automated Machine Learning) refers to the use of automated tools and techniques to automatically select, preprocess, and optimize machine learning models. It aims to streamline and simplify the machine learning pipeline, making it more accessible to users without extensive knowledge of machine learning algorithms. * **Adversarial Machine Learning**: Adversarial machine learning focuses on studying and defending against adversarial attacks on machine learning models. Adversaries intentionally manipulate or perturb the input data to deceive or mislead the model. Adversarial techniques aim to improve model robustness and security, especially in critical applications like cybersecurity and image recognition. * **Ethical Considerations**: As machine learning models have a growing impact on society, ethical considerations become essential. Issues like privacy, bias, transparency, and accountability need to be addressed to ensure the responsible development and deployment of machine learning systems. * **Natural Language Processing (NLP)**: Natural Language Processing focuses on enabling computers to understand, interpret, and generate human language. It involves techniques such as text classification, sentiment analysis, named entity recognition, machine translation, question answering, and language generation. NLP plays a crucial role in applications like chatbots, virtual assistants, and language understanding systems. * **Computer Vision**: Computer Vision is a field that deals with enabling computers to interpret and understand visual information from images or videos. It involves tasks such as object detection, image segmentation, image classification, facial recognition, and image generation. Computer Vision finds applications in autonomous vehicles, surveillance systems, medical imaging, and augmented reality. * **Deep Reinforcement Learning**: Deep Reinforcement Learning combines deep learning techniques with reinforcement learning. It involves training deep neural networks to learn policies that maximize rewards through interactions with an environment. Deep reinforcement learning has achieved remarkable success in complex tasks like playing games (e.g., AlphaGo and OpenAI Five) and robotic control. * **Generative Models**: Generative models are machine learning models that can generate new samples similar to the training data. Variational Autoencoders (VAEs) and Generative Adversarial Networks (GANs) are popular generative models. They have applications in image synthesis, text generation, and data augmentation. * **One-shot Learning**: One-shot learning focuses on the problem of learning from a single or a few examples per class. It aims to develop models that can generalize and recognize new instances from limited training data. Few-shot learning and zero-shot learning are extensions of one-shot learning, addressing scenarios with a small number of examples per class. * **Autoencoders**: Autoencoders are unsupervised learning models that aim to learn efficient representations or encodings of input data. They consist of an encoder network that compresses the input data into a lower-dimensional representation and a decoder network that reconstructs the input from the encoded representation. Autoencoders are used for tasks like dimensionality reduction, denoising, and anomaly detection. * **Bayesian Learning**: Bayesian learning is an approach to machine learning that incorporates prior knowledge or beliefs about the problem. It uses Bayesian inference to update and refine the model's predictions based on the observed data. Bayesian learning helps in situations with limited data and provides uncertainty estimates for predictions. * **Neuroevolution**: Neuroevolution combines neural networks and evolutionary algorithms. It involves using evolutionary techniques like genetic algorithms or genetic programming to optimize the architecture or parameters of neural networks. Neuroevolution is commonly used in reinforcement learning tasks and neural architecture search. * **Privacy-Preserving Machine Learning**: Privacy-preserving machine learning techniques aim to protect the privacy of sensitive data while allowing for analysis and model training. Techniques like federated learning, differential privacy, and secure multi-party computation enable collaborative learning without sharing raw data. * **Quantum Machine Learning**: Quantum machine learning explores the intersection of quantum computing and machine learning. It investigates how quantum algorithms and quantum computers can enhance or accelerate various machine learning tasks, such as optimization, clustering, and feature selection. * **Graph Neural Networks**: Graph Neural Networks (GNNs) are a class of neural networks designed to process and analyze data represented as graphs. GNNs can capture relationships and dependencies among nodes and edges in complex networks. They find applications in social network analysis, recommendation systems, and molecular chemistry. * **Time Series Forecasting**: Time series forecasting involves predicting future values or trends based on historical data that is ordered chronologically. Techniques like autoregressive integrated moving average (ARIMA), exponential smoothing methods, recurrent neural networks (RNNs), and Long Short-Term Memory (LSTM) networks are commonly used for time series forecasting. * **Anomaly Detection**: Anomaly detection focuses on identifying rare or abnormal instances in a dataset that deviate from the norm. It finds applications in fraud detection, network intrusion detection, system monitoring, and outlier detection. Approaches for anomaly detection include statistical methods, clustering, and supervised or unsupervised learning algorithms. * **Transfer Learning in NLP**: Transfer learning has had significant impact in natural language processing (NLP). Pre-trained language models like BERT, GPT, and RoBERTa, trained on massive amounts of text data, can be fine-tuned for specific NLP tasks with limited labeled data. Transfer learning has revolutionized various NLP tasks, including text classification, named entity recognition, and text generation. * **Causal Inference**: Causal inference aims to understand cause-and-effect relationships between variables. It involves identifying causal effects from observational or experimental data. Causal inference techniques, such as propensity score matching, instrumental variable analysis, and causal graphical models, help in making informed decisions and understanding the impact of interventions. * **Semi-Supervised Learning**: Semi-supervised learning combines labeled and unlabeled data to train machine learning models. It leverages the unlabeled data to improve the model's performance and generalization. Semi-supervised learning is useful when acquiring labeled data is expensive or time-consuming, as it can make use of large amounts of readily available unlabeled data. * **Domain Adaptation**: Domain adaptation focuses on transferring knowledge learned from one domain (source domain) to another domain (target domain). It addresses the challenge of model performance degradation when the training data and test data come from different distributions. Domain adaptation techniques aim to align the source and target domains to improve model performance on the target domain. * **Capsule Networks**: Capsule Networks (CapsNets) are a type of neural network architecture introduced as an alternative to traditional convolutional neural networks (CNNs). CapsNets aim to capture hierarchical relationships between visual entities and handle spatial relationships more effectively. They show potential in tasks like object recognition, pose estimation, and image synthesis. * **Responsible AI**: Responsible AI focuses on developing machine learning systems that are fair, transparent, and unbiased, and that mitigate potential risks and ethical concerns. It involves addressing issues such as fairness, accountability, transparency, interpretability, and robustness in machine learning models and systems. * **Adaptive Learning**: Adaptive learning refers to machine learning systems that can adapt and change their behavior based on user feedback or changing environments. These systems continuously learn from user interactions and update their models to personalize the learning experience or improve performance over time. * **Metaheuristic Optimization**: Metaheuristic optimization algorithms are used to solve complex optimization problems that cannot be easily solved by traditional optimization techniques. These algorithms, such as genetic algorithms, particle swarm optimization, and simulated annealing, are inspired by natural processes and can efficiently explore large search spaces to find near-optimal solutions. * **Multi-modal Learning**: Multi-modal learning deals with data that comes from multiple sources or modalities, such as text, images, audio, or sensor data. It aims to integrate and learn from different modalities to improve performance or gain a more comprehensive understanding of the data. Multi-modal learning finds applications in areas like multimedia analysis, healthcare, and autonomous systems. * **Meta-learning**: Meta-learning, also known as learning to learn, focuses on developing models or algorithms that can learn how to learn new tasks or adapt quickly to new environments. Meta-learning algorithms aim to acquire knowledge and meta-knowledge from previous learning experiences to facilitate the learning of new tasks with limited data. It finds applications in few-shot learning, hyperparameter optimization, and domain adaptation. * **Data Augmentation**: Data augmentation techniques involve creating new training examples by applying transformations or perturbations to the existing data. These techniques help increase the diversity of the training set, improve model generalization, and reduce overfitting. Common data augmentation methods include rotation, translation, scaling, flipping, and adding noise to the data. * **Privacy-Preserving Techniques**: Privacy-preserving techniques in machine learning aim to protect the privacy of sensitive data during model training or inference. Techniques like secure multi-party computation, federated learning, and homomorphic encryption allow for collaborative model training or prediction while preserving data privacy. * **AutoML for Neural Architecture Search**: AutoML techniques, including neural architecture search (NAS), automate the process of designing or discovering optimal neural network architectures for a given task. NAS algorithms explore and optimize the architecture space by using reinforcement learning, evolutionary algorithms, or gradient-based methods. * **Model Compression**: Model compression techniques aim to reduce the size and computational complexity of machine learning models, making them more efficient for deployment on resource-constrained devices or systems. Techniques like pruning, quantization, knowledge distillation, and model factorization help in compressing models while maintaining performance. * **Knowledge Distillation**: Knowledge distillation is a technique where a smaller, more compact model, known as a student model, is trained to mimic the predictions or behavior of a larger, more complex model, known as a teacher model. This helps transfer knowledge from the teacher model to the student model, enabling the student model to achieve similar performance while being more lightweight. * **Human-in-the-Loop Machine Learning**: Human-in-the-loop machine learning involves integrating human expertise and feedback into the machine learning pipeline. It combines automated learning algorithms with human decision-making to create interactive and iterative learning systems. Human-in-the-loop approaches are used in tasks like active learning, interactive data labeling, and model debugging. * **Neurosymbolic AI**: Neurosymbolic AI combines the power of neural networks and symbolic reasoning. It aims to integrate deep learning models with symbolic representations and reasoning capabilities, enabling machines to learn from data while also leveraging human-like symbolic reasoning and logic. * **Transfer Learning in Computer Vision**: Transfer learning has also made significant strides in computer vision tasks. Pre-trained convolutional neural networks (CNNs) such as VGG, ResNet, and Inception are often used as feature extractors or fine-tuned for specific vision tasks, allowing for effective learning with limited labeled data. * **Data Bias and Fairness**: Data bias and fairness have gained considerable attention in machine learning. It involves addressing biases in training data that may lead to unfair or discriminatory outcomes. Techniques such as bias detection, bias mitigation, and fairness-aware learning aim to ensure equitable and unbiased machine learning models. * **AutoML for Hyperparameter Optimization**: AutoML techniques extend beyond neural architecture search. They also encompass automating the process of hyperparameter optimization, which involves selecting the best values for hyperparameters that control the learning process. Techniques like Bayesian optimization, genetic algorithms, and random search are employed for automated hyperparameter tuning. * **Active Learning**: Active learning is a semi-supervised learning approach that involves an iterative process of selecting the most informative or uncertain examples from a pool of unlabeled data and requesting labels from an oracle (e.g., a human expert). This approach helps optimize the learning process by actively selecting the most valuable data points to label, reducing labeling effort. * **Meta-reinforcement Learning**: Meta-reinforcement learning combines ideas from reinforcement learning and meta-learning. It focuses on training agents that can adapt to new and unseen reinforcement learning tasks with minimal learning or fine-tuning. Meta-reinforcement learning algorithms learn a meta-policy that generalizes across multiple tasks and guides the learning process for new tasks. * **Online Recommender Systems**: Online recommender systems aim to provide personalized recommendations in real-time, often in online platforms such as e-commerce, streaming services, or social media. These systems leverage techniques such as collaborative filtering, content-based filtering, and reinforcement learning to make timely and relevant recommendations to users. * **Model Interpretability and Explainability in Deep Learning**: As deep learning models become increasingly complex, interpretability and explainability have become critical. Techniques such as attention mechanisms, gradient-based methods (e.g., Grad-CAM), and rule extraction aim to provide insights into the decision-making process of deep learning models, making them more transparent and understandable. * **Weakly Supervised Learning**: Weakly supervised learning deals with scenarios where the training data is only partially labeled or labeled at a coarse level. It aims to learn models with limited supervision, leveraging techniques such as multiple instance learning, co-training, or self-supervised learning. Weakly supervised learning is useful when acquiring fine-grained labels is expensive or impractical. * **Gaussian Processes**: Gaussian Processes (GPs) are probabilistic models that can be used for regression, classification, and uncertainty estimation. GPs provide a flexible framework for modeling data and making predictions, offering non-parametric and Bayesian approaches. They find applications in diverse areas such as surrogate modeling, Bayesian optimization, and time series analysis. * **Analogical Reasoning**: Analogical reasoning involves solving problems by finding similarities and relationships between different examples or domains. It is inspired by human reasoning processes and has applications in areas such as natural language processing, image recognition, and cognitive modeling. * **Meta-learning for Few-shot Learning**: Meta-learning approaches are widely used for few-shot learning, where the goal is to learn new concepts or tasks with limited training data. Meta-learning algorithms aim to acquire knowledge from previous tasks to adapt quickly to new tasks with only a few examples per class. * **Adversarial Machine Learning**: Adversarial machine learning focuses on understanding and defending against adversarial attacks on machine learning models. Adversarial attacks involve intentionally manipulating input data to deceive or mislead the model's predictions. Defenses against adversarial attacks include robust optimization, adversarial training, and input sanitization techniques. * **Automated Feature Engineering**: Automated feature engineering involves the automatic generation or selection of features from raw data. It leverages techniques such as feature extraction, dimensionality reduction, and feature selection algorithms to transform raw data into meaningful and informative representations for machine learning models. * **Graph Representation Learning**: Graph representation learning focuses on learning meaningful and informative representations of nodes and edges in graph-structured data. Techniques like graph neural networks (GNNs), graph embedding methods, and random walk-based algorithms enable effective analysis and modeling of complex relational data. * **Causal Discovery**: Causal discovery aims to identify causal relationships between variables in a dataset. It involves inferring cause-and-effect relationships from observational or experimental data. Causal discovery algorithms help uncover underlying mechanisms and dependencies in complex systems. * **Metaheuristic Learning to Optimize Hyperparameters**: Metaheuristic algorithms, such as genetic algorithms, particle swarm optimization, or simulated annealing, can be used to optimize hyperparameters of machine learning models. These algorithms explore the hyperparameter space efficiently and find near-optimal combinations of hyperparameters. * **Automated Machine Learning (AutoML)**: AutoML involves automating the process of designing, training, and optimizing machine learning models. It encompasses techniques like neural architecture search, hyperparameter optimization, feature selection, and model selection. AutoML aims to simplify the machine learning workflow and make it accessible to non-experts. * **Unsupervised Domain Adaptation**: Unsupervised domain adaptation deals with scenarios where labeled data is available in a source domain but not in the target domain. It focuses on learning representations that can generalize well across domains without relying on labeled data from the target domain. Unsupervised domain adaptation techniques bridge the gap between different domains and improve model performance on the target domain. * **Machine Learning in Robotics**: Machine learning plays a crucial role in robotics, enabling robots to perceive and interact with the environment, learn from data, and make intelligent decisions. Machine learning techniques, such as reinforcement learning, imitation learning, and computer vision, are used for robot control, perception, navigation, and manipulation tasks. * **Agent, Agency, or Agentic Behavior**: The concept of an agent, an LLM with agency, or an LLM presenting agentic behavior can be described as tool exposure and tool use by an LLM through iterative prompting and augmentation of prompting based on results of tool use. The LLM decides the path and which tools to use - the system is acting as an independent agent in the truest sense of the word. * **Emergent Behaviors**: Emergent behaviors are behaviors for which a model was not specifically trained or fine-tuned for, but inherently acquire through their initial training data. There is a high correlation with the number of parameters a model is built on and many of these emergent behaviors. Some examples of emergent behaviors include the ability to do math, summarization, code completion, joke explanations, logical chain of thought reasoning, and many others. Some of these behaviors may be harmless, some may be sought after, others may be hazardous or exploitable. * **Transformer Model**: A type of neural network that does away with recurrent neural network (RNN) and convolutional neural network (CNN) concepts and replaces them entirely with attention mechanisms. These are further broken down into encoder models, decoder models, and encoder-decoder models. Training a transformer model takes less time and ends up generalizing its training much better than RNN or CNN models. * **Return Augmented Generation (RAG)**: RAG is a technology that allows conventional applications to augment a prompt for an LLM or other model. This concept is often associated with data enrichment as discussed by conventional ML researchers and engineers. You are effectively enriching the data, the content of the prompt, with additional information from an outside system. In the case of an LLM based chatbot, RAG may automatically enrich the user’s prompt with a wikipedia entry based on the subject extracted from the user’s prompt. This gives the LLM more information to work with and remain factual by becoming heavily biased by the input prompt with RAG content. * **Token**: The atomic representation of any data element that is fed into a transformer model and some other forms of NLP based machine learning models. Tokens for LLMs can consist of short phrases, words, word parts, letters, numbers, symbols, whitespace, and other non-printable characters. An LLM never sees the text itself, only an integer/float representation of this textual object; consequently, LLMs can’t spell (at least not very well). Tokens can also take the form of individual pixel information, discrete audio samples, or any other form of digitized information you train a transformer model or other type of model that uses tokens on. * **Tokenizer**: For LLMs, there are two basic types of tokenizers: greedy tokenizers and NLP tokenizers. The effectiveness of the tokenizer is heavily correlated with how well the model performs at abstracting information from training data. Greedy tokenizers for LLMs just match a list of words, wordparts, letters, numbers, symbols, and whitespace against the input text and convert this input text into an array of tokens (integers). NLP based tokenizers are often trained models themselves that break down language and label the constituent parts with context related to the language itself. This additional context often allows smaller models to converge faster in training. * **Special Tokens**: Special tokens are used by almost all LLMs and other transformer based models. These tokens can be used as delimiters, end-of-string, end-of-line, and other similar ideas that may or may not be represented by the data itself as it is fed in. These special tokens may also allow LLMs and other transformer based models to output specific styles in their text or delimit other aspects of data output in the case of language based transformers. * **Prompt**: Input into an LLM. Prompts can be instructions, text for completion, and other textual information. There are subcategories of prompting such as system prompting or pre-prompting, RAG prompting, and user prompting. These are often delimited through textual delimiters or special tokens. Prompting is used to bias an LLM towards a specific output, output style, or behavior. * **Prompt Engineering**: Heavily biasing an LLM to handle a specific task through prompt formatting, language, augmentation, and examples. Prompt engineering is inexorably tied to RAG as all of this information is contained within a prompt. Depending on how you delimit and present your information to a model and how this model was fine-tuned or trained can drastically impact how it is able to respond to your prompting. Prompt engineering is currently far more art than science and requires brute force attempts to convince the model to perform the task you are requesting until you learn its capabilities and what works. The science portion would be reading the papers about how the original model was trained, what data it was trained on, and how this training information was presented to the model. In some cases, prompt engineering will not turn up with effective responses and you’ll be forced to fine-tune or further fine-tune your your model. In other cases, you’ll determine a model is not sufficient for your use-case. * **n Shot Prompting**: This can take the form of zero shot, one shot, two shot, few shot, and other forms of prompt engineering. Zero shot prompting is prompting an LLM using unique structure and language that was not seen in training but may have been seen in fine-tuning. One shot prompting gives an LLM an example prompt and example response and then another prompt to address. Two shot prompting gives an LLM two example prompts and example responses and then a third prompt to address. Few shot prompting is 3-5 example prompts and example responses followed by an additional prompt to address. Typically using more than 3 examples provides diminishing returns in response quality and providing 5+ examples indicates you should probably further train/fine-tune your model or that your model may not be capable of handling the complexity of your request. # Generative AI in a Nutshell # (WIP) AI/ML/LLM Application Security Testing Any AI/ML/LLM-related application should undergo pen testing and red teaming activities according to size and needs. We should pay the same attention to traditional applications here. There is no difference in the risk perspective here. Due to the model's capabilities and improvision nature, high-profile risks are more likely. In the application security testing, I prefer to examine under two headlines:\ \- Predictive AI System Security Testing\ \- Generative AI System Security Testing In general, both have a model and data the models are trained on. However, the first-category systems involve using machine learning models to forecast future events or behaviours based on historical data. The second-category systems are working on generating entirely new results using the experience they got from the provided data.\ \ Both sides share some architectural parts and joint attack surfaces. For example, both have training data, models, and tokenization as their central parts. For the attack surface parts, both have similar tokenization issues, privacy attacks, and model bias risks on their attack surfaces. All experienced and skilled security testers have a common approach: Know your target! So, whether you're Predictive or Generative, you should know your target. First questions to know your target: * What kind of model does it use? * How it's trained, and how the fine-tuning cycles will go, if there are any? * What is the data source? Could it be poisoned? * Where is the tokenization? What kind of tokens can you use for manipulating the model via inputs? Are there any special tokens that can trigger some behaviours like prediction? For example, MASK token. * What kind of data storage is used?
# (WIP) Offensive Approach for Prompt Injection Attacks As we discussed earlier, Prompt Injection can be a way to exploit many different vulnerabilities in your Generative AI Application. All the security engineers and folks may not be scientists, but security can be practical in fascinating ways. The best way to secure complex stuff is to test them in practical and simplified ways. Let's jump on the approach. 1. Analyze the architecture - Recon Offensive testing starts with good reconnaissance techniques. Security is still security, so you should know your target. I assume you're going through approved white-box testing for your role. * Read all the architectural references, even for little plugins that may impact the system. Then, re-draw the architecture in your way. In my way, I track down the user input's journey and re-draw that journey in the security language. 2. Track down the Input In every testing discipline, input manipulation-response detection is the most helpful tactic for discovering weaknesses. It's the same in LLM-integrated apps. You need to identify if your input changes after you enter it. This change can increase the context quality with extra reasoning, safety filter masking/shaping before feeding it to the LLM, etc. 3. Code Review is your BFF In a good code review, you should identify prompt filters, reasoning points, RAG search queries/field names, agency relationships like executions, data searches, prompt template details like template-supported filters/guardrails, etc. Identifying all those critical points in the code review may give you better visibility for your tracked input. If you track down your input like in the 3rd step, now you can plan which point you can manipulate. # Standard Input: Prompt Injection This blog aims to explain the nature of the prompt injection attacks in LLM-based apps and how a security engineer can approach an LLM-based app for testing. ### What is Prompt Injection? | | The prompt injection is a type of security vulnerability that can occur when large language models (LLMs) are used in applications. It occurs when an attacker can inject malicious text into the prompt that generates the LLM's output. **This can cause the LLM to perform unintended actions, such as revealing sensitive information, generating harmful content, or even taking control of the application itself.** | | ------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: | ## Strategic Significance of Prompt Injection In the offensive security area, the primary mindset revolves around manipulating inputs as an initial strategy. It is commonly believed that every cybersecurity journey begins with a single quote, which can potentially lead to injection attacks. This concept forms the most fundamental starting point in the field. I also accept this as a mindset for my approach to almost everything as a security-minded person. If there is a possibility of altering the input effectively, that rings the bells for possible weaknesses. Prompts constitute the principal input for most Large Language Model (LLM)-based architectures. These prompts can take various forms, including text, images, voice, or a combination thereof. LLM-based architectures are often responsible for processing sensitive data and may exhibit agentic behaviours, such as executing commands or creating resources in response to prompts. A comprehensive understanding of the attack surface in this context should encompass the capabilities of LLM interactions, the nature of the data, and its processing methods. Therefore, a well-crafted prompt could potentially expose sensitive information or introduce malicious code into the generated outputs, contingent upon the system's capabilities and architecture. ## Short Approach of the Prompt Injection Methods Thinking with the offensive security approach again here. Let's start with the testing styles: Blackbox, Graybox, and Whitebox. As a summary for the people who are not familiar with those terms, **Blackbox:** as a security tester, you have no information about the system. You need to discover everything on your own. You cannot access source code or any other private resources, only public ones. **Graybox:** you may have access to some private resources like API explanations, extra parameters that you may submit, etc. It may be different in the different types of tests like web security, network security, etc. **Whitebox:** You have access to everything, such as source code, architecture design documents, access to different layers in the system, etc. You can evaluate the system with all the aspects.
The same structure may apply while testing the LLM-based apps. While testing the LLM-based applications, you may use architectural information to improve your tests. Otherwise, it may stay in Blackbox methods if you only send prompts and observe the answers without any information about what is behind them. Still, it can be impactful; however, if you examine the training cycles, training data, tokenization, prompt template, data processing, data storage, etc, you may hit the right spots and catch more issues on your target LLM-based application. I'd redefine the approaches like the below: **Blackbox:** No data about what's behind like in the traditional way. **Graybox:** Has access to some resources like tokenization, prompt template, etc. **Whitebox:** Has access to all the resources and full access to *the model*. ### What attackers can do over Prompt Injections? There are various cases in which a prompt injection can lead. 1. **Information Extraction**: Attackers can carefully craft prompts to extract confidential, proprietary, or otherwise sensitive information from models trained on private datasets. For instance, trying to get pieces of code, algorithms, or specific details that shouldn't be disclosed. 2. **Model Misdirection**: An attacker might manipulate prompts to make the LLM produce incorrect, misleading, or harmful information, leading users astray or causing them to make poor decisions based on the output. 3. **Model Behavior Revelation**: Attackers can systematically probe the model to understand its inner workings, biases, and training data specifics. This can expose the model's weaknesses, making it susceptible to more targeted attacks. 4. **Generating Offensive Content**: Crafted prompts might coax the model into generating inappropriate, discriminatory, or offensive content, which can be used to discredit the deploying organization or harm users. 5. **Amplifying Biases**: Attackers can intentionally create prompts that highlight and amplify inherent biases in the LLM to exploit these biases or demonstrate the model's lack of neutrality. 6. **Social Engineering Attacks**: By understanding how the model responds, attackers can craft messages that appear legitimate and use them in phishing or other types of social engineering attacks. 7. **Misrepresentation**: Attackers can misuse the model's output to misrepresent facts, views, or beliefs, potentially spreading misinformation. 8. **Automated Attacks**: Knowing how an LLM responds, automated systems can be designed to continually exploit the model by overwhelming it with requests or systematically extracting information. 9. **Evasion of Content Filters**: If LLMs are used in content moderation, attackers could craft input evading detection by understanding the model's blind spots. ### Prompt Engineering for Injection into Prompts Prompt engineering refers to the art and science of crafting input prompts to effectively guide a machine learning model, especially a Large Language Model, towards producing a desired output. The term often emerges in the context of few-shot or zero-shot learning scenarios where fine-tuning on task-specific data is either unavailable or not desired. Prompt engineering can involve: 1. **Rephrasing Questions**: Sometimes, rewording a question can lead to clearer, more accurate answers from the model. For instance, instead of "Tell me about X," asking, "What is the definition of X?" might yield a more concise and direct response. 2. **Providing Context**: Providing additional context or elaborating on the specific aspect of a query can help guide the model in generating more relevant answers. 3. **Explicit Instructions**: It can be beneficial to give the model clear instructions on the format or kind of answer expected. For example, ask, "List five examples of X" instead of just "Examples of X." 4. **Use of Examples**: In few-shot learning, providing one or more examples can help specify the task. For instance, if you're looking to translate English to French, providing an example like "Translate the following English sentences to French: ..." can guide the model. 5. **Iterative Querying**: If the first response from the model isn't satisfactory, refining the prompt or asking follow-up questions based on the model's initial response can be effective. 6. **Temperature and Max Tokens**: Beyond just the text prompt, parameters like "temperature" and "max tokens" can be adjusted to influence the randomness and length of the model's outputs. Prompt engineering is essential because Large Language Models do not always understand the context as humans do. The right prompt can bridge the gap between the model's raw capability and the user's needs. As LLMs become more prevalent, developing expertise in prompt engineering is crucial for extracting maximum utility from these models. # (WIP) Training Issues ### Understanding AI Training Cycles AI systems, particularly those based on machine learning (ML) and deep learning (DL), require extensive training cycles using large datasets to learn and make predictions or decisions. These cycles can be broadly categorized into three phases: data collection, model training, and model validation. Each phase presents unique security challenges. Let's generalize some possible attacks under those three phases. #### Possible Data Collection **Security Issues**: * **Data Poisoning**: Malicious actors may introduce corrupted data into the dataset, aiming to skew the AI model's learning process. Discovery typically involves data validation techniques and anomaly detection to identify outliers that do not fit the expected data distribution. * **Privacy Leaks**: Collecting data from individuals without adequate consent or security measures can lead to privacy breaches. Discovery involves auditing data collection processes and employing privacy-preserving techniques like differential privacy. **Risks**: * Skewed AI decisions, potentially causing financial loss or reputational damage. * Legal repercussions from privacy violations. #### Possible Model Training **Security Issues**: * **Adversarial Attacks**: During training, models may be susceptible to adversarial examples designed to mislead AI predictions. These can be discovered through robustness testing, where the model is exposed to various manipulated inputs to assess its response. * **Overfitting to Sensitive Data**: If a model overfits its training data, it might inadvertently reveal sensitive information through its predictions. Techniques like model auditing and implementing generalization measures (e.g., regularization) can help identify and mitigate this issue. **Risks**: * Compromised decision-making leads to security vulnerabilities. * Unintentional data leakage, compromising user confidentiality. #### Possible Model Validation **Security Issues**: * **Insufficient Testing**: Failing to thoroughly test the model against a wide range of scenarios can leave unseen vulnerabilities. This can be discovered through comprehensive testing, including stress and scenario-based tests, to evaluate the model's performance across diverse conditions. * **Bias and Fairness**: Models might exhibit biased behaviour if not properly validated for fairness, which can be discovered through fairness assessments and bias mitigation techniques. **Risks**: * Inadequate model performance under unexpected conditions, potentially endangering users. * Ethical and legal issues from biased decision-making. ### (WIP) How can you find the vulnerabilities in the training parts? \- Training cycle frequency and how data is provided into those cycles\ \- Feedback mechanisms\ \- Platform related issues\ \- Privacy matters\ \- Synthetic vs. Real Data\ \- # (WIP) Multi-Modal LLM Application Security Testing # (WIP) Resources # What is this AREA? I've started a challenge to improve myself. I'll try to record my readings and research regularly. I'll collect some little notes and thoughts of mine here that I like when I read. So; These are the voyages of a security enthusiast. Its continuing mission: is to explore strange new knowledge. To seek out new ideas and new technics. To boldly go where no one has gone before! # Phishing with MS Office Docs # VSTO and Malicious Office Docs These are the voyages of a security enthusiast. Its continuing mission: to explore strange new knowledge. To seek out new ideas and new technics. To boldly go where no one has gone before! It's not a bug, it's a feature! What is the Visual Studio Tools for Office (VSTO)? * **Visual Studio Tools for Office** (**VSTO**) is a set of development tools available in the form of a [Visual Studio](https://en.wikipedia.org/wiki/Visual_Studio) add-in (project templates) and a [runtime](https://en.wikipedia.org/wiki/Runtime_library) that allows [Microsoft Office](https://en.wikipedia.org/wiki/Microsoft_Office) [2003](https://en.wikipedia.org/wiki/Microsoft_Office_2003) and later versions of Office applications to host the [.NET Framework](https://en.wikipedia.org/wiki/.NET_Framework) [Common Language Runtime](https://en.wikipedia.org/wiki/Common_Language_Runtime) (CLR) to expose their functionality via .NET. {% embed url="" %} {% embed url="" %} {% embed url="" %} \ \ Any Malicious Payload can be emplaced into any office documents using VSTO. You can find detailed explanations in the below links. {% embed url="" %} {% embed url="" %} {% embed url="" %} Quick practice in a YouTube video: {% embed url="" %} **PS:** * More powerful than Macro and formula injection! You can write anything with C# smoothly. * It can update itself and .NET libraries in every execution. (OMG! It can be a self-updated dropper!) * EDRs can not detect in the same way as macros. **Prevention:** * Disable all add-ins for office in the Group Policy (This may not be useful for large companies.) ![](/files/FtTpRr3PFk6RR8IC9Yra) * Require that application add-ins are signed by Trusted Publisher in the Group Policy. (There are many bypass ways for this(^^,) ) ![](/files/HDn14QCWo4Z7nVwBMjFl) * I have not done it yet, but behavioral analysis can be done for this situation. Also, C# projects could be restricted with the company signature. If a subprocess of a C# project compiling and executing without a signature is observed under an office document, it can be blocked by the EDR. * Also, the below blog post should be read to create a prevention scenario against this kind of phishing attack. {% embed url="" %} -EOF # Malware Analysis & Development # Malware Development These are the voyages of a security enthusiast. Its continuing mission: to explore strange new knowledge. To seek out new ideas and new technics. To boldly go where no one has gone before! How can you understand the movements of malware? The answer is so simple. It would help if you learned how you could develop malware. You should think like an attacker! I'm comparing malware analysis with penetration testing. So, if you know how to develop an application in the target technology, you can analyze the behavior of a system, and you can find vulnerabilities in a system. Every time I start to learn or test new technology, I'm practicing to build a basic application in the target language. Useful knowledge might be found inside the below links about malware development. A mix of everything: :) {% embed url="" %} Step by step and technic by technic, Cocomelonc is giving wonderful details about malware development and analysis in his blog with a good perspective. {% embed url="" %} Thanks to [@aas\_s3curity](https://twitter.com/aas_s3curity) 's tweet I found a valuable malware development blog series in [0xpat's blog](https://0xpat.github.io/). {% embed url="" %} 0xpat's first post to track others: {% embed url="" %} Online training:
{% embed url="" %} {% embed url="" %} {% embed url="" %} Both might be more specific than the above but I like them: {% embed url="" %} {% embed url="" %} ​ Also, I found a continuing playlist on YouTube made by ActiveXsploit that hasn't finished yet. {% embed url="" %} These are my resources to understand malware development. Also, I'm trying to cosubstrate with Red Teaming technics besides working only on Malware development. Because; malware is a coded version of the Red Teaming technics in order from my perspective. So, I'll make a rich list for Red Teaming later.
-EOF # 3rd Party Recon with Javascript - Part1 These are the voyages of a security enthusiast. Its continuing mission: to explore strange new knowledge. To seek out new ideas and new technics. To boldly go where no one has gone before! In a security test engagement, there are many ways to find 3rd Party Libraries like browser and burp extensions. So, I'm using them in my tests. But I noticed that I couldn't see all the libraries except the vulnerable ones in these extensions. So, I'm checking with JavaScipt for complete visibility, especially while JS lib recon. You can see all the JS files, such as custom coded for the project. If you are lucky or in a vulnerable environment, you can find sensitive data, logic problems, etc., in these JS files. Easily developer console is a blessing!\ \ Let's see my little notes. \ \ It's getting all the JS files that are called on the page. querySelectorAll is the key in this tiny script. As you guessed, it's selecting all the \