Keywords
offensive AI
cybersecurity
deepfake attacks
phishing automation
AI-generated malware
Abstract
The growing role of artificial intelligence (AI) in cybersecurity presents both opportunities and challenges. While AI is widely used to enhance security mechanisms, it is also being leveraged by cybercriminals to orchestrate sophisticated, adaptive, and large-scale attacks. This paper provides a comprehensive review of AI-powered cyber threats, including deepfake-based deception, AI-driven phishing campaigns, auto-mated vulnerability exploitation, and the autonomous generation of malware. Additionally, the paper ex-amines recent attack techniques such as prompt injection, model stealing, and data poisoning, which pose systemic risks to AI security infrastructure. By analyzing current research and real-world incidents, we explore the implications of these threats and evaluate strategic defense mechanisms such as explainable AI (XAI), anomaly detection, and AI-based awareness training. This study aims to support the cybersecu-rity community in understanding and anticipating AI-enabled threats and emphasizes the urgent need for proactive and collaborative defense strategies.
References
Zdrojewski K.: Impact of Artificial Intelligence on Computer Networks. Advances in IT and Electrical En-gineering, (2024), vol 30, pp. 49-59.
Giudice O, Guarnera L, Battiato S.: Fighting Deepfakes by Detecting GAN DCT Anomalies. Journal of Im-aging. 2021; 7(8):128. https://doi.org/10.3390/jimaging7080128.
Jabbarova K., Jafarov S.: AI and Cybersecurity - New Threats and Opportunities. Pakistan Journal of Life and Social Sciences (PJLSS), vol. 22, (2023), doi: 10.57239/PJLSS-2024-22.2.00752.
Alanezi M., AL-Azzawi R. M. A.: AI-Powered Cyber Threats: A Systematic Review. Mesopotamian Journal of CyberSecurity, vol. 4(3), pp. 166-188, (2024). https://doi.org/10.58496/MJCS/2024/021.
ENISA: Threat Landscape 2024. (2024). https://doi.org/10.2824/0710888.
Page M. J., Moher D., Bossuyt P. M., Boutron I., Hoffmann T. C., Mulrow C. D. et al.: PRISMA 2020 explanation and elaboration: updated guidance and exemplars for reporting systematic reviews. BMJ 2021, vol. 372, doi: 10.1136/bmj.n160.
CEPS Task Force report: Artificial intelligence and cybersecurity. Technology, Governance and Policy Chal-lenges. European Defence Agency, pp. 30-35, (2021). https://eda.europa.eu/docs/default-source/docu-ments/ceps-tfr-artificial-intelligence-and-cybersecurity.pdf
Mcafee: Artificial Imposters - Cybercriminals Turn to AI Voice Cloning for a New Breed of Scam. (2023). mcafee.com/blogs/privacy-identity-protection/artificial-imposters-cybercriminals-turn-to-ai-voice-cloning-for-a-new-breed-of-scam.
Alali A., Theodorakopoulos G.: Partial Fake Speech Attacks in the Real World Using Deepfake Audio. Jour-nal of Cybersecurity and Privacy. 2025; 5(1):6. https://doi.org/10.3390/jcp5010006.
Paul O.: Deepfakes Generated by Generative Adversarial Networks. Honors College Theses. 671, (2021). digitalcommons.georgiasouthern.edu/honors-theses/671.
Majchrzak A.: Rosyjska dezinformacja i wykorzystanie obrazów generowanych przez sztuczną inteligencję (deepfake) w pierwszym roku inwazji na Ukrainę. Media Biznes Kultura, pp. 73-86, (2023). Doi: 10.4467/25442554.MBK.23.005.18028.
Allen D.: Deepfake Fight: AI-Powered Disinformation and Perfidy Under the Geneva Conventions. Notre Dame Journal on Emerging Technologies, 2022, http://dx.doi.org/10.2139/ssrn.3958426.
European Parliament: Directorate-General for Parliamentary Research Services, Huijstee M., Boheemen P., Das D., Nierling L., Jahnel J., Karaboga M., Fatun M.: Tackling deepfakes in European policy. European Parliament, 2021, https://data.europa.eu/doi/10.2861/325063.
Regulation (EU) 2024/1689: Artificial Intelligence Act. Official Journal of the European Union. (2024, July 12), https://artificialintelligenceact.eu.
CISA: Risk in Focus: Generative A.I. and the 2024 Election Cycle. (2024). https://www.cisa.gov/resources-tools/resources/risk-focus-generative-ai-and-2024-election-cycle.
Soudy A. H., Sayed O., Tag-Elser H., Ragab R., Mohsen S., Mostafa T., Abohany A., Slim S.: Deepfake detection using convolutional vision transformers and convolutional neural networks. Neural Comput & Ap-plic vol. 36, pp. 19759–19775 (2024). https://doi.org/10.1007/s00521-024-10181-7.
Pinhasov B., Lapid R., Ohayon R., Sipper M., Aperstein Y.: XAI-Based Detection of Adversarial Attacks on Deepfake Detectors, (2024), https://arxiv.org/abs/2403.02955.
Doloriel, C. T., Cheung, N. M.: Frequency masking for universal deepfake detection. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 13466-13470, IEEE.
Bao H., Zhang X., Wang Q., Liang K., Wang Z., Ji S., Chen W.: Pluggable watermarking of deepfake models for deepfake detection. In Proceedings of the Thirty-Third International Joint Conference on Artificial Intel-ligence (IJCAI '24). Article 37, pp. 331–339. https://doi.org/10.24963/ijcai.2024/37.
Zhang Y., Gao W., Miao C., Luo M., Li J., Deng W., Chu Q: Inclusion 2024 Global Multimedia Deepfake Detection: Towards Multi-dimensional Facial Forgery Detection. arXiv preprint arXiv:2412.20833.
Say T, Alkan M, Kocak A.: Advancing GAN Deepfake Detection: Mixed Datasets and Comprehensive Ar-tifact Analysis. Applied Sciences. 2025; 15(2):923. https://doi.org/10.3390/app15020923.
Tolosana R., Vera-Rodriguez R., Fierrez J., Morales A., Ortega-Garcia J.: DeepFakes and Beyond: A Survey of Face Manipulation and Fake Detection, (2020), doi: 10.48550/arXiv.2001.00179.
Heidari A., Navimipour N., Dag H., Unal M.: Deepfake detection using deep learning methods: A systematic and comprehensive review. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery. (2023), vol 14, doi: 10.1002/widm.1520.
Frąszczak D.: RPaSDT - Rumor Propagation and Source Detection Toolkit. SoftwareX, 2022. Doi: 10.1016/j.softx.2022.100988.
Frąszczak D.: Detecting rumor outbreaks in online social networks. Soc. Netw. Anal. Min. 13, 91 (2023). https://doi.org/10.1007/s13278-023-01092-x.
Frąszczak D., Frąszczak E.: NetCenLib: A comprehensive python library for network centrality analysis and evaluation. SoftwareX, 2024. Doi: 10.1016/j.softx.2024.101699.
Zhang J., Bu H., Wen H., Liu Y., Fei H., Xi R., Li L., Yang Y., Zhu H., Meng D.: When llms meet cyberse-curity: A systematic literature review. Cybersecurity, vol. 8(1), pp. 1-41, (2025).
Schmitt M., Flechais I.: Digital deception: generative artificial intelligence in social engineering and phish-ing. Artif Intell Rev vol. 57, 324, (2024). https://doi.org/10.1007/s10462-024-10973-2.
Kumar S., Menezes A., Giri S., Kotikela S.: What The Phish! Effects of AI on Phishing Attacks and Defense. International Conference on AI Research, vol. 4, pp. 218-226, (2024), doi: 10.34190/icair.4.1.3224.
Heiding F., Schneier B., Vishwanath A., Bernstein J., Park P. S.: Devising and Detecting Phishing Emails Using Large Language Models. IEEE Access, vol. 12, pp. 42131-42146, 2024, doi: 10.1109/AC-CESS.2024.3375882.
Fairbanks J., Serra E.: Generating Phishing Attacks and Novel Detection Algorithms in the Era of Large Language Models. 2024 IEEE International Conference on Big Data (BigData), Washington, DC, USA, 2024, pp. 2314-2319, doi: 10.1109/BigData62323.2024.10825007.
Koide T., Nakano H., Chiba D.: ChatPhishDetector: Detecting Phishing Sites Using Large Language Models. IEEE Access, vol. 12, pp. 154381-154400, (2024), doi: 10.1109/ACCESS.2024.3483905.
Misra K., Rayz J. T.: LMs go Phishing: Adapting Pre-trained Language Models to Detect Phishing Emails. 2022 IEEE/WIC/ACM International Joint Conference on Web Intelligence and Intelligent Agent Technology (WI-IAT), Niagara Falls, ON, Canada, 2022, pp. 135-142, doi: 10.1109/WI-IAT55865.2022.00028.
Çolhak F., Ecevit M. İ., Uçar B. E., Creutzburg R., Dağ H.: Phishing website detection through multi-model analysis of html content. (2024), doi: https://arxiv.org/abs/2401.04820.
Bauskar S. R., Madhavaram C. R., Galla E. P., Sunkara J. R., Gollangi H. K.: AI-Driven Phishing Email Detection: Leveraging Big Data Analytics for Enhanced Cybersecurity. Library Progress International, vol. 44(3), pp. 7211-7224, 2024.
Chirra B. R.: Intelligent Phishing Mitigation: Leveraging AI for Enhanced Email Security in Corporate En-vironments. International Journal of Advanced Engineering Technologies and Innovations, vol. 1(2), pp. 178-200, (2021), doi: https://ijaeti.com/index.php/Journal/article/view/736.
Ansari M. F., Sharma P., Dash B.: Prevention of Phishing Attacks Using AI-Based Cybersecurity Awareness Training. International Journal of Smart Sensor and Adhoc Network, vol. 3, pp. 61-72, (2022), doi: 10.47893/IJSSAN.2022.1221.
Europol & UNICRI: Malicious uses and abuses of artificial intelligence. Europol, (2020). https://www.euro-pol.europa.eu/cms/sites/default/files/documents/malicious_uses_and_abuses_of_artificial_intelligence_eu-ropol.pdf
Vassilev A, Oprea A, Fordyce A, Anderson H, Davies X, Hamin M: Adversarial Machine Learning: A Tax-onomy and Terminology of Attacks and Mitigations. (National Institute of Standards and Technology, Gaithersburg, MD) NIST Trustworthy and Responsible AI, NIST AI 100-2e2025, (2025). https://doi.org/10.6028/NIST.AI.100-2e2025
Kosinski M. IBM, (2024). https://www.ibm.com/think/insights/defend-against-ai-malware
HYAS: BlackMamba: Using AI to Generate Polymorphic Malware. 31 July 2023. https://www.hyas.com/blog/blackmamba-using-ai-to-generate-polymorphic-malware
HYAS: EyeSpy Proof-of-Concept. 01 August 2023. https://www.hyas.com/blog/eyespy-proof-of-concept
Cohen S., Bitton R., Nassi B.: Here comes the AI worm: Unleashing zero-click worms that target GenAI-powered applications. (2024), https://doi.org/10.48550/arXiv.2403.02817.
Beckerich M., Plein L., Coronado S.: Ratgpt: Turning online llms into proxies for malware attacks. (2023), arXiv preprint arXiv:2308.09183.
Zhong F., Cheng X., Yu D., Gong B., Song S., Yu J.: MalFox: Camouflaged adversarial malware example generation based on conv-GANs against black-box detectors. IEEE Transactions on Computers, vol. 73(4), pp. 980-993.
Janjeva A., Gausen A., Mercer S., Sippy T.: Evaluating Malicious Generative AI Capabilities: Understanding inflection points in risk. CETaS Briefing Papers (July 2024). https://cetas.turing.ac.uk/publications/evaluat-ing-malicious-generative-ai-capabilities
Papernot N., McDaniel P., Goodfellow I., Jha S., Celik Z. B., Swami A.: Practical black-box attacks against machine learning. Proceedings of ACM ASIACCS, 2017.
Tramer F., Zhang F., Juels A., Reiter M. K., Ristenpart T.: Stealing machine learning models via prediction apis. USENIX Security Symposium, 2016.
Lee J., Han S., Lee S.: Model Stealing Defense against Exploiting Information Leak through the Interpreta-tion of Deep Neural Nets. In IJCAI, pp. 710-716, 2022.
Hu C., Hu Y. -H. F.: Data Poisoning on Deep Learning Models. 2020 International Conference on Compu-tational Science and Computational Intelligence (CSCI), Las Vegas, NV, USA, 2020, pp. 628-632, doi: 10.1109/CSCI51800.2020.00111.
Shao Y., Lin X., Luo H., Hou C., Xiong G., Yu J., Shi J.: POISONCRAFT: Practical Poisoning of Retrieval-Augmented Generation for Large Language Models. (2025), arXiv preprint arXiv:2505.06579.
Gao Y.: Cyber Attacks and Defense: AI-Driven Approaches and Techniques. Academic Journal of Compu-ting & Information Science (2024), vol. 7(7), pp. 41-46. https://doi.org/10.25236/AJCIS.2024.070706.
Gaddam N.: AI-enhanced honeypots for advanced cyber deception strategies. QIT Press - International Jour-nal of Cyber Security Research and Development (QITP-IJCSRD), vol. 5(1), pp. 9–19, (2025). https://doi.org/10.63374/QITP-IJCSRD_05_01_002.
Liu P., Tao Q., Zhou J. T.: Evolving from Single-modal to Multi-modal Facial Deepfake Detection: Progress and Challenges. (2024), arXiv preprint arXiv:2406.06965.
ENISA: Multilayer framework for good cybersecurity practices for AI. European Union Agency for Cyber-security. (2023). https://www.enisa.europa.eu/sites/default/files/publications/Multilayer%20Frame-work%20for%20Good%20Cybersecurity%20Practices%20for%20AI.pdf.
SlashNext Security: The state of phishing: 2024 mid-year assessment. (2024). https://slashnext.com/wp-con-tent/uploads/2024/05/SlashNext-The-State-of-Phishing-24-Midyear-Report.pdf.