--- title: Transforming Phishing Campaigns with Advanced Language Models - Topic Discovery description: Jericho finds an interesting and potentially innovative gap in the form of pretext recommendations via topic discovery. image: //www.jerichosecurity.com/hubfs/topic%20discover.webp --- [![Jericho Security logo](https://www.jerichosecurity.com/hubfs/assets/images/Jericho-Security.svg)](https://www.jerichosecurity.com/) [![Jericho Security logo](https://www.jerichosecurity.com/hubfs/assets/images/Jericho-Security.svg)](https://www.jerichosecurity.com/) - Why Jericho Why choose Jericho Security? See what sets Jericho Security apart from the rest. [About Us Our Mission](https://www.jerichosecurity.com/about) [Customers What customers say](https://www.jerichosecurity.com/customers) - Plans - Plans - [Lite](https://www.jerichosecurity.com/lite) - [Plus](https://www.jerichosecurity.com/plus) - [Premium](https://www.jerichosecurity.com/premium) - [Free Trial](https://www.jerichosecurity.com/7-day-free-trial) - Solutions See how Jericho Security can help keep your team safe. We equip individuals with the knowledge and skills needed to identify, prevent, and respond to cyber threats. Platform [Cybersecurity Dashboard Manage Your Security](https://www.jerichosecurity.com/platform/cybersecurity-dashboard) [Phishing Simulator Train Your Employees](https://www.jerichosecurity.com/platform/phishing-simulator) [Performance Analytics Gain Key Insights](https://www.jerichosecurity.com/platform/employee-performance-analytics) [Customized Training Tailored to Your Organization](https://www.jerichosecurity.com/platform/training-content) Solutions [Cybersecurity Awareness Training Train Your Team](https://www.jerichosecurity.com/solutions/cybersecurity-awareness-training) [Anti-Phishing Training Protect Your Business](https://www.jerichosecurity.com/solutions/anti-phishing-training) [Enterprise Cybersecurity Training Defend Against Threats](https://www.jerichosecurity.com/solutions/enterprise-cybersecurity-training) [SMB Cybersecurity Training Empower Your Team](https://www.jerichosecurity.com/solutions/smb-cybersecurity-training) Industries [Retail Keep retail threats away](https://www.jerichosecurity.com/industries/retail) [E-Commerce Protect your E-Com Business](https://www.jerichosecurity.com/industries/e-commerce) [Technology Prevent sensitive breaches](https://www.jerichosecurity.com/industries/technology) [Financial Services Safeguard your assets](https://www.jerichosecurity.com/industries/financial-services) [Healthcare Secure sensitive information](https://www.jerichosecurity.com/industries/healthcare) [Manufacturing Keep your operation running](https://www.jerichosecurity.com/industries/manufacturing) [Energy Prevent the next breach](https://www.jerichosecurity.com/industries/energy) [Federal Government Prevent largescale attacks](https://www.jerichosecurity.com/industries/federal-government) [State & Local Government Protect local officials](https://www.jerichosecurity.com/industries/state-local-government) [Higher Education Keep staff & students safe](https://www.jerichosecurity.com/industries/higher-education) - Resources Jericho Security resources hub. Find everything you need to defend against emerging Generative AI threats. [News Latest news headlines](https://www.jerichosecurity.com/resources/tag/news) [Literature White papers & solution briefs](https://www.jerichosecurity.com/resources/tag/white-papers-solution-briefs) [Case Studies Jericho success stories](https://www.jerichosecurity.com/resources/tag/case-study) [Events Upcoming security events](https://www.jerichosecurity.com/resources/tag/events) [Webinars Upcoming & On-Demand](https://www.jerichosecurity.com/resources/tag/webinars) [Podcast Leading CISO Discussions](https://www.jerichosecurity.com/ciso-diaries) [Community Spotlight To recognize and appreciate](https://www.jerichosecurity.com/resources/tag/community-spotlight) [Cybersecurity Glossary Cyber-Related Definitions](https://www.jerichosecurity.com/glossary) [Support Advice and Answers](https://support.jerichosecurity.com/) [View All Resources](https://www.jerichosecurity.com/resources) - [Blog](https://www.jerichosecurity.com/blog) - [Partners](https://www.jerichosecurity.com/partner) - [Sign In](https://app.jerichosecurity.com/users/sign_in) - [Free Trial](https://www.jerichosecurity.com/7-day-free-trial) # Transforming Phishing Campaigns with Advanced Language Models - Topic Discovery Written by ![Jericho Security Contributor](https://www.jerichosecurity.com/hubfs/assets/images/Jericho-Security-Thumbnail.png) Jericho Security Contributor Published on May 31, 2024 ![](https://www.jerichosecurity.com/hubfs/topic%20discover.webp) *We previously discussed* [*semantic similarity*](https://www.jerichosecurity.com/blog/how-jericho-security-provides-realistic-phishing-simulations-through-semantic-similarity) *and how Jericho tracks metrics such as diversity and realism of our generated emails using that concept. *   **Introduction**   **Semantic similarity** is calculated from a vector representation of whole documents, which is instrumental for a variety of natural language processing tasks. Today, we will show that a natural extension of the semantic similarity workflow is doing **Topic Discovery** on a corpus of emails.   Topic discovery has traditionally been performed using a bag-of-words assumption. This assumption means that documents are treated as a collection of individual words, and contextual information is mostly assumed not to play a role. That is why common techniques such as latent semantic analysis (LSA) or latent Dirichlet allocation (LDA) require a transformation of the documents into what are functionally word co-occurrence matrices.    From there, LSA uses singular value decomposition and cosine similarities to determine document similarity, while LDA trains a Bayesian network with a Dirichlet distribution as its posterior to infer topics. While the assumption is relatively naive, the methodologies work reasonably well to generate topics, and many research papers have shown this to work for large text corpora.  For example, in the applied stats/political science space, [Martin and McCrain (2018)](https://web.stanford.edu/~gjmartin/papers/localnews_final_apsr.pdf)used LDA to parse and qualitatively pick out 15 topics that were characteristic of local and national news.   **How Jericho improves pretext automation** As Jericho researched how to improve the experience of our users when crafting campaigns, we found an interesting and potentially innovative gap in the form of **pretext recommendations** via topic discovery.    In a **traditional** security awareness training program, security analysts write templates of phishing emails caught in the security gateway or found through trawling the internet. This process is t**ime-intensive** and **costly to do in-house**, and if given over to traditional third-party phishing simulation providers would be difficult to perform at scale.    Our thesis is that this process could be automated by **using topic discovery to label clusters of emails in an unsupervised fashion.** The topic labels can be recommended as commonly occurring pretexts at various levels such ascompany-specific, industry-specific, and even global.   ![](https://www.jerichosecurity.com/hs-fs/hubfs/image-png-May-31-2024-09-31-11-1525-PM.png?width=698&height=346&name=image-png-May-31-2024-09-31-11-1525-PM.png)   From the semantic similarity analysis (shown in Figure 1), we showed that there was reasonable clustering that could be obtained based on semantic embeddings of each document.    The next step is finding representations of these clusters, ideally using the words present in the document. As explained above, this is usually done with a bag-of-words assumption and using word-count vectorizers to find similar documents.   However, since we already have clusters, we can instead use a simple metric of term-frequency inverse-document frequency (TF-IDF) at the cluster level to find significant keywords in the clusters. This method was proposed by Grootendorst (2022), and he also started an open-source library called BERTopic (See the algorithm Grootendorst proposed [here](https://maartengr.github.io/BERTopic/algorithm/algorithm.html)) that abstracts much of this process, allowing quick and easy experimentation of the topic discovery pipeline.    ![](https://www.jerichosecurity.com/hs-fs/hubfs/image-png-May-31-2024-09-31-52-8247-PM.png?width=791&height=617&name=image-png-May-31-2024-09-31-52-8247-PM.png)   Following this basic workflow of embedding the documents, projecting the embeddings to a lower dimensional space and clustering them, and finally using clustered-TF-IDF (c-TF-IDF) on the word-count vectorized documents, we show in Table 1 a simple run of the above using the seed dataset described in the previous post results in the topics.   Some topics seem reasonable, such as row 5 related to phishing emails about Citibank security issues. However, on the whole, they are not readable from a natural language perspective and require human parsing to make sense of the topic.    **Integrating advanced language models to improve topic readability **   With LLM technology, we may not require a human to do the parsing. If given sufficient information about the document cluster and keywords, an LLM may generate a good representation of the topic, making it **human-readable** and useful for future LLM downstream tasks.    BERTopic provides an easy-to-use hook at the end of the pipeline to integrate it with various LLMs, and we hooked our pipeline up to a Mistral model we have running. Using a custom prompt to inject the document and keyword information into Mistral, we were able to come up with better representations of the documents while keeping the process relatively fast.    These results can be seen in Table 2, where the names of the topics now are much more human-readable and could even be passed into our email phishing campaign generation process.    ![](https://www.jerichosecurity.com/hs-fs/hubfs/image-png-May-31-2024-09-32-15-2830-PM.png?width=782&height=600&name=image-png-May-31-2024-09-32-15-2830-PM.png)   **Conclusion**   There is still room for exploration, for example, experimenting with larger LLMs or different prompt techniques and examining those effects on the final topic representation. Overall, this is a **significant step in pretext automation/recommendation** and **improving user experience** during the phishing campaign generation process. ##### Categories - [News (45)](https://www.jerichosecurity.com/blog/tag/news) - [Thought Leadership (15)](https://www.jerichosecurity.com/blog/tag/thought-leadership) - [Phishing (14)](https://www.jerichosecurity.com/blog/tag/phishing) - [Product Updates (10)](https://www.jerichosecurity.com/blog/tag/product-updates) - [Announcements (8)](https://www.jerichosecurity.com/blog/tag/announcements) - [Reports (5)](https://www.jerichosecurity.com/blog/tag/reports) - [Case Study (4)](https://www.jerichosecurity.com/blog/tag/case-study) - [Community Spotlight (3)](https://www.jerichosecurity.com/blog/tag/community-spotlight) - [Events (3)](https://www.jerichosecurity.com/blog/tag/events) - [White Papers & Solution Briefs (1)](https://www.jerichosecurity.com/blog/tag/white-papers-solution-briefs) #### Subscribe to our newsletter Get the latest updates on phishing trends and generative AI. Subscribe ##### Popular Posts Platform - [Cybersecurity Dashboard](https://www.jerichosecurity.com/platform/cybersecurity-dashboard) - [Phishing Simulator](https://www.jerichosecurity.com/platform/phishing-simulator) - [Performance Analytics](https://www.jerichosecurity.com/platform/employee-performance-analytics) - [Customized Training](https://www.jerichosecurity.com/platform/training-content) Solutions - [Anti-Phishing Training](https://www.jerichosecurity.com/solutions/anti-phishing-training) - [SMB Cybersecurity Training](https://www.jerichosecurity.com/solutions/smb-cybersecurity-training) - [Enterprise Cybersecurity Training](https://www.jerichosecurity.com/solutions/enterprise-cybersecurity-training) - [Cybersecurity Awareness Training](https://www.jerichosecurity.com/solutions/cybersecurity-awareness-training) Industries - [Retail](https://www.jerichosecurity.com/industries/retail) - [E-Commerce](https://www.jerichosecurity.com/industries/e-commerce) - [Technology](https://www.jerichosecurity.com/industries/technology) - [Financial Services](https://www.jerichosecurity.com/industries/financial-services) - [Healthcare](https://www.jerichosecurity.com/industries/healthcare) - [Manufacturing](https://www.jerichosecurity.com/industries/manufacturing) - [Energy](https://www.jerichosecurity.com/industries/energy) - [Federal Government](https://www.jerichosecurity.com/industries/federal-government) - [State & Local Government](https://www.jerichosecurity.com/industries/state-local-government) - [Higher Education](https://www.jerichosecurity.com/industries/higher-education) Resources - [Cybersecurity Glossary](https://www.jerichosecurity.com/glossary) - [Support](https://support.jerichosecurity.com/) - [Blog](https://www.jerichosecurity.com/blog) - [News](https://www.jerichosecurity.com/resources/tag/news) - [Literature](https://www.jerichosecurity.com/resources/tag/white-papers-solution-briefs) - [Case Studies](https://www.jerichosecurity.com/resources/tag/case-study) - [Events](https://www.jerichosecurity.com/resources/tag/events) - [Webinars](https://www.jerichosecurity.com/resources/tag/webinars) - [Community Spotlight](https://www.jerichosecurity.com/resources/tag/community-spotlight) Company - [About Us](https://www.jerichosecurity.com/about) - [Customers](https://www.jerichosecurity.com/customers) - [Partners](https://www.jerichosecurity.com/partner) - [Careers](https://jericho-security.breezy.hr/) - [Sign In](https://app.jerichosecurity.com/users/sign_in) - [Get a Demo](https://www.jerichosecurity.com/demo) - [Privacy Policy](https://www.jerichosecurity.com/privacy-policy) - [Terms of Service](https://www.jerichosecurity.com/terms-of-service) [![Jericho Security footer logo](https://www.jerichosecurity.com/hubfs/assets/images/bootstrap/jericho-logo.svg)](https://www.jerichosecurity.com/) AI-Powered Cyber Security Training for Employees. ![AICPA SOC](https://www.jerichosecurity.com/hs-fs/hubfs/assets/images/bootstrap/AICPA-SOC-Logo.png?width=54&height=54&name=AICPA-SOC-Logo.png) ![SOC 2 TYPE 1](https://www.jerichosecurity.com/hs-fs/hubfs/assets/images/bootstrap/SOC-2-TYPE-1.png?width=54&height=54&name=SOC-2-TYPE-1.png) ![SOC 2 TYPE 2](https://www.jerichosecurity.com/hs-fs/hubfs/assets/images/bootstrap/SOC-2-TYPE-2.png?width=54&height=54&name=SOC-2-TYPE-2.png) Toggle theme - Light - Dark - Auto Copyright © Jericho Security, Inc. ```json { "@context" : "https://schema.org", "@type" : "BlogPosting", "author" : { "@type" : "Person", "name" : "Jericho Security Contributor", "url" : "http://www.jerichosecurity.com/blog/author/jericho-security-contributor" }, "dateModified" : "2024-05-31T21:40:12.319Z", "datePublished" : "2024-05-31T21:27:08.000Z", "headline" : "Transforming Phishing Campaigns with Advanced Language Models - Topic Discovery", "image" : [ "//www.jerichosecurity.com/hubfs/topic%20discover.webp" ], "mainEntityOfPage" : { "@id" : "http://www.jerichosecurity.com/blog/transforming-phishing-campaigns-with-advanced-language-models-topic-discovery", "@type" : "WebPage" }, "publisher" : { "@type" : "Organization", "logo" : { "@type" : "ImageObject", "url" : "//www.jerichosecurity.com/hubfs/assets/images/Jericho-Security.svg" }, "name" : "Jericho Security" } } ```