--- title: How Jericho Security Provides Realistic Phishing Simulations Through Semantic Similarity description: At Jericho Security, we strive to prepare people against the next generation of phishing attacks by providing realistic generative phishing simulations. image: //www.jerichosecurity.com/hubfs/Untitled%20presentation%20(1).png --- [![Jericho Security logo](https://www.jerichosecurity.com/hubfs/assets/images/Jericho-Security.svg)](https://www.jerichosecurity.com/) [![Jericho Security logo](https://www.jerichosecurity.com/hubfs/assets/images/Jericho-Security.svg)](https://www.jerichosecurity.com/) - Why Jericho Why choose Jericho Security? See what sets Jericho Security apart from the rest. [About Us Our Mission](https://www.jerichosecurity.com/about) [Customers What customers say](https://www.jerichosecurity.com/customers) - Plans - Plans - [Lite](https://www.jerichosecurity.com/lite) - [Plus](https://www.jerichosecurity.com/plus) - [Premium](https://www.jerichosecurity.com/premium) - [Free Trial](https://www.jerichosecurity.com/7-day-free-trial) - Solutions See how Jericho Security can help keep your team safe. We equip individuals with the knowledge and skills needed to identify, prevent, and respond to cyber threats. Platform [Cybersecurity Dashboard Manage Your Security](https://www.jerichosecurity.com/platform/cybersecurity-dashboard) [Phishing Simulator Train Your Employees](https://www.jerichosecurity.com/platform/phishing-simulator) [Performance Analytics Gain Key Insights](https://www.jerichosecurity.com/platform/employee-performance-analytics) [Customized Training Tailored to Your Organization](https://www.jerichosecurity.com/platform/training-content) Solutions [Cybersecurity Awareness Training Train Your Team](https://www.jerichosecurity.com/solutions/cybersecurity-awareness-training) [Anti-Phishing Training Protect Your Business](https://www.jerichosecurity.com/solutions/anti-phishing-training) [Enterprise Cybersecurity Training Defend Against Threats](https://www.jerichosecurity.com/solutions/enterprise-cybersecurity-training) [SMB Cybersecurity Training Empower Your Team](https://www.jerichosecurity.com/solutions/smb-cybersecurity-training) Industries [Retail Keep retail threats away](https://www.jerichosecurity.com/industries/retail) [E-Commerce Protect your E-Com Business](https://www.jerichosecurity.com/industries/e-commerce) [Technology Prevent sensitive breaches](https://www.jerichosecurity.com/industries/technology) [Financial Services Safeguard your assets](https://www.jerichosecurity.com/industries/financial-services) [Healthcare Secure sensitive information](https://www.jerichosecurity.com/industries/healthcare) [Manufacturing Keep your operation running](https://www.jerichosecurity.com/industries/manufacturing) [Energy Prevent the next breach](https://www.jerichosecurity.com/industries/energy) [Federal Government Prevent largescale attacks](https://www.jerichosecurity.com/industries/federal-government) [State & Local Government Protect local officials](https://www.jerichosecurity.com/industries/state-local-government) [Higher Education Keep staff & students safe](https://www.jerichosecurity.com/industries/higher-education) - Resources Jericho Security resources hub. Find everything you need to defend against emerging Generative AI threats. [News Latest news headlines](https://www.jerichosecurity.com/resources/tag/news) [Literature White papers & solution briefs](https://www.jerichosecurity.com/resources/tag/white-papers-solution-briefs) [Case Studies Jericho success stories](https://www.jerichosecurity.com/resources/tag/case-study) [Events Upcoming security events](https://www.jerichosecurity.com/resources/tag/events) [Webinars Upcoming & On-Demand](https://www.jerichosecurity.com/resources/tag/webinars) [Podcast Leading CISO Discussions](https://www.jerichosecurity.com/ciso-diaries) [Community Spotlight To recognize and appreciate](https://www.jerichosecurity.com/resources/tag/community-spotlight) [Cybersecurity Glossary Cyber-Related Definitions](https://www.jerichosecurity.com/glossary) [Support Advice and Answers](https://support.jerichosecurity.com/) [View All Resources](https://www.jerichosecurity.com/resources) - [Blog](https://www.jerichosecurity.com/blog) - [Partners](https://www.jerichosecurity.com/partner) - [Sign In](https://app.jerichosecurity.com/users/sign_in) - [Free Trial](https://www.jerichosecurity.com/7-day-free-trial) # How Jericho Security Provides Realistic Phishing Simulations Through Semantic Similarity Written by ![Jericho Security Contributor](https://www.jerichosecurity.com/hubfs/assets/images/Jericho-Security-Thumbnail.png) Jericho Security Contributor Published on April 25, 2024 ![](https://www.jerichosecurity.com/hubfs/Untitled%20presentation%20(1).png) **Background** At Jericho Security, we strive to prepare people against the next generation of phishing attacks by providing generative phishing simulations similar to real-life phishing and spear phishing scenarios. To that end, we have always prioritized having a wide breadth of realistic phishing simulations - that mimic real-life situations to an extraordinary degree.    To track our performance, however, we need **measurable metrics** to determine both the breadth and realism of the generations.    This report is meant to shed light on our data science research efforts in conceptualizing, operationalizing, and measuring those metrics.    **How did we decide on Semantic Similarity?** Diving deep into what it means for our simulations to have breadth and realism, we decided that our simulations would on average be different from each other whereas an individual simulation would be similar to a real phishing email. Again, this required a concept of how much a piece of text overlaps with another - fortunately, machine learning literature already has fairly grounded operationalizations for this concept.    After experimenting with various algorithms and scores, we landed on using **semantic similarity**. We chose this for a couple of reasons, namely because algorithms such as lexical matching would rely on explicit n-gram overlapping which is much more prone to shift over time, while the underlying semantics might not. There are also technical reasons for this choice, for example being able to easily parallelize the embedding computation, which is also reusable for other tasks.    **Calculation of Semantic Similarity** To calculate the semantic similarity between two emails, we rely on SentenceBERT to first embed the whole text into an embedding space. From there, **breadth** is calculated as the **average pairwise cosine similarity** between a generated email with all other generated emails while **realism** is calculated as the **max pairwise cosine similarity** between a generated email with all real phishing emails.    Currently, we are using a **seed dataset** from various open-source phishing dumps (SpamAsssassin, the Enron dumps, etc.) as our representation of real phishing emails, and we are actively expanding this dataset to include more up-to-date phishing emails.   The above process creates a distribution of the diversity and realism scores for this particular sample of generated and real emails. Figure (1) shows how the distribution of the diversity scores of a random sample of 200 generated emails with varying inputs related to the target information, pretext, sender information, and attack type compares to a distribution of diversity scores from real phishing emails.    ![](https://www.jerichosecurity.com/hs-fs/hubfs/image-png-2.png?width=539&height=492&name=image-png-2.png)   Ideally, the generated phishing emails should be trending toward the lower end of the average cosine similarity range, as lower cosine similarity values mean the pairwise comparison is more dissimilar on average.    For the **realism score distribution**, there could be a better point of comparison as we only have one set of emails to compare against our real phishing email dataset. That being said, Figure (2) highlights the distribution of these scores; over time we want these scores to **trend up** as any given generated phishing email has a corollary with a real phishing email.    ![](https://www.jerichosecurity.com/hs-fs/hubfs/image-png-3.png?width=482&height=422&name=image-png-3.png)   **Dimensionality Reduction Based Visualization** Another informative way of visualizing these embeddings is through dimensionality reduction-based visualizations. T-SNE and UMAP are common techniques of nonlinear dimensionality reduction that also uses clustering as an added benefit.    We elected to use UMAP in this case due to allowing more modifications of hyperparameters. Figure (3) shows the results of running both generated and real phishing emails through the same UMAP transform.    ![](https://www.jerichosecurity.com/hs-fs/hubfs/image-png-4.png?width=586&height=358&name=image-png-4.png)   The results help explain the trend we saw above in the raw distributions of diversity scores. Specifically, the generated emails tend to be a **subsetted cluster** within the phishing emails. This can be due to a variety of reasons, one of them being that pretexts not overlapping as much as they should. For example, one of the main clusters in the UMAP visualization is the prototypical Nigerian-prince-type phish, which we haven’t included as a default pretext in our simulation.    **Conclusion** Ultimately, we recognize that the generated emails are meant to be more akin to spear phishing, which is a niche that won’t be able to be captured by public phishing email dumps. As we grow this dataset and our functionalities related to generation variation, we will continue monitoring these metrics and visuals to make sure we are moving toward our end goal of making increasingly **diverse** and **realistic phishing simulations**. ##### Categories - [News (45)](https://www.jerichosecurity.com/blog/tag/news) - [Thought Leadership (15)](https://www.jerichosecurity.com/blog/tag/thought-leadership) - [Phishing (14)](https://www.jerichosecurity.com/blog/tag/phishing) - [Product Updates (10)](https://www.jerichosecurity.com/blog/tag/product-updates) - [Announcements (8)](https://www.jerichosecurity.com/blog/tag/announcements) - [Reports (5)](https://www.jerichosecurity.com/blog/tag/reports) - [Case Study (4)](https://www.jerichosecurity.com/blog/tag/case-study) - [Community Spotlight (3)](https://www.jerichosecurity.com/blog/tag/community-spotlight) - [Events (3)](https://www.jerichosecurity.com/blog/tag/events) - [White Papers & Solution Briefs (1)](https://www.jerichosecurity.com/blog/tag/white-papers-solution-briefs) #### Subscribe to our newsletter Get the latest updates on phishing trends and generative AI. Subscribe ##### Popular Posts Platform - [Cybersecurity Dashboard](https://www.jerichosecurity.com/platform/cybersecurity-dashboard) - [Phishing Simulator](https://www.jerichosecurity.com/platform/phishing-simulator) - [Performance Analytics](https://www.jerichosecurity.com/platform/employee-performance-analytics) - [Customized Training](https://www.jerichosecurity.com/platform/training-content) Solutions - [Anti-Phishing Training](https://www.jerichosecurity.com/solutions/anti-phishing-training) - [SMB Cybersecurity Training](https://www.jerichosecurity.com/solutions/smb-cybersecurity-training) - [Enterprise Cybersecurity Training](https://www.jerichosecurity.com/solutions/enterprise-cybersecurity-training) - [Cybersecurity Awareness Training](https://www.jerichosecurity.com/solutions/cybersecurity-awareness-training) Industries - [Retail](https://www.jerichosecurity.com/industries/retail) - [E-Commerce](https://www.jerichosecurity.com/industries/e-commerce) - [Technology](https://www.jerichosecurity.com/industries/technology) - [Financial Services](https://www.jerichosecurity.com/industries/financial-services) - [Healthcare](https://www.jerichosecurity.com/industries/healthcare) - [Manufacturing](https://www.jerichosecurity.com/industries/manufacturing) - [Energy](https://www.jerichosecurity.com/industries/energy) - [Federal Government](https://www.jerichosecurity.com/industries/federal-government) - [State & Local Government](https://www.jerichosecurity.com/industries/state-local-government) - [Higher Education](https://www.jerichosecurity.com/industries/higher-education) Resources - [Cybersecurity Glossary](https://www.jerichosecurity.com/glossary) - [Support](https://support.jerichosecurity.com/) - [Blog](https://www.jerichosecurity.com/blog) - [News](https://www.jerichosecurity.com/resources/tag/news) - [Literature](https://www.jerichosecurity.com/resources/tag/white-papers-solution-briefs) - [Case Studies](https://www.jerichosecurity.com/resources/tag/case-study) - [Events](https://www.jerichosecurity.com/resources/tag/events) - [Webinars](https://www.jerichosecurity.com/resources/tag/webinars) - [Community Spotlight](https://www.jerichosecurity.com/resources/tag/community-spotlight) Company - [About Us](https://www.jerichosecurity.com/about) - [Customers](https://www.jerichosecurity.com/customers) - [Partners](https://www.jerichosecurity.com/partner) - [Careers](https://jericho-security.breezy.hr/) - [Sign In](https://app.jerichosecurity.com/users/sign_in) - [Get a Demo](https://www.jerichosecurity.com/demo) - [Privacy Policy](https://www.jerichosecurity.com/privacy-policy) - [Terms of Service](https://www.jerichosecurity.com/terms-of-service) [![Jericho Security footer logo](https://www.jerichosecurity.com/hubfs/assets/images/bootstrap/jericho-logo.svg)](https://www.jerichosecurity.com/) AI-Powered Cyber Security Training for Employees. ![AICPA SOC](https://www.jerichosecurity.com/hs-fs/hubfs/assets/images/bootstrap/AICPA-SOC-Logo.png?width=54&height=54&name=AICPA-SOC-Logo.png) ![SOC 2 TYPE 1](https://www.jerichosecurity.com/hs-fs/hubfs/assets/images/bootstrap/SOC-2-TYPE-1.png?width=54&height=54&name=SOC-2-TYPE-1.png) ![SOC 2 TYPE 2](https://www.jerichosecurity.com/hs-fs/hubfs/assets/images/bootstrap/SOC-2-TYPE-2.png?width=54&height=54&name=SOC-2-TYPE-2.png) Toggle theme - Light - Dark - Auto Copyright © Jericho Security, Inc. ```json { "@context" : "https://schema.org", "@type" : "BlogPosting", "author" : { "@type" : "Person", "name" : "Jericho Security Contributor", "url" : "http://www.jerichosecurity.com/blog/author/jericho-security-contributor" }, "dateModified" : "2024-04-25T22:41:27.142Z", "datePublished" : "2024-04-25T22:41:27.000Z", "headline" : "How Jericho Security Provides Realistic Phishing Simulations Through Semantic Similarity", "image" : [ "//www.jerichosecurity.com/hubfs/Untitled%20presentation%20(1).png" ], "mainEntityOfPage" : { "@id" : "http://www.jerichosecurity.com/blog/how-jericho-security-provides-realistic-phishing-simulations-through-semantic-similarity", "@type" : "WebPage" }, "publisher" : { "@type" : "Organization", "logo" : { "@type" : "ImageObject", "url" : "//www.jerichosecurity.com/hubfs/assets/images/Jericho-Security.svg" }, "name" : "Jericho Security" } } ```