Simulated phishing tests are one of the few security tools that can make your business measurably safer or measurably more suspicious of you, depending on how you run them. Send the right test and people get better at spotting fraud. Send the wrong one and you spend the next year explaining yourself, while your team quietly decides the safest move is to tell you nothing.
We use simulations with clients, and we also talk clients out of them fairly often. The difference is not the software. It is whether the program is built to teach or built to catch. Here is our honest take on what separates the two, and how to run tests that leave people more willing to raise their hand rather than less.
What a Simulation Actually Measures
A phishing simulation is a fake fraudulent email your own organization sends, so you can see what happens next. That is genuinely useful. It shows you whether people notice, whether they know where to report, and whether your reporting process works when someone tries to use it under mild stress.
What it does not measure well is how careful your people are, and that is where most programs go wrong. When NIST published its Phish Scale research in 2020, it made the point that a low click rate for a particular phishing email can have several causes, and that looking at click rates without understanding how hard the email was to detect can create a false sense of security. NIST built the Phish Scale to help trainers judge whether a phishing email is harder or easier for a particular audience to detect, rating elements tied to the scenario’s premise. NIST also noted that a training email may produce a low click rate simply because it was too easy, because it did not provide relevant context to the user, or because it resembled a previous exercise.
Translated into plain terms: a good score can mean your team is sharp, or it can mean you sent an easy test. A bad score can mean your team needs practice, or it can mean you sent something nearly impossible to catch. The number alone tells you very little.
Why Click Rate Is the Wrong Headline Number
Click rate is popular because it is easy to graph and it goes down over time, which looks like progress. But in a real incident, nobody wins by clicking slightly less often. You win because somebody told you fast enough to act. So measure that.
- Reporting rate. What share of people who received the message reported it. This is the number that maps to real defense, because reports are what start your response.
- Time to first report. How many minutes from send to the first person raising a hand. Ten minutes is a very different Tuesday than four hours.
- Whether reporting worked at all. Did the button exist, did the message reach a monitored inbox, did anyone answer the person who reported. Tests reveal broken plumbing more often than careless people.
- Difficulty, recorded every time. Note how hard each test was before you send it, so you are not comparing an easy campaign against a hard one and calling the difference improvement.
Click rate is still worth watching. It just belongs in the footnotes, not on the scoreboard.
What a Good Program Looks Like
The strongest programs we run share four traits, and the first one surprises people.
- Announced in advance. Tell everyone, in plain language, that simulated phishing emails will show up throughout the year and that the purpose is practice. You are not trying to catch anyone by surprise. You are trying to build a reflex, and a reflex practiced knowingly still works when the real thing arrives.
- No shaming, ever. No public lists, no leaderboards with names, no jokes in the all-hands about who fell for it. Aggregate results only.
- Coaching, not discipline. Someone who clicks gets a short, friendly explanation of what the message was doing and what to watch for next time. Repeat clicks signal that a person needs help or that their role is genuinely high risk, not that they are a bad employee.
- Reporting is the point. Every test ends with a thank you to the people who reported, and the reporting path is one button that takes ten seconds. CISA’s guidance for handling suspicious messages is to report it, resist clicking any link or attachment, and delete it, and to look up another way to contact the company or person directly if the message might be real. That is exactly the habit a good simulation rehearses.
The Tests That Destroy Trust
Some simulations are effective in the narrow sense that lots of people click, and catastrophic in every other sense. The common thread is that they use an employee’s personal hopes or fears about their job as bait.
- Fake bonus, raise, or reimbursement notices. The click rate will be spectacular. So will the resentment when people learn their employer dangled money that was never real.
- Fake layoff, restructuring, or severance emails. This is the one that ends up in the news. It teaches people that company communications about their livelihood might be a trick, which is a lesson you cannot unteach.
- Fake HR investigations or benefits changes. Same problem. You are borrowing credibility from the channel employees most need to trust, and you do not get it back cheaply.
- Anything exploiting a personal crisis. Charity drives for a sick colleague, family emergencies, hardship funds. Do not.
The contrarian argument for these tests is that real attackers use exactly these themes, so your simulations should too. That argument is not wrong about attackers. It is wrong about you. Attackers are not trying to preserve a working relationship afterward. Your goal is a workforce that reports quickly, and you cannot get there by proving you will use their anxiety as a training aid. Plenty of realistic themes remain: shipping notices, invoice updates, shared documents, password expiration warnings, vendor bank changes. Those mirror the attacks we described in our look at the surge in phishing attacks without costing you trust.
Rolling It Out Without Losing the Room
- Announce the program before the first test. Explain what it is, why you are doing it, and that results are never used for discipline. Put it in writing so it survives management changes.
- Fix reporting first. A one-click report button, a monitored destination, and a person who replies. Running tests before this exists just measures your own gap.
- Start easy and get harder slowly. The first test should be catchable. You are building confidence, not a trap.
- Share aggregate results within a week. Reporting rate, time to first report, what the message was doing, and thanks to the people who flagged it.
- Fix the system, not just the people. If a test shows a fake vendor bank change could have worked, the answer is a payment verification step, not more scolding. If credentials would have been usable, the answer is multi-factor authentication.
- Run them quarterly, not monthly. Monthly tests turn into background noise and breed cynicism. Quarterly keeps it fresh and keeps it small.
The Bottom Line
Phishing simulations are a teaching tool when they are announced, coached, and measured by how fast people report. They are a trust killer when they are secret, scored, and baited with someone’s paycheck. The honest test of your program is simple: after a round of simulations, is your team more likely to forward you something odd, or less? If it is less, the program is making you less safe no matter what the click chart says. And remember that no amount of testing substitutes for the controls behind it, a point that comes through clearly in what the Stryker cyberattack tells us about the threats facing every business.
If you want a simulation program that builds reporting instead of resentment, we can set one up, write the announcement, handle the coaching, and report the numbers that actually matter. Contact us today.
Sources:
Comments are closed