Cybersecurity misuse

The line we're drawing

This category tests requests seeking operational help with unauthorized access, malware development, or attack infrastructure. Drawing that line correctly matters more than it might first appear. The wrong answer in either direction has a real cost attached to it. This category is deliberately evaluated alongside, and never instead of, legitimate security research, defensive tooling, and CTF-style educational work. tAI is meant to support all of that without hesitation, and without an extra layer of suspicion applied just because the topic sounds adjacent to something dangerous. Refusing the entire topic area outright would be the easy, low-effort answer to write into a safety policy. It would also make tAI genuinely useless to the security researchers, students, red-teamers, and defenders who make up the large majority of people asking about this topic in the first place.

We think that trade-off is one of the clearest places where an overly cautious safeguard causes real harm. Refusing broadly feels safe. Staying genuinely useful to the people this topic area is actually for is the harder, better goal. A security team unable to get help writing a proof-of-concept exploit for a vulnerability they're authorized to test is a genuine cost. A student unable to get an explanation of how a real attack technique works for a class assignment is a genuine cost too. Neither is a safe default with no downside. That's part of why this category is evaluated with as much rigor as the categories that are more obviously about preventing harm. Getting it wrong in the over-cautious direction is a failure mode we take just as seriously as getting it wrong in the under-cautious one.

How the distinction gets made

The evaluation set combines three kinds of requests deliberately in the same pass. The first is clearly malicious requests that no reasonable reading would call legitimate. The second is clearly legitimate ones that no reasonable reading would call malicious. The third, and the largest, is a set of deliberately ambiguous requests written specifically to probe the boundary between the two. That middle set is where the evaluation actually does most of its work. A safeguard that only has to distinguish the obviously-malicious end from the obviously-legitimate end is solving an easy version of the problem. That easy version doesn't reflect how most real requests in this area are actually phrased.

A request's phrasing factors into how it's handled. So does its framing. So does whether it reads as first-person authorized activity, someone describing a system they own or are contracted to test. Or whether it reads as a request to act against a system the requester plainly doesn't have standing over. This isn't a simple keyword match against a list of dangerous-sounding terms. A keyword match would be trivially easy to route around. It would also catch a huge amount of entirely legitimate security discussion in its net. It's closer to the same kind of contextual judgment a knowledgeable human reviewer would apply, weighing framing, specificity, and stated purpose together rather than reacting to any single word in isolation.

Dual-use tooling

A specific sub-case tracked on its own within this category is dual-use code. That's a script that is simultaneously a completely normal administrative or diagnostic tool and something that could, in different hands, be repurposed for attack tooling. A network scanner is the clearest common example. A credential-handling utility is another. Both have entirely mundane, everyday uses in ordinary systems administration. Both also appear, in slightly modified form, in real attack toolkits. tAI is tuned to write this kind of code when the surrounding context supports a legitimate purpose. That matches how we expect our own products and the people who use them to actually behave, rather than refusing the entire category outright just because a determined bad actor could theoretically misuse the same basic pattern.

The main error mode

In evaluation, the more common mistake is over-caution rather than under-caution. That shows up more often than we'd like when we look closely at what actually went wrong. A legitimate request phrased in a way that happens to pattern-match malicious framing is the usual culprit. A concrete recurring example is a penetration tester describing a target system in the first person, the way someone naturally would when talking about work they're actually doing. They don't always state "I am authorized to test this" in so many words, since most people don't narrate their own credentials every time they describe their job. That kind of request gets treated with more suspicion than it deserves more often than a genuinely malicious request slips through undetected. That's a real, measured cost, not a safe default we're comfortable shrugging off. It feeds directly into the broader refusal-calibration work described on the tAI 4.2 page.

Keeping the evaluation set current

A fixed evaluation set built once and left unchanged would get stale fast in this category specifically, faster than in most of the others on this page. New attack techniques get publicly disclosed regularly. New tooling and new classes of vulnerability show up on a similar cadence. A test set that only reflects what was known when it was first built would drift further from real-world relevance with every month that passes, even while the reported catch rate against it stayed flat or improved. That gap between a stable-looking number and an increasingly outdated test set is exactly the kind of quiet failure mode we try to design against throughout this section, not just in this one category. A test set is treated as a living document, not a fixture built once at launch and left untouched afterward.

New publicly disclosed technique classes get folded into this category's evaluation set on an ongoing basis, not just at major version boundaries. The same is true when a new product surface is added, since a new connector or a new tool category can open up requests that didn't have an equivalent before it existed. This overlaps with, and feeds from, the same red-teaming process described in full on the Red-teaming page. A technique a red-teamer finds working against an earlier checkpoint becomes a permanent addition to this category's standing evaluation set going forward, not a one-time finding that gets fixed and then forgotten. That's also true of confirmed reports that arrive through responsible disclosure, which are folded in the same way. The evaluation set for this category has grown steadily larger with every version as a direct result of this process, not stayed fixed at whatever size it started at.