Why Testing Matters More Than Claims

Security products are often evaluated on the strength of their claims rather than their behaviour. Marketing promises, feature lists, and headline metrics provide limited insight into how systems perform under realistic conditions. Meaningful testing focuses on what happens when controls fail, how quickly failures are detected, and how much damage can occur before a response begins. This article explains why evidence matters more than assertion, what good testing reveals, and how realistic evaluation leads to better security decisions over time.

It is easy to make claims. It is harder to test them. But security vendors often demand your money without providing solid evidence that they will solve your security problems, which is why testing matters so much.

In cyber security, the gap between what products promise and how they behave in practice can be significant. This is not necessarily due to bad faith. Complex systems behave unpredictably, and controlled demonstrations rarely reflect real environments.

Bad faith claims

That said, some controlled demonstrations are so intentionally biased to provide a favourable result that bad faith does start to creep in around the edges. Or, in one memorable case with a so-called ‘next-gen’ anti-virus company, an entire and central marketing project was based on bad faith.

Testing matters because it replaces assumptions and assertions with credible observation.

A useful test does not ask whether a product blocks an idealised threat. It asks how the product behaves when conditions are imperfect. What happens when credentials are stolen? Or when a user makes a mistake? Or when an attacker adapts?

Limitations

Equally important is understanding what a product does not do well. Every system has limits. Knowing where those limits lie allows organisations to manage things deliberately, rather than discovering limits accidentally.

If you need a product but know it isn’t very good at a certain task, you can shore up its failings with something demonstrably better. You don’t have to throw everything out and start again. But to do this effectively you need to be aware of the limitations.

Repeat and be resilient

Testing should be repeatable and transparent. If you can’t reproduce or understand results then they provide little value. The aim is not to produce flattering outcomes, but to reveal behaviour.

There is also a distinction between prevention and resilience. Preventive controls will eventually fail. Resilient systems limit the consequences of that failure and make recovery easier.

Organisations that base decisions on evidence tend to be less surprised by incidents. Their expectations are grounded in observed behaviour rather than marketing narratives.

Over time, this approach produces better outcomes. Not because it eliminates risk, but because it aligns understanding with reality.