Skip to content

How We Test

How We Test

Every product reviewed on TheCyberPicks goes through a structured, hands-on testing process. This page explains exactly what we do, how we measure results, and how we assign scores — so you can judge whether our methodology is sound.

We believe the best review methodology is one you can scrutinize and challenge. If you think we’re missing something, tell us.

General Principles

These principles apply to all product categories we review:

  • Real devices, real networks. We test on physical hardware connected to real ISP connections, not lab environments with artificial conditions.
  • Multiple platforms. We test on Windows, macOS, iOS, and Android where the product is available. If a product only has apps for certain platforms, we note the limitation.
  • Extended use. We use each product for a minimum of one to two weeks before writing a review. Quick installs and 15-minute tests don’t reveal real-world issues.
  • Controlled comparisons. When comparing products (e.g., speed tests), we test them under the same conditions: same device, same network, same time window.
  • Dated results. All test data includes the date it was collected. Product performance changes over time, so dated results let you judge currency.

VPN Testing Methodology

Testing setup

  • Primary test device: Windows 11 desktop (wired Ethernet)
  • Secondary devices: MacBook Air, iPhone, Android phone (Samsung Galaxy)
  • Base connection: Residential broadband (speed and ISP disclosed per review)
  • Testing period: Minimum 7 days of active use per VPN

What we measure

Speed performance

We run speed tests to the same set of server locations for every VPN to ensure fair comparison:

  • Local server (same country) — measures overhead added by VPN encryption.
  • Mid-range server (different continent) — measures real-world international browsing performance.
  • Long-distance server (opposite hemisphere) — tests worst-case latency.

For each location, we run three tests at different times of day (morning, afternoon, evening) and report the average. We also record the protocol used (WireGuard, OpenVPN, IKEv2) since protocol choice significantly affects speed.

Results are reported as a percentage of base connection speed retained, making comparisons fair regardless of how fast your own ISP connection is.

Privacy and logging policy

We don’t take “no-log” claims at face value. Our privacy evaluation includes:

  • Full privacy policy read. We read the entire privacy policy and terms of service — not just the marketing page.
  • Data collection specifics. We document exactly what data the VPN collects: connection timestamps, bandwidth usage, IP addresses, DNS queries, or nothing at all.
  • Jurisdiction analysis. We note where the company is incorporated, which data retention laws apply, and whether they’re part of intelligence-sharing alliances (Five Eyes, Nine Eyes, Fourteen Eyes).
  • Independent audit history. We check whether the VPN has undergone third-party security or no-log audits (e.g., PwC, Deloitte, Cure53) and when the most recent one was published.
  • Warrant canary and transparency reports. We check for published transparency reports and whether they document government data requests.
  • Real-world incidents. We research whether the VPN has been involved in data breaches, server seizures, or legal cases that tested their no-log claims.

Streaming and geo-unblocking

We test each VPN’s ability to access geo-restricted content on major platforms:

  • Netflix (US, UK, Canada, Australia libraries)
  • BBC iPlayer (UK)
  • Disney+
  • Amazon Prime Video
  • Hulu (US)

We verify access by actually loading and playing content, not just connecting to a server in the correct country. Streaming access is tested at the time of review and may change — VPN providers and streaming services play an ongoing cat-and-mouse game.

Security features

We evaluate:

  • Kill switch — does it actually block traffic when the VPN drops? We test by forcibly disconnecting the VPN mid-session.
  • DNS leak protection — we run DNS leak tests while connected to verify no queries leak to the ISP’s DNS servers.
  • WebRTC leak protection — we check for IP leaks via WebRTC in Chrome and Firefox.
  • Split tunneling — if available, we test that it works reliably and doesn’t leak traffic.
  • Protocol options — we note which protocols are available and which are default.

Apps and usability

We evaluate the user experience on each platform:

  • Installation process — how many steps, how long, any confusing prompts.
  • Interface clarity — can a non-technical user figure it out without a guide?
  • Server selection — is it easy to find and connect to specific countries?
  • Settings accessibility — can you access important features (kill switch, protocol selection) without digging through menus?
  • Stability — does the app crash, freeze, or fail to connect during the testing period?

Customer support

We contact customer support with a real question and evaluate:

  • Response time (for live chat and email/ticket)
  • Quality and accuracy of the answer
  • Whether we reached a human or a scripted bot
  • Availability (24/7 or limited hours)

Antivirus Testing Methodology

Our antivirus testing methodology will be published in full when we launch antivirus reviews (Phase 2). In the meantime, here is a summary of our planned approach:

  • Malware detection rates — cross-referenced with independent lab results from AV-Test and AV-Comparatives, supplemented by our own controlled tests using known malware samples.
  • System performance impact — measured as percentage increase in boot time, file transfer time, and application launch time with the antivirus running versus baseline.
  • Real-time protection — tested against simulated threat scenarios including phishing URLs, malicious downloads, and drive-by exploits.
  • False positive rate — how often the product flags legitimate software as malicious.
  • Feature set — evaluation of extras (firewall, password manager, VPN, parental controls) relative to their quality and added cost.
  • Pricing and value — total cost per device per year, including renewal pricing (which is often significantly higher than the introductory rate).

Password Manager Testing Methodology

Our password manager testing methodology will be published when we launch password manager reviews (Phase 2). Key evaluation areas will include:

  • Encryption architecture and zero-knowledge claims
  • Cross-platform sync reliability
  • Browser extension quality and autofill accuracy
  • Password generation and strength analysis
  • Breach monitoring and dark web scanning
  • Emergency access and recovery options
  • Pricing per user, including family plans

Scoring System

We rate every product on a scale of 1 to 5, scored to one decimal place (e.g., 4.3). The overall score is a weighted average of category scores:

VPN scoring weights

Category Weight What it measures
Privacy & Security 30% Logging policy, encryption, kill switch, leak protection, jurisdiction, audits
Speed 20% Download/upload speeds, latency across multiple server locations
Features 20% Server count, simultaneous connections, split tunneling, protocol options, extras
Ease of Use 15% App quality, installation, interface clarity, cross-platform availability
Value 15% Price per month, plan flexibility, money-back guarantee, free tier quality

Score interpretation

Score Meaning
4.5–5.0 Excellent — best in class, minor or no drawbacks
4.0–4.4 Very good — strong recommendation with minor compromises
3.5–3.9 Good — solid product with some notable weaknesses
3.0–3.4 Average — acceptable but better alternatives exist
2.0–2.9 Below average — significant issues; not recommended for most users
Below 2.0 Poor — avoid; serious privacy, security, or quality concerns

What We Don’t Do

  • We don’t inflate ratings to protect affiliate relationships.
  • We don’t copy results from other review sites.
  • We don’t use automated bots to test products — every test involves a real person using the real product.
  • We don’t guarantee results — your experience may differ based on your location, device, ISP, and the specific server you connect to.

Limitations and Transparency

No testing methodology is perfect. Here are the limitations we acknowledge:

  • Speed test variability. Internet speeds fluctuate based on time of day, ISP congestion, and server load. Our three-test averaging mitigates this but doesn’t eliminate it.
  • Streaming access changes. A VPN that unblocks Netflix today may not tomorrow. We note test dates so you know how recent our results are.
  • We can’t verify “no logs” claims independently. Short of being subpoenaed by a government, we can’t prove a VPN truly keeps no logs. We rely on independent audit reports, transparency reports, and real-world legal cases as proxies.
  • Single-tester perspective. As a small team, our reviews reflect one person’s experience. We mitigate this by using consistent, reproducible testing procedures.

If you believe our methodology has a blind spot, please let us know. We’re always looking to improve.

Last updated: [SET ON PUBLISH]