Methodology: How We Score AI Tools
The full methodology behind our AI tool scoring. Detailed 1-10 rubric for each of 5 dimensions, 3 worked examples, the weighted average formula, and how we handle edge cases. This page is more technical than our editorial guidelines - it is for readers who want to understand exactly how we arrive at a score.
Last updated: July 24, 2026 · Version: 2.3 · Next review: October 2026
The 5-dimension framework (overview)
Every tool in our directory is scored on 5 dimensions, each on a 1-10 scale. The dimensions are:
- Output quality (30% weight) - what the tool produces
- Ease of use (20% weight) - the learning curve and day-to-day UX
- Pricing transparency (20% weight) - clarity and predictability of cost
- Support quality (15% weight) - human help when things break
- Privacy and data handling (15% weight) - what happens to your data
The final score is the weighted average, rounded to one decimal place. The minimum to be listed is 7.0 overall AND at least 6.0 in every dimension (7.0 in privacy and output quality specifically).
Why these 5 dimensions (and not others)
We tested 12 candidate dimensions in 2025. The 5 we kept are the ones that:
- Predict user satisfaction - the 5 dimensions correlate with the question "would I keep using this tool?" at r=0.78 (n=87 users, March 2026 survey)
- Are observable without insider access - we can measure them with a free trial, no vendor cooperation required
- Are not redundant with each other - we ran a factor analysis and these 5 explain 71% of variance with no significant cross-loading
- Are stable over time - scores for a tool change by less than 0.5 points over 6 months, unless the tool itself has a major update
The dimensions we rejected: integration ecosystem (too vendor-specific), community size (varies by category), brand reputation (we test the tool, not the brand), and update frequency (a tool can update often and still be bad).
Dimension 1: Output quality (30% weight)
This is the most important dimension. A tool that produces bad output is bad, regardless of how nice the interface is.
What we test
The reviewer uses the tool on their normal work for at least 2 weeks. We do not use synthetic prompts. We use real projects, real deadlines, real stakeholders. The reviewer keeps a daily log of what they used the tool for, what worked, and what did not.
What we measure
- Accuracy - is the output factually correct? For code: does it compile? For writing: are the claims supported? For image: does it match the prompt?
- Usefulness - does the output help the user accomplish their goal, or do they have to redo it?
- Consistency - is the quality the same across many uses, or does it vary wildly?
- Polish - is the output presentation clean, or does it need heavy editing to be presentable?
Scoring rubric
| Score | Meaning | Example |
|---|---|---|
| 9-10 | Indistinguishable from top human work | Claude Pro: ship-quality drafts on first try |
| 7-8 | Production-ready with light editing | Midjourney v7: 70% ship as-is, 30% need minor tweaks |
| 5-6 | Useful starting point, needs editing | Generic AI writer: 50% useful, 50% needs rewrite |
| 3-4 | Below average, frequent errors | Hallucinates 30%+ of the time |
| 1-2 | Not usable, consistently wrong | Output is unrelated to input |
Minimum to list
7.0
Dimension 2: Ease of use (20% weight)
A great tool that takes 6 months to learn is worse than a good tool that takes 6 hours. Most users do not have 6 months.
What we test
The reviewer signs up for a new account and times themselves on the first 3 tasks they would normally do with this tool. They rate the learning curve, the documentation, and the day-to-day UX.
What we measure
- Time to first useful output - how long from sign-up to producing something good?
- Documentation quality - are the docs complete, accurate, and findable?
- Onboarding - does the tool guide new users to the most useful features?
- Daily friction - what is annoying about using the tool every day?
- Error handling - when you mess up, does the tool help you recover?
Scoring rubric
| Score | Meaning |
|---|---|
| 9-10 | Intuitive from minute one, no tutorial needed |
| 7-8 | Easy to learn, 1-2 hours for advanced features |
| 5-6 | Has a learning curve, tutorial recommended |
| 3-4 | Steep learning curve, significant time investment |
| 1-2 | Not usable without extensive training |
Minimum to list
6.0
Dimension 3: Pricing transparency (20% weight)
The worst kind of tool is one that surprises you with charges. We score pricing transparency specifically because it is the dimension most likely to hurt users.
What we test
The reviewer signs up for the cheapest paid plan, uses it for a month, and tries to predict next month's bill. They check for: cancellation policies, refund policies, auto-renewal settings, usage limits, and any overage charges.
What we measure
- Clarity - is the pricing page clear and easy to understand?
- Predictability - can you predict your bill before the end of the month?
- Fairness - is the pricing fair relative to the value delivered?
- Cancellation - is it easy to cancel? Are there dark patterns?
- Refunds - is there a fair refund policy?
Scoring rubric
| Score | Meaning |
|---|---|
| 9-10 | Clear pricing, easy to predict, no hidden costs |
| 7-8 | Clear pricing, a few edge cases not obvious upfront |
| 5-6 | Documented but complex, multiple tiers |
| 3-4 | Opaque, hard to predict, surprise charges |
| 1-2 | Misleading, predatory, or no clear pricing page |
Minimum to list
6.0
Dimension 4: Support quality (15% weight)
When something breaks - and it will - how fast and how well does the vendor help?
What we test
The reviewer sends a real support ticket during the evaluation period. We measure response time, helpfulness, and whether the issue was actually resolved. We also test: live chat (if available), community forums, documentation quality, and bug tracker activity.
What we measure
- Response time - how fast do they reply?
- Helpfulness - do they actually solve the problem?
- Knowledge - do the support people know the product?
- Documentation - is there a real, maintained knowledge base?
- Community - is there an active user community that helps each other?
- Status page - do they have a public status page with incident history?
Scoring rubric
| Score | Meaning |
|---|---|
| 9-10 | 24/7 human support, real person, within 4 hours |
| 7-8 | Email support, 24-hour response, knowledgeable |
| 5-6 | Email or chat, 48-hour response, mixed quality |
| 3-4 | Slow, mostly bot-driven, multiple escalations needed |
| 1-2 | No support, or actively unhelpful or hostile |
Minimum to list
6.0
Dimension 5: Privacy and data handling (15% weight)
This is the deal-breaker dimension. A tool that mishandles your data does not get listed, no matter how good the other scores are.
What we test
The reviewer reads the privacy policy, terms of service, data retention policy, and security page. We check for: training on inputs, data sharing, data location, encryption, SOC 2 compliance, GDPR compliance, breach history.
What we measure
- Training on inputs - does the tool train on user data by default? Can you opt out?
- Data sharing - does the tool share or sell data to third parties?
- Data location - where is the data stored? Is it cross-border transferred?
- Encryption - is data encrypted in transit and at rest?
- Compliance - is the tool SOC 2 / GDPR / HIPAA compliant?
- Breach history - has the tool had a security breach in the last 3 years?
Scoring rubric
| Score | Meaning |
|---|---|
| 9-10 | No training by default, SOC 2, GDPR, encrypted |
| 7-8 | No training by default, clear privacy, GDPR |
| 5-6 | Trains on inputs but offers opt-out |
| 3-4 | Trains by default, opt-out hard to find |
| 1-2 | No privacy policy, sells data, or major breach |
Minimum to list
7.0 (higher than other dimensions because privacy failures are deal-breakers)
The scoring formula
Final score = (output_quality x 0.30) + (ease_of_use x 0.20) + (pricing_transparency x 0.20) + (support_quality x 0.15) + (privacy x 0.15)
Where each dimension is scored 1-10. The result is rounded to one decimal place. A tool needs:
- Final score of 7.0 or higher
- Output quality of 7.0 or higher
- Privacy of 7.0 or higher
- All other dimensions of 6.0 or higher
To be listed.
3 worked examples
Example 1: ChatGPT Plus (the highest-scored tool in our directory)
- Output quality: 9.0 (best general-purpose assistant, multimodal, plugins)
- Ease of use: 9.0 (intuitive from minute one)
- Pricing transparency: 8.0 (clear, but Plus vs Team vs Enterprise is complex)
- Support quality: 7.5 (email + community, 24-hour response)
- Privacy: 7.5 (no training by default, opt-out available, GDPR)
Final score: (9.0 x 0.30) + (9.0 x 0.20) + (8.0 x 0.20) + (7.5 x 0.15) + (7.5 x 0.15) = 2.70 + 1.80 + 1.60 + 1.125 + 1.125 = 8.35
Verdict: Listed. Top of the writing category. Justified by all dimensions above threshold.
Example 2: Jasper (a tool that did not make the directory)
- Output quality: 6.5 (decent but not better than Claude at the same price)
- Ease of use: 7.0 (good templates, easy to start)
- Pricing transparency: 5.0 (complex tier system, hard to predict monthly cost)
- Support quality: 6.5 (email support, 48-hour response)
- Privacy: 8.0 (no training by default, clear privacy policy)
Final score: (6.5 x 0.30) + (7.0 x 0.20) + (5.0 x 0.20) + (6.5 x 0.15) + (8.0 x 0.15) = 1.95 + 1.40 + 1.00 + 0.975 + 1.20 = 6.525
Verdict: Not listed. Final score below 7.0, and pricing transparency below 6.0 threshold. The tool has its uses (marketing teams that need brand voice training) but does not pass our overall bar.
Example 3: A tool that passes the threshold but with a privacy warning
- Output quality: 8.5 (excellent for the category)
- Ease of use: 8.0 (intuitive, good documentation)
- Pricing transparency: 8.0 (clear, predictable)
- Support quality: 7.0 (responsive, knowledgeable)
- Privacy: 6.5 (trains on inputs by default, but offers opt-out - opted out is not great)
Final score: (8.5 x 0.30) + (8.0 x 0.20) + (8.0 x 0.20) + (7.0 x 0.15) + (6.5 x 0.15) = 2.55 + 1.60 + 1.60 + 1.05 + 0.975 = 7.775
Verdict: Listed, but with a privacy warning at the top of the review. The tool is good, but the privacy dimension is a real concern. We note it explicitly: "This tool trains on your inputs by default. We recommend opting out of training in settings, and we do not recommend using it for sensitive work."
How we handle edge cases
Some tools do not fit cleanly into the rubric. Here is how we handle them.
Edge case 1: Open-source tools with no support
For open-source tools, there is no vendor support. The "support quality" dimension is replaced by "community support quality" - is there an active Discord, GitHub issues, or community forum where users help each other? Stable Diffusion is the example here: scored 9.0 on community support despite having no vendor support at all.
Edge case 2: Free-only tools
For free tools, the "pricing transparency" dimension is 10.0 by default (you cannot get a surprise charge from a free tool). The other dimensions are scored normally. Stable Diffusion, Ideogram free tier, and ChatGPT free are examples.
Edge case 3: Enterprise-only tools
For enterprise-only tools (no consumer pricing), the "pricing transparency" dimension is scored on the enterprise pricing page and contract clarity. Custom pricing is OK if it is clear what you are paying for. Palantir, Databricks, and Salesforce Einstein are examples.
Edge case 4: Tool with a major update between reviews
If a tool has a major update between our reviews (e.g. a new model launch, a major UI overhaul, a pricing change), we re-test within 30 days. The score may change up or down. The last-updated date is always current.
Edge case 5: Tool gets acquired
If a tool is acquired, we add a banner at the top of the review: "This tool was acquired by [Company] on [date]. We are re-testing under the new ownership." The re-test takes 4-6 weeks. The score may change based on the new owner's policies.
How we calibrate the rubric
Every 6 months, we run a calibration exercise to make sure the rubric is consistent across reviewers:
- All 5 reviewers independently score the same 10 tools (selected from our directory).
- We compare the scores. The standard deviation across reviewers should be less than 0.5 points per dimension.
- If the standard deviation is higher, we run a calibration session to align on what each score means.
- The last calibration: April 2026. Standard deviation: 0.31 points per dimension (good).
This is how we ensure the rubric is fair, consistent, and not dependent on which reviewer happens to test a tool.
How we handle disagreements with vendors
Vendors sometimes disagree with our scores. Here is how we handle it.
- Vendors can request a factual review. We will correct any factual errors (wrong pricing, wrong feature name, wrong release date). The correction is fast (1-3 business days).
- Vendors cannot request a score change. Our scores are based on our testing, not on vendor preferences. If the vendor disagrees, we will add a "vendor response" section to the review with their perspective.
- Vendors cannot remove a review. Once published, a review stays published. If a tool's quality drops, the score drops. If it improves, the score improves.
- Vendors can submit corrections in writing. If the vendor wants to add context, we will publish it in a clearly labeled "Vendor Response" section at the bottom of the review.
How to verify our scoring
You can verify any score by:
- Sign up for the tool yourself and use it for a week. Most of the tools have free tiers or trials. Compare your experience to our review.
- Read the reviewer's notes. Every review includes specific examples of what worked and what did not. If our examples do not match your experience, we want to hear about it.
- Check the last-updated date. The score is current as of the last review. If the tool has changed since then, let us know.
- Submit your own experience. If you have used the tool extensively and your assessment differs, email us. We consider every reader submission.
More about our process
For the broader editorial guidelines (how we pick tools, how we write reviews, what we will not do), see our editorial guidelines page. For the team behind the reviews, see our about page. For the changelog of what we have published, see our changelog.