A practical template for comparing AI tools.
A useful comparison starts with one real task and a fair test—not a leaderboard or a single impressive answer. Use this lightweight protocol to compare tools on work you actually need done.
1. Choose one task with a clear outcome
Pick a task you understand well enough to judge. Examples include summarizing a lecture section, extracting action items from a meeting note, explaining a concept for a beginner, or drafting a short email. Avoid changing the task between tools.
Write down what a successful answer must include before testing. This helps prevent you from changing the rules to favor the output you happen to like.
2. Use matched inputs and instructions
Give each tool the same source material and the same core prompt. Record any differences that cannot be avoided, such as a file-size limit or a feature available only in one product. Do not quietly give one tool more context or extra revision rounds.
Replace “the task” with your exact task and paste the same source material into each tool. Keep the original outputs unchanged in your records.
3. Score against prewritten criteria
Use a simple scale from 1 to 5. Write what each score means before reading the outputs; otherwise, the scoring can drift from one tool to another.
| Criterion | Question to ask |
|---|---|
| Accuracy | Are claims supported, and are errors or invented details present? |
| Coverage | Did it include the required points without important omissions? |
| Clarity | Can the intended reader understand and use the answer? |
| Organization | Is information structured in a way that helps the task? |
| Revision effort | How much work was needed to make the result usable? |
Not every task needs equal weights. For source-based study notes, accuracy and coverage may matter more than style. For brainstorming, variety and usefulness may matter more. Choose weights that match the task and publish them with the results.
4. Record time, cost, and constraints
A response that looks good but takes longer to verify may not be the best option. Track how long you spend checking and editing each output, plus any usage limits, paid features, or privacy constraints relevant to your decision.
- Tool and model/version, if visible
- Date of test and exact prompt
- Source material or a reproducible description of it
- Original output and number of revision rounds
- Scores, reviewer notes, and editing time
- Known missing information and test limitations
5. Avoid overstating the conclusion
A small test tells you how tools performed on that task under those conditions. It does not prove that one tool is universally better. If you have one reviewer, say so. If output identities are hidden, do not guess which model produced each answer. If records are incomplete, label the result provisional.
When possible, run more than one task and repeat the comparison. Preserve ties when the evidence does not support a clear winner.
Reusable results table
| Record | What to write down |
|---|---|
| Task | One-sentence task and intended reader |
| Setup | Prompt, source, tools, date, and versions |
| Evaluation | Criterion scores with a short evidence note for each |
| Effort | Verification time, editing time, and revision rounds |
| Limitations | Missing records, uncertainty, and what the test cannot show |
For the longer method, read How to compare AI tools fairly. If the task involves study material, use the source-checking workflow before relying on the generated notes.