How well do AI systems produce technical revision notes?
An exploratory qualitative review of three outputs created from a technical report about generative AI in automated curricular synthesis.
Research question
When given a technical source report, how well do generated revision notes preserve source meaning, cover important content, communicate clearly, organize information, and identify actionable revision issues?
Method at a glance
- Three materials were reviewed against the same supplied source report.
- Two qualitative evaluations were conducted: an initial rubric-based audit and a second evaluator's assessment.
- The rubric used five criteria scored from 1 to 5, with fixed weights.
- The second evaluator's totals were recalculated after arithmetic inconsistencies were identified.
- The exact original shared task prompt was not available, limiting assessment of instruction-following.
Provisional independent-evaluator scores
The table reproduces the second evaluator's criterion ratings. Weighted totals below are corrected using each rating divided by 5, multiplied by the criterion weight.
| Criterion | Weight | Output A | Output B | Output C |
|---|---|---|---|---|
| Accuracy and source fidelity | 35% | 5/5 | 4/5 | 5/5 |
| Coverage | 25% | 5/5 | 5/5 | 5/5 |
| Clarity | 15% | 4/5 | 4/5 | 5/5 |
| Organization | 10% | 4/5 | 3/5 | 5/5 |
| Revision usefulness | 15% | 1/5 | 3/5 | 5/5 |
| Corrected weighted total | 100% | 83/100 | 80/100 | 100/100 |
Calculation: Σ (criterion rating ÷ 5 × criterion weight). The second evaluator initially reported 78 for B and 98 for C; both totals were corrected to 80 and 100 respectively. Correct arithmetic does not resolve uncertainty about material authenticity.
What the review suggests
Output A
The evaluator found that A reproduces the source verbatim. That makes it unsuitable as revision notes, but its origin is unresolved: it may be a model repeating the input or an accidental source-text substitution. Its score should not be treated as comparable evidence of model performance.
Output B
The evaluator described B as a broad structured summary, with some formatting problems and limited explicit criticism of the source report.
Output C
The evaluator rated C highest for readable structure and source-critical analysis, including attention to missing empirical measures, unclear configurations, wording defects, and tension between source-fidelity instructions and reported external contextualization.
Source-report concerns identified
- The source report does not provide clear sample sizes, statistical tests, or numerical performance measurements.
- Exact model versions and configurations are unclear.
- Phrases such as “tiered tier features” and “summarized summaries” appear to need editorial correction.
- The report restricts external concepts while also describing Gemini as adding external context; the relationship needs clarification.
- The phrase “high-retention study guides” is not supported by reported student-retention measurements in the supplied source.
Limitations
- Only one source report and three materials were considered.
- The original shared task prompt is unavailable.
- The model-to-output mapping has not been verified for publication.
- Output A's authenticity is unresolved, which undermines direct comparison across all three outputs.
- Scores are subjective rubric judgments; no independent human replication or inter-rater reliability analysis has been completed.
- This experiment evaluates revision-note outputs only. It does not test student learning, retention, or general model quality.
Scoring method
Each criterion is rated from 1 to 5. Its contribution to the total is the rating divided by 5, multiplied by the criterion weight. The five weights sum to 100%.
| Criterion | Weight | Evaluation focus |
|---|---|---|
| Accuracy and source fidelity | 35% | Faithfulness to source claims and qualifications; unsupported additions. |
| Coverage | 25% | Coverage of important sections, findings, limitations, and recommendations. |
| Clarity | 15% | Readability, precision, and explanation. |
| Organization | 10% | Logical structure, headings, and scannability. |
| Revision usefulness | 15% | Actionable edits and identification of source weaknesses. |
Formula: Weighted total = Σ [(criterion rating ÷ 5) × criterion weight]. For example, a rating of 4/5 on a 35%-weighted criterion contributes 28 points. The published totals for B and C were corrected from 78 to 80 and 98 to 100 after arithmetic review.
Reproducibility record
This record distinguishes information available in the current audit from details that could not be verified from the materials reviewed. Missing details are not inferred.
| Item | Current status |
|---|---|
| Source report | Title and text are available in the working conversation; a downloadable public research archive has not been added. |
| Exact shared task prompt | Not recovered / not verified. |
| Model names and versions for A/B/C | Output-to-model mapping is unverified; do not attribute the scores to named models. |
| Run dates and settings | Exact run dates, model versions, and generation settings have not been verified. |
| Complete outputs A/B/C | Some output files are available in the working materials; authenticated, labeled originals have not all been confirmed for a public archive. |
| Evaluation records | An initial qualitative audit and a second evaluator's written ratings are available. A second independent human scoring pass has not been completed. |
Research archive checklist
The following items remain to be assembled before this experiment can be described as reproducible: (1) the exact shared prompt; (2) the source report as supplied; (3) each complete, original output labeled A/B/C; (4) model/version and run date for each output where recoverable; (5) the rubric and evaluator score sheets; and (6) a short change log recording the corrected weighted totals. Until then, this page is a transparent preliminary record, not a complete replication package.
Conclusion
In the second evaluator's assessment, Output C received the highest provisional rating, followed by A and B. Because A may be the source report itself and the shared prompt and model attribution remain unresolved, these scores are best treated as an exploratory record of the supplied materials—not a definitive model ranking or validated benchmark.
Next steps
- Recover the original shared prompt and original unedited outputs.
- Verify the identity of each output and document model/version and run date where available.
- Have an independent human evaluator score the authenticated outputs blind to model identity.
- Repeat the test on additional source documents before drawing broader conclusions.
Research status: exploratory qualitative comparison. Scores last arithmetically audited 23 October 2026. No claim of statistical significance or improved student retention is made.
Back to CurioLab ↗