A research assistant that reviews its own answers

How a question is answered

1. Retrieve. MiniLM embeddings through ONNX Runtime, with the weights checked by SHA-256, and an exact cosine search for the top 5 passages. A paper is a few hundred passages, so exact search costs milliseconds and an approximate index would only cost recall.
2. Answer. The model sees the passages numbered [1] to [5] and replies in a JSON schema: a short answer, the passages that support it, and a flag for "the passages don't answer this". Made-up passage numbers are dropped.
3. Review. A second call reads the question, the answer and the full text of what it cites, marks each part supported or not, and if something is missing writes one search query to find it.
4. Again. New passages are merged with the old ones and the answer is redone from all of them.

Model calls go through one small interface, LLMClient, with retries in my code rather than the SDK, so every retry shows up in the trace and the tests can count them.

How a run ends

One of five ways, and the result says which. The deadline and the budget are checked before every call.

How it's measured

In QASPER (Dasigi et al., NAACL 2021), annotators answered each question after reading the whole paper and marked the paragraphs their answer rests on, so a system is scored on what it cites as well as what it says.

• Each question is asked against its own paper, split into the dataset's paragraphs, so a citation matches the annotators' evidence exactly.
• 150 dev questions for reading errors; 150 test questions, run once after the last change.
• Same retrieval and models for both systems: gpt-6-luna answers and reviews, gpt-6.1-sol judges. The loop's first round replays the single pass's request, so any difference from the 5-passage single pass comes from the review rounds and their extra searches.
• Official answer and evidence F1, ported from allenai/qasper-led-baseline, plus judged accuracy. The judge is checked against 40 answers I graded myself, blind to its verdicts.
• Paired differences with 95% bootstrap intervals, the first failed stage for every wrong answer, and a spending cap, $25 by default and $5 for these runs, that refuses any call that could cross it.

Seed 0 throughout. The single pass and the loop ran at commit daac2a1 on 2026-10-28. The two wider single passes ran on 2026-11-02 at aac699e, and the loop and single-pass rows replayed from the recording. Counting every recorded call, original and matched runs, the recording comes to $0.89 of a $5 cap across 1,986 calls. The spend line in the results files is a running total, so the dev file's $0.65 is the total when its matched run finished, not what dev cost. make check-replay reproduces every row with no key.

Paired by question, the 95% interval on the answer F1 difference is -0.011 to +0.037 on dev and -0.015 to +0.029 on test. On test, evidence F1 is +0.024 (-0.003 to +0.051) and judged accuracy is +2.7 points (0.000 to +0.060), which is 5 questions the loop got right and the single pass got wrong against 1 the other way. With six comparisons I don't read any of that as a finding.

The loop made 2.5 calls per question against 1. Answering cost about three times as much per question ($0.00028 against $0.00009 on test), the slowest 5% of questions took about 11 seconds against 2 to 3, and with judging included the bill differs by about 20%, because the judge is most of the spend. On this sample I can't say the loop is worth its extra calls.

Against a single pass that stops at the first search's top 5 passages, the loop is a tie; it starts from that same retrieval but can search again, and averages 5.6 passages a question. Against one that reads 10 or 15 in a single call, it loses. Judged correct, loop minus baseline, paired by question: -7.3 points against the top 10 and -13.3 against the top 15 on dev (the loop right and the baseline wrong on 5 and 3 questions, the other way on 16 and 23), and -9.3 points against both on test (9 and 10 against 23 and 24). The 95% intervals exclude zero in all four, and the exact sign-test p is 0.027, below 0.001, 0.020 and 0.024. Answer F1 points the same way. The wider single pass costs about the same per question, judging included, and its 95th-percentile latency is 2.2 seconds against 11.0.

These baselines are matched to the most the loop can read, not to what it reads on average: the loop averages 5.6 passages a question, so the baselines read more than it does. They show that this loop's retrieval budget is better spent on a wider first search, not that a review loop can't help. They were added after I had seen the original test results, so the test comparison is post hoc, and the dev comparison, run once with the same code, agrees with it. The failure table fits a retrieval problem: on test the loop never retrieved the evidence on 20 of 150 questions, against 11 for the top 10 and 8 for the top 15. I haven't tested that explanation.

The judge agreed with my grades on 35 of the 40 answers (87.5%, Cohen's kappa 0.75): more lenient than me on 2 and stricter on 3. I graded them by hand on 2 November 2026 without seeing its verdicts, one grader in one pass over 37 distinct questions, and the grader also ran the experiments, so this is not external validation. An earlier draft of the labels, which was not an independent grade, is kept in the repo and is not scored.

Rerunning without a key

Every model reply is recorded under a hash of its request, and every search goes to a run log with its scores. research-assistant check-replay reruns the evaluation from those files with no API key and lists any answer, citation or score that changes, so changing a prompt or the chunking fails the replay.

MiniLM runs in float32, so on another CPU a similarity can move in the sixth decimal and swap two near-tied paragraphs. A replay keeps the logged ranking when every score agrees to 1e-4 and fails when retrieval really changed. A reply the API cut off is billed, recorded as incomplete and never replayed as an answer.

Limits

• QASPER papers are all NLP, and each question searches only its own paper.
• Figures and tables aren't indexed.
• With 150 test questions the interval on judged accuracy is 8 points either side, so small differences will show up as ties, and they did.
• The judge is a model. My 40 grades show it is broadly in line with me; one grader and 37 distinct questions don't settle how far to trust a difference of a few points.
• One seed, one sample of 150 questions per split, one model pair and one embedding model.

What I'd do next: give the loop a stronger first retrieval and see whether review adds anything on top of it, since the failure table says retrieval, not review, is where this loop loses. A larger independent sample would also tighten the dev and test comparisons, and a second grader would give the judge check an inter-rater number. None of this has been run. Re-checking the recorded runs costs nothing (make check-replay), but new questions would need new API calls.

Run it yourself

make install   # the hashed lock, then the package and the test tools
make demo      # three sample papers, offline stub model, no key

research-assistant ingest path/to/papers/
research-assistant research "Which datasets do these papers evaluate on?"
research-assistant research "..." --max-rounds 1    # single pass

make qasper         # dev and test sets, with the release tarballs' SHA-256 recorded
make eval-dev       # 150 dev questions through both systems
make eval-test      # 150 test questions, once
make check-replay

The sample papers are short synthetic ones I wrote so the demo runs offline. The HTTP API adds key auth, per-caller rate limits and Prometheus metrics, and the Docker image is pinned by digest.