Measuring whether AvtoPilot actually works

This site now has a Research section, where we write up how our products work, with real numbers and honest limits. The first two pieces are an engineering deep dive on QuizPilot's extraction pipeline and our first efficacy study, an attempt to answer the only question that really matters about a learning product: do the people who use it actually get better?

The study starts from an uncomfortable observation. Over a learner's first ten active days on AvtoPilot, almost every number in the app drifts down. Raw practice accuracy falls from 84 to 78 percent, and accuracy on never-before-seen questions falls from 81 to 73 percent. If we wanted a flattering dashboard, we would not show you this.

The decline is the method working. AvtoPilot's FSRS spaced repetition engine deliberately resurfaces the questions a learner is about to forget, and the easy material gets exhausted early, so what remains is harder by construction. You cannot measure learning by the difficulty of what you are surviving, so we needed a yardstick the system cannot bend.

That yardstick is the fixed mock exam: twenty questions drawn at random from the full 1,220-question bank, scored the same way every time and never adaptively hardened. There, the picture inverts. Across ten attempts, the average score rises from 14.0 to 15.7 out of 20, and the pass rate climbs from 20 to 31 percent. The result holds under the strictest cut too: the 1,416 learners who took at least five mocks improved from 14.01 to 14.72, with the share reaching a passing 18 or better going from 19.8 to 26.1 percent.

The piece is equally clear about what it is not. This is observational data with no control group, on our own low-stakes mocks rather than the government exam, over about five months of a young product. Learners who practiced improved on our measures; we do not claim AvtoPilot caused it. That is the standard we want to hold in public, and the full study is in the Research section.