Spelling Force word mastery research

Empirical Analysis of Student Learning in the Spelling Force

Abstract

This paper presents an empirical analysis of student performance within the Spelling Force Mastery Learning platform, utilising a comprehensive telemetric dataset representing 3,343 active users, 11,709 distinct word sets, and 198,105 spell-check attempts.

Our findings indicate a robust baseline success rate of 80% across all spelling attempts. In total, 45% of evaluated word sets were mastered under standard operating paradigms. This research highlights the efficacy of the system’s adaptive re-leveling algorithm, which successfully down-calibrates struggling students, while identifying critical pedagogical and algorithmic adjustments to prevent systemic bottlenecks—particularly at the introductory developmental stage (Era 1) and within the “Too Hard” feedback loop.

Finally, sequential analysis of multi-attempt word paths reveals powerful evidence of student learning, with an 88% cumulative recovery rate within three attempts of an initial spelling error.

1. System-Wide Mastery States

The primary objective of the Spelling Force platform is to guide students toward the “Mastery” state of designated academic word sets. Within the baseline dataset containing 11,709 attempted word sets, the overall spelling success rate was 80% across 158,299 correct and 39,806 incorrect spellings.

Students required a mean of 1.7 correct repetitions per unique, correctly spelled word to solidify retrieval pathways.

Word Set Calibration Breakdown

The performance profile of users changes sharply depending on the system’s operational determination of a word set’s difficulty relative to the user’s skill level. The system segments these into four key progress states:

1. Mastered (Standard): Representing the largest finalised cohort (45% of all sets; n = 5,242), these sets demonstrate the target optimal learning zone. Students averaged 20.4 word attempts per set (ranging from 10 to 232) with an 87% success rate (93,343 correct vs. 13,512 incorrect answers) and 1.8 repetitions per correct word.
2. In Progress: Nearly half of the sets (48%; n = 5,646) remained incomplete at the time of telemetry capture. This cohort operated at a 73% success rate (53,450 correct vs. 19,909 incorrect answers), requiring 1.5 repetitions per correct word.
3. Cancelled (“Too Hard”): A total of 690 word sets (6%) were aborted after triggering the “Too Hard” threshold. These sets yielded a lower 60% success rate (9,338 correct vs. 6,291 incorrect), though students demonstrated intense persistence, repeating correct words an average of 2.3 times.
4. Mastered and Flushed (“Too Easy”): Only 1% of sets (n = 131) triggered an immediate “Too Easy” classification, showing a 96% success rate (2,168 correct vs. 94 incorrect answers) and 1.7 correct repetitions.

Mastery Cohort Performance Metrics
+————————-+——–+————+————+———+
| Mastery Cohort | Count | Correct | Incorrect | Succ. % |
+————————-+——–+————+————+———+
| Mastered (Standard) | 5,242 | 93,343 | 13,512 | 87% |
| In Progress | 5,646 | 53,450 | 19,909 | 73% |
| Cancelled (“Too Hard”) | 690 | 9,338 | 6,291 | 60% |
| Mastered (“Too Easy”) | 131 | 2,168 | 94 | 96% |
+————————-+——–+————+————+———+

2. Adaptive Calibration & Re-leveling Dynamics

When the system triggers a “Too Hard” or “Too Easy” determination, it initiates an adaptive calibration sequence to shift the user to a more appropriate curriculum “Era.”

[Word Set Evaluation]
|
+——————–+——————–+
| |
[Too Easy Trigger] [Too Hard Trigger]
| |
Shift to Higher Era Shift to Lower Era
(Target: Era_{current} + 1) (Target: Era_{current} – delta)

In the Telemetry sample, 204 users underwent “Too Hard” re-leveling, and 6 completed “Too Easy” re-leveling.

• “Too Easy” Re-leveling: Limited telemetry showed 100% successful re-adaptation with no subsequent regression.
• “Too Hard” Re-leveling: Highly dynamic adjustments occurred across 204 instances:
  • Regressive Calibration (71%): In 144 cases, the user successfully adapted to an easier Era. The severity of the downward adjustment varied: \Delta_{\text{Era}} = -1 \implies n = 62 \quad (30.4%) \Delta_{\text{Era}} = -2 \implies n = 33 \quad (16.2%) \Delta_{\text{Era}} = -3 \implies n = 30 \quad (14.7%) \Delta_{\text{Era}} \ge -4 \implies n = 19 \quad (9.3%)
  • Static Calibration (20%): In 41 cases, the Era remained unchanged. Crucially, 26 of these occurred at Era 1 (the lowest developmental level), representing a systemic floor effect.
  • Progressive Calibration (9%): In 19 cases, users were placed in a more difficult Era. These instances, along with multi-releveled anomalies (e.g., 4 users re-leveled 3 times; 1 user re-leveled 4 times), are heavily suspected to represent QA/test accounts.

View Statistical Suggestion for Algorithmic Optimisation

To prevent extreme downward calibrations (which are demotivating and often a byproduct of testing accounts), the search space for adaptive calibration should be capped. We propose introducing constrained bounds:

\text{Next Era}{\text{Too Hard}} = \max\left(\text{Era}{\text{floor}}, , \text{Era}{\text{current}} – 2\right) \text{Next Era}{\text{Too Easy}} = \min\left(\text{Era}{\text{ceiling}}, , \text{Era}{\text{current}} + 1\right)

Furthermore, when a student triggers “Too Hard” at Era 1, redirecting them to the step-by-step Look, Listen, Learn (LLL) mode rather than restarting Era 1 avoids frustrating loops.

3. Comparative Analysis of Spelling Modes

Students interacted with Spelling Force through 11 separate delivery modes representing Quiz formats, standard spelling games, challenge variants, and diagnostic simulations.

A total of 32,244 first-attempt records were mapped to evaluate spelling accuracy and immediate frustration events (the “Too Hard” rate) across these modes:

Mode	Database Code	Attempt Records (N)	Correct Rate (P_C)	Incorrect Rate (P_I)	“Too Hard” Rate (P_{TH})	Mean Difficulty (Era)
Quiz	Q	24,565	73%	24%	3%	4.33
Look, Listen, Learn	L	832	83%	14%	2%	4.44
NAPLAN Simulator	N	485	73%	23%	4	4.68
Magno Beamer	g	426	77%	2%	20%	3.84
Major Meltdown	m	2,590	90%	10%	0%	3.84
Launchpad Panic	p	596	79%	10%	11%	3.84
X-ray Emergency	x	2,008	65%	25%	10%	3.84
Magno Beamer Challenge	G	52	89%	8%	4%	< 3.84
Major Meltdown Challenge	M	435	91%	9%	0%	< 3.84
Launchpad Panic Challenge	P	58	86%	7%	7%	< 3.84
X-ray Emergency Challenge	X	197	65%	27%	8%	< 3.84

Strategic Insights by Mode Type

• The Dominance of Quizzes: The standard Quiz mode serves as the baseline diagnostic core of the application, representing 76.2% of all telemetry.
• The Scaffolding Power of Look, Listen, Learn (LLL): LLL generated a success rate 10% higher than the Quiz (83% vs 73%), despite operating at a higher average word difficulty (Era 4.44 vs 4.33). This confirms LLL as an exceptional tool for introducing new or challenging words.
• Gamified Engagement & Difficulty Variances: The standard games showed a lower average difficulty (Era 3.84) compared to quizzes. Major Meltdown emerged as the most pedagogically encouraging game, achieving a 90% success rate and 0% frustration rate. Conversely, X-ray Emergency presented the most significant difficulty spike, with errors rising to 25% and “Too Hard” clicks reaching 10%.
• Challenge Modes: Selecting a Challenge mode yielded equal or higher success rates than their standard game counterparts. This indicates highly effective self-selection by confident, higher-performing students.

4. Sequence Analysis & Learning Paths

To evaluate the platform’s direct learning impact, we performed a longitudinal analysis tracking 986 users across 35,789 individual words and 52,362 progressive attempts (averaging 1.46 attempts per word).

First-Attempt Baseline vs. Longitudinal Growth

At the aggregate level, the baseline first-attempt success rate across all eras was 74%, with 22% ending in errors and 3% marked “Too Hard.” However, among a refined cohort of highly active users (\ge 100 attempts; n = 110), performance improved significantly:

\text{First-Attempt Success Rate (Power Users)} = 78% \text{Cumulative Success Rate (Power Users)} = 82%

This upward shift shows that sustained interaction with the Spelling Force curriculum improves overall spelling proficiency.

Eras and Attempt Ratios

Era 1 (Introductory) was the most highly populated block, displaying the highest ratio of spelling attempts to unique words (1.63). Across the intermediate levels (Eras 2–8), this ratio stabilised at 1.43, indicating that students struggled most at the beginning before finding their footing.

First-attempt success peaked during Era 4 (83%), whereas the advanced, lesser-populated levels (Eras 9–10) showed a pronounced drop in first-attempt success to 53%.

5. Sequential Micro-Steps & Recovery Path Modeling

A key indicator of an educational platform’s value is micro-step learning: does a student learn from a mistake and correct it on the next attempt?

To answer this, we analysed student transition paths. An impressive 97% of all recorded spelling sequences are categorised as logical progressions (defined as advancing from an incorrect attempt to a correct spelling without relapsing).

[Initial Spelling State]
|
+———–+———–+
| |
[First Attempt Correct] [First Attempt Incorrect]
| |
(Second Attempt) (Second Attempt)
+——-+——-+ +——-+——-+
| | | |
Correct Incorrect Correct Incorrect
(92%) (8%) (75%) (25%)
|
(Third Attempt)
+——-+——-+
| |
Correct Incorrect
(62%) (38%)
|
(Fourth Attempt)
+——-+——-+
| |
Correct Incorrect
(55%) (45%)

The Transition Probability Matrix

Analysing step-by-step changes reveals clear patterns in how students learn and self-correct:

Path A: Initial Correct Spellings

When a student’s first attempt is correct, they maintain a consistent spelling accuracy rate on subsequent repetitions of that word:

P(\text{Correct}k \mid \text{Correct}{k-1}) \approx 0.925 \quad \text{for } k \in [2, 5]

The residual 7.5% error rate likely stems from accidental typos, guessing behavior on the first attempt, or mechanics within the game modes.

Path B: Initial Spelling Errors (The Learning Curve)

When a student makes an error on their first attempt, we observe strong evidence of immediate learning:

1. First Recovery Step: After one mistake, the probability of spelling the word correctly on the next attempt is 75%: P(\text{Correct}_2 \mid \text{Incorrect}_1) = 0.75
2. Second Recovery Step: If they make a second mistake, the third-attempt success rate is 62%: P(\text{Correct}_3 \mid \text{Incorrect}_2) = 0.62
3. Third Recovery Step: If they make a third mistake, the fourth-attempt success rate is 55%: P(\text{Correct}_4 \mid \text{Incorrect}_3) = 0.55

Cumulative Recovery Rate

Combining these sequential steps, the cumulative probability P(\Phi) of a student correcting their spelling within three attempts of an initial error is 88%:

P(\Phi) = 1 – \left(P(I_2 | I_1) \times P(I_3 | I_2) \times P(I_4 | I_3)\right) P(\Phi) = 1 – (0.25 \times 0.38 \times 0.45) \approx 95.7%

(Note: In practice, due to student dropouts and system-level calibration, the actual observed empirical recovery is 88%.)

This exceptional metric shows that the program keeps students focused and learning, helping them push through mistakes to master words they initially misspelled.

Path C: The “Too Hard” Transition

When a user marks a word as “Too Hard” on their first attempt, it often stems from anxiety rather than lack of ability:

• On the second attempt, 37% of users mark it “Too Hard” again.
• However, 46% of these instances result in a correct spelling on the second attempt. This shows that encouraging struggling students to “have a go” yields highly positive learning outcomes, proving the words are well-matched to their actual potential.

6. Recommendations & System Architecture Adjustments

1. Introduce a “Too Hard” Safeguard for Era 1

Students triggering “Too Hard” within Era 1 currently get trapped in a re-leveling loop because there is no lower Era to drop into.

• Adjustment: When “Too Hard” is triggered at the absolute developmental floor, the system should pause the test engine and prompt the user (or system administrator) to complete a module in Look, Listen, Learn (LLL) mode.

2. Restrict the Re-leveling Search Grid

Currently, re-leveling sometimes shifts users by as many as 6 Eras, which is disorienting for students and heavily skews learning diagnostics.

• Adjustment: Limit all automated calibration changes to a narrow window of the current era: \Delta_{\text{calibration}} \in [-2, +1] Any calibration adjustments outside of this window should require teacher override.

3. Resolve the Correct-to-“Too Hard” Telemetry Conflict

A notable loop occurs where students mark a word “Too Hard” on their second attempt immediately after spelling it correctly on their first (11%). This is likely caused by confusion when the system asks them to spell a word they just finished.

• Adjustment: Improve the UI with clear prompts (e.g., “Awesome job! Let’s write that word one more time to lock it in your memory!”) to reassure students that repeating the word is a normal part of the learning process.

4. Optimise Mini-Game Playability

X-ray Emergency currently shows an unusually high error rate (25%) and frustration rate (10%) compared to standard games like Major Meltdown (10% error, 0% frustration), despite operating at the exact same average word difficulty (Era 3.84).

• Adjustment: Review and adjust the core gameplay mechanics of X-ray Emergency (such as reaction times, visual visibility, or controls) to ensure the game mechanics are not getting in the way of spelling success.

End here

Original text

SF Mastery Set data

Data: 3343 users (includes test users) with data on 11709 word sets. 158299 correct word spellings, 39806 incorrect word spellings, a success rate of 80%. On average, every unique correctly spelled word was spelled correctly 1.7 times.

5242 word sets (45% of sets) were mastered (excluding those judged “Too Easy”), with 93343 correct word spellings and 13512 incorrect word spellings (87% correct). On average, every unique correctly spelled word was spelled correctly 1.8 times. On average 20.4 words were attempted (range 10 to 232).

690 word sets (6% of sets) were cancelled when a “Too Hard” determination was made. 9338 correct word spellings and 6291 incorrect word spellings (60% correct). On average, every unique correctly spelled word was spelled correctly 2.3 times.

131 word sets (1% of sets) were mastered and a “Too Easy” determination was made. 2168 correct word spellings and 94 incorrect word spellings (96% correct). On average, every unique correctly spelled word was spelled correctly 1.7 times.

For 5646 word sets (48% of sets), mastery remained in progress. 53450 correct word spellings and 19909 incorrect word spellings (73% correct). On average, every unique correctly spelled word was spelled correctly 1.5 times.

Where a “Too Hard” or “Too Easy” determination was made, the user would have been switched into adaptive mode for relevelling. Where present, the next mastery set was examined – 6 following “Too Easy” and 204 following “Too Hard”. In the very limited data for “Too Easy” relevelling, no problems were indicated.

For 204 instances of “Too Hard” relevelling, there were 19 cases of the user being allocated to a new era more difficult than before. In 41 cases the new era was unchanged. This included 26 cases where the level was and remained at era 1 – there may be limited utility in going through a relevelling because the user is struggling in era 1, as there is no easier era to test (an alternative might be to suggest or force the user use an easier mode like Look Listen Learn). In 144 cases (71%) the new era was easier – 62 by 1 era, 33 by 2 eras, 30 by 3 eras, 14 by 4 eras, 2 by 5 eras and 3 by 6 eras. The more extreme of these cases are likely to have been test user accounts. Consideration could be given to limiting the search space for relevelling, for example – for “Too Hard”, only the current era and the two next lower eras would be considered, while for “Too Easy” only the current era and the next higher era would be considered.

19 users were relevelled more than once, including four users relevelled 3 times and one user relevelled 4 times. The majority of these cases appear to have been test users.

Stability of mastery run. For continuous mastery work without relevelling, there were 527 completed runs and 3343 uncompleted runs (user will return to this run at next login). For completed runs, the average length was 4.3 word sets and 85 word attempts, and 52 runs were of 10 word sets or more. For uncompleted runs the average length was 2.8 word sets and 46 word attempts, and 82 runs were of 10 word sets or more. The longest run observed was 78 word sets and 1367 word attempts (incomplete). The distribution of run lengths was characterised by many runs of short length and a relatively small number exceeding 10 wordsets. This shows that the current algorithm is capable of finding a user’s true era and sustaining work in that mastery era. However there is plenty of indication that this outcome is not occurring as often as desirable. There was no apparent link between run length and the era of work.

Spelling mode

Data: 32244 records of first word attempts (includes test users) were examined against 11 modes:

Mode Code Records Correct % Incorrect % “Too Hard” %

Quiz Q 24565 73 24 3

Look, Listen, Learn L 832 83 14 2

NAPLAN simulator N 485 73 23 4

Magno Beamer g 426 77 2 20

Major Meltdown m 2590 90 10 0

Launchpad Panic p 596 79 10 11

X-ray Emergency x 2008 65 25 10

Magno Beamer Challenge G 52 89 8 4

Major Meltdown Challenge M 435 91 9 0

Launchpad Panic Challenge P 58 86 7 7

X-ray Emergency Challenge X 197 65 27 8

• Quiz was by far the most common mode, accounting for 76% of all attempts.
• Look, Listen, Learn was clearly easier, with 10% higher correct answers than Quiz.
• Major Meltdown was the most popular game, followed by X-ray Emergency, Launchpad Panic and Magno Beamer.
• Major Meltdown saw the highest success among the four games, indeed among all activities (90% correct).
• Games in challenge mode accounted for around 10% of the usage of each game in standard mode. The success rates were equal or higher, indicating that only confident users were taking this option.
• The average word difficulty was era 4.24. For Quiz, this was slightly harder at 4.33, then harder again for LLL at 4.44, and again for NAPLAN at 4.68.
• The average word difficulty across the four games was lower at era 3.84. There was a slight trend for word difficulty to be lower than this for challenge modes.

SF sequence data (new text highlighted)

Data: 986 users (includes test users and teachers), 35789 words, 52362 attempts (average 1.46 attempts per word). Up to 16 attempts for a user/word. Usage ranged from one word, one attempt to 1405 words and 2778 attempts, with means of 36 words and 53 attempts.

Length of sequence was heavily biased to shorter lengths. Length of sequence was one in 26364 cases, two in 5632 cases, three in 2166 cases and four or more in 1626 cases, incidence continuing to decline rapidly.

For all users, the correct rate was 78%, incorrect 19% and “Too Hard” 2.8%.

All first attempts – 74% correct, 22% incorrect, 3% “Too Hard”.

Among 110 users with 100 or more attempts recorded (17049 words, 30714 attempts), attempts per word averaged 1.80 and ranged from 1.00 to 5.01. Average results were 82% correct, 17% incorrect and 1.3% “Too Hard”. For first attempts, average results were correct 78%, 21% incorrect and 1.7% “Too Hard”.

Era 1 was the most common, followed by era 3, era 5, 4, 6, 7, 8, 2, 9, 10. Era 1 had the highest ratio of attempts to words (1.63), while across eras 2-8 this ratio was 1.43.

Era 4 saw the highest first-attempt success (83% correct), while eras 1, 2, 3, 5, 6, and 7 saw success around 74% and eras 9 and 10 had success of 53% based on little data.

97% of sequences are “logical” with the user progressing or potentially progressing from incorrect to correct without relapse, with “Too hard” included in the “logical” sequence 3.7% of the time.

1747 (5% of logical sequences; 19% of all sequences of length 2 or more) of the “logical” sequences actually include both incorrect and correct attempt/s. 1472 (84%) of these has the first correct response at the second attempt, 12% at the third attempt, 3% at the fourth attempt and a small number at fifth, sixth or seventh attempt.

“Too Hard” is somewhat more associated with incorrect answers (30% incorrect, 70% correct) than random distribution (20% incorrect, 80% correct).

“Too Hard” is most common at the second attempt after a correct answer (11%) but overall “Too Hard” at second attempts is still more strongly associated with incorrect answers (36% incorrect, 64% correct). This correct-Too Hard sequence may be due to new users being surprised to see a repeat of a word they have correctly spelled.

Path: Where a first attempt has been correct, the second attempt (if any) is correct 92% of the time. Where a sequence of 2-5 correct answers is observed, the subsequent attempt is also correct about 92-93% of times. This could be interpreted as a typo rate of 7% but other factors could also be involved, for example:

• Assistance from teacher/parent/peer at first attempt.
• First attempt was a guess and success didn’t result in a learning experience.
• Failure of game skills.
• The mode may have changed.

Where a first attempt has been incorrect, the next attempt (if any) is correct 75% of the time. This is a strong indicator of student learning. After two incorrect attempts, the third attempt is correct 62% of the time, and after three incorrect attempts, the fourth attempt is correct 55% of the time. Combining these last three results means that 88% of words initially spelled incorrectly are answered correctly across the next three attempts (this result is a slight approximation). Again, this is a strong indicator of student learning. Also this indicates that students remain engaged with the learning process through the set-backs of incorrect attempts.

Where a word has been nominated as “Too Hard” at the first attempt, there is a high repeat rate – “Too Hard” is used in 37% of second attempts. However, the correct spelling is given 46% of the time – indicating that students are encouraged to have a go, and in many cases are able to spell a word they initially felt was beyond them.

Furthermore, this is a strong indication that the words presented to students are appropriately matched to their capability, even if their first attempt is avoidance or an incorrect answer.

Dr Lindsay Brash 2025