Does 3音 help beginners read and understand Chinese?

Participants

Thirty-eight foreign learners of Chinese were screened (participant codes S10–S47). The protocol admits learners at HSK 0–3, so two who reported HSK 4 or above (S12, S39) were stopped after screening and are excluded, leaving 36 participants: HSK 0 = 6, HSK 1 = 13, HSK 2 = 13, HSK 3 = 4. 15 were women, 12 men, and gender was not recorded for 9. Self-reported characters known had a median of 30 (IQR 12.5–60, range 0–700); 4 of 33 said they could read 汉字.

Procedure and measures

In one facilitated session each participant completed the tasks below, and the facilitator recorded the answers.

• Comprehension (Q1–2). The participant viewed Chinese content in Microsoft Translator and then in the 3音 interpreter (My3中文) and said what they understood. The facilitator estimated the share understood (0–100%, in 5% steps) and whether the participant understood it (Yes/No). The protocol metric C01 is the ratio 3音% ÷ MS Translator%.
• Legibility (Q3). The highest line (1–9, smaller print on each line) the participant could read on a chart in regular PinYin and in 3音 format. The protocol metric C02 = 2.25^{(3音 line − PinYin line)}, which assumes each line carries 1.5² = 2.25 times the information of the previous one. That factor is a property of the chart design; it was not measured here.
• Acceptance (Q4–9). Awareness of similar solutions, liking, ease of use and helpfulness (1–10), intention to use and recommend (Yes/No), expected frequency of use, stated chance of subscribing at $6.99 per month (0–100%), and helpfulness for reading, pronunciation, more information and more understanding (1–10). The usefulness average is the mean of those four items when at least two were answered.

Data preparation

The data were checked for missing values, out-of-range and non-numeric entries, duplicates and internal inconsistencies before analysis. No duplicate participants were found. Each issue and how it was handled is listed in Appendix B. The Yes/No comprehension answers often disagreed with the percentage (for example “No” at 60%), so the percentage is the primary comprehension measure.

Statistical analysis

All tests are two-sided at α = .05. Most variables are bounded ratings or percentages and several distributions are skewed (Shapiro–Wilk on the comprehension differences, p = .023), so rank-based tests are used, with parametric results given as a check.

• Wilcoxon signed-rank for the paired 3音-versus-comparison outcomes, because each participant was measured under both conditions and the differences are not normal. Effect size is the matched-pairs rank-biserial correlation (r_rb); Cohen’s d_z and a paired t-based 95% CI of the mean difference are also given.
• McNemar exact test for the paired Yes/No comprehension answers, because it uses only the participants whose answer changed.
• Geometric means with log-scale CIs, and bootstrap CIs of the median (10,000 resamples) for the ratio metrics C01 and C02, because ratios are right-skewed and a few small denominators dominate an arithmetic mean.
• One-sample Wilcoxon against the scale midpoint (5.5) for the ratings, to test whether opinion leans positive.
• Wilson score intervals for Yes/No proportions, because they stay accurate at small n and near 0% or 100%.
• Friedman test with Holm-corrected Wilcoxon post-hoc tests to compare the four helpfulness items, because the same people rated all four.
• Kruskal–Wallis (effect size ε²), Spearman ρ and Mann–Whitney (r_rb) for exploratory differences by HSK level, characters known and gender. The 24 exploratory group tests are Holm-corrected as one family; five exploratory correlations with willingness to pay form a separate Holm family.

Analyses were run in Python (SciPy 1.17) on the available cases for each test, and each result reports its own n.

With Microsoft Translator, participants understood a mean of 13.8% of the content (median 10%). With 3音 they understood 64.4% (median 70%). The mean paired gain was +50.6 pp (95% CI 42.6–58.6; median +55 pp, bootstrap 95% CI 45–60). All 29 participants whose score changed understood more with 3音, and 2 were unchanged (Wilcoxon W = 0, p < .001, r_rb = 1.00). A paired t-test agrees (p < .001, d_z = 2.32). On the Yes/No question, 2 of 26 said they understood the content with Microsoft Translator and 20 of 26 with 3音 (McNemar exact, p < .001).

The protocol ratio C01 had a median of 6.0× (bootstrap 95% CI 4.0×–10.0×) and a geometric mean of 5.9× (4.3×–8.1×). Its arithmetic mean (8.5×) is pulled up by two of the three “1%” Microsoft Translator scores (S13 at 15× and S32 at 50×), so it should not be quoted. If values below 5% are raised to 5%, the geometric mean is 5.3× and the median is unchanged (Table 3).

On the regular PinYin chart the smallest line read had a mean of 3.74 (median 4); in 3音 format it was 6.48 (median 6). All 31 participants read further in 3音, by a mean of 2.74 lines (95% CI 2.43–3.06; W = 0, p < .001, r_rb = 1.00). The gains were +1 line for 2 participants, +2 for 10, +3 for 13 and +4 for 6. Using the protocol’s 2.25× per line, this is a geometric mean of 9.2× more readable information (C02, 95% CI 7.2×–11.9×). The line gain is the measured result; the C02 multiple holds only if the chart’s 2.25× step is right.

Every rating was above the scale midpoint (Wilcoxon against 5.5, all p ≤ .013). Liking (mean 8.69; 89% rated 8 or higher) and overall helpfulness (8.69) were highest. Ease of use was lowest (mean 6.36, 95% CI 5.85–6.87); only 25% rated it 8 or higher, and one participant (S21) rated it 2. Liking and ease were not correlated (ρ = .10, p = .553), so participants liked the tool even when they found it hard to use.

The four helpfulness items differed (Friedman χ²(3) = 37.2, p < .001, Kendall’s W = 0.69, n = 18 with all four answered; complete-case means 8.39, 8.06, 6.89 and 6.50). Only half the sample answered all four items, and “more understanding” was skipped most often, so this comparison may not represent everyone. Using all available answers, reading (8.35) and pronunciation (7.96) were rated above “more information” (7.19) and “more understanding” (6.54); all four of those pairwise comparisons held after Holm correction (Appendix A). Participants therefore saw 3音 mainly as a reading and pronunciation aid. “More understanding” also had the most missing answers (12 of 36).

27 of 31 participants (87%, 95% CI 71%–95%) would use 3音, and 23 of 27 (85%) would recommend it. None of 31 knew a similar tool they could use now (95% CI 0–11%), so that question has no variation to analyse. Expected use had a median of 2 times a day (IQR 0.5–4.25).

The mean stated chance of subscribing at $6.99 a month was 44.8% (95% CI 34.1–55.6; median 50%). 20 of 33 (61%) gave 50% or more and 7 (21%) gave 0%. Stated purchase probabilities usually overestimate real conversion, so this figure is an upper bound on demand. Of five exploratory associations tested with it, only liking reached p < .05 (ρ = .35, p = .046), and not after Holm correction across those five (adjusted p = .232); the comprehension gain was not (ρ = .11, p = .571).

The comprehension gain rose with HSK level: a median of +7 pp at HSK 0 (n = 4), +49 at HSK 1, +60 at HSK 2 and +55 at HSK 3 (Kruskal–Wallis H = 14.1, p = .003, ε² = .47). The monotonic trend held after Holm correction for HSK level (ρ = .60, Holm-adjusted p = .008) and for characters known (ρ = .55, Holm-adjusted p = .036). Two of the four HSK 0 participants gained nothing. One of them (S17) also does not know PinYin, so lack of PinYin is an equally plausible explanation, and the percentage-point gain has less room at higher levels (HSK 3 baseline median 35%). With cells this small, the pattern is an association to confirm, not an established moderator.

Several other patterns reached p < .05 before correction but not after it, and should be treated as hypotheses. Ease of use rose with HSK level (ρ = .37) and was higher for women (median 7 vs 5.5). The usefulness average fell with HSK level (ρ = -.44; HSK 3 mean 6.2 vs 8.2 at HSK 1). Liking and the legibility gain did not differ by group. Descriptively, the HSK 3 group gave the highest subscription chance (mean 75%, n = 4), but this difference was not significant even before correction (Kruskal–Wallis p = .099).

Under test conditions 3音 produced a large, consistent improvement on both measured tasks. Participants understood about six times as much content (median C01) as with Microsoft Translator (roughly +50 percentage points), and every participant could read smaller print in 3音 format than in regular PinYin. Effects this uniform (no participant did worse on either task, with S22’s out-of-range reading entry excluded) are unlikely to come from sampling noise alone, even with n ≈ 30, and the conclusions do not depend on the exact test chosen. Chance is not the only concern, though: an unblinded facilitator and a fixed tool order could both inflate a uniform effect, and this design cannot rule them out (§4).

The group analysis suggests who benefits most. The comprehension gain rose with HSK level and characters known, and the complete beginners gained little. This is consistent with a tool that helps learners use knowledge they already have, and it points to learners at roughly HSK 1–3 as the best fit. The evidence is thin at the extremes: only four HSK 0 and three HSK 3 participants had comprehension data, and one of the HSK 0 non-gainers did not know PinYin. The pattern should be confirmed with a larger sample.

Opinions were very positive on value (liking, helpfulness, intention to use) and weakest on ease of use, the only rating whose confidence interval sits near the scale midpoint. Ease was descriptively lowest at HSK 0 (mean 5.2, n = 6), but the difference between HSK groups was not significant, so whether beginners in particular find 3音 hard to use is a hypothesis for the next study. Participants rated 3音 most helpful for reading and pronunciation.

Stated willingness to pay was moderate: a mean chance of 45% at $6.99 a month, with one in five at 0%. Stated intentions tend to overstate actual purchases, so this is evidence of interest rather than a conversion estimate.

1. Measurement. Comprehension was a facilitator’s estimate of what the participant said they understood, not a scored test, and the facilitator knew which tool was in use. The Yes/No answers were often inconsistent with the percentages, and some values (1%) fall outside the protocol’s 5% grid.
2. Order and familiarity. The results record does not say whether the order of the two tools, or the text used with each, was counterbalanced. If 3音 always came second, or used the same passage, part of the gain may reflect repeated exposure.
3. Derived metrics. C02 depends on the assumed 2.25× information step between lines. C01 is a ratio with small denominators, so it is unstable for learners who understood almost nothing with Microsoft Translator (for example, 1% to 15% is recorded as 15×).
4. Sample. This was a small convenience sample, probably recruited through the developer’s network, which invites courtesy bias in ratings and intentions. Subgroups were small (HSK 3, n = 4), gender was missing for a quarter of participants, and several items had 20–33% non-response.
5. Scope. This was a single session with no follow-up, so it measures immediate reading support, not learning over time. The only comparison was Microsoft Translator; no other learner-focused tools were tested. Willingness to pay was a stated probability, not a purchase.
6. Multiple testing. Group comparisons were exploratory. Only the HSK and character-count trend in comprehension gain held after correction.

Recommended next steps: counterbalance tool order and passages, score comprehension with fixed questions marked by someone who does not know which tool was used, record the reading-chart font sizes so C02 can be checked, and recruit at least 10 participants per HSK level.