Benchmarks¶
Generated by make bench on Apple M3, Python 3.11.13. Every number comes from the scripts in bench/; rerun them to check.
Speed¶
| Check | p50 | p95 | p99 | calls |
|---|---|---|---|---|
PrivacyGuard.check (74 replies, 6.6 words on average) |
0.053 ms | 0.208 ms | 0.246 ms | 2960 |
ApprovalGuard.start + reply |
0.024 ms | 0.026 ms | 0.032 ms | 1000 |
Voice ID on LibriSpeech¶
LibriSpeech test-clean (CC BY 4.0), 40 speakers, 3 enrollment + 8 test utterances each, seed 7. speechbrain/spkrec-ecapa-voxceleb + Silero VAD, CPU on Apple M3.
Every cut of speech scored against every enrolled speaker, at the packaged thresholds:
| Speech | Owner accepted | Stranger accepted | EER | No score | Score time (p50) |
|---|---|---|---|---|---|
| 0.5 s | 68 of 320 (21.2%) | 0 of 12480 (0%) | 5.9% | 0 of 12800 (0%) | 10.9 ms |
| 1.0 s | 235 of 320 (73.4%) | 1 of 12480 (0.008%) | 1.6% | 40 of 12800 (0.3%) | 16.1 ms |
| 2.0 s | 308 of 320 (96.2%) | 1 of 12480 (0.008%) | 0.6% | 0 of 12800 (0%) | 26.0 ms |
| 3.0 s | 315 of 320 (98.4%) | 2 of 12480 (0.016%) | 0.4% | 0 of 12800 (0%) | 36.0 ms |
End to end through ApprovalGuard (20 other speakers, voice tier, reply 'yes', command 3.0 s, both from the same person). Calibrated thresholds (accept 0.293, reject 0.201) come from saysafe.calibrate on 20 dev speakers, 1.0 s cuts (159 owner, 3021 stranger scores).
| Thresholds, reply length | Owner approved by voice | Stranger approved |
|---|---|---|
| packaged, reply 0.5 s | 126 of 160 (78.8%) | 0 of 760 (0%) |
| packaged, reply 1.0 s | 151 of 160 (94.4%) | 0 of 760 (0%) |
| calibrated, reply 0.5 s | 147 of 160 (91.9%) | 0 of 760 (0%) |
| calibrated, reply 1.0 s | 159 of 160 (99.4%) | 2 of 760 (0.3%) |
An owner who isn't approved by voice isn't refused: the step goes to a phone tap. A stranger who is approved has passed the voice tier, which the default policy keeps for purchases under $75 and emails to known contacts.
LibriSpeech is clean read speech, close to the best case. A wearable mic in a noisy room will do worse, and recordings or voice clones of the owner aren't tested here: calibrate on your own recordings, and read the threat model.
Private replies¶
Not measured yet: 0 of 140 labels reviewed. The benchmark only counts replies a person has labelled.