I ran into this while writing the Rust T-one example. With
enable_endpoint on, the segment after an endpoint came out wrong on the
test file 0.wav:
before: "давайя жду" (start_time 0.6)
after: "давай я жду" (start_time 4.8)
There were three things going on, all T-one only:
1. Reset() put the model back to its initial states in the middle of
the audio. T-one needs some silence after the initial states (that's
why we add left padding at the start of a stream), so the first
words of the next segment were decoded badly. Feeding a fresh stream
from the same point gives the exact same bad output, and adding 0.3s
of padding fixes it. Now we keep the states for T-one and only clear
the decoding result.
2. start_time used 30ms per frame, but GetNumFramesSinceStart() counts
300ms input frames for T-one, so it was 10x too small.
3. IsEndpoint() had the same mix-up: num_processed_frames is in 300ms
frames but was multiplied by 0.03s, so the utterance length was 10x
too short. That made rule2 (must contain non-silence) never fire and
rule3 would need ~200s instead of 20s.
Checked with a local build: the token timestamps in the second segment
now line up with the ones you get without endpointing. Other models are
not touched.
std::to_string(float) keeps six decimals, so a tiny reward such as
1e-7 was forwarded as "0.000000". The decoder already parses and
validates the option, so the CLI now forwards the string unchanged.
Hypotheses of a TDT model advance by their predicted durations, so the
beam holds hypotheses at different frames. They were ranked and pruned
together, although a hypothesis that is ahead has not yet paid for the
frames the others still have to consume. Hypotheses are now kept in
per-frame buckets and only compared within a bucket. Identical token
sequences are merged (log-add). For models without durations, expansion
on a frame stops, as in NeMo's beam search, once max_active_paths
hypotheses on later frames are better than the best one left on it.
Expansion follows NeMo's TDT beam search: blank is always expanded,
with each of the top-k non-zero durations, and non-blank tokens are
chosen as the top-k (token, duration) pairs instead of using only the
argmax duration.
The hotword boost used to select candidate tokens was also added to the
hypothesis score, on top of the context graph score, and was never taken
back when a hotword was not completed. It is now used for the selection
only.
Even with this search, the model gives paths that skip the onset of a
sentence after a pause and then emit blank with near certainty a higher
score than the correct path, so beam search drops whole sentences. Add
length_reward, a reward per emitted token, to counter it (default 0,
i.e. no reward).
Fixes#3267
* CI: smoke-test modified_beam_search with a NeMo TDT model
* Reject non-finite length_reward; keep expanding a frame when the length reward or hotwords can raise its hypotheses
* Pass length_reward per stream via SetOption instead of OfflineRecognizerConfig
Co-authored-by: Hugo Montenegro <hugofmsaiz@gmail.com>
The Kokoro decoder's generator computes the phase of its forward STFT
with atan2(imag, real), which the exported graph implements as an
unguarded imag / real followed by Atan. imag and real are produced by
two Conv nodes. quantize_dynamic also quantized these two convs; at DC
and Nyquist the quantized imag is exactly 0, and with int8 arithmetic
real can become exactly 0 there as well, giving 0 / 0 = NaN.
On macOS arm64 the NaN spreads over the whole output, so short pieces of
text come back as all-NaN audio. On x64 the following
DynamicQuantizeLinear happens to drop the NaN, which hides the problem
but does not remove it.
Pass the two convs as nodes_to_exclude, for v1.0 and v1.1-zh. v0.19
computes the STFT without Conv nodes and is not affected.
The released int8 models kokoro-int8-multi-lang-v1_0 and
kokoro-int8-multi-lang-v1_1 need to be regenerated with this change.
Fixes#4086
VoxLock keeps the voice of one speaker (the caller) and removes the other
voices. It runs in the offline and online speech denoisers next to GTCRN
and DPDFNet, with two ONNX models: an enroll model that runs once on a
voice sample of the caller and a chunk model that runs on each chunk of
the audio and carries the LSTM state.
Without a voice sample, the offline denoiser uses the loudest 3 seconds
in the first 15 seconds of the input, and the online denoiser passes
audio through until it has heard 3 seconds of speech. A voice sample can
be given with --speech-denoiser-voxlock-voice-sample.
The voice sample is optional, so instead of exiting, log a warning and
take the voice sample from the input audio as if none was given.
The prompt is fed at positions 0..n-1, but generated tokens were fed at
1, 2, ... instead of n, n+1, ..., overlapping the prompt. Feed them at
decoder_input.size() + i - 1, as NeMo's GreedySequenceGenerator does.
Fixes#4096
Fixes#3635
It is for v2.0.0 and breaks the ABI.
CircularBuffer::head_ and tail_ are absolute sample counters that grow
forever. As int32_t they overflow after 2^31 samples (~37 h at 16 kHz);
tail_ % capacity then goes negative and std::copy() writes to an invalid
iterator, crashing long-running VAD streams.
Widen every absolute stream position to int64_t. Counts and capacities
(Push/Pop/Get's n, Size(), capacity) stay int32_t — they are bounded by
buffer memory, not stream duration.
Anmol Guragain
Zachary Alexander
XDcobra
Fangjun Kuang
matiaslin
Julian Pscheid
Hugo Montenegro
Nexmoe
Arbaz Siddiqui
Daniil Sterkhov
Hugo MontenegroandHugo Montenegro
Stef
Cassian433
Hassan Kazi
Manickavela
DR1N0
sunerpy