The Kokoro decoder's generator computes the phase of its forward STFT
with atan2(imag, real), which the exported graph implements as an
unguarded imag / real followed by Atan. imag and real are produced by
two Conv nodes. quantize_dynamic also quantized these two convs; at DC
and Nyquist the quantized imag is exactly 0, and with int8 arithmetic
real can become exactly 0 there as well, giving 0 / 0 = NaN.
On macOS arm64 the NaN spreads over the whole output, so short pieces of
text come back as all-NaN audio. On x64 the following
DynamicQuantizeLinear happens to drop the NaN, which hides the problem
but does not remove it.
Pass the two convs as nodes_to_exclude, for v1.0 and v1.1-zh. v0.19
computes the STFT without Conv nodes and is not affected.
The released int8 models kokoro-int8-multi-lang-v1_0 and
kokoro-int8-multi-lang-v1_1 need to be regenerated with this change.
Fixes#4086
VoxLock keeps the voice of one speaker (the caller) and removes the other
voices. It runs in the offline and online speech denoisers next to GTCRN
and DPDFNet, with two ONNX models: an enroll model that runs once on a
voice sample of the caller and a chunk model that runs on each chunk of
the audio and carries the LSTM state.
Without a voice sample, the offline denoiser uses the loudest 3 seconds
in the first 15 seconds of the input, and the online denoiser passes
audio through until it has heard 3 seconds of speech. A voice sample can
be given with --speech-denoiser-voxlock-voice-sample.
The voice sample is optional, so instead of exiting, log a warning and
take the voice sample from the input audio as if none was given.
The prompt is fed at positions 0..n-1, but generated tokens were fed at
1, 2, ... instead of n, n+1, ..., overlapping the prompt. Feed them at
decoder_input.size() + i - 1, as NeMo's GreedySequenceGenerator does.
Fixes#4096
Fixes#3635
It is for v2.0.0 and breaks the ABI.
CircularBuffer::head_ and tail_ are absolute sample counters that grow
forever. As int32_t they overflow after 2^31 samples (~37 h at 16 kHz);
tail_ % capacity then goes negative and std::copy() writes to an invalid
iterator, crashing long-running VAD streams.
Widen every absolute stream position to int64_t. Counts and capacities
(Push/Pop/Get's n, Size(), capacity) stay int32_t — they are bounded by
buffer memory, not stream duration.
NeMo's AudioToMelSpectrogramPrefab computes log(x + 2**-24), while the
shared fbank extractor applies a hard floor, log(max(x, FLT_EPSILON)).
On quiet recordings a large share of the mel bins sit at that floor and
per-feature normalization turns them into constant columns the model
never saw in training, which degrades accuracy a lot (issue #3997).
For NeMo models (nemo_normalize_type is set), disable use_log_fbank and
apply log(x + 2**-24) to the raw mel energies in GetFrames() before the
per-feature normalization. Other models keep the existing behavior.
Bump the dependency and select the additive guard with eps = FLT_EPSILON / 2 for NeMo fbank models. The manual log pass in GetFrames() is no longer needed, which also keeps MFCC models such as TeleSpeech untouched.
The wav2vec2-style convolutional front end of omnilingual ASR models needs at
least 400 samples. For shorter input onnxruntime throws an Ort::Exception from
DecodeStream that nothing catches, so sherpa-onnx-offline and C API users abort,
Python users get a RuntimeError, and a batch passed to DecodeStreams loses the
results of every stream. Empty input also reached f.size() / feat_dim with
feat_dim == 0. Return an empty result for such input instead.
Both examples passed each 0.1 s read straight to accept_waveform(), which expects
window-sized input. Buffer the reads and feed window_size samples per call, as the
other VAD examples do.
A lead byte announcing a 2-4 byte character near the end of the input was
copied as a whole character even when fewer bytes remained, reading past
the end of the string. This happens for TTS text that ends in the middle
of a character, which reaches SplitUtf8 through the lexicon front ends.
Treat such a lead byte like any other invalid byte and skip it.
The timestamps were built as str(timedelta(seconds=t))[:-3]. That only
works when t has a fractional part: for a whole number of seconds
str(timedelta) has no ".ffffff" suffix, so the slice cuts off the
seconds instead, e.g. a segment starting at 0 s was written as
"0:00 --> 0:00:01,500". Hours were also written with a single digit,
while SRT expects HH:MM:SS,mmm.
Format the timestamps explicitly as HH:MM:SS,mmm for both the .srt
and the .txt output.
Fangjun Kuang
Stef
Cassian433
Hassan Kazi
Manickavela
Arbaz Siddiqui
DR1N0
sunerpy
Kyozzz
Emmanuel BORGES
losewayy
Sergio Fernández
Hulk
dxhome
Anthony
olerast
Mohammad Hijjawi
Pratik Gandhi