We introduce Haru-4B, a full-duplex continuous spoken dialogue foundation model capable of listening and speaking simultaneously.

Haru-4B focuses on natural human-AI interaction, learning conversational behaviors such as backchannels, interruptions, and turn-taking from large-scale real-world speech, rather than relying on synthetic conversations or explicitly scripted turn-taking behavior. Our goal is to make interacting with AI feel more like a natural human conversation, unlike interacting with agentic assistants.

We are releasing Haru-4B as a pretrained base model, providing a foundation for further fine-tuning and research on real-time spoken interaction. The model weights are available on Hugging Face, and the code is available on GitHub.

Haru-4B at a Glance

  • Ultra-Low-Latency Interaction: About 65 ms end-to-end latency on a MacBook.
  • 7.5 Hz Native Full-Duplex Speech-to-Speech: Simultaneous processing of incoming and outgoing speech every 133 ms through synchronized audio-text streams.
  • Single-Step Continuous Audio Generation: A lightweight MLP head with one-step MeanFlow sampling, eliminating the need for an additional acoustic Transformer.
  • 40 Million Hours of Real Speech: Large-scale multilingual pretraining without incorporating LLM- or TTS-generated turn-based synthetic dialogues.
  • Multilingual Speech Capabilities: Training data spanning over 160 languages, with a primary focus on nine languages: English, Spanish, Portuguese, German, French, Chinese, Japanese, Russian, and Korean.
  • Long-Form Full-Duplex Training: Trained on full-duplex conversations up to 60 minutes long, enabling the model to learn extended conversational dynamics and long-range interaction patterns.
  • Controllable Character and Voice: Supports character customization through text prompts and reference audio, enabling control over persona, speaking style, and voice characteristics.
  • Open Pretrained Base Model: We are releasing model weights (Hugging Face), code (GitHub), and demos to support further research.

1. Architecture: Native Full-Duplex at 7.5 Hz

Haru-4B is built on Qwen3.5-4B 1 and designed to process incoming and outgoing speech simultaneously within a unified language model. Its architecture combines low-rate continuous audio representations, synchronized multi-stream processing, and efficient single-step generation.

Cascaded, turn-based, interleaved, and multi-stream modeling of spoken dialogue

Native full-duplex multi-stream modeling. Haru processes three synchronized streams: incoming user audio, outgoing model audio, and model text 2 3. Their embeddings are combined through element-wise addition and processed by a shared language model. This enables simultaneous listening and speaking 4 without interleaving the three streams token by token 5 6. Compared with interleaving, this reduces the context length to one third while keeping the text stream uninterrupted.

Discrete, iterative-flow, and single-step continuous speech generation

Continuous speech tokens at 7.5 Hz. Most spoken language models rely on discrete audio tokens, often produced through Residual Vector Quantization (RVQ) 7 8, where each audio frame is represented by multiple discrete codebook indices. In contrast, Haru uses 64-dimensional continuous acoustic latents at just 7.5 Hz 9, representing approximately 133 ms of speech per token. By avoiding discrete codebooks and operating at a low token rate, Haru reduces the number of sequential LLM modeling steps required for training and inference.

Single-step continuous speech generation. Haru generates continuous acoustic tokens directly from language-model hidden states using a lightweight Multi-Layer Perceptron (MLP) within a next-latent diffusion framework 10 11 12. We primarily train the generation head with Flow Matching 13, followed by MeanFlow distillation 14 15 for training and inference efficiency. At inference time, each acoustic token is generated in a single sampling step, without iterative flow sampling or an additional acoustic Transformer.

Token-level and chunk-level text alignment

Chunk-level text alignment. Instead of aligning text and audio at the individual token level, Haru represents text in chunks 3, followed by padding positions and a dedicated end-of-padding token for predicting speech initiation. This accommodates the different temporal resolutions of text and audio while maintaining synchronization across streams and mitigating catastrophic forgetting of pretrained language-model knowledge.

Together, these design choices enable efficient large-scale speech pretraining while reducing inference to just 7.5 autoregressive audio steps per second, making low-latency real-time full-duplex interaction practical even on local hardware.

2. Scaling Pretraining with 40 Million Hours of Real Speech

We believe large-scale real-world speech is essential for learning the dynamics of natural interaction. Unlike synthetic conversations, real recordings contain spontaneous interruptions, hesitations, backchannels, overlapping speech, and irregular turn boundaries.

Haru-4B is pretrained using over 40 million hours of speech, covering more than 160 languages, with nine primary languages. To learn natural human conversational patterns, we intentionally exclude LLM-generated dialogues and TTS-synthesized speech from our training mixture.

Our pretraining follows two stages.

Stage 1: Single-Channel Speech Pretraining

We first adapt Qwen3.5-4B to speech using a mixture of:

  • Automatic speech recognition, connecting speech and linguistic representations.
  • Speech and text continuation, learning to model and generate continuous speech.
  • Text-only rehearsal, mitigating catastrophic forgetting of pretrained language-model knowledge 16.

Speech restoration with Sidon. We process most of our speech data using Sidon 17 to restore clean audio for pretraining. We have also open-sourced optimized Triton kernels for Sidon to accelerate large-scale speech preprocessing: Sidon-Triton.

Stage 2: Full-Duplex Pretraining

Full-duplex data reconstruction. Large-scale full-duplex training requires synchronized speech streams, but most real-world conversational recordings are available only as single-channel mixtures. To address this, we developed FastDialogueSidon, a fast joint speech separation and restoration model that converts mixed conversational recordings into temporally aligned full-duplex streams. The FastDialogueSidon model and implementation are also publicly available: FastDialogueSidon.

Large-scale long-form full-duplex training. We use over 13 million hours of separated full-duplex dialogue data. Training samples include conversations up to 60 minutes long, allowing the model to learn long-range conversational dynamics, including extended turn-taking and overlapping speech.

Character and voice conditioning. We generate a character prompt for each training sample and use reference audio from the same speaker to condition speech generation. This enables Haru to control character personality and voice characteristics through text prompts and audio references.

3. Efficient Large-Scale Pretraining and Real-Time Inference

3.1. Preprocessing and Training Optimization

Accelerated speech preprocessing. Processing speech at this scale requires substantial computational resources. We optimize preprocessing models, including the neural audio codec, Sidon, and FastDialogueSidon, using custom Triton kernels 18 to accelerate large-scale data preparation.

Large-scale distributed training. Haru-4B was pretrained using 256 NVIDIA B200 GPUs over approximately 8 days. Our training infrastructure is built on Megatron-LM 19, with optimizations for efficient large-scale speech pretraining.

3.2. Real-Time Inference Optimization

Ultra-low-latency inference on Apple Silicon. On a MacBook with Apple M1 Max, our native Swift app with optimized custom Metal kernels achieves an end-to-end latency of approximately 65 ms, including acoustic encoding, language-model inference, audio decoding, and wired microphone and headphone I/O, excluding audio framing. With a 7.5 Hz token rate, audio framing introduces up to 133 ms of additional algorithmic latency.

The breakdown below is for int8 weights.

Stage Latency (int8)
Input (microphone → app) 5.3 ± 0.0 ms
One model step 34.2 ± 0.5 ms
Socket and handoff 4.1 ± 0.3 ms
Output wait (15 ms margin and slack) 13.3 ± 1.5 ms
Output (app → headphones) 8.1 ± 0.2 ms
Total (p50) 65.2 ± 1.5 ms

High-performance GPU serving. We also support GPU deployment through vLLM-Omni 20 21. On an NVIDIA B300 GPU, audio encoding and decoding together with one language-model step take about 15 ms.

Together, these optimizations enable efficient large-scale pretraining and low-latency real-time full-duplex interaction on both Apple Silicon and GPU servers.

4. Demo

Without training on any natively recorded two-channel conversations, our model pretrained only on separated speech achieves natural conversation, including backchannels and overlapping speech.

Investment advice (EN). A conversation with a financial advisor about investing. Haru understands the content and intent of the user’s remarks and offers logical, balanced advice. It maintains a natural conversational pace while providing consistent, context-aware responses.

5. Future Work

We are releasing Haru-4B as a large-scale pretrained foundation model. Going forward, we are exploring several directions to further improve its conversational abilities and extend real-time interaction beyond speech.

Beyond separated speech. Separated speech makes full-duplex dialogue pretraining scalable, but separation errors and noise can remain, especially in overlapping speech. To further improve natural full-duplex interaction, we are fine-tuning Haru on native full-duplex two-channel conversational recordings. We are also developing full-duplex dialogue and speech generation models to create natural synthetic full-duplex data that enhances response quality, conversational diversity, and character-specific dialogue capabilities.

Scaling up to 100B+ parameters. Although Haru-4B demonstrates efficient full-duplex speech modeling, its relatively small language-model backbone limits its response quality. We are already pretraining larger full-duplex models with over 100 billion parameters, aiming to combine natural conversational timing with stronger language understanding and knowledge.

Toward omni-modal full-duplex interaction. Human interaction involves more than speech. Visual context, facial expressions, and nonverbal cues all contribute to natural conversation. We are extending our architecture to take real-time video input, moving toward omni-modal full-duplex dialogue models that can see, listen, and speak continuously.

Open foundations for future research. We plan to release a more detailed technical report and training code.

We believe native full-duplex interaction is an important step toward more natural human-AI communication. If you’re interested in building the next generation of full-duplex models with us, we’d love to hear from you: contact@higurashi.ai

6. Acknowledgments

The development of Haru-4B was supported by Amazon Web Services (AWS) and by the Generative AI Accelerator Challenge (GENIAC), a project led by Japan’s Ministry of Economy, Trade and Industry (METI) and the New Energy and Industrial Technology Development Organization (NEDO).

References

  1. Qwen Team, “Qwen3.5-4B,” model card, Hugging Face, 2026. [Online]. Available: https://huggingface.co/Qwen/Qwen3.5-4B ↩︎

  2. A. Défossez et al., “Moshi: A speech-text foundation model for real-time dialogue,” arXiv preprint arXiv:2410.00037, 2024. ↩︎

  3. Y. Yao et al., “FLM-Audio: Natural monologues improves native full-duplex chatbots via dual training,” arXiv preprint arXiv:2509.02521, 2025. ↩︎ ↩︎

  4. T. A. Nguyen et al., “Generative spoken dialogue language modeling,” Transactions of the Association for Computational Linguistics, vol. 11, 2023. ↩︎

  5. B. Veluri, B. N. Peloquin, B. Yu, H. Gong, and S. Gollakota, “Beyond turn-based interfaces: Synchronous LLMs as full-duplex dialogue agents,” in Proc. Conference on Empirical Methods in Natural Language Processing (EMNLP), 2024. ↩︎

  6. Q. Zhang et al., “OmniFlatten: An end-to-end GPT model for seamless voice conversation,” in Proc. Annual Meeting of the Association for Computational Linguistics (ACL), 2025. ↩︎

  7. N. Zeghidour, A. Luebs, A. Omran, J. Skoglund, and M. Tagliasacchi, “SoundStream: An end-to-end neural audio codec,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 30, 2022. ↩︎

  8. A. Défossez, J. Copet, G. Synnaeve, and Y. Adi, “High fidelity neural audio compression,” Transactions on Machine Learning Research, 2023. ↩︎

  9. Z. Peng et al., “VibeVoice: Expressive podcast generation with next-token diffusion,” in Proc. International Conference on Learning Representations (ICLR), 2026. ↩︎

  10. T. Li, Y. Tian, H. Li, M. Deng, and K. He, “Autoregressive image generation without vector quantization,” in Advances in Neural Information Processing Systems (NeurIPS), 2024. ↩︎

  11. Y. Sun et al., “Multimodal latent language modeling with next-token diffusion,” arXiv preprint arXiv:2412.08635, 2024. ↩︎

  12. S. Rouard, M. Orsini, A. Roebel, N. Zeghidour, and A. Défossez, “Continuous audio language models,” in Proc. International Conference on Learning Representations (ICLR), 2026. ↩︎

  13. Y. Lipman, R. T. Q. Chen, H. Ben-Hamu, M. Nickel, and M. Le, “Flow matching for generative modeling,” in Proc. International Conference on Learning Representations (ICLR), 2023. ↩︎

  14. Z. Geng, M. Deng, X. Bai, J. Z. Kolter, and K. He, “Mean flows for one-step generative modeling,” in Advances in Neural Information Processing Systems (NeurIPS), 2025. ↩︎

  15. Z. Geng, Y. Lu, Z. Wu, E. Shechtman, J. Z. Kolter, and K. He, “Improved mean flows: On the challenges of fastforward generative models,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2026. ↩︎

  16. A. Ibrahim et al., “Simple and scalable strategies to continually pre-train large language models,” Transactions on Machine Learning Research, 2024. ↩︎

  17. W. Nakata, Y. Saito, Y. Ueda, and H. Saruwatari, “Sidon: Fast and robust open-source multilingual speech restoration for large-scale dataset cleansing,” in Proc. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2026. ↩︎

  18. P. Tillet, H. T. Kung, and D. Cox, “Triton: An intermediate language and compiler for tiled neural network computations,” in Proc. ACM SIGPLAN International Workshop on Machine Learning and Programming Languages (MAPL), 2019. ↩︎

  19. M. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro, “Megatron-LM: Training multi-billion parameter language models using model parallelism,” arXiv preprint arXiv:1909.08053, 2019. ↩︎

  20. W. Kwon et al., “Efficient memory management for large language model serving with PagedAttention,” in Proc. ACM Symposium on Operating Systems Principles (SOSP), 2023. ↩︎

  21. vLLM-Omni Team, “vLLM-Omni technical report: A unified serving runtime for omni-modality generation,” arXiv preprint arXiv:2610.09307, 2026. ↩︎