aud22_38.yaml

- en: Device ASR with Emformer RNN-T
  prefs:
  - PREF_H1
  type: TYPE_NORMAL
- en: 原文：[https://pytorch.org/audio/stable/tutorials/device_asr.html](https://pytorch.org/audio/stable/tutorials/device_asr.html)
  prefs:
  - PREF_BQ
  type: TYPE_NORMAL
- en: Note
  prefs: []
  type: TYPE_NORMAL
- en: Click [here](#sphx-glr-download-tutorials-device-asr-py) to download the full
    example code
  prefs: []
  type: TYPE_NORMAL
- en: '**Author**: [Moto Hira](mailto:moto%40meta.com), [Jeff Hwang](mailto:jeffhwang%40meta.com).'
  prefs: []
  type: TYPE_NORMAL
- en: This tutorial shows how to use Emformer RNN-T and streaming API to perform speech
    recognition on a streaming device input, i.e. microphone on laptop.
  prefs: []
  type: TYPE_NORMAL
- en: Note
  prefs: []
  type: TYPE_NORMAL
- en: This tutorial requires FFmpeg libraries. Please refer to [FFmpeg dependency](../installation.html#ffmpeg-dependency)
    for the detail.
  prefs: []
  type: TYPE_NORMAL
- en: Note
  prefs: []
  type: TYPE_NORMAL
- en: This tutorial was tested on MacBook Pro and Dynabook with Windows 10.
  prefs: []
  type: TYPE_NORMAL
- en: This tutorial does NOT work on Google Colab because the server running this
    tutorial does not have a microphone that you can talk to.
  prefs: []
  type: TYPE_NORMAL
- en: 1\. Overview[](#overview "Permalink to this heading")
  prefs:
  - PREF_H2
  type: TYPE_NORMAL
- en: We use streaming API to fetch audio from audio device (microphone) chunk by
    chunk, then run inference using Emformer RNN-T.
  prefs: []
  type: TYPE_NORMAL
- en: For the basic usage of the streaming API and Emformer RNN-T please refer to
    [StreamReader Basic Usage](./streamreader_basic_tutorial.html) and [Online ASR
    with Emformer RNN-T](./online_asr_tutorial.html).
  prefs: []
  type: TYPE_NORMAL
- en: 2\. Checking the supported devices[](#checking-the-supported-devices "Permalink
    to this heading")
  prefs:
  - PREF_H2
  type: TYPE_NORMAL
- en: Firstly, we need to check the devices that Streaming API can access, and figure
    out the arguments (`src` and `format`) we need to pass to [`StreamReader()`](../generated/torchaudio.io.StreamReader.html#torchaudio.io.StreamReader
    "torchaudio.io.StreamReader") class.
  prefs: []
  type: TYPE_NORMAL
- en: We use `ffmpeg` command for this. `ffmpeg` abstracts away the difference of
    underlying hardware implementations, but the expected value for `format` varies
    across OS and each `format` defines different syntax for `src`.
  prefs: []
  type: TYPE_NORMAL
- en: The details of supported `format` values and `src` syntax can be found in [https://ffmpeg.org/ffmpeg-devices.html](https://ffmpeg.org/ffmpeg-devices.html).
  prefs: []
  type: TYPE_NORMAL
- en: For macOS, the following command will list the available devices.
  prefs: []
  type: TYPE_NORMAL
- en: '[PRE0]'
  prefs: []
  type: TYPE_PRE
- en: We will use the following values for Streaming API.
  prefs: []
  type: TYPE_NORMAL
- en: '[PRE1]'
  prefs: []
  type: TYPE_PRE
- en: For Windows, `dshow` device should work.
  prefs: []
  type: TYPE_NORMAL
- en: '[PRE2]'
  prefs: []
  type: TYPE_PRE
- en: In the above case, the following value can be used to stream from microphone.
  prefs: []
  type: TYPE_NORMAL
- en: '[PRE3]'
  prefs: []
  type: TYPE_PRE
- en: 3\. Data acquisition[](#data-acquisition "Permalink to this heading")
  prefs:
  - PREF_H2
  type: TYPE_NORMAL
- en: Streaming audio from microphone input requires properly timing data acquisition.
    Failing to do so may introduce discontinuities in the data stream.
  prefs: []
  type: TYPE_NORMAL
- en: For this reason, we will run the data acquisition in a subprocess.
  prefs: []
  type: TYPE_NORMAL
- en: Firstly, we create a helper function that encapsulates the whole process executed
    in the subprocess.
  prefs: []
  type: TYPE_NORMAL
- en: This function initializes the streaming API, acquires data then puts it in a
    queue, which the main process is watching.
  prefs: []
  type: TYPE_NORMAL
- en: '[PRE4]'
  prefs: []
  type: TYPE_PRE
- en: The notable difference from the non-device streaming is that, we provide `timeout`
    and `backoff` parameters to `stream` method.
  prefs: []
  type: TYPE_NORMAL
- en: When acquiring data, if the rate of acquisition requests is higher than that
    at which the hardware can prepare the data, then the underlying implementation
    reports special error code, and expects client code to retry.
  prefs: []
  type: TYPE_NORMAL
- en: Precise timing is the key for smooth streaming. Reporting this error from low-level
    implementation all the way back to Python layer, before retrying adds undesired
    overhead. For this reason, the retry behavior is implemented in C++ layer, and
    `timeout` and `backoff` parameters allow client code to control the behavior.
  prefs: []
  type: TYPE_NORMAL
- en: For the detail of `timeout` and `backoff` parameters, please refer to the documentation
    of `stream()` method.
  prefs: []
  type: TYPE_NORMAL
- en: Note
  prefs: []
  type: TYPE_NORMAL
- en: The proper value of `backoff` depends on the system configuration. One way to
    see if `backoff` value is appropriate is to save the series of acquired chunks
    as a continuous audio and listen to it. If `backoff` value is too large, then
    the data stream is discontinuous. The resulting audio sounds sped up. If `backoff`
    value is too small or zero, the audio stream is fine, but the data acquisition
    process enters busy-waiting state, and this increases the CPU consumption.
  prefs: []
  type: TYPE_NORMAL
- en: 4\. Building inference pipeline[](#building-inference-pipeline "Permalink to
    this heading")
  prefs:
  - PREF_H2
  type: TYPE_NORMAL
- en: The next step is to create components required for inference.
  prefs: []
  type: TYPE_NORMAL
- en: This is the same process as [Online ASR with Emformer RNN-T](./online_asr_tutorial.html).
  prefs: []
  type: TYPE_NORMAL
- en: '[PRE5]'
  prefs: []
  type: TYPE_PRE
- en: '[PRE6]'
  prefs: []
  type: TYPE_PRE
- en: 5\. The main process[](#the-main-process "Permalink to this heading")
  prefs:
  - PREF_H2
  type: TYPE_NORMAL
- en: 'The execution flow of the main process is as follows:'
  prefs: []
  type: TYPE_NORMAL
- en: Initialize the inference pipeline.
  prefs:
  - PREF_OL
  type: TYPE_NORMAL
- en: Launch data acquisition subprocess.
  prefs:
  - PREF_OL
  type: TYPE_NORMAL
- en: Run inference.
  prefs:
  - PREF_OL
  type: TYPE_NORMAL
- en: Clean up
  prefs:
  - PREF_OL
  type: TYPE_NORMAL
- en: Note
  prefs: []
  type: TYPE_NORMAL
- en: As the data acquisition subprocess will be launched with “spawn” method, all
    the code on global scope are executed on the subprocess as well.
  prefs: []
  type: TYPE_NORMAL
- en: We want to instantiate pipeline only in the main process, so we put them in
    a function and invoke it within __name__ == “__main__” guard.
  prefs: []
  type: TYPE_NORMAL
- en: '[PRE7]'
  prefs: []
  type: TYPE_PRE
- en: '[PRE8]'
  prefs: []
  type: TYPE_PRE
- en: 'Tag: [`torchaudio.io`](../io.html#module-torchaudio.io "torchaudio.io")'
  prefs: []
  type: TYPE_NORMAL
- en: '**Total running time of the script:** ( 0 minutes 0.000 seconds)'
  prefs: []
  type: TYPE_NORMAL
- en: '[`Download Python source code: device_asr.py`](../_downloads/8009eae2a3a1a322f175ecc138597775/device_asr.py)'
  prefs: []
  type: TYPE_NORMAL
- en: '[`Download Jupyter notebook: device_asr.ipynb`](../_downloads/c8265c298ed19ff44b504d5c3aa72563/device_asr.ipynb)'
  prefs: []
  type: TYPE_NORMAL
- en: '[Gallery generated by Sphinx-Gallery](https://sphinx-gallery.github.io)'
  prefs: []
  type: TYPE_NORMAL