Contributions:AudioExtension

Synopsis

An environment extension which manages multichannel, low latency audio I/O.

Location

http://www.bci2000.org/svn/trunk/src/contrib/Extensions/AudioExtension

Versioning

Authors

Griffin Milsap (griffin.milsap@gmail.com)

Jordan Powell (jpow7@outlook.com)

Version History

• 2012/06/11: Initial public release;

Source Code Revisions

• Initial development: 4095
• Tested under: 8516
• Known to compile under: 8516
• Broken since: --

Todo

• Fix Known Issues
• Add per-sample resolution to envelopes

Known Issues

• Using DirectSound when suspending and resuming states can cause an issue where the file recorded drops samples, this can be fixed by suspending and resuming until the audio clears up. Luckily, AudioExtension plays back what has been recorded so its easy to detect when this issue happens, just restart the trial to fix or use ASIO where no known issues exist.

• When compiling in Debug mode the audio clips and some data may be lost, this DOES NOT occur in release mode.

Functional Description

Experiments which require audio input or real-time audio synthesis based on system state are now possible with the AudioExtension. This extension is capable of recording multiple channels of audio input, synthesizing tones or noise, and reading encoded audio files. These channels are input to a mixing matrix which mixes those inputs to multiple channels of audio output. Both input and output are run through a simple filterbank, then they have their envelope extracted and logged into states via the bcievent interface. Audio input and output channels can be recorded into audio files losslessly and can be resynchronized offline. The mixing matrix is a matrix of expressions which can be used to dynamically change audio mixing based on the system state.

Integration into BCI2000

Compile the extension into your source module by enabling contributed extensions in your CMake configuration. You can do this by going into your root build folder and deleting CMakeCache.txt and re-running the project batch file, or by running cmake -i and enabling BUILD_AUDIOEXTENSION. Once the extension is built into the source module, enable it by starting the source module with the --EnableAudioExtension=1 command line argument.

Building with ASIO support

ASIO is a driver that allows for recording from devices with up to four input channels. It also can provide lower latency than other audio drivers. To compile with ASIO support, visit https://www.steinberg.net/en/company/developers.html and download the ASIO SDK. Extract the downloaded SDK zip file to src/extlib/portaudio and rename it asio. Enable the AudioExtension in CMake and click "Configure". Make sure the "Advanced" option is checked in the CMake GUI and enable PORTAUDIO_ENABLE_ASIO. Click "Generate" and recompile BCI2000. ASIO will now appear as an option under the EnableAudioExtension parameter when BCI2000 is run with the AudioExtension enabled.

Block Diagram

Parameters

The AudioExtension is configured in the Source tab within the AudioExtension section. The configurable parameters are:

EnableAudioExtension

Enables/Disables the AudioExtension.

AudioExtensionHostAPI

This parameter is an audio host API selector. The following values of this parameter are valid. NOTE: Not all audio APIs are available on all platforms.

• • [0] - auto
  • [1] - DirectSound
  • [2] - MME
  • [3] - ASIO
  • [4] - SoundManager
  • [5] - CoreAudio
  • [6] - Disabled
  • [7] - OSS
  • [8] - ALSA
  • [9] - AL
  • [10] - BeOs
  • [11] - WDMKS
  • [12] - JACK
  • [13] - WASAPI
  • [14] - AudioScienceHPI
  • [15] - AudioIO
  • [16] - PulseAudio
  • [17] - Sndio

When set to 0 (auto), the audio extension will choose a preferred host API. On Windows, DirectSound is chosen over WASAPI although WASAPI allows for lower latency because DirectSound is more compatible across multiple devices for input and output.

AudioBufferSize

The size of the audio buffer, in audio frames. When set to "auto", defaults to 2048 (50ms). Enlarge in steps of 1024 if you experience audio frame loss.

AudioMixer

The Audio Mixer is represented as an N x M Matrix, where N is the number of input channels on the selected input device, and M is the number of output channels on the selected output device.

If the input device has 2 inputs and the output device has 2 outputs, the user must open the AudioMixer and set the matrix size to 2 x 2. To specify which input will be mapped to a specific output you place a 1 at the intersection of the row (input) and column (output).

For the simplest configuration set the number of inputs and outputs and place a 1 in a diagonal line from the top left hand corner to the bottom right hand corner.

   row:1, column:1; row:2, column:2; row:3, column:3; ... , row:(N-1), column:(N-1); row:N, column:N;

By Default the Matrix will have numeric values for all the labels. To specify a different label, double click on the label and type the specified input type.

Below are a list of valid input labels:

• X - This is automatically interpreted as INPUT[X], where x is the input channel on the device.
• INPUT[X] - This input will come from channel X on the sound capturing device.
• FILE[X] - This input will come from channel X in the specified AudioInputFile listed in the Source Tab of BCI2000 Config.
• TONE[X] - This input will be a synthesized sine wave with the frequency of X Hz.
• NOISE[X] - This input will be generated white noise at X Hz. NOTE: NOISE[] is white noise at the audio sampling rate (which defaults to 44100)

AudioInputDevice

Requires a number, which corresponds to an input device ID. Each Audio Recording Device connected to the computer has an associated number. To select a specific device, enter the number into the corresponding box. To view a list of detected Audio Input Devices in BCI2000 click on Set Config and the devices will be listed below 'Audio Extension Enabled' in the operator log.

Format:

             Audio Input Device ID [ i ] : [Name of Audio Device] supports  N Input Channels

Where i is a number that corresponds to the Name of the Audio Device. A value of -1 selects the default input device on this host API.

Where N is the number of input channels that can capture audio. This is also used as the number to set up the AudioMixer during configuration.

AudioOutputDevice

Requires a number, which corresponds to an output device ID. Each Audio Playback Device connected to the computer has an associated number. To select a specific device, enter the number into the corresponding box. To view a list of detected Audio Input Devices in BCI2000 click on Set Config and the devices will be listed below 'Audio Extension Enabled' in the operator log.

Format:

             Audio Output Device ID [ i ] : [Name of Audio Device] Supports N Output Channels

Where i is a number that corresponds to the Name of the Audio Device. A value of -1 selects the default output device on this host API.

Where N is the number of output channels that where audio can be stored. This is also used as the number to set up the AudioMixer during configuration.

AudioInputFile

Audio file to use as audio input to AudioMixer. The selected file can have any non-zero number of channels and be encoded in almost any format (except MP3), but MUST be encoded at 44100 Hz.

AudioRecordInput

Enables/Disables recording of audio data to a file in the DataDirectory.

AudioRecordOutput

Enables/Disables recording of audio data to a file in the DataDirectory.

AudioRecordingFormat

Changes the file format and encoding options of the recorded output files. This parameter has the following three options:

• Raw - Records to 16 bit Microsoft formatted WAV files with no compression. These files open directly in MATLAB if that's interesting to you.
• Lossless - Records to FLAC formatted files. These files are slightly smaller than RAW files, but have no quality loss.
• Lossy - Records to Ogg Vorbis files. These files are similar to MP3 but do not have the associated licensing issues. They are compressed using a lossy algorithm, so the resulting files are very small but sound slightly worse than lossless encoding. This format is good for long recordings where perfect quality is not necessary.

Output files are located in the current BCI2000 output directory, and bear the .dat file's name, with its extension replaced with _in.wav or _out.wav, respectively (.flac and .ogg for the remaining two file format options).

AudioInputFilterbank, AudioOutputFilterbank

A filterbank which filters audio input and output before rectification/smoothing for envelope extraction. These butterworth filters will not be applied to the audible signal. The format of the filter bank is as follows:

• Type - The characteristic of the filter. The following values are valid.
  • Lowpass - Creates a low pass filter
  • Highpass - Creates a high pass filter
  • Bandpass - Creates a band pass filter *See Known Issues*
  • Bandstop - Creates a band stop, or notch filter
• Order - The order of the filter model. Higher order filters are more accurate but more expensive computationally.
• Cutoff1 - The cutoff frequency for Lowpass and Highpass filters, and the cut-on frequency for Bandpass and Bandstop filters.
• Cutoff2 - The cut-off frequency for Bandpass and Bandstop filters.

The matrix can have as many rows as necessary to filter the signal. Filters can be applied in any order and their transfer functions are multiplied before filtering occurs.

AudioEnvelopeSmoothing

The cutoff frequency for the low pass filter which is applied to the filtered and full-wave rectified audio data. This should be set to the highest frequency you want to see in the resulting audio envelope.

ComputeEnvelopes

When set to 0 (false), the relatively CPU intensive envelope computation for input and output signal is omitted. This may be useful if you experience audio sample loss.

State Variables

The AudioExtension outputs the following state variables:

Audio[In/Out]Envelope[0-3]

These are the envelope values of each channel (up to channel 4) of the audio inputs and outputs (in the AudioMixer matrix). These 16 bit unsigned values correspond to the resulting envelope after the audio envelope extraction. For architectural reasons, it is not possible to publish states after system startup, so you are limited to four channels of input and output. The AudioExtension can be easily modified to change the number of channels by editing the #define NUM_INPUT_ENVELOPES 4 and #define NUM_OUTPUT_ENVELOPES lines in AudioExtension.cpp, and recompiling your source module.

AudioFrame

This 32 bit unsigned number corresponds to the current frame of audio data in the recorded output files. This can be used to resynchronize the lossless audio to the resulting .dat file offline. Audio is sampled internally at 44100 Hz, so this number will roll over once every 27 hours or so.

Configuring AudioExtension for 4-Channel Recording

Overview

The following instructions describe how to configure the AudioExtension in BCI2000 to record from four separate audio input channels. This configuration requires building the extension with ASIO support for multi-channel low-latency audio.

Building with ASIO Support

Download the ASIO SDK from Steinberg’s developer site. Extract the downloaded archive into:

src/extlib/portaudio/asio

Run CMake and ensure the following options are set:

BUILD_AUDIOEXTENSION = ON

PORTAUDIO_ENABLE_ASIO = ON (requires "Advanced" view)

Click Generate and recompile BCI2000. After compilation, ASIO will appear as an available audio host API when running the source module with the AudioExtension enabled.

Enabling the Extension

Start your source module with the following command-line argument:

 --EnableAudioExtension=1 

Parameter Configuration

In the Source → AudioExtension section of the BCI2000 Configuration dialog, set the following parameters:

EnableAudioExtension

EnableAudioExtension = 1

AudioExtensionHostAPI

Select the audio host API. For 4-channel recording, choose ASIO:

 AudioExtensionHostAPI = 3 

AudioInputDevice

Each connected audio device is listed in the Operator Log after clicking Set Config. Locate the entry similar to:

 Audio Input Device ID [ i ] : [Device Name] supports N Input Channels 

Choose a device with N ≥ 4 and set:

 AudioInputDevice = i 

Use -1 to select the default device.

AudioOutputDevice

Select an output device (optional, for monitoring). If unused, you may set:

 AudioOutputDevice = -1 

AudioBufferSize

Defines the audio buffer length in frames:

 AudioBufferSize = auto 

If you experience frame loss, increase this in steps of 1024 (e.g., 3072, 4096).

AudioRecordInput

Enable input recording:

 AudioRecordInput = 1 

AudioRecordingFormat

Select the desired file format:

Raw — 16-bit WAV, uncompressed.

Lossless — FLAC, smaller size, no quality loss.

Lossy — compressed.

Example:

 AudioRecordingFormat = Lossless 

Recorded files are saved to the data directory as:

_in.wav / .flac / .ogg — input recording

_out.wav / .flac / .ogg — output recording

ComputeEnvelopes

If envelope states are not required or CPU load is high:

 ComputeEnvelopes = 0 

Otherwise, leave set to 1 (default) to compute AudioInEnvelope[0–3].

AudioMixer

The AudioMixer defines input–output routing as an N × M matrix, where:

N = number of input channels (4)

M = number of output channels

For a 4-channel input and stereo output:

 Matrix size: 4 × 2 

For simple monitoring, set the diagonal entries to 1 and label each input:

 INPUT[1] → OUTPUT[1] INPUT[2] → OUTPUT[2] INPUT[3] → OUTPUT[1] INPUT[4] → OUTPUT[2] 

If you do not need to hear playback, set all matrix values to 0.

State Variables

AudioInEnvelope0..3 — Input channel envelopes (0–3)

AudioOutEnvelope0..3 — Output channel envelopes (0–3)

AudioFrame — Current frame index (44.1 kHz)

Notes

Use WASAPI for devices supporting ≥4 inputs.

Debug builds may drop audio—use, use Release for deployment.

Increasing AudioBufferSize can reduce dropouts.

Troubleshooting

If only two input channels are available, ensure the device driver and host API are WASAPI.

For stability issues, increase buffer size or disable envelope computation.

See also

User Reference:Logging Input, Contributions:Extensions