Contributions:WebcamLogger

Synopsis

An extension that records video from one or more webcams to an mp4 file, and stores frame numbers as (event-type) state variables.

Location

http://www.bci2000.org/svn/trunk/src/contrib/Extensions/WebcamLogger

Versioning

Authors

Adam Wilson (adam.wilson@uc.edu)
Alexander Belsten (belsten@neurotechcenter.org)
Juergen Mellinger (mellinger@neurotechcenter.org)

Version History

07/06/2011 - Full commit with support for MSVC and MinGW.

07/13/2011 - Bugfixes

11/28/2018 - Updated openCV library to v2.4.3 for x86/x64 Windows and added functionality for recording from multiple cameras

3/16/2021 - Updated openCV library to v4.5.1 for x86/x64 Windows.

2/2/2022 - Rewrite WebcamLogger to use a helper app that records camera data to a file and allows previewing

2/28/2023 - Add support for Webcam URLs.

Source Code Revisions

Initial development: --
Tested under: 7227
Known to compile under: 7227
Broken since: 5828 for MinGW

Known Issues

None.

Functional Description

For many experiments, visual information about how a subject is behaving during an experiment is desired. This extension acquires video from most webcams, and writes to an MP4 video file. It also records the current frame number as a state variable, so that the video can be synchronized with brain data at sample resolution (max. 1ms). Other options include the ability to overlay the date and time directly on the video (useful for long-term monitoring), and time decimation (i.e., save every Nth frame).

The video file bears the name of the BCI2000 data file, with the extension removed and _M_video.mp4 appended where M is the index of the camera in question. For the format of the video file, see the notes below.

NOTE: This module can also be used to perform screen capture with a virtual screencast webcam driver.

Integration into BCI2000

Compile the extension into your source module by enabling contributed extensions in your CMake configuration. In your root build folder, run Configure.sh.cmd in order to bring up the CMake GUI; there, enable EXTENSIONS_WEBCAMLOGGER.

Once the extension is built into the source module, enable it by starting the source module with the --LogWebcam=1 command line argument.

Usage

Webcams should be installed and configured per the device instructions. On Windows, the Webcam logger uses the Windows Media Foundation (WMF) library for display and saving. Desired resolution may be chosen per webcam, and will be matched agains available resolutions.

Camera options are found in the Source tab of the Configuration window.

Parameters

The Webcam logger is configured in the Source tab within the WebcamLogger section. The configurable parameters are:

LogWebcam

Enables/Disables logging of Webcam states.

CameraNumber

Used to select one camera to record from on systems with multiple cameras. Use value 0 for the default camera.

RecordAll

Enables the recording of cameras from StartIndex to EndIndex. Note that, unlike other indices in BCI2000, camera indices are 0-based rather than 1-based.

StartIndex

Index to start searching for cameras at.

EndIndex

Index to stop searching for cameras at.

Decimation

The decimation factor for saving and viewing video; the logger saves/displays every Nth frame.

DateTimeLocation

The location of the date/time overlay in saved video (0: none, 1: UpperRight, 2: UpperLeft, 3: LowerRight, 4: LowerLeft). If DateTimeLocation is 5, the current frame number will be displayed in the upper right corner.

DisplayStream

Display the video stream from each connected camera. Display stream is identical to what is being recorded in the .mp4 file.

WebcamFormat

For each camera index, you may specify a video width, height, and fps. These are matched against the formats provided by the camera, and the closest one is chosen. The encoding profile column allows you to choose a H.264 encoding profile for video output, which may be a number from 1 to 3. 1 corresponds to the "baseline" profile, 2 corresponds to the "main" profile, and 3 corresponds to the "high" profile as defined in the ITU-T H.264 standard.

WebcamURL

In the first column of the WebcamURL matrix parameter, you may specify a camera index, and a corresponding URL in the second column. If a URL is given for a certain index, video data will be acquired from the URL rather than a webcam with the specified index attached to the machine.

When reading video data from an URL, only streaming formats supported by the Microsoft Media Foundation library are supported. This includes MPEG4, h.264, and h.265 codecs.

For URL-based video streams, the Microsoft Media Foundation library allows to configure both an initial buffering time (MFNETSOURCE_BUFFERINGTIME), and a maximum buffer-ahead time (MFNETSOURCE_MAXBUFFERTIMEMS). In the WebcamURL matrix parameter, these are represented by the BufferingTime and MaxBufferTime columns. For clarity, these values must be entered as time values with a trailing "s" or "ms". Note that BufferingTime will be rounded to full seconds, whereas MaxBufferTime will be rounded to full milliseconds.

For typical streaming webcams, it seems not possible to obtain a delay below 1s. Also, setting MaxBufferTime to values below 200ms will typically result in unstable connections.

Event Variables

WebcamFrame<0-n>

The frame number of the video recorded on camera with index i. i must be less than or equal to n due to the nature of event declarations. The default value of n is 3, allowing for the recording of up to 4 cameras. To record from more cameras, change the NUM_OF_WEBCAM_EVENTS macro in WebcamLogger.cpp to the number of cameras you want to record from and recompile. Event values can be used to synchronize the video with the recorded neural data. It is a 32-bit value, allowing 4294967295 frames before overflowing. If the webcam can capture at 30 frames/sec, this allows more than four years of video before it will go back to frame 0. In other words, this should not be an issue for most users.

Format of the Video File

Each camera writes its own video file. For most webcams, actual video frame rate may deviate considerably from nominal video frame rate. Also, video frame rate is often not constant but may change over the recording. What makes things even more difficult is that files in mp4 format (and related formats) do not allow to seek to locations that represent original frames when writing the file. Rather, the file's nominal frame rate and length are used in conjunction with interpolation to construct a frame at an ideal location. Thus, it is not possible to write individual frames using the camera's original timestamp, at the original frame rate.

Instead, the file is written with a simplified framerate as close as possible to the camera's nominal framerate, with differences between frame timestamps matching ideal frame duration. This way, loading the video into e.g. Matlab, you will obtain a sequence of frames where each frame index corresponds to the frame index in the WebcamFrameN state (minus 1).

Note: Do not expect nominal framerate and/or nominal duration of the video file to correspond to the camera's actual framerate or duration. Also, in case of variable framerate cameras, video display may be jerky when playing back the video in a standard video player such as VLC. This is not an error but an artifact of how video frames are stored.

For a strategy how to synchronize video with brain signal data, and possibly audio data, see this page.

Troubleshooting

You may experience an error code stating that the remote procedure call failed when trying to launch the webcam logger. This is likely a driver issue. To fix this, try rolling back the camera driver in the device manager by following this tutorial.

Webcam Driver Error

Roll Back Driver