Jump to content

User Tutorial:UnityBCI2000

From BCI2000 Wiki
Red text blocks contain detailed instructions for the higher-level instructions preceding them.

This example comprehensively demonstrates many of the capabilities of UnityBCI2000, and as such, is more than what is necessary for many usecases. If you just want to use BCI2000 as a logging backend for an existing Unity application, see User Tutorial:UnityBCI2000 Barebones.

Download Unity

Download Unity Hub from Unity.

Setting Up BCI2000

Follow the instructions starting with [1] to download BCI2000.

Setting Up UnityBCI2000

Download UnityBCI2000 from its GitHub page, [2]. 

Click on the "Releases" section on the right side of the page. This tutorial is intended to be used with UnityBCI2000 version 2.0.0, so scroll to that release and click on the files BCI2000RemoteNETStandard.dll and UnityBCI2000.cs to download them.

Download Tutorial Project

Download the CursorTask3 repository from GitHub. 

Either:
	-Clone the git repository
	-Download the respository directly from GitHub
		-> Click the green "Code" button 
		-> Select "Download ZIP"
		-> Extract the zip file

The download directory contains two versions of the CursorTask3 project. One, CursorTask3-BCI2K, contains an example implementation of the task with BCI2000 integration. The other, CursorTask3-NoBCI2K, contains the same task without BCI2000 integration. This tutorial is a step by step guide on turning that second project into the first.

Opening the project

Open the project in the Unity Editor, and open the CursorTask scene. 

Open Unity Hub.
Click the "Add" button in the top right corner.
Navigate to the CursorTask3 directory downloaded previously.
Select the CursorTask3-NoBCI2K directory to add it to Unity Hub.
Click the CursorTask3-NoBCI2K project in the Unity Hub.
If prompted to select a version of Unity to use or install, select Unity 2022.3. The specific release number of Unity 2022.3 should not matter, so select the one labeled "LTS" for consistency.
The lower panel (the asset browser) should already show the Assets/Scenes directory, which contains a single scene called SampleScene.
Double-click SampleScene. This should open a scene with a box containing various lights and objects.

If you wish to view the completed implementation, repeat these steps, but instead open the CursorTask3-BCI2K directory. Note that in order to use the completed implementation, you will still need to set the Operator Path value of the UnityBCI2000 component of the BCI2000 game object.

Adding UnityBCI2000 to the project

Add BCI2000RemoteNETStandard.dll and UnityBCI2000.cs to your project's assets. 

In the Unity Editor's menu bar, click Assets > Open Containing Folder to open your project directory.
Place the BCI2000RemoteNETStandard.dll and UnityBCI2000.cs files in the Assets folder.

Add an empty GameObject called 'BCI2000'. This will hold the scripts for controlling BCI2000. 

In the Unity Editor's menu bar, click GameObject > Create Empty to create an empty GameObject, and name it 'BCI2000'. (The object's specific name does not actually matter, it is only needed in order to reference it within other scripts)
You will notice that it now appears in the Scene Hierarchy panel, on the left side of the screen.

Add the UnityBCI2000 component to the object. 

Click on the object in the Hierarchy panel. This will open it in the Inspector panel (the panel on the right side of the screen)
In the inspector, below the new object's Transform component, click the "Add Component" button, which will open up a small window for adding components.
Within this window, select Scripts > Unity BCI2000 to add the UnityBCI2000 component.

Configuring UnityBCI2000

The inspector panel now contains the configuration options for UnityBCI2000. Set the options as follows:

Start Local Operator: Checked

This will start the operator on your computer when the Unity scene initializes. If we were instead connecting to an already-running instance of BCI2000, or an instance on another computer, this box would be deselected.

Operator Path: The path to the Operator executable. This will look something like this on Windows (C://path/to/bci2000/prog/Operator.exe)

Operator Address: 127.0.0.1

This is the address of the machine on which BCI2000 is running. Since we are running BCI2000 on the same computer as Unity, we leave it as 127.0.0.1, the loopback address.

Operator Port: 3999

This is the port on which BCI2000 is listening for commands. By default, it is 3999.

Start Modules: Checked

This tells BCI2000 to start the requested Signal Source, Signal Processing, and Application modules when the Unity scene initializes. Similarly to Start Local Operator, we would deselect this box if connecting to an already-running instance.

Start With Scene: Unchecked

This tells BCI2000 to start a data collection run when the scene starts. Since we will be using BCI2000 itself to set experiment parameters, we will instead wait for BCI2000 to start from Unity, and thus will leave the box unchecked.

Stop With Scene: Unchecked

This tells BCI2000 to stop collecting data when the scene stops. Since we will be controlling BCI2000 directly, rather than entirely through Unity, we will leave this unchecked.

Shutdown With Scene: Unchecked

This tells BCI2000 to shut down alongside the Unity scene. Whether or not this value is set ultimately doesn't matter much, especially if Start Local Operator is checked. The data will be saved whether or not BCI2000 shuts down.

Module 1: SignalGenerator

The signal source module to start. We will use SignalGenerator, which generates a signal without any connected hardware.

Module 2: DummySignalProcessing

The signal processing module to start. We will use DummySignalProcessing, as there is no processing to do.

Module 3: DummyApplication

The Application module to start. Since we are using Unity, we will use DummyApplication.

Setting References

In order for the game's scripts to communicate with BCI2000, they need to hold a reference to the UnityBCI2000 component. There are three scripts which will need to communicate with BCI2000. They are GameControl.cs, BallControl.cs, and MCursorControl.cs. These scripts are each located within the Assets directory of the Unity project. 

As before, select Assets > Open Containing Folder to open the project directory, then open the Assets folder. 
For each script, open it in a text editor.
Add a data member of type UnityBCI2000 to the class, like so:
	UnityBCI2000 bci;
Place the member definition above the Awake() method, for readability.
Within the Awake() method, set this reference to the UnityBCI2000 component.
	bci = GameObject.Find("BCI2000").GetComponent<UnityBCI2000>();

Each of the three scripts should contain a section like this: 

	...
	UnityBCI2000 bci;
	void Awake() {
		bci = GameObject.Find("BCI2000").GetComponent<UnityBCI2000>();
		...

Adding Events

Events are the primary way that non-signal experiment data is recorded in BCI2000. They are timestamped integer values which are encoded alongside the signal data in BCI2000 output files. Due to BCI2000's design, events must be added during a very specific part of its startup sequence, which is immediately after the BCI2000 operator starts, and before any of the modules start. As such, we cannot simply call AddEvent() whenever we want. Furthermore, the order in which Unity objects initialize is undefined, so it cannot even be guaranteed that calling AddEvent at a specific time will be consistent across multiple projects. As such, UnityBCI2000 provides a couple of methods for sending commands at well-defined points within the startup sequence. These two methods, OnIdle and OnIdle and OnConnected allow BCI2000 commands to be sent while the operator is in the state Idle (immediately before starting its modules) and when the operator is in the state Connected (after starting and connecting to the modules. Below is an example of using those methods to add and show an event in BCI2000. 

class Script : MonoBehaviour {
    UnityBCI2000 bci;
    void Awake() {
	bci = GameObject.Find("BCI2000Object").GetComponent<UnityBCI2000>();
	bci.OnIdle(remote => {
	    remote.AddEvent("AnEvent", 32);
	});
    }
    void Start() {
	bci.OnConnected(remote => {
	    remote.Visualize("AnEvent");
	}
    }
}

As seen above, OnIdle and OnConnected take a delegate (C#'s term for a callback/closure/functor/etc.) with a single parameter of type BCI2000Remote (in this case called "remote"). The lambda expression given to the call to OnIdle uses the method BCI2000Remote.AddEvent() to add an event called "AnEvent" with a bit width of 32 bits. The call to OnConnected tells BCI2000 to show the event's value in a graphical window. A description of the BCI2000Remote class can be found here, and API documentation can be found here.

We will now add the events relevant to the Cursor Task. Open the GameControl.cs script, and modify its Awake() to add the events PreFeedback, Feedback, PostFeedback, TargetHit, and Timeout with bit width 1. Additionally, add the events TrialNumber, TargetPositionX, and TargetPositionY with width 16. These will encode the task state and number of trials. We also need to set a parameter value so that the signal will be set to the mouse position. Using BCI2000Remote.SetParameter(), we will set the ModulateAmplitude parameter to 1.

Open the BallControl.cs script, and within the Awake() function, add events CursorPositionX and CursorPositionY, with width 16. Show these events in a visualization window with Visualize

Your two scripts should now look like this: 

>>> GameControl.cs 
    ...
    void Awake() {
	...
	bci.OnIdle(remote => {
	    remote.AddEvent("PreFeedback", 1);
	    remote.AddEvent("Feedback", 1);
	    remote.AddEvent("PostFeedback", 1);
	    remote.AddEvent("TargetHit", 1);
	    remote.AddEvent("Timeout", 1);
	    remote.AddEvent("TrialNumber", 16);
	    remote.AddEvent("TargetPositionX", 16);
	    remote.AddEvent("TargetPositionY", 16);
	    remote.SetParameter("ModulateAmplitude", "1");
	    });
    }

>>> BallControl.cs
    ...
    void Awake() {
	bci.OnIdle(remote => {
	    remote.AddEvent("CursorPositionX", 16);
	    remote.AddEvent("CursorPositionY", 16);
	    });
	bci.OnConnected(remote => {
	    remote.Visualize("CursorPositionX");
	    remote.Visualize("CursorPositionY");
    }


Using BCI2000 Parameters

Adding Parameters

BCI2000's operator contains an interface for changing experiment parameters before running a task. By making use of this interface we can change the behavior of the task dynamically, without needing to rebuild the Unity project each time. In order to do this we need to define the parameters and add them to BCI2000, and later read their values after the operator has a chance to change them. This section is not strictly necessary. The project will function the same whether this is done or not, just without the option to change parameters at runtime.

First, we define add parameters to BCI2000. Similarly to events, parameters must be added while the Operator is in the Idle state, so we add them inside of OnIdle, like below: 

public int someParameter = 5;
void Awake() {
    bci.OnIdle(remote => {
	remote.AddParameter("Application:ParameterPlace", "SomeParameter", someParameter.ToString()); 
    });
}

The first argument to the method is the section into which the parameter is placed, that is, on which tab and under which heading the parameter will appear within the Operator's parameter editor window (note that the section is not a scope, all parameter names must be unique). The second argument is the parameter name itself. The third argument is the default value of the parameter. We pass in the value of the parameter, so when it appears in the Operator, its default value is 5 (or another value if it has been changed in the Unity Editor). Note that all parameters are handled as strings, so we need to convert its value to a string when sending it.

For <code>GameControl.cs</code>, we will add parameters corresponding to the Pre-Feedback, Feedback, and Post-Feedback durations, Target Radius, and Number of Trials.
For <code>BallControl.cs</code>, we will add parameters corresponding to the Acceleration Scale, Coefficient of Restitution, Coefficient of Drag, and Attraction Curve Coefficient (how quickly the Ball moves to follow the Mouse Cursor).
For <code>MCursorControl.cs</code>, we will add a parameter corresponding to the Rolling Maximum Amount.

Your scripts should look like this:
<pre>
>>> GameControl.cs
void Awake() {
	bci = GameObject.Find("BCI2000").GetComponent<UnityBCI2000>();
	...

	bci.OnIdle(remote => {
		remote.AddEvent("PreFeedback", 1);
		...

		remote.AddParameter("Application:Task", "PreFeedbackDuration", preFeedbackDuration.ToString());
		remote.AddParameter("Application:Task", "FeedbackDuration", feedbackDuration.ToString());
		remote.AddParameter("Application:Task", "PostFeedbackDuration", postFeedbackDuration.ToString());
		remote.AddParameter("Application:Task", "TargetRadius", targetRadius.ToString());
		remote.AddParameter("Application:Task", "Trials", n_trials.ToString());
		});
}

>>> BallControl.cs
void Awake() {
	...
	bci.OnIdle(remote => {
		remote.AddParameter("Application:Physics", "AccelerationScale", accelerationScale.ToString());
		remote.AddParameter("Application:Physics", "CoeffOfRestitution", coefficientOfRestitution.ToString());
		remote.AddParameter("Application:Physics", "CoeffOfDrag", coefficientOfDrag.ToString());
		remote.AddParameter("Application:Physics", "AccelerationCurve", attractionCurveCoefficient.ToString());
		})}
}

>>> MCursorControl.cs
void Awake() {
	...

	bci.OnIdle(remote => {
		AddParameter("Application:Physics", "RollingMaximumAmount", rollingMaximumAmount.ToString());
		});
}
		

====Reading and Validating Parameters====
Since we set the parameters within the BCI2000 operator, we need some way to read them back into Unity after they have been set. Specifically, we need to read them after the operator clicks the "Set Config" button within the BCI2000 Operator. How this timing synchronization is covered in the section [[Reacting to BCI2000 state changes]]. In this section we will cover reading and validating parameters.

A method called <code>SetConfig</code> is provided in each of the scripts in which we added parameters previously. This method will be called after the operator changes parameters. In this method we will read in parameters and parse them into valid values. Using the same example parameter as before:
<pre>
void SetConfig() {
    try {
	someParameter = int.Parse(bci.Control.GetParameter("SomeParameter"));
    } catch (FormatException e) {
	bci.Error("Could not parse parameter SomeParameter as int");
	throw e;
    }
}

Note that since we are now working outside the BCI2000 startup sequence, we use UnityBCI2000.Control to call BCI2000Remote methods directly, rather than using OnIdle or OnConnected. We also wrap the call within a try/catch block so that we can report erroneous parameters back to the operator. This is optional, but it keeps your project from failing silently and is otherwise useful for doing things like distributing your Unity project as a standalone executable, where logging errors can only be done through the BCI2000 operator. You can also write a helper function which handles parsing and error reporting for you: 

int GetParseParameterInt(string paramName) {
    int p_val = bci.Control.GetParameter("SomeParameter");
    try {
	someParameter = int.Parse(p_val);
    } catch (FormatException e) {
	bci.Error($"Could not parse parameter {paramName} (value {p_val}) as int");
	throw e;
    }
}

Unfortunately, since the version of C# used by Unity does not support type-parameterized parsing, you must write one of these helper functions for each type of data you want to store in a parameter (int, float, double, etc.)


Read back the parameters added in the previous step. Your scripts should look like this: 

>>> GameControl.cs 
void SetConfig() {
	try {
	   preFeedbackDuration = float.Parse(bci.Control.GetParameter("PreFeedbackDuration"));
	   feedbackDuration = float.Parse(bci.Control.GetParameter("FeedbackDuration"));
	   postFeedbackDuration = float.Parse(bci.Control.GetParameter("PostFeedbackDuration"));
	   targetRadius = float.Parse(bci.Control.GetParameter("TargetRadius"));
	} catch (FormatException e) {
	   bci.Error("Could not parse one of PreFeedbackDuration, FeedbackDuration, PostFeedbackDuration, or TargetRadius as a float");
	   throw e;
	}
	try {
	   n_trials = int.Parse(bci.Control.GetParameter("Trials"));
	} catch (FormatException e) {
	   bci.Error("Could not parse Trials as an int");
	   throw e;
	}

	if (preFeedbackDuration < COUNTDOWN_DURATION) {
	   bci.Error("preFeedbackDuration must be greater than or equal to 2.25");
	   throw new Exception("preFeedbackDuration must be greater than or equal to 2.25");
	}
    }

>>> BallControl.cs
    public void SetConfig() {
	try {
	   accelerationScale = float.Parse(bci.Control.GetParameter("AccelerationScale"));
	   coefficientOfRestitution = float.Parse(bci.Control.GetParameter("CoeffOfRestitution"));
	   coefficientOfDrag = float.Parse(bci.Control.GetParameter("CoeffOfDrag"));
	   attractionCurveCoefficient = float.Parse(bci.Control.GetParameter("AccelerationCurve"));
	} catch (FormatException e) {
	   bci.Error("Could not parse one of AccelerationScale, CoeffOfRestitution, CoeffOfDrag, AccelerationCurve as float");
	   throw e;
	}
    }

>>> MCursorControl.cs
public void SetConfig() {
    try {
       rollingMaximumAmount = int.Parse(bci.Control.GetParameter("RollingMaximumAmount"));
    } catch (FormatException e) {
       bci.Error("Could not parse parameter RollingMaximumAmount as int");
       throw e;
    }
    signalsX = new int[rollingMaximumAmount];
    signalsY = new int[rollingMaximumAmount];
}

Note that in GameControl.cs, we also verify that the Pre-Feedback Duration is longer than the duration of the countdown that happens before each trial, and report errors to BCI2000 accordingly.


Reacting to BCI2000 state changes

Since we will be controlling the Unity task from BCI2000, we need some way to wait for BCI2000 to be in a specific state, for example, waiting for the operator to click Start and put the Operator Module into the Running state. UnityBCI2000 provides a method for this, the UnityBCI2000.PollSystemState method. In order to wait for the Operator Module to be in the Running state: 

bci.PollSystemState(BCI2000Remote.SystemState.Running);

This will check once per frame whether or not the Operator is in the Running state, and only continue once BCI2000 is in the Running state. Note that since SystemState is a member of BCI2000Remote, we also need to add a using BCI2000RemoteNET statement to our script.

In our case, we need to react to the operator setting the parameters, starting the data collection, and stopping the data collection. These correspond to the Operator Module entering the Resting state, entering the Running state, and exiting the Running state.

In the GameControl.cs script, we will wait for the system to be in the Resting state at the start of the ControLoop method, and wait for the system to be in the Running state immediately before the entering the trial loop. Note that this code is within a coroutine and we do not want it to block, so we use a yield return statement. 

>>> GameControl.cs

    IEnumerator ControlLoop() {
	while (True) {

	   yield return bci.PollSystemState(BCI2000Remote.SystemState.Resting);

	   try {
		...
	   } catch (Exception e) {
		continue;
	   }
	
	   int trials = 0;

	   yield return bci.PollSystemState(BCI2000Remote.SystemState.Running);


	   yield return new WaitForSeconds(preRunDuration);
	   while (IsContinue() && trials < n_trials) {
		...
	   }
	}
    }

We also need to check for the operator stopping data collection, so we modify the IsContinue() method. 

bool IsContinue () {
	return bci.Control.GetSystemState() == BCI2000Remote.SystemState.Running;
}

Remember to add using BCI2000RemoteNET; to the top of the script, so you can access the SystemState enum.

Reading the control signal

We will be using the control signal to control the cursor. Open the MCursorControl.cs script.

The commented out section of the GetPos() method contains the code required to turn the control signal waveform coming from the Signal Source and Signal Processing modules into screen coordinates. Uncomment the commented part and delete the line Ray r = camera.ScreenPointToRay(Input.MousePosition);. Change the double signalX = 0; and double signalY = 0; to.

Your GetPos() method should look like this: 

    Vector3 GetPos() {
	double signalX = bci.Control.GetSignal(1, bci.CurrentSampleOffset());
	double signalY = bci.Control.GetSignal(2, bci.CurrentSampleOffset());
	signalsX[signalIndex] = signalX;
	signalsY[signalIndex] = signalY;
	
	signalIndex = signalIndex + 1 >= rollingMaximumAmount ? 0 : signalIndex + 1;
	double max_x = signalsX.Max();
	double max_y = signalsY.Max();
	Ray r = camera.ScreenPointToRay(new Vector3((float) max_x * Screen.width, (float) max_y * Screen.height, 0));
	float p;
	if (!plane.Raycast(r, out p)) {
	   throw new Exception("error casting ray to plane, invalid mouse position?");
	}
	return r.GetPoint(p);
    }

Note the use of the UnityBCI2000.CurrentSampleOffset() method. This is a special method which gets the offset into the current block such that the sample is exactly one block length later than when it was collected by the hardware. This is to normalize the latency between when the hardware collects the signal and the software receives the signal, due to how BCI2000 processes data in blocks.

Sending events back to BCI2000

The primary way to communicate game state back to BCI2000 is via the use of Events, which are integer values encoded alongside the signal data.

We will send the events that we added in a previous section.

First, we will send back the current position of the cursor. Within the BallControl.cs script, within the Update() method, immediately after Move() is called, set the CursorPositionX and CursorPositionY events. 

>>> BallControl.cs
void Update() {
    ...
    if (isTrialRunning) {
	...
	Move();
	bci.Control.SetEvent("CursorPositionX", (uint) ((transform.position.x + 7) * 1000));
	bci.Control.SetEvent("CursorPositionY", (uint) ((transform.position.y + 4.5) * 1000));
    }
    ...

Notice that we transform the value of the cursor's position. This is because events in BCI2000 are represented as unsigned integers, so, for example, the cursor's range of movement in the x axis, -7 to 7, would not be directly representable within a BCI2000 event, so we add 7 so it is positive, and multiply by 1000 so that we have a more precise measure of the cursor's position. The range of [-7,7] becomes [0,14000].


We will also set the events corresponding to the game state within GameControl.cs. First we will set PreFeedback, Feedback, and PostFeedback to represent when the game is in these states. To do this, at the beginning and end of PreTrial(), Trial(), and PostTrial(), we will set the corresponding event values to 1 and 0, respectively. 

>>> GameControl.cs
IEnumerator PreTrial() {
    bci.Control.SetEvent("PreFeedback", 1);
    ...
    bci.Control.SetEvent("PreFeedback", 0);
}
IEnumerator Trial() {
    bci.Control.SetEvent("Feedback", 1);
    ...
    bci.Control.SetEvent("Feedback", 0);
}
IEnumerator PostTrial() {
    bci.Control.SetEvent("PostFeedback", 1);
    ...
    bci.Control.SetEvent("PostFeedback", 0);
}

We will also set the events which happen at the end of each trial. The TargetHit event is activated when the subject hits the target, and the Timeout event is activated when the subject runs out of time. 

>>> GameControl.cs
IEnumerator Trial() {
    ...
    bci.Control.SetEvent("Feedback", 0);
    if (lastTrialSucceeded) {
	bci.Control.PulseEvent("TargetHit", 1);
    } else {
	bci.Control.PulseEvent("Timeout", 1);
    }
}

Notice that we use <code>BCI2000Remote.PulseEvent()</code> rather than <code>BCI2000Remote.SetEvent()</code>. This results in the event being set to the value <code>1</code> for exactly one sample duration, then returning to zero.

We will also record the position of the target. Similarly to the cursor, we will transform the target's coordinates to be a positive integer.

<pre>
>>> GameControl.cs
IEnumerator PreTrial() {
    ...
    target.SetActive(true);
    bci.Control.SetEvent("TargetPositionX", (uint) ((target.transform.position.x + 7) * 1000));
    bci.Control.SetEvent("TargetPositionY", (uint) ((target.transform.position.y + 4.5) * 1000));
    ...
}

The last event we need to set is the trial number. 

IEnumerator ControlLoop() {
    ...
    while (IsContinue() && trials < n_trials) {
	bci.Control.SetEvent("TrialNumber", trials + 1);
	...
    }
}

Unity Player Settings

Additionally, due to how Unity detects changes in BCI2000 state, it must be allowed to run in the background. 

In the Unity menu bar:
    Edit > Project Settings > Player > Resolution and Presentation
Check the "Run In Background" box.


Usage

Now, when the Unity application runs, BCI2000 will open. From here, parameters can be set via the Config window, then clicking Set Config and Start will start data collection.

Retrieved from "http://www.bci2000.org/mediawiki/index.php?title=User_Tutorial:UnityBCI2000&oldid=11773"