Buckle up readers, because this will be a long one.
Week 5 was all about the finishing touches: improving the eye-tracking setup and preparing the science fair. This blogpost is all about improving the setup.
Rotating the setup
Tuesday, we started improving our eye-tracking setup. We started by rotating the setup ninety degrees. We did this, because by doing this we could use the already existing frame on the wall to place the projector right above the participant’s head. In our old setup, the projector was placed behind the eye-tracker (see image below). However, by placing the projector above the participant, the painting can be moved forward so the participant stands way closer to the painting (see image below). We expected to get more precise eye-tracking data out of this setup.
Testing precision, part 1
After changing the setup, it was time to put it to the test. We used the same sort of precision test, that we described in our blogpost about week 4. What was different though, was the precision sheet we used. Instead of using different sized blocks, we used dots that were placed in a grid. The test subject had to look at the dots one by one in a fixed order. Afterwards, we could analyze the deviation between what the test subject looked at and what the Tobii eye-tracker thought they looked at. By changing the setup slightly between every test, we tried to get an optimal test setup. In the tables below, you can see what we did, with the corresponding results and conclusions.
We started with the setup exactly the way we rotated it. This resulted in the following:
Then we realized we forgot to take the maximum gaze angles into account (see image below). We changed the setup in such a way that the projection would fit within the maximum gaze angles. This meant that we could not use all dots anymore, because our sheet was too big. As we did not necessarily need all the dots for it to work, we decided to zoom in and work with less dots, instead of making a new precision sheet. The results can be seen in the table below.
There was one thing we were not sure about. Because the projector was so close to the screen, while placed fairly high, the image got relatively distorted. When putting the dimensions into Tobii, we were not sure what to fill in for the width. In the last two tables, we used the maximum width. In the table below, we use the average width to see whether this would influence the results.
We decided that the last setup in the table above was as good as we would be able to get it. Therefore, that was going to be the setup to continue our precision testing process with.
Testing precision, part 2
Thursday, we decided to put that final setup to the test, using excel. We all did the test with the same setup and we used excel to compare the results. Earlier this week, we took some time to get to know the excel files that Tobii produces. In an attempt to make everything clear, we are going to tell you about our entire process.
When looking at the slick Tobii data visualisations, created by applying inverse heat maps and gaze plots to the raw eyetrack data, the results seem rather precise and good looking. However, it is still debatable if these slick visualisations actually provide a proper view of reality.
We thought of a way to try to determine the actual precision of the Tobii x60 eyetrack software to answer this question. By using the raw data Tobii exports to a .xls file (Excel) we want to conduct an experiment to look into the precision of the resulting visualisations gained by performing eye tracking experiments using our set up.
The raw excel data file is really big and messy. The biggest problem is filtering all the necessary information out of these files without spending too much time on interesting, yet for our purpose not important, additions (thirty seconds of film provides thousands of rows of data distributed about over thirty columns with possibly useful characteristics).
Excel sheets such as these give some kind of a “quantitative” value to the data measured by the Tobii. An example in which this data is visually displayed is given below, this is no “qualitative” visualization as can be seen in for example the gaze plot option Tobii provides.
In attempt to make the raw data shown in the excel file more understandable for ourself, we tried to use a simple test in which a test subject will look at an origin (0,0) and a dot located at (15;7.5). These coordinate values are given in cm, from now on all the values given in the context of this precision experiment will be given in cm.
The first fixation point (at a properly measured and matched time) Tobii indicated, will be set as the origin. The second fixation point will be the second point.
When looking at how much the location of this second point differs from the manually chosen origin we can try to determine the precision of the raw eyetrack data. On average Tobii locates the second point at (15.2;8.1) which is actually not too bad for a first try.
Tobii distincts saccade and fixations points and notes, among others, indices and amplitudes to both types of gaze data. Most of the data Tobii provides contains measurements in both pixels and mm, for simplicities sake and mostly to save time we neglected the pixel data and only looked at the data in mm.
With “tidying-up” of the elaborate excel file we gained much understanding of the data. An example of this process has been displayed in the image below.
The raw data also includes a lot of information about the eyes of the participant. This kind of data remains almost constant over the full test, for now it would cost way too much time to have a closer look at these results but the data is too interesting to leave unmentioned.
Because we now have a greater understanding of the excel data visualization, we can conduct a more extensive second experiment.
For the fixation points, we used the same dots setup we used in the tests:
From these 5 dots, the bottom left point, point 0, will be the origin (0,0), the top left point will be located at (2.5;22.5), the top right point at (17.5;22.5), the bottom right point at (22.5; 0) and the fourth point, will be located in the middle at (10,10).
Conducting this five point test went well. When looking at the raw data this specific experiment provided, it is important to look at the useful fixation points of each test (the most accurate point located at the coordinates listed above).
Theoretically it should be possible to create results with only as many fixation point as there are points to look at, however, since our own eyes turn out to have limits on this aspect we selected each desirable fixation point manually in the excel file.
In this manual selection of fixation points, fixation point 2 at the picture above, for example, is not to be included.
Using the proper tests with proper calibrations for Ronja, Jerome and Celine, a raw data excel sheet was imported based on this five point test.
Setting point 0 to be the origin and by calculating the average position for each meaningful fixation point for both the left and the right eye of each test participant, an impression of the precision of the Tobii x60 eyetracker while using our test setup can be made. We visualized this impression for every point in the graphs below.
When analysing the results of this Tobii precision experiment, we think it that the overall precision is pretty accurate. Sadly, some big deviations (between 0.5 and 1 cm) are certainly undesirable when looking at a detailed and precise object as a painting.
Several factor are playing a role when explaining the accuracy in using the Tobii x60 device in combination with our test setup.
- The precision of the calibration parameters. These parameters (in terms of measurements) need to be set manually and because of this a mistake (measuring error) is easily made. We suspect we made some small mistakes when setting the parameters of our precision experiment.
- The precision of the human eye. Usually Jerome wears glasses but he did not wear them during the test. His right eye (-0.5) turned out slightly less accurate compared to his left eye (-0.25). An interesting, yet logical result.
- Physical factors, bigger and darker eyes tend to perform better.
- Psychological factors, distracted test subjects perform less accurate in our specific test.
- The method of this specific experiment. Strictly taken, it is not correct to set the first point in the point test as the origin of an axes system. If this first point is, by accident, flawed all the other points can falsely be perceived as flawed as well. Despite this possibly incorrect origin, we still chose to conduct this specific experiment because it is an efficient and fast way to receive fairly trustworthy information about the precision of the Tobii x60 device combined with our test setup. Using 3 test subjects instead of one increases this trustworthiness even more.
- The actual precision restrictions of the Tobii x60 eye tracking device.
In the case that test subjects are supposed to look at a painting, and the precision of the calibration parameters is set as precisely as possible, the precision of the human eye and the actual precision of the eye tracker device are playing important parts. If the eye turns out to be not precise enough, this would not influence the test results.
Because we think it is safe to say the actual precision restriction of the Tobii x60 eye tracking device is limited to a few millimeters, using our setup and the x60 will give results accurate enough to perform proper art tracking experiments.
In short, we think it is safe to say our specific setup combined with the x60, is accurate enough and thus can be used in an experiment to see where experts and ordinary people look at when asked to perceive gloss. Goal achieved.