Pilot: the results (week 5, part 1)

This week was all about the finishing touches and tying all loose ends together. We started our week by analyzing the pilot we conducted last Friday (20-10-2017). We used three main methods to analyse the acquired information:

  1. We used SPSS to find relations between different variables;
  2. We compared the individual Tobii gazeplots with the drawings the participants made of what they thought they were focussing on;
  3. We compared the Tobii gazeplots with each other.

As we had twenty participants, we expected to find at least a few correlations between different variables. However, there was little to be found. After analysing every possible variable combination, we only found two significant correlations:

  1. For the real painting, there was a positive relationship between the attention participants paid to the gloss and the perceived realism of the painting. The more people paid attention to the gloss, the more realistic the painting was perceived. This correlation was non-existent for the 3D printed painting.
  2. For the 3D printed painting, there was a negative relationship between the perceived influence of the frame and the perceived realism of the painting. The more people thought the frame influence their way of looking at the painting, the less realistic the painting was perceived.
  3. For the real painting as well as the 3D printed painting, the main thing that caught peoples attention was high contrast. Most participants said their eyes were drawn to spots on the painting with high contrast between colors or between light and dark.

We made some graphs to visualize the test results to give you and ourselves a better understanding of the test results.

Influence of the gloss

Knowledge about art

Comfort level

Influence of the frame

Does it look realistic?

What caught attention?

Gazeplot data vs. experience
To compare the gazeplot data with the drawings of what the test subjects thought they focussed on, we had to analyse every test subject individually. We came up with a way to broadly categorize the differences test results. Keep in mind that this method is based on estimates, as the drawings were estimates also. Therefore, they could not be compared to the gazeplots in an exact way. We categorized the test subjects in the following groups:

  1. People who estimated less than 25% of their focus points correctly
  2. People who estimated more than 25% of their focus points correctly
  3. People who estimated more than 50% of their focus points correctly
  4. People who estimated more than 75% of their focus points correctly

In the picture below, two examples are shown: one in the less than 25% category and one in the more than 75% category.

>75% estimated correctly

<25% estimated correctly

Amount of participants per category visualized

The interesting thing about these results, was that many participants were incorrect. Almost half of the participants (45%) estimated less than 50% correctly. This means almost half of the people were not really aware of what they are really looking at.

Gazeplots compared to each other
To compare the way people looked at the different paintings, we layered all the gazeplots of each category: the perceived focus points, the gazeplots of the real painting, and the gazeplots of the 3D printed painting. The results are shown in the image below.

Gazeplot comparison

A few things stand out in this picture:

  1. The actual focus points are way more dispersed than just the spots people thought they were looking at. This could be because of several reasons. Firstly, it might be that the participants were not really consciously thinking about what they were looking at. Therefore, they did not know what they had been focussing on when filling in the survey. Secondly, it could be that the participants simply forgot what they were looking at. Thirdly, they might have been disinterested or in a hurry so they might have filled in the survey in a rush. Lastly, the act of looking at a picture of the painting could have distorted their memory. On the picture other things might seem interesting all of a sudden. Maybe this made them think they were actually looking at those things instead of the spots they were actually looking at.
  2. The gazeplot of the 3D printed painting is more dispersed than the gazeplot of the real painting. This might be because the 3D printed painting does not have gloss details. Therefore, it might look more flat and equally interesting across the whole surface. The real painting does have gloss detailing, which causes certain points to stand out more than others.

The pilot left us with a few improvement points for our setup as well as the pilot itself. The pilot would be better if we changed/added the following features:

  • In the survey, the question about the realism of the painting could have been misleading. Test subjects thought of a simple, flat, paper surface when reading the words ‘printed painting’. Therefore, these test subjects thought the 3D printed painting was real. Had we used the word ‘3D printed painting’ the results might have been different.
  • The questions in the survey and the actual test should have been more focussed on precision of the test setup instead of the perceived realism of the paintings. That would have been more useful when improving the setup.
  • We forgot to ask a ‘why’ question about the comfort. We now have an average comfort level, but we do not know why our participants experienced a certain level of comfort.

The setup can be improved by the following:

  • The head support could be a bit more comfortable. Firstly, the cushion should not be able to wiggle. Because it is so wiggly right now, it does not offer much support. Secondly, it might be better to use a kind of foam with a dent for the chin. This minimises to possibility to move even more.
  • The distance between the test subjects and the painting is a bit too big. To really see the detailing of the painting, you should be standing a bit closer to the painting.
  • The calibrations were a bit off almost every time. There is much room for improvement here.

All these research results lead us to the next step: improving the setup, mainly focussing on improving the precision of the eye-tracking results. We will focus on these aspects in our next blogpost.


Comments are closed.