For further information, please contact Yunus Emre KARA.

Head Pose Annotations Dataset

The Head Pose Annotations Dataset is a crowdsourced annotation dataset with known ground truth values, which is collected for the purpose of evaluating crowd-labeling methods. Apart from consensus estimation, this dataset can be used in a multitude of ways in machine learning and artificial intelligence research. If you use this dataset in your work, please cite the following article:
  • Kara, Yunus Emre
  • Genc, Gaye
  • Aran, Oya
  • Akarun, Lale
(2018). Actively Estimating Crowd Annotation Consensus, Journal of Artificial Intelligence Research, 61, 363-405.

                    

Collection of this dataset was funded by "Multimodal Computational Modeling of Nonverbal Social Behavior in Face to Face Interaction (SOBE), SNSF Ambizione Fellowship Project".

Annotated Data (Head Pose Image Database)

We asked the annotators to annotate the samples of the Head Pose Image Database*. The dataset has both "pan" and "tilt" ground truth values for a total of 2790 photos. The "tilt" values in the dataset are {-90, -60, -30, -15, 0, +15, +30, +60, +90} degrees and the "pan" values are {-90, -75, -60, -45, -30, -15, 0, +15, +30, +45, +60, +75, +90} degrees. The pan-tilt pairs used in the dataset result in 93 unique head pose configurations. There are two series of photos in which 15 subjects portrayed all of these configurations. 6 subjects in the dataset wear glasses in one of their photo series. Some samples from Head Pose Image Database:
Head pose samples

(*) Gourier, N., Hall, D., & Crowley, J. L. (2004). Estimating face orientation from robust detection of salient facial features. In ICPR International Workshop on Visual Observation of Deictic Gestures.

Annotation Task

Due to budgetary constraints, we submitted only a subset of these images to the CrowdFlower (rebranded as Figure Eight) platform for annotation. We chose only one photo series for each subject. If available, we chose the photo series with glasses, otherwise the first series was used. We tried to choose a balanced combination of images with and without glasses. For pan and tilt values, we chose the photos with {-90, -60, -30, 0, +30, +60, +90} degrees in both dimensions. A total of 555 photos were annotated. For each photo, we asked the participants to annotate three questions: The following figure shows the annotation task instructions shown to the annotators:
Annotation task instructions shown to the annotators
The figure below shows a sample of what the annotators see when they are working on our head orientation tagging task.
The Annotations Dataset sample question

Annotation statistics

In the table below, we present the annotation frequency of the samples. Out of 555 samples, 475 have 9 annotations, with other samples having as few as 7 and as many as 17 annotations.
Number of annotations per sample for the dataset
Sample annotation count 7 8 9 15 16 17
Number of samples 10 10 475 6 34 20
The next table shows the annotation frequency of the annotators, which we call annotator workload. A total of 189 annotators participated in the annotation tasks. Most common annotator workloads are multiples of 10 since many annotators completed the batch tasks assigned to them. For example, 61 annotators annotated 10 samples and 2 annotators annotated 100 samples.
Annotator workloads (the number of annotations made by an annotator) for the dataset
Annotator workload 5 10 17 20 24 30 39 40 45 50 55 60 70 75 80 84 90 100
Number of annotators 1 61 1 45 1 26 1 15 2 13 1 7 5 1 4 1 2 2

Download

There are five files in the archive:
  1. headpose_groundtruth.csv: Defines the groundtruth values for the samples. Each row of the file has semi-column separated ground truth values, respectively, as follows
    • sample id
    • whether the person is wearing glasses or not,
    • tilt orientation angle of the head
    • pan orientation angle of the head
    Glasses column has binary values in the form of yes/no. Pan and Tilt columns have values in the [-90, 90] range.
    Sample rows from the headpose_groundtruth.csv file
    SampleIDGlassesTiltPan
    person01100-90+0.jpgyes-900
    person01101-60-90.jpgyes-60-90
    person01103-60-60.jpgyes-60-60
    ...
    ...
  2. headpose_annotations.csv: Defines the annotations of the samples. Each row of the file has semi-column separated annotation values, respectively, as follows
    • id of the annotator of this annotation
    • id of the sample of this annotation
    • glasses annotation
    • head tilt orientation annotation
    • head pan orientation annotation
    Glasses column has binary values in the form of yes/no. Pan and Tilt columns have values in the 1-7 range.
    Sample rows from the headpose_annotations.csv file
    AnnotatorIDSampleIDGlassesTiltPan
    29899677person05111-60+60.jpgyes44
    29899677person09120-30+0.jpgno34
    28962469person15179+60-90.jpgno61
    28976121person15179+60-90.jpgno61
    28962469person01116-30-60.jpgyes32
    ...
    ...
  3. kara2018acl.pdf: The paper introducing this dataset
  4. kara2018acl.bib: BibTeX database for the above paper
  5. readme.txt: A copy of these descriptions