A Dataset for Analysing Complex Document Layouts in the Digital Humanities and Its Evaluation with Krippendorff’s Alpha

Abstract

We introduce a new research resource in the form of a high-quality, domain-specific dataset for analysing the document layout of historical documents. The dataset provides an instance segmentation ground truth with 19 classes based on historical layout structures that stem (a) from the publication production process and the respective genres (life sciences, architecture, art, decorative arts, etc.) and, (b) from selected text registers (such as monograph, trade journal, illustrated magazine). Altogether, the dataset contains more than 52,000 instances annotated by experts. A baseline has been tested with the well-known Mask R-CNN and compared to the state-of-the-art model VSR [55]. Inspired by evaluation practices from the field of Natural Language Processing (NLP), we have developed a new method for evaluating annotation consistency. Our method is based on Krippendorff’s alpha (K-$\alpha$), a statistic for quantifying the so-called “inter-annotator-agreement”. In particular, we propose an adaptation of K-$\alpha$ that treats annotations as a multipartite graph for assessing the agreement of a variable number of annotators. The method is adjustable with regard to evaluation strictness, and it can be used in 2D or 3D as well as for a variety of tasks such as semantic segmentation, instance segmentation, and 3D point cloud segmentation.

Appendices

Appendix 1

In this part some example calculations for K-$\alpha$ are illustrated. It shows different forms of how the metric can be adapted. As a helpful resource for further understanding we recommend reading [33]. There are two aspects that we assume will affect most adaptation possibilities: the creation of the cost matrix, which would change the creation of the reliability data and the handling of missing data when calculating K-$\alpha$.

Figure 5 is used as an example and the resulting reliability matrix after the graph matching would look as shown in the matrix below. For this matching a IoU threshold of 0.5 is used and the normal matching is used.

	Unit 1	Unit 2	Unit 3	Unit 4	Unit 5
$\varOmega _A$	$a_1$	$a_2$	$a_3$	$a_4$	$\varnothing$
$\varOmega _B$	$b_1$	$b_2$	$b_3$	$b_4$	$\varnothing$
$\varOmega _C$	$\varnothing$	$c_2$	$c_3$	$c_4$	$c_1$

Fig. 5.

Three annotator visual examples of matching with $IoU>0.5$ threshold. Image source: https://commons.wikimedia.org/wiki/File:Smile_at_a_stranger.jpg

1.1 Strict Matching, Don’t Allow Missing Data

After the reliability data is calculated all values $a_n$, $b_m$ and $c_p$ are replaces with their classes and for this case which doesn’t allow missing data $\varnothing$ is replaced with the filler class 0. Hence, the matrix would look as follow:

	Unit 1	Unit 2	Unit 3	Unit 4	Unit 5
$\varOmega _A$	1	1	3	2	0
$\varOmega _B$	1	1	3	2	0
$\varOmega _C$	0	2	3	2	1

From here K-$\alpha$ is calculated the regular way, by first creating the coincidence matrix. Unit 1 contains $3(3-1)=6$ pairs, 2 matching 1–1 pairs, 2 mismatching 1–0 pairs and 2 mismatching 0–1 pairs, it contributes $\frac{2}{3-1}=1$ to the $o_{1,1}$ cell, $\frac{2}{3-1}=1$ to the $o_{1,0}$ cell and $\frac{2}{3-1}=1$ to $o_{0,1}$ cell. Unit 2 contains $\frac{3}{3-1}=6$ pairs, 2 matching 1–1 pairs, 2 mismatching 1–2 pairs and 2 mismatching 2–1 pairs, it contributes $\frac{2}{3-1}=1$ to the $o_{1,1}$ cell, $\frac{2}{3-1}=1$ to the $o_{1,2}$ cell and $\frac{2}{3-1}=1$ to the $o_{2,1}$ cell. Unit 3 contains $\frac{3}{3-1}=6$ pairs, 6 matching 3–3 pairs, $\frac{6}{3-1}=3$ to the $o_{3,3}$ cell. Unit 4 contains $\frac{3}{3-1}=6$ pairs, 6 matching 2–2 pairs, it contributes $\frac{6}{3-1}=3$ to the $o_{2,2}$ cell. Unit 5 contains $\frac{3}{3-1}=6$ pairs, 2 matching 0–0 pairs, 2 mismatching 0–1 pairs and 2 mismatching 1–0 pairs, it contributes $\frac{2}{3-1}=1$ to the $o_{0,0}$ cell, $\frac{2}{3-1}=1$ to the $o_{0,1}$ cell and $\frac{2}{3-1}=1$ to the $o_{1,0}$ cell. As an example the first value in the coincidence matrix $o_{0,0}$ is the sum of all value in the five units related to $o_{0,0}$, which is rather straight forward since only unit 5 contains 0–0 pairs, hence $o_{0,0}=1$. The coincidence matrix is as follows:

	0	1	2	3
0	1	2	0	0	3
1	2	2	1	0	5
2	0	1	3	0	4
3	0	0	0	3	3
	3	5	4	3	15

Computing K-$\alpha$ is now done via Eq. 3, which means for our example:

$$\begin{aligned} \alpha = \frac{(15-1)(1+2+3+3)-[ 3(3-1) + 5(5-1) + 4(4-1) + 3(3-1) ]}{15(15-1)-[ 3(3-1) + 5(5-1) + 4(4-1) + 3(3-1) ]} = 0.49 \end{aligned}$$

1.2 Strict Matching, but Allow Missing Data

A second possible version build on the same example shown in Fig. 5 that allows missing data, would transfer the reliability data slightly different. Instead of 0 a * will be used indicating missing data, which won’t be included in the calculation of $\alpha$.

	Unit 1	Unit 2	Unit 3	Unit 4	Unit 5
$\varOmega _A$	1	1	3	2	*
$\varOmega _B$	1	1	3	2	*
$\varOmega _C$	*	2	3	2	1

Calculating the coincidence matrix would be done in the same way as before for Unit 2, Unit 3 and Unit 4, but Unit 1 and Unit 5 are different. Unit 1 contains $2(2-1)=2$ pairs, which are 2 matching 1–1 pairs it contributes $\frac{2}{2-1}=2$ to the $o_{1,1}$ cell. Since Unit 4 only contains a single entry, no pairable unit can be found. The coincidence matrix would therefore be:

	1	2	3
1	3	1	0	4
2	1	3	0	4
3	0	0	3	3
	4	4	3	11

This results in a calculation of alpha with the following values:

$$\begin{aligned} \alpha = \frac{(11-1)(3+3+3)-[4(4-1)+4(4-1)+3(3-1)]}{11(11-1)-[4(4-1)+4(4-1)+3(3-1)]} = 0.75 \end{aligned}$$

Appendix 2

Further information to the historical sources:

The selected documents were already available as digitized sources. They all come from publicly accessible digital collections. These are: the digital collections of University Library Heidelberg (“Pädagogisches Skizzenbuch” [29], “Das Kunstgewerbe” [5] and “Fliegende Blätter” [2]), the Internet Archive (“Zeitschrift für Physiologie und Psychologie der Sinnesorgane” [4]), the Virtual Laboratory (“Lehre von den Tonempfindungen” [25]) and the digital collections of the Berlin State Library (“Centralblatt der Bauverwaltung” [3]).

The Pedagogical Sketchbook by Paul Klee is part of the artistic-experimental domain. It is the second volume in the Bauhaus book series. The so-called Bauhaus books are a series of books published from 1925 to 1930 by Walter Gropius and Lazlo Moholy-Nagy. Although the books appeared as a series in the same publishing house (Albert Langen Verlag), the respective layout varied widely [11]. The publication sequence was also irregular. While in 1925 alone eight publications of the series could be published, in 1926 there were only two and in 1927, 1928, 1929 and 1930 one more volume each. The publication of Klee presented not only his artwork but also presented his art theoretical knowledge. At the same time, it presented aspects of his extensive lectures on visual form at the Bauhaus and conveyed his way of thinking and working on this topic.

Both the journal “Physiology und Psychologie der Sinnesorgane” and Hermann von Helmholtz’s publication “Lehre von den Tonempfindungen” are part of the domain of life sciences. The different types of publications (journal and monograph) have different but typical layout components within their domain, which is why they were both integrated into the dataset.

The journal “Das Kunstgewerbe” appeared every fourteen days from 1890 to 1895 and belongs to the domain of applied arts. The individual issues had a length of 10 pages and a manageable number of illustrations, but the pages were often designed with decorative frames and ornaments.

The illustrated magazine “Fliegende Blätter” appeared from 1844 to 1944, at first irregularly several times a month, later regularly once a week. The humorous-satirical publication was richly illustrated and held in high esteem among the German bourgeoisie. At the same time, the “Fliegende Blätter” are significant both artistically and in terms of printing, due to the high quality of its layout [50].

The “Centralblatt der Bauverwaltung” was a professional journal intended to satisfy the need for information in the construction sector. The journal was first published in April 1881 by the publishing house Ernst & Sohn, in 1931 it was merged with the “Zeitschrift für Bauwesen”, in 1944 the publication was discontinued. The Ministry of Public Works acted as publisher until 1919, and from 1920 to 1931 the Prussian Ministry of Finance. The journal was to serve as a supplement to the existing trade journals and, in contrast to these, was to have a faster publication schedule. Information about construction projects and competitions, projects currently being implemented, new technologies and amended legal framework conditions were to reach the readership more quickly than before and also address international developments. At the same time, however, the journal was to be less elaborately designed than the existing trade organs and art journals. Although the Ministry of Public Works was the editor and the structure of the journal was divided into “official” and “non-official” parts, it can nevertheless not be characterized as a purely official journal of authorities [19].

Appendix 3

See Table 4.

Table 4. Benchmark results on TexBiG comparing Mask R-CNN and VSR for object detection and instance segmentation. This is the extended version of the table, including the different classes.