Identifying Edits from Metadata.

We begin with the edit metadata collected by Hale [7] for the largest 46 language editions of Wikipedia and extract the data for three of the largest Wikipedia editions, in terms of the number of articles: English (en), German (de), and Spanish (es). The metadata includes article titles, language editions, timestamps, editor ids, and URLs to the content of each edit captured from a near real-time broadcast on Internet Relay Chat. The extracted data comprises 2,799,729 edits by 146,616 distinct editors. We discard edits to non-article pages including article talk pages and user pages. We further discard edits that are indicated as being made by Wikipedia (ro)bots. In contrast to Hale [7], we retain edits marked as “minor” because what one considers as minor may vary from person to person and language to language.

Article Edit Session.

Even though the easiest way to measure the edit activity on Wikipedia is simply counting each edit when changes are submitted, this does not accurately reflect editors’ behavior because of individual differences in activity patterns. For example, some editors may submit a few large edits while others may make a series of smaller edits saving the pages more frequently as they work. To account for these individual differences, we adopt the idea of edit sessions [26], which measure the labor of Wikipedia editors whose contributions tend to occur in bursts. In our work, we limit each edit session to a single document, and so we rename it article edit session. We use one hour as the cutoff between intra-session and inter-session edit activities.

In other words, from the collected dataset, we define an article edit session as a sequence of continuous editing activity without a break of more than one hour for a specific article by a single editor.

Primary vs. Non-Primary Editors.

Next, we identify multilingual editors from the metadata and retrieve the content of all the edits made by multilingual editors from Wikipedia using the Wikipedia API. We define a multilingual editor as an editor who edits two or more language editions. Using this definition, we identified 12,577 multilingual editors with 427,529 total article edit sessions composed of 622,766 distinct edits. Following Hale [7], we operationally define primary and non-primary users as follows:

• We identify the primary language of an editor as the language of the edition that the multilingual editor edits most frequently.
• We identify the non-primary languages of an editor as all the other languages that they edit.
• For each language edition X, we define primary editors as the set of all users with primary language X.

Thus, an editor can be a primary editor in only one language edition, but can be a non-primary editor in multiple language editions.

Fig 1a shows the distribution of the number of languages per user in our multilingual editor dataset. We found that most multilingual editors contribute to two or three language editions. There are also a small number of editors with edits in more than 10 languages. Table 1 contains the statistics of the edit history data we use, and Fig 1b shows the proportions of the primary languages of editors who are contributing to the English (en), German (de), and Spanish (es) editions. The data for the three languages are available online [27].

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1.

(a) Number of languages multilinguals edit in Wikipedia. Regarding the number of languages they edit, 77.3% of multilingual editors are bilingual, followed by 11.4% trilingual and 4.1% and quadrilingual editors. We discard editors who edited more than 10 languages (these editors account for 2.3% of all multilingual editors). (b) Primary languages of multilingual editors for three of the largest language editions. The English edition has the largest, yet the most varied, number of multilingual editors by primary language. 32.9% of the multilingual editors who edited the English edition primarily edited English. In comparison, 49.9% of the multilingual editors in the German edition primarily edited the German edition. Multilingual editors who primarily edit English are the second-largest proportion of multilingual editors in the Spanish and German editions. (c) Second most used languages for the three primary editor groups. English primary editors have much more diverse language usage, compared to German and Spanish primary editors. Most of German and Spanish primary editors contribute to English edition as their second most edited version.

https://doi.org/10.1371/journal.pone.0155305.g001

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Number of editors, article edit sessions, edits, and edit paragraphs.

There is more activity in the English edition (en) than in either the German (de) or Spanish (es) edition. In all three language editions there are more primary editors (p) than non-primary editors (np) and primary editors are more active than non-primary editors.

https://doi.org/10.1371/journal.pone.0155305.t001

Collecting Edit Text.

Wikipedia provides the previous and current versions of each article. To look at the actual edited text, we download the “diffs,” which are the parts of the text that have changed from the previous version of the article for each article edit session we identified. Fig 2 shows an example of the diff provided by the Wikipedia Web interface. For each article edit session, we extract the pairs of the changed paragraphs and convert the edit from Wiki markup to plain text.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Two example diffs of edits to a Wikipedia article 2015 FIFA Women’s World Cup.

(a) Edit paragraph containing a visible edit. (b) Edit paragraph containing a non-visible edit. We define an edit paragraph as one line of Wikipedia markup, utilizing the diff from Wikipedia. Pre-edit text and post-edit text are shown on the left and right sides, respectively, with text differences highlighted in yellow (deleted text) and blue (inserted text).

https://doi.org/10.1371/journal.pone.0155305.g002

In this way, we retain only the visible text from edits, discarding all non-visible and non-text information including URL, multimedia, metadata, and document structure. We regard an edit as non-visible if there is no visible text change. An example of such an edit is adding a link to existing text but otherwise making no other changes.

Distribution of Article Edit Sessions.

We examine the distribution of multilingual editors’ contributions by language and the number of article edit sessions. Fig 3 shows the distribution of editors by the number of article edit sessions. The plot, on a log-log scale, shows that the distributions for primary and non-primary users in all three editions is heavy-tailed; most users perform only a few edits while a few users perform many edits.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Distribution of editors and article edit sessions for the three language editions—English (EN), German (DE), and Spanish (ES)—for primary (P) and non-primary (NP) multilingual editors.

We display the number of article edit sessions up to 100. Dots in the same color denote the same language edition. The plot, on a log-log scale, shows that the distributions for primary and non-primary users in all three editions is heavy-tailed; most users perform only a few edits while a few users perform many edits.

https://doi.org/10.1371/journal.pone.0155305.g003

Examining Edits in Four Aspects.

We define an edit paragraph as a line of Wiki markup in an article, utilizing the difference of the text before and after the edit from Wikipedia as shown in Fig 2. In an article edit session, all of the edit paragraphs before revisions, shown in the left side of Fig 2, are paired with their corresponding revised edits, shown in the right side of Fig 2. With those paired paragraphs, we examine the edit paragraphs in four ways as follows:

• Pre-edit: Edit paragraph (text) before revision
• Post-edit: Edit paragraph (text) after revision
• Diff: The actual revision including both inserted and deleted text in an edit
• Delta: The difference between a Pre-Edit paragraph and its corresponding Post-Edit paragraph for each computed measure

The Diff includes both inserted and deleted text and is used to measure an editor’s direct contributions. The difference between computed measures for Pre-edits and Post-edits, defined as Delta, is an indirect approach that considers the context of an edit.

Comparing Editor Behaviors within Languages.

Rather than comparing each editor’s behavior across languages, we compare editing behavior of primary versus non-primary editors within each language, as defined in the previous section. That means, for example, we take all editors in our dataset who contributed to any articles in the English edition, divide them into two groups: 1) those who contributed to the English edition more than any other language edition, and 2) those who contributed to some other language edition more than the English edition, and then we compare the behaviors of those two groups. This way, we do not need to worry about the inherent differences among languages (e.g., German tends to have longer words than English).

Editor Engagement.

Since we categorized multilingual editors into two groups (primary and non-primary) on the basis of the number of article edit sessions they had across language editions, we measure the level of quantitative engagement of editors within an article edit session, such that the measure becomes independent of the number of article edit sessions.

We measure the amount of engagement based on four metrics. First, we count the number of revisions committed within an article edit session, only including the sessions containing more than 1 edit. Second, we measure the session length in minutes by the difference of timestamps between the first and the last revision in each article edit session. Third, we measure the amount of text added and deleted in terms of characters, words, and sentences for all the Delta in a session. Last, we count the number of edit sessions that only include non-visible edits indicating the changes the editor made did not affect the text of the article (i.e, modifying URL link, images, etc.). By computing this, we can examine whether an editor’s engagement results in article content changes. These four metrics are summarized in Table 2.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Measures of engagement, interest, and language proficiency.

https://doi.org/10.1371/journal.pone.0155305.t002

For each metric, we compute the mean over all article edit sessions for each editor, and then again compute the mean of primary and non-primary editors to represent the level of engagement for each groups. We perform two-tailed independent samples t-test to examine if the differences between the means of the two groups are significant.

Editor Interest.

Assuming that the topics of the edited articles represent editors’ interests in specific fields, we measure the interests of multilingual editors using a Bayesian topic model. We first determine the main topic category for each article a multilingual user edited and assign each article a single topic label. We then compute the proportion of interest from primary and non-primary editors for each topic. By comparing the distributions, we can compare the interests of primary and non-primary editors.

• Topic Modeling. We use a Bayesian topic model as an automatic method to categorize Wikipedia articles into different topics. Although Wikipedia already provides “categories”, there are many cases where one Wikipedia article is assigned to multiple categories. Moreover, Wikipedia categories form a complex network in which the relationships between categories are often unclear [28]. Instead of using these Wikipedia-defined categories, we develop a more consistent and replicable methodology. We first model the topics using a widely-used topic model, the latent Dirichlet allocation (LDA) [29] using the Python Gensim [30] library with the online variational Bayes algorithm. We set the number of topics (k) to 100 and use the default values for the hyperparameters.
• Clustering and Labeling Articles. With the 100-dimensional topic proportion vectors generated by LDA from the step above, we cluster the articles using DBSCAN [31] in the Python Scikit-learn [32] package. The generated clusters show consistency in the topics of the articles; hence, we consider each cluster as representing a single topic. We manually examine the articles in each cluster, assign a topic label to the cluster, and assign the same topic label to each article in the cluster. This process results in 20 clusters and their topic labels for each language edition with all articles labeled with one of the 20 topic labels. To verify the clustering process, we simply compare the inter-cluster and intra-cluster distances using Euclidean distance. In Eq (2), the numerator is the average of all pairwise distances of the articles in a cluster, representing the intra-class distance. The denominator is the mean of the pairwise distances of the cluster medoids for all clusters, representing the inter-class distance. For both the numerator and the denominator, c_i denotes cluster i in the set of all clusters C, and x denotes each article in c_i. The resulting I_ci represents the ratio of intra-class to inter-class distance; so, we would expect this ratio to be significantly below one if the clustering is done well. The average value of I_ci for ith cluster is 0.59 with a standard deviation 0.22. (1) (2)
• Comparing Levels of Interest. The previous step provides topic labels for each article. With those labels, we can define, for each multilingual editor, the level of interest on a topic as the proportion of the article edit sessions for articles in that topic over all article edit sessions by the editor. Then, we average the interest levels of editors in the primary and non-primary groups. We define this as the “normalized frequency of edits for each topic” (also listed in Table 2). To compare the interest levels between the two groups, we perform a two-tailed independent samples t-test for each topic.

Language Proficiency.

In order to measure the language proficiency of multilingual editors, we focus on three aspects of their edits: (1) Pre-Edits, (2) Delta, and (3) Diff.

Using Pre-Edits, we analyze the language complexity of the paragraphs of the articles that multilingual editors choose to revise. Using Delta and Diff we quantify the editors’ contributions in terms proficiency to estimate editors’ linguistic abilities. With Pre-Edits and Delta, the textual contents are sufficient in length to perform both lexical diversity and syntactic complexity measures, which have previously been used as estimates of language proficiency in language acquisition research [33]. Meanwhile, for Diff, which often consist of just a few words, we focus on the definite and indefinite articles edited (e.g., ‘a’, ‘an’, and ‘the’ in English). These have also been shown by previous research to be a good proxy for language proficiency in English, German, and Spanish [34–36].

• Lexical Diversity Measures. We first compute the entropy of unigram, bigram, and trigram frequencies as complexity measures [37]. These measures are used to quantify the richness of word usage in Wikipedia articles [18]. To calculate the entropy for each edit paragraph, we average the metrics over all unigrams, bigrams, and trigrams that appear in the paragraph.
• Syntactic Complexity Measures. We compute the entropy of parts-of-speech (POS) frequency. Analyzing the sequence of POS has advantages over analyzing the sequence of words. POS entropy can ignore extremely trivial edits (e.g., correcting typos) as well as meaningless bot-produced edits [18]. In addition, looking for diverse combinations of POS is a good approach for detecting complex syntactic structure [38]. We count the occurrence of POS unigrams, bigrams, and trigrams in each edit paragraph and then compute an entropy measure based on the frequency distributions of POS that captures the amount of information and thus indicates the diverse use of POS in the edits.
  For automatic POS tagging on edits, we employ a maximum entropy POS tagger [39] trained on the Penn Treebank corpus for English edits with Penn Treebank tagset [40]. For German edits, we use the Stanford log-linear POS tagger [41] trained on the NEGRA corpus with Stuttgart-Tübinger Tagset (STTS) [42]. For Spanish edits, we use the tagger trained on the AnCora corpus with its tagset [43].
• Usage of Articles. A number of previous studies investigate the difficulty of learning the definite and indefinite articles for language learners. For example, Butler [34] finds that children who are learning English as their second language (L2) have more difficulty in understanding articles compared to children who are acquiring English as their first language (L1). They show that L1 learners make lower frequency of errors than L2 learners. In addition, Japanese-speaking English learners have the most difficulty in understanding the article system and even proficient English learners only record about a 90% of success rate when given a task to choose the proper article [44]. In addition, the difficulty of the article system in Spanish and German are comparable to English [35, 36].
  The difficulty comes from the complex usage of articles: i.e., that there is not a one-to-one correspondence between languages. Such difficulty imposes a challenge for language learners of a second language [34]. Since it is computationally challenging to automatically test the appropriateness of the articles editors use, we approximate the level of understanding of the article system in each language by measuring the frequencies with which definite and indefinite articles are used. We count the number of definite and indefinite articles in each Diff by primary and non-primary users. The definite and indefinite articles we count in each language are shown in Table 3. The number of articles in a Diff is divided by total number of words of the Diff, and we refer to this quantity as the “fraction of articles in added tokens.”
• Computing Language Complexity Measures for Each Editor. The language proficiency measures were summarized with a maximum value for each editor rather than the mean value. We compute the maximum value of the proficiency measures for all edits belonging to the same editor as a representation of that editor’s highest displayed level of linguistic ability to produce complex edits. The maximum value is a fairer estimate than the mean or the minimum value as not all possible revisions to an article always require an editor’s maximum linguistic ability. For this reason, widely used central tendency measures (e.g., mean) would not reflect editors’ true proficiencies properly. Summarizing each editor’s edits with the maximum value assumes that each editor shows his/her maximum linguistic ability to edit a paragraph at least once, which is more reasonable.
  Since the maximum value may be merely affected by the number of edits, we control the number across editors to three. We uniformly sample three edits within the entire edits an editor made, repeating 100 times for editors and average the evaluated 100 maximum values.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. The definite and indefinite articles tracked in English, German, and Spanish.

https://doi.org/10.1371/journal.pone.0155305.t003

All of the engagement, interest, and proficiency measures described in this section are summarized in Table 2.

Pre-Edit.

We first examine the complexity of Pre-Edits to understand the text which multilingual editors choose to edit in their primary and non-primary languages.

Fig 7a shows that the entropy of unigrams, bigrams, and trigrams is always higher for the edits of multilingual users writing in English or Spanish as a primary language compared to those writing in English or Spanish as a non-primary language. The same is true for unigram entropy in German, while bigram and trigram entropy show a similar pattern although the difference is not significant. In short, we observe a difference between primary and non-primary editors for all conditions, indicating that primary editors use more diverse terms while editing.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 7.

(a) Language complexity of pre-edits for primary and non-primary editors. Both lexical diversity measures and syntactic diversity measures show primary users edit more complex articles. (b) The increase of lexical and syntactic diversity per paragraph per edit (delta). The higher delta complexity scores for primary users indicate that multilinguals have higher linguistic abilities and make more complex edits in their primary languages. (c) Fraction of article words in added tokens. Primary editors use more article words, both definite and indefinite articles, than non-primary editors. (⁺ p < 0.08; *p < 0.05; **p < 0.01; ***p < 0.001.).

https://doi.org/10.1371/journal.pone.0155305.g007

Moreover, we observe that the entropy of part-of-speech unigrams, bigrams, and trigrams is significantly higher for edits by primary editors in English and Spanish. Likewise, the POS entropy of edits is always higher for primary than non-primary editors in the German edition. The difference for bigrams and trigrams is significant while the difference in unigram POS entropy is not. This indicates that syntactic complexity also differs between primary and non-primary editors in all three languages in general. Thus, we conclude that primary editors edit more complex parts of articles compared to non-primary editors. In addition, the result implies that multilingual editors writing in a non-primary language may face a complexity barrier whereby they shy away from editing more complex sections of articles.

Delta.

Delta complexity measures the difference in complexity before and after each article edit session and thus provides a measure of how the edits by the multilingual user changed the complexity of the article.

Fig 7b shows that there are significant differences between primary and non-primary groups, but that these differences are not consistent across the three language editions. Specifically, for the German and Spanish editions, we find the average increase in entropy between article edit sessions is higher from primary than non-primary editors. Surprisingly, however, we do not observe a significant difference between editors writing in English as a primary or non-primary language for the lexical diversity and syntactic complexity. Opposite to the other language editions, all entropy measures are slightly higher for non-primary editors of the English edition compared to primary editors.

These results implies that primary editors in German and Spanish editions possess higher linguistic proficiency, in terms of writing ability at least, than non-primary editors. This is consistent to the findings from the analysis of user interest. However, this does not apply to English, where there is no evidence that primary and non-primary editors of English possess different levels of linguistic proficiency.

Diff.

Fig 7c shows that there are significant differences in the use of articles between by primary and non-primary editors. For all three language editions, primary editors use more articles—both definite and indefinite—than non-primary editors. Given that choosing the proper article is a difficult task—even proficient learners only have an accuracy of ∼90% [44]—the difference in the use of articles between primary and non-primary users likely stems from different levels of proficiency. It is possible that non-primary editors simply skip using article words when confused as to which article to use.