An Empirical Analysis of Formality in Online Communication

What it is about

The paper studies sentence-level formality. Crowd workers rated sentences from four genres (news, blogs, emails and Yahoo Answers) on a scale from -3 (very informal) to +3 (very formal), and the agreement among raters is measured. A ridge regression model is then trained on eleven groups of features: n-grams, word embeddings, POS tags, parse productions, dependency tuples, entity types, casing and punctuation, a frequency-based formality score, readability, language-model perplexity and a lexicon. Cross-validated, the full model beats the strongest n-gram baseline by up to 11 points, and the feature ablation shows that n-grams and embeddings carry most of the signal. As a test of real style rather than topic, a pairwise task asks which of an original sentence and its rewrite is more formal: the model is right 88 percent of the time, while readability alone gets 57 percent. Applied to online debates, formality tracks lower shared context and other pragmatic ratings, as the theories predict.

Ideas

• I1. Build a count-based formality score for each meta token from function-word and POS-tag frequencies, in the spirit of the paper's frequency-based score, and sum it over a frame. Counts are deterministic and easy to audit.
• I2. Use the pairwise test as a measure. Given a source sentence and a standard rewrite, check whether the score prefers the author's version. Keep the readability-only baseline as the negative control, since 57 percent is the floor the paper reports.
• I3. Train and score per genre, as the paper does, so a register delta is learned per style class (open problem 8). Do not pool genres when the labels differ.
• I4. Run a feature-group ablation for the translator's meta-token features (open problem 4): drop one group at a time and report the change on held-out verses.

For Mova

The code is tools/translate (meta.rs for the meta-token features, measure.rs for the pairwise test). The ridge model is not for Mova, but the feature groups and the pairwise check are. They address open problem 4 (function words and boundaries as features) and open problem 8 (style deltas per class).

Topics: Part-of-speech tagging · Machine translation

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00083

@article{pavlick2016empirical,
  title = {{An Empirical Analysis of Formality in Online Communication}},
  author = {Pavlick, Ellie and Tetreault, Joel},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2016},
  doi = {10.1162/tacl\_a\_00083},
  url = {https://doi.org/10.1162/tacl_a_00083}
}

References

9 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Fadi Abu Sheikha and Diana Inkpen. 2010. Automatic classification of documents by formality. In Interntional Conference on Natural Language Processing and Knowledge Engineering (NLP-KE), pages 1–5. IEEE. find
2. Fadi Abu Sheikha and Diana Inkpen. 2011. Generation of formal and informal sentences. In Proceedings of the 13th European Workshop on Natural Language Generation, pages 187–193, Nancy, France, September. Association for Computational Linguistics. find
3. Cristina Battaglino and Timothy Bickmore. 2015. Increasing the engagement of conversational agents through co-constructed storytelling. Eighth Workshop on Intelligent Narrative Technologies. find
4. Julian Brooke and Graeme Hirst. 2014. Supervised ranking of co-occurrence profiles for acquisition of continuous lexical attributes. In Proceedings of The 25th International Conference on Computational Linguistics. find
5. Julian Brooke, Tong Wang, and Graeme Hirst. 2010. Automatic acquisition of lexical formality. In Coling 2010: Posters, pages 90–98, Beijing, China, August. Coling 2010 Organizing Committee. find
6. Penelope Brown and Colin Fraser. 1979. Speech as a marker of situation. In Social Markers in Speech, pages 33–62. Cambridge University Press. find
7. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006. The PASCAL recognising textual entailment challenge. In Machine Learning Challenges. Evaluating Predictive Uncertainty, Visual Object Classification, and Recognising Textual Entailment, pages 177–190. Springer. find
8. Cristian Danescu-Niculescu-Mizil, Michael Gamon, and Susan Dumais. 2011. Mark my words!: Linguistic style accommodation in social media. In Proceedings of the 20th International Conference on World Wide Web, pages 745–754. ACM. find
9. Cristian Danescu-Niculescu-Mizil, Lillian Lee, Bo Pang, and Jon Kleinberg. 2012. Echoes of power: Language effects and power differences in social interaction. In Proceedings of the 21st International Conference on World Wide Web, pages 699–708. ACM. Cristian Danescu-Niculescu-Mizil, Moritz Sudhof, Dan Jurafsky, Jure Leskovec, and Christopher *Figure 5 : Effect size of prior posts’s formality* on current post’s formality for posts at different positions in a thread. Effect size can be interpreted as the expected increase in a post’s formality corresponding to an increase of 1 in the prior post’s formality, all else being equal. sions may tend to adapt better to one another as the discussion progresses. We leave further investigation for future work. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/pavlick-2016-empirical-analysis-formality-online-communication/