Cross-Domain Generalization of Neural Constituency Parsers

What it is about

The paper trains constituency parsers on the WSJ and tests them zero-shot on Brown, Genia and the English Web Treebank, and on Chinese treebank domains. It compares non-neural parsers (Berkeley, BLLIP, ZPar) with neural ones (a self-attentive chart parser and an in-order shift-reduce parser). The neural and non-neural parsers lose similar relative accuracy when the domain changes, and the parsers that do best in-domain also do best out-of-domain. Pretrained encoders (BERT) remove 25% to 55% of the error in every domain, but the gain is not larger out of domain. Structured decoding raises exact-match accuracy in all domains and helps longer spans, where the curves for the two decoders diverge after about 30 to 40 words. Generalization is measured as the relative increase in error against the in-domain score.

Ideas

• I1. en parse-eval: report the relative error increase against the in-domain score for EWT dev and for GUM, GENTLE, PUD and ATIS (the last four as test only, once per release), so that a drop can be read as a domain effect or a model effect.
• I2. Add the share of fully correct sentences (exact match) to en parse-eval, bucketed by sentence length, since the paper finds the benefit of structure in long sentences.
• I3. gparse/src/eisner.rs: compare Eisner decoding with greedy decoding per length bucket on dev, using exact match as well as LAS.
• I4. Do not expect domain gains from pretrained encoders; Mova ships none, so the quantity to watch is the relative error increase of the from-scratch gparse out of domain, not its in-domain score.

For Mova

Mova is dependency-based, so only the method transfers, not the constituency models. The ideas touch 20. Syntax: the Universal Dependencies frame, the en parse-eval report and gparse. The measure with the most value for Mova is I1, because it tells a domain loss from a parser loss.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/p19-1031

@inproceedings{fried2019cross,
  title = {{Cross-Domain Generalization of Neural Constituency Parsers}},
  author = {Fried, Daniel and Kitaev, Nikita and Klein, Dan},
  booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1031},
  url = {https://doi.org/10.18653/v1/p19-1031}
}

References

3 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Martı́n Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. 2016. Tensorflow: A system for large-scale machine learning. In 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16), pages 265–283. find
2. Eugene Charniak. 2000. A maximum-entropy-inspired parser. In 1st Meeting of the North American Chapter of the Association for Computational Linguistics. find
3. Eugene Charniak and Mark Johnson. 2005. Coarse-to-fine n-best parsing and maxent discriminative reranking. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL’05), pages 173–180, Ann Arbor, Michigan. Association for Computational Linguistics. find our card

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/acl-2019-fried-cross-domain-generalization-neural-constituency/