Semanlink - [1907.07355] Probing Neural Network Comprehension of Natural Language Arguments

Tags:

Au sujet de ce document

sl:arxiv_author :
- Hung-Yu Kao
- Timothy Niven
sl:arxiv_firstAuthor : Timothy Niven
sl:arxiv_num : 1907.07355
sl:arxiv_published : 2019-07-17T06:26:20Z
sl:arxiv_summary : We are surprised to find that BERT's peak performance of 77% on the Argument Reasoning Comprehension Task reaches just three points below the average untrained human baseline. However, we show that this result is entirely accounted for by exploitation of spurious statistical cues in the dataset. We analyze the nature of these cues and demonstrate that a range of models all exploit them. This analysis informs the construction of an adversarial dataset on which all models achieve random accuracy. Our adversarial dataset provides a more robust assessment of argument comprehension and should be adopted as the standard in future work.@en
sl:arxiv_title : Probing Neural Network Comprehension of Natural Language Arguments@en
sl:arxiv_updated : 2019-09-16T04:07:54Z
sl:bookmarkOf : https://arxiv.org/abs/1907.07355
sl:creationDate : 2019-07-24
sl:creationTime : 2019-07-24T01:34:54Z
sl:relatedDoc : http://www.semanlink.net/doc/2019/07/bert_s_success_in_some_benchmar

Infos sur le fichier

Bookmark of: https://arxiv.org/abs/1907.07355

Documents with similar tags (experimental)

[2002.12327] A Primer in BERTology: What we know about how BERT works

Tags:

2020-02-28 A propos

[2003.05473] Investigating Entity Knowledge in BERT with Simple Neural End-To-End Entity Linking (CoNNL 2019)

Tags:

Training BERT-base-uncased on English Wikipedia and then fine-tuned and evaluating it
on an entity linking (EL) benchmark (EL implemented as a token classification over the entity vocabulary)

> BERT+Entity is a straightforward extension on top
of BERT, i.e. we initialize BERT with the publicly
available weights from the BERT-base-uncased
model and add an output classification layer on
top of the architecture. Given a contextualized token,
the classifier computes the probability of an
entity link for each entry in the entity vocabulary.

Can BERT’s architecture learn all entity
linking steps jointly? To answer:

> an extreme
simplification of the **entity linking setup that
works surprisingly well**: simply cast it as **a
per token classification over the entire entity
vocabulary** (over 700K classes in our case).

> the model
is the first that performs entity linking without any
pipeline or any heuristics, compared to all prior
approaches. We found that with our approach we
can learn additional entity knowledge in BERT that
helps in entity linking. **However, we also found
that almost none of the downstream tasks really
required entity knowledge**.

### Related work

- > [Durrett and Klein (2014)](/doc/2020/01/a_joint_model_for_entity_analys) were the first to propose
jointly modelling Mention detection, Candidate generation and Entity disambiguation in a graphical
model and could show that each of those steps are
interdependent and benefit from a joint objective

This paper uses neural techniques instead of CRF.

- > [Yamada](/showprop.do?pptyuri=http%3A%2F%2Fwww.semanlink.net%2F2001%2F00%2Fsemanlink-schema%23arxiv_author&pptyval=Ikuya%2BYamada) (2016, 2017) was the first to
investigate neural text representations and entity
linking, but their approach is limited to ED.

cf. [#Wikipedia2Vec](tag:wikipedia2vec). Compare with [newer work by Yamada](doc:2020/09/1909_01259_neural_attentive_b)

2020-01-09 A propos

[1906.04341] What Does BERT Look At? An Analysis of BERT's Attention

Tags:

2019-06-21 A propos

[1906.02715] Visualizing and Measuring the Geometry of BERT

Tags:

2019-06-07 A propos