[1910.10843] Relation Module for Non-answerable Prediction on Question Answering
In addition we augment the context vector with plausible answers, allowing us to extract objects focused on the proposed answer span, and differentiate from other objects that are not as relevant in the context
Abstract: Machine reading comprehension(MRC) has attracted significant amounts of
research attention recently, due to an increase of challenging reading
comprehension datasets. In this paper, we aim to improve a MRC model's ability
to determine whether a question has an answer in a given context (e.g. the
recently proposed SQuAD 2.0 task). Our solution is a relation module that is
adaptable to any MRC model. The relation module consists of both semantic
extraction and relational information. We first extract high level semantics as
objects from both question and context with multi-head self-attentive pooling.
These semantic objects are then passed to a relation network, which generates
relationship scores for each object pair in a sentence. These scores are used
to determine whether a question is non-answerable. We test the relation module
on the SQuAD 2.0 dataset using both BiDAF and BERT models as baseline readers.
We obtain 1.8% gain of F1 on top of the BiDAF reader, and 1.0% on top of the
BERT base model. These results show the effectiveness of our relation module on
MRC
‹Figure 1: An example of non-answerable question in SQuAD 2.0. Highlighted words are the output from the BERT base model. The true answer is “None”. (Introduction)Figure 2: Tokens highlighted by the object extractor (Introduction)Figure 3: Relation Module on BERT. S and E are hidden states trained by plausible answers. We then concatenate S and E with the contextual representation to feed into the object extractor. After we obtain the extracted objects, we then feed into a Relation Network and pass it down for NA predictions. (Introduction)Figure 4: Illustration of a Relation Network. The gθ is a MLP to score relationships between pairs (Object Extractor)Figure 5: Relation Module applied on BiDAF. (BiDAF) (Analysis)Figure 6: In each subplot, each row represents one object from our object extractor; for each object we highlight the top 5 tokens with highest weights in the entire context and question. We show the two windows where the majority of these top 5 weights occur. For example, the top purple object in the context looks at key phrases such as “##ridge earthquake” in the top subplot and “billion” in the middle subplot; the blue object in the question looks at “20 million in” in the bottom subplot. (Analysis) (Analysis)Figure 7: In each subplot, each row represents one object from our object extractor; for each object we highlight the top 5 tokens with highest weights in the entire context and question. We show a window where the majority of the top 5 weights occur. For example, there are numerous objects in the context window that look at the key phrase “some concrete” in the top subplot; the two objects in the question look at the key phrase “the abstract” in the bottom subplot. (Analysis)›
