Abhik Naskar

14 February 2019

Text Summarisation
Text summarisation can be done through either:
1. Extraction : Copying and pasting key and relevant phrases from the origi-
nal into sentences.
Pro: robust since natural language from original piece is used
Con: not flexible since co new connectors and novel words are introduced,
can’t paraphrase.
2. Abstraction: Actually abstract from original piece.
Pro: pretty obvious
Cons : hard to come up with a model to generate coherent phrases etc and
they make a lot of mistakes.
Problems of actually implementing abstractive text summarisation before: the
output “lacked a sense of general coherence, flow and readability”. This paper
tries to address this by designing an abstractive model that address this problem.
Basic Building Blocks
Encoding/Decoding :
Recurrent neural networks (RNNs) are deep learning models that can
process sequences (e.g. text) of variable length and compute useful representa-
tions (or hidden state) for each phrase. These networks process each element of
the sequence (in this case, each word) one by one; for each new input in the se-
quence, the network outputs a new hidden state as a function of that input
and the previous hidden state. In this sense, the hidden state calculated at each
word is a function of all the words read up to that point.
RNNs can also be used to generate output sequences in a similar fashion.
At each step, the RNN hidden state is used to generate a new word that is added
to the final output text and fed in as the next input.
The input (reading) and output (generating) RNNs can be combined in a joint
model where the final hidden state of the input RNN is used as the ini-
tial hidden state of the output RNN. Combined in this way, the joint mod-
el is able to read any text and generate a different text from it. This framework is
called an encoder-decoder RNN (or Seq2Seq) and is the basis of our
summarisation model. In addition, we replace the traditional encoder RNN
by a bidirectional encoder, which uses two different RNNs to read
the input sequence: one that reads the text from left-to-right and
another that reads from right-to-left. This helps our model to have a
better representation of the input context.

Training the Model:

Common method for training: Teacher Forcing Algorithm.(https://machine-
learningmastery.com/teacher-forcing-for-recurrent-neural-networks/) The model generates
a summary and uses the reference summary to check error rate word by word.
Cons : observes the reference summary too much and it isn’t flexible.
Pros : helps ensure good language flow.
Preferred algorithm : Reinforcement Learning(RL). The RL generates
the summary and uses the reference summary to score on how good the gener-
ated summary is in comparison. The score guides what the model should update
in itself.
Pros: improves overall recall-all important information are indeed summarised.
Model Evaluation :
ROUGE : Recall oriented understudy for gifting evaluation is used. It
works by comparing sub-phrases in the generated summaries against sub-phras-
es in the ground truth reference summaries. Various variations of Rogue score
are used. Like any other score the higher the better.