Question on the term "pretrained BertSum"

[robinsongh381]

On your paper, you mentioned "The encoder is pretrained BertSum" and im not entirely sure on the meaning of "pretrained BertSum".

For an input document, you modify BERT as follows: (1) Add CLS tokens in the beginning of each sentence (2) interval segment embeddings

This leads to BertSum.

On what task(s), do u perform and call the trained model "pretrained BertSum" ?

Just MLM task ? or borth MLM and NSP just like in BERT pretrain ?

Thanks in advance

[huseinzol05]

BERTSUM just a feeding technique, i am serious, I study the code to translate to Tensorflow.

[robinsongh381]

Are you sure ?

If you just change the way of constructing an input (i.e/ (1) Add CLS tokens in the beginning of each and (2) interval segment embeddings) and use the pretrained BERT whose input involves in only one CLS token for each input, how do we get meaningful output ?

I thought that we need to "pretrain again" with the BERTSUM feeding method against MLM and NSP ?!

What do u think @huseinzol05 ?

[huseinzol05]

Nope, no need. It is literally just constructing the inputs. I reversed engineered the code to Tensorflow already.

[robinsongh381]

@huseinzol05 I am surprised ! Is there any chance that you could possibly share the tensorflow code ? That would be greatly helpful for my understanding

[huseinzol05]

This is preprocessing and tokenization, https://github.com/huseinzol05/NLP-Models-Tensorflow/blob/master/extractive-summarization/preprocessing-data-bert.ipynb

This is the model, https://github.com/huseinzol05/NLP-Models-Tensorflow/blob/master/extractive-summarization/4.bert-base.ipynb

I got bad result for now,

1. I disable use_position_embeddings inside https://github.com/huseinzol05/NLP-Models-Tensorflow/blob/master/extractive-summarization/modeling.py#L190

Original implementation is using use_position_embeddings. Problem if we implemented use_position_embeddings, max length BERT can accept is 512 length, and obviously, some texts we want to summarize longer than that.

If we check how nlpyang/PreSumm tackle this issue, https://github.com/nlpyang/PreSumm/blob/master/src/models/model_builder.py#L150 , nlpyang/PreSumm repeat after 512, and that pytorch code is unable to done it tensorflow.

[robinsongh381]

Thanks for your reply.

Can we still use use_position_embeddings, with the input sequence length greater than 512 ? In my opinion, 512 is just a hyper-parameter and we can simply change it to another value.

Back to my question, you still do not provide answers to my question.

As you said, if BertSum is just a way of modifying input, rather than a new way of pre-training, I don't see that the former definition of BertSum can capture the true meaning of CLS token.

Without re-pre-training Bert with BertSum input architecture, your suggestion of BertSum won't understand the true meaning of CLS tokens for inputs longer than 2 sentences.

Please correct me if I am wrong.

Regards

[huseinzol05]

Can we still use use_position_embeddings, with the input sequence length greater than 512 ? In my opinion, 512 is just a hyper-parameter and we can simply change it to another value. We can't, use_position_embeddings in Tensorflow code will exception if sequence longer than 512.

As you said, if BertSum is just a way of modifying input, rather than a new way of pre-training, I don't see that the former definition of BertSum can capture the true meaning of CLS token.. It will learn during transfer learning extractive / abstractive summarization. We know original BERT can capture 2 sentences by putting CLS token, example, text similarity transfer learning. BERTSUM just added CLS after N sentences, and during transfer learning, we give indices of CLS to let know BERTSUM when to gather output sequences to multiple output N sequences depends on CLS counts.