LISA: Linguistically-Informed Self-Attention

This is the original implementation of the linguistically-informed self-attention (LISA) model described in the following paper:

Emma Strubell, Patrick Verga, Daniel Andor, David Weiss, and Andrew McCallum. Linguistically-Informed Self-Attention for Semantic Role Labeling. Conference on Empirical Methods in Natural Language Processing (EMNLP). Brussels, Belgium. October 2018.

This code is based on a fork of Timothy Dozat's open source graph-based dependency parser, and the code to run ELMo is copied from AI2's TensorFlow implementation. Thanks Tim and AI2!

This code is released for exact replication of the paper. You can find a work-in-progress but vastly improved re-implementation of LISA here.

Requirements:

• Python 2.7
• >= TensorFlow 1.1
• h5py (for ELMo models)

Quick start:

Data setup (CoNLL-2005, GloVe):

1. Get pre-trained word embeddings (GloVe):

   wget -P data http://nlp.stanford.edu/data/glove.6B.zip
   unzip -j data/glove.6B.zip glove.6B.100d.txt -d data/glove
   

2. Get CoNLL-2005 data in the right format using this repo. Follow the instructions all the way through preprocessing for evaluation.
3. Make sure data_dir is set correctly, to the root directory of the data, in any config files you wish to use below.

Download and evaluate a pre-trained model:

Pre-trained models are available for download via Google Drive here.

To download the model from Google Drive on the command line, you can use the following bash function (from here):

function gdrive_download () {
  CONFIRM=$(wget --quiet --save-cookies /tmp/cookies.txt --keep-session-cookies --no-check-certificate "https://docs.google.com/uc?export=download&id=$1" -O- | sed -rn 's/.*confirm=([0-9A-Za-z_]+).*/\1\n/p')
  wget --load-cookies /tmp/cookies.txt "https://docs.google.com/uc?export=download&confirm=$CONFIRM&id=$1" -O $2
  rm -rf /tmp/cookies.txt
}

The gdrive_download function takes two parameters: Google Drive ID and filename. You can find model names and their corresponding ids in this table:

To download and evaluate e.g. the lisa-conll05 model using GloVe embeddings:

mkdir -p models
gdrive_download 1XyfyjjjQVJK16XhqHoY9GK8GDOqZX2qk models/lisa-conll05.tar.gz
tar xzvf models/lisa-conll05.tar.gz -C models
python network.py --load --test --test_eval --config_file models/lisa-conll05/config.cfg 

To evaluate it on the Brown test set:

python network.py --load --test --test_eval --load_dir models/lisa-conll05 --config_file models/lisa-conll05/config.cfg --test_file $CONLL05/test.brown.gz.parse.sdeps.combined.bio --gold_test_props_file $CONLL05/conll2005-test-brown-gold-props.txt --gold_test_parse_file $CONLL05/conll2005-test-brown-gold-parse.txt

Evaluating with ELMo embeddings:

First, download the pre-trained ELMo model and options into a directory called elmo_model:

wget -P elmo_model https://s3-us-west-2.amazonaws.com/allennlp/models/elmo/2x4096_512_2048cnn_2xhighway/elmo_2x4096_512_2048cnn_2xhighway_weights.hdf5
wget -P elmo_model https://s3-us-west-2.amazonaws.com/allennlp/models/elmo/2x4096_512_2048cnn_2xhighway/elmo_2x4096_512_2048cnn_2xhighway_options.json

In order for the model to fit in memory you may want to reduce max_characters_per_token to 25 in the options file:

sed 's/"max_characters_per_token": [0-9][0-9]*/"max_characters_per_token": 25/' elmo_model/elmo_2x4096_512_2048cnn_2xhighway_options.json

Run the D&M+ELMo parser

Code for running the D&M+ELMo parser that we use in our paper can be found in the elmo-parser-stable branch of this repo:

git checkout elmo-parser-stable

If you haven't already, download the pre-trained ELMo model.

Pre-trained parser models are available via Google Drive. Here are the corresponding ids if you wish to download via command line:

To download and evaluate e.g. the dm-conll05-elmo parser, you can do the following:

git checkout elmo-parser-stable
mkdir -p models
gdrive_download 1uVJna6ddiJCWelU384ssJrnBJGWlPUML models/dm-conll05-elmo.tar.gz
tar xzvf models/dm-conll05-elmo.tar.gz -C models
python network.py --load --test --test_eval --config_file models/dm-conll05-elmo/config.cfg 

Evaluate LISA using the D&M parse

Once you've run the D&M+ELMo parser, you may want to provide that parse to LISA. You can do this by: (1) creating a new test file that contains the D&M predicted parses rather than gold parses, then (2) providing this new test file to LISA and passing a command line flag which instructs LISA to predict using the provided parse information (rather than its own predicted parses, which is the default behavior).

To create a new test file, first run the parser with the test file for which you would like to predict parses. If you ran the parser under models/dm-conll05-elmo, then there should be a new file generated: models/dm-conll05-elmo/parse_preds.tsv. Below we assume the original test file resides under the directory defined in the environment variable $CONLL05. To create a new test file with parse heads and labels replaced with the predicted ones, you can use the following script:

./bin/replace_gold_parse.sh $CONLL05/test.wsj.gz.parse.sdeps.combined.bio models/dm-conll05-elmo/parse_preds.tsv > test.wsj.gz.parse.sdeps.combined.bio.predicted

Now, you can run LISA evaluation using this test file instead of the original one, and tell LISA to use gold parses:

python network.py --load --test --test_eval --config_file models/lisa-conll05/config.cfg --test_file test.wsj.gz.parse.sdeps.combined.bio.predicted --gold_attn_at_train False 

Note that when you change the command line params (as above), config.cfg will be rewritten with the latest configuration settings. You can change this by specifying a directory to --save_dir which differs from --load_dir. (For simplicity, in all the above examples they are the same.)

Evaluate LISA using the gold parse

Similarly, you can easily evaluate LISA using the original, gold parses. To do so on the dev set only, omit the --test_eval flag:

python network.py --load --test --gold_attn_at_train False --config_file models/lisa-conll05/config.cfg

Train a model:

We highly recommend using a GPU.

To train a model with save directory model using the configuration file lisa-conll05.conf:

python network.py --config_file config/lisa-conll05.conf --save_dir model

Results

CoNLL-2005 results for released models w/ predicted predicates (dev, WSJ test, Brown test):

CoNLL-2005 results for released models w/ gold predicates (dev, WSJ test, Brown test):

CoNLL-2012 results for released models (dev, test):

CoNLL-2005 dependency parsing results for released models (dev (section 24), WSJ test, Brown test):

CoNLL-2012 dependency parsing results for released models (dev, test):