“SRL in Practice” Assignment
CSCI 4460, Spring 2014
Due: April 1, 2014 electronically via email to ylierler@unomaha.edu
Assignment Overview
Each student in class is assigned 20 sentences (see later in the file the list of sentences assigned to you).
You are asked to run Semantic Role Labeler LTH (http://nlp.cs.lth.se/) and Illinois Semantic Role Labeler ISRL
(http://cogcomp.cs.illinois.edu/page/software_view/SRL) on each one of these sentences. In 2008 CoNLL
shared task, LTH obtained the best results. ISRL was among best performers at CoNLL-2005.
To run LTH use online demo posted at http://barbar.cs.lth.se:8081/ .
To run ISRL use online demo posted at http://cogcomp.cs.illinois.edu/demo/srl/ .
For each sentence
1. obtain the output of LTH as well as ISRL
2. briefly describe how (I) LTH interpreted the sentence and (II) ISRL interpreted the sentence, in case when the
interpretation of ISRL differ, and (III) how would you annotate the sentence
3. mark LTH as well as ISRL annotations as right or wrong for the sentence
4. describe what is “wrong”
Once you analyze all the sentences, compute precision, recall, and F1 score of LTH and ISRL
Find the definitions of precision/recall/F1 here: http://en.wikipedia.org/wiki/F1_score
Elaborate on the following questions:
Q1. How does your number for LTH compare with the number reported at CoNLL-2008 for this system?
Q2. How do LTH and ISRL compare and differ in their assignments?
Q3. Can one account on these similarities differences based on the technological ideas implemented in the
systems?
Note especially interesting sentences and corresponding analysis that you observed. Anything that caught your
eye. On April 2 we will compare the findings across the students. Each student will share the summary of his
findings with other class members. This way we will carry out our in house evaluation of modern SRL
technology.
Jason Gaines:
He painted the house on the hill.
He painted the house on paper.
He painted the house on Friday.
I watch clowns on bikes.
I watch clowns on Friday.
I read a book on physics.
I read a book on the plane.
I ate a pickle in the library.
I ate a pickle in a hamburger.
I see a boat with binoculars.
I see a boat with an engine.
I read a book about turtles.
I read a book about noon.
I read a brochure for Paris.
I read a brochure for fun.
I gave away my key to the car.
I gave away my key to Mary.
I lived in a house with a porch.
I lived in a house with my brother.
I read a book by the fire.
Benjamin Greer:
I read a book by Pratchett.
I brush my teeth twice a day.
The show is about to start.
We had a few beers after the game.
I planted a row of lillies around the statue.
She went around the track fifty times.
I left the house around 10 this morning.
The pages from the notebook were scattered around the room.*
You are not as tall as me.
The movie features Al Gore as a streetwise pimp.
Meet at Jim's house.
He threw the ball at me.
It is growing at the rate of 3% a year.
The mailbox is by the bus stop.
He fell asleep during the meeting.
I want to go swimming during the weekend.
I was bored during the whole film.
They ran down the hill.
The post office is down the road.
They have had many wars down the years.
Brian Hiserote:
We went swimming despite the cold water.
He passed the test despite being ill.
Monkeys live in the jungle.
John is the man with his hand in his pocket.
I live in an apartment.
She lives in Bangkok.
Tara was born in 1977.
Trains were invented in the nineteenth century.
I'll come back in two weeks.
Let's meet in the morning.
There are 60 seconds in a minute.
The price is $70 including $10 for delivery.
There were four of us, including the baby.
John went into that shop.
If you heat ice it turns into water.
We cut the cake into ten pieces.
Five into ten makes two.
It was dark inside the tunnel.
My modem is inside my computer.
Do you live with your parents?
Ajay Medury:
He's been with EnglishClub.com for two years.
I discussed it with her.
With your permission, I'd like to go.
If you mix red with yellow you get orange.
Muriel is the girl with black hair.
Do you want to come with us?
I made this chair with my own hands.
You'll forget her with time.
There is a modem within the computer.
I will finish within 30 minutes.
He lives within 10 miles of his work.
It's not within my power to help you.
He is at school.
We first met at a party.
Let's start the meeting at 9 o'clock.
He started work at 17 (years of age).
We are aiming at sales of $1,000,000.
I will see you on Monday.
The week begins on Sunday.
My plane leaves at noon.
John Oerter:
The movie starts at 6 p.m.
He likes to read in the afternoon.
The days are long in August.
The book was published in 1999.
The flowers will bloom in spring.
She has been gone since yesterday.
I'm going to Paris for two weeks.
The movie showed from August to October.
The decorations were up from spring until fall.
I watch TV during the evening.
We must finish the project within a year.
There is a wasp in the room.
Put the present inside the box.
I left your keys on the table.
She was waiting at the corner.
He threw the ball over the roof.
Hang that picture above the couch.
The rabbit burrowed under the ground.
The child hid underneath the blanket.
We relaxed in the shade beneath the branches.
Steven Schoonover:
The valley is below sea-level.
She lives near the school.
There is an ice cream shop by the store.
An oak tree grows next to my house
The house is between Elm Street and Maple Street.
I found my pen lying among the books.
The bathroom is opposite that room.
She glanced at her reflection.
She glanced in the mirror.
You didn't laugh at his joke.
I'm looking at the computer monitor.
We rejoiced at his safe rescue.
That pretty girl smiled at you.
Stop staring at me.
I don't approve of his speech.
My contribution to the article consists of many pages.
He came home smelling of alcohol.
I dream of finishing college in four years.
Can you think of a number between one and ten?
I am thinking about this problem.
William Switzer:
Did someone call for a taxi?
He hopes for a raise in salary next year.
I'm looking for my keys.
We'll wait for her here.
You go buy the tickets and I'll watch for the train.
If you wish for an "A" in this class, you must work hard.
The sandwich [on the counter] looks delicious.
The plane flew [over the Atlantic Ocean].
The man read [with the light on].
The present [under the Christmas tree] was large.
The clock stopped working [over an hour ago].
The man wanted to build a bridge [over the river].
The teacher lives in a brand new apartment.
The baseball catcher moved his hand inside his glove, and the pitcher saw the signal easily.
They went to the Aquarium to see the sea otters.
The capital of Saskatchewan is Regina.
The people in the park are walking.
The loveliest area to live in Vancouver is near Stanley Park.
Rain is frequent in Vancouver.
My friend John wore a hat with a button of the Earth saying "home."