; Text Retrieval for Video Search
Learning Center
Plans & pricing Sign in
Sign Out
Your Federal Quarterly Tax Payments are due April 15th Get Help Now >>

Text Retrieval for Video Search


  • pg 1
									crowd a quarter Sony she nighttime Isn't it become much about democracy in the wallets and minus teeth of people and I guess excitement tough said democrat CN and half of them makes sense There is as a junior had the explosive pine crest prep school in Fort Lauderdale his journey may have been inspired by an assignment in his english class the textbook that noon new journalism encourages writers to the stories they cover his classmates can get over his improbable adventure One can just decided to go Lakes Erie and and actually see friends up here rack the election ellen but even think and that's I think people are more shocked than anything out Harris's mother says her son is intelligent not only when compared to the kids at the school here but really when compared to kids across the country she says he recently took the SAT test and scored in the top one percent of kids in the country intelligent but maybe not so smart for water There was a teen ager made his presence known in Baghdad he was picked up by the one hundred first airborne it's taken back to the embassy and while all might be an adventure perhaps on year rise of rival disruptive series business for US soldiers and officials NBC's Richard Angle has more on that aspect from Baghdad US And missy and military officials here have not been amused by this trip saying it diverted valuable resources and describing how some as ninety Sixteen year old Paris costs on made it this far the Palestine hotel downtown Baghdad home to many western reporters itself attack two months ago Costs on checked into room ten twenty six and stayed two nights before walking down the hall to the Bureau Of The Associated Press little green men had walked into my office eleven less surprised them ferris walking in says he left the hotel to go to a nearby restaurant but not speaking any arabic he had trouble for it moved route when he used a phrase book said he wanted to be an aid worker but most aid agencies have pulled foreign staff out of a rock And going in alone was deadly for american mall a receipt got killed by a car bomb last april on the airport road while trying to help a Rocky war victims received compensation AP reporters told House on how much danger he'd put himself in Once he was in Baghdad he understood more clearly that if you walk down a certain street it was quite possible that you is going to be grab by hooded men and put in the back of the truck The journalists then alerted the US Embassy US officials stressed that the rock is not yet ready for tourists for revenge worst and hope that others will not follow hot funds example richer engel NBC News Baghdad In the Middle East a border crossing between Egypt and the Gaza Strip had been reopened it was closed after about a hundred palestinian police stormed the crossing and set up an arms blockade they were reportedly angry about the death of a comedy One at a Northern California where the nonstop rain has residents dealing with flooding and mudslides in the downpour is expected to continue into the weekend and if NBC jennifer London is live in Guerneville California didn't break the all that money and water behind jail Well good morning we got a lot of mud we've got a lot of rain it's been a steady rain all morning and let me give you a little background first guerrillas located about fifty nine miles north several cisco this small town has a population of less or just over one thousand and it sits on the banks of the russian Red River which flows into the Pacific now this river has a rich history of flight into the people here and a live along the river came very close attention to the weather today and the concern is if we see the continued heavy rain Like we've been seen all morning like we're seeing now the russian river could fly in fact when we got here last night this area where i'm standing was covered in water you can see that the water has receded but we've got a lot of money here and if you look out over the river hat is not the case things are pretty dry across Texas right now and they will remain at way where we are instead getting all the wet weather out deeply in the northwest ersey police officers are preparing to pay their final goodbyes after christmas tragedy officers Robert Plan and John Carlson were both killed after they accidentally drove off and open vertical lift drawbridge christmas night this morning hundreds of fellow police officers and citizens came out to honor Carson in the funeral procession last night police divers finally recover the body as partner for the Hackensack River Both officers are being awarded the police department's highest accolade the medal of honor the Vatican has announced that pope benedict the sixteenth will visit one of the most notorious nazi death camps in the upcoming year german born pope will toward the Outfits concentration camp during his schedule affected Poland today some one point five million people most of them jews died in alsace under nazi rule the pope has also been invited to visit the birthplace of pope John Paul sets Some artwork in Salem Massachusetts is trying a lot of stares and a couple of complaint if the statue of the naked woman and the sculptor Terry back in place turned on at his home for all fifty eight People look at go oh my gosh it's a naked sculpture pop this is terrible but you know you walk into the church to see naked sculptures did that terrible now The latter day hearing that then it added A naked lady is really money I hindu goddess of allusion and he says that the public artists he wants to make the World Bank despite a few complaints Miami has not stirred up too much controversy coming up your nemeth NBC the year in terror is the battleground shifting and where is some of the lot a look at care in two thousand five and terrorism expert Steve Emerson And the new year means it changes for people on medicare and medicaid try to cut the confusion that's next Hello my name is on best R. and amenable to Baghdad Aaron I was rendered and you run on the dating from what's Oklahoma and they have been Holland and the guy in Tonight it's the it or all of it Any had to call so we have runny nose a tahoe over and you have a really big cold uni Alka Seltzer plus liquid tells the big cold medicine With multiple cold fighting ingredients to stop the big cold symptoms you feel all over and how to sell surplus liquid gels Yeah I read in their minimum March 2007 22 monthly payment on credit card balances be doubling if you're a homeowner budget cleveland didn't well you could use some of your home equity to pay off your high and high tech and lovely total monday evening just color log on Di and I can't checkout dyke takes forty nine dollars blackfeet home equity loan you and I took one hundred percent of your home's value i'm not indict and I can't recall one eight hundred seventy one It Let us like just the families and I went to see injection is my own age she went back into black I think it's not England After recommended lakes an edge but no one more strides as for help here parents you know Nevertheless left the band of men and nelson land Yeah I was the great picture on it I really would it mean anything Now get a screen picture on three just in time for the holidays the best gift givers give sprint the bill would sprint or nextel stowers sprint the subpoena The americans and bring them will be certain that our among the top movie things and looting you better only on Starr Green who like the aviator and what they get that now it did and that all right here and some zoo with a version of the blues after he did it any better than that porter now and start enjoying the ultimate movie in your own living room but they're all or what you get Are you get a little break channel over three hundred fifty different movies alone there not all being held on monday when I do behind the scenes that had to make a star that will be in it laura border guard today did that is that would not come call one eight eight eight eight one thirty that won eighty eight in 1 one thirty or simply pick your remote and run like now to orders Farmers Terrorist attacks tore through cities in Europe and the Middle East this year but not the United States the fourth year in a row we have not enhanced but the question of why is still open as evidenced by the debate over the effectiveness of the way the Bush Administration is running the war on terror and as NBC analyst Steve Emerson is a terrorism expert he joins me now Steve thinks are being with

The Digital Universe

Text Retrieval for Video Search
Bouke Huurnink
Intelligent Systems Lab Amsterdam
University of Amsterdam

• Digital video has reached critical mass

• How do we search through all this video?
Use surrounding text? Analyse content....Multimodal Search


Multimodal Search Ingredients
Audio (sound)

Kinds of Multimodal Search
Intensive user interaction

• Interactive
Video (images) Motion (spatio-temporal)


Higher Level Features
Automatic Speech Recognition
today sniper fire disrupted the funeral of an eleven year old ethnic albanian boy he was killed yesterday while while chopping wood his family blames serb police before his death louisiana state police now say six workers were killed after a natural gas well exploded and caught fire about forty five miles east of shreveport four others were injured in yesterday's blast a police spokesman says the derek started to melt in the intense

• Automatic
Minimal user interaction

Detected Concepts



Interactive Multimodal Search
For example – Finding shots of George W. Bush entering or leaving a vehicle
Query User System

Automatic Multimodal Search
Finding shots of George W. Bush entering or leaving a vehicle
Query User
“Find shots of George Bush entering or leaving a vehicle”




“George Bush”

Feedback Results Results



Recognising Speech

• Objects can occur in different shots in different
Saddam Hussein

The Temporal Mismatch Problem


Saddam Hussein ✔

as specific as possible because here this is I much a court of public opinion in those who were watching but inside it is a legal proceeding in everything must be proven himself as often as possible the judge is trying to get names of those that were doing the torturing also trying to alleviate many in those stocks the defendants directly to the crimes being alleged by the witness is now one of the witnesses did point Finger bars on this on


• How can we retrieve the first shot if we are only
searching on speech?
7 8

Taking Advantage of Video Structure
Saddam Hussein Trial Price of Jellybeans Ad

• One way of combining is document expansion

Combining Shots and Stories
Use extra information to create a pseudo-document, dʹ, with adjusted word counts

• Video is made up of shots • Shots combine to make stories
But we still want to retrieve shots

shot = uninterrupted video sequence

count(w,dʹ) = α∙count(w,s) + (1-α)∙count(w,S)
w = the current word s = shot, S = surrounding story α = weighting constant

More natural unit for speech-based retrieval

• Can we improve performance by combining shot and
story speech?

• Let’s try this on real data...

Testing Shot-Story Combination

Shot-Story Combination Results
count(w,dʹ) = α∙count(w,s) + (1-α)∙count(w,S)
0.08 0.07 0.06 0.05
dirichlet, mu=0 dirichlet, mu=1000 dirichlet, mu=2000 dirichlet, mu=3000 dirichlet, mu=4000 dirichlet, mu=5000 jm, lambda=0.8 jm, lambda=0 jm, lambda=0.2 jm, lambda=0.4 jm, lambda=0.6 jm, lambda=0.8 jm, lambda=1

• TRECVID data sets 2003 - 2006
300 hours of news video 190 000 shots, 14 000 stories
the Tomb of the Unknown Soldier at Arlington National Cemetery
Condoleeza Rice


road with one or more cars

0.04 0.03 0.02 0.01 0.00
0.0 0.1 0.1 0.2 0.2 0.3 0.3 0.4 0.4 0.5 0.5 0.6 0.6 0.7 0.7 0.8 0.8 0.9 0.9 1.01.0


94 natural language topics with ground truth Standard evaluation measure is MAP

Shot MAP: 0.0448 Combined MAP: 0.0742 Increase: 65.6%


MAP Change Per Topic
Topic analysis at optimal shot weight = 0.9 Measure change in MAP between baseline and combined
Find shots of the Spinx

Where are we now?

• We looked at speech-based video retrieval • Story information important for shot retrieval • Weighting fairly constant over smoothing methods
65.6% improvement!

Find shots of the Tomb of the Unknown Soldier at Arlington National Cemetery
13 14

• Test combination under different retrieval models • Document expansion • Combine speech with other features
Concept detectors Low-level features Optical character recognition Other metadata

More Speech Opportunities


Why are we doing it?
The largest audiovisual archive of the Netherlands


Add terms to shots from external corpus Correct for out-of-vocabulary and badly recognised terms

• •

Approximately 700.000 hours Yearly increase of approximately 20.000 hours

The ‘business archive’ of the Dutch public broadcasters Make cultural heritage data accessible for general public

The End



To top