Topic

Implementation to an improved

CoBoost algorithm for NEC





CMPT 825

Project Presentation

Yudong Liu

12/9/2003

Preface

 CoBoost is proposed by (Collins and Singer, 1999)

as an unsupervised frame for Name Entity

Classification.

 It is based on AdaBoost algorithm

 Classifying names with over 91% accuracy on

10000 rounds.

 My work is implementing an improved CoBoost

algorithm, which is motivated by a feature selection

strategy for BoostLoss in Parsing Reranking task

(Collins, 2000), and applying it to the original NER

task. It turns out the efficiency can be improved

much.

Agenda

 Problem description



 Algorithm description

 AdaBoost algorithm

 CoBoost algorithm

 More Observations

 Improved CoBoost algorithm



 Experiment Result



 Conclusion

Problem Description

 The original task that CoBoost applied to is to learn a function

from an input string to its type, which is assumed to be one of

the categories: Person, Organization, or Location.

 Given 89305 examples, the only supervision is in the form of 7

seed rules (namely, the New York, California and U.S. are

locations; that any name containing Mr. is a person; that any

name containing Incorporated is an organization; and I.B.M

and Microsoft are organizations). So we get 8932 labelled and

80370 unlabeled examples.

 An example: …, Mr. Doernberg, a professor of …

 Both a spelling feature( that the string contains Mr. ) and a

contextual feature (that professor modifies the string) are strong

indications that Mr. Doernberg is a type of Person.

Problem Description(cont.d)

 Actually unlabeled data can give many such

indications that two features should predicate the

same label.

 CoBoosting just takes advantage of such redundancy

in the unlabeled data to build two classifiers in

parallel.

 Based on this, every example in the data set is

represented by a pair of (spelling, context). And

either the spelling or context alone is sufficient to

classify an example.

Problem Description(cont.d)

 Features: Full-string=x, contains(x), allcap1, allcap2,

nonalpha=x , context=x.

 Example 44123:

X0_Ronald-Berke X01_president X2_Ronald X2_Berke X3_LEFT

 Classifier 1 X0_Ronald-Berke X2_Ronald X2_Berke

 Classifier 2 X01_president X3_LEFT

 So for CoBoost, the learning task is to find two

classifiers f1 and f2 such that their predictions on

labelled data should be the same and predictions on

unlabeled data should be the same as often as

possible.

Algorithm Description



 AdaBoost

 CoBoost Algorithm

 More Observations

 Improved CoBoost Algorithm

AdaBoost--Base of CoBoost

 Proposed by (Freuned and Schapire, 1997)

 Aim to produce a more accurate prediction rule by

iteratively calling a weak supervised learner

 Main Idea:

 maintain a distribution or set of weights over the training

set

 Initially, the weights are set equally, but on each round, the

weights of incorrectly classified examples are increased (and

accordingly, the weights of correctly classified examples are

decreased) .That means the weight tends to concentrates

on “hard” examples.

AdaBoost- Base of CoBoost

 The weak learner‟s job is to find a weak hypothesis ht

appropriate for the distribution on instances. The

goodness of a weak hypothesis is measured by its

error on classification.

 Once the weak hypothesis ht has been received, a

parameter t, which intuitively measures the

importance is assigned to ht.

 Then the distribution is next updated, which effort is

to increase the weight of examples misclassified by ht,

and to decrease the weight of correctly classified

examples.

 Finally output

CoBoost Algorithm (1)

 New target function----

“error bound”



 Decompose:







 Define:

CoBoost Algorithm(2)







 Some other definitions:

 Pseudo-label :









 For a hypothesis:

CoBoost Algorithm(3)





 According to the derivation of (Schapire and Singer,

1999) with W+ > W-, equation above is minimized by

setting(  is for smoothing):



Accordingly, Z co  W0  2 WW  Z  ( W  W )

1 2









where

More Observations(1)

 Those instances with their pseudo-labels being zero

will not contribute to target function:







 So in this case the weak learner is free to pick labels

for these instances, which could speed up the

boosting process.

More Observations(2)

 Suppose on round t ,feature k* is selected

 Then in round t+1:

---if AdaBoost for supervised learning, the weights

of those instances on which k* abstains will not be

updated.

Dt 1 (i)  Dt (i) exp(  t yi ht ( xi ))

---but in CoBoost, we have to check pseudo-label for

each instance even in case of k* abstains

Dt j (i )  exp(  ~i g tj 2  ~i t 1ht j 1 )

y y 

More Observation(3)

 In each round W+,W- have to recalculated for each

feature. This computation involves about 70000 features

and 90000 instances. It is expensive!

 Actually only need consider a small number of features

---the features co-occurring with k*

---the features whose instances have their pseudo-labels changed

eg: X2_Mr. 3 56 450 ………

 Due to the sparseness of the feature space, the number

of features that are taken into account will be much

smaller (about 3 to 10000) than the number of features

in feature space, which is about 70000.

Improved CoBoost Algorithm(1)

 Define some new functions:

1. Sign of k

for any feature k:

if feature occurring in labelled data

sign(k)=1 if more instances that containing k are

labelled 1

sign(k)=-1 otherwise

if feature only occurring in unlabeled data

sign(k)=-1 if instances contains k

sign(k)=1 otherwise

Improved CoBoost Algorithm(2)

2. A,k A, B, Bi

k i

 Ak ={x: x contains k and k has the same prediction

with the pseudo-label of x}

 Ak ={x: x contains k and k has the opposite

prediction with the pseudo-label of x}

 Bi ={k: k has the same prediction on instance i with

pseudo-label of i};

 Bi ={k: k has the opposite prediction on instance i

with pseudo-label of i}.

3. wt for each instance i

 weight(xi)=exp(-1 *wt(i))

Improved Coboost Algorithm(3)

 For t=1 and j=1,2

--set pseudo-labels

--find the best hypothesis with maximum W   W 

--freely pick labels for instances with pseudo-label 0

--update g

 For t=2 and j=1,2



--set pseudo-labels

--update those instance weights with their pseudo-

label changed

--Init Ak Ak Bi Bi

--find best hypothesis with naïve way

--update g

Improved CoBoost Algorithm(4)

 For t=3 to T and j=1,2

--set pseudo-labels

--update Ak Ak Bi Bi

--for instances in Ak Ak

.calculate delta of weights

.update W+,W- for the according features in Bi and Bi

--for instances which have their weights changed and

not in Ak Ak take the same process as above

--for those features which have their W+ and W-

changed , reconsider for minimizing Zco

--update g



2

 Final hypothesis: f ( x)  sign( j 1 g j ( x j ))

T

Multi-Class Problem



 Three candidate labels:loc,org and per

 Do CoBoost over 3 labels in turn --an innermost loop

 For the final label of a particular instance, if there is

some disagreement occurring, e.g. the signs of

“Person” and “Location” are positive at the same

time, we will pick the one with a larger

“confidence”(abs(g)) as the final label.

Experimental Evaluation

 Training data set(5.4 M): 89305 examples

8932 labelled examples

 Testing data set: 431 examples randomly picked from

training data set

 Features involved: 56649 for classifier 1

11824 for classifier 2

 In pre-procesing, I produce a hash table with each

key being each feature and the corresponding value

the list of instances that contain the feature in their

representation. Such hash tables are kept for two

classifiers respectively.

Experimental Evaluation(Cont.d)

 The CoBoost algorithm was run for 10000 rounds on

the training data. I record g1T, g 2, g1T  g 2 and final

T T





label (1 or –1) for each instance on the 3 labels and

get output file at round 10,100,1000 and 10000.

 = 1/2n, n=89305

 The computation took roughly 20 hours on an Athlon

1.2GHz server (shared CPU)

Final Result

 On round 10 ,

 the accuracy, the coverage and the agreement are

34%,95% and 5%

 „Org‟ prediction:coverage 90.2% ,agreement 89.6%

 „Location‟ prediction:coverage 91.5%, agreement 52.5%

 „Person‟ prediction:coverage 93.1% agreement 42.1%

 On round 100,round 1000 and round 10000, these

results keep unchanged.

 The result is far different from the original result

Conclusion and future work

 An improved CoBoost algorithm is proposed which is

based on the original CoBoost algorithm and a feature

selection strategy in parsing reranking problem.

 Result is frustrating compared with the original one.

 buggy implementation

 Improper presumption

 Incomplete testing data

 Based on the redundancy of features, CoBoost gives an

impressive result in NER task. Its idea suggests us

explore other issues with the similar property,

especially in the semi-supervised and unsupervised

learning task.

References

1. Michael Collins 2000 discriminative Reranking for natural

language parsing in international conference on machine

learning

2. Steven Abney ,Robert E. Schapire & Yoram Singer Boosting

Applied to Tagging and PP Attachment , In Proc. of the

Joint SIGDAT Conference on Empirical Methods in Natural

Language Processing and Very Large Corpora

3. Yoav Freund & Robert E. Schapire A short Introduction to

Boosting In Journal of Janpanese society for Artificial

Intelligence, 14(5): 771-780. September,1999

4. Robert. E. Schapire & Yoram Singer Improved Boosting

Algorithm using Confidence-rated Predictions ,Machine

Learning 37(3):297-336,1999