Tan Jamal Correspondence

Document Sample
Tan Jamal Correspondence Powered By Docstoc
					           Editor’s initial letter responding to initial submission of Tan and Jamal

Professor Hun-Tong Tan
Nanyang Business School
Room S3-01B-51
Nanyang Technological University
Nanyang Avenue
Singapore 639798
Republic of Singapore

MS #99-0226

Dear Hun-Tong:

I have enclosed a thoughtful review of your manuscript APerceived Subordinate Competence and
Reviewer Evaluation,@ Ms. 99-0226, and a marked-up copy of the paper from the reviewer. I
also asked Consulting Editor Michael Bamber to evaluate the paper, given his expertise in this
area, and his letter is also enclosed.

I agree with the reviewer and the Consulting Editor that whether managers= assessments of the
quality of work performed by their senior-subordinates depends on managers= prior impressions
of those subordinates (and whether higher-rated managers are more immune from any such
propensity) is an interesting and important question. Using real-world senior-manager teams to
address this question is an important and novel strength of the study. We appreciate the
difficulty of obtaining such real-world subject pairings, and we understand the importance of this
strength.

Notwithstanding these strong points, the paper is subject to some rather serious limitations. The
review and Consulting Editor=s letter are quite clear so there is no need for me to repeat the
details here. However, I do agree with the Consulting Editor that two major concerns must be
carefully addressed if the paper is to provide credible evidence on the research question. First, as
the Consulting Editor points out, it is important to pin down the features of the subordinate that
appear to influence the managers. What exactly is driving the results? What is the precise nature
of the difference between the two groups of seniors (and managers) that the study captures?

The second issue is the reliability of the study=s inferences. I must admit that even for a
behavioral study using real-world senior-manager pairs, I have a hard time justifying publication
of a study that is based on a sample of only 14 managers. My first reaction would be to do your
best to obtain more subject-pairs. If this is not possible, the reason should be explained, and the
limitation acknowledged forthrightly. In this case, the reviewer=s requests for power analyses
and robustness tests become even more critical. I am particularly concerned by the inconsistent
and selective choice of reported statistical results. With a sample size of only 14 managers,
consistent and complete disclosure is imperative if readers are to have any confidence at all in the
results.
                                                                                                  1
If you obtain additional subject-pairs, it is possible that the inferences may change. One
attractive feature of your study is that in principle it should be possible to frame the results in an
interesting way, however they turn out. That is, if upon further analysis managers appear not to
be significantly affected by their prior impressions of senior subordinates, it would seem in
principle feasible to make the case that this outcome is of interest to the journal=s readers
(provided that you can demonstrate that the tests have sufficient power to detect a material effect
if one exists).1

The review and the Consulting Editor=s letter express a number of other credible concerns that
warrant careful consideration in any further work in this area.

I share the reviewer=s and Consulting Editor=s general enthusiasm for the study because it is
quite creative in using real-world senior-manager pairs to investigate a topical issue. Thus, I have
decided to accept the Consulting Editor=s recommendation to encourage you to revise and
resubmit your paper to The Accounting Review if you can develop a revision that carefully and
thoroughly addresses the concerns raised in the review and in the Consulting Editor=s letter.
Obtaining additional data would in my opinion materially enhance the probability of ultimate
success, although at this point I am not requiring this as a necessary condition for resubmission.
Please also bear in mind that whenever there are such substantial concerns about the reliability of
a study=s inferences B and indeed about what is actually driving the study=s results B any
revision would be subject to outcome risk. If you are willing and able to address the concerns
expressed in the enclosed feedback, I would encourage you to revise the paper for resubmission
here, particularly if you are able to obtain more subject-pairs.

After you have had a chance to consider the enclosed review and Consulting Editor=s letter, I
would appreciate it if you would send me an email to let us know whether you plan to take
advantage of the opportunity to revise the paper for resubmission here, and if so, approximately
when we might expect to receive the revision.

Thank you for considering The Accounting Review as an outlet for your research. I sincerely
hope that you are able to find a creative path to a responsive revision.

Sincerely,

                                        Linda Smith Bamber


       1
         Should the primary results turn out to lack statistical significance, it is even more
important to demonstrate that the tests were sufficiently powerful to detect a difference, had one
existed. See Greenwald (Psychological Bulletin, January 1975) for suggestions on how to
gracefully fail to reject the null hypothesis. See also Cready and Mynatt (1991, TAR) in an
accounting context.

                                                                                                     2
                   Consulting Editor’s letter responding to initial submission


January 28, 2000


Professor Hun-Tong Tan
Nanyang Business School
Room S3-01B-51
Nanyang Technological University
Nanyang Avenue
Singapore 639798
Republic of Singapore

MS #99-0226

Dear Hun-Tong:

The editor asked that I take responsibility for your paper, APerceived Subordinate Competence
and Reviewer Evaluation,@ MS #99-0226. I have enclosed the sole referee=s report. The
referee=s report is insightful and clear so I will not repeat the details here. While the referee
raises a number of concerns with the paper, the referee also provides constructive suggestions for
dealing with most of these concerns. In addition to satisfactorily addressing the other issues
raised in the referee=s report, my own reading of the paper suggests that your ability to address
two fundamental issues with the paper will ultimately determine the publication outcome.

The first issue involves providing closure on what exactly we learn from the study: what feature
of the senior appears to influence managers? The paper=s title refers to Aperceived subordinate
competence,@ but the paper refers to the managers= Ageneral impression@ of the seniors, while
seniors are described as Atop@ and Aoutstanding.@ The debriefing questionnaire elicits
information only on managers= familiarity with seniors. While you may argue that finding that
some attribute affects managers= evaluations of seniors= memos is an important finding,
irrespective of the actual seniors= attribute influencing managers= evaluations, the implications
are a function of the underlying attribute(s).

The description of how subjects were selected for the study is not particularly helpful in
understanding what is driving the study=s results. The criteria provided the staff partner is that
for each manager, one senior should be outstanding and the other mediocre Aas per the firm=s
performance evaluation system.@ [As an aside, I find it hard to believe that an international
accounting firm would continue to employ mediocre seniors and managers. What especially
calls into question the meaning of Amediocre@ is that the training partner was able to identify 14
mediocre seniors who worked with the 14 selected managers.] Did the terms Aoutstanding@ and
Amediocre@ come from the firm=s performance evaluation scale? Footnote one simply says that
auditors from the upper and lower percentiles of the performance evaluation ratings were
                                                                                                    3
selected. Do you have assurance that the training manager successfully followed this procedure?
 Do you know that auditors at the lower percentile are still not evaluated, for example, as good as
distinct from mediocre? In sum, we need greater assurance about what difference between the
two groups of seniors (and managers) is being captured and how well is it being captured.

Ideally, and as recognized by the reviewer, we should also know the managers= own evaluations
of the seniors. At present, we do not know whether the manager is influenced by his or her
knowledge of seniors= formal firm evaluations, direct knowledge of the seniors= abilities, a
general impression of the seniors, or some other factor. An alternative explanation is that the
managers= are influenced by the seniors= experience. Mediocre managers have significantly less
experience than top mangers. The paper does not report the statistics for seniors; it needs to
provide this information. A similar relation for seniors might suggest that inexperienced
managers tend to rely more on experienced seniors. The implications of this interpretation are
less than those discussed in the paper.

The second fundamental issue is the robustness of the paper=s results. In private
correspondence, the referee says that Athe study suffers from a very small sample size, which
raises concerns about the meaningfulness of the findings.@ The referee suggests a variety of
steps to help convince the reader of the study=s results. Each of the referee=s suggestions has
merit, although the most straightforward way to address this issue would be to collect more data.
 Fourteen independent data points make an unusually small data set for The Accounting Review.
Further, in considering the referee=s suggestions, I recommend more rather than less disclosure.
A related issue is that the selective disclosure of statistics in the paper does not increase the
reader=s confidence in the results. Examples of selective disclosure include reporting mean
experience levels but not familiarity ratings in footnote three, and reporting medians (with t-
tests!) for some partner ratings and means for others. Moreover, no statistical test is provided for
the claim that Athere is no difference in the quality of memos evaluated by top and mediocre
managers@ (p. 17).

Additional concerns include the following:

$      Be careful how you motivate the paper and place it in the context of prior research. At
       times the paper gets very close to criticizing (unnecessarily) prior research. For example,
       on p. 7 we are told that much prior research (e.g., Ramsay 1994) does Anot capture a real
       world feature.@ However, if this feature is not particularly relevant to the study=s
       research question, nothing is lost by not capturing it and internal validity considerations
       would in fact suggest not capturing it. Similarly, we are told on p. 8 that Atwo studies
       attempt to investigate.@ Why not simply say Atwo studies investigate.@ We are also
       informed on p. 8 that Asare and McDaniel (1996) do not examine how the reviewers=
       impression of actual preparers influences the review process. This is strictly correct, but
       rather than implying what is wrong or incomplete with their paper, a more desirable
       approach is to identify what that paper did and how their results raise a question which
       you go on to address. That is, the focus should be on how you are extending prior
       research. No paper provides the complete answer. Note that while Asare and McDaniel
                                                                                                   4
       do examine the effect on the review process, this paper does not. That is, this paper
       examines managers= evaluation of memos, not how this evaluation affects the review
       process or subsequent audit work. (See also the referee=s specific comment 2.)

$      Clarify the roles of objectivity and effectiveness in review. While a given
       memo/workpaper should be evaluated objectively, I do not think you mean to imply that a
       reviewer should not modify the review process in response to specific contextual factors,
       including the competence of the preparer. This is not clear in the paper.

$      On p. 6 the example of familiarity with the client has nothing to do with the objectivity of
       the review process.

$      The theoretical discussion is sufficiently clear to question the need for p.10. I do not
       think the hypotheses need this discussion of the paper=s method. Discuss the method in
       the method section.

$      I agree with the referee that the wording of H1 is not clear. I also think the wording of H2
       needs revision. H2 is more clearly stated on p. 16.

$      The description of the seniors= task in writing the memo on p. 14, Ajustifying their
       stand,@ is inconsistent with the instrument=s description.

$      If a revision is resubmitted please provide a copy of the original test instrument. The
       materials provided seem to be only parts of the original instrument. Subjects must have
       been given some instructions. Did the manager=s review of the seniors= memos only
       consist of the two scales provided and were they presented on the same page? There are
       other questions that only an examination of the original instrument can satisfy.

$      How did you prevent seniors and managers discussing the experiment in the three weeks
       between stages?


These concerns make an evaluation of the paper=s incremental contribution difficult to assess. I
agree with the reviewer that you should be given an opportunity to revise the paper for The
Accounting Review. The study provides a creative approach to examining an important problem.
 However, please note that significant outcome risk remains. My concern is whether, without
expanding the sample, what remains after addressing the referee=s concerns and those noted
above will be a sufficient incremental contribution to justify publication in The Accounting
Review.

Sincerely,


E. Michael Bamber
                                                                                                  5
6
                             Referee’s Report on initial submission


                                          Reviewer Comments

             Manuscript No. 990226 “Perceived Subordinate competence and Reviewer
                                      Evaluation”

        The purpose of the study is to investigate whether auditors’ assessment of the quality of
work performed by their subordinates are influenced by their prior impressions of these
subordinates and whether those considered to be outstanding managers are less susceptible to this
effect. This was done by having two sets of managers, outstanding and mediocre, first evaluate
memos written by seniors who were identified, and then evaluate the same memos without the
identification of the senior.

        The authors motivate the study by claiming it is important that reviewers objectively
assess seniors’ work. They state that objectivity of review is important for its effectiveness as a
quality control mechanism, and for effectiveness of the feedback. They also state that lack of
subjectivity subjects the evaluation and promotion process to impression management activities.

       The study is well written, and the experiment is generally well done. I agree that the issue
is important to auditors, and I believe the findings of differences in effects of impressions on
reviews between top managers and mediocre managers (H2) fulfills the definition of a significant
incremental contribution to the literature.

        The strength of the paper is the use of natural senior/manager teams, and this should be
mentioned in the abstract. However, the study suffers from its small sample size (NMANAGER=14),
which makes one wonder if the results are influenced by one or two managers making large
changes between stages one and two. First, the authors should provide power calculations (see
Burgstahler, AR Jan. 1987). Also, I would like to see another row for each panel in Table 1
indicating ratings change between stages. I would also like to see the N and the range of the
observations for all cells, especially the ratings change cells. The small sample size also makes
the statistical analysis less robust to violations of ANOVA requirements. The authors should
perform the analyses with nonparametric tests and report those results as well. Finally, the
authors might consider showing the results of each observation, given the small sample size.



       More specific comments follow:

1. P. 2—An additional motivation may be the effect of a lack of objectivity on effectiveness and
       efficiency of the audit in that managers may over rely or under rely on subordinate’s work
       in determining additional procedures. I think this is different than the quality control
       mechanism mentioned.

2. P. 5—I’m not sure that the study demonstrates “a potential cost” of relying on impressions—
                                                                                                      7
       only that they do it.

3. P. 11—The wording of H1 is not exactly accurate. I can appreciate the authors’ difficulty in
       formulating a precise statement of the hypothesis. Consider “The amount by which
       ratings of a outstanding senior exceed that of a mediocre senior will be greater when the
       seniors’ identities are known…..” (Note: I would stick with the term “outstanding”
       instead of “top” throughout the paper.)

4. P. 12—I wonder why subjects first read memos that identified the seniors and then read the
       without the identification. The authors acknowledge in footnote 8 that the style of the
       memos may have been recognized by the managers. I would like the authors to explain
       why they adopted this sequence rather that the reverse. I do not think, however, that this
       choice significantly effects the results.

5. P. 15—The use of “Netrating” as both the dependent variable and an independent variable in
       the discussion is confusing. It would help if the authors used STAGE as a within subjects
       variable in the discussion.

6. P. 16, 4th line—you should indicate here and in the introduction and conclusion that the main
       effect of Netrating is only marginally significant

7. P. 16—Please provide degrees of freedom with the F scores.

8. P. 17—The comparison of differences between memos from outstanding seniors and
       mediocre seniors is not needed.

9. P. 18—As the authors did not measure managers’ impressions of the seniors, there is little
       justification for the discussion of justification and self-promotion activities. The findings
       may have been due to managers’ knowledge of seniors’ abilities, rather than their overall
       impressions. This discussion should be toned down.

10. P. 19—The small sample size should be acknowledged as a significant limitation of the
        study.

11. Table 1—Label the stages as “Senior identified” and “Senior not identified” or something
       similar.

12. Table 1—Any thoughts about why top managers increased their ratings from stage one to
       stage two while mediocre managers decreased their ratings?




                                                                                                    8
                                    FIRST REVISION
                     Authors’ response to Editor accompanying revision

April 26, 2000

Professor Linda Smith Bamber
Editor, The Accounting Review
J.M. Tull School of Accounting
University of Georgia
Athens, Georgia 30602-6253
USA

Dear Linda

                                        MS 990226
                 Perceived Subordinate Competence and Reviewer Evaluation

Enclosed are three copies of my revised manuscript “Perceived Subordinate Competence and
Reviewer Evaluation” with Karim Jamal, along with memos to the Consulting Editor and the
reviewer describing the changes made.

We have revised the paper based on the comments by the reviewer, the Consulting editor, and
you. These changes are described in detail in the enclosed memos, and I summarize the major
ones here:

1. What are the features of the subordinate that influences the manager? What are the
   differences between the two groups of seniors and managers?

Managers and seniors are classified as ‘outstanding’ or ‘mediocre’ by our contact in the CPA
firms, based on the subjects’ performance evaluations.

In collecting the data from Firm 1, we do not ask the participating managers to rank their paired
seniors. The participating firm is very concerned about the sensitivity and confidentiality of the
performance evaluation ratings, and subjects are not aware of the basis upon which they have
been selected to participate in this study. Asking the managers to rank the seniors may reveal
this basis (i.e., based on their performance ratings). We receive assurance from the firm that the
managers are aware of the performance ratings of their paired seniors since these managers work
frequently with the seniors and evaluate their performance.

In response to your comment, we obtain permission from Firm 2 for us to ask the managers to
rank the seniors and discuss their attributes with us. Specifically, we ask the six Firm 2
managers, after they have evaluated the memos, to identify which of their two seniors is more
outstanding based on their performance on prior engagements. All the managers are able to
successfully identify the more outstanding senior. Most managers also identify technical

                                                                                                 9
competence and the ability to see the big picture as the defining attributes of the outstanding
senior.


2. Increase the sample size

In line with your suggestion, we have collected more data from additional 12 seniors and 6
managers. Although this additional number is not large, we hope that you appreciate the
constraints we face. Our relatively small sample size is a reflection of two conditions imposed
by our research design –auditors in the study must have worked with each other for a period of
time, and these audit managers and seniors have to be ranked high or low based on their firm’s
performance evaluation scores. These criteria impose constraints on the number of subjects
available. Other than the design constraints described above, CPA firms are also very sensitive
to the nature of the data we requested.

The results from the new data we collected are similar to those that we had previously collected,
which gives comfort on the reliability of our results.

3. Consistent and complete disclosure

We have done so in this version of the paper. It is not our intention not to completely disclose
data, and we apologize if we indeed gave that impression. The inconsistencies mentioned by the
consulting editor have been corrected.

We believe that these changes have improved the paper, and thank you for your consideration.

Sincerely




Hun-Tong Tan



Enclosures




                                                                                                  10
                 Authors’ response to Consulting Editor accompanying revision


April 26, 2000

Professor Michael Bamber
Consulting Editor, The Accounting Review
J.M. Tull School of Accounting
University of Georgia
Athens, Georgia 30602-6253
USA

Dear Michael

                                         MS 990226
                  Perceived Subordinate Competence and Reviewer Evaluation

Thank you for your insightful comments. We have found them very helpful in revising the paper.
They key changes are summarized below:

What feature of the senior appears to influence managers? Do we know whether the
manager is influenced by his or her knowledge or seniors’ formal firm evaluations, direct
knowledge of seniors’ abilities, general impression, or experience?

In collecting the data from Firm 1, we do not ask the participating managers to rank their paired
seniors. The participating firm is very concerned about the sensitivity and confidentiality of the
performance evaluation ratings, and subjects are not aware of the basis upon which they have
been selected to participate in this study. Asking the managers to rank the seniors may somehow
reveal this basis of selection (i.e., based on performance ratings). We receive assurance from the
firm that the managers are aware of the performance ratings of their paired seniors since these
managers work frequently with the seniors and evaluate their performance.

In response to your comment, we obtain permission from Firm 2 for us to ask the managers to
rank the seniors and discuss their attributes with us. Specifically, we ask the six Firm 2
managers, after they have evaluated the memos, to identify which of their two seniors is more
outstanding based on their performance on prior engagements. All the managers are able to
successfully identify the more outstanding senior. When asked to explain why they think one
senior is better than the other, the majority indicate technical competence (5 managers) and the
ability to see a big picture (4 managers) as key attributes of the more outstanding senior. Other
attributes mentioned include experience (2 managers), confidence (1 manager), and common
sense (1 manager). 2

2 Experience was mentioned by one outstanding manager and one mediocre manager, and there is no clear pattern
that the seniors’ experience is particularly salient for mediocre managers.
                                                                                                                11
The data from Firm 2 indicates that managers have knowledge of the seniors’ technical
competence and abilities. Few managers mention experience as an attribute that influence their
opinion, and none mention only experience to the exclusion of technical competence or other
abilities. We now discuss this issue in Footnote 4.

In our debriefing, managers also indicate that group level discussions are held by managers to
determine the annual performance evaluations of each senior.

We also now report the experience levels of the outstanding and average auditors on pages 11
and 12. On average, outstanding and average managers have 109 months and 88 months of
experience, respectively (F=1.103, p=.308). Outstanding and average seniors have 43 and 31
months of experience, respectively (t=3.551, p=.001). It is possible that the average managers
are less experienced and rely more heavily on the more experienced seniors. However, our
debriefing with managers from Firm 2 suggests that it is less likely that experience is used as a
cue (or the only cue). We do think that it is entirely plausible that relative to outstanding
managers, the average managers are less able to objectively assess the work of seniors, and rely
on the seniors’ performance evaluation as a cue on the quality of the assessed work. We discuss
this point on page 11 of the manuscript.

Did the term ‘outstanding’ and ‘mediocre’ come from the firm’s performance evaluation
scale?

No, the terms do not directly correspond to those used in the firms, although the firms do have
equivalent terms. For example, the term ‘outstanding’ corresponds to ‘exceeds expectations’ and
‘mediocre’ corresponds to ‘at expectations.’ Given your comment, we now clarify the meaning
of the term ‘mediocre,’ given that it can mean either ‘poor’ or ‘average.’ We describe in
Footnote 1 that the ‘mediocre’ auditors in our sample are deemed to be average, and they are
considered to be making a positive contribution to the CPA firm (i.e., their performance ratings
are still satisfactory), despite their lower performance ranking relative to the others. We also
clarify that at the date of participation in the experiment, none of the auditors has received a
rating of ‘below average’ or lower (1=unacceptable; 2=below average; 3=average; 4=above
average; 5=outstanding).

We are told that some auditors may receive a ‘below average’ rating in the next assessment, and
that those who are deemed below average or lower are encouraged to leave the firm.

Is there assurance that the auditors are from the upper and lower percentiles of the
performance evaluation ratings?

As part of the confidentiality agreement, we do not have access to the performance evaluation
records of the firms. Hence, we have no direct evidence about whether the training manager
(partner) followed the procedure of selecting subjects from the upper and lower end of the
performance evaluation ratings (but see response to preceding comment). We did receive

                                                                                                 12
assurance that they did as promised, and we have no reason to suspect that they did not. That the
managers from Firm 2 are able to identify the more outstanding senior (and the results for Firm 2
are similar to those from Firm 1) suggests that the subjects indeed differ in their performance
ratings as intended.

Sample size is small. Collect more data.

Our small sample size is a reflection of two conditions imposed by our research design –auditors
in the study must have worked with each other for a period of time, and these audit managers and
seniors have to be ranked high or low based on their firm’s performance evaluation scores.
These criteria impose constraints on the number of subjects available.

In line with your suggestion, we have collected more data from 12 seniors (6 outstanding and 6
mediocre seniors; experience=43 and 37 months, respectively) and 6 managers (3 outstanding
and 3 mediocre managers; experience=138 and 125 months, respectively). Although this
additional number is not large, we face constraints in collecting the data – other than the design
constraints described above, CPA firms are also very sensitive to the nature of the data we
requested.

The results from the new data we collected are similar to those that we had collected, which
gives comfort on the reliability of our results.

More disclosure of data

We have done so in this version of the paper. It is not our intention to selectively disclose data,
and we apologize if we indeed gave that impression. The inconsistencies you mentioned have
been corrected.

 Report both experience levels and familiarity ratings in footnote 3

 Done. Please see footnote 6.

 Why report medians (for t-tests) for partner ratings and means for others?

 In the previous version of the manuscript, we erroneously use the word ‘medians’ (line 6, first
 full paragraph on page 17 of previous manuscript) instead of the word ‘means’ (these are the
 mean values). We apologize for this. However, there is a reason for reporting t-tests. Recall
 that for Firm 1, we have four partners’ ratings for each senior; hence, we use the median rating
 for each senior in our analyses (similar results are obtained if we use the mean rating instead of
 the median rating; p > .276). Our t-tests are used to ascertain whether the average (mean) of
 these median ratings are different for memos written by outstanding seniors and mediocre
 seniors. Our results are not different if we use the Wilcoxon signed-rank test (smallest
 p=.610). We have tried to be clearer in our exposition in this version (please see page 17 and
 Footnote 9 of the revised manuscript).

                                                                                                      13
 No statistical tests are provided for the claim “there is no difference in the quality of memos
 evaluated by top and mediocre managers” (page 17).

 Our original intention is that the two t-tests (showing insignificant results) provide justification
 for this conclusion. We agree that running an ANOVA would provide better justification, and
 we have included this result in this version (please see page 17 of manuscript).

Be careful in placing study in context of other studies (e.g., Ramsay, on page 7; Asare and
McDaniel, on page 8).

Done. We have removed the said statements, and are more careful in terms of the positioning of
our paper relative to other studies.

Clarify the role of objectivity and effectiveness.

Done. On page 4 (paragraph 1) and page 19 (first paragraph), we note that while there are merits
to adjusting the review strategy in response to information about the preparer’s competence,
over- or under-reliance on the subordinates’ work can have adverse implications on audit
effectiveness and/or efficiency.

On page 6, the example of familiarity with the client has nothing to do with the objectivity of
the review process.

This example has been removed.

The theoretical discussion is sufficiently clear to question the need for page 10. The hypotheses
may not need this discussion of the paper’s method. Discuss the method in the method section.

We have removed the discussion of the paper’s method on page 10.

Wordings of H1 and H2 need to be clarified.

Done. See pages 10 and 11.

Submit copy of original instrument

Please see attached instrument.

Did managers’ review of the seniors’ memos only consist of the two scales provided, and were
they presented on the same page?

Yes. We document this on page 12 of the manuscript.


                                                                                                   14
How did you prevent seniors and managers from discussing the experiment in the three weeks
between stages?

We could not think of any way to prevent seniors and managers from discussing the experiment,
but we took certain precautions in the design of the experiment to reduce the likelihood of any
discussion effect. First, seniors write the memos under supervised conditions, and return the
memos to the researchers immediately after completion. Therefore, they do not have copies of
their own memos. Second, managers evaluate the memos also under controlled conditions, the
memos are returned after evaluation to the researchers, and the managers are unaware, at Stage 2,
that they will be asked to evaluate the memos a second time. This reduces the likelihood that they
will make an attempt to record the content of the memos or the evaluations given, or talk to the
seniors to learn more about the memos. We acknowledge these points in Footnote 5 of the paper.


We believe that these changes have improved the paper, and appreciate your very helpful
comments.


Sincerely



Hun-Tong Tan




                                                                                               15
                    Authors’ response to Reviewer accompanying revision

                               Response to Reviewer’s Comments

Thank you for your very helpful comments. We have incorporated them in the revised
manuscript, and summarize them below:

Mention the use of natural senior/manager team in the abstract.

Done.

Provide power calculations, rating changes, N and range of observations, and non-parametric
tests. Consider showing results for each observation.

As suggested by the consulting editor, we adopted the most straightforward way of addressing
the robustness of our data by collecting more data. The additional data are from 12 seniors (6
outstanding and 6 mediocre seniors; experience=43 and 37 months, respectively) and 6 managers
((3 outstanding and 3 mediocre managers; experience=138 and 125 months, respectively).
Although this additional number is not large, we face constraints in collecting the data. Our
small sample size is a reflection of two conditions imposed by our research design – auditors in
the study must have worked with each other for a period of time, and these audit managers and
seniors have to be ranked high or low based on their firm’s performance evaluation scores. Other
than the design constraints described above, CPA firms are also very sensitive to the nature of the
data we requested. These criteria impose limits on the number of subjects available.

Our results are now more significant, and the power of our tests are correspondingly stronger. Given
=.05, power > .80 for each of the two hypotheses. In this version of the paper, we also report rating
changes, number of observations, and range values in Table 1. We also perform non-parametric tests
for each of our hypotheses. Using the Wilcoxon signed-rank test, the significance levels for H1 is
.011. For Hypothesis 2, the significance levels for tests related to mediocre and outstanding
managers are .009 and .573, respectively (please see Footnote 8).

Although you suggested reporting a line by line report of the results for each observation, we are
unable to do so. As part of the confidentiality agreement with the participating firm, we can only
report aggregate results; we cannot show results of each observation as this will reveal
information the firm’s performance evaluation of specific auditors.

Specific comments

1. Put in additional motivation about implications of over- or under-reliance on
   subordinates’ work.

Done. Please see page 4, paragraph 1.


                                                                                                   16
2. On page 5, it is not clear that paper demonstrates a potential cost of relying on
   impressions.

We have removed this line.

3. Reword H1. Use the term ‘outstanding’ instead of ‘top.’

Done.

4. Why did subjects read memos with identification first?

An alternative is to provide memos without the seniors’ identification in Stage 1, but this is
associated with certain problems. Given that the memos are handwritten and managers work
closely with these seniors, removal of the seniors’ identification from these handwritten memos
in Stage 1 would not be helpful as managers are likely to recognize the handwriting. Providing
typewritten memos without the seniors’ identification to the managers would address this issue.
However, managers in our study are aware that their paired seniors would be writing memos
specifically addressed to them; removal of the seniors’ identification in Stage 1 (in addition to
typing the memos) may potentially alert the managers to the issue we are investigating and lead
to unintended demand effects. By providing managers with handwritten memos containing
seniors’ identification in Stage 1, we preserve the natural setting under which managers operate
under, and avoid the potential demand effects described above. To minimize the likelihood that
managers will recall or make a special effort to remember the evaluations given, we maintain a
three-week interval between the Stage 1 and Stage 2 evaluations, and do not inform managers, at
Stage 1, about the necessity of making a second evaluation.

We discuss this issue in Footnote 3.

5. Use the word ‘Stage’ instead of ‘Netrating’ when discussing the within-subject results.

Done.

6. Indicate that the effect of Netrating is only marginally significant.

With the inclusion of the additional data, the effect of Netrating is now statistically significant at
p=.012, and we report so on page 15 of the manuscript.

7. Provide degrees of freedom with the F scores.

We appreciate the need to report the degrees of freedom, but felt that this would be rather
repetitive if reflected in the text. As a result, we now include the ANOVA table together with the
degrees of freedom. We hope that this is satisfactory.

8. The comparison of differences between memos from outstanding seniors and mediocre

                                                                                                    17
   seniors is not needed (p. 17).

Given comment 12 (see below), we have decided to retain this comparison (see page 16) along
with an expanded discussion of its implications.

9. Tone down discussion of justification and self-promotion activities.

In response to your suggestion, we have toned down this discussion. We now simply highlight
that once a favorable impression of a subordinate has been formed (either through seniors’
impression management or actual good performance), subsequent behaviors may be interpreted
as positive behaviors (see last paragraph of page 18).

10. Acknowledge the small sample size as a limitation.

Done. Please see page 19, last paragraph.

11. Table 1 – label the stages as ‘Senior identified’ and ‘Senior not identified.’

Done.

12. Why do top managers increase their ratings while bottom managers decrease their
    ratings across stages?

In our analysis on page 16 (see also response to comment 8), we show that the difference
between outstanding managers’ ratings are not significantly different between stages, while
mediocre managers’ ratings are lower only for those related to work by outstanding seniors but
not for work by mediocre seniors. This result may suggest that mediocre managers are
particularly influenced by knowledge of the identities of outstanding seniors, while outstanding
managers are not influenced by such knowledge. We have included a discussion of this point on
page 17 of the manuscript.




                                                                                               18
                      Editor’s letter responding to submission of revision

Professor Hun-Tong Tan
Nanyang Business School
Room S3-01B-51
Nanyang Technological University
Nanyang Avenue
Singapore 639798
Republic of Singapore

MS #99-0226/R1

Dear Hun-Tong:

I have enclosed a review of your manuscript APerceived Subordinate Competence and Reviewer
Evaluation,@ Ms. 99-0226/R1. The enclosed (heavily) marked-up copy contains editorial
suggestions from a writing consultant (in pen) and me (pencil and Xeroxed pencil). Consulting
Editor Michael Bamber and I both read your paper in light of the review, and this letter expresses
our joint thoughts about the paper.

We all agree that this revision is a significant improvement over the original submission. As the
reviewer indicates, the additional data go a long way toward increasing the reader=s confidence
in the inferences. The remaining concerns are largely editorial, although some (particularly 1, 6,
7, and 10) will require careful thought.

1. Familiarity with a subordinate can increase efficiency and effectiveness in the audit process,
but the nature of such gains in review is less obvious. The paper=s introduction makes several
general references to such gains:

       p. 3 AThis familiarity with subordinates can lead to efficiency and effectiveness gains in
       the review process.@

       p. 4 AFirst, inability to objectively evaluate the quality of work performed ..... raises
       questions about the effectiveness of the audit review process as a quality control
       mechanism.@

       p. 6 AOur results .... suggest that some of the process gains documented in prior studies
       may be conditional ....@

The paper must clarify and provide specific examples of such process gains. Relatedly, the first
reason why reviewers= ability to objectively evaluate the preparer=s work (second quote above)
is important and requires more elaboration.

2. When discussing managers= familiarity with subordinates (e.g., either at the bottom of p. 5 or
                                                                                                    19
on p. 7) a footnote stating that managers participated in group evaluations of subordinates (as
indicated in the response to the consulting editor) would buttress the familiarity argument.

3. Is the first paragraph under the hypothesis development section necessary? The Treadway
example seems to be a tangential distraction.

4. The motivation of H2 should begin almost immediately with the discussion of tacit managerial
knowledge. The reader does not need half a page discussing general technical knowledge if the
paper simply relies on Tan and Libby (1997) to claim all managers should possess this
knowledge.

Also, the paper should make it clearer that average managers do not necessarily make less
accurate assessments of their subordinates, because the assessments are shared among managers
(or actually made as a group). The paper=s argument is that mediocre managers are more
inclined to rely on these shared assessments rather than evaluating the subordinate=s specific
work on its own merits.

5. Footnote 1 suggests that Aaverage@ is a more appropriate label than Amediocre.@

6. As the reviewer notes, seniors= performance evaluations are confounded with their
experience. The paper must recognize this limitation. The paper cannot determine whether the
average managers= reliance on the better seniors is attributable to: (1) the higher-rated seniors=
perceived better performance, or (2) the better seniors= greater experience. The second part of
footnote 4 suggests it is more than simple experience. This part of footnote 4 should also be
moved to the final section of the paper and included in the text when discussing the confound=s
implications.

7. The discussion of the experiment=s procedure is confusing, especially because of the
placement of footnote 3. This footnote explains why stages 1 and 2 were not randomized. It
would make more sense to put this after the discussion of both stages 1 and 2, not in the middle
of the discussion of stage 1 and before discussing stage 2.

Footnote 3 makes too much of this issue. To the extent that any carryover effects exist they work
against the hypotheses. As the paper recognizes, the opposite order may induce a demand effect.

8. Is the last sentence on p. 12 necessary?

9. Page 13 refers to Aseveral@ design features, but the paper identifies only two. Moreover, the
second one is not clear. Did you Aensure@ this (how?) or just request this from the participating
firms?

10. The fact that partners= independent evaluation of the memos found no quality difference
between the two groups of seniors warrants some comment on why better seniors did not write
better memos.
                                                                                                  20
11. Footnote 6 should be included in the text, and the reviewer=s related point on differences
between seniors in addition to managers also incorporated.

12. The test for firm differences could be relegated to a footnote.

13. The implications of the study discussed on p. 19 over-reach. This study does not examine
whether managers= reliance on seniors= evaluations differs across tasks. The discussion should
focus on the implications of the study=s actual results.

14. The enclosed marked-up copy has a number of editorial suggestions designed to enhance the
paper=s communication. Please use these as a starting point for polishing the exposition. That
is, the comments are a starting point, not a complete set of comments designed to yield a polished
manuscript. Our common goal is the highest quality paper that communicates as clearly and
concisely as possible. Please take this opportunity to make any additional changes that you
believe would enhance the paper, and certainly do not make any suggested changes that you
believe would detract from the paper.

15. I have enclosed a copy of The Accounting Review style guidelines. Please ensure that the
revision conforms to all these guidelines. Also, please look at tables in the April 2000 issue of
The Accounting Review and make sure your table formats correspond to those as closely as
possible.

16. Please check each of the references against the original sources to ensure that there are no
errors in titles, spelling, page numbers, etc.

I hope that you will be able to prepare a careful and thoroughly responsive revision within about
3-4 weeks. When the revised manuscript is ready, please email it to us at accounting-
review@terry.uga.edu. If you can get us a close-to-publishable manuscript by about July 17, we
will have a shot at the January 2001 issue. Given the magnitude of revisions on this round, I
would anticipate at least one more round of (hopefully only editorial) revisions.

I have kept a copy of the marked-up manuscript. Please do not hesitate to contact me if you
believe doing so would facilitate your revision. (I prefer email, but I also often work pretty late
at night so you may even be able to get me by phone despite the time difference.)

I will now turn to some formalities. Given the delay in international courier service, I am
sending some information that I would normally not send until I was certain that the paper was in
a publishable state. Assuming that the paper will become publishable, when the final version is
ready, please return the following to me:

Two hard copies of the manuscript (double-spaced).

A 3-1/2" diskette with the manuscript file. A diskette preparation form is enclosed. Note that the instructions for the
        diskette are not the same as for the hard copy of the manuscript (e.g., single-spacing is requested). Please
                                                                                                                    21
        ensure that the label of the diskette includes the authors= surnames, the paper=s title, and indicates that the
        paper will appear in The Accounting Review.

A signed Copyright Transfer Agreement. All authors must sign this agreement. At this point I hope that the paper
        will be able to appear in the January 2001 issue of The Accounting Review, but this depends on whether you
        can deliver a close-to-publishable paper by email no later than July 17.

After you have had a chance to think about the letters and the enclosed marked-up copy of the
manuscript, please let me know whether you believe that you can email us a very careful revision
no later than July 17, 2000. I would appreciate it if you could let me know a day or two before
you plan to email the revision, to give me a chance to clear the decks so that I can concentrate on
your manuscript as soon as you send it.

Thank you for considering The Accounting Review as an outlet for your research. You and your
coauthor have done an excellent job in capitalizing on the reviewers= comments to improve the
manuscript. Your paper will be an interesting addition to the literature on the audit review
process and audit performance evaluation. We look forward to receiving your revision.

Sincerely,



Linda Smith Bamber




                                                                                                                     22
                         Referee’s Report on submission of revision

                                     Reviewer Comments

                                Manuscript 990226 rev1
               Perceived Subordinate Competence and Reviewer Evaluation

I believe the authors have done a good job of addressing the concerns I raised regarding the
previous version of the paper. Of course, obtaining additional subjects goes a long way in
improving the persuasiveness of the paper. I have some minor comments below that I would like
to see addressed.

However, I do not believe the authors have addressed the first issue raised by the Consulting
Editor. Specifically the paper does not comment on whether the reported effect might be a result
of seniors’ experience. Moreover, as noted below, the authors do not report differences in
familiarity and time working together between outstanding and mediocre seniors. I believe the
authors should be more forthcoming in discussing possible alternative reasons for the differences
in first stage ratings of top seniors and mediocre seniors.

Other comments:
p. 4, 2nd para. 3rd line—delete second “with”.
p. 4, 2nd to last line—firms
p. 28, top line, delete second “under”
  p. 14—It seems that note 6 should address whether familiarity and time working together is
  different between outstanding and mediocre seniors.
P. 16—I believe that the separate post hoc analysis by manager rank should be a Tukey or similar
    test.
p. 18, 1st full para., 7th line—“suggests”
p. 19—please include the power calculation in the sample size discussion.
Table 1—Remove “tests of difference” from the column 3 heading.
  Table 1—Re “Range.” It seems the highest netrating score in the first row would be 8. That is,
  the highest outstanding senior (9) less the lowest mediocre senior (1). The same is true for
  other rows.
Table 1, Panel A, Row 3—Remove “Test of” from the row heading.
Table 2—given note 7, the * is not necessary.




                                                                                               23
                                  SECOND REVISION
                    Authors’ response to Editor accompanying revision

June 28, 2000


Professor Linda Bamber
Editor, The Accounting Review
J. M. Tull School of Accounting
University of Georgia, Athens GA 30602-6252

Dear Professor Bamber

                           Revision of Manuscript MS 990226rev1

Attached to this message is a file (TanJamalMS990226.doc) containing the revised copy of my
paper “Perceived Subordinate Competence and Reviewer Evaluation” with Karim Jamal.

We are very grateful to you for your very thorough reading of the paper, and the extensive
suggestions you have made. We have incorporated all these comments, and summarize them
below:

1a. Clarify and provide specific examples of gains from familiarity with preparers, and
from the review process

       Done. Please see page 3, where we document examples of gains from familiarity with
       preparers: less reperformance of the preparer’s work, and greater detection of conclusion
       and documentation errors. On page 4, we discuss the gains from the review process:
       enhanced attention to inconsistent evidence, as well as the detection of conclusion and
       documentation errors.

1b. Elaborate on why reviewers’ ability to objectively evaluate the preparer’s work is
important.

   Done. Please see page 4, where we explain that reviewers who over-rely on the work of a
   preparer perceived to be competent may become less vigilant in finding inconsistencies and
   let errors go undetected, thereby impairing audit effectiveness. Reviewers who under-rely on
   the work of a preparer perceived to be less competent may spend excessive time scrutinizing
   or reperforming the preparer’s work, impairing audit efficiency.

2. Add footnote to say that managers participate in group evaluations.

 Done. Please see footnote 1.


                                                                                              24
3. Deleted first paragraph of hypothesis development.

 Done.

4a.   Significantly reduce the discussion of technical knowledge.

 Done. Please see page 9.

4b. Average managers do not necessarily make less accurate assessments if assessments
are shared.

 Done, please see page 17, where we elaborate on how concerns about the quality of average
 managers’ evaluations may be mitigated by two environmental features within public
 accounting firms:
   (1) Audit seniors generally work with different managers on different assignments, and they
       should, over time and across assignments, benefit from receiving feedback and
       performance evaluations of different managers (both average and outstanding).
   (2) In some public accounting firms, annual performance evaluations are shared among
       managers, or made in a group setting. This suggests that average managers can benefit
       from learning about the performance evaluations made by other managers (particularly
       the outstanding ones). Moreover, the overall group evaluation of the subordinate’s
       performance should incorporate outstanding managers’ opinions (which are relatively
       more objective).

5. Use the word ‘average’ rather than ‘mediocre’ throughout the paper.

 Done.

6. Acknowledge the limitation that outstanding seniors are also more experienced. Move
   part of footnote 4 to this discussion.

 Done. See page 18, last paragraph.

7. Condense and re-position Footnote 3.

 Done. The footnote (now labeled as footnote 6) is now inserted at the end of the second full
 paragraph on page 11.

8. Is last sentence on page 12 necessary?

 This has been deleted.

9. Clarify paragraph on design features.


                                                                                                25
   Done. Please see end of page 11 and beginning of page 12.

10. Comment on why better seniors do not write better memos.

 Done. Please see footnote 10.

11. Include footnote 6 in main text, and discuss differences between outstanding and
    average seniors’ familiarity ratings.

 Done. Please beginning of page 13.

12. Relegate test for firm differences to a footnote.

 Done. Please see footnote 8.

13. Tone down discussion of implications of study. This study does not examine whether
    managers’ reliance on seniors’ evaluations differs across tasks. The discussion should
    focus on the implications of the study’s actual results.

 In response to your comment, we have substantially reduced our discussion on task differences.
  We did retain a line to highlight the limitation of using past performance as an indicator of
 future performance when subordinates perform tasks that require different skills than those
 currently employed

Other changes

14. We have made the changes in the marked-up copy of the manuscript. We have also rewritten
    paragraph 1 of the introduction to make it more succinct. We are happy to retain the original
    introduction if you think that is better.

15. P. 15, last paragraph. We rewrote part of the discussion pertaining to the partners’ ratings.
    We now talk about computing a difference score, Partner ratingoutstanding senior - Partner
    ratingaverage senior, so that it will be clearer to the readers. We also found a mistake in the
    partners’ ratings for the outstanding managers, and the correct figures should be: means =
    4.67 and 5.17, respectively; paired t-test = 0.56, p = 0.593. We apologize for the error.

16. We moved Footnote 9 to page 16, line 1 from its former position in the middle of the last
    paragraph, page 15.

Many of the reviewer’s suggestions have been included in your comments to us. For
completeness, we discuss some of the other comments below:

Conduct post-hoc analysis.


                                                                                                      26
       The p-value for the test concerned (see beginning of page 15) is highly significant (p =
       0.003), and using Bonferroni’s adjustment makes no difference to the conclusions (p =
       0.0015).

Include power calculation
       Adding the power calculation in the conclusion is fine, but we are not sure about the
       value of showing the power calculations, given that the tests are highly significant and are
       invariant to the use of non parametric tests (which are reported). Also, when the
       statistical tests are highly significant, power is almost necessarily high.

Remove the words “tests of” from table 1.
      We feel that the words are necessary because the columns and rows associated with these
      words are the only ones where we perform statistical tests.

Range – highest netrating score should be the difference between ratings of highest outstanding
senior less ratings of lowest average senior.
        The outstanding senior with the highest rating need not necessarily have been paired with
        the average senior with the lowest rating, so the highest netrating score cannot be
        obtained this way. Nonetheless, we did through our data again, and found an error in the
        range for row 1, column 3 (it should be “-3 to 7”). We have corrected this figure. We
        also checked all the other figures, and found no other errors.

Remove * from Table 2, given footnote 7.
      Done.

Thank you once again for your insightful suggestions. They have helped to improve the paper.


Sincerely


Hun-Tong Tan




                                                                                                  27

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:4
posted:7/4/2012
language:
pages:27