Combined Feedback on CAUL Communication Portal by tyndale


									Combined Feedback on CAUL Communication Portal 2006

Ambit Insights has sought feedback from CAUL on the CAUL (Portal) with a view to reviewing and improving the functionality of the Portal.

The purpose of the CAUL Portal is to communicate and compare results, and to provide a venue for the exchange of ideas among members.
The cost of developing and maintaining the Portal is included in the cost of CAUL surveys. Ambit Insights (incorporating Rodski Survey
Research) sees the Portal as an important part of their engagement with their clients, and seeks to improve the usefulness of the Portal

The suggestions provided by Ambit Insights were circulated for comment amongst CAUL members. Seventeen responses were received. A
summary of the responses is provided.

                                                         Rodski statement provided for comment
                                                                                                           Changes to the “Benchmark
                                      Changes to the “News and Updates” section
                                                                                                               Reports” section
                      It has been proposed that the section       The section would still show the         Improvements to the look and
                      show both current results and results       importance, performance and gap          feel of the charts might make
                      from previous years. Users reported         performance means across the various     them easier to understand and
                      that trend data would be very useful,       university libraries. However it was     explain. Font sizes in particular
                      and that many Universities are              suggested that we should also include    could be increased.
                      currently generating these data             the lowest performance means from
                      manually.                                   across the database.
Auckland              We agree with this proposal.                This suggestion would be useful to us.   Great.

Bond                  Yes, it would be good to have the trend     Yes, this would be very useful.          The font sizes are OK.
                      data as currently only 2005 is displayed.
Canterbury            Agree.                                      Agree.                                   Agree.
CDU                   See comments provided later
CQU                   See comments provided later

CSU           Good suggestion. The ready availability       Another good suggestion. I suspect that         Improvements would be
              of this data would be very useful for         the lowest performance means may be             appreciated especially increased
              CSU                                           similar for many university libraries (ie not   font size.
                                                            enough computers).
Edith Cowan   See comments provided later
Flinders      SFU -Yes this would be useful. Trend          SFU – Yes, more trend data would be of          SFU – They are a little busy at
              data on the quartile lines would also be      use.                                            times some more contrast in the
              of use, these are changing over time.                                                         colours might also help.
La Trobe      Results from previous years (perhaps                                                          I would like to know the actual
              via archive link) would be useful. So not                                                     values (average of all CAUL for
              just 2005 as at present. Trend data at                                                        each category) – not just the graph
              the individual institutional level is very
              important, and some is already being
              provided in the individual survey reports
              from Rodski. Perhaps this is best done
              on a case by case request to Rodski?
              More generally, national trend data
              would be interesting.
Macquarie     Yes I agree on that. And I think it will be   That could be an idea but don‟t know            The font size is OK I think, but
              better to have graphical representation       whether certain libraries would like to see     personally I don‟t think the
              of trends too to tell us whether there are    themselves showing up in the list. Also         benchmark report tells you much
              any shifts in terms of importance,            the picture would be fairer if we have the      over the performance. Particularly
              performance etc across the field. Also        user number and response number ratio           we don‟t have an “industrial
              for the top performance scores, it will be    listed side by side.                            standard” to compare against the
              better to provide the ratio of user                                                           actual performance, it seems to me
              number (e.g. student plus staff / the total                                                   that if we are benchmarking
              number of active users in the Library                                                         against each other, we could end
              system) and the user response no. That                                                        up have low expectations for things
              could provide a clearer picture of                                                            that we are not doing well across
              whether this ratio could affect the                                                           the industry because “everyone is
              results.                                                                                      not doing that well anyway”.
                                                                                                            Unless there is an industrial
                                                                                                            standard set there, it is difficult to
                                                                                                            have real benchmarking. It will not
                                                                                                            make a difference to the
                                                                                                            usefulness of the page whether we
                                                                                                            have bigger or smaller fonts.

Monash      Yes, that‟s a great idea.                                                             Agreed.
Newcastle   This would be a useful change – trends     I guess everyone is interested in seeing   I don‟t have any problems with the
            over time are more meaningful than one     where they stand comparatively to not      font sizes, perhaps an increase in
            off results, and automation rather than    just the highest performance mean, but     the actual size of the charts
            manual generation of this data would be    also the lowest performance mean.          themselves would be useful?
            useful.                                    Familiarity with the libraries that have
                                                       received both the highest and lowest
                                                       scores may be a good indicator as to
                                                       what staff and students at various
                                                       Universities perceive as good service as
                                                       well as bad service, as opposed to what
                                                       Library staff perceive as good and bad

Swinburne   Agree. Trend data would be very useful                                                Benchmark Reports is pretty thin
            – though perhaps its not “news” -                                                     given the amount of data available.
            maybe a new section. The data about                                                   From my perspective, its not so
            the year‟s collection could be expanded.                                              much the format as the extent of
            The pdf on responses is a good model –                                                the data. I like tables too – makes
            why not a pdf of all performance scores.                                              it easier to take the data and use it
                                                                                                  for something else.
UNE         I agree that this would be very useful     This would be of interest                  Yes
UNSW        This section should stick to news, and     Showing the lowest performance means       A bigger display would be helpful,
            keep the information changing. 18          across the board would not be helpful.     especially as the middle two
            month old news is not news any more        Results are most helpful when libraries    groups are small and it is easy to
                                                       make comparisons with their peer sub-      misread/misconstrue information
                                                       group, not across the whole cohort.
                                                       Lowest performance information would
                                                       not be constructive

USQ         Support production of trend data.          Yes, that would be useful.                 Agree
UWS         See comments provided later

                                                             Rodski statement provided for comment
                                                                                       Changes to the “Costing”                 Changes to the “Tutorials”
                Changes to the “Other Universities Data” section
                                                                                               section                                  section
             Additional programming would allow users to compare                    Clearer, more accessible                 It has been clear that this section
             different University results side by side. The universities            information about the costs of the       is not of great importance to
             could be selected from a drop down menu and the analysis               various levels of service could be       users. It is little used, time
             run in the background. This upgrade would automate a                   made available on the Portal.            consuming to update, and we
             process that some Universities are currently performing                Users should have open access to         recommend its removal.
             manually.                                                              information of this type to allow
                                                                                    better informed decisions.
Auckland     This would be appreciated.                                             Any extra information would be           Agree.
Bond         Yes – this would be good if it could be done on similar lines to the   Yes, including fees about add-on         Yes, remove.
             CAUL Online stats managed by CAVAL                                     products such as comments analysis,
Canterbury   Could be very useful. Similar to the use currently made of CAUL        Would be useful.                         Agree
CSU          This feature would be very useful. CSU for example, would be           No comment.                              CSU agrees with this
             interested in obtaining comparative data for institutions with a                                                recommendation.
             similar profile – eg DE, rural, etc.
Flinders     SFU – We are not running such processes on an ongoing basis            SFU – If in doubt we would ask.          SFU – Not used by us.
             but have done some snapshots so there could be merit in this
             suggestion depending on the complexity and cost of the
La Trobe     That would be very useful – e.g. for like groups such as GO8,                                                   Agree – of no use to LTU
             IRUA, ATN libraries Depends on cost of course
Macquarie    I think that will be a good enhancement as by doing so we can          I think that is a good idea because      I think this part is good but if it is not
             compare among ourselves according to our needs. For example,           then we will be able to cross-check      being used, it is difficult to justify its
             smaller university libraries can compare among themselves to           other services already introduced by     presence. Maybe a little more
             see how they are doing, which will give them a better picture of       other libraries and make our own         marketing would help? Being new to
             their performance on a similar level of available resources. Also it   budgeting easier if we want to include   this page, I think I won‟t know the
             helps us to set our goals and targets if we want to catch up with      those new services in the years to       presence of this part if I am not
             or surpass other libraries in the years to come.                       come.                                    actually filling this questionnaire. I will
                                                                                                                             just thought tutorial is all about using
                                                                                                                             the web-page, which I think most
                                                                                                                             information professional they will just
                                                                                                                             jump to the pages they need to go to
                                                                                                                             instead of going through a “tutorial”

Monash      Yes, this would be great.                                                                                        Could this possibly be replaced with
                                                                                                                             a section relevant to running surveys
                                                                                                                             (eg. Information on any latest
                                                                                                                             information on methods to boost
                                                                                                                             response numbers etc.)?

Newcastle   Now this would be handy – being able to compare results side by                                                  I have to admit I have only skimmed
            side rather than having to either print or cup/paste etc to compare                                              over what‟s available in the tutorial
            various Universities (i.e. Libraries that you might be in a                                                      section. I would not oppose its
            partnership with etc) would be a great tool.                                                                     removal from the site.
Swinburne   Yes, it would be fantastic to be able to create a set from the list   Agree.                                     Agree.
            and generate some analysis of the set. I suspect that is not
            easily done however. Giving a table with the results for each
            institution would be easier and make life easier for people who
            want to generate subsets. It would be great to be able to drill
            down further, to be able to compare results for specific questions
            from different subsets.

UNE         We would do more comparisons if this feature were added               Yes good idea                              No objections to its removal

UNSW        Strongly endorse this proposal. Most libraries select a               This section is OK. If a library needs     Agree
            comparison group of peers. Comparisons are generally not made         specifics, then a specific request is in
            regularly against all libraries. Automation would replace currently   order.
            practiced manual processes. Automation should also allow for
            comparisons against the full list for the occasions when this is
            appropriate, and for searching for atypical data for a specific

USQ         Yes, would be very useful                                             Agree                                      Agree. Don‟t need this sort of
                                                                                                                             resource on a portal of this kind –
                                                                                                                             available elsewhere if needed.

                                                             Rodski statement provided for comment
               Changes to the “Quizzes” section                                      Addition of the “Employee Survey” Section
             It has been clear that this section is not of    All respondents requested that          We hope that in the fullness of time this section
             great importance to users. It is little used,    more information be made                would be enhanced to mirror the functionality of the
             time consuming to update, and we                 available on Employee Surveys. At       Library Client Survey online results pages. This
             recommend its removal.                           the very least, a description of our    would enable Clients to compare and contrast their
                                                              offering and the associated cost        employee survey results with those of others, isolate
                                                              would be useful. Users could            areas of best practice, and network with their peers
                                                              download sample survey forms            to increase performance.
                                                              and reports.
Auckland     Agree.                                           No comment.                             No comment.
Bond         Yes, remove.                                     Yes, this would be good.                Great.
Canterbury   Agree                                            Definitely. Do not use Employee         Would be a great asset.
                                                              survey at present, but are interested
                                                              in pursuing.
CSU          CSU agrees with this recommendation.             This additional information would be    No comment.
Flinders     SFU – Not used by us.                            SFU – The university has a process
                                                              for employee surveys and we are not
                                                              presently considering more surveying
                                                              in this area.

La Trobe     Agree – of no use to LTU                         Agree                                   Agree
Macquarie    Same as the last section, but I think it will    That could be useful if the Library     Ideally helpful but needs time to prove its validity,
             have more potential if we group the two          wants to measure their own staff over   because I don‟t know whether individual libraries want to
             together and rename it as work related issues    different issues.                       spell out their problems publicly on these pages. The
             or something like that to raise the awareness                                            concept is good, but would we be able to find out the
             of the users about it. “Quizzes” doesn‟t                                                 truth about their internal performances?
             explain its contents at all.

Monash       Agreed.                                          Yes, it would be interesting to see
                                                              sample survey forms and reports on
                                                              the portal.

Newcastle                Same goes for “Tutorials”                     Sounds like this could be a handy        Any tool that allowed comparison / improvement of
                                                                       addition. The University recently ran    performance would be useful, along with the opportunity
                                                                       a broad employee survey so not sure      to network.
                                                                       if we could justify one in the near
                                                                       future but would be useful to have the
                                                                       information there should the option
                                                                       become available.
Swinburne                Agree.                                        I am not aware of the “Employee
                                                                       Survey” section.
UNE                      Not something I‟ve used enough to remember    Yes                                      Good idea
                         it‟s there
UNSW                     Agree                                         Agree

USQ                      Agree – as above.                             Yes, would be very useful.               Agree

                                                        Rodski statement provided for comment
                                  Addition of an “Offshore Students Survey” Section                                                      CAUL Portal Updates
Last year a number of CAUL members ran separate projects for “offshore students”. There are actually four major groups            As is currently the case, the Portal
of students who use CAUL Libraries: 1. Australian students at Australian Universities 2. Overseas students physically             will be updated twice a year.
studying in Australia 3. Overseas students studying at overseas campuses of Australian universities 4. Overseas students
using Australian University online Libraries while possibly using another University‟s physical facilities.
It is proposed that more significant work needs to be done around this point in coming years, but that in the interim those
institutions that ran an „Offshore Student survey‟ should have access to their results online.

Auckland                 Not applicable.                                                                                           We are fine with that.
Bond                     NA                                                                                                        Since Bond has three semesters
                                                                                                                                   (and many public universities have
                                                                                                                                   summer semesters) and the twice
                                                                                                                                   yearly update seems aligned with a 2
                                                                                                                                   semester year, it would be good if the
                                                                                                                                   portal could be updated at least 3,
                                                                                                                                   maybe 4 times a year, particularly if
                                                                                                                                   one wants to compare up-to-date
                                                                                                                                   trend data. In 2004 we had to wait
                                                                                                                                   several months after the survey
                                                                                                                                   before our data was available.

Canterbury   Not applicable to this Library.
CSU          This would be very useful. An additional category of interest for CSU is CSU students studying at affiliate    No comment.
             institutions both onshore and offshore. It would be beneficial to gauge if the physical Library resources
             and services provided by affiliate institutions (coupled with CSU‟s online services and resources) meet the
             needs of these students.
La Trobe     Agree more work needs to be done in this area. We have looked at the current survey for 3, and have            Agree
             decided it would need major modification if we were to use (e.g. we don‟t have overseas La Trobe
             campuses, but we do have overseas La Trobe students studying at partner institutions and we would like
             to survey them)
Macquarie    I think this is a good idea as we are moving into a real digital age to support overseas students / offshore   If the portal is going to include current
             students. It will be good to be able to compare with other universities to know what and how they are doing    findings, should it be updated more
             in this area.                                                                                                  frequently to show new changes?
                                                                                                                            Also whenever there are updates,
                                                                                                                            would the libraries be notified? This
                                                                                                                            could increase the usage of the portal
                                                                                                                            and maybe even sections that are
                                                                                                                            under utilized at the moment.
                                                                                                                            Sections like the Tutorial and
                                                                                                                            Quizzes will benefit a lot if we have
                                                                                                                            regular updates and current
                                                                                                                            information and new developments in
                                                                                                                            the industry being posted there. It
                                                                                                                            could greatly help the Australian
                                                                                                                            libraries to move forward in different
Newcastle    Not having run an offshore student survey the interim measure would have little use for us.                    Sufficient
Swinburne    Yes. Agree. I‟d be very interested to track progress of the other institutions results from this exercise. I   OK
             think these results will need more contextualization – the very rudimentary summary charts in the
             benchmark style will be misleading for the Offshore results.
UNE          Not something we have used, but I agree with this. We do identify distance students in our surveys. They       OK More often could be good, but not
             usually complain that some questions are not relevant to them, but I don‟t actually agree with that            vital
UNSW         This is an area of increasing interest to many Australian universities, so this sounds useful                  Acceptable
USQ          Agree                                                                                                          Yes

                                                       Rodski statement provided for comment
                     CAUL Portal Update notifications                                            Additional Comments
              We propose to send an email to CAUL
              members each time an update occurs, and
              perhaps one other time per semester. This
              would encourage Portal usage and give us
              another opportunity to add value.
Auckland      That would be useful.

Bond          Yes.
Canterbury    Would be an excellent prompt.                           I do not look at the portal very often, but when I do I find it useful, and if the
                                                                      suggestions above are implemented it would be even more useful. Thank you
                                                                      for the work already done, and for seeking feedback.
CDU                                                                   General feeling at this end is that the portal is realy useful at the moment, and
                                                                      any of the improvements already indicated would be a bit of a bonus! So
                                                                      nothing further to add.
CQU                                                                   CQU concurs with the statements made in the document.
CSU           Good idea.                                              No additional comments.
Edith Cowan                                                           Provision of more marketing assistance such as templates for posters.
La Trobe      Agree

Macquarie     That basically covers what I raised in the last         It will be good if we can have regular discussions on certain topics, maybe
              question. I think an increase in communication          have the surveyed libraries to put follow up actions in response to their
              channel will be good. If the portal is going to serve   surveys in the portal? So other libraries can use them as references in the
              as an area for us to exchange ideas and                 future if they encounter certain similar results in their own surveys. Otherwise
              benchmarking, then regular updates and                  the portal will only be a place where we compare figures, which, with no
              communication is vital for the success.                 context are just numbers instead of information.

Monash        Once again, a great idea so that we know that an
              update has occurred.
Newcastle     Email notification would be a good reminder to
              check the updates.

Swinburne   Yes – very useful to know if there is an update.
UNE         I thought this already happened.
UNSW        This would be useful. Current use includes             Having access to peer institutional data in an efficient and quick way is very
            periods of fairly intense usage, and then extended     useful. Having all data for occasional access is also useful. Most of the rest
            inactivity, so alerts would be helpful for prompting   of the portal is of limited value and could go if maintenance of the critical
            awareness                                              areas needs more resources
USQ         Agree

UWS                                                                Successive surveys benchmark against the cohort of libraries undertaking the
                                                                   survey at that time – the results do not refer back to the previous survey
                                                                   cohort. This method has resulted in UWS Library‟s 2 survey reporting
                                                                                                                              nd               st
                                                                   improvement between surveys, with movement from 2 quartile 2004 to 1
                                                                                                                           st                     nd
                                                                   quartile 2006…when we were in 2004 in fact in the 1 quartile rather than 2
                                                                   as indicated. This comparison is meaningless. There needs to be clearer
                                                                   comparison made between successive surveys, the „membership‟ of that
                                                                   survey period, and reference to previous surveys. It would be useful to have
                                                                   survey results compared with all organizations using the instrument, and with
                                                                   university libraries only, thus comparing our sector with the wider base of
                                                                   organizations using Rodski.


To top