These days it is no surprise to see mainstream and niche programs making use of tech-based platforms: web-based self-help tools, mobile applications, SMS-based reminder systems, viral videos, conversations on social media... the list is much longer than this, and ever growing.
There is a need to develop capacity among evaluators to work confidently in this environment, designing and executing sound evaluations that understand what these technologies are, how they can be used and how their impact can be measured.
There are also great opportunities for using technology in our evaluations – wikis, online forums, online surveys, social media monitoring... again the list is long and growing.
Spilling over from one of the parallel sessions at the 2011 Australasian Evaluation Society Conference a crew of around 15 people has started pulling together a new AES Special Interest Group around this intersection between evaluation and technology: AES tech-eval.
Join the mailing list
To connect with others in the AES tech-eval group, add your email address to the mailing list.
Go on, sign up! If technology freaks you out, swap fear of the unknown with curiosity and see where it takes you. If you’re already working comfortably in this space, help lead your colleagues forward.
If you have trouble with the mailing list, contact Duncan Rintoul.
Resources and links
This site also serves as a repository of relevant published evaluations, resources and links.
If you can add resources to this list, contact Duncan Rintoul.
Use of iPads for personal interviewing in Australian Aboriginal and Torres Strait Islander communities.
AES 2012 conference paper by Kylie Brosnan, Colmar Brunton and Your Source.
Kylie’s presentation slides, minus the videos (pdf)
“Using technology to evaluate technology”: paper on eHealth evaluation by Dr Rachael Holbrook (Australian National Prescribing Service) at the 2012 AES conference.
|Conference paper slides (pdf)|
Conference paper on questionnaire design for online surveys, looking at the impact of gamification and interactive Q&A techniques on respondent behaviour and data quality.
By Duncan Rintoul and Jon Puleston. Winner of Best Paper at the 2012 AMSRS conference.
Poster from the 2012 AES conference, presenting evaluation findings re an online information resource and guide to the rights of people with mental illness. From Nick Roberts, Australian Mental Health Coordinating Council.
|Poster - the NSW Mental Health Rights Manual (pdf)|
2011 report by FSG for the Knight Foundation (US): Measuring the Online Impact of Your Information Projects. Includes material on analytics in practice (eg what bounce rate, visits per unique visitor, and other metrics really mean in terms of online reach or engagement).
A presentation by Andrew Hawkins (ARTD) on evaluating websites. Delivered at the American Evaluation Association conference 2011, then at a NSW AES Workshop, May 2012
|Andrew’s slides (pdf)|
The Reach Out Website (Inspire Foundation) aims to promote mentally healthy behaviours among young people. It has been evaluated by Atari Metcalfe over several years.
|Atari’s slides from the 2011 AES conference (pdf)|
The wellbeing toolbox is a self-help website for veterans and their families. Del Lloyd (Australian Centre for Posttraumatic Health) presented an evaluation of the site at the 2011 AES conference.
|Del’s slides from the 2011 AES Conference (pdf)|
Bursting the bubble is a website targeting young people, designed to prevent relationship violence. It was evaluated by Brad Shrimpton and colleagues at Melbourne University Centre for Program Evaluation.
|The evaluation findings are summarised online. The findings were also summarised in a guide to developing websites for young people (first half, second half).|
TAFE NSW used to run an intranet-based Professional Development Network. Larraine Larri (Renshaw-Hitchen & Associates) presented a poster on its evaluation at the 2003 AES Conference (Auckland).
|Key elements of this poster (pdf)|
Social software includes things like wikis, web forums, online focus groups, online surveys, digital photostories and video-blogging.
Larraine Larri (Renshaw-Hitchen & Associates) has contributed to several resources on use of social software, in evaluation and otherwise.
2006 Australian Flexible Learning Framework paper co-authored by Larraine with Val Evans: Networks, connections and community: Learning with social software
Larraine’s 2007 and 2009 AES Conference roundtable session on use of social software in evaluation practice:
B.J. Fogg's work at the Stanford Persuasive Technology Lab sometimes falls outside the scope of traditional public health and social marketing, but is well worth a look.
FuelZone aimed to increase the uptake of school meals among children in Glasgow aged 11-16.
The project used a mix of canteen redesign and a new cashless payment card system which provided users with points (that varied on the ‘healthiness’ of their food choices) that could be used to redeem prizes online (eg i-pods, cinema tickets) as an incentive.
FuelZone full case study (pdf)
FuelZone summary (pdf)
Other case studies on ShowCase have had web based elements (eg a website), but the performance of these technologies has not often been measured all that well in the evaluation.
Three that have included some form of measure of their tech-based interventions are listed here. However, none of the evaluations are particularly precise about the tech-based aspects.
Take Charge. Take the Test. – a one-year social marketing campaign in Cleveland (OH) and Philadelphia (PA) to increase HIV testing in African American women at high risk for HIV infection. The www.hivtest.org website showed people where they could find out where they could find the closest testing facility was.
Be a star aimed to promote breastfeeding amongst 16- to 25-year-old mothers in central Lancashire. It included a two-way SMS system to provide 24-hour advice and support to mothers via text in the 12 weeks following hospital discharge, as well as a website and blog to provide support, information and signposting for women, partners and families on a range of breastfeeding and wider issues.
There is a tremendous amount of material about energy efficiency and the smart grid starting to appear in Europe, North America, and other parts of the world.
In addition to the links here, other potential avenues include:
See various resources on the American Council of Energy Efficient Economy website, eg:
Also see the proceedings from the International Energy Program Evaluation Conference (IEPEC) from 1997-2010
Doug Evans (George Washington University) has several new pieces coming out related to mobile and social media, and can share in future.
In the mean time, here’s one of his recent papers about evaluating sexual communication messages:
A survey of Minnesota NGOs about their use of technology (and evaluation of its impact) was fielded in September 2011.
|Link to follow once results are in.|
Any ideas for other resources, publications or links that should be added?
Send your suggestions to Duncan Rintoul.
Latest media coverage
The power of love - interview with Hugh Mackay, Illawarra Mercury, 4 May 2013
Den einflussreichsten Konsumenten finden (“Find the most influential consumers”) - interview with A/ Prof Ulrike Gretzel, Salzburger Nachrichten (in German, Austrian newspaper), 21 April 2013
Master's degree kicks off at Wollongong - Research News Magazine, April 2013 issue
The mobile phone turns 40 - interview with A/ Prof Ulrike Gretzel, The Wire radio, 4 April 2013
Digital detox: the rise of tech-free tourism - interview with A/ Prof Ulrike Gretzel, The Age, 11 March 2013
Technology improves emergency warnings - interview with Dr Katina Michael, ABC TV News, 27 Jan 2013
Concern people without latest technology will miss fire warnings - interview with Dr Katina Michael, ABC Radio News, PM with Mark Colvin, 23 Jan 2013