AES Tech-Eval

Use of iPads for personal interviewing in Australian Aboriginal and Torres Strait Islander communities.

AES 2012 conference paper by Kylie Brosnan, Colmar Brunton and Your Source.
 

Kylie’s presentation slides, minus the videos (pdf)

“Using technology to evaluate technology”: paper on eHealth evaluation by Dr Rachael Holbrook (Australian National Prescribing Service) at the 2012 AES conference.
 

Conference paper on questionnaire design for online surveys, looking at the impact of gamification and interactive Q&A techniques on respondent behaviour and data quality.

By Duncan Rintoul and Jon Puleston. Winner of Best Paper at the 2012 AMSRS conference.
 

Beyond colour and movement: Measuring the impact of dynamic response formats in online surveys (pdf)

Poster from the 2012 AES conference, presenting evaluation findings re an online information resource and guide to the rights of people with mental illness. From Nick Roberts, Australian Mental Health Coordinating Council.
 

2011 report by FSG for the Knight Foundation (US): Measuring the Online Impact of Your Information Projects. Includes material on analytics in practice (eg what bounce rate, visits per unique visitor, and other metrics really mean in terms of online reach or engagement).

http://www.knightfoundation.org/publications/measuring-online-impact-your-information-projects

A presentation by Andrew Hawkins (ARTD) on evaluating websites. Delivered at the American Evaluation Association conference 2011, then at a NSW AES Workshop, May 2012

The Reach Out Website (Inspire Foundation) aims to promote mentally healthy behaviours among young people. It has been evaluated by Atari Metcalfe over several years. 

The wellbeing toolbox is a self-help website for veterans and their families. Del Lloyd (Australian Centre for Posttraumatic Health) presented an evaluation of the site at the 2011 AES conference.

Bursting the bubble is a website targeting young people, designed to prevent relationship violence. It was evaluated by Brad Shrimpton and colleagues at Melbourne University Centre for Program Evaluation. 

TAFE NSW used to run an intranet-based Professional Development Network. Larraine Larri (Renshaw-Hitchen & Associates) presented a poster on its evaluation at the 2003 AES Conference (Auckland).

Social software includes things like wikis, web forums, online focus groups, online surveys, digital photostories and video-blogging.

Larraine Larri (Renshaw-Hitchen & Associates) has contributed to several resources on use of social software, in evaluation and otherwise.

2006 Australian Flexible Learning Framework paper co-authored by Larraine with Val Evans:  Networks, connections and community: Learning with social software

Larraine’s 2007 and 2009 AES Conference roundtable session on use of social software in evaluation practice:

• Using social software to gather evidence, two case studies in education for sustainability (pdf)
• Web 2.0 / Social Software – What is it and what are the implications for Evaluation theory and practice? (pdf)

B.J. Fogg's work at the Stanford Persuasive Technology Lab sometimes falls outside the scope of traditional public health and social marketing, but is well worth a look.

FuelZone aimed to increase the uptake of school meals among children in Glasgow aged 11-16.

The project used a mix of canteen redesign and a new cashless payment card system which provided users with points (that varied on the ‘healthiness’ of their food choices) that could be used to redeem prizes online (eg i-pods, cinema tickets) as an incentive.

This case study is taken from the ShowCase database of evaluated social marketing programs, curated by the (UK) National Social Marketing Centre .

FuelZone case study page on ShowCase 

FuelZone full case study (pdf)

FuelZone summary (pdf)

Other case studies on ShowCase have had web based elements (eg a website), but the performance of these technologies has not often been measured all that well in the evaluation.

Three that have included some form of measure of their tech-based interventions are listed here. However, none of the evaluations are particularly precise about the tech-based aspects.

Take Charge. Take the Test. – a one-year social marketing campaign in Cleveland (OH) and Philadelphia (PA) to increase HIV testing in African American women at high risk for HIV infection. The www.hivtest.org website showed people where they could find out where they could find the closest testing facility was.

Fight Back aimed to reduce smoking prevalence in male routine and manual workers aged 35 to 55 living in a deprived part of the UK. The campaign included this viral video.

Be a star aimed to promote breastfeeding amongst 16- to 25-year-old mothers in central Lancashire. It included a two-way SMS system to provide 24-hour advice and support to mothers via text in the 12 weeks following hospital discharge, as well as a website and blog to provide support, information and signposting for women, partners and families on a range of breastfeeding and wider issues.

There is a tremendous amount of material about energy efficiency and the smart grid starting to appear in Europe, North America, and other parts of the world.

In addition to the links here, other potential avenues include:

• papers appearing in different IEEE Journals
• Paul Ryan (EnergyConsult Australia)
• Lloyd Harrington (Energy Efficient Strategies Australia).

See various resources on the American Council of Energy Efficient Economy website, eg:

• 2011 Behavior, Energy and Climate Change Conference
• 2011 Symposium on Market Transformation
• 2011 Summer Study on Energy Efficiency in Industry

Also see the proceedings from the International Energy Program Evaluation Conference (IEPEC) from 1997-2010

Doug Evans (George Washington University) has several new pieces coming out related to mobile and social media, and can share in future.

In the mean time, here’s one of his recent papers about evaluating sexual communication messages:

Evaluation of Sexual Communication Message Strategies (pdf)

A survey of Minnesota NGOs about their use of technology (and evaluation of its impact) was fielded in September 2011.