In a 2016 paper in the journal Evaluation and Program Planning, Frank Materia et al. provide a high-level overview of some of the advantages and considerations for performing program evaluation with mobile and internet technologies. Below is a bullet point summary of the issue they raise for those who don’t have access to, or the time to read, the full paper. Materia et al’s overarching message is that these technologies can reduce the cost of evaluation, while simultaneously enhancing recruitment, retention and improving the efficiency of outcomes evaluation. They state that:
Utilizing technology that exists, is built upon a well-organized infrastructure, and is widely accessible to evaluators and participants can help ease the burdens associated with program evaluation by reducing the resources needed (e.g., time, money, and staff) while increasing the reach of the evaluation.
BENFITS OF INTERNET-BASED PROGRAM EVALUATION
The authors note the following benefits of web-based surveys in program evaluation: Recruitment – websites can be used to advertise programs and distribute program content at a lower cost than traditional methods – websites allow for immediate sign up for participants – email invitations can be automated Tracking & Retention – web platforms can help streamline communications between the research team – personalized emails can boost response rates – automated email reminders can be scheduled based on time or fulfillment of previous survey assignments Process & Outcomes – response rates to web-based surveys do not in general differ from those to paper-based surveys except among the highly educated who are more likely to respond to web-based surveys – web-based surveys take less time to complete than paper-based one – they may also produce more valid responses as they are more private and there is less risk of social desirability bias in the responses– researchers can make some or all of the answers required to reduce missing data elements – researchers can add branching and complex skip logic to make the survey more user-friendly and easier to navigate – researchers can restrict responses to certain data types (e.g. a date) to eliminate data errors – there are lower costs, less administrative time and data entry is eliminated Some benefits not mentioned by Materia et al (2016) but provided by the ilumivu system: – precise response times can be measured for each survey element –multimedia stimuli can be presented as part of an online survey and response times captured – researchers have control over how many questions are presented at one time on the screen – participants’ cursor movements over the screen as they respond to the survey can be tracked – most computers have a built in camera, this can be integrated to capture facial expressions during survey completion.
BENEFITS OF MOBILE APPS FOR PROGRAM EVALUATION
The authors note the following benefits of using mobile apps in program evaluation: – alerts can be sent at scheduled times to complete a survey – numerous graphic displays (e.g. visual analogue scales) are easier for participants to complete than responding to multiple text messages – collect data beyond self-report by accessing phone sensors or wearable devices – trigger surveys based on these other data sources (e.g. trigger a survey when heart rate reaches a predefined threshold or a participant arrives at a specific location) – the authors also mention mobile web-based applications for content presentation and note that these require an internet connection and rely on its speed for accurate content presentation. The ilumivu apps can present multimedia content in the native app so they run offline whenever needed. The authors caution that,
Mobile apps hold substantial potential for program evaluation; however, careful planning is required to ensure that the significant time and resources invested in app development pay off for program evaluation.
This is the advantage of the ilumivu System. The infrastructure is already in place, you don’t need to manage the spiraling costs and timelines associated with new software development. The system is set-up for non-technical users to create surveys and alert schedules. The costs are known upfront and we have decades of data management and security experience under our belt to make sure the system will get through your IRB and keep your study data safe. The original Materia et al article can be found here For more information on the ilumivu system for program evaluation https://ilumivu.com/solutions/patient-monitoring/ and on our mobile mEMA apps https://ilumivu.com/solutions/ecological-momentary-assessment-app/ To receive new editions of the Researcher’s Guide to Wearables and other product news please sign up for our Newsletter.