How to Run a Successful EMA/EMI Study

On Time and On Budget Without Hiring a Developer

If you are a researcher involved in digital health or using ecological momentary assessment or intervention methodologies, then I want to show you how to successfully run your grant-funded study on time and on budget without hiring a developer.

We’ve helped hundreds of researchers over the past ten years to do just that

Some Examples of our Work

Suicide Ideation Phenotyping 

 

This study was done by a group at Harvard Medical School and was published in the journal “Anxiety and Depression” in April 2018. It was led by Evan Kleiman who is now an Assistant Professor at Rutgers University.

 

Before this study was done suicidal thinking had always been studied as a homogeneous construct – which was only due to the methodologies available. They only allowed for snapshot assessments to be taken days, weeks or months apart missing the variation in suicidal thoughts that happens through the course of a single day. So our understanding of suicidal ideation and its relationship to the act of suicide was blunted by the available methodologies.

 

This group used the mEMA System which allowed them the opportunity for “digital phenotyping” – real-time characterization and quantification of human thought in situ. The research team used the mEMA app to ask the same four questions of their participants multiple times throughout the day for 28 days each.  

 

They found that instead of suicide ideation being a homogenous construct there were actually five previously unknown distinct phenotypes of suicidal thought. Each phenotype differed in terms of intensity and variability. They also found a significant relationship between having more severe and persistent suicidal thoughts and a recent suicide attempt. 

 
This work has opened up new avenues in the understanding of suicide and it’s prevention that were previously unavailable. 

Ecological Momentary Intervention for Marijuana Use 

This study was done by a group at Boston Children’s Hospital and was published in the journal “mHealth” in 2018. It was led by Lydia Shrier (MD, MPH) who is an attending physician at Boston Children’s. 

This group developed a motivational counseling intervention that they had previously found to be effective in reducing marijuana use among adolescents. They wondered if they could make it more effective by adding an ecological momentary intervention component. 

 

The researchers used the mEMA System to provide personalized motivational messages to patients throughout their day – after asking them questions about who they were with, what they were doing and how they were feeling. The messages were tailored to the individual based on the counselors’ in-person interactions with the patients and additionally tailored to the moment based on how they answered questions at each survey instance throughout the day.

 

They found that patients in both the motivational counseling condition (MET) and the EMI assisted (MOMENT) condition showed both lower marijuana use and desire to use at the end of the study. Furthermore, they found that the desire to use marijuana in the moment during targeted contexts (such as hanging out with friends) decreased even more in the EMI assisted condition. They found their patients enjoyed using the app and reading the intervention messages.  

This work opens the door to increasing the efficacy of substance abuse treatment programs in our adolescents – a highly vulnerable population.  

Dynamics of Prescription Opioid Use

This study was done by a group at Johns Hopkins University and was published in the “Journal of Pain” in January 2018. It was led by Patrick Finan who is an Assistant Professor at Johns Hopkins.

 

They worked with people who have Sickle Cell Disease- a chronically painful condition often treated with opioids. Opioid prescriptions of course come with the risk of addiction and abuse but very little was previously understood about how patients make decisions about which prescription opioids to take and when.  

 

The researchers used the mEMA System to provide patients with a daily diary questionnaire each evening. They asked about which opioids they had taken that day – short or long-acting, how they were feeling and probed some of their thought patterns around the pain they were experiencing.

 

They found that higher levels of pain and of catastrophizing about their pain led to greater use of short-acting opioids. Whereas negative affect led to more use of long-acting opioids. They also found that catastrophizing about their pain led patients to use more opioids even when pain levels were relatively low. 

 

So they were able to shed light on the momentary cognitive and affective dynamics underlying prescription opioid use behaviors that had previously been hidden from view by outdated methodologies. 


None of these groups hired a developer and they all delivered their grant funded studies on time and on budget. 

Who is the mEMA System for?

If you are an academic researcher in the field of psychology, behavioral medicine, social science or public health looking to run an ecological momentary assessment or intervention study then you need to find a way to collect those data within the timeframe and budget dictated by your grant. In that past that might have meant hiring a developer or hobbling together a system with open-source software. 

 

For this study you are looking for a solution that is going to work without you having to do any hiring or figuring out how various pieces of software function or might talk to each other. You’re looking for something that is seamless and allows you and your research team to focus on research not on the technical headaches of running an EMA or EMI study. 

 

You’ve worked hard to get to where you are in your career. Grant funding is very competitive. You have an idea for a study that is innovative and will be a significant contribution to the current literature. You know if you can get this study completed successfully you will be able to apply for larger grants to do even more innovative work in the future. You’re excited about the intersection of your field with technology and want to lead the wave of research that is leveraging this new technology to make a positive impact for the populations you work with. If this is you then you are in the right place.

 

Receiving a grant is a big responsibility. You are accountable for delivering on what you promised. You have to hire the right people, find the right measures, recruit participants, keep them engaged, make sure your data are secure and oversee all the analysis and writing. It’s a lot of moving parts! The last thing you need on top of all that is trying to figure out how to make various pieces of unfamiliar or old software work so you can collect your data.


You may have learned this the hard way in the past. That’s part of the learning curve the entire field has been going through as we’ve been bridging the gap between academic research and ever-changing consumer technology. Behavior change research methods have undergone a revolution in the last ten years as Smartphones, wearable and connected devices have become more and more available. It’s understandable that this has caused some turbulence and created opportunities for developers who maybe want to help but don’t understand the needs of a grant funded research study.

As the Smartphone assisted EMA and EMI field has been developing researchers have tried all sorts of things to capture data from people as they go about their daily lives – some have worked, some….well… not so much! The promise of being able to collect not only self-report data but also objective contextual data and physiological data in parallel is alluring but to a researcher without a technical background the task can seem overwhelming. 

 

The truth is that you don’t need to custom build an app and backend database to run your EMA or EMI study. 

And the fact is that even if your grant budget and timeline allowed you to do a custom build by the time you’ve built the thing it’s out of date anyway and you’ll have to build another one for the next study. 

 

The long grant funding cycle doesn’t match up with the rate of development of consumer technology. They are two completely different markets with different drivers. Trying to use your research study to straddle those two canoes is going to leave you soaking wet! 


We’re not the only people around dedicated to solving this particular problem for researchers but we’re the ones innovating the fastest to keep up with the consumer tech market so you don’t have to.

Who are we?

Before we go on and I show you how not to end up straddling those two canoes and easily run a successful EMA or EMI study let me introduce myself. My name is Kat Houghton and I co-founded ilumivu in 2009 – one year before I went back to school to get my PhD in Psychology. 


At the time I was working intensively with children with autism and their families with play-based, parent-mediated interventions in a private treatment center. I saw every day how families were struggling to keep track of all the therapies and treatments they were using and trying to make sense of what was working and not working. I co-founded the company with Mark Tuomenoksa (our CTO) because I knew there was a more effective way to help parents track data and share it with all their child’s care providers. Mark has been developing software since the 1980s, he worked at Bell Labs on part of the original Unix kernel and has one of those minds that thinks in systems and seems to solve every software problem he comes across by making the system more efficient and broader reaching than it was before.

 
The first platform we created back in 2009 was a HIPAA compliant website for parents of children with autism. We quickly realized what we had built had a much broader application and when we stripped out all the autism specific content it could be used to support any human subjects research study. We started to get requests from researchers to use the platform for a wide range of research needs – substance abuse, suicide prevention, anxiety and depression, mindfulness interventions and so on.

Kat Houghton PhD, Co-Founder & CEO
Mark Tuomenoksa, Co-Founder, CTO

I went back to school to get the PhD because I also knew there was a more effective, and more cost-effective way to train parents to play with their kids in a way that facilitates socio-emotional and communication development. I learned a variety of play-based interventions methods for autism and distilled them down into their essential components so they could be quickly taught to parents across the world in underserved communities. I used research methodologies grounded in complex systems theory to better assess change in family systems. I worked in the Bronx, in Poland, the Netherlands, Oman, Argentina and across the US, Canada and the UK. Through that process I learned a lot about behavior change and how to facilitate it in a wide variety of populations.

 

I don’t recommend running a tech startup and getting a PhD simultaneously in case you know anyone considering it! 

 

They say that what doesn’t kill you makes you stronger – I survived! And now Mark and I get to co-pilot the world’s most innovative EMA and EMI platform. We combine his deep expertise in software development with my understanding of creating and delivering behavioral interventions for humans. 


So let’s get back to learning how you can run a successful grant funded EMA or EMI study on time and on budget without hiring a developer. 

The Value in Your Research is NOT in the Technology

The value in your research is not in the technology – it’s in the research design and what you do with the data.

 

You’ve spent years, perhaps decades, becoming an expert in your field. That field of research is relying on you to keep probing the important questions and making new breakthroughs in theory and implementation. You can’t do that effectively if simultaneously you are attempting to figure out how to guide a development team to build an app. There are too many unknown and frequently changing variables for you to be able to effectively manage both projects.

 

Stop spending your time on technology development and focus on what you do best – research design and data analysis. 

 

The research project you are planning has very specific requirements, there are certain variables you need to measure in specific ways at particular points in time. But you don’t need a custom built app to do that – you simply need a system that allows you to define what you want to measure, how you want to measure it and at what point in time for each of your study participants. Assuming that system allows you to do all that without any coding expertise then you are good to go!

 

We recently started working with a hospital system research center. They came to us after wasting $50K with third party developer who built for them, from scratch, an app to deliver an Ecological Momentary Intervention that still wasn’t able to collect data reliably after 8 months. We got them up and running in 1 month. Now they are collecting data and delivering intervention content via our app to patients that come through their eating disorders clinic. How did we manage that? 

 

Because our platform has been in production since 2009 and has been designed to be agnostic to content we don’t need to build any software from scratch. You just type in your content – your survey items, intervention content, notification schedules, and decision logic –  and off you go.


So if you are thinking of custom building an app, please consider, is this really what you want to spend your limited time on?

Fast, Easy, Iteration is Key

Easy, fast, iteration is key to designing a study and especially an intervention. 

Being able to test your hypotheses quickly, collect data from participants then refine and test again is a critical part of any design process. When you are paying by the hour for each change to your custom built app your budget is going to quickly limit how much you can iterate. 

 

If you are moving from traditional data collection or intervention delivery methods to Smartphone or wearable data collection and delivery then you will need to go through the build-test-refine cycle a few times – even if it’s just with your own team – before you launch your official study. You need the freedom to test and refine your protocols. 

 

We supported a study out of USC that was focused on smoking cessation with Korean youth in California. They knew they wanted to deliver a smoking cessation intervention for this specific demographic group but didn’t yet have all the data they needed to do that. So they designed a two phase study with mEMA. In Phase 1 they simply collected data from participants about where they were, what they were doing and who they were with each time they smoked. Once they had collected these data from each individual in the study they moved into Phase 2 where they changed the content each participant saw every day on the app to deliver tailored intervention messages when the app knew they were at those trigger locations, doing those activities or with those people. 

 

They were able to go through that entire process with the same app because the app is agnostic to content – they were able to change the content as they needed it, and to see the data coming back from participants in real-time so they knew when to change the content. Any changes they made made to content were automatically delivered to that participants’ app they next time they connected to the server – which was happening multiple times a day.
No lag time, no waiting for developers to make changes, no astronomical coding bill. 

 

It’s easy to get caught in short-term thinking and looking for a solution to solve the problem of delivering your upcoming project. But take a moment to think forward and imagine the evolution of this work:
if technology wasn’t a constraint – you could make it do whatever you want – how would you design the next study?

Mobile Operating Systems are Dynamic

Expect Mobile Operating Systems to Change

You might think that if you invest the time and resources into building an EMA app then it will save you time and money long term. Once it’s built then it’s built and there’s nothing more to do right? 

 

No! Both the Android and Apple operating systems are updated frequently – by which I mean multiple times a year – on each platform. Each update can bring unknown changes that may affect your app. Complex apps, like mMEA, that tap into many of the instruments on the phone – for instance, the location services, the notification service, the camera, microphone and so on – are especially likely to be impacted by operating system updates. When you add in integrations with other devices and wearables then there’s a lot of maintenance to take care of.

 

It’s a full time job – literally! If you are thinking about building your own EMA or EMI app then you’ll need to factor in hiring both Android and Apple developers to continue to maintain the apps once in production. That is a significant expense.


Again, think about the long-term evolution of your work. Do you want to be managing a team of developers or spending your time focused purely on research?

So to run a successful EMA or EMI study on time and on budget without hiring a developer you just need to use a system that allows you to:

> Decide which variables you want to capture, how you want to measure them and when

> Iterate frequently on your design and add new data streams as you need them

> Plan ahead to keep up with changing technology without having to budget for custom software develop

 

The DIY Method is Always Available

Instead of hiring a developer you could attempt to patch together a system from various pieces of software out there. You could find an online survey provider to do the baseline and follow up questionnaires, maybe try a free, made by researchers EMA app or a text messaging survey system and try to find a provider that will give you data from a wearable sensor. 

 

But then you are left with the headache of finding all these different parts – that is going to suck up a lot of your time

Then you have to make all the parts talk to each other or at least be in enough alignment that you can recognize one individual’s data in all the systems. Something as seemingly simple as having a shared subject ID# likely will not work across different systems. Leaving you with a mass of datasets you have to somehow match manually.

 

Similarly different systems use different naming and timestamping protocols – it’s hard to make sense of datasets that are timestamped in different ways, maybe one uses Unix time while another uses data/time format in UTC and another translates UTC into local time to the phone. 

With a system like this who do you turn to when you have questions or something is not working?
There’s no one, it’s up to you to figure it out.
 

The mEMA System

The alternative is to use an end-to-end EMA and EMI platform that handles all your data streams in a cohesive manner bringing everything into one database with a consistent naming and timestamping protocol. Ideally a system that can accessed by anyone with a Smartphone and quickly customized to capture exactly the data you need.  

That’s why we created mEMA in 2009. It’s the most innovative EMA and EMI platform in the world today. We can get your complete study up and running on time and on budget. 

 

The mEMA System is for researches planning an EMA or EMI study who want to successfully complete you data collection within the timeline and budget you promised on your grant application.  

So remember, the Harvard Medical group were able to discover previously unknown phenotypes in suicide ideation by taking repeating measures at multiple times a each day and using latent profile analysis


The Boston Children’s’ Hospital group took an already effective marijuana abuse treatment program and made it even better by sending personalized messages to their patients at just the right moment. 

 

And the Johns Hopkins group revealed a much deeper understanding of the cognitive and affective mechanism that lead Sickle Cell Disease patients to choose short or long-acting prescription opioids throughout their day.  

These are just three examples from dozens of studies the mEMA System has supported over the last ten years.  

Using the mEMA System You will be able to:

  • Allow your participants to use their own phones, no burden or cost of additional devices
  • Add objective, contextual data to self-report measures – such as location, phone acceleration, ambient light and sound, phone use and so on
  • Collect data from wearable devices – for instance we are fully integrated with Garmin wearables so can capture real-time heart rate, HRV, stress, steps and get daily summaries of various other metrics including sleep
  • Easily move through the IRB and university tech audit – we’ve done it hundreds of times
  • Put your study on auto-pilot and watch the data roll in
  • Access support whenever you need it for troubleshooting with participants
  • Know how much data you have at any time and who is in compliance

How We Work

A subscription to the mEMA System allows you to iterate on your design at no additional cost. Once you have access to the mEMA System you can build-test-refine as many times as you need to without incurring additional costs or waiting for someone else do to the work. You just do it yourself.

 

Our business model has always been to do all the development heavy lifting and provide the system to multiple researchers at once so no group is paying the full load of employing app developers. This allows us to offer the functionality you need at a much more affordable price. It also means that researchers get the benefit of updates and additional features added for others at no extra cost.

 

If you need a feature that mEMA doesn’t already carry and it is something that other researchers are likely to use then we will add it for you. This is how we’ve built the entire system – in response to the requests of researchers. It is continually growing as we interact with more researchers doing more and more innovative work. 


For instance, for a group from the University of Minnesota we recently added the ability to capture audio recordings of participants reading a standard passage so the recordings can be used for voice analysis work. They are interested in capturing emotional states through voice feature detection. This is now a standard part of our Professional License.   

mEMA System Components

The mEMA System is comprised on two main components – the web portal where you log in to create your surveys, notification schedules and find your data – and the mobile apps on Apple and Android with which your participants interact to provide data. 

 

It allows you to capture self-report data, contextual data from the phone sensors – such as ambient light and noise, acceleration, phone usage, and local humidity. You can integrate various wearable devices to capture Heart Rate, HRV, Stress, Steps, Sleep and activity. You can also run cognitive tests through the mEMA apps which collect reaction time data to millisecond precision. 

 

You don’t need any programming experience – you simply type and click icons to set everything up. Your data are stored securely and in our HIPAA and GDPR compliant platform. You can send unlimited surveys and notifications to your participants and include up to 1,500 participants at any time. 


We start each license with an online training session to show you how to use everything you need to set up your study. Then you will have access to our online Help Desk which provides written and video tutorials. Our staff are available by email and phone to make sure you have everything you need to run a successful EMA or EMI study

Let's Talk!

If you have an upcoming EMA or EMI study, then I’d love to talk to you about the requirements of your study and see if the mEMA System will be a good fit for you.

Our pricing is term-based; you purchase a subscription to the system for 6 months, 1 year or more. Subscriptions start at $2,650 for the 6-month Basic License.

We are a small company, which allows us to stay agile and keep ahead of the technology development curve and it limits the number of new projects we can support. We take only 10 new research projects each month.

If you’d like to talk to me about your upcoming EMA or EMI study please book a call. As soon as you book a call we will send you more videos that describe some of the mEMA platform features in more detail so by the time we get on the phone together you’ll know exactly what you need to ask. 

Thank you for exploring using the mEMA platform. 

I look forward to talking with you soon. 

Kat Houghton PhD

https://www.linkedin.com/in/kathoughton/