When a new concept is being developed, verifying its acceptability is a vital challenge for any company. When hearing UX research, most people immediately think of qualitative research, and only that. At Uism however, we have abundant experience in both qualitative and quantitative research. From our wealth of experience and know-how, we can provide concept validation support ranging from the deep psychological understanding for qualitative research to specific conceptual verifications in quantitative research, attuning to the issues and needs of our client.
To showcase this in detail, we would like to introduce an actual research case we conducted, in which the client developed new service concepts with the goal of encouraging change in user behavior.
To form the base of our research, the questions we first asked were:
- How much cognitive empathy would users have for these concepts?
- How would concepts be able to influence existing cognitive and behavioral patterns?
- How could the development process be improved?
- How should we prioritize the multiple concepts in making development decisions?
Starting from Qualitative Research
As an important first step in verifying the acceptability of a new service, we normally begin with qualitative research. The main purpose of this phase is to gain a deeper understanding of user behavior, feeling, and pain points, to help guide quantitative research later on. The details of the example case are below:
Details:
- Sample size: n=12
- Method: One-on-one remote interview (IDI)
- Activity: Conduct exercises to explore participants’ expectations and pain points in detail using an online whiteboard tool (Miro)
- Stimuli: Collect concept reactions and feedback
Results showed positive responses to the concept, in addition to user type categorization and type-specific journey visualization. In particular, the results suggested that for some user types, the new service has the potential to promote behavioral change. The results of these qualitative studies will serve as the foundation for the quantitative study.
Verify Acceptability with Quantitative Research
While the results obtained from qualitative research are very important and valuable in understanding how new service concepts address user expectations and pain points, due to sample size limitations and the presence of multiple concepts at once, more extensive and specific validation is necessary. Therefore, for this case, we conducted a quantitative survey to build on the results of the qualitative study. We developed a preliminary analysis plan, conducted an online quantitative questionnaire, and employed three main analytical methods on a larger user scale.
①Verification of Overall Concept Impact
Using a statistical technique called Analysis of Variance (ANOVA), we analyzed how three different factors (in this case, user type, experience, and changes in opinion) affected a particular outcome. This analysis is a powerful tool for understanding complex interactions and for assessing in detail how much each factor contributes to an outcome. This method of analysis is a way to examine how multiple factors affect outcomes simultaneously on the opinions of certain user types, and can consider not only individual factor effects, but interactions between factors as well. We also examined how much the overall service contributed to the customer satisfaction score (NPS: Net Promoter Score).
②Overall Concept Behavioral Change Effects
In order to evaluate in detail the specific effects of the overall concept on behavioral change, we asked the subjects to rate their intention to use a specific service or take a specific action on a 10-point scale. Scores of 6 or higher, indicating an intention to take some action, were used to analyze how the presentation of the concept affected the user’s intention to take action.
This analysis determined the extent to which the presented concept was effective in encouraging behavioral change by measuring the percentage of subjects who changed from a no-action intention to an action intention after the concept was presented.
③Assessing the Impact of each Concept
In sections ① and ②, we evaluated multiple concepts collectively. In section ③, however, we assessed and compared the individual influence of each concept. Usually, evaluating numerous concepts would require presenting a single concept to a person, necessitating a high volume of samples. In this study, to ensure a statistically significant sample size, we had groups where all concepts were presented and some where not all were shown, helping us gauge the varying impact levels of each concept through logistic regression analysis.Logistic regression analysis is a method that quantifies the relationships between variables, estimating the factors that increase the likelihood of particular outcomes (e.g., “behavioral change occurs” or “doesn’t occur”). We used an odds ratio to measure the effect magnitude, indicating the likelihood that a specific event (in this case, the presentation of each concept) induces a certain outcome (in this case, a change in behavioral intention). Additionally, we formulated hypotheses for opinion improvements based on responses about usefulness, allowing us to delineate how each concept influences behavioral change in detail.
Results and Final Words
Through qualitative research (emotional analysis) and quantitative research (effectiveness analysis), we were able to assist the client in developing new service concepts more clearly, resulting in a better picture of which user types to focus on and which concepts to prioritize for development, and helping to improve overall development efficiency.
Depending on which phase of the development stage you are in, the challenges you face will vary in many ways. Our job at Uism, as your research partner, is to assist you in finding and developing solutions using the most appropriate methods, taking into account your budget, your timeline, and all of the challenges you face. We are not just a tool, but a wealth of knowledge and experience, both qualitative and quantitative. We use this to provide custom-tailored research to meet the needs of each project and help contribute to your business’ success. Please feel free to contact us with any inquiries you may have.
When a new concept is being developed, verifying its acceptability is a vital challenge for any company. When hearing UX research, most people immediately think of qualitative research, and only that. At Uism however, we have abundant experience in both qualitative and quantitative research. From our wealth of experience and know-how, we can provide concept validation support ranging from the deep psychological understanding for qualitative research to specific conceptual verifications in quantitative research, attuning to the issues and needs of our client.
To showcase this in detail, we would like to introduce an actual research case we conducted, in which the client developed new service concepts with the goal of encouraging change in user behavior.
To form the base of our research, the questions we first asked were:
- How much cognitive empathy would users have for these concepts?
- How would concepts be able to influence existing cognitive and behavioral patterns?
- How could the development process be improved?
- How should we prioritize the multiple concepts in making development decisions?
Starting from Qualitative Research
As an important first step in verifying the acceptability of a new service, we normally begin with qualitative research. The main purpose of this phase is to gain a deeper understanding of user behavior, feeling, and pain points, to help guide quantitative research later on. The details of the example case are below:
Details:
- Sample size: n=12
- Method: One-on-one remote interview (IDI)
- Activity: Conduct exercises to explore participants’ expectations and pain points in detail using an online whiteboard tool (Miro)
- Stimuli: Collect concept reactions and feedback
Results showed positive responses to the concept, in addition to user type categorization and type-specific journey visualization. In particular, the results suggested that for some user types, the new service has the potential to promote behavioral change. The results of these qualitative studies will serve as the foundation for the quantitative study.
Verify Acceptability with Quantitative Research
While the results obtained from qualitative research are very important and valuable in understanding how new service concepts address user expectations and pain points, due to sample size limitations and the presence of multiple concepts at once, more extensive and specific validation is necessary. Therefore, for this case, we conducted a quantitative survey to build on the results of the qualitative study. We developed a preliminary analysis plan, conducted an online quantitative questionnaire, and employed three main analytical methods on a larger user scale.
①Verification of Overall Concept Impact
Using a statistical technique called Analysis of Variance (ANOVA), we analyzed how three different factors (in this case, user type, experience, and changes in opinion) affected a particular outcome. This analysis is a powerful tool for understanding complex interactions and for assessing in detail how much each factor contributes to an outcome. This method of analysis is a way to examine how multiple factors affect outcomes simultaneously on the opinions of certain user types, and can consider not only individual factor effects, but interactions between factors as well. We also examined how much the overall service contributed to the customer satisfaction score (NPS: Net Promoter Score).
②Overall Concept Behavioral Change Effects
In order to evaluate in detail the specific effects of the overall concept on behavioral change, we asked the subjects to rate their intention to use a specific service or take a specific action on a 10-point scale. Scores of 6 or higher, indicating an intention to take some action, were used to analyze how the presentation of the concept affected the user’s intention to take action.
This analysis determined the extent to which the presented concept was effective in encouraging behavioral change by measuring the percentage of subjects who changed from a no-action intention to an action intention after the concept was presented.
③Assessing the Impact of each Concept
In sections ① and ②, we evaluated multiple concepts collectively. In section ③, however, we assessed and compared the individual influence of each concept. Usually, evaluating numerous concepts would require presenting a single concept to a person, necessitating a high volume of samples. In this study, to ensure a statistically significant sample size, we had groups where all concepts were presented and some where not all were shown, helping us gauge the varying impact levels of each concept through logistic regression analysis.Logistic regression analysis is a method that quantifies the relationships between variables, estimating the factors that increase the likelihood of particular outcomes (e.g., “behavioral change occurs” or “doesn’t occur”). We used an odds ratio to measure the effect magnitude, indicating the likelihood that a specific event (in this case, the presentation of each concept) induces a certain outcome (in this case, a change in behavioral intention). Additionally, we formulated hypotheses for opinion improvements based on responses about usefulness, allowing us to delineate how each concept influences behavioral change in detail.
Results and Final Words
Through qualitative research (emotional analysis) and quantitative research (effectiveness analysis), we were able to assist the client in developing new service concepts more clearly, resulting in a better picture of which user types to focus on and which concepts to prioritize for development, and helping to improve overall development efficiency.
Depending on which phase of the development stage you are in, the challenges you face will vary in many ways. Our job at Uism, as your research partner, is to assist you in finding and developing solutions using the most appropriate methods, taking into account your budget, your timeline, and all of the challenges you face. We are not just a tool, but a wealth of knowledge and experience, both qualitative and quantitative. We use this to provide custom-tailored research to meet the needs of each project and help contribute to your business’ success. Please feel free to contact us with any inquiries you may have.