CARE OPINION Research chat 6: Perfect ratings with negative comments

Our research chat on 11 May 2021 featured the work of Andrew Gallan, associate professor at Florida Atlantic University in the US. We were delighted to have Andrew join us for this chat.

Research from the US and the UK describes the phenomenon of patients giving healthcare providers perfect ratings (10 out of 10!), yet contradicting this with comments which may include negative experiences and suggestions for improvement.

In this chat we discuss why this happens, what we can learn about the nature of patient feedback, and what implications this might have for policy and practice.

Watch the session

Perfect Ratings with Negative Comments: Learning from Contradictory Patient Survey Responses

Perfect Ratings with Negative Comments: Learning from Contradictory Patient Survey Responses

Lead Research Article in Patient Experience Journal


This research explores why patients give perfect domain scores yet provide negative comments on surveys. In order to explore this phenomenon, vendor-supplied in-patient survey data from eleven different hospitals of a major U.S. health care system were utilized. The dataset included survey scores and comments from 56,900 patients, collected from January 2015 through October 2016. Of the total number of responses, 30,485 (54%) contained at least one comment. For our analysis, we use a two-step approach: a quantitative analysis on the domain scores augmented by a qualitative text analysis of patients’ comments. To focus the research, we start by building a hospital recommendation model using logistic regression that predicts a patient’s likelihood to recommend the hospital; we use this to further evaluate the top four most predictive domains. In these domains (personal issues, nurses, hospital room, and physicians), a significant percentage of patients who rated their experience with a perfect domain score left a comment categorized as not positive, thus giving rise to stark contrasts between survey scores and comments provided by patients. Within each domain, natural language analysis of patient comments shows that, despite providing perfect survey scores, patients have much to say to health care organizations about their experiences in the hospital. A summary of comments also shows that respondents provide negative comments on issues that are outside the survey domains. Results confirm that harvesting and analyzing comments from these patients is important, because much can be learned from their narratives. Implications for health care professionals and organizations are discussed.

Recommended Citation

Gallan, Andrew S.; Girju, Marina; and Girju, Roxana (2017) “Perfect ratings with negative comments: Learning from contradictory patient survey responses,” Patient Experience Journal: Vol. 4 : Iss. 3 , Article 6.
Available at:

Download the article for FREE here:

Learning from Patient Narratives for Human-Centered Design to Transform PX

Andrew Gallan, PhD, Assistant Professor of Marketing at the Kellstadt Graduate College of Business at DePaul University, and Senem Guney, PhD, CPXP, Founder and Chief Experience Officer at NarrativeDx discuss the interplay between “Top Box” domain scores and patients’ comments in measuring patient experience.

Dr. Guney shares her expertise in how to bring a mixed-methods approach (qualitative/quantitative analysis) to metrics and measurement in patient experience, and Dr. Gallan shares insights from one of his recent studies on the contradictions between perfect scores in distinct PX domains (such as nurse or doctor communication) and comments related to these distinct domains.

This webinar also involves the implications of such contradictions for working with PX data, specifically to determine the drivers for patients’ willingness to recommend a hospital.

To access video of the webinar, please go here:

Physician Transparency Requires Understanding Patients’ Comments

This is a guest blog post by Andrew Gallan, PhD, a teacher, researcher, and adviser to organizations on designing services to improve patient experience.

Now that over 50 healthcare organizations regularly publish patient evaluations of physicians on their public-facing websites (Ed Bennett Consulting 2017), transparency around physician ratings and reviews has become a growing focus in healthcare. University of Utah Health Care is a pioneering institution in today’s transparency movement.

While transparency around ratings can (understandably) be stressful for physicians, it can be very useful for patients
searching for the right physician for their own care (Patient Engagement HIT 2017). Transparency is about presenting digestible data to patients to help inform their purchasing decisions.

To that end, survey scores are converted into a five-star system; and all comments, positive and negative, are posted on a hospital’s website with redacted identifying or personal information.

Why are healthcare organizations adopting physician rating transparency?
It is now imperative for health care organizations and professionals to manage their online presence and reputation (PatientTrak2017). While Yelp and other third-party online sources capture attention, physician profiles on providers’ websites typically outrank other sources of information and provide more valid and reliable data (Mahoney 2016). CMS-mandated data, such as those collected from CAHPS surveys, are from verified patients of a specific provider, and the questions have been developed with good psychometric qualities.

When a healthcare organization transitions toward online physician transparency, they motivate physicians to improve performance, explains Vivian Lee, M.B.A., M.D., Ph.D. in a recent article for the New England Journal of Medicine. When physicians are compared to one another, very often they seek out counsel to improve the service they provide to patients. This can lead to improved patient-centered care and quality outcomes.

Then, the question for physicians and healthcare leaders becomes: If I’m being held accountable to certain standards, how can I improve my scores?

The answer to this question lies in patients’ comments. Patients might give high survey scores to physicians, but they describe their true care experiences in their comments. Physicians can learn from these first-person accounts about what patients need and expect of their care. It’s also true that people identify with stories over numbers, so a sample of patient comments that represent a specific trend out of aggregate qualitative data would resonate very well with physicians about the improvements they need to make.

How can healthcare organizations manage what is now transparent to potential patients?
As a result of physician transparency, it is more important now than ever to bring a scientific approach to understanding what patients are saying in their own words. Healthcare organizations need to make sense of both structured data (survey scores) and unstructured data (patient comments). By analyzing patient comments, providers can learn about patients’ experiences above and beyond CAHPS domains. Artificial Intelligence (AI) and Natural Language Processing (NLP) are methodologies that provide the capability to analyze and make sense of large amounts of text. When this analytical capability is coupled with intuitive visualizations on dashboards and easy-to-digest reports, such as those offered by NarrativeDx, it becomes so much easier for providers to learn from and build improvement actions on what patients are saying.

The goal of patient experience improvement is to gain a better understanding of what is important to patients.

Some examples that bring this issue to life include:

  1. If patients complain about food, what are they saying? Are they commenting about taste, temperature, choice, or other issues?
  2. If patients complain about noise, what are they hearing? Alarms, conversations, other patients, visitors?
  3. If patients don’t understand discharge information, what do they recall? Do they remember the timing of when the information was given, the amount of information provided, instructions surrounding new medications, follow-up appointments, or other issues?

Patient experience professionals can now engage new techniques and tools to unearth additional insights from existing data that will help improve care experiences, and happy patients and families will in turn give much higher ratings to physicians.

Andrew Gallan, PhD specializes in researching, teaching, and advising organizations on designing and delivering patient-centered care. He has worked with organizations including Mayo Clinic (AZ), University of Chicago Medical Center, UPMC, and Advocate Health Care on their journey toward delivering world-class care. Andrew has engaged audiences as a keynote speaker, breakout presenter, and committee facilitator and advisor. His graduate class on Patient Experience, the first in the nation, is among the most highly rated classes at DePaul. He is a Beryl Institute speaker.


How Home Health Can Achieve the ‘Wow Factor’ with Patients

When home health providers think of the “patient experience,” high-quality care is likely what comes to mind first. In reality, physical care may just be one aspect of an outstanding patient experience.

The real definition of patient experience can be beyond what providers and caregivers are used to, Joyce Boin, principal consultant at Strategic Health Care Solutions, explained to Home Health Care News.

“All patients assume as a home health provider you are sending a competent caregiver. The ‘wow factor’ comes in when you expand that idea even further than just great care,” Boin said. “The above and beyond comes from compassion and empathy.”

Patient Experience Project

In addition to caregivers showing compassion and empathy, another way home health providers should train caregivers to go above and beyond is by teaching how to engage patients in the care process, Andrew Gallan, an assistant professor at DePaul University who researches the patient experience, told HHCN.

This engagement could be as simple as asking the patient one or two extra questions to learn more about them, or explaining certain aspects of the patient’s recovery to keep them engaged in what’s happening.

Gallan has made the patient experience in health care his focus. He began his research in hospitals, but moved into the home care sector through a project with an Illinois home health provider, Advocate at Home, where he and Boin partnered together.

The project consisted of three phases that took just over two years. First, a survey was conducted among home health patients to see where caregivers stood in terms of quality of care. Then Gallan pinpointed the top drivers of the patient experience and created an index to track for those things. The items he tracked included communication with home health staff members, taking care of emotional needs and overall perception.

Second, Gallan did phone interviews with the patients to try to find out more details about the areas in phase one. In phase three, Gallan attended ride-alongs with home health staff to witness the caregiver/patient relationship firsthand. In total, Gallan made about 40 patient visits in the city of Chicago, the suburbs and rural areas of Illinois.

The results of the project found that patients overall had very good relationships with their home health providers, but they were reluctant to talk about their emotional state.

“Home health providers did a very good job of delivering the actual care, but the relationships formed could have been better,” Gallan explained. “The caregivers could have been better active listeners. Something as small as putting away computers or tablets and taking a few extra minutes to ask a few more questions could help create a clearer picture of the patients’ emotional state.”

Making goals very clear can also improve the overall patient experience, added Gallan.

“If patients can identify a goal, they can see a light at the end of the tunnel,” he said. “It’s a highly motivating thing. If caregivers can help them identify those goals, patients can make significant progress.”

Though the project is over now, Gallan continues to work with Advocate at Home to help implement the full patient experience and build upon the project’s findings.

Investing the Upfront Time

Caregivers can improve the patient experience is to work on building a strong relationship with the caregiver by encouraging patients to become engaged. Allowing caregivers to spend more time on visits sounds like it could be inefficient, but investing time upfront to build relationships and develop goals with patients actually has a positive impact on recovery times, according to Gallan.

“I know some of the pushback about being more engaging with patients is because it will take more time, but my response is that if it’s done well, it’s not going to be that much more time down the road,” he said. “And then, if the patient knows what’s expected of them from the beginning, once the nurse comes in they don’t have to reinvent the wheel.”

Building a strong relationship between caregiver and patient can also help put family members at ease.

“It’s a cause of concern and anxiety for some families when their loved ones are receiving in-home care,” said Boin. “The idea of having caregivers as somewhat of a support system is what value-based purchasing is really going to buy. If caregivers can deliver on that, the patients and caregivers will become more efficient.”

Written by Alana Stramowski


Calculating and Understanding the Drivers of a Net Promoter Score in Health Care

The Beryl Institute Blog Series:

In 2016, Advocate Health Care, the largest health system in the Chicagoland area, integrated into its performance measures a Net Promoter-like score, which they call a Patient Loyalty Score (PLS). Net Promoter Score (NPS) is a valuable metric, and it has been adopted by many companies in almost every industry. NPS is a simple, easy to use, and easily calculated metric that is intuitively associated with business health by assessing a respondent’s likelihood to recommend an organization to a friend or colleague.[1] Health care organizations are beginning to see its value, and are exploring how it is best calculated and used.

For Advocate Health Care, PLS is constructed using data from CAHPS and vendor surveys, and utilizes the likelihood to recommend question. Only a top-box score is defined as a promoter, and varying bottom scores are defined as representing a detractor. That is, for a five-point scale (ED vendor survey) the bottom three responses are categorized as detractors; for a four-point scale (HCAHPS) the bottom two are detractors; and, for a three point scale (CG-CAHPS) only the bottom score is a detractor.

Some issues with the measure include the referent (CG-CAHPS asks about likelihood to recommend the provider’s office, ED refers to the department, and HCAHPS asks about the hospital), and the limited scale width (the original NPS scale is 11 points). However, for me, having a patient-provided measure outweighs the issues, and I commend the organization for holding people accountable for patient perceptions of care. The strength of this metric is to create system-wide responsibility for a patient-provided measure, thereby ensuring that the patient’s voice is heard.

Like most organizations, Advocate Health Care is interested in earning increased rates of positive word-of-mouth recommendations. As a result, I recently engaged with Advocate as an Academic-in-Residence. In this role, championed by EVP & COO Bill Santulli, SVP & CNO Susan K. Campbell, and VP Information and Technology Innovation Tina Esposito, I performed analytics to identify drivers of PLS. The two important research questions that drove this project were:

  1. Which variables are the most important drivers of PLS?
  2. What can we learn from patient comments about potential drivers of PLS?

In order to investigate these questions, I was provided with almost two years of HCAHPS, CG-CAHPS, and ED survey data and patient comments. Top line results included the following:

Inpatient (HCAHPS): Nurses and personal issues (privacy, pain, and emotional issues) had by far the most impact on patients. Positive comments centered on comfort, communication, and care. Negative comments focused on food.

Outpatient (CG-CAHPS): The face-to-face interaction between a patient and physician is the “moment of truth,” and as such is what the patient apparently will use to evaluate the entire experience. Positive comments centered on comfort and communication. Negative comments focused on waiting and rude treatment.

Emergency Department (Vendor Data): When patients are in the ED, taking care of personal issues will have the greatest impact on PLS. These issues include keeping patients informed about delays, caring about patients as people, pain control, and providing information about caring for yourself at home. Positive comments centered on comfort, communication, and care. Negative comments focused on feeling vulnerable and afraid in a busy and foreign environment.

As a result of this project, Advocate Health Care is now embarking on disseminating the results, integrating insights into daily practice, and evaluating additional questions that emerged from the analysis. I’d be interested in hearing more about what your organization thinks about NPS, how you use it, and what you have learned as a result!

[1] NPS was first proposed by Fredrick F. Reichheld, (2003), “The One Number You Need to Grow,” Harvard Business Review, 81 (December), 46-54. For more on advantages and issues utilizing NPS in health care, see

Andrew S. Gallan PhD is an assistant professor at DePaul University in Chicago, a member of the Editorial Review Board of Patient Experience Journal, and principal of Dignity in Action, Inc., a PX analytics and advisory company ( Andrew can be contacted via email:


Five Reasons Why Practices Should Track Their Net Promoter Score

Happy patients not only keep coming back, they also recommend your medical practice to their friends and family members.

Innovation at DePaul: Predictive Analytics for Health Care

Innovation at DePaul: Predictive Analytics for Health Care


What’s the practical value of big data in the health care industry?“It’s limitless,” says Raffaella Settimi (above), an associate professor in the College of Computing and Digital Media (CDM). “The first objective among stakeholders—doctors, hospitals, clinics, insurance companies, and pharmacies—is better patient outcomes, and data is playing a big role there. But it’s also being used to address other challenges, such as reducing costs and improving communication. Every health care organization is being touched by data analytics.”

And that’s why CDM and the Kellstadt Graduate School of Business combined their expertise to design a health care concentration within the MS in Predictive Analytics. “Companies are looking for data scientists who also understand the industry. And there’s a huge gap between demand and supply,” says Bamshad Mobasher, a professor in CDM and director of the Center for Data Mining and Predictive Analytics. “Our students are having no problem finding work.”

Marty Martin (above), an associate professor in the Kellstadt Graduate School of Business, agrees: “Provider organizations, like Advocate and Presence, want our students. Insurance companies, like Blue Cross Blue Shield, want our students. Technology companies, like McKesson and Cerner, want our students. They’re all looking for data analysts who can help answer hard questions: What variables predict disease, infections, and injury? Which treatments work best, for whom and why? Which populations are most at risk for illnesses?”

Megan Lipps (MS ’16) chose the health care concentration because of these opportunities “to make a difference,” she says: “Retailing has been using data science forever, but in health care it’s really, really new. How can we make people healthier?  That’s a compelling question.”

Bridging the Gap

The use of big data is all about context, says Settimi: “A data analyst in health care has to ask the right questions and identify the right data sources. It’s not about ‘crunching the data’—it’s about solving problems.” Domain knowledge is very important, and “our students learn a lot that’s above and beyond data science,” says Martin:

“In this industry, each data inquiry raises issues—practical, personal, ethical, legal, and regulatory. We teach our students about HIPAA [Health Insurance Portability and Accountability Act] so they can appreciate whether and how patient data can be used. They take classes in epidemiology, which is the distribution of disease and illness, because it’s fundamental to predicting risk factors. They learn about pathology, diagnostic methods, treatment plans, and disease classification systems.”

Also, the health care industry needs people who can take the science of data into a conference room and speak English, not statistics, adds Andrew S. Gallan, an assistant professor in the Kellstadt Graduate School of Business: “At the end of the day, health care managers don’t want to hear about algorithms, methodologies, and model parameters: All they care about are insights.”

Gallan invites speakers to bring real-world perspective and actual data into the classroom. For example, Blue Cross Blue Shield lets students work on claims data to answer pressing, practical questions, such as “How can we help members be healthy? How can we reduce premiums and control the rising cost of healthcare by gaining insights into members’ behaviors?” Advocate Health shows students how data can be used to improve operational efficiency within medical centers. And Walgreens introduces them to clinical dashboards that help physicians track their compliance with care and reporting protocols.

Jobs, Jobs, Jobs

“Every guest speaker who comes to my class says ‘We have positions open,’ ” says Gallan. “While the industry generates tremendous volumes of data, only a small, small amount of it—maybe three or four percent—is analyzed in a deep way. So, our graduates are well positioned to do meaningful work and affect real change in health care delivery and outcomes.”

Lipps would agree. Last year, she was an intern at Press Ganey, helping analyze patient surveys for hospitals, medical groups, and other service providers. “In meetings I’d say to myself, ‘I know this!’  Because of the program, I was so well-prepared.” She recently started a job as senior healthcare informatics analyst with Village MD, a startup that provides analytics and operational support to primary care doctors.

Two 2016 graduates, Andrew Clark and David Thomas, chose the MS as a way to change careers. Clark, a former math teacher, says he was drawn to the health care concentration because he “wanted to be on the cutting edge of a new, growing industry where the likelihood of finding a satisfying, well-paying job is high.” Thomas agrees:  “Non-profit sectors are just beginning to see the value of analytics, and they don’t have enough talent.” He appreciated the cross-cultural, cross-pollination quality of the MS program, as well as having access to real data to work with: “That was really helpful.”

Gallan sums up the value of the MS in Predictive Analytics generally and the health care concentration specifically:  “I think our graduates get perspectives and, therefore, opportunities that they couldn’t get any other place in Chicago.”

DePaul University is one of the country’s most innovative schools, according to U.S. News & World Report. This story highlights one way that faculty innovate in research, in the classroom, and in the community.