Simpson's Paradox

From the brilliant education blog Are We Doing Anything Today? comes No Dentist Left Behind.

My dentist is great! He sends me reminders so I don’t forget checkups. He uses the latest techniques based on research. He never hurts me, and I’ve got all my teeth. When I ran into him the other day, I was eager to see if he’d heard about the new state program. I knew he’d think it was great.

“Did you hear about the new state program to measure effectiveness of dentists with their young patients?” I said.

“No,” he said. He didn’t seem too thrilled. “How will they do that?”

“It’s quite simple,” I said. “They will just count the number of cavities each patient has at age 10, 14, and 18 and average that to determine a dentist’s rating. Dentists will be rated as excellent, good, average, below average, and unsatisfactory. That way parents will know which are the best dentists. The plan will also encourage the less effective dentists to get better,” I said. “Poor dentists who don’t improve could lose their licenses to practice.”

“That’s terrible,” he said.

“What? That’s not a good attitude,” I said. “Don’t you think we should try to improve children’s dental health in this state?”

“Sure I do,” he said, “but that’s not a fair way to determine who is practicing good dentistry.”

“Why not?” I said. “It makes perfect sense to me.”

“Well, it’s so obvious,” he said. “Don’t you see that dentists don’t all work with the same clientele, and that much depends on things we can’t control? For example, I work in a rural area with a high percentage of patients from deprived homes, while some of my colleagues work in upper middle-class neighborhoods. Many of the parents I work with don’t bring their children to see me until there is some kind of problem, and I don’t get to do much preventive work. Also, many of the parents I serve let their kids eat way too much candy from an early age, unlike more educated parents who understand the relationship between sugar and decay. To top it all off, so many of my clients have well waterwhich is untreated and has no fluoride in it. Do you have any idea how much difference early use of fluoride can make?”

“It sounds like you’re making excuses,” I said. “I can’t believe that you, my dentist, would be so defensive. After all, you do a great job, and you needn’t fear a little accountability.”

“I am not being defensive!” he said. “My best patients are as good as anyone’s, my work is as good as anyone’s, but my average cavity count is going to be higher than a lot of other dentists because I chose to work where I am needed most.”

“Don’t get touchy,” I said.

“Touchy?” he said. His face had turned red, and from the way he was clenching and unclenching his jaws, I was afraid he was going to damage his teeth. “Try furious! In a system like this, I will end up being rated average, below average, or worse. The few educated patients I have who see these ratings may believe this so-called rating is an actual measure of my ability and proficiency as a dentist. They may leave me, and I’ll be left with only the most needy patients. And my cavity average score will get even worse. On top of that, how will I attract good dental hygienists and other excellent dentists to my practice if it is labeled below average?”

“I think you are overreacting,” I said. “‘Complaining, excuse-making and stonewalling won’t improve dental health’… I am quoting from a leading member of the DOC,” I noted.

“What’s the DOC?” he asked. “It’s the Dental Oversight Committee,” I said, “a group made up of mostly lay persons to make sure dentistry in this state gets improved”

“Spare me,” he said, “I can’t believe this. Reasonable people won’t buy it,” he said hopefully.

The program sounded reasonable to me, so I asked, “How else would you measure good dentistry?”

“Come watch me work,” he said. “Observe my processes.”

“That’s too complicated, expensive and time-consuming,” I said. “Cavities are the bottom line, and you can’t argue with the bottom line. It’s an absolute measure.”

“That’s what I’m afraid my parents and prospective patients will think This can’t be happening,” he said despairingly.

“Now, now,” I said, “don’t despair. The state will help you some.”

“How?” he asked.

“If you receive a poor rating, they’ll send a dentist who is rated excellent to help straighten you out,” I said brightly.

“You mean,” he said, “they’ll send a dentist with a wealthy clientele to show me how to work on severe juvenile dental problems with which I have probably had much more experience? BIG HELP!”

“There you go again,” I said. “You aren’t acting professionally at all.”

“You don’t get it,” he said. “Doing this would be like grading schools and teachers on an average score made on a test of children’s progress with no regard to influences outside the school, the home, the community served and stuff like that. Why would they do something so unfair to dentists? No one would ever think of doing that to schools.”

I just shook my head sadly, but he had brightened. I’m going to write my representatives and senators,” he said. “I’ll use the school analogy. Surely they will see the point.”

He walked off with that look of hope mixed with fear and suppressed anger that I, a teacher, see in the mirror so often lately.