Metric Maladies: The Disease of elevating Quantity over Quality

As Jerry Z. Muller notes in his book "The Tyranny of Metrics", we live in an age of "metric fixation". We use quantitative data to give us the assurance of success (or failure) because it simplifies what can be a bewildering sea of qualitative information and experience. But as helpful as some quantitative data can be at some times, it also can be deceptive. Over-reliance on quantitative data can become a poor replacement for the inherently messy, multi-causal process of discernment. Quantities can be mis-attributed, mis-assigned, and mis-counted to give decision makers a false assurance.

Meaning and/or Metrics? In place of the frequently quoted assumption that "whatever is not measured is not meaningful", we must reply with "what is measurable is meaningless". This is not a rejection of measurement, but an acknowledgment that it simply is not a source of value or meaning. This is because meaning and measurement are two separate categories that only partially overlap (if at all). Quantitative facts carry no values or meaning within them. Only persons construct values, and values largely come from the narratives they rely on to provide value to the facts of their lives. When we make meaning and value, we do not count things, we tell stories. And those who do try to count things for meaning-- whether it is the number of digits in their bank account, the number of awards on the mantle, or the number of likes on social media-- wind up profoundly empty and unhappy.

The statement "what is measurable is meaningless" suggests that raw data or quantitative measurements lack inherent meaning or value. These measurements gain significance only when people interpret them through the lens of their own values and narratives. Here are some significant examples:

Economic Growth (GDP):

• Measurement: Gross Domestic Product (GDP) measures the total value of goods and services produced in a country.
• Meaningless without context: A high GDP doesn't automatically equate to societal well-being. It doesn't account for income inequality, environmental impact, or other factors that contribute to quality of life.
• Values and narratives shape interpretation:
• Pro-growth narrative: A rising GDP signifies success, progress, and improved living standards.
• Environmentalist narrative: A rising GDP may indicate unsustainable resource consumption and environmental degradation.

Standardized Test Scores:

• Measurement: Tests like the SAT or ACT quantify a student's knowledge and skills in certain areas.
• Meaningless without context: A high score doesn't reflect a student's creativity, work ethic, or potential in fields not covered by the test.
• Values and narratives shape interpretation:
• Meritocratic narrative: High scores indicate intelligence and future success.
• Critical narrative: Test scores are biased, favoring privileged students and not measuring the full range of human potential.

Social Media Metrics (Likes, Shares):

• Measurement: These metrics quantify engagement with online content.
• Meaningless without context: High numbers don't necessarily mean content is valuable, informative, or accurate.
• Values and narratives shape interpretation:
• Popularity narrative: Lots of likes and shares equal social validation and influence.
• Critical narrative: These metrics can be manipulated, and viral content often prioritizes entertainment over substance.

Gaming the System: As a result, we find that "whatever can be measured can be gamed". We manipulate what is measured and how it is measured in order to provide a false sense of success or failure according to the values imposed on them. The phrase "whatever can be measured can be gamed" highlights the potential for manipulating metrics to achieve desired outcomes, even if those outcomes don't reflect genuine success or improvement. In addition to the ways that the metric collections above illustrate how metrics can be gamed for a false sense of success, here are other examples drawn from "The Tyranny of Metrics", as well as other research sources:

Healthcare:

• Measurement: Hospitals and doctors are often evaluated based on metrics like patient satisfaction surveys and readmission rates.
• Gaming the system: This can lead to doctors prioritizing quick fixes and avoiding complex cases to maintain high satisfaction scores and low readmission rates, potentially compromising patient care in the long run. Even the goal of patient death rates after surgery can (and have) been gamed: If there is a demerit for losing patients within 30 days after surgery, this can (and has) been solved by simply keeping patients artificially alive for 31 days.

Policing:

• Measurement: Police departments may be judged by arrest numbers or crime reduction statistics.
• Gaming the system: This can incentivize focusing on minor offenses or over-policing certain communities, while neglecting underlying issues like poverty and inequality that contribute to crime.

Scientific Research:

• Measurement: Scientists are under pressure to publish a high volume of papers in prestigious journals.
• Gaming the system: This can lead to "p-hacking" (manipulating data to achieve statistically significant results) or prioritizing flashy research topics over less publishable but potentially more impactful work.

Military:

• Measurement: During the Vietnam War, the U.S. military focused on "body counts" (number of enemy soldiers killed) as a measure of success.
• Gaming the system: This led to inflated body counts, misrepresentation of battlefield realities, and ultimately a flawed understanding of the war's progress.

Education:

• Measurement: Schools are often evaluated based on standardized test scores.
• Gaming the system: Teachers may "teach to the test," narrowing the curriculum and neglecting critical thinking skills or creativity, in order to boost scores.

Business:

• Measurement: Companies may rely heavily on metrics like quarterly earnings reports to gauge performance.
• Gaming the system: This can lead to short-term thinking, sacrificing long-term investments or innovation or even safety to meet immediate financial targets.

These examples illustrate how an overemphasis on metrics can distort priorities, incentivize harmful behavior, and undermine the very goals the metrics were intended to measure. It's crucial to recognize the limitations of metrics, use them thoughtfully, and prioritize qualitative factors alongside quantitative data to gain a more comprehensive understanding of complex situations.

Metric obsession and over reliance on quantitative data often falls into the following traps:

• Prioritizing Easily Measurable Over Important: We often focus on what's easy to quantify, neglecting aspects that truly matter but are harder to measure.
• Oversimplifying Complex Realities: Reducing multifaceted goals and responsibilities to a few metrics can lead to misleading results.
• Focusing on Inputs, Not Outcomes: Measuring resources spent rather than results achieved provides a distorted view of success.
• Sacrificing Information Quality for Standardization: Simplifying data for comparison often strips away context and nuance, leading to misleading conclusions.
• Gaming Metrics by Choosing Easier Targets: Selecting simpler cases or clients to boost metrics artificially inflates success rates.
• Lowering Standards to Improve Numbers: Easing criteria or definitions allows for improved metrics without genuine progress.
• Manipulating Data through Omission or Distortion: Excluding or misrepresenting data points creates a false impression of performance.
• Outright Cheating: When the stakes are high, individuals or organizations may resort to outright fraud to achieve desired metric outcomes.

Overall, we find that quantitative numbers and measurements are tools, not ends; indicators, not goals. They can provide valuable information, but their meaning is always shaped by the values and narratives we apply to them. Metrics may be more applicable to assess certain tasks, such as material production, and almost useless for assessing other tasks, such as social or spiritual experience. It's essential to critically evaluate metrics, considering their limitations and the potential for manipulation, to ensure they are serving our true goals and values.