Not everything that is important can be
counted, and much that can be counted is not worth knowing. (USAID
Center for Development Information and Evaluation,1998)
While Indicators tell you how well you may
be doing against an arbitrary measurement, it does not tell you how to change
the process to make it better.
Introduction
An Indicator is "something that provides a "measure"
of some process or event and nothing more. An indicator can be a number such as
"10" or a descriptive word such as "good". But an indicator by itself carries no
more meaning.
Example: Indictor = 50
Ah, but you say, this is nice, we have an indicator of 50. Now
we know that it is suppose to mean something but what? In general, a number by
itself carries no meaning. What are we going to do with it" Rather what
can we do with it. It is a measure but in is not information. To give it a
meaning we must but a label to it....say, number of visits to our healthcare
organization...Again, you say, this is nice...we had 50 registrations in our
healthcare organization...and again, we ask our selves, what can we do with this
information...
What we need is more attributes given to the "Indicator" such
as:
- What time period
- What gender
- For what services
- Who provided that service
- How long were they in our healthcare clinic
- How sick where they
- What was the outcome of those visits
- How does his number compare to the same time period last
year
As we can see, without "metadata"...data about the data...then
there is not much we can do with "Indicators"....they are just a number that
indicate "something."
While they are often called "performance
measurements", again, these indicators are not very useful by themselves.
There are many different types of indicators, in fact, there
are organizations that create "indicators" and then "copyright"
them...interesting work!
A problem occurs when an indicator is defined in such broad
terms that it fails to provide a direction. These indicators are often
"composite" indicators.
One example of this type of indicator is used for the monitoring and
evaluation of "other people's performance." Examples, of
these indicators are standard accounting balance sheet indicators such as the
ratio of debt to equity. While they tell you where you are and how well you may
be doing in meeting your goals they do not tell you what to do about it....how
to use the indicator to Manage."
... their production does not by itself constitute an
analysis that helps select courses of action aimed at accomplishing specific
objectives. (PRICOR Methods Paper 1, 1985)
Indicator are NOT designed to be used by management in managing
for results.
It is not enough to ask "How well is the logistics system
functioning?" We must ask "How can we make the system better?"
When developing indicators, it must be very clear for what
purpose the indicators are to be used. While "the number of indicators used
should be the minimum necessary to "easy"...it is nothing more. It is not suitable for management.
History of Indicators in Health Care
In the past, before the Berlin Wall came
down and the end of the cold war, there was plenty of money to be had by
developing countries and research organizations in certain areas of the world.
Very little accounting for that money was required and very little was done to
see if the recipients actually did what they said they were going to do.
However, now that is not the case. These same donors are worried about their own
poverty, problems at home and their funding agencies’ budgets are often less now
than they were a few years ago. Now
these funding agencies have to justify to their governments why they are
giving away their money. Just as in business, donors now want their projects
to be successful and sustainable. Funding organizations are demanding that the
recipients of these funds prove that they are getting the results that they
have promised. They are demanding indictors to measure these results. In summary, funding
organizations are demanding cost-benefit analysis and accountability, just
like a business.
Without any previous way to measure these projects, many
organizations, such as USAID, have given top priority to developing methods to
accomplish this task. The buzzword in public health projects is
“Indicators.” This is
automatically tied to two other words, “Monitoring and Evaluating.”
Sometimes the word “management” will be thrown in to be such that all the
bases are covered.
However, while there is now a focus on "collecting" indicators
there is still no effort to develop Indicators for the management of the project
or organization itself...somewhere this has not seem important!
In the Health Facility Survey developed by WHO the
following criteria for indicators were given:
-
The total number of indicators should be
limited in number
-
Indicators should "flag" problems and
achievements, not provide a detail and comprehensive picture of
implementation.
-
Indicators should be measurable with
low-cost approaches...
-
Indicators should provide results that
are meaningful and easy to interpret...and
Indicators are developed by WHO and other
organizations not to manage but only to indicate wither an organization is
meeting its goals. They are general used by the measuring "Watch dog". not the
organization itself.
No where does USAID state (or even imply)
that the reasons for Indicators is that they are to be used to "Manage" an
organization; In fact, very few of the indicators used by USAID and it's
agencies can be used for Management.
However, giving USAID credit, they do state
that the indicators that they develop are for "our intended use"
in their measure of performance." What they and their project developers do not
seem to understand, is that these indicators cannot be used by the project
itself for management. What is consistently done is that these USAID indicators
are emphasized at the project level instead of developing good
management indicator for the project.
"Indicators must be actionable"
The first type can be defined
as something we are trying to measure directly. In other words, it
"closely tracks the result it is intended to measure" These
"indicators should either be widely accepted for use by specialists in a
relevant subject area, exhibit readily understandable face validity...or be
supported by a specific body of technical research" (USAID, 1998).
One example of the first indicator is given by the measurement of the variable "number of
visits." In this example, the variable means that an individual made x number of visits to the
organization or that the organization had a total of x visits. We can measure it directly but it does not mean anything
else or should it be taken to mean anything else.
It is recommended that a "Baseline Evaluation"
be performed in order to identify the current values of the indicators chosen.
However, it is also often stated that the reason for this baseline evaluation is
to "help plan programme strategies and address barriers and problems in
advance"...but how can a single indicator do that? If for example, we use the
previous example of "number of visits"....what does this mean?...is the value we
obtained from the measure "good" or "bad"...and what can do to influence its
value? The answer, of course, is that you must have other data hanging around
somewhere that the user uses for comparison...it is trends that are
interesting...not the value themselves.
This type of indicator may be used to used as a (1) baseline
evaluation and (2) at a follow up evaluation. However, it must be
understood, that the follow up evaluation is taken too late to be used for
management (except maybe by the project creators for their next project). If the
interventions were not going as intended, then no-one would know that until it
was too late!
The Health Facility Survey states:
If minimum standards of implementation are not met, then
it is unlikely that changes in the quality of care will be seen. In many
countries it is necessary to wait two or three years between
surveys conducted in the same area in order to see changes in
practices...[this is a joke surely!]
The second type is a variable that has
a relationship with another variable that we are trying to measure. It is
called an indirect or proxy indicator.
This second type of indicator is a bit fuzzy. It means that we know that
there is some form of relationship between variable "A" and
variable "B." Variable B is what we would like to measure but it is
either impossible to measure directly or very difficult to measure correctly.
This type of indicator may also be called a "substitute variable" or a
"process variable" and
we may describe how close the first variable measures the second, or how close the
relationship between variable A and variable B, its
"validity."
An example of this second indicator is given by measuring "the number
of family planning teaching sessions given by the staff to the client
(variable A), determines the number of births prevented by the client
(variable B)." How close a particular measurement of
variable A comes to the actual value of variable B is called is "reliability."
Sometimes a particular value of variable A may measure variable B exactly, and at
other times it may not.
The relationship between variable A and variable B may or may not be a
relationship that can be mathematically determined. What this implies is, if we do choose a "good"
variable A to measure variable B, this act along, says
that we will be unable to evaluate the success or failure of a project (i.e. if variable A has
no relationship to variable B, then we cannot determine anything about
variable B and we certainly cannot tell if the project changed variable B).
What this implicitly implies is that
we cannot always define Indicators before the project begins.
We may not know all the relationships, or if there is a relationship, how
strong that relationship is.
In some cases, we may discover that there is an "indicator" that was
unknown at the beginning of the project that better predicts the results than
that actually chosen.
By limiting projects to measure only those indicators that are defined at the beginning of the program, other
valuable variables (indicators) may go undetected.
A research organization was performing a surveillance that included over 100
variables. It appeared that some of these variables were inappropriate, such
as "what is the roof of your house made of." Now suppose that the
disease the surveillance was attempting to track came from a parasite that
resided in "that particular type" of material and that their
droppings was what was causing the disease. In this case "type of
roof" was a very important indicator, yet it would not have been
discovered until after the surveillance had started.
By measuring and using multiple variables instead of only one, interesting
relations between them may be discovered. Some variable may act to reinforce
each other, while another variable may decrease the effect of another
variable. In practice, combining these variables may made the difference
between success and failure. By measuring only one or a few variables this
information may never surface.
This implies that we should collect as much data as possible and in as much
detail as possible. Afterwards, if analysis does not show that there is a
relationship, then that particular variable may be dropped (but only for this
analysis-it may be significant for some other variable).
Subjective indicators should be avoided (except for the clients perception
of services). Instead objective variables (as described above) should be chosen
in order to "approximate" the information that we need. An example
is that of attempting to measure "Leadership." Many millions
of books have been written on the subject and a million more will probably be
written. In fact there is no single definition of leadership so it is
impossible to measure the quality itself. Give it up. Measure something else.
[My favorite theory on leadership was by Machiavelli...but I don't think today
his attitudes would win many accolades.]
Sufficient quantity to be useful. The number
of indicators collected should depend on the amount of information needed to
be able to use the information in managing. Example: Measuring the
number of caesarians births gives no information in regards to how to manage
this procedure. Additional information is needed, as an example, what are the
demographics of the population (percent of births), risks factors of the women
(why were was the caesarian performed?), and outcomes of the procedure (risk
to the patient due to this procedure)?
In many articles you read, "We must be prudent about
how much and what information we collect and use for decisions...more is not
always better." This may have been true in the past but it is not today.
Computers and data analysis programs make data collection and analysis cost
effective. However, the prime reason for the collect of more vs. less
information is that it may be impossible to determine in advance what data
will be important to the decision making process. It is always least costly
and more effective to collect data in the beginning than to go back and try to
recreate it later.
Health Care Organizations should move from the calculations of indicators to
actually using these indicators in managing.
Indicators and Computers
Computers are making obsolete former
systems of information collection. This is not only true in the use of computers for data collection, but also in the ability to
develop indicators (more of them and in greater detail), and reporting
techniques. A "few key indicators" are not sufficient for
management. | 
