AIDS action Issue 32 Page 1 2
Issue 32 March - May 1996
All of us involved in HIV prevention and sexual health promotion, or in caring for people with AIDS, want to know how useful our work is. This issue of AIDS Action looks at simple ways to collect information about our work and its effect on people, help plan future activities and adapt to changing situations.
Monitoring and evaluation
The words planning, monitoring and evaluation can sound technical and off-putting. Often projects are evaluated by outsiders in order to justify continued funding or political support and this can make an evaluation very threatening.
However, monitoring and evaluating do not need to be difficult. If monitoring is built into daily activities, and if evaluation is carried out regularly with the involvement of the target audience, projects become more rewarding and effective.
Both monitoring and evaluation are necessary for project management and planning.
Monitoring means collecting information and keeping records about activities to check whether the work is being carried out as planned and to assess reactions of people receiving the services or involved in the project.
Evaluation means asking 'Did we achieve what we set out to do?' and comparing the present situation with the past in order to find out to what extent original purposes have been achieved. People working in community-based organisations often have a very good idea of what they are doing and how well it is being done. But it is very important for them to review their work regularly in order to answer two key questions:
are project activities achieving what they set out to do?
is the project making a real and positive difference to people's lives?
It is also important for projects to be planned and monitored well so that senior managers or funders understand how the aims of the project are being achieved. This makes it easier for staff to participate in external evaluations and develop proposals for funding.
Community participation is essential, but with whom and how?
Participation means consulting everyone who is affected by the work - staff, clients and the wider community - in the process of deciding what the project aims to do, how it is done and how its effect can be measured.
Most projects aim to improve a particular situation for a particular group or community. It is essential to involve people in identifying their concerns and problems, deciding which of these concerns are most important and possible to do, and what activities can best achieve this. Participatory approaches are only possible when senior managers and decision makers are willing to accept the opinions of the people affected at community level. However, it is impossible to involve everybody all the time!
Asking the right questions
It is useful to think about the following points to help decide who should be involved at which stage:
What do you need to know and whose views and experiences are necessary when planning your project or making it more effective? The more directly affected a person is, the more important their opinion. It is necessary to involve those who often have less influence, such as people who are sick, young women or men, or those who are discriminated against by their community, such as injecting drug users or sex workers.
Whose active support is essential for the success of the project, such as funders, managers, or leaders in the community? Try to include them in the process as much as possible.
Who is likely to feel threatened by the possibility of change? Think about how you can gain their support or challenge their opposition if necessary.
Who takes decisions about the activities and who will be expected to act on the decisions? They need to be involved, especially in planning new activities that will need more resources or training to implement.
All of us!
Communities are made up of different people with many different concerns and priorities. People may see themselves as belonging to several different groups - defined by their family, religion, race, workplace or age for example.
Within any community some groups of people are discriminated against. Women often have lower status and power in decision making. In considering your project's impact on women and their role, changes in the following areas may be useful to record:
numbers of women staff and volunteers on the project
roles of women and men in decision making and skilled work in project and community activities
numbers of women who are literate, and who receive primary and secondary education compared to men
roles of women and girls at home and in the community - for example, often it is women and girls who care for sick people and children
possibilities for employment for women and men locally
women's and men's views on ability to negotiate safe sexual behaviour in relationships
level of reported and unreported domestic violence
legal status of women, for example in terms of inheritance and divorce
numbers of unmarried and married women with access to sexual and reproductive health information and services
women's views on the acceptability of attending sexual health services
women's income levels and access to earning potential, such as land rights or credit facilities. It is useful to talk with women to ensure activities respond to their needs.
The diagram below shows the different stages in a project's life, using an imaginary example where staff at a health centre decide to develop a new HIV/STD project with the local community.
What do you want to achieve? This is the overall purpose or aim of the project.
Find out what the situation is using baseline surveys, needs assessments and priority setting exercises. Issues to survey in any project include HIV/STDs, factors increasing risk such as male or female migration and poverty, condom availability and sexual activity and risk.
How are you going to achieve the aim? These are sometimes called objectives and refer to broad strategies that can be measured over time.
What specific activities will achieve these objectives? Each activity needs to have a timeframe, a budget (including volunteer costs) and staff time allocated.
What results do you want to see from these activities? These are sometimes referred to as outcomes and must aim to fulfill your objectives. They can be numerical targets (e. g. number of people to be trained in six months)
How will you measure or monitor how far activities are achieving objectives? You need to choose markers or indicators which will show any changes that occur. These suggest, but do not prove, how far an objective has been achieved.
How will you use this information for planning? Regular review of information will help you see whether activities are going as planned and within budget. You may decide to stop the activity, change it, or start again. Occasional in-depth evaluations are also needed, followed by review of aims, objectives and activities. The planning process then starts again.
1a. The new project aims to:
improve sexual health among potentially vulnerable groups
strengthen their coping strategies for reducing the spread of STD/HIV
2a. Staff prepare a baseline survey looking at
clinic STD figures
STDs treated by traditional healers, private doctors and pharmacists
deaths from HIV/AIDS, where known
clinic attenders' assessment of service
3a. Staff decide that the project can achieve its aims by:
improving STD care for women
providing sexual and reproductive health education in clinic
promoting and providing condoms
4a. The clinic staff plan to:
train staff on HIV/AIDS/STDs
provide STD education and treatment for women and men
improve links with hospital e. g. training, drug supply and laboratory tests
5a. In the clinic, activities will result in:
better STD diagnosis and treatment
increased client knowledge about HIV/STDs and satisfaction with services
increased contact tracing
6a. Clinic indicators could include:
numbers of women seeking STD treatment
numbers of women with STDs at ante-natal clinic
numbers of condoms distributed
reported client satisfaction and reduced stigma
7a. Tools include:
medical supervisor checklist of counselling and diagnostic skills
focus group discussions with clinic attenders every six months
8a. At regular project meetings staff look at records, discuss any difficulties and review progress and possible improvements.
An evaluation includes surveys, interviews and record analysis for comparison with the baseline survey. They are used for planning future activities. The results are discussed internally and, with advisory group, written up for funders and national managers.
2b. Staff set up meetings with groups including clinic attenders, women, religious leaders, teachers, traditional midwives/healers and young people, to discuss the current situation and prioritise key concerns. An advisory group of representatives is selected to meet regularly with project staff at every stage, and co-ordinate community activities.
3b. With staff support, the advisory group decide on additional objectives e. g.
improve economic situation, especially of women
encourage better communication, especially between young men and women
4b. The advisory group plan to:
select and train community educators
improve condom access via shops, pharmacies and community educators
have community discussions about STDs and sexual relationships
set up income generation schemes
5b. In the community, results will include:
improved communication between men and women about sexual matters
increased use of condoms
increased actions to reduce poverty
6b. Community indicators could include:
reported domestic violence/arguments
condom sales in bars
establishment of women's credit schemes
more reports of sexual responsibility and willingness to use condoms
7b. Tools include:
community questionnaire to ask about condom use, STDS etc.
observation of condoms displayed in pharmacies, shops and bars
focus group discussions and community meetings
8b. Advisory group attends project meetings and discuss any issues with community.
Results are fed back to community meetings where the activities are assessed and the exercise of setting priorities is repeated.
Most organisations are already collecting information about their activities - and often collecting more information than can be used! The important points to remember are:
collect information systematically - record sheets are not useful if they are not filled in regularly
be selective - only collect information that refers specifically to the ways you have decided to measure whether the activity is going as planned
make sure that the recording system is easy to use and that the person using it understands and is committed to the process
make sure that the information is collected by the person most involved in the activity. For example. supervisors who visit village health clinics are in the best position to do regular checks on drug supplies. Young peer educators are often the best people to fill in questionnaires with participants in a peer education session.
What to collect
Information that can be measured in numbers is often called quantitative. This means asking questions about when, who, which, how much or how often. Information about people's feelings or attitudes that cannot be measured in numbers is often called qualitative.
Both of these methods are needed to be able to explore reasons for both expected and unexpected results and to examine some of the possible causes.
There are a variety of simple tools for collecting information:
records, for example of purchase and distribution of supplies, number of peer educators trained or community meetings held
questionnaires or surveys, used to gather a broad range of answers to specific questions
diaries, kept by participants in the activity
group discussion of case studies or events, such as clinic staff analysing why a child was not referred to hospital in time
observing in the community, looking at events or people's behaviour and recording these observations
focus group discussions where people with specialist knowledge or common interests discuss a specific topic in depth
interviews with groups or individuals to collect views on a particular issue, such as whether people with HIV feel that their quality of care is improved following a training programme for their carers.
After collecting information it is important to check your conclusions with the people involved, before planning further activities. A monitoring system can cost money and must be included in the budget. For example, training and supervising, printing forms and staff time all cost money. Community and volunteers also have their own costs in time and energy.
Finding out about feelings
Monitoring and evaluation can be especially difficult for projects working on sensitive issues such as sexuality and relationships, or challenging views about how men and women behave. It is hard to explore changes in people's sexual behaviour, values or attitudes. People may not want to share personal and sensitive information or may not feel able to say what they really think or do.
However, changes in public behaviour may be influenced by the same factors that affect private behaviour and these are easier to observe. For example, activities aiming to increase young women's confidence in negotiating sexual relationships can be monitored by recording whether young women speak out more in meetings and by asking them what they feel about their confidence levels.
Focus group discussions
A focus group discussion (FGD) involves talking about a particular topic in detail with a small group of people who share a common experience or area of interest, and are often of the same sex and similar age. Holding a series of FGDs on the same topic or activity at different stages of a project can show how thoughts, feelings and reported practices are changing. Once a topic for discussion has been decided and the objectives for the group set, a question guide should be drawn up by project staff. The questions need to be general and open-ended to let people talk about their views and concerns.
For example, a question guide for an FGD which aims to find out the needs of women carers might include:
One or two introductory questions to help people feel at ease and start a general discussion on the topic, such as: When someone is sick in your family, who usually takes care of them? What are the advantages? What are the difficulties?
Two to four general questions to encourage people to talk about the main issues in their own words, for example asking what they did yesterday to help the sick person in the family and checking details, such as what tasks they did at different times of the day.
More detailed questions to find out responses to particular ideas or suggestions, such as: How do you feel about taking care of someone who is sick at home? What is the most difficult thing about taking care of someone with AIDS? Where can you go for help if it is hard to care for someone alone? These questions should pick up key issues raised earlier. It is sometimes useful to have prepared a picture or a written list of questions to hand out.
Finally it is important to ask a general question such as: Are there any other comments that you would like to make about taking care of someone who is sick at home?
A facilitator leads the FGD, encourages people to talk freely and ensures that the conversation does not stray too far from the question guide. Another person records the key points of the conversation.
Ideally a group should be between 6 and 12 people to allow everyone to participate. A discussion should not normally last more than an hour and a half. It is important to check that every one agrees with what is being recorded and hears or sees the final report.
A questionnaire or survey can be useful for collecting information from a larger number of people. Everyone is asked the same questions so that the results can be analysed and compared.
Designing the questionnaire
Decide what you want to find out and from whom, and think about what questions are needed to obtain this information. Closed questions are those that only need a 'yes' or 'no' answer or a number, for example: Have you received any AIDS education at school? Try not to use questions which lead to a particular answer, such as Do you agree that the home care programme should visit you every month? Open-ended questions require the respondent to give a longer answer in their own words, for example: What have you learnt at school about HIV and AIDS? It is useful to include some open-ended questions to gather opinions that you may not have expected.
Keep questions brief and use simple language. A question is easiest to understand when it addresses only one idea. Use exact words which cannot be misunderstood to obtain precise answers, such as: How many times in the last week? It is useful to collect only information which can be remembered accurately. For example, instead of How many times did you feel sick last year? ask How many times have you been to the clinic in the past four weeks?
Use words which are non-judgemental, such as person with HIV rather than AIDS victim, or more than one sexual partner rather than promiscuous.
Most importantly, keep the questionnaire short by avoiding unnecessary questions. If doing the questionnaire takes longer than 15 minutes then people may become restless.
Preparation The questionnaire should be checked (or pre-tested) with a small group of people who are similar to those who will be interviewed. This ensures that the questions are easy to understand, that people are willing to answer them and that the interviewers know how to fill it in correctly. The pre-test will also show whether the information collected is relevant and can be easily analysed.
All in a day's work
Community care groups in Zambia find that recording their work helps them to plan better.
In 1995 Chikankata Hospital in Zambia began supporting the work of small groups, called Care and Prevention Teams (CPTs) which provide care and support to people affected by HIV.
Starting with the community
At community meetings people discuss ways in which HIV affects the village - such as increasing numbers of deaths and people with AIDS without close relatives to care for them. Villagers suggest possible solutions and elect CPT members. CPTs include a range of people such as headmen. church leaders, teachers, women's club representatives, traditional healers and midwives. Community health workers and community counsellors are key members of the CPTs. The CPT's role is to develop and co-ordinate activities. Firstly they review and summarise needs identified at community meetings. For example, when thinking about how to support sick people in the community the group will analyse their needs by drawing a picture of a sick person and then drawing the person's needs on the picture. These may include food, clothes, clean environment, medicine, nursing care, transport to hospital and spiritual comfort and friendship.
They then decide on appropriate activities, what each activity should achieve and how to assess it.
Chikankata encourages each group to keep simple records of their activities. This is important for maintaining motivation and remembering why activities were prioritised and what they were intended to achieve. Groups are encouraged to measure quantitative changes, such as noting numbers of condoms requested, and qualitative ones. For example, one of the problems commonly identified is high levels of alcohol drinking by men, which is thought to lead to unsafe sex, domestic arguments and spending of money needed for the family. After discussing the issue with the community, some groups have chosen to record numbers of disturbances caused by drunkenness and court hearings for marital disputes.
Each CPT decides how to monitor their own work. They form an administrative committee including someone who can read and write. CPT members develop a list of areas where they record changes observed by villagers, such as monthly village meetings to discuss HIV prevention, numbers seeking STD treatment, infection control practices used by traditional healers and midwives, numbers of unplanned pregnancies, alcohol-related disturbances, ritual cleansing methods used after death and whether people are providing nursing care or support to children. Traditional healers, for example, are members of the CPT and their client registers provide useful information. Traditional village courts report numbers of cases related to HIV to the CPT.
The Chikankata team visit the CPTs regularly and ask them how they feel about the work. If they feel that they are not doing as well as expected they discuss why this might be so. If they feel that they are doing better than expected they explore what has helped them achieve this.
Daphetone Siame and Simon Mutonyi, AIDS Management and Training Services, Chikankata Hospital, Private Bag S. 2, Mazabuka, Zambia
Learning all together
Students and teachers can see the changes when they become involved in planning their own AIDS programmes.
GAPA-Bahia is an NGO working in a very poor region of Brazil. One of its projects, started in 1992, aims to help young people decide how they can have a healthy sexual life that is pleasurable and safe, as well as to encourage acceptance of and support for people living with HIV and AIDS.
Planning in groups
Teachers and young people have been consulted in designing the project and deciding how to measure its success. This has made it easier to adapt the project and make it useful. First, information was collected from both teachers and students about issues such as sexuality, death, fear, prejudice and how men and women should, and do, behave. Methods included discussions and drama sessions, where teachers and students role-played characters such as an activist with HIV, an uncooperative headmaster. shocked mother or rebellious student.
This information helped in planning the curriculum and teacher training. The project staff provide training in curriculum activities for the teachers and meet them each month for support and supervision. The supervisors ask the teachers the following questions: What teaching activities have you done? What made you decide to do this activity? What did you think about the content and the type of activity? Did this activity lead to further activities? How could you improve the teaching materials? Comments are recorded and are referred to when planning future activities.
From the beginning the project has made use of daily record-keeping activities which have been used for project planning and review. The variety of methods used have made monitoring activities enjoyable, for example an exercise called 'the day's impressions'. This involves both teachers and students drawing pictures with words each evening at home to express their personal feelings about the day, the issues covered and what impact they think the activity might have on their lives.
Review sessions involve trainers, teachers and some students who discuss issues such as: What helped you to introduce this work in school? What has made the work difficult to do? How can these difficulties be overcome?
One activity which helps planning involves writing all the project activities on large pieces of paper and sticking them on the wall. Supervisors, teachers and students then write their opinions about the activities and pin them to the sheets. Everyone has a chance to share their ideas for discussion and this helps prioritise the important issues. Sessions are also held with groups of young people where teaching materials and a students' magazine are developed.
In order to generate support from parents and to involve young people who are not at school and other people in the community, the project organises 'mobilisation days' for parents, community members, headmasters, priests and local health workers. People who attend the day are invited to participate in some activities themselves. Small discussions and workshops are held to allow people to express their ideas about the project.
The most useful thing about including regular and simple monitoring in the project is that the work can be adapted easily. People suggest positive changes which can be acted on. They contribute to the activities at all stages, and are not just 'recipients' of information.
GAPA-BA, CEP 40.155-260, Salvador, Bahia, Brazil.
AIDS action Issue 32 7 Page 8
Letter / Resources
Home visits for HIV and TB
I liked the section on community-based treatment in AA30 (page 11). Our hospital serves about half a million people. Community outreach teams provide support to people with TB and HIV in their homes. Services for HIV and TB are integrated to share limited transport and to enable each team to work in specific areas. We find that HIV becomes less stigmatised as people accept that more and more individuals have TB and HIV.
Once a person is diagnosed with TB and is well enough to receive their treatment at home, then he or she chooses a supervisor. This is any responsible person who lives close by and is willing to supervise the treatment - for example, a shop-keeper, traditional healer, teacher or nurse. The person's family may also be involved in making the decision. The supervisor is trained by the hospital team and community health workers. Problems are discussed with the community health worker and the hospital team which makes a monthly visit to each supervisor. The team goes to the person's home where treatment, preventive therapy for close contacts and health education are discussed with the patient and his or her family. Counselling and training in caring for the sick is given where appropriate. Patients are introduced to our other services and asked to return to the hospital for sputum smear tests when necessary.
In order to co-ordinate services we have weekly meetings for all people involved - community doctors, matrons, clinic sisters, community health facilitators and community psychiatric nurses. The AIDS Action Team also includes elected community representatives.
N P Mcanyana and Dr P Haselau, Ngwelezune Hospital, KwaZulu Natal, South Africa
Resources for planning and monitoring
Community action on HIV is a resource manual for NGOs, containing guidelines on how to plan and evaluate HIV prevention programmes.
Available free plus postage from HIDNA, c/o ACFOA, Private Bag 3, Deakin ACT 2600, Australia.
Community HIV prevention handbook covers the key steps for action planning with communities.
For more details contact UNAIDS, 1211 Geneva 27, Switzerland.
HIV/AIDS project planning manual for NGOs is a step-by-step guide for NGOs with little or no experience in project planning and development.
Available free from UNDPl Regional Project on HIV/AIDS, 55 Lodi Estate, New Delhi 110 003, India.
Manual of group interview techniques to assess the needs of people with AIDS gives guidelines on running group interviews to plan HIV care programmes.
Available free from WHO Documentation Centre, 1211 Geneva 27, Switzerland.
Partners in evaluation is a practical handbook for community-based programmes on monitoring and evaluation with local communities.
Available for £3.20 from TALC, PO Box 49, St. Albans, AL4AX, UK.
Self-evaluation is a simple guide for rural community development programmes.
Available for US$5.00 plus postage from World Neighbours, 4127 NW 122 Street, Oklahoma City, OK 73 112, USA.
Toolkits is a practical guide to planning, monitoring and evaluation for development workers.
Available for £6.95 from Publications Sales, Save the Children Fund, 17 Grove Lane, London SE5 8RD, UK.
Stepping stones is a training package on HIV/AIDS, gender, relationships and communication skills, for running workshops with community members.
Available for £85.00 (4 manuals 1 video), £47.50 ( 1 manual 1 video) or £12.50 (manual only) from TALC.
Contact TALC for details of Swahili, Luganda and French language editions. Limited numbers available free to organisations in sub-Saharan Africa on written request to TALC.
Facing the challenges of HIV/ADS/STDs: a gender-based response contains an outline of the issues that affect women and men, and practical training exercises.
Available free from KIT Mauritskade 63, Amsterdam 1092 AD, The Netherlands.
Executive editor Nel Druce
Assistant editor Sian Long
Design and production Ingrid Emsden
Editorial advisory group Calle Almedal, Kathy Attawell, Nina Castillo, Professor E M Essien, Dr Sam Kalibala, Ashok Row Kavi, Dr Ute Küpper, Professor Keith MacAdam, Dr Tuti Parwati Merati, Dr Claudia Garcia Moreno, Dr Chandra Mouli, Dr Anthony Pinching, Dr Peter Poore, Barbara Wallace, Dr Michael Wolff
Special issue with thanks to Peter Ssebanja, TASO, Uganda - Hillary Hughes, Sexual Health Project, IPPF - Louisa Gosling, Save the Children Found, UK
Publishing partners ABIA (Brazil) Colectivo Sol (Mexico) ENDA (Senegal) HAIN (the Philippines) Consultants based at University Eduardo Mondlane (Mozambique)
AHRTAG's AIDS programme is supported by CAFOD, Charity Projects, Christian Aid, FINNIDA, HIVOS, ICCO, Memisa Medicus Mundi, Misereor, Norwegian Red Cross, ODA, Oxfam, Save the Children Fund, SIDA, UNICEF and WHO/GPA.
The International Newsletter on AIDS Prevention and Care
This English edition of AIDS action was produced and distributed by Healthlink Worldwide.
Healthlink Worldwide works in partnership with organisations in developing countries to improve the health and well-being of poor and vulnerable communities by strengthening the provision, use and impact of information.
Reproducing articles and images
Healthlink Worldwide encourages the reproduction of articles in this newsletter for non-profit making and educational uses. Please clearly credit AIDS Action/Healthlink Worldwide as the source and send us a copy of the reprinted article and any uses made of the material. Permission to reproduce images must be obtained from the photographer/artist or organisation as shown in the credit. Contact details are available from Healthlink Worldwide.
Search over 20,000 health and disability resources...
...using the online bibliographic database of Source International Information Support Centre. Access is free of charge. www.asksource.info
Healthlink Worldwide is a partner in Source International Information Support Centre.