Acknowledgements Understanding people's behaviour Communicating for change Working with groups Assessing change Teaching tools Check your facts Resource list

	Why evaluation is important
	Stages in evaluation
	Collecting information
	Activities Community mapping Structured group discussion Using checklists Using questionnaires

Why evaluation is important

In order to evaluate projects effectively, we need to ask questions and seek answers at different stages of the project:

Before the project starts
What is the situation like? What are people's priorities and what needs to be done? This is called a needs assessment or base-line survey.
 

Stages in evaluation

Evaluation starts in the planning phase of the project and continues throughout the programme. 

1. Assessing the situation before the project starts e. g. recording numbers of clinic visits and sexually transmitted infections, and running focus group discussions, listening surveys, etc.

2. Setting a broad project goal or aim e.g.
    

to increase the use of STD services among workers in a particular area and raise awareness of the need for safer sex

and clear and realistic objectives (targets) which can be measured e.g.
 

	to provide at least 10 condoms per month per adult and improve treatment for sexually transmitted infections
	to carry out educational and training activities.

3. Planning detailed activities to achieve the goal and the objectives 

4. Selecting simple monitoring methods (collecting information) to measure changes that have occurred as a result of your activities and check that targets are being achieved e. g. reviewing clinic records, setting up focus group discussions and interviews.

5. Doing an evaluation at the end of the project to explore if objectives have been achieved and carry out more in-depth activities to assess any overall changes.

6. Finding ways to involve people and feed the information back to the community.
    

Collecting information

Evaluation involves collecting and analysing information about the project. Most organisations are already collecting information about their project activities -and often collecting more information than can be used! Whatever methods you decide to use, think about how to:
 

	collect information systematically - record sheets and checklists are only useful if they are filled in regularly
	be selective - only collect information that refers specifically to what you want to measure
	ensure that the recording system is easy to use and that the person using it understands what it is for.

Information can be collected in a number of ways: 
 

	listening surveys and focus group discussions (see Part 1)
	community mapping
	through personal interviews
	by questionnaire
	through group discussions
	by observation
	through looking at records at clinics or pharmacies
	diaries and checklists kept by educators or health workers.

Information that can be measured in numbers is called quantitative. Information that concerns people's feelings or attitudes and cannot be measured in numbers is often called qualitative.

This information is sometimes more difficult to analyse but it provides a very useful guide to what people are really saying, thinking, feeling and doing.

The information can be collected by people working in the project as health workers or trainers, or by external researchers.

Activities in this section describe how to collect:
 

	baseline information using community mapping (See also the listening survey in Part 1)
	information about the process of the project using a structured group discussion and a checklist
	end-point information using a questionnaire.

Activity:  Community Mapping

Aim:       to gather information about what is happening in the community MATERIALS large sheets of paper and
              marker pens
Time:     1 day

Community maps are simple diagrams which show information about an area in a way that is easy to understand e. g the location of the clinic, the schools, the number of houses in an area, the beerhall, the bus terminus etc. They are prepared by community members and can be used to analyse the situation and the problem.

Example  Programme staff wanted to explore where people felt at risk of unsafe sex, pressure or violence. Small groups (young men, older men, young women, older women) were invited to draw maps of their community. The drawings showed places where people lived and worked as well as places such as bus stops, bars, hotels and other meeting places where risky or unsafe sexual behaviour can occur. For example, men listed places where they met in their leisure time, such as beer shops or bars, or special houses where they were able to meet women. Information was also gathered about places where condoms were available. The group then discussed what steps could be taken to ensure that condoms are made available at the local stalls, family planning clinic, bars and beer shops. They also realised that neither the women nor the men had easy access to treatment for sexually transmitted infections, and that school girls were being approached by men. The programme staff -decided to look at addressing these issues in more detail. At the end of the year the group looked again at the maps they had drawn earlier. Participants drew a new map showing any changes such as where condom outlets had been set up.

Activity:  Structured group discussion

Aim:            to assess whether the project activities are being carried out as planned (project monitoring or process evaluation) 
Materials:   large sheets of paper and marker pens
Time:          about half a day  

1. Prepare a few questions for the discussion:
    

	What are the main successes and problems that have been experienced during the project?
	What changes in attitudes or behaviour have been observed in the community since the project started?
	What are the possible barriers to change?
	How should the project develop?

2. Ask the participants to form groups of six to eight people.

3. Give each group a pen and a large sheet of paper.

4. Ask each group to discuss the questions and write their answers in order of importance.

5. Re-group with the other workshop participants, and display all the response sheets.

6. Discuss the similarities and differences in the large group. 

7. Reach agreement about the project priorities in either small groups or as the large group.

8. Make a final list which will be displayed in the project offices as a reminder of the discussion.
    

Activity:  Using Questionnaires for a survey

Aim:       to collect information for an end point evaluation 

Questionnaires are often used in surveys to collect information from a large number of people. If people are literate, then they can complete the questionnaire themselves. In non-literate communities, trained researchers will use the questionnaire to collect information through a personal interview. Which ever method is used, everyone must be asked the same questions in the same way so that the results can be analysed and compared.

Designing a questionnaire

Decide:
 

	what you want to find out
	who will collect the information
	from whom and how many people
	when you need to collect the information
	how the information will be collected (personal interviews or written questionnaires)
	how the information will be analysed
	what will be done with the information you have collected.

Then, think carefully about what information is needed, in relation to the aims and objectives of the project. 
 

	Keep questions brief and use simple language. A question is easiest to understand when it addresses one idea at a time.
	Use exact words which cannot be misunderstood to obtain precise answers and accurate information - ask How many times have you had diarrhoea in the last week? rather than How many times recently?
	Use words which are non-judgemental, such as person with HIV rather than AIDS victim, or more than one sexual partner rather than promiscuous.
	Use closed questions if a 'yes' or 'no' answer or a number is needed. For example Have you received any AIDS education at school?

	Open-ended questions require the respondent to give a longer answer in their own words. For example: What have you learnt at school about HIV and AIDS? Open-ended questions help to gather opinions that you may not have expected.
	Ask straight-forward questions rather than ones which threaten or make a person feel uncomfortable, such as: In your opinion, how do you feel about the treatment which you receive from health workers at this clinic rather than Do you think that the health worker at this clinic is polite and caring?
	Most importantly, keep the questionnaire short by avoiding unnecessary questions.

Before finalising the questions 
Always go through the questions (pre-test) with a small group of people who are similar to the people who will be interviewed. This ensures that the questions are easily understood and that people are willing to answer them - and that the interviewers know how to complete the questionnaires. The pre-test will also show whether the information collected is useful and can be easily analysed.

Carrying out the survey
 

	Choose interviewers who are trusted and with whom people can easily talk.
	Go through the questionnaire in a place where people feel comfortable and have privacy.
	Interviewers must be sure to explain why the survey is being conducted and what the findings will be used for.
	Ensure that respondents have time to answer questions
	Invite further comments at the end of the interview.