Introduction
This chapter describes the steps undertaken to conduct the formative evaluation.
A. Purpose of Report
The purpose of this report is to provide the results of a formative evaluation of the Office of the Procurement Ombudsman (OPO). The scope of the evaluation covers the Office's activities from 2008-09 to 2010-11.
B. Methodology
The formative evaluation was undertaken in two phases. The purpose of the first phase was to prepare an evaluation methodology report while the second phase consisted of a detailed formative evaluation. To prepare the evaluation methodology report, we:
- Undertook a preliminary document review to become familiar with the activities and intended outcomes of the Office of the Procurement Ombudsman.
- Reviewed the logic model contained in the Evaluation Framework developed by the Office of the Procurement Ombudsman.
- Undertook initial interviews with some representatives of OPO to discuss the logic model contained in the Evaluation Framework. Based on these discussions, we prepared a revised logic model for evaluation purposes that more accurately reflects the intended outcomes of the OPO.
- Reviewed the evaluation questions in the Evaluation Framework prepared by the OPO. We prepared revised evaluation questions to reflect the intended outcomes included in the revised logic model and reformulated some evaluation questions to reduce duplication and focus the formative evaluation.
- Identified the most appropriate indicators and sources of data to address the evaluation questions and issues.
- Prepared a revised evaluation matrix based on the results of the previous steps.
- Assessed the most appropriate evaluation design and specific methodologies that should be used to undertake a formative evaluation of the OPO.
- Prepared a data analysis plan to determine the methods to be used to analyze the evaluation data and how the data from multiple sources will be synthesized to address each evaluation issue.
- Prepared methodological tools and data collection instruments to undertake a formative evaluation of the OPO.
- Prepared a Methodology Report describing the methodology required to undertake a formative evaluation of the Office of the Procurement Ombudsman.
The second stage of the assignment consisted of the conducting the formative evaluation as specified in the Methodology Report developed in the first phase. The methodologies that we employed to conduct the formative evaluation are described in the following paragraphs.
1. Survey of Suppliers
A survey of suppliers that contacted the OPO was conducted using an online survey with follow-up telephone calls to increase the response rate for suppliers who did not complete the survey online. The population for the survey consisted of suppliers and supplier representatives that made an inquiry or complaint to the OPO during the period from 2008-09 to 2010-11. The data source that we used for the population of suppliers and supplier representatives is the comprehensive list of suppliers maintained by the OPO. This list consisted of a total of 548 suppliers, representatives of suppliers and supplier associations. However, some suppliers on this list were listed more than once and there were several suppliers for which there were no email addresses or other contact information such that we could not obtain their email addresses even by contacting the individuals by telephone. By removing these suppliers from the list, the total population of suppliers with complete email addresses and telephone numbers was reduced to 342. We sent out emails to all of the suppliers on this revised list. The email was written in both official languages and provided a link to the online survey questionnaire (which was also provided in both English and French) to the entire population of 342 suppliers. However, 67 of the email addresses were no longer in operation. We attempted to contact the suppliers with inoperable emails by telephone and were successful in obtaining new email addresses for 20 out of the 67 organizations. As a result, the total population of suppliers that were sent an email containing a link to the online survey questionnaire was 295.
A total of 99 surveys were initially completed online by the suppliers that were sent an email containing the link to the online survey. In addition, 16 suppliers refused to complete the survey. To increase the response rate, we sent a reminder email to non-respondents each week for three weeks. We also conducted follow-up telephone calls with non-respondents to encourage them to either complete the online survey or to provide their responses to the survey in a telephone interview with a representative of Ference Weicker & Company. At least six attempts of phone calls or emails were made before we stopped contacting the potential respondents. A total of 164 completed surveys were obtained from suppliers which results in a survey response rate of 56%. This response rate provides survey results at a satisfactory level of statistical significance (i.e. confidence interval of 5.1 with a confidence level of 95%).
2. Survey of Federal Government Officials
A survey of federal government officials was conducted using an online survey with follow-up telephone calls to increase the response rate. The population for the survey consisted of federal government officials who were involved in the procurement practice reviews conducted by the OPO or were involved in complaints, investigations or alternative dispute resolution processes undertaken by the OPO. By removing individuals that email addresses could not be found on the Government of Canada staff directory (GEDS) or by contacting individuals by telephone, this list consisted of a total population of 137 federal government officials. We sent out emails in French and English containing the link to the online survey questionnaire (provided in both official languages) to the entire population of 137 federal government officials. However, 25 of these emails were no longer in operation. We attempted to contact the federal government officials with inoperable emails by telephone and were successful in obtaining new email addresses for 10 of the 25 individuals. As a result, the total population of federal government officials that were sent an email containing the link to the online survey was 122.
A total of 42 surveys were initially completed online by the federal government officials that were sent an email containing the link to the online survey. In addition, three individuals refused to complete the survey. To increase the response rate, we sent a reminder email to non-respondents each week for three weeks. We also conducted follow-up telephone calls with non-respondents to encourage them to either complete the online survey or to provide their responses to the survey in a telephone interview with a representative of Ference Weicker & Company. Consequently, a total of 68 completed surveys were obtained from suppliers which results in a survey response rate of 56%. This response rate provides survey results at a satisfactory level of statistical significance (i.e. confidence interval of 7.9 with a confidence level of 95%).
Of the 68 federal government officials surveyed, 22 were from PWGSC while the remainder were from 46 other government departments and agencies. The rationale for the relatively large number of PWGSC officials surveyed compared to other government departments and agencies is that many of the procurement practice reviews impacted activities undertaken by PWGSC since the department is a common service organization for the procurement of goods and services for the Government of Canada.
3. Key Informant Interviews
We conducted key informant interviews with a total of 31 internal and external stakeholders in order to gather information on relevant evaluation questions. These interviews also helped contextualise findings obtained from other data collection methods. We conducted interviews with several categories of key informants. Table 1.1 indicates the different types of key informants and the number of key informants in each group that were interviewed. As indicated below, we would conduct interviews with federal government procurement officials and suppliers were involved in an investigation resulting from complaints received by the OPO as well as those who were involved in an ADR process. Where possible, we interviewed federal procurement officials and suppliers who were involved on same file to get both perspectives.
| Type of Key Informant | Number of Interviews |
|---|---|
| Current and former OPO staff and management | 5 |
| Federal government procurement officials involved in procurement practice reviews, investigations or ADR processes | 9 |
| Suppliers involved in OPO investigations or ADR processes | 5 |
| Senior federal government staff | 7 |
| Other stakeholders (professional associations, industry associations, etc.) | 5 |
| Total | 31 |
4. Document and File Review
We conducted a review of the available documents and administrative files necessary to address the evaluation issues pertaining to the relevance and performance of the OPO.
5. Literature Review
The literature review consisted of a review of the available information on the need for the OPO and the environment that the OPO operates in. It also included a review of the literature on best practices of government procurement organizations.
6. Review of Similar Organizations in Other Jurisdictions
We conducted a detailed literature and Internet review of similar organizations in other jurisdictions. We examined the activities undertaken by the most similar organizations in Canada and other countries including the Australia, New Zealand, the United Kingdom and other European countries. Based on the information compiled, we undertook a comparative analysis of the OPO with the organizations investigated.
C. Challenges and Limitations
Non-response error poses a challenge for the surveys undertaken as part of the formative evaluation. A non-response error exists when the characteristics of respondents that participate in the survey is significantly different from the characteristics of those which do not. The potential for non-response exists in this survey because potential respondents who chose not to participate in the survey could share common reasons for not participating, such as lack of reachability, indifference or disinterest in the evaluation, or a misunderstanding of the survey objectives.
To increase the response rate and reduce the potential for non-response errors, we carefully explained the purpose of the survey to the suppliers and federal government officials surveyed, provided them with multiple means to completing the survey, contacted suppliers at varying times and attempted up to six follow-up telephone calls or emails to encourage the suppliers and federal government officials to participate in the evaluation. In addition, we cross-tabulated the results by various characteristics of suppliers to assess how lower response rates from certain sub-groups may have affected the overall results.
D. Structure of the Report
The next chapter provides a brief profile of the Office of the Procurement Ombudsman. Chapter 3 provides the findings and conclusions regarding each of the evaluation questions used to conduct the formative evaluation. Copyright in the Material belongs to Canada © Her Majesty the Queen in right of Canada (2012).
- Date modified: