Evaluation Finding and Conclusions

Document Navigation for "Evaluation Finding and Conclusions" Previous Page Table of Contents

This chapter summarizes the evaluation findings and conclusions regarding each of the evaluation questions employed to undertake a formative evaluation of the Office of the Procurement Ombudsman.

A. Relevance

This section provides evaluation findings and conclusions for the evaluation questions dealing with the need for and relevance of the OPO.

Evaluation Question #1: To what extent is there an ongoing need for the OPO?

As indicated in Figure 3.1, the suppliers and federal government officials surveyed indicated an ongoing need for the OPO. When asked to indicate to what extent there is an ongoing need for the OPO, on a scale of 1 to 5 where 1 is not at all, 3 is somewhat and 5 is to a great extent, 72% of suppliers indicated that the need for the OPO was significant or to a great extent while an additional 10% indicated that there was somewhat of a need. The average response obtained from the suppliers surveyed was 4.1 out of 5. When federal government officials were asked the same question, the average response obtained was 3.4. The average response obtained from the key informants interviewed is 3.7 which is between the average response of the suppliers and the federal government officials surveyed.

Figure 3.1: Extent of the Ongoing Need for the OPO

	Federal Government Officials (N=68 )	Suppliers (N=164)
1 - Not at all	2	15
2 - Minimal	10	4
3 - Somewhat	24	17
4 - Significant	20	29
5 - Great extent	9	89
N/A	3	10

The majority of the suppliers surveyed stated that there is a tremendous need for the services provided by the OPO because it plays a crucial role in maintaining the credibility and fairness of the procurement process in the federal government. Not only does the OPO help in providing independent investigation in matters arising from procurement and resolving issues so that the supplier does not have to engage a lawyer, it also enables suppliers to provide feedback on the procurement practices of the federal government departments for future improvements to the procurement process. The two most frequent reasons given by suppliers why there exists a need for the OPO are to act as an independent party to ensure fairness and avoid abuses in federal procurement as well as to help small businesses by acting as an impartial referee and dealing with issues regarding contract award and administration. Another reason mentioned frequently by suppliers is that the OPO provides another option to resolve issues because small businesses cannot afford to hire a lawyer to deal with issues resulting from federal government procurement activities. A few respondents stated that they need an alternative to the Canadian International Trade Tribunal (CITT) as a method of resolving disputes with the federal government because it is too costly for small and medium sized businesses.

3.1: Most Frequent Responses Provided by Suppliers Regarding the Need for the OPO

Question: To what extent is there an ongoing need for the Office of the Procurement Ombudsman, on a scale of 1 to 5 where 1 is not at all, 3 is somewhat and 5 is to a great extent?

Suppliers (# provided comments = 134)

Note: excluding 10 suppliers who did not provide a rating and 20 who did not provide any comment for their rating

Those who provided higher rating

Act as an independent third party to ensure fairness and prevent abuses in federal government procurement (40)
Help small business by acting as an impartial referee to deal with issues regarding contract award and administration (22)
Provide another option to resolve issues because small business cannot afford to hire a lawyer (11)
Need an alternative to CIISD because it is too costly for small and medium sized businesses (2)

Those who provided lower rating

OPO does not have sufficient power to bring real change to government procurement practices (4)
OPO not useful because it can only handle very small contracts (3)
OPO not useful because of limited jurisdiction (3)

As indicated in Figure 3.1, 12% of the suppliers surveyed stated that the need for the OPO was "minimal" or "not at all". One of the most frequent reasons why these respondents did not see a substantial need for the OPO is that they did not feel that the OPO had sufficient power to bring change to federal government procurement (i.e. does not have authority amongst government departments to pursue corrective actions). Several respondents stated that the mandate of the OPO is too narrow to be of value to small and medium sized businesses because the OPO can only review complaints with respect to the award of a contract for the acquisition of goods below the value of $25,000 and services below the value of $100,000. Some other suppliers indicated that the jurisdiction of the OPO is too limited and needs to be expanded to be able to addresses a wider range of issues experienced by suppliers (e.g. prior to contract award, standing offers, etc.).

As indicated in Figure 3.1, 29 or 43% of federal government officials indicated that the need for the OPO was significant or to a great extent while an additional 24 or 35% indicated that there was somewhat of a need. The two most frequent reasons given by these federal government officials why there is an ongoing need for the OPO are to provide an independent and centralized venue for suppliers involved in small contracts to have their complaints addressed and to undertake procurement practice reviews to enhance federal government procurement practices. Some other federal government officials stated that the OPO is needed to ensure transparency, fairness and integrity in government procurement while others indicated that the OPO provides a useful review and challenge mechanism regarding government procurement.

As indicated in Figure 3.1, 12 or 18% of the federal government officials surveyed stated that the need for the OPO was "minimal" or "not at all". The most frequent reasons why these respondents did not see a substantial need for the OPO are that there are already multiple oversight bodies (e.g. OAG, internal audit) that examine procurement and that the procurement practice reviews of the OPO do not provide substantive recommendations. Some other respondents stated that there are already multiple complaint mechanisms for vendors (e.g. CIISD, federal court, PWGSC, departmental processes) while others indicated that the need for the OPO is not significant because federal government procurement is already open and transparent.

3.2: Most Frequent Responses by Federal Government Officials Regarding the Need for the OPO

Question: To what extent is there an ongoing need for the Office of the Procurement Ombudsman, on a scale of 1 to 5 where 1 is not at all, 3 is somewhat and 5 is to a great extent?

Federal Government Officials (# provided comments = 61)

Note: excluding 3 officials who claimed they are not in position to provide a rating and 4 who did not provide any comment for their rating

Those who provided higher rating

Provide an independent and centralized venue to address complaints by suppliers for small contracts (23)
Need to undertake procurement practice reviews to enhance procurement practices (15)
Ensure transparency, fairness and integrity in government procurement (8)
Provide a review and challenge mechanism regarding government procurement (3)

Those who provided lower rating

Already multiple oversight bodies (e.g. OAG, internal audit) that examine procurement (6)
Procurement practice reviews do not provide substantive recommendations (3)
Already multiple complaint mechanisms for vendors (e.g. CIISD, federal court, PWGSC, departmental processes) (2)
Federal government procurement is already open and transparent (2)

Almost all of the key informants interviewed indicated a great need for the OPO. Some of the most frequent reasons they indicated that the OPO is needed is to address the complaints of suppliers, particularly the small contracts that do not fall within the mandate of the CIISD and to identify areas for improvement in departmental procurement practices.

Based on a document review, the OPO is needed because it fills a gap in handling supplier complaints below CIISD's threshold which is a minimum of $25,000 for the award of a contract for goods and $76,500 for the award of a contract for services. Prior to the establishment of the OPO, the options for suppliers to deal with issues related to the award of contracts lower than the minimum threshold of the CIISD included hiring a lawyer, dealing with the department directly or accepting the contract award decision.

Conclusion: There is an ongoing need for the OPO in order to provide an independent body to address the complaints of suppliers and to identify areas of improvement in federal government procurement practices.

Evaluation Question #2: To what extent are the objectives and activities of the Office of the Procurement Ombudsman aligned with federal government priorities?

In 2003, an interdepartmental Task Force was put in place to carry out a government-wide review of procurement. The Task Force made recommendations for a procurement system that demonstrates transparency, accountability and fiscal responsibility. One of the Task Force's recommendations was "the government should consider the potential need for a body to provide advice and oversight to provide increased transparency on the management of the government's procurement process and insight on evolution in the procurement community of practice". It was the Task Force report and the 2006 Federal Accountability Act, and subsequent amendments to the Department of Public Works and Government Services Act that established federal government priorities regarding the need for openness, transparency and accountability of federal government procurement. The Department of Public Works and Government Services Act stipulates that the Governor in Council may appoint a Procurement Ombudsman for a term of not more than five years and that the Procurement Ombudsman shall, in accordance with the regulations:

Review the practices of departments for acquiring material and services to assess their fairness, openness and transparency and make any appropriate recommendations to the relevant department for the improvement of those practices;
Review any complaint respecting the compliance with any regulations made under the Financial Administration Act of the award of a contract for the acquisition of materiel or services by a department to which the Agreement, as defined in section 2 of the Agreement on Internal Trade Implementation Act, would apply if the value of the contract were not less than the amount referred to in article 502 of that Agreement;
Review any complaint respecting the administration of a contract for the acquisition of materiel or services by a department; and
Ensure that an alternative dispute resolution (ADR) process is provided, on request of each party to such a contract.

In 2007, the Government announced the creation of the OPO and stated that OPO's work will enhance procurement practices across government by ensuring that the highest standards of ethical conduct are followed and increasing Canadians' confidence in the accountability and transparency of federal procurement activities. The Office of the Procurement Ombudsman became fully operational after the Procurement Ombudsman Regulations came into effect in May 2008.

The stated objective of the OPO is to improve the fairness, openness and transparency in federal government procurement. The activities undertaken by the OPO include the investigation of supplier complaints, provision of alternative dispute resolution services and the review of procurement practices of government department and agencies. This objective and activities of the OPO are aligned with the federal government priorities of increased openness, transparency and accountability.

The key informants interviewed were asked to indicate to what extent are the objectives and activities of the Office of the Procurement Ombudsman aligned with federal government priorities, on a scale of 1 to 5 where 1 is not at all, 3 is somewhat and 5 is to a great extent. The average rating of the key informants is 4.0 out of 5 which indicates that they feel that the OPO objectives and activities are aligned to a significant extent with federal government priorities. Several key informants stated that the investigation of supplier complaints contributes to the federal government priorities of openness and transparency and that the procurement practice reviews contribute to increased accountability of federal government procurement.

Conclusion: The objectives and activities of the Office of the Procurement Ombudsman are aligned with federal government priorities.

Evaluation Question #3: To what extent are the activities of the Office of the Procurement Ombudsman aligned with federal roles and responsibilities?

The Federal Accountability Act and Action Plan brought forward specific measures to help strengthen accountability and increase transparency and oversight in government operations. These measures included steps to clean up procurement of government contracts by enshrining in law a commitment to fairness, transparency, and openness in the procurement process, and by appointing an independent Procurement Ombudsman to provide additional oversight of the procurement process. In addition, the Government of Canada committed to strengthen and streamline its management policies and consult with stakeholders on reducing barriers that inhibit access to government, including a review of its procurement and financial management policies to identify where they could be streamlined, and where the Government could replace transaction-level requirements and rules with basic principles of management accountability and transparency. The objective of the OPO is to improve the fairness, openness and transparency in federal government procurement as well as the activities undertaken by the OPO to investigate supplier complaints, provide alternative dispute resolution services and review government procurement practices are aligned with the federal government roles and responsibilities as outlined in the Federal Accountability Act and Action Plan.

The key informants interviewed were asked to indicate to what extent are the objectives and activities of the Office of the Procurement Ombudsman aligned with federal roles and responsibilities, on a scale of 1 to 5 where 1 is not at all, 3 is somewhat and 5 is to a great extent. The average rating of the key informants is 4.6 out of 5 indicating that they feel the OPO objectives and activities are aligned to a great extent with federal government priorities.

Conclusion: The activities of the Office of the Procurement Ombudsman are aligned with federal roles and responsibilities.

Evaluation Question #4: Do the activities of the Office of the Procurement Ombudsman complement or overlap those of other programs?

A review of the available documentation indicates that the OPO complements other government programs. OPO's primary mandate is to investigate supplier complaints, provide alternative dispute resolution services and review the procurement practices of government department and agencies. The Department of Public Works and Government Services Act specifies that the OPO will review complaints below the the Agreement on Internal Trade (AIT) thresholds of $25,000 for goods and $100,000 for services. The Canadian International Trade Tribunal (CITT) is a quasi-judicial institution and has authority to inquire into complaints by potential suppliers concerning procurement by the federal government that is covered by the North American Free Trade Agreement (NAFTA), the Agreement on Internal Trade (AIT), the World Trade Organization (WTO) Agreement on Government Procurement (AGP), the Canada-Chile Free Trade Agreement (CCFTA), and any other applicable trade agreement. The minimum dollar value of service contracts handled by the CITT is $25,000 for goods contracts (AIT minimum threshold) and $76,500 for service contracts (NAFTA minimum threshold) which is, for the most part, greater than the maximum dollar value of contracts that can be handled by the OPO (i.e. the award of a contract for the acquisition of goods below the value of $25,000 and services below the value of $100,000). Consequently, there is a small overlap in the mandate of the OPO and the CITT with regard to services valued between $76,500 and $100,000. To deal with this overlap, the OPO checks with the CITT for complaints in the range between $76,500 and $100,000 and does not deal with the complaint if it is being addressed by the CITT.

Some differences between the CITT and the OPO are indicated below:

The CITT does not handle complaints regarding contract administration while the OPO can investigate complaints dealing with contract administration at any dollar value.
Suppliers may file a complaint to the CITT before contract award whereas complaints to the OPO may be only filed after contract award.
CITT can recommend cancelation of a contract while the OPO cannot.
CITT does not address systemic procurement issues while the mandate of the OPO includes undertaking procurement practice reviews to address systemic issues regarding federal government procurement.
CITT provides does not provide alternative dispute resolution services such as mediation and facilitation such as that provided by the OPO.

The mandate of the Office of Small and Medium Enterprises (OSME) administered by PWGSC is to support SMEs by working to reduce barriers and by simplifying requirements for SMEs that want to do business with the Government of Canada. The stated mission of OSME is to:

Encourage and assist small and medium enterprises (SMEs) to participate in the federal government procurement process;
Improve the links between supply and demand and influencing change within government acquisitions; and
Conduct economic analysis of Government of Canada procurement and the private sector.

OSME's mandate does not include the investigation of supplier complaints such as that undertaken by the OPO.

The role of the Office of the Auditor General of Canada (OAG) is to hold the federal government accountable for its stewardship of public funds. The OAG conducts performance audits of federal departments and agencies, annual financial audits of the government's financial statements, and special examinations and annual financial audits of Crown corporations. In addition, the OAG has a specific environmental and sustainable development mandate, established through amendments to the Auditor General Act. As a result, OAG has a much broader role than the OPO.

When key informants were asked if the activities of the OPO complement or overlap those of other programs (e.g. CITT, OSME, OAG), 35% of the key informants stated that OPO activities both complement and overlap other programs while 35% of respondents stated that they complement other programs (Figure 3.2). Most respondents stated that the OPO complements the CITT because CITT handles larger value contracts (i.e. over $76,500) while the OPO handles smaller dollar value contracts (i.e. the award of a contract for the acquisition of goods below the value of $25,000 and services below the value of $100,000). A few respondents noted the small overlap between the mandate of CITT and OPO because the OPO handles complaints regarding the award of contracts for services for less than $100,000 while the CITT's mandate is to handle contracts for goods and services over $76,500. Some respondents noted that OPO complements the CITT because the CITT does not handle complaints regarding contract administration while the OPO can investigate complaints dealing with contract administration at any dollar value when the issue has to do with obligations set out in the contract (for example, timelines to be met, documents to be provided, and payment terms).

Figure 3.2 Relationship of OPO Activities with Other Programs

	Key informants (N=31)
Complement	11
Overlap	2
Complement and Overlap	11
Don't know	7

Most key informants stated that the OPO complements OSME because OSME does not have a formal process for dealing with supplier complaints and that the role of OSME is an advocate for small and medium sized businesses which is quite different than OPO's role as an independent neutral body.

The most frequent responses regarding the organization that the OPO overlaps with the most is the OAG. Some respondents stated that both organizations have looked at similar topics relating to government procurement. However, most key informants stated that the OAG has a broader scope by undertaking high level audits and doesn't focus solely on procurement. A few respondents stated that if we just relied on the OAG, there would not be enough focus to improve the procurement process. A few respondents indicated that there is a small overlap between their departmental internal audit function and OPO. However, other respondents indicated that internal audit departments examine a much broader range of topics than just procurement.

Conclusion: The activities of the Office of the Procurement Ombudsman complement and do not overlap other federal government programs.

B. Performance - Achievement of Outcomes

This section provides evaluation findings and conclusions for the evaluation questions dealing with the extent to which the OPO has achieved its intended outcomes.

Evaluation Question #5: To what extent has the OPO been able to address the complaints received regarding contract award or administration?

During the three year period from 2008-09 to 2010/11, the OPO received a total of 335 complaints regarding contract award or administration (Figure 3.3). Complaints regarding contract award accounted for about 72% of the total complaints while the remainder consisted of complaints regarding contract administration. During this time OPO also received numerous general procurement related inquiries.

Figure 3.3: Number and Type of Complaints Received by the OPO

	Complaint about Contract Award (Total = 241)	Complaint about Contract Administration (Total = 94)
2008-09	71	39
2009/10	89	32
2010/11	81	23

OPO's approach in handling contractual complaints is prescribed by the Procurement Ombudsman Regulations, which provide the parameters for the Office's activities. In instances where the Office receives a complaint filed in accordance with the criteria established in the Regulations (i.e. filed in writing), the complaint is assessed by a team of OPO procurement experts and a senior legal advisor to ensure adherence to all prescribed criteria of the Regulations.

Most of the complaints received by the OPO do not meet all of the criteria prescribed in the Procurement Ombudsman Regulations. The most common regulatory criteria not met, which resulted in the complaint being outside the OPO's jurisdiction, include the following:

The service was not covered by the Agreement on Internal Trade (for example, Aboriginal set-aside programs);
The complaint dealt with a standing offer or supply arrangement; or
The contract was above OPO's dollar value thresholds ($25,000 for goods and $100,000 for services)

Despite issues falling outside the OPO prescribed regulatory criteria, the OPO has strived to help address these complaints. As an illustration, with the written permission of the complainant, the OPO submits the complaint to the attention of the deputy head of the federal organization involved. OPO staff also work with the federal organization involved to try and help resolve the complaint. The team of procurement investigators may suggest other courses of action for consideration, including encouraging departments to actively participate in addressing supplier concerns through dialogue and providing suppliers with information and assistance to increase their knowledge and understanding of the Government of Canada's procurement processes.

In instances where the OPO receives a complaint filed in accordance with the Procurement Ombudsman Regulations, which meets all of the prescribed criteria, the OPO undertakes an investigation of each complaint. As indicated in Table 3.3, the OPO has undertaken nine investigations of complaints from 2008-09 to 2010/11. Most of these investigations concluded that the complaint was not substantiated. In most instances where the complaint was not substantiated, other observations were noted in the procurement process which led to some provided recommendations for future corrective action to be taken by the department involved.

Table 3.3: Number of OPO Investigations of Supplier Complaints from 2008-09 to 2010/11

Fiscal Year	Number	Outcome
2008-09	1	Complaint was not substantiated but recommendations provided for future corrective action
2009/10	5	Complaint was not substantiated but recommendations provided for future corrective action (4) Complaint withdrawn by supplier (1)
2010/11	3	Department took corrective action (1) Recommendations provided for future corrective action (1) Complaint was not substantiated but recommendations provided for future corrective action (1)
Total	9

In order to obtain feedback from suppliers who have dealt with the OPO, we conducted a survey of all suppliers and supplier representatives that made an inquiry or complaint to the OPO during the period from 2008-09 to 2010-11. The data source that we used for the population of suppliers and supplier representatives was the comprehensive list of suppliers maintained by the OPO. By removing the suppliers that were listed more than once and those for which there were no email addresses or other contact information, the total population of suppliers that were sent an email containing a link to the online survey questionnaire was 295. A total of 164 completed surveys were obtained from these suppliers which results in a survey response rate of 56%.

Of the suppliers that responded to the survey conducted as part of the formative evaluation, the most frequent purpose of the contact made by the supplier with the OPO was to complain about a contract award which accounted for the purpose of contact for 62 or 38% of the total number of 164 suppliers surveyed (Figure 3.4). The second most frequent reason why the suppliers surveyed contacted the OPO was regarding a complaint about contract administration (29% of suppliers). The third most frequent purpose of the contact made by the supplier with the OPO was an inquiry (17% of suppliers). The other reasons why suppliers contacted the OPO included a variety of matters including those outside of the mandate of the OPO (e.g. complaints about large contracts, concerns about standing offers, etc.).

Figure 3.4: Purpose of Contact with the OPO by the Suppliers Surveyed

	Suppliers (N=164)
Don't know	3
Other	24
Inquiry	27
Complaint about contract administration	47
Complaint about contract award	63

The suppliers surveyed have divergent opinions regarding the extent to which their complaints and inquires have been addressed. When asked to indicate the extent to which the OPO was able to address their complaint or inquiry, on a scale of 1 to 5 where 1 is not at all, 3 is somewhat and 5 is to a great extent, the average rating was 2.3 for suppliers that made a complaint about a contract award, 2.7 for suppliers that made a complaint about contract administration and 3.0 for suppliers who made an inquiry to the OPO. As indicated in Figure 3.5, the responses varied considerably within each of these three sub-groups of suppliers.

Figure 3.5: Suppliers Responses Regarding the Extent to Which Their Complaints and Inquiries Have Been Addressed by the OPO

	Supplier complaints about contract award (N=63)	Supplier complaints about contract administration (N=47)	Supplier inquiries (N=27)
1 - Not at all	22	15	6
2 - Minimal	6	7	3
3 - Somewhat	11	7	4
4 - Significant	2	3	6
5 - Great extent	8	10	5
N/A	14	5	3

As indicated in Figure 3.5, 50 or 45% of the suppliers surveyed that made a complaint to the OPO about a contract award or administration stated that the OPO was not able to address their complaint at all or to a minimal extent. The most frequent reason given by suppliers for their relatively low ratings of the extent to which their complaint was addressed is that the matter was outside the jurisdiction of the OPO (e.g. contract larger than the maximum value that can be handled by the OPO, dealt with issues related to standing offers, etc.). The second most frequent reason given for their relatively low rating is that the OPO found that the supplier had no claim. Some less frequent responses why suppliers provided a low rating regarding the extent to which their complaint was addressed are that the OPO did not have the authority to provide a solution; the OPO was slow to act; they were never advised of the results of the OPO investigation; and the response obtained from the OPO seemed to favour the government department.

3.4: Most Frequent Responses by Suppliers Regarding the Handling of their Complaints Regarding Contract Award and Contract Administration by the OPO

Question: To what extent was the OPO able to address your complaint, on a scale of 1 to 5 where 1 is not at all, 3 is somewhat and 5 is to a greater extent?

Suppliers complained about contract award or contract administration (# provided comments = 71)
Note: excluding 19 suppliers who did not respond and 20 that did not provide any comment for their rating

Those who provided higher rating

Dealt with the complaint in an open and pragmatic manner and resolved the issue (16)
Even though the matter was outside their jurisdiction, the OPO helped to resolve the matter (4)

Those who provided lower rating

Matter was outside the jurisdiction of the OPO (27)
OPO did an investigation and found that the supplier had no claim or did not resolve the issue (12)
OPO did not have the authority to provide a solution (4)
Slow to act or were never advised of the results of the OPO investigation (4)
Response seemed to favour the government department (3)

As indicated in the Figure 3.5, 23 or 21% of the suppliers that made a complaint to the OPO about a contract award or administration indicated that the OPO was able to address their complaint to a significant or great extent while an additional 18 or 16% of suppliers indicated that the OPO was somewhat able to address their complaint. The most frequent reasons given by suppliers for their higher ratings regarding the extent to which their complaint about a contract award or contract administration was addressed is the OPO dealt with the complaint in an open and pragmatic manner. The second most frequent reason for the high rating is that even though the matter was outside their jurisdiction, the OPO helped to resolve the matter (e.g. by sending a letter to the Deputy Minister of the department).

As indicated in the Figure 3.5, 11 or 41% of the suppliers that made an inquiry indicated that the OPO was able to address their inquiry to a significant or great extent while an additional 4 or 15% of suppliers indicated that the OPO was somewhat able to address their inquiry. As shown in Figure 3.5, the most frequent reason given by suppliers for their high ratings is that the OPO answered their questions very well. Some respondents also stated that OPO staff were very responsive and helpful.

As indicated in Figure 3.5, 9 or 33% of the suppliers surveyed that made an inquiry indicated that the OPO was not able to address their inquiry at all or to a minimal extent. The most frequent reason given by suppliers for their relatively low ratings of the extent to which their inquiry was addressed is that the information received from the OPO did not help or nothing happened as a result of their inquiry. Some other suppliers stated that they did not receive any new information that they didn't already know. A few suppliers indicated that their low rating was due to the fact that the OPO could not be of any assistance because the matter was outside the jurisdiction of the OPO.

3.5: Most Frequent Responses by Suppliers Regarding the Handling of their Inquiries by the OPO

Question: To what extent was the OPO able to address your inquiry, on a scale of 1 to 5 where 1 is not at all, 3 is somewhat and 5 is to a greater extent?

Suppliers contacted OPO for inquiries (# provided comments = 21)
Note: excluding 3 suppliers who did not respond and 3 that did not provide any comment for their rating

Those who provided higher rating

Answered their questions very well (10)
Were very responsive and helpful (2)

Those who provided lower rating

Information received did not help/nothing happened (4)
Did not receive any new information (3)
Matter was outside the jurisdiction of the OPO (3)

A further analysis of the survey data was undertaken that focussed on those complaints filed in accordance with the Regulations (i.e. filed in writing). This analysis resulted in the survey of a sample of 9 suppliers whose complaints met all criteria of the Regulations and a sample of 24 suppliers whose complaints did not meet all criteria of the Regulations.

The average rating provided by suppliers regarding the extent to which their complaint was addressed was 2.6 for the sample of 9 suppliers whose complaints met all the criteria of the Regulations and 3.2 for the sample of 24 suppliers whose complaints did not meet all criteria of the Regulations resulting in an overall overage of 3.0 for all 33 complaints included in the analysis.

Of the sample of 9 suppliers whose complaints did meet all criteria in the Regulations, 5 or 56% of these suppliers indicated that their complaints were addressed or somewhat addressed (i.e. rating of 3 or higher). The most frequent reasons for the lower ratings given by the suppliers are they felt the OPO defended the government position, and that the OPO did not have sufficient powers to ensure compliance by the department.

Of the sample of 24 suppliers whose complaints did not meet all criteria of the Regulations, 16 or 67% of the suppliers indicated that their complaints were addressed or somewhat addressed (i.e. rating of 3 or higher). The most frequent reasons for the lower ratings given by suppliers is the matter was outside the jurisdiction of the OPO; the OPO indicated that they did not have a valid claim; the OPO intervened but the outcome remained the same; and the OPO had no power to require a department to resolve the issue.

Conclusion: According to the suppliers surveyed, the majority of the complaints and inquiries received by the OPO have been addressed to some extent. However, a considerable proportion of the complaints and inquiries received from suppliers are outside of the mandate of the OPO.

Evaluation Question #6: To what extent has the provision of alternative dispute resolution (ADR) processes been able to facilitate settlement of contractual disputes?

Pursuant to the Department of Public Works and Government Services Act and the ARCHIVED - Procurement Ombudsman Regulations, the Procurement Ombudsman ensures that an ADR process is provided to resolve contractual disputes with respect to the interpretation or application of the terms and conditions of a contract. In order to facilitate a fair and quick resolution of contractual disputes, the Office has ensured that facilitation capabilities are available internally, with minimal or no charge to the parties. The ADR services are organized along a continuum where at one end, the options afford the parties control over the process and outcome (facilitation and mediation) and the other end, the parties are provided a written decision (non-binding arbitration) on the likelihood of success should one party engage in court action. Both parties must agree to participate in the ADR process before it is commenced.

A total of 10 requests for ADR processes were received by the OPO during the three year period from 2008/9 to 2010/11. Five of these requests resulted in an ADR process being conducted and a settlement was reached; the request for an ADR process was withdrawn in four instances and for one of the requests, the matter was unresolved. As a result, a settlement was reached or the request was withdrawn in 9 of the 10 ADR processes undertaken by the OPO.

In addition, there were eight requests for an ADR process by a supplier but the department involved declined to participate in the ADR process. As a result, the ADR process did not proceed. The number of instances where the department declined (8) to participate in an ADR process was almost equal to the total number of ADR processes (10) undertaken by the OPO. There is no consequence or impact on the department for not participating in the ADR process.

When key informants were asked to indicate to what extent the provision of ADR processes had been able to facilitate settlement of contractual disputes, on a scale of 1 to 5 where 1 is not at all, 3 is somewhat and 5 is to a great extent, the average rating is 3.9. However, most key informants were not very familiar with the ADR processes as only 9 respondents provided a rating. The low response rate could be due to the fact that most of the key informants surveyed were not directly involved in an ADR process. Some respondents indicated that that the suppliers and departments that have participated in an ADR process have been able to arrive at a facilitated resolution of the complaint. Some respondents indicated that there have been several instances where the applicable department has declined to participate in the ADR processes because the department stated that they had their own internal ADR process. A few respondents indicated that it should not be an option for a department to refuse to participate in an ADR process initiated by the OPO.

Conclusion: The provision of alternative dispute resolution (ADR) processes by the OPO has been able to facilitate a settlement or result in the withdrawal of the contractual dispute in almost all instances. However, almost one half of the requests for alternative dispute resolution processes by suppliers have not proceeded because the government departments involved have declined to participate in the ADR processes of the OPO since they had their own processes as well.

Evaluation Question #7: To what extent has the OPO been independent, neutral and helpful with regard to federal government procurement?

The Office of the Procurement Ombudsman is part of the portfolio of the Minister of Public Works and Government Services Canada, but operates at arms-length from the department. The Procurement Ombudsman submits an annual report to the Minister of PWGSC who is required to table it in Parliament. To ensure independence of the OPO from government departments and agencies, the following steps have been undertaken:

A Memorandum of Understanding has been signed by the Ombudsman and the Deputy Minister of PWGSC to clearly identify the roles and responsibilities of the OPO and the Department: it clearly recognizes and states the independence of the Ombudsman from the Department;
The budget for the OPO was approved directly by the Treasury Board, separately from the departmental budgeting process;
PWGSC and OPO have negotiated service-level agreements for human resources, financial and information technology services – these services are provided on a fee for service basis; and
The OPO has retained its own legal advisor and is responsible for its own internal audits, risk management and communication services.

The key informants interviewed were asked to what the OPO has been independent, neutral and helpful with regard to federal government procurement, on a scale of 1 to 5 where 1 is not at all, 3 is somewhat and 5 is to a great extent. The average rating was 3.7 out of 5 (17 respondents) which indicates that the key informants feel that the OPO has been independent, neutral and helpful with regard to federal government procurement to a considerable extent. Several respondents, particularly federal government officials, stated that the OPO is viewed as independent and neutral and that the OPO has been helpful because their procurement practice reviews and investigations have highlighted areas for improvement in government procurement practices.

Conclusion: The OPO has been independent, neutral and helpful with regard to federal procurement.

Evaluation Question #8: To what extent has there been acceptance of the recommendations resulting from OPO procurement practice reviews and investigations resulting from complaints received by the OPO?

In 2011, the OPO followed up with 16 federal government departments and agencies to determine the extent to which these organizations have assessed the recommendations contained in the first five OPO procurement practices reviews (i.e. published in August 2009) and developed plans to improve their existing practices. During the follow-up analysis undertaken by the OPO, many of these organizations provided comprehensive action plans and reported on significant changes already implemented to improve their procurement practices. The majority of the planned actions were close to completion. The follow-up report noted considerable progress in introducing rigour in new and previously established senior procurement review committees, which play an important role in the broader procurement management framework. Many of the introduced changes reflect an increased focus on risk-based decision-making and also on plans to assess the effectiveness of their methods for managing procurement risks.

As part of the formative evaluation, federal government officials were asked to indicate to what extent there has been acceptance by their organization of the recommendations resulting from OPO procurement practice reviews and investigations resulting from complaints received by the OPO, on a scale of 1 to 5 where 1 is not at all, 3 is somewhat and 5 is to a great extent. The average of the responses received is 3.8 which indicates that OPO recommendations have been accepted to a significant extent by the federal government organizations surveyed. As indicated in Figure 3.6, 33 or 49% of the federal government officials surveyed indicated that the OPO recommendations have been accepted to a significant or great extent while an additional 13 or 19% of the federal government officials indicated that OPO recommendations have been somewhat accepted by their organization.

Figure 3.6: Extent to Which Recommendations of OPO Procurement Practice Reviews and Investigations Resulting from Complaints have been Accepted

	Federal Government Officials (N=68)
1 - Not at all	0
2 - Minimal	7
3 - Somewhat	13
4 - Significant	17
5 - Great extent	16
N/A	15

When asked to provide the rationale for their relatively high rating of the extent to which the OPO recommendations have been accepted, the most frequent response of federal government officials is that they have no issues with the recommendations of the procurement practice reviews they have been involved with. The second most frequent response is that they have not only accepted the recommendations but have already made improvements to their business processes as a result of the recommendations. A few respondents stated that they have agreed with most but not all of the OPO recommendations.

As indicated in Figure 3.6, 7 or 10% of the federal government officials surveyed indicated that the OPO recommendations have been accepted to a minimal extent while an additional 13 or 19% of the federal government officials indicated that OPO recommendations have been somewhat accepted by their organization. When asked to provide the rationale for their relatively low rating of the extent to which the OPO recommendations have been accepted, the most frequent response is that some of the recommendations are not valid or useful. Some other frequent responses are that they do not agree with the recommendations; the recommendations have not resulted in any significant improvements in their processes; and they have some issues with the quality of OPO reports (e.g. recommendations are not substantiated).

3.6: Most Frequent Responses by Federal Government Officials Regarding the Extent to which the Recommendations of OPO have been Accepted

Question: To what extent has there been acceptance by your organization of the recommendations resulting from OPO procurement practice reviews and investigations resulting from complaints received by the OPO, on a scale of 1 to 5 where 1 is not at all, 3 is somewhat and 5 is to a great extent?

Federal Government Officials (# provided comments = 50)
Note: excluding 15 officials who did not provide a rating and 3 did not provide any comment for their rating

Those who provided higher rating

Accepted the recommendations/have no issues with the recommendations (22)
Have made changes as a result of recommendations (15)
Agreed with most of the recommendations (6)

Those who provided lower rating

Some recommendations are not valid/useful (5)
Do not agree with recommendations (4)
Recommendations have not resulted in any significant improvement in their processes (4)
Some issues with the quality of OPO reports (3)

Conclusion: The recommendations resulting from OPO procurement practice reviews and investigations have been accepted to a significant extent by government departments.

Evaluation Question #9: To what extent have the fairness, openness and transparency of departmental procurement practices improved as a result of the OPO?

As indicated in the logic model provided in the previous chapter, one of the intended outcomes of the OPO is as follows: Improved departmental procurement practices with regard to fairness, openness and transparency. This intended outcome is shown as an intermediate outcome rather than an immediate outcome. Since the OPO has only recently been established, it is expected that it will have had some impact in achieving its immediate outcomes but not its intermediate outcomes.

When the federal government officials were asked to indicate the extent the fairness, openness and transparency of the procurement practices of their organization has improved as a result of the OPO, the average rating of the responses received is 2.4, on a scale of 1 to 5 where 1 is not at all, 3 is somewhat and 5 is to a great extent. As indicated in Figure 3.7, 32 or 47% of the federal government officials surveyed indicated that the fairness, openness and transparency of their department's procurement practices have not improved at all or to a minimal extent.

Figure 3.7: Extent to Which the Fairness, Openness and Transparency of the Department's Procurement Practices have Improved as a Result of the OPO

	Federal Gvernment Officials (N=68)
1 - Not at all	8
2 - Minimal	24
3 - Somewhat	16
4 - Significant	5
5 - Great extent	1
N/A	14

A more detailed analysis of the survey results were undertaken and the responses of the federal government officials were divided into senior government versus other government staff as well as by procurement versus non procurement personnel. However, the average rating of these sub-groups did not vary significantly from the overall average rating of 2.4.

When the federal government officials were asked to provide the rationale for their relatively low rating of the extent to which the fairness, openness and transparency of their procurement practices have improved as a result of the OPO, the two most frequent responses are that there has been no substantial change in their procurement practices and that their procurement practices are already fair, open and transparent. Some other federal government officials indicated that OPO recommendations have resulted in only minor improvements to the fairness, openness and transparency of their procurement practices.

3.7: Most Frequent Responses by Federal Government Officials Regarding the Extent to Which the Fairness, Openness and Transparency Of the Department's Procurement Practices have Improved as a Result of the OPO

Question: To what extent have the fairness, openness and transparency of procurement practices of your organization improved as a result of the OPO, on a scale of 1 to 5 where 1 is not at all, 3 is somewhat and 5 is to a great extent?

Federal Government Officials (# provided comments = 51)
Note: excluding 14 officials who did not provide a rating and 3 who did not provide any comment for their rating

Those who provided higher rating

Made process improvements as a result of the OPO recommendations (8)
Paid more attention to procurement as a result of the OPO (3)

Those who provided lower rating

No substantial change in procurement practices as a result of the OPO (14)
Procurement practices are already fair, open and transparent (13)
OPO recommendations resulted only in minor improvements (8)

As indicated in Figure 3.7, 6 or 9% of the federal government officials surveyed indicated that the fairness, openness and transparency of their department's procurement practices have improved significantly or to a great extent while another 16 or 24% of respondents stated that their procurement practices have improved somewhat as a result of the OPO. Some of the reasons given by these respondents for their higher rating are that they have made process improvements as well as senior management devotes more attention to procurement as a result of OPO recommendations.

As indicated previously, some of the suppliers who stated that they complaint was not addressed because the OPO did not have the sufficient powers to compel departments to change their practices. In other words, they indicated that the OPO needs additional powers to be able to ensure departments comply with OPO recommendations regarding supplier complaints.

Conclusion: It is premature to develop conclusions regarding the extent to which the OPO has contributed to an improvement in the fairness, openness and transparency of departmental procurement practices. The OPO has raised the awareness of the need for fair, open and transparent procurement processes among senior management in some government departments. However, the impact of the OPO in improving departmental procurement practices has been limited by the recent establishment of the OPO and the limited powers and authority of the OPO as it can only make recommendations and departments don't have to comply.

Evaluation Question #10: To what extent has the OPO contributed to enhanced knowledge of suppliers and the federal government procurement community?

The three major activities undertaken by the OPO are to examine government procurement practices; review supplier complaints about contract award or administration; and provide alternative dispute resolution processes. While the OPO does communicate with the supplier and procurement communities to enhance knowledge and understanding of federal procurement, this is not a key activity of the OPO and does not consume a significant portion of OPO resources. In addition, there are many other organizations involved that are more focused on enhancing the knowledge of suppliers and the federal government procurement community.

The OPO has had a greater impact on enhancing the procurement knowledge of suppliers than the federal government officials surveyed. When these two groups of respondents were asked to what extent the OPO has enhanced their knowledge of federal government procurement, on a scale of 1 to 5 where 1 is not at all, 3 is somewhat and 5 is to a great extent, the average rating of the suppliers surveyed is 2.8 as compared to 2.2 for federal government officials. As indicated in Figure 3.8, 31 or 39% of suppliers who indicated that their complaint or inquiry was within the OPO mandate stated that the OPO enhanced their knowledge of federal government procurement to a significant or great extent while an additional 15 or 19% of suppliers indicated the OPO somewhat enhanced their knowledge of procurement.

Figure 3.8: Extent to Which OPO has Enhanced Knowledge of Procurement

	Federal Government Officials (N=68)	Suppliers (N=79)
1 - Not at all	15	22
2 - Minimal	20	11
3 - Somewhat	17	15
4 - Significant	3	22
5 - Great extent	1	9
N/A	12

The most frequent reasons given by suppliers why their knowledge of procurement has been enhanced as a result of the OPO are: the OPO provided a thorough explanation of the procurement process; they received guidance in terms of what steps to take next; and the OPO was very helpful in answering their questions.

As indicated in Figure 3.8, 33 or 42% of suppliers stated that the OPO did not enhance their knowledge of federal government procurement at all or to minimal extent. The most frequent reasons for their relatively low ratings are: they did not get answers to their questions; the information they obtained did not improve their knowledge; and they did not receive any information due to lack of follow-up.

3.8: Most Frequent Responses by Suppliers Regarding Whether the OPO Enhanced their Knowledge of Federal Government Procurement

Question: To what extent has the OPO contributed to enhanced knowledge of procurement within your organization, on a scale of 1 to 5 where 1 is not at all, 3 is somewhat and 5 is to a great extent?

Suppliers (# provided comments = 43)
Note: excluding 36 suppliers who did not provide any comment for their rating

Those who provided higher rating

Provided thorough explanation of the procurement process (16)
Received guidance in terms of what steps to take next (10)
OPO was very helpful in answering their questions (3)

Those who provided lower rating

Did not improve their knowledge (8)
Did not receive any information due to lack of follow up/did not receive any help (6)

As indicated in Figure 3.8, 35 or 51% of federal government officials stated that the OPO did not contribute to enhanced knowledge of procurement within their organization at all or to a minimal extent. The two most frequent reasons for these low ratings are they stated that their departmental personnel are already very knowledgeable regarding procurement or they did not learn anything new.

3.9: Most Frequent Responses by Federal Government Officials Regarding Whether the OPO Contributed to Enhanced Knowledge of Procurement within their Organization

Question: To what extent has the OPO contributed to enhanced knowledge of procurement within your organization, on a scale of 1 to 5 where 1 is not at all, 3 is somewhat and 5 is to a great extent?

Federal Government Officials (# provided comments = 45)
Note: excluding 12 officials who did not provide a rating and 11 did not provide any comment for their rating

Those who provided higher rating

Improved their knowledge of procurement processes considerably (7)
Learned best practices (4)

Those who provided lower rating

Already very knowledgeable regarding procurement (9)
Did not learning anything new (8)

As indicated in Figure 3.8, 4 or 6% of the federal government officials indicated that the OPO did contribute to enhanced knowledge of procurement within their organization to a significant or great extent while an additional 17 or 25% of the federal government officials indicated that the OPO enhanced their knowledge somewhat. The most frequent reasons given by these respondents for their relatively high ratings are the OPO improved their knowledge of procurement processes considerably or they learned about best practices as a result of OPO reports.

A more detailed analysis of the survey results were undertaken and the responses of the federal government officials were divided into senior government versus other government staff as well as by procurement versus non procurement personnel. As shown in Table 3.10, the average ratings of these sub-groups only varied slightly from the overall average rating of 2.2. The highest average ratings were provided by less senior and non-procurement staff.

Table 3.10: Average Rating of Sub-Groups of Federal Government Officials Surveyed

Analysis	Type of Sub-Group
Procurement Versus Non Procurement Position	Procurement Staff	Non-Procurement Staff
Procurement Versus Non Procurement Position	2,1	2,3
Seniority of Staff	More Senior	Less Senior
Seniority of Staff	2,0	2,4

Conclusion: Despite not being a primary mandate, the OPO has contributed to the knowledge of suppliers and the federal government procurement community.

Evaluation Question #11: What, if any, are the unintended outcomes of the OPO?

Very few key informants described any unintended outcomes of the OPO. A few key informants stated that one positive intended outcome was the sharing of best practices regarding procurement among government departments.

Conclusion: There have not been any major unintended outcomes of the OPO.

C. Performance – Efficiency, Effectiveness and Economy

This section provides evaluation findings and conclusions for the evaluation questions dealing with performance (i.e. efficiency, effectiveness and economy).

Evaluation Question #12: To what extent is the mandate and structure of the OPO adequate?

Fifty years ago, the word ombudsman and the institution were relatively unknown. There are now ombudspersons in over 100 countries. The main reason for the phenomenal growth has been the emergence of big government, and the consequent need for someone to protect the citizen from unfair decisions by big government. The operational model employed by ombudspersons varies greatly according to the activity and environment. However, there are essentially two models:

Reactive ombudsman – waits for complaints and acts on what has been brought forward. In Britain, for example, ombudsmen tend to be reactive offices that can only respond to complaints or grievances (i.e. the last port of call in a formal complaints procedure).
Proactive ombudsman – seeks out matters of concern, inspects and initiates investigations. In Scandinavia, an ombudsman may initiate action and has an ongoing inspection role, such as the Public Justice Ombudsman who keeps a watchdog eye on public administration and tackles action to ensure acceptable quality and the maintenance of standards.

The organizational structure of the OPO includes both of the above models. In other words,the OPO fulfils the role of a reactive ombudsman by investigating complaints while it also plays the role of a proactive ombudsman by conducting procurement practice reviews. A review of similar organizations was undertaken as part of the formative evaluation to compare the organization structure of the OPO with similar organizations.

While Canada has no single national ombudsman, the following nine Canadian provinces and one territory have parliamentary ombudsman who oversee the provincial government and receive and investigate public complaints:

Alberta Ombudsman (established in 1967)
Office of the Ombudsperson, British Columbia (1979)
Ombudsman Manitoba (1970)
New Brunswick Ombudsman`s Office (1967)
Citizen's Representative of Newfoundland (2002)
Nova Scotia Office of the Ombudsman (1970)
Ontario Ombudsman (1975)
Quebec Ombudsman (1969)
Ombudsman Saskatchewan (1972)
Office of the Yukon Ombudsman and Information and Privacy Commissioner (1996)

The above organizations are responsible for handling complaints by businesses regarding provincial and territorial government procurement as well as a wide variety of other matters. As an illustration of the activities undertaken by provincial and territorial ombudsman, the Office of the Ombudsperson in BC can investigate complaints of actions and decisions by provincial ministries, provincial boards and commissions, provincial Crown corporations, local governments, health authorities, school boards, colleges and universities, and other provincial authorities including self-regulating professions and pension boards of trustees. The mandate of the provincial and territorial ombudspersons is much broader than the OPO because they handle more than procurement-related complaints. In addition to investigating complaints, the BC Ombudsperson also has a proactive role in examining systemic issues. As an illustration, the BC Ombudsperson recently completed the largest and most complex investigation of its 30 year history which consisted of a detailed examination and comprehensive report on home and community care services delivered to seniors.

Even though Canada has no single national ombudsman, a number of specialist ombudsman and ombudsman-like organizations, such as the OPO, have been established by the Government of Canada including the following:

Office of the Ombudsman for National Defense and the Canadian Forces (Defence Ombudsman)
Federal Ombudsman for Victims of Crime
Office of the Veterans Ombudsman
Taxpayers' Ombudsman
Office of the Commissioner of Lobbying of Canada
Office of the Information Commissioner
Office of the Public Sector Integrity Commissioner of Canada

The mandate of most of the above federal ombudsman-like organizations is similar to the OPO in that it is much more focussed and narrow than provincial ombudsmen. One of the primary activities undertaken by most of these organizations is the investigation of complaints. In addition, the mandate of several of these organizations is also proactive and includes the examination of systemic issues. For example, the mandate of the Federal Ombudsman for Victims of Crime includes identifying and exploring systemic issues that impact negatively on victims of crime. Similarly, the mandate of the Office of the Veterans Ombudsman is to address complaints and emerging system issues related to programs and services provided or administered by the Department of Veterans Affairs, as well as systemic issues related to the Veterans Review and Appeal Board. In addition to investigating complaints from taxpayers, the mandate of the Taxpayers' Ombudsman is to identify and review systemic and emerging service-related issues within the Canada Revenue Agency that have a negative impact on taxpayers.

Based on an extensive literature review and Internet search, there does not appear to be any organization with a similar mandate to the OPO in other countries. Instead, the investigation of complaints regarding procurement is handled by an ombudsman with a much broader mandate than the OPO. As an illustration in Australia, complaints by businesses regarding federal government procurement are handled by the Commonwealth Ombudsman which is a parliamentary ombudsman that is appointed and funded by the Government of Australia to handle complaints about the administrative actions of federal government agencies. As indicated below, the Commonwealth Ombudsman has both a reactive and proactive role:

Complaint investigations – investigating and reviewing the administrative actions of Australian Government officials and agencies, upon receipt of complaints
Own motion investigations – investigating, on the initiative of the Ombudsman, the administrative actions of Australian Government agencies – often arising from insights gained from handling individual complaints.

The Parliamentary and Health Service Ombudsman in the United Kingdom is the official ombudsman institution responsible for investigating complaints regarding whether government departments, agencies and other public bodies in the United Kingdom have acted properly fairly, or have provided poor service. The mandate of the organization is primarily reactive. Another ombudsman organization in the United Kingdom is the Local Government Ombudsmen (LGO) which has a broad mandate and is primarily reactive. LGO's jurisdiction includes a wide range of public bodies providing local services, including local authorities, adult social care providers, national parks, police authorities, education appeal panels and some schools. The primary activity of the LGO is to investigate complaints by members of the public.

According to a recent article in the Journal of Public Sector Management entitled Ombudspersons as Producers of Governance, the mandate of the ombudsman must be not only be reactive but also proactive in order to be of significant value. This article states that the role of the ombudsman should be to tackle the issues raised by the complaints that it investigates in order to "unearth and expose the source of the problem, and to become the architect of better governance arrangements capable of eradicating the causes of the difficulties". The article also states that the independence, accessibility, informality, cheapness, and speed of the ombudsing process, together with the powers of investigation and inquiry, and some form of statutory base of operation, make ombudsing better suited to deal with governance failures than the more traditional legal and political processes. As indicated in the previous paragraphs, many of the ombudsman organizations in Canada and other countries are aligned with this role as they have both a reactive mandate to investigate complaints as well as a proactive mandate to identify and address systemic issues. The mandate of the OPO also conforms to this best practice because it includes the review of complaints with respect to contract award and administration as well as reviewing departmental procurement practices to identify and correct system-wide issues.

OPO's mandate to review departmental procurement practices, to review complaints with respect to contract award and administration, and to provide alternative dispute resolution services was well received by most federal government officials. Many found the mandate of OPO to be adequate and that the services provided by OPO complement other oversight bodies such as CITT and OAG. When federal government officials were asked to rate the adequacy of the mandate and structure of the OPO, on a scale of 1 to 5, where 1 is not at all, 3 is somewhat and 5 is very adequate, the average rating is 3.6 (Figure 3.9).

Figure 3.9: Adequacy of the Mandate and Structure of the OPO

	Federal Government Officials (N=68)
1 - Not at all	1
2 - Minimal	5
3 - Somewhat	15
4 - Adequate	28
5 - Very Adequate	7
N/A	12

The most frequent reason provided by federal government officials why they agreed with the mandate and structure of the OPO is they stated that the mandate and structure is adequate for its purpose and meets a need in the federal government. Some other reasons mentioned frequently are the mandate meets the needs of the supplier community by providing access to an independent entity to address supplier complaints and the OPO has a role to validate departmental procurement processes.

Some federal government officials who provided lower ratings stated the mandate of the OPO needs to be clarified, particularly with regard to reducing the overlap with other oversight bodies. Other officials felt that the OPO needs greater authority and strategic focus to impact change while some other federal government representatives stated that the mandate of the OPO is too limited and the mandate should be expanded to handle larger dollar value contracts.

3.11: Most Frequent Reasons for Responses by Federal Government Officials Regarding the Adequacy of the Mandate and Structure of the OPO

Question: How adequate is the mandate and structure of the OPO, on a scale of 1 to 5 where 1 is not at all, 3 is somewhat and 5 is very adequate?

Federal Government Officials (# provided comments = 44)
Note: excluding 12 officials who did not provide a rating and 12 that did not provide any comment for their rating

Those who provided higher rating

OPO's mandate and structure is adequate for its purpose and meets a need in the federal government (13)
Meets the needs of the supplier community by providing access to an independent entity to address supplier complaints (6)
OPO has a role to validate departmental procurement processes (3)

Those who provided lower rating

Mandate needs to be clarified/need to reduce duplication with other oversight bodies (6)
OPO needs greater authority and strategic focus to impact change (5)
Mandate is too limited/should handle larger contracts (3)

While suppliers were not specifically asked to comment on the mandate and structure of the OPO, several suppliers did offer a number of comments in this regard when responding to other questions or when asked if they had any other comments. As indicated in Table 3.12, the most frequent responses received by suppliers are that the mandate of the OPO is too limited (e.g. should be allowed to handle larger contracts, be engaged before the contract stage) and the OPO needs greater power to order government agencies to act (i.e. be more than an advisory organization). A few suppliers stated that the OPO should be more independent from PWGSC to enhance its credibility with the supplier community.

Table 3.12: Most Frequent Responses by Suppliers Regarding the Mandate and Structure of the OPO

Response	Number of Responses
Mandate/jurisdiction of the OPO is too limited (e.g. should handle larger contracts, needs to be engaged before the contract stage, etc.)	36
OPO needs greater power to be of value/needs authority to make determination and order government agencies to act/needs to be more than an advisory organization	28
OPO should be more independent from PWGSC	5

Conclusion: The mandate and structure of the OPO are adequate. The mandate and structure are adequate because the OPO fills a gap in handling supplier complaints below CITT's minimum threshold. In addition to addressing supplier complaints, the OPO performs a useful proactive function in addressing systemic issues which is necessary to enhance the fairness, openness and transparency of federal government procurement practices and to reduce supplier complaints in the future.

Evaluation Question #13: To what extent has the OPO been implemented as intended?

The Procurement Ombudsman Regulations dated May 1, 2008 indicate that the OPO shall undertake the following activities: review of departmental practices for acquiring materiel and services and make any necessary recommendations; review of complaints regarding the award of a contract or the administration of a contract and provide findings and recommendations including recommended compensation, where appropriate; and provide an alternative dispute resolution process. As described in responses to previous evaluation questions, all of the above activities have been implemented by the OPO.

Conclusion: The OPO has been implemented as intended.

Evaluation Question #14: To what extent do OPO activities reach the intended stakeholders?

The OPO has undertaken a number of activities to reach its intended stakeholders including outreach presentations, participation in conferences, events and workshops, development and maintenance of an Internet website, etc. In addition, the OPO has collaborated with PWGSC to ensure that reference of the OPO occurs in procurement contracts so that suppliers are aware of the OPO in case they have issues with their contract. The OPO recently prepared a communications strategy which includes an extensive outreach program. The OPO is currently implementing this communications strategy to raise the awareness of OPO's role and to increase access to OPO services.

As indicated in Figure 3.10, the three most frequent methods by which suppliers surveyed became aware of the OPO are an internet search (30% of total responses), referral by a government representative (22%) and a referral by a business colleague (13%). About 9% of the suppliers surveyed stated that they became aware of the OPO as a result of communications undertaken by the OPO. Some of the other methods mentioned less frequently by respondents regarding how they became aware of the OPO are industry associations, business advisors, newspapers and personal research by the respondents.

Figure 3.10: How did you become aware of the OPO?

	Suppliers (N=164)
Don't know	11
Other	25
Business advisor	3
Industry association	10
Communications by the OPO	16
Reference from a business colleague	23
Government representative	40
Internet search	56

Note: Some respondents provide more than one means of becoming aware of the OPO. Total count of responses is 184.

In August 2009, the Canadian Federation of Independent Business (CFIB) conducted a survey of 1,450 SMEs thought to be most likely involved in federal procurement. It included those companies that indicated in CFIB's 2008 survey that they had recently sold to the federal government. The survey sample was also increased to focus on those sectors that were more likely to engage in federal procurement (e.g. construction, manufacturing, wholesale, retail, and enterprise and administrative management). Of the respondents to the CFIB survey, 68.6% indicated that they had never used the services of the Procurement Ombudsman, while 0.9% indicated that they had used OPO services while the remaining 30.5% stated that they were unaware of the existing of the OPO (Figure 3.11). This low percentage of suppliers using OPO services is appropriate because it is likely that only a very small proportion of the suppliers will have to make a complaint to the OPO and this does not indicate the level of awareness of the OPO. While Figure 3.11 indicates that over two thirds of respondents appear to be aware of the Procurement Ombudsman's existence, a representative of CFIB indicated that caution should be used in coming to this conclusion because the survey results could indicate instead that over two thirds of respondents are aware of some type of ombudsman but not necessarily the OPO in particular.

Figure 3.11: Have you ever used the services of the Procurement Ombudsman?

	Column 1
Yes	0.9%
No	68.6%
Unaware of existence	30.5%

When the key informants interviewed were asked to indicate to what extent OPO activities reach the intended stakeholders, on a scale of 1 to 5 where 1 is not at all, 3 is somewhat and 5 is to a great extent, the average rating was 2.8 (20 respondents) which indicates that the OPO has been somewhat successful in reaching the intended audience. Many key informants indicated that the degree of awareness of the OPO is high among the federal government procurement community, particularly in Ottawa, but suppliers are less aware of the OPO. Some factors limiting the awareness of the OPO are that it was established only a few years ago and it takes a lot of communication resources to create awareness of the OPO given the large number of suppliers in Canada.

Conclusion: The OPO has been somewhat successful in reaching the intended audience.

Evaluation Question #15: To what extent does the OPO have staff with the required skills sets and competencies to effectively deliver results?

Federal government officials and suppliers were asked to rate the extent to which the OPO has staff with the required skills sets and competencies to effectively deliver results, on a scale of 1 to 5 where 1 is not at all, 3 is somewhat, and 5 is to a great extent. Federal department officials and suppliers provided average ratings of 3.5 and 3.3, respectively, while a breakdown of their ratings is shown in Figure 3.12.

Figure 3.12: Extent to which the OPO Has Staff with the Required Skills Sets and Competencies to Effectively Deliver Results

	Federal Government Officials (N=68)	Suppliers (N=79)
1 - Not at all	0	11
2 - Minimal	8	7
3 - Somewhat	13	13
4 - Significant	22	17
5 - Great extent	4	14
N/A	21	17

As indicated in Figure 3.12, a total of 31 or 39% of the suppliers who made complaints or inquiries within the OPO mandate indicated that OPO staff had the required skill sets and competencies while an additional 13 or 16% stated that OPO staff were somewhat qualified. However, many of these respondents did not provide any explanation for their ratings of OPO staff. The most frequent responses for the higher ratings are that OPO staff are very knowledgeable, competent and helpful while other suppliers indicated that OPO staff understood procurement and provided guidance regarding what steps should be taken. A total of 18 or 23% of suppliers provided a rating of 1 or 2 regarding OPO staff skill skills sets and competencies. However, very few of these respondents provided a rationale for their rating. A few of these respondents indicated that reasons for their low rating are that the staff did not provide an answer or any help. Some other suppliers did not directly address the question and stated that the reason for their low rating is that the staff did not have the power to resolve their complaint.

3.13: Most Frequent Responses by Suppliers Regarding OPO Staff Skill Sets and Competencies

Question: To what extent does the OPO have staff with the required skills sets and competencies to effectively deliver results, on a scale of 1 to 5 where 1 is not at all, 3 is somewhat and 5 is to a great extent?

Suppliers (# provided comments = 42)
Note: excluding 27 suppliers who did not provide any comment for their rating

Those who provided higher rating

Staff are knowledgeable/competent/helpful (16)
Understand procurement (9)
Provided good guidance regarding next steps (6)
Have expertise in dispute resolution (3)

Those who provided lower rating

Staff did not provide an answer or any help (6)
Staff did not have skills to assess their industry (2)

Approximately one third of the federal government officials surveyed indicated that they are not sufficiently aware of OPO staff skills to be able to rate their qualifications. As indicated in Figure 3.12, 26 or 38% of the federal government officials indicated that the extent to which OPO has staff with the required skills sets and competencies to effectively deliver results is significant or to a great extent. Several respondents indicated that not only are the staff very knowledgeable about procurement practices and processes but they are also highly competent, professional, communicative and collaborative with government departments. The most frequent reasons given by federal government officials who provided relatively lower rating regarding the skills sets and competencies of OPO staff are as follows: some staff could benefit by having more procurement knowledge; staff turnover appears to be an issue; and some staff have the knowledge and experience but lack the analytical and writing skills in carrying out the work.

3.14: Most Frequent Responses by Federal Government Officials Regarding OPO Staff Skill Sets and Competencies

Federal Government Officials (# provided comments = 42)
Note: excluding 15 officials who did not provide a rating and 11 who did not provide any comment for their rating

Those who provided higher rating

Staff are very competent, collaborative, and knowledgeable in the field of procurement (23)

Those who provided lower rating

Some staff could benefit by having more procurement knowledge (6)
Staff turnover appears to be an issue (4)
Some staff have the knowledge and experience but lack the analytical and writing skills in carrying out the work (3)

When OPO staff were asked to indicate to what extent have the roles and responsibilities of OPO staff been clearly delineated and matched with the requirements of the mandate, on a scale of 1 to 5 where 1 is not at all, 3 is somewhat and 5 is to a great extent, the average rating was 3.8, indicating that staff feel the staff roles are aligned with the requirements of the mandate. Several staff indicated that the OPO has an appropriate mixture of staff with procurement and audit experience. Some staff indicated that there is opportunities to more clearly delineate the duties and responsibilities of the different corporate groups. A few respondents indicated that it might be more effective for ADR processes to be handled by the corporate group responsible for handling supplier complaints and investigations rather than another corporate group.

Conclusion: The OPO has staff with the required skills sets and competencies to effectively deliver results.

Evaluation Question #16: To what extent have OPO activities met OPO service standards?

The Procurement Ombudsman Regulations set out for the OPO establish a number of specific service standards the OPO is expected to meet. These services standards are listed below:

OPO Service Standards

Procurement Reviews

One year to complete
Recommendations take into account government policies and resource implications
Departments are provided the opportunity to provide comments before OPO report is finalized

Formal Complaints: Contract Awards

Suppliers have 30 days to file a complaint which may be extended to 90 working days
OPO has 10 working days to determine whether to review the complaint (contracting department and complainant are notified without delay)
Simple review
Contracting department has 15 working days to comment on the written complaint
OPO has 95 working days to provide a Final Report
Expanded review
Contracting department has 25 days to comment on the written complaint
Complainant has 10 working days to respond
Contracting department has 10 working days to provide final comments if any
OPO has 65 working days to produce the Final Report

Formal Complaints: Contract Administration

Suppliers have 30 days to file a complaint which may be extend to 80 working days
OPO has 10 working days to determine whether to review the complaint (contracting department and complainant are notified without delay)
Contracting department is provided a copy of the complaint
OPO has 95 working days to provide a Final Report

Alternative Dispute Resolution (ADR) Mechanisms

Upon request from a part to a contract, OPO has 10 working days after the request to ask the other parties if they agree to participate in such a process and to bear the costs of that process
The other parties are permitted 10 working days to respond to OPO
If the parties agree to participate, OPO has 10 working days after receipt of the last of the other partied responses to submit to the parties a proposal for an ADR process

The OPO is currently working on developing service standards below the Regulatory ones. It is recognized that it is more difficult to set service standards in a service environment. Additional factors such as quality of service and client satisfaction must be considered in a service environment.

Detailed information is maintained on the project files for each inquiry, complaint and investigation as well as each procurement practice review in order to determine whether the applicable service standards have been met. However, this information is not reported making it difficult to determine whether the relevant service standards have been met without going through each individual file.

When OPO staff were asked to indicate to what extent OPO activities (inquiries and complaints, ADR and practice reviews) have met OPO service standards, on a scale of 1 to 5 where 1 is not at all, 3 is somewhat and 5 is to a great extent, the average rating is 4.2. Several staff indicated that the service standards with regards to inquires and complaints have been met to a greater extent than the service standards for procurement practice reviews.

Conclusion: Data is maintained to determine whether OPO services standards have been met but this information needs to be reported regularly.

Evaluation Question #17: To what extent does the OPO have the appropriate performance indicators to report on impacts and effects of its activities?

The performance indicators that were initially established for the OPO and included in PWGSC's Report on Plans and Priorities are as follows:

Percentage of suppliers whose confidence in federal procurement has increased
Percentage of procurement officials who recognize that OPO's efforts have resulted in an increase in procurement community professionalism (training, tools, knowledge of best practices
Percentage of senior officials and parliamentarians attesting to the usefulness of the Procurement Ombudsman Report to Parliament in helping them arrive at a meaningful assessment of the performance of the federal procurement process
Percentage of suppliers attesting to the simplification of the procurement process and reduction of paper burden

The initial OPO performance indicators were subsequently revised and changed to the following as reported in PWGSC's 2012/2013 Report on Plans and Priorities.

Percentage of formal complaints, requests for ADR and procurement practice reviews addressed within the timelines stipulated in the Regulations
Percentage of all inquiries and complaints addressed by the provision of information, referrals to appropriate authorities and facilitation

The OPO has also prepared some additional indicators that are in draft form.

Based on a review of the existing indicators, some progress has been made in developing performance indicators but some additional indicators are required and work is underway. In particular, some additional performance indicators are required to ensure that all key immediate outcomes in the OPO logic model are addressed. In addition, it is important to determine the most appropriate source of data for each performance indicator.

When OPO staff were asked to what extent does the OPO has the appropriate performance indicators to report on impacts and effects of its activities, on a scale of 1 to 5 where 1 is not at all, 3 is somewhat and 5 is to a great extent, the average rating was 2. Most staff indicated that more appropriate performance indicators have to be developed.

Conclusion: While some progress has been made in developing performance indicators, it would be beneficial to identify some additional indicators. Some additional performance indicators are required to ensure that all key immediate outcomes in the OPO logic model are addressed.

Evaluation Question #18: To what extent does the OPO use the most efficient and effective means to achieve its outputs and outcomes?

Almost half of the federal government officials surveyed indicated that they were not able to rate the extent to which the OPO uses the most efficient and effective means to achieve its outputs and outcomes. Other federal government officials provided an average rating of 3.1, when they were asked to provide a rating on a scale of 1 to 5 where 1 is not at all, 3 is somewhat and 5 is to a great extent (Figure 3.13).

Figure 3.13 : Extent to which the OPO Use the Most Efficient and Effective Means to Achieve Its Outputs and Outcomes

	Federal Government Officials (N=68)
1 - Not at all	1
2 - Minimal	10
3 - Somewhat	15
4 - Significant	10
5 - Great extent	2
N/A	30

The federal government officials who provided a higher rating believe that given the resources and tools available, the OPO has been quite efficient and effective in achieving its outputs and outcomes. Some officials stated that the OPO makes great use of face-to-face and email transfer of knowledge while others felt that OPO reports on best practices and practice reviews are helpful. A few respondents recognized that the OPO has been present at different forums where contracting officers meet for workshops or other meetings. The federal government officials who provided a lower rating believe there exists room for improvement in OPO's means to effectively and efficiently achieve its outputs and outcomes. Some respondents stated that the OPO does not provide department's sufficient time to review OPO reports while others indicated that the effectiveness of the OPO could be enhanced by placing less emphasis on procurement practice reviews and more emphasis on the investigation of supplier complaints. A few respondents stated that the OPO should consider a less formal approach for the investigation of supplier complaints. Other respondents felt that an excessive amount of departmental resources was required to address OPO procurement practice reviews and investigations. In order to reduce the impact on departmental personnel, some federal government officials suggested that the OPO should space procurement practice reviews more evenly throughout the year.

3.15: Efficiency and Effectiveness of OPO

Question: To what extent does the OPO use the most efficient and effective means to achieve its outputs and outcomes, on a scale of 1 to 5 where 1 is not at all, 3 is somewhat and 5 is to a great extent?

Federal Government Officials (# provided comments = 30)
Note: excluding 30 officials who did not provide a rating and 8 who did not provide any comment for their rating

Those who provided higher rating

OPO seems to operate in an effective and efficient way given the resources and tools available (11)
OPO has been present at different forums where contracting officers meet for workshops or other meetings (2)
Makes great use of face-to-face and email transfer of knowledge (2)

Those who provided lower rating

Does not provide department sufficient time to review OPO reports (5)
Effectiveness could be enhanced by placing less emphasis on procurement practice reviews and more emphasis on the investigation of supplier complaints (4)
Should consider a less formal approach for the investigation of supplier complaints (3)
Need to space procurement practice reviews more evenly throughout the year to reduce impact on departmental personnel (3)
Excessive amount of departmental resources required to address OPO procurement practice reviews and investigations (3)

Table 3.16 provides a comparison of the level of resources of the OPO with similar organizations. Because the type of activity undertaken by other organizations is quite different than the OPO, it is difficult to compare the efficiency of these organizations with the OPO. However, the table does indicate that the current level of resources of the OPO is in line with organizations that perform somewhat similar functions. The annual expenditures of the OPO in 2010/11 were approximately $3.4 million which is between the range of $2.8 million and $6.9 million for the organizations that perform similar ombudsman-like functions.

Table 3.16: Comparison of Resources and Activities of Similar Organizations in 2010-11

Organization	Annual Expenditures ($ millions)
OPO	$3.4
CITT	$6.9
Office of the Veterans Ombudsman	$3.6
Defence Ombudsman	$4.6
Taxpayers Ombudsman	$2.8
Office of the Commissioner of Lobbying	$4.7
Office of the Public Sector Integrity Commissioner of Canada	$5.3
Average

Conclusion: Based on a limited analysis, the OPO appears to use efficient and effective means to achieve its outputs and outcomes.

Document Navigation for "Evaluation Finding and Conclusions" Previous Page Table of Contents

Date modified:: 2018-11-01

Evaluation Finding and Conclusions

A. Relevance

Evaluation Question #1: To what extent is there an ongoing need for the OPO?

Figure 3.1: Extent of the Ongoing Need for the OPO

3.1: Most Frequent Responses Provided by Suppliers Regarding the Need for the OPO

Question: To what extent is there an ongoing need for the Office of the Procurement Ombudsman, on a scale of 1 to 5 where 1 is not at all, 3 is somewhat and 5 is to a great extent?

Suppliers (# provided comments = 134)

Those who provided higher rating

Those who provided lower rating

3.2: Most Frequent Responses by Federal Government Officials Regarding the Need for the OPO

Question: To what extent is there an ongoing need for the Office of the Procurement Ombudsman, on a scale of 1 to 5 where 1 is not at all, 3 is somewhat and 5 is to a great extent?

Federal Government Officials (# provided comments = 61)

Those who provided higher rating

Those who provided lower rating

Evaluation Question #2: To what extent are the objectives and activities of the Office of the Procurement Ombudsman aligned with federal government priorities?

Evaluation Question #3: To what extent are the activities of the Office of the Procurement Ombudsman aligned with federal roles and responsibilities?

Evaluation Question #4: Do the activities of the Office of the Procurement Ombudsman complement or overlap those of other programs?

Figure 3.2 Relationship of OPO Activities with Other Programs

B. Performance - Achievement of Outcomes

Evaluation Question #5: To what extent has the OPO been able to address the complaints received regarding contract award or administration?

Figure 3.3: Number and Type of Complaints Received by the OPO

Table 3.3: Number of OPO Investigations of Supplier Complaints from 2008-09 to 2010/11

Figure 3.4: Purpose of Contact with the OPO by the Suppliers Surveyed

Figure 3.5: Suppliers Responses Regarding the Extent to Which Their Complaints and Inquiries Have Been Addressed by the OPO

3.4: Most Frequent Responses by Suppliers Regarding the Handling of their Complaints Regarding Contract Award and Contract Administration by the OPO

Those who provided higher rating

Those who provided lower rating

3.5: Most Frequent Responses by Suppliers Regarding the Handling of their Inquiries by the OPO

Those who provided higher rating

Those who provided lower rating

Evaluation Question #6: To what extent has the provision of alternative dispute resolution (ADR) processes been able to facilitate settlement of contractual disputes?

Evaluation Question #7: To what extent has the OPO been independent, neutral and helpful with regard to federal government procurement?

Evaluation Question #8: To what extent has there been acceptance of the recommendations resulting from OPO procurement practice reviews and investigations resulting from complaints received by the OPO?

Figure 3.6: Extent to Which Recommendations of OPO Procurement Practice Reviews and Investigations Resulting from Complaints have been Accepted

3.6: Most Frequent Responses by Federal Government Officials Regarding the Extent to which the Recommendations of OPO have been Accepted

Those who provided higher rating

Those who provided lower rating

Evaluation Question #9: To what extent have the fairness, openness and transparency of departmental procurement practices improved as a result of the OPO?

Figure 3.7: Extent to Which the Fairness, Openness and Transparency of the Department's Procurement Practices have Improved as a Result of the OPO

3.7: Most Frequent Responses by Federal Government Officials Regarding the Extent to Which the Fairness, Openness and Transparency Of the Department's Procurement Practices have Improved as a Result of the OPO

Those who provided higher rating

Those who provided lower rating

Evaluation Question #10: To what extent has the OPO contributed to enhanced knowledge of suppliers and the federal government procurement community?

Figure 3.8: Extent to Which OPO has Enhanced Knowledge of Procurement

3.8: Most Frequent Responses by Suppliers Regarding Whether the OPO Enhanced their Knowledge of Federal Government Procurement

Those who provided higher rating

Those who provided lower rating

3.9: Most Frequent Responses by Federal Government Officials Regarding Whether the OPO Contributed to Enhanced Knowledge of Procurement within their Organization

Those who provided higher rating

Those who provided lower rating

Table 3.10: Average Rating of Sub-Groups of Federal Government Officials Surveyed

Evaluation Question #11: What, if any, are the unintended outcomes of the OPO?

C. Performance – Efficiency, Effectiveness and Economy

Evaluation Question #12: To what extent is the mandate and structure of the OPO adequate?

Figure 3.9: Adequacy of the Mandate and Structure of the OPO

3.11: Most Frequent Reasons for Responses by Federal Government Officials Regarding the Adequacy of the Mandate and Structure of the OPO

Those who provided higher rating

Those who provided lower rating

Table 3.12: Most Frequent Responses by Suppliers Regarding the Mandate and Structure of the OPO

Evaluation Question #13: To what extent has the OPO been implemented as intended?

Evaluation Question #14: To what extent do OPO activities reach the intended stakeholders?

Figure 3.10: How did you become aware of the OPO?

Figure 3.11: Have you ever used the services of the Procurement Ombudsman?

Evaluation Question #15: To what extent does the OPO have staff with the required skills sets and competencies to effectively deliver results?

Figure 3.12: Extent to which the OPO Has Staff with the Required Skills Sets and Competencies to Effectively Deliver Results

3.13: Most Frequent Responses by Suppliers Regarding OPO Staff Skill Sets and Competencies

Those who provided higher rating

Those who provided lower rating

3.14: Most Frequent Responses by Federal Government Officials Regarding OPO Staff Skill Sets and Competencies

Those who provided higher rating

Those who provided lower rating

Evaluation Question #16: To what extent have OPO activities met OPO service standards?

OPO Service Standards

Procurement Reviews

Formal Complaints: Contract Awards

Simple review

Expanded review

Formal Complaints: Contract Administration

Alternative Dispute Resolution (ADR) Mechanisms

Evaluation Question #17: To what extent does the OPO have the appropriate performance indicators to report on impacts and effects of its activities?