Procurement Practice Review: Follow-up Report on the 2016-2017 “Review of Bid Evaluation Processes”

What we reviewed

1. In 2016-2017, the Office of the Procurement Ombudsman (OPO) conducted a procurement practice review entitled review titled Review of Bid Evaluation Processes.

2. Three federal organizations were included in the scope of this review: Canada Border Services Agency (CBSA), Parks Canada (Parks), and Royal Canadian Mounted Police (RCMP). The Procurement Ombudsman’s recommendations were addressed only to two organizations: CBSA and Parks.

3. In September 2018, OPO asked CBSA and Parks to provide information regarding actions taken in response to the recommendations in the original 2016-2017 review.

4. The purpose of the follow-up exercise was to determine whether CBSA and Parks considered and took action, or developed plans, in response to the Procurement Ombudsman’s recommendations. In this regard, OPO assessed the information provided by the organizations for overall reasonableness and credibility. This report provides a summary, as well as specific examples, of progress made by the organizations in implementing the recommendations from the original review.

Why it’s important

5. There are three main reasons why reporting on the progress made in response to the Procurement Ombudsman’s recommendations is important:

• it informs interested stakeholders of specific actions organizations have taken to improve procurement practices
• by sharing information on changes being implemented by the organizations whose practices were reviewed, OPO facilitates other federal organizations’ ability to introduce similar improvements; and
• the information on the nature and extent of responses to the recommendations provides an indication of the usefulness of OPO’s reviews in promoting fairness, openness, and transparency in federal procurement

What we found

6. CBSA stated it took two actions in response to the recommendation made by the Procurement Ombudsman. Using a scale provided by OPO^{Footnote 1}, CBSA self-assessed the level of implementation of its actions as “full implementation” (Level 5).

7. Parks stated it took three actions in response to the recommendation made by the Procurement Ombudsman. Using the scale provided by OPO, Parks self-assessed the level of implementation of its actions as “full implementation” (Level 5).

8. Upon reviewing the information provided by the organizations, OPO assessed the level of implementation of the Procurement Ombudsman’s recommendations by CBSA as “full implementation” (Level 5) and by Parks as “substantial implementation” (Level 4).

Objectives

10. The objectives of this follow-up review were to determine:

• whether CBSA and Parks considered the recommendations made by the Procurement Ombudsman in the November 2016 review with respect to their procurement practices
• whether action plans to respond to the recommendations were prepared and approved; and
• what actions were undertaken in response to the recommendations, and the extent to which each action had been monitored and completed

11. OPO expected the two organizations to have introduced changes to improve their procurement practices in response to the Procurement Ombudsman’s recommendations.

Scope, Methodology, and Timing

12. OPO requested CBSA and Parks provide information on actions planned or implemented as a result of the recommendations in the original (i.e. November 2016) review. This report reflects actions implemented by the two organizations up to September 2018.

13. The approach OPO uses for follow-up exercises differs from the approach used in OPO’s procurement practice reviews. The assessment of progress made against recommendations was based upon the Department’s self-assessment and assertions regarding its plans and actions, coupled with supporting (i.e. substantiating) documentation. For each recommendation in the original review, OPO reviewed the information provided for overall reasonableness and credibility. This was done by:

• verifying whether any contradiction existed between an organization’s assertions and information available from publicly accessible sources or obtained during the original review
• analyzing the organization’s responses to understand how its actions addressed the recommendations and whether there were plans to monitor the results or effectiveness of these actions or changes; and
• seeking clarification, as required, to ensure a clear understanding of the information and supporting documentation provided by the organization

14. This report provides an overview of the CBSA’s and Parks’ assertions, as well as OPO’s assessment, on their progress in implementing changes in response to the recommendations contained in the original review.

Summary of original review findings

15. The original review covered key elements of a bid evaluation process such as: consistency of the guidance in place to support bid evaluation processes with the Treasury Board Contracting Policy (TBCP); bid evaluation criteria established as per applicable policies and guidance; and bid evaluations and supplier selection conducted as per applicable policies and guidance. The original review’s findings for each of these elements are summarized below.

16. At all three organizations (CBSA, Parks, and RCMP), guidance was in place to support bid evaluation processes, and was consistent overall with the TBCP. Although monitoring activities occurred, each organization varied in formality and the level of quality assurance (QA) provided.

17. At CBSA, the Material Management Volume that stood as its internal procurement policy had not been updated since 2005 (approximately ten years). Multiple versions of bid evaluation plans and scoring sheet templates were also identified. Peer reviews and self-verifications were encouraged but not mandatory.

18. Although not in place during the original review, CSBA was also implementing a Contracting QA program. Upon its full implementation, this QA program would include three features:

• established criteria for file review, with increasing levels of review dependent on risk
• file close-out checklists; and
• service standards for the QA group

19. At Parks, staff were advised to use the Public Works and Government Services Canada Supply Manual in lieu of an internal procurement policy. Parks also had only one delegated contracting authority level, which was not consistent with the risk management approach applied across other federal organizations. Peer reviews were informal and at the discretion of the contracting authority. No formal challenge mechanism (e.g. a contract review board) was in place at the time of the original review.

20. All three organizations had provided evidence of contracting officers giving advice and performing a challenge function with respect to establishing bid evaluation criteria.

Original review conclusion

21. The original review had concluded:

• overall, the reviewed organizations had policies and guidelines in place to support bid evaluation processes; roles and responsibilities were clear and communicated; and monitoring and QA activities were noted
• the bid evaluation processes for the majority of files reviewed were conducted in a manner consistent with applicable legislation and policies, and supported the principles of fairness, openness and transparency. However, inconsistencies and areas for improvement were noted, including the need for organizations to strengthen file documentation

22. The Procurement Ombudsman recommended that:

1. CBSA complete the implementation of its Contracting QA program and the updating of policies, templates, and guidance; and
2. Parks strengthen its oversight of bid evaluation processes through:
   1. the continued updating of procurement guidance
   2. the development and implementation of formal QA activities
   3. the establishment of an appropriate review mechanism, such as a contracting review board; and
   4. a review of its contracting authority delegations

Summary of organizational response to follow-up

23. CBSA and Parks stated they took actions to address the recommendations made in the original review, and that all actions had been fully implemented. The organizations provided substantiating documentation for most of these actions.

CBSA’s Response to recommendation

24. In response to the Procurement Ombudsman’s recommendation, CBSA stated it had implemented two actions to ensure it completed the implementation of its Contracting QA program, as well as the updating of its policies, templates and guidance.

25. First, CBSA completed the implementation of its formal QA Program. To support this, CBSA also created a database to track QA requests, including the requests’ status, amendments, and approvals; and a formal Pre and Post Contract Award Review checklist exported from the database and signed by the reviewing officer and applicable procurement authorities for compliance with CBSA’s QA criteria.

26. Second, CBSA updated and communicated its QA procedures and guidance. Formal communication on updated QA requirements was shared with Contracting Officers. An informal QA checklist, which included approval from the Review Officer and standardized review questions, was also developed. CBSA also developed procedural documentation that was further intended to be reviewed and updated as part of its 2018-2019 Financial Policy Instrument Suite Reset.

Assessment of CBSA’s response to recommendation

31. CBSA stated it had implemented two actions to respond to the Procurement Ombudsman’s recommendation. Both actions were reported by CBSA as being fully implemented. CBSA provided documentation to substantiate the actions taken.

32. First, to assess the assertion that its QA Program had been fully implemented, CBSA’s documentation was reviewed. The necessary features of its QA Program were substantiated through the organization’s clear communiqués on QA requirements, formal and informal checklists, and comprehensive procedural documentation. An ability to monitor results was also substantiated through the details tracked by its QA database, as well as the documented changes required by the formal QA review checklist.

33. Second, relating to the updating of policies, templates, and guidance, CBSA provided communiqués substantiating its formal communication of QA requirements that confirmed the up-to-date status of some guidance, and an intent to send another communiqué to new staff following changes to its organizational structure.

34. CBSA also provided its QA Program’s procedural documentation, which contained a reference to guidance from 2003, and multiple references to the Agreement on Internal Trade (AIT) without referencing the Canadian Free Trade Agreement (CFTA), which replaced the AIT in 2017. However, this documentation is scheduled for review during CBSA’s Financial Policy Instrument Suite Reset, which is currently underway and extended to a multi-year effort.

Assessment of Parks’ response to recommendation

35. Parks stated it had implemented three actions to respond to the Procurement Ombudsman’s recommendation. All three actions were reported by Parks as being fully implemented. Parks provided documentation to largely substantiate the actions taken.

36. First, Parks provided screenshots of its intranet to identify its available procurement communiqués, guides, templates, forms, and tools. A communiqué to staff substantiated how Parks monitored the need for further updates through a detailed explanation of the factors leading to its new requirements and the objectives of each change. To demonstrate the continued nature of its updates, Parks also provided four internal procurement guidance documents that were recently published to its intranet.

37. Second, Parks provided an approved three-year proposal that outlined a conceptual plan for its Quality Assurance and Training (QA&T) section, and identified roles and staff responsible for the document. This plan helped substantiate the QA&T section’s existence, and established the deliverables Parks would expect to produce as per the planned activities, including QA compliance reviews and procurement training presentations. The existence of these deliverables were substantiated through a checklist for QA compliance reviews, a multi-year schedule of QA activities (e.g. reviews, training, site visits), and the full presentations for Parks’ basic and enhanced procurement training sessions.

38. Third, Parks provided a 2018 communiqué announcing an organization-wide exercise to validate signature card attestations as a result of its migration to a new financial system, and because all existing financial signature cards would expire, new contracting authority delegations would be assigned among three basic authority levels. Requests for one of four enhanced authority levels would be subject to a case-by-case assessment only after the requesting individual had been proven to meet multiple requirements, including those for training and endorsements from relevant financial managers.

39. Parks did not provide documentation specific to establishing an appropriate review mechanism, as had been recommended by the Procurement Ombudsman.