Follow-up report to the 2019-20 procurement practice review of Environment and Climate Change Canada
1. As a standard practice, the Office of the Procurement Ombudsman (OPO) follows up on the recommendations made in Procurement Practice Reviews (PPRs) two years after those recommendations are published.
2. Follow-up reviews allow OPO to provide assurance that actions to improve federal procurement processes in response to PPR recommendations have been effectively implemented, or that senior management has accepted the risk of not taking action; and provide an organizational report card that facilitates comparisons across organizations over time.
3. In 2019-2020 OPO conducted a PPR entitled Procurement Practice Review of Environment and Climate Change Canada (ECCC). The final report was issued in July 2020.
4. The initial review was undertaken to determine whether ECCC’s procurement practices pertaining to evaluation criteria and selection plans, solicitation, and evaluation of bids and contract award, supported the principles of fairness, openness and transparency. To make this determination, OPO examined whether ECCC’s procurement practices were consistent with Canada’s obligations under applicable sections of national and international trade agreements, the Financial Administration Act and regulations made under it, the Treasury Board Contracting Policy and, when present, departmental guidelines.
5. The following three lines of enquiry (LOE) were used to assess the highest-risk procurement elements:
- LOE 1: Evaluation criteria and selection plans were established in accordance with applicable laws, regulations and policies;
- LOE 2: Solicitation documents and organizational practices during the bid solicitation period were consistent with applicable laws, regulations and policies; and
- LOE 3: Evaluation of bids and contract award were conducted in accordance with the solicitation.
6. Regarding LOE 1, OPO’s initial review found that evaluation criteria and selection plans met requirements set out in applicable laws, regulations and policies in the majority of procurements reviewed. In all files reviewed, mandatory criteria were not overly restrictive and did not unnecessarily favour or penalize any particular bidder. Most mandatory and point-rated criteria were clearly communicated in a manner supporting a fair and transparent procurement process. However, in several files, issues were identified regarding unclear or inconsistent evaluation criteria, instructions for awarding points for point-rated criteria and selection methodologies. These issues can lead to incorrect bids and improper evaluations, thus impacting the fairness of the procurement process.
7. Regarding LOE 2, OPO’s initial review found that solicitation documents and actions taken by ECCC during the solicitation process were, in the majority of cases, consistent with applicable rules. However, several issues regarding clarity and consistency of instructions to bidders, documentation and the communication of information to suppliers were identified. Certain files contained incomplete records and did not demonstrate that relevant information had been shared with suppliers to ensure fairness and encourage competition.
8. Regarding LOE 3, OPO’s initial review found that evaluation of bids and contract award in a majority of files were performed in accordance with the solicitation. However, several evaluations were not consistently carried out in accordance with the planned approach, including 2 contracts that were awarded to bidders that should have been deemed non-responsive. File documentation was also found to be incomplete in several files.
9. In order to address the issues identified during the initial review, the Procurement Ombudsman made 4 recommendations to ECCC.
10. In October 2022, OPO asked ECCC to self-assess their progress in implementing the initial review’s 4 recommendations using an established progress scale, ranging from “no progress” (level 1) to “full implementation” (level 5), as outlined in Appendix 1. ECCC was also asked to provide supplementary information and documentation to explain and support their self-assessment.
11. When reviewing departments’ self-assessments, OPO may conduct additional testing of procurement files, if required, to obtain evidence that demonstrates structures and processes have been implemented and positive results have been identified. During this review, it was determined that additional testing of procurement files was not necessary because ECCC’s self-assessment and supporting documentation provided sufficient evidence regarding the actions taken to address the recommendations noted in the initial review.
12. OPO reviewed and assessed the information provided by ECCC in its self-assessment for its overall reasonableness and credibility. This report also includes a summary, a report card and an overall conclusion regarding progress made by ECCC in implementing the recommendations made by the Procurement Ombudsman in the initial report.
Management Action Plan Assessment Summary
13. Using the scale provided by OPO, ECCC self-assessed the level of implementation of its actions at a level 5, that is “Full implementation”, for all 4 recommendations in the initial report. However, substantiating documentation provided by ECCC demonstrated structures and processes being fully implemented for 3 of the 4 recommendations, namely recommendation 2, 3, and 4, but not for recommendation 1. Therefore, OPO is not in full agreement with ECCC’s self-assessment, and assessed ECCC’s level of implementation at a level 4 (“substantial implementation”) for recommendation 1.
14. Overall, ECCC developed and implemented new and enhanced products and initiatives that support fairness, openness, and transparency in the procurement process, and meet the intent of each recommendation. These initiatives included a mandatory training program with enhanced tools that were shared with procurement officers via professional development activities, including information sessions and focus groups. As well, ECCC implemented a revised Quality Assurance (QA) program, with additional management oversight and an independent monitoring function. The revised QA program and the mandatory training program for procurement officers were comprehensive enough to cover the intent of multiple recommendations.
15. To further support the effectiveness of its initiatives, ECCC should formally monitor attendance at its mandatory training program to ensure training is completed by all required staff. This additional measure would assist in the mitigation of future procurement risks and help ensure that the intent of recommendation 1, i.e., establishing a mechanism to ensure clarity and accuracy in evaluation criteria, selection methodologies and bidder instructions to avoid discrepancies in its bidding processes, is fully met. Additional details of OPO’s overall assessment of ECCC’s response to each recommendation are included in Appendix 3.
16. PPR follow-up reviews include a report card with a rating that depicts the organization’s performance with regard to the 3 LOEs, resulting in 3 assessments. In determining these assessment ratings, OPO takes into consideration the results from the initial review and the actions taken by the organization to implement the recommendations under each LOE. The assessment ratings are as follows:
- Satisfactory +
- Partially Satisfactory
17. Note that a Satisfactory + assessment rating is only possible when the initial review resulted in no recommendations being issued under a particular LOE. As the initial review contained recommendations under all 3 LOEs, a Satisfactory + rating was not possible in this follow-up review, and the highest possible score ECCC could obtain for each LOE was Satisfactory. Assessment definitions and criteria are outlined in Appendix 2.
After a review of progress made toward implementing the action plans for each recommendation and in consideration of the established assessment ratings, a rating of Satisfactory was determined for each LOE. ECCC’s descriptions of actions taken and its supporting documentation demonstrate that reasonable and credible progress has been achieved in response to all recommendations.
|Line of Enquiry (LOE)||Rating||Assessment|
Evaluation criteria and selection plans were established in accordance with applicable laws, regulations and policies.
Solicitation documents and organizational practices during the bid solicitation period were consistent with applicable laws, regulations and policies.
Evaluation of bids and contract award were conducted in accordance with the solicitation.
Table 1 Note
ECCC’s overall report card rating is “Satisfactory”. The key factors contributing to this rating include:
ECCC’s response is supported with documentation, but the lack of formal monitoring of training attendance precludes the achievement of level 5 “full implementation” for recommendation 1.
18. ECCC has implemented new initiatives and enhanced existing training, policies, and processes that support fairness, openness, and transparency in the procurement process. The evidence provided demonstrates a series of advancements designed to meet the intent of each recommendation stemming from the initial review.
19. ECCC provided supporting documentation demonstrating that it has considered the Procurement Ombudsman’s recommendations from the initial review and has taken the necessary action to respond to these recommendations. ECCC implemented a comprehensive mandatory training program for procurement officers and communicated the changes to procurement processes and tools via multiple channels including written direction and through professional development programs. However, attendance or a completion rate for the mandatory training program was not tracked.
20. The follow-up review concluded that ECCC’s self-assessment is reasonable and credible. As such, OPO accepts most aspects of ECCC’s self-assessment and concludes that the progress noted is a combination of level 4 “substantial implementation” and 5 “full implementation”.
21. Finally, the Procurement Ombudsman is encouraged by the progress made to date and takes the opportunity to commend ECCC for its commitment to supporting fairness, openness, and transparency in the procurement process.
|Level 1||No progress or insignificant progress. Actions such as establishing a new committee, conducting meetings, and generating informal plans have not advanced, or insignificant progress has been made.|
|Level 2||Planning stage. Formal plans for organizational changes have been created and approved.|
|Level 3||Preparations for implementation. Preparations for implementing a recommendation are in progress—for example hiring or training staff, developing necessary resources, etc.|
|Level 4||Substantial implementation. Structures and processes are in place and integrated within at least most parts of the organization, and some achieved results have been identified.|
|Level 5||Full implementation. Structures and processes are in place and integrated within at least some parts of the organization, and some achieved results have been identified.|
|Obsolete||Recommendation is no longer applicable due to new policies, procedures, etc.|
|1. ECCC should establish a mechanism to ensure clarity and accuracy in evaluation criteria, selection methodologies and bidder instructions to avoid discrepancies in its bidding processes||
ECCC activities included the development of a new mandatory training program and an improved management review for a bid solicitation process. ECCC designed and delivered a new mandatory and comprehensive training package, composed of 12 training modules, for procurement officers during 2021. The training program covered topics such as the statements of work, evaluation criteria, selection methodologies, and key aspects of bid solicitation. Other training topics such as contract management, trade agreements, managing client expectations or the basis and method of payment, etc., were developed in addition to the list of training topics identified in the initial action plan. Evidence showed that the training program was well received from procurement officers and that lessons learned were identified following its completion. However, the lack of formal attendance monitoring for the mandatory training should be corrected to ensure training is completed by all required staff.
In addition, the revised management review for bid solicitation ensures that all RFPs, whether through limited or electronic tendering, are reviewed at the appropriate level (for example, PG-04 or PG-05) as per the assigned financial thresholds. It also provides for additional levels of review for higher dollar bid solicitations (for example over $1M), which are now reviewed by the QA team.
|Level 4—Substantial implementation|
|2. ECCC should establish mechanisms to: 1) ensure that relevant information is shared with all suppliers simultaneously; and 2) ensure all relevant communications with suppliers are properly documented||ECCC activities included training for procurement officers addressing the importance of sharing information with suppliers simultaneously, and a revised QA framework. Evidence showed that ECCC’s mandatory training program emphasized the importance of properly documenting every stage of the solicitation process, documenting every decision of business value, and maintaining up-to-date and complete procurement files. Furthermore, the revised QA framework, launched in March 2022, was designed to ensure procurement legislation, directives, guidelines, policies and best practices are being followed, and to meet the recommendations of central agency reviews. The QA Framework provides an overview of the various processes that are carried out during all stages of the procurement process including pre-solicitation, pre-contract award, and post-contract award, and by relocating the functions from Procurement and Contracting Division to the Policy Team should ensure efficiency and a more independent QA function. The File Documentation Checklist and the Quality Assurance Review Checklist for competitive requirements also addressed the requirement that communications with suppliers be properly documented on file and facilitate this task for the procurement officers. Finally, the QA team is required to produce a quarterly summary report and an annual validation exercise which further support proper file documentation.||Level 5—Full implementation|
|3. ECCC should establish mechanisms to ensure bid evaluations: 1) adhere strictly to the evaluation criteria in solicitations; 2) are carried out in accordance with planned approaches; and 3) are appropriately documented||ECCC activities included evaluation guidelines and instructions to be shared with evaluators prior to release of bids; training for procurement officers on the evaluation process; and the need to award contracts in accordance with planned approaches. Enhanced guidance such as Bid Evaluation Guide, Evaluation Team Basic Guidelines or the Bid Evaluation Basis of Selection template address this recommendation and demonstrate that ECCC has implemented new evaluation guidelines and instructions to be shared with evaluators prior to release of bids. In addition, ECCC provided training to procurement officers on the evaluation process and the need to award contracts in accordance with planned approaches as part of its mandatory training program for procurement officers conducted during 2021.||Level 5—Full implementation|
|4. ECCC should establish a mechanism to enforce the requirement to document every decision of business value and maintain up-to-date and complete procurement files||ECCC activities included a revised quality assurance and monitoring program, a more comprehensive (a non-mandatory) File Documentation Checklist, and Quality Assurance Review Checklists for competitive requirements (mandatory) and non-competitive requirements (mandatory above a certain threshold). The follow-up review found that the initiatives implemented properly address this recommendation. The revised QA program supports proper file documentation by making QA functions mandatory throughout all the procurement stages and enhancing the monitoring functions. As noted, the QA functions were moved under the Policy Team, an independent team which is likewise responsible for producing a quarterly summary report and an annual validation exercise. In addition, a Fairness Monitor can be employed as needed to provide independent assurance that specific procurements are conducted in a fair, open and transparent manner. Also the File Documentation Checklist, and Quality Assurance Review Checklists for competitive and non-competitive requirements support procurement officers in ensuring that every decision of business value is documented and that procurement files are complete and up-to-date. Collectively, the actions taken substantially address the intent of Recommendation 4 which puts the emphasis on the mechanism that enforces proper documentation, up-to-date, and complete procurement files.||Level 5—Full implementation|
- Date modified: