Follow-up report to the 2019-20 procurement practice review of Environment and Climate Change Canada

October 2023

On this page

Introduction

1. As a standard practice, the Office of the Procurement Ombudsman (OPO) follows up on the recommendations made in Procurement Practice Reviews (PPRs) two years after those recommendations are published.

2. Follow-up reviews allow OPO to provide assurance that actions to improve federal procurement processes in response to PPR recommendations have been effectively implemented, or that senior management has accepted the risk of not taking action; and provide an organizational report card that facilitates comparisons across organizations over time.

3. In 2019-2020 OPO conducted a PPR entitled Procurement Practice Review of Environment and Climate Change Canada (ECCC). The final report was issued in July 2020.

4. The initial review was undertaken to determine whether ECCC’s procurement practices pertaining to evaluation criteria and selection plans, solicitation, and evaluation of bids and contract award, supported the principles of fairness, openness and transparency. To make this determination, OPO examined whether ECCC’s procurement practices were consistent with Canada’s obligations under applicable sections of national and international trade agreements, the Financial Administration Act and regulations made under it, the Treasury Board Contracting Policy and, when present, departmental guidelines.

5. The following three lines of enquiry (LOE) were used to assess the highest-risk procurement elements:

  • LOE 1: Evaluation criteria and selection plans were established in accordance with applicable laws, regulations and policies;
  • LOE 2: Solicitation documents and organizational practices during the bid solicitation period were consistent with applicable laws, regulations and policies; and
  • LOE 3: Evaluation of bids and contract award were conducted in accordance with the solicitation.

6. Regarding LOE 1, OPO’s initial review found that evaluation criteria and selection plans met requirements set out in applicable laws, regulations and policies in the majority of procurements reviewed. In all files reviewed, mandatory criteria were not overly restrictive and did not unnecessarily favour or penalize any particular bidder. Most mandatory and point-rated criteria were clearly communicated in a manner supporting a fair and transparent procurement process. However, in several files, issues were identified regarding unclear or inconsistent evaluation criteria, instructions for awarding points for point-rated criteria and selection methodologies. These issues can lead to incorrect bids and improper evaluations, thus impacting the fairness of the procurement process.

7. Regarding LOE 2, OPO’s initial review found that solicitation documents and actions taken by ECCC during the solicitation process were, in the majority of cases, consistent with applicable rules. However, several issues regarding clarity and consistency of instructions to bidders, documentation and the communication of information to suppliers were identified. Certain files contained incomplete records and did not demonstrate that relevant information had been shared with suppliers to ensure fairness and encourage competition.

8. Regarding LOE 3, OPO’s initial review found that evaluation of bids and contract award in a majority of files were performed in accordance with the solicitation. However, several evaluations were not consistently carried out in accordance with the planned approach, including 2 contracts that were awarded to bidders that should have been deemed non-responsive. File documentation was also found to be incomplete in several files.

9. In order to address the issues identified during the initial review, the Procurement Ombudsman made 4 recommendations to ECCC.

Methodology

10. In October 2022, OPO asked ECCC to self-assess their progress in implementing the initial review’s 4 recommendations using an established progress scale, ranging from “no progress” (level 1) to “full implementation” (level 5), as outlined in Appendix 1. ECCC was also asked to provide supplementary information and documentation to explain and support their self-assessment.

11. When reviewing departments’ self-assessments, OPO may conduct additional testing of procurement files, if required, to obtain evidence that demonstrates structures and processes have been implemented and positive results have been identified. During this review, it was determined that additional testing of procurement files was not necessary because ECCC’s self-assessment and supporting documentation provided sufficient evidence regarding the actions taken to address the recommendations noted in the initial review.

12. OPO reviewed and assessed the information provided by ECCC in its self-assessment for its overall reasonableness and credibility. This report also includes a summary, a report card and an overall conclusion regarding progress made by ECCC in implementing the recommendations made by the Procurement Ombudsman in the initial report.

Management Action Plan Assessment Summary

13. Using the scale provided by OPO, ECCC self-assessed the level of implementation of its actions at a level 5, that is “Full implementation”, for all 4 recommendations in the initial report. However, substantiating documentation provided by ECCC demonstrated structures and processes being fully implemented for 3 of the 4 recommendations, namely recommendation 2, 3, and 4, but not for recommendation 1. Therefore, OPO is not in full agreement with ECCC’s self-assessment, and assessed ECCC’s level of implementation at a level 4 (“substantial implementation”) for recommendation 1.

14. Overall, ECCC developed and implemented new and enhanced products and initiatives that support fairness, openness, and transparency in the procurement process, and meet the intent of each recommendation. These initiatives included a mandatory training program with enhanced tools that were shared with procurement officers via professional development activities, including information sessions and focus groups. As well, ECCC implemented a revised Quality Assurance (QA) program, with additional management oversight and an independent monitoring function. The revised QA program and the mandatory training program for procurement officers were comprehensive enough to cover the intent of multiple recommendations.

15. To further support the effectiveness of its initiatives, ECCC should formally monitor attendance at its mandatory training program to ensure training is completed by all required staff. This additional measure would assist in the mitigation of future procurement risks and help ensure that the intent of recommendation 1, i.e., establishing a mechanism to ensure clarity and accuracy in evaluation criteria, selection methodologies and bidder instructions to avoid discrepancies in its bidding processes, is fully met. Additional details of OPO’s overall assessment of ECCC’s response to each recommendation are included in Appendix 3.

Report Card

16. PPR follow-up reviews include a report card with a rating that depicts the organization’s performance with regard to the 3 LOEs, resulting in 3 assessments. In determining these assessment ratings, OPO takes into consideration the results from the initial review and the actions taken by the organization to implement the recommendations under each LOE. The assessment ratings are as follows:

  • Satisfactory +
  • Satisfactory
  • Partially Satisfactory
  • Unsatisfactory

17. Note that a Satisfactory + assessment rating is only possible when the initial review resulted in no recommendations being issued under a particular LOE. As the initial review contained recommendations under all 3 LOEs, a Satisfactory + rating was not possible in this follow-up review, and the highest possible score ECCC could obtain for each LOE was Satisfactory. Assessment definitions and criteria are outlined in Appendix 2.

After a review of progress made toward implementing the action plans for each recommendation and in consideration of the established assessment ratings, a rating of Satisfactory was determined for each LOE. ECCC’s descriptions of actions taken and its supporting documentation demonstrate that reasonable and credible progress has been achieved in response to all recommendations.

Table 1: Report Card
Line of Enquiry (LOE) Rating Assessment

Evaluation criteria and selection plans were established in accordance with applicable laws, regulations and policies.

 Satisfactory
  • Recommendations 1 and 4 applied to this LOE.
  • In response to these recommendations ECCC developed an action plan which included a revised training module for procurement officers and a revised management review process for bid solicitation.
  • ECCC addressed the first part of Recommendation 1 by making the revised training program mandatory for all existing and new procurement officers. 12 mandatory training modules were delivered to procurement officers during 2021. According to ECCC all existing procurement officers were invited and completed the mandatory training program. However, ECCC is unable to confirm the attendance/ completion rate for this mandatory training due to the lack of a formal tracking or monitoring mechanism. A formal monitoring of training attendance should be implemented to ensure training is completed by all required staff.
  • The new training program builds on the priority areas identified in the action plan and covers all the stages of the solicitation process including planning, procurement strategy, developing a statement of work and evaluation criteria. It provides procurement officers with the necessary knowledge and tools to ensure clarity and accuracy in evaluation criteria, selection methodologies and bidder instructions with the purpose of avoiding discrepancies in its solicitation documents.
  • ECCC addressed the second part of Recommendation 1 by implementing the revised management review process for the bid solicitation process, including:
    • all Requests For Proposals (RFP) are reviewed at the assigned level;
    • increased management accountability in accordance with financial thresholds at each stage of the solicitation process;
    • an electronic review system, where management approval is needed to move to the next step following a review process to ensure that evaluation criteria and selection plans are respected; and
    • reassigned QA responsibilities from the Procurement and Contracting Division to the Policy Team, to enhance the independence of the quality review process.
  • In addition to the above, ECCC created a mandatory QA checklist for competitive and non-competitive processes (above financial thresholds), and having the checklist integrated in the electronic review system, ensures that: procurement officers follow the process according to the plan; all documentation is on the file; and management sign-off on the file to ensure that laws, regulations and policies are properly followed.
  • Collectively, the above initiatives meet the intent of Recommendation 1, that is establishing a mechanism to ensure clarity and accuracy in evaluation criteria, selection methodologies and bidder instructions to avoid discrepancies in its bidding processes.

Solicitation documents and organizational practices during the bid solicitation period were consistent with applicable laws, regulations and policies.

 Satisfactory
  • Recommendations 1, 2 and 4 applied to this LOE.
  • ECCC designed, communicated, and delivered several actions to address this LOE, including a new mandatory training program for procurement officers, a revised QA framework and monitoring process, QA checklists for competitive and non-competitive requirements, and a revised File Documentation Checklist. It should be noted that the File Documentation Checklist was developed as a guide to ensure proper documentation that supports decisions made during the procurement process and that all relevant documentation is uploaded to the corresponding file.
  • Both new and revised tools were designed and implemented. Changes to internal policies and procedures were documented and communicated to procurement officers via multiple channels, including in writing and through training and information sessions.
  • Roles and responsibilities pertaining to matters of QA and monitoring process have been properly defined, documented, and communicated.
  • Collectively, these initiatives provide the necessary tools for procurement officers to ensure that relevant information with all suppliers is shared simultaneously and communications with suppliers are properly documented.

Evaluation of bids and contract award were conducted in accordance with the solicitation.

 Satisfactory
  • Recommendations 3 and 4 applied to this LOE.
  • To address these recommendations ECCC designed, communicated, and delivered several actions including new tools, templates and products such as the Evaluation Guide, Evaluation Team Basic Guidelines, Bid Evaluation Basis of Selection template, QA checklists for competitive and non-competitive requirements, and a File Documentation Checklist.
  • A new training module on bid evaluation and contractor selection methodology was designed and delivered to procurement officers.
  • A revised management, monitoring and QA process was put in place, combined with QA functions now relocated within the Policy Team to ensure independence.
  • The new QA team must produce a quarterly summary report and an annual validation exercise which allows ECCC to ensure that the evaluation of bids and contract award are conducted in accordance with the planned approach, and supports continuous improvement.
  • Collectively, the above initiatives better equip procurement officers to ensure that bid evaluations adhere strictly to evaluation criteria, are carried out based on the planned approach and are properly documented.

Table 1 Note

ECCC’s overall report card rating is “Satisfactory”. The key factors contributing to this rating include:

  • The initial review resulted in one or more recommendations under each of the three LOEs, thus precluding a rating higher than Satisfactory;
  • Action plan(s) have achieved level 5 implementation or a combination of level 4 and level 5 that is deemed to be satisfactory, that is significant progress has been made to implement action plan(s) that address recommendation(s)

ECCC’s response is supported with documentation, but the lack of formal monitoring of training attendance precludes the achievement of level 5 “full implementation” for recommendation 1.

Conclusion

18. ECCC has implemented new initiatives and enhanced existing training, policies, and processes that support fairness, openness, and transparency in the procurement process. The evidence provided demonstrates a series of advancements designed to meet the intent of each recommendation stemming from the initial review.

19. ECCC provided supporting documentation demonstrating that it has considered the Procurement Ombudsman’s recommendations from the initial review and has taken the necessary action to respond to these recommendations. ECCC implemented a comprehensive mandatory training program for procurement officers and communicated the changes to procurement processes and tools via multiple channels including written direction and through professional development programs. However, attendance or a completion rate for the mandatory training program was not tracked.

20. The follow-up review concluded that ECCC’s self-assessment is reasonable and credible. As such, OPO accepts most aspects of ECCC’s self-assessment and concludes that the progress noted is a combination of level 4 “substantial implementation” and 5 “full implementation”.

21. Finally, the Procurement Ombudsman is encouraged by the progress made to date and takes the opportunity to commend ECCC for its commitment to supporting fairness, openness, and transparency in the procurement process.

Appendix 1

Table 2: Assessment of implementation level
Implementation level Assessment
Level 1 No progress or insignificant progress. Actions such as establishing a new committee, conducting meetings, and generating informal plans have not advanced, or insignificant progress has been made.
Level 2 Planning stage. Formal plans for organizational changes have been created and approved.
Level 3 Preparations for implementation. Preparations for implementing a recommendation are in progress—for example hiring or training staff, developing necessary resources, etc.
Level 4 Substantial implementation. Structures and processes are in place and integrated within at least most parts of the organization, and some achieved results have been identified.
Level 5 Full implementation. Structures and processes are in place and integrated within at least some parts of the organization, and some achieved results have been identified.
Obsolete Recommendation is no longer applicable due to new policies, procedures, etc.

Appendix 2

Table 3: Overall Performance Assessment Scale
Overall performance Assessment
Satisfactory +
  • Findings from the initial PPR resulted in no recommendations being issued under the line of enquiry
  • A rating of Satisfactory + is based on results from the initial PPR
Satisfactory
  • Initial PPR resulted in one or more recommendations under the line of enquiry
  • Action plan(s) have achieved level 5 implementation or a combination of level 4 and level 5 that is deemed to be satisfactory, that is significant progress has been made to implement action plan(s) that address recommendation(s)
  • Organization's response is supported with adequate documentation and/or testing results
Partially satisfactory
  • Initial PPR resulted in one or more recommendations under the line of enquiry
  • Some progress has been made in implementing action plan(s), but not enough to adequately address all recommendations under the line of enquiry, for example one or more recommendations achieved level 3, but work remains to be done
  • Organization's response is supported with some documentation and/or testing results
Unsatisfactory
  • Initial PPR resulted in one or more recommendations under the line of enquiry
  • No or insignificant progress made to implement action plan(s)
  • Inadequate supporting documentation provided or testing does not demonstrate that positive results have been achieved
  • Recommendation(s) not addressed and remain(s) outstanding.

Appendix 3

Table 4: Assessment of Recommendations
Recommendation OPO Assessment Rating
1. ECCC should establish a mechanism to ensure clarity and accuracy in evaluation criteria, selection methodologies and bidder instructions to avoid discrepancies in its bidding processes

ECCC activities included the development of a new mandatory training program and an improved management review for a bid solicitation process. ECCC designed and delivered a new mandatory and comprehensive training package, composed of 12 training modules, for procurement officers during 2021. The training program covered topics such as the statements of work, evaluation criteria, selection methodologies, and key aspects of bid solicitation. Other training topics such as contract management, trade agreements, managing client expectations or the basis and method of payment, etc., were developed in addition to the list of training topics identified in the initial action plan. Evidence showed that the training program was well received from procurement officers and that lessons learned were identified following its completion. However, the lack of formal attendance monitoring for the mandatory training should be corrected to ensure training is completed by all required staff.

In addition, the revised management review for bid solicitation ensures that all RFPs, whether through limited or electronic tendering, are reviewed at the appropriate level (for example, PG-04 or PG-05) as per the assigned financial thresholds. It also provides for additional levels of review for higher dollar bid solicitations (for example over $1M), which are now reviewed by the QA team.

 Level 4—Substantial implementation
2. ECCC should establish mechanisms to: 1) ensure that relevant information is shared with all suppliers simultaneously; and 2) ensure all relevant communications with suppliers are properly documented ECCC activities included training for procurement officers addressing the importance of sharing information with suppliers simultaneously, and a revised QA framework. Evidence showed that ECCC’s mandatory training program emphasized the importance of properly documenting every stage of the solicitation process, documenting every decision of business value, and maintaining up-to-date and complete procurement files. Furthermore, the revised QA framework, launched in March 2022, was designed to ensure procurement legislation, directives, guidelines, policies and best practices are being followed, and to meet the recommendations of central agency reviews. The QA Framework provides an overview of the various processes that are carried out during all stages of the procurement process including pre-solicitation, pre-contract award, and post-contract award, and by relocating the functions from Procurement and Contracting Division to the Policy Team should ensure efficiency and a more independent QA function. The File Documentation Checklist and the Quality Assurance Review Checklist for competitive requirements also addressed the requirement that communications with suppliers be properly documented on file and facilitate this task for the procurement officers. Finally, the QA team is required to produce a quarterly summary report and an annual validation exercise which further support proper file documentation. Level 5—Full implementation
3. ECCC should establish mechanisms to ensure bid evaluations: 1) adhere strictly to the evaluation criteria in solicitations; 2) are carried out in accordance with planned approaches; and 3) are appropriately documented ECCC activities included evaluation guidelines and instructions to be shared with evaluators prior to release of bids; training for procurement officers on the evaluation process; and the need to award contracts in accordance with planned approaches. Enhanced guidance such as Bid Evaluation Guide, Evaluation Team Basic Guidelines or the Bid Evaluation Basis of Selection template address this recommendation and demonstrate that ECCC has implemented new evaluation guidelines and instructions to be shared with evaluators prior to release of bids. In addition, ECCC provided training to procurement officers on the evaluation process and the need to award contracts in accordance with planned approaches as part of its mandatory training program for procurement officers conducted during 2021. Level 5—Full implementation
4. ECCC should establish a mechanism to enforce the requirement to document every decision of business value and maintain up-to-date and complete procurement files ECCC activities included a revised quality assurance and monitoring program, a more comprehensive (a non-mandatory) File Documentation Checklist, and Quality Assurance Review Checklists for competitive requirements (mandatory) and non-competitive requirements (mandatory above a certain threshold). The follow-up review found that the initiatives implemented properly address this recommendation. The revised QA program supports proper file documentation by making QA functions mandatory throughout all the procurement stages and enhancing the monitoring functions. As noted, the QA functions were moved under the Policy Team, an independent team which is likewise responsible for producing a quarterly summary report and an annual validation exercise. In addition, a Fairness Monitor can be employed as needed to provide independent assurance that specific procurements are conducted in a fair, open and transparent manner. Also the File Documentation Checklist, and Quality Assurance Review Checklists for competitive and non-competitive requirements support procurement officers in ensuring that every decision of business value is documented and that procurement files are complete and up-to-date. Collectively, the actions taken substantially address the intent of Recommendation 4 which puts the emphasis on the mechanism that enforces proper documentation, up-to-date, and complete procurement files. Level 5—Full implementation
Date modified: