Follow-up report to the 2019-2020 procurement practice review of Employment and Social Development Canada

March 2024

On this page

Introduction

1. As a standard practice, the Office of the Procurement Ombud (OPO) follows up on the recommendations made in Procurement Practice Reviews (PPRs) approximately two years after those recommendations are published.

2. Follow-up reviews allow OPO to assess whether departmental actions responded to PPR recommendations. They also provide an organizational report card that facilitates comparisons across organizations over time.

3. In 2019-2020 OPO conducted a PPR entitled Procurement Practice Review of Employment and Social Development Canada (ESDC). The final report was issued in October 2020.

4. The initial review was undertaken to determine whether ESDC’s procurement practices pertaining to evaluation criteria and selection plans, solicitation, and evaluation of bids and contract award, supported the principles of fairness, openness and transparency. To make this determination, OPO examined whether ESDC’s procurement practices were consistent with Canada’s obligations under applicable sections of national and international trade agreements, the Financial Administration Act and regulations made under it, the Treasury Board Contracting Policy and, when present, departmental guidelines.

5. The following three lines of inquiry (LOE) were used to assess the highest-risk procurement elements:

  • LOE 1: Evaluation criteria and selection plans were established in accordance with applicable laws, regulations and policies
  • LOE 2: Solicitation documents and organizational practices during the bid solicitation period were consistent with applicable laws, regulations and policies
  • LOE 3: Evaluation of bids and contract award were conducted in accordance with the solicitation

6. Regarding LOE 1, which reviewed evaluation criteria and selection plans, OPO’s initial review found that the evaluation criteria were sometimes not communicated in a clear, precise and measurable manner. Several instances were noted in which the use of ambiguous or undefined terms may have impacted transparency by hampering the ability of bidders to determine how evaluation factors would be used to determine the successful bid, as well as the ability of evaluators to apply the evaluation factors as intended.

7. Regarding LOE 2, which reviewed solicitation documents and ESDC’s actions during the solicitation process, OPO found that the design and execution of competitive solicitation processes were mostly consistent with applicable rules, regulations and policies.

8. Regarding LOE 3, which reviewed the evaluation of bids and contract award, OPO found that while ESDC had established a standard process for evaluating proposals complete with detailed guidance for evaluators, evaluations were not consistently carried out in accordance with the planned approach.

9. In order to address the issues identified during the initial review, the Procurement Ombud made 3 recommendations to ESDC, one for each LOE.

Methodology

10. In December 2022, OPO asked ESDC to self-assess progress in implementing the 3 recommendations from the initial review using an established progress scale, ranging from “no progress” (level 1) to “full implementation” (level 5), as outlined in Appendix 1. ESDC was also asked to provide information and documentation to explain and support their self-assessment.

11. When reviewing departments’ self-assessments during a follow-up exercise, OPO may conduct additional testing of procurement files, if required, to obtain evidence that demonstrates that structures and processes have been implemented and positive results have been identified. During this review, it was determined that additional testing of procurement files was not necessary because ESDC provided sufficient documentation to allow OPO to review and assess the actions taken to address the recommendations provided in the initial review.

12. OPO reviewed and assessed the information provided by ESDC in its self-assessment for its overall reasonableness and credibility. This report includes a summary of the assessment of ESDC’s management action plan, a report card, and an overall conclusion regarding progress made by ESDC in implementing the recommendations made by the Procurement Ombud in the initial review.

Management Action Plan Assessment – Summary

13. Using the scale provided by OPO, ESDC self-assessed the level of implementation of its actions at level 4, “Substantial Implementation” for Recommendation 1, and at level 5, “Full Implementation” for Recommendations 2 and 3. However, information and substantiating documentation provided by ESDC did not clearly demonstrate that structures and processes were substantially implemented for Recommendation 1 or fully implemented for Recommendations 2 and 3. Therefore, OPO assessed ESDC’s level of implementation at level 3 “Preparation for Implementation” for all 3 recommendations because positive results have not yet been achieved or identified.

14. Overall, ESDC has updated existing policies and implemented new and enhanced products that support fairness, openness, and transparency in the procurement process. The evidence provided by ESDC demonstrated a series of advancements designed to meet the intent of each recommendation through several initiatives, including: revised Bid Evaluation Instructions and Bid Evaluation Guide; a bid evaluation refresher session; a requirement for evaluators to certify that they comply with all of the requirements set out in the bid evaluation instructions; and an improved electronic filing system. In addition, ESDC stated it is currently implementing a ‘Lessons Learned’ process for complex procurements as a tool to leverage the knowledge gained from previous procurement processes. The updated information and newly developed products are posted on ESDC’s intranet (iService) and shared with procurement officers.

15. Despite these positive advancements, there remains an opportunity for ESDC to adopt better oversight, provide regular training sessions, and enhance the monitoring function around documentation practices to ensure the existing initiatives and practices are implemented. A detailed summary of OPO’s overall assessment of ESDC’s response to each recommendation is included in Appendix 3.

Report Card

16. PPR follow-up reviews include a report card with a rating that depicts the organization’s performance with regard to the LOEs, resulting in 3 assessments. In determining these assessment ratings, OPO takes into consideration the results from the initial review and the actions taken by the organization to implement the recommendations under each LOE. The assessment ratings are as follows:

  • Satisfactory Plus
  • Satisfactory
  • Partially Satisfactory
  • Unsatisfactory

17. Note that a Satisfactory Plus assessment rating is only possible when the initial review resulted in no recommendations being issued under a particular LOE. As the initial review contained recommendations under all 3 LOEs, a Satisfactory Plus rating was not possible in this follow-up review, and the highest possible score ESDC could obtain for each LOE was Satisfactory. Assessment definitions and criteria are outlined in Appendix 2.

18. After the review of the progress made toward implementing the action plans for each recommendation, a rating of “Partially Satisfactory” was awarded for each LOE. ESDC’s descriptions of actions taken and the supporting documentation provided demonstrate that some progress has been achieved in response to all 3 recommendations, but not enough to adequately address the recommendations under each LOE. More oversight, regular training sessions/programs and formal documentation would serve to strengthen ESDC’s efforts and help ensure fair, open and transparent procurement processes. Furthermore it would help ESDC to be better prepared for the scrutiny of future internal audit activity and to defend against external challenges.

Table 1: Report Card
Line of Enquiry (LOE) Rating Assessment
Evaluation criteria and selection plans were established in accordance with applicable laws, regulations and policies. Partially Satisfactory
  • Recommendation 1 applied to LOE 1.
  • A series of initiatives and products were developed, and were in the process of implementation to address this LOE and help to ensure evaluation criteria and selection plans are in compliance with applicable laws, regulations and policies. These include a mandatory training session, updated instructions and guidelines, a new Quality Assurance (QA) Framework, and a Lessons Learned process.
  • Full implementation of the Lessons Learned exercise and the establishment of regular training sessions will further improve the department’s evaluation criteria and selection plans.
Solicitation documents and organizational practices during the bid solicitation period were consistent with applicable laws, regulations and policies Partially Satisfactory
  • Recommendation 2 applied to LOE 2.
  • A series of initiatives and products were developed, and were in the process of implementation to address this LOE, which helps to ensure solicitations are conducted in compliance with laws, regulations and policy requirements. These include the implementation of a mandatory training session, and updated instructions and guidelines.
  • Additional guidance regarding questions and appropriate time extensions, establishment of regular training, and proper documentation to measure the results of certain practices’ implementation suggested in the existing policies will support consistency and compliance in this area.
Evaluation of bids and contract award were conducted in accordance with the solicitation. Partially Satisfactory
  • Recommendation 3 applied to LOE 3.
  • A series of initiatives and products were developed, communicated and were in the process of implementation to address this LOE. These include a mandatory training session, updated instructions and guidelines, a new Quality Assurance (QA) Framework, and upgraded documentation process.
  • Full implementation of the Quality Assurance Framework and the establishment of regular training will enhance the department’s evaluation processes.

Table 1 Note

ESDC’s overall report card rating is “Partially Satisfactory”. The key factors contributing to this rating include:

  • The initial review resulted in one or more recommendations under each of the three LOEs, thus precluding a rating higher than “Satisfactory”;
  • ESDC’s Management Action Plan has achieved a level 3 for all 3 recommendations, meaning that preparations for implementation are in progress;
  • ESDC's response is supported with documentation and some progress has been made in implementing the action plan, but not enough to adequately address all recommendations under the three lines of inquiry.

Conclusion

19. ESDC has implemented some new initiatives to enhance existing processes that support fairness, openness, and transparency in the procurement process. The evidence provided showed that ESDC is in the preparation stage of implementing multiple initiatives designed to meet the intent of each of the 3 recommendations stemming from the initial review.

20. ESDC provided documentation demonstrating that it has considered the Procurement Ombud’s recommendations from the initial review. ESDC’s actions have partially addressed OPO’s recommendations, and its action plan is still being implemented. In order to ensure the necessary long-term improvements in the areas identified under each LOE, ESDC should provide training on a more regular basis, and full implementation of its planned ‘Lessons Learned’ exercises and Quality Assurance Framework should further enhance the fairness, openness, and transparency of ESDC’s procurement processes.

Appendix 1

Table 2: Assessment of implementation level
Implementation level Assessment
Level 1 No progress or insignificant progress. Actions such as establishing a new committee, conducting meetings, and generating informal plans have not advanced, or insignificant progress has been made.
Level 2 Planning stage. Formal plans for organizational changes have been created and approved.
Level 3 Preparations for implementation. Preparations for implementing a recommendation are in progress—for example, hiring or training staff, developing necessary resources, drafting and adopting policy documents, etc.
Level 4 Substantial implementation. Structures and processes are in place and integrated within at least most parts of the organization, and some achieved results have been identified.
Level 5 Full implementation. Structures and processes are fully implemented, and positive results have been identified.
Obsolete Recommendation is no longer applicable due to new policies, procedures, etc.

Appendix 2

Table 3: Overall performance assessment scale
Overall performance Assessment of performance
Satisfactory Plus
  • Findings from the initial PPR resulted in no recommendations being issued under the line of inquiry.
  • A rating of Satisfactory plus is based on results from the initial PPR.
Satisfactory
  • Initial PPR resulted in one or more recommendations under the line of inquiry.
  • Action plan(s) have achieved level 5 implementation or a combination of level 4 and level 5 that is deemed to be satisfactory, that is significant progress has been made to implement action plan(s) that address recommendation(s).
  • Organization's response is supported with adequate documentation and/or testing results.
Partially satisfactory
  • Initial PPR resulted in one or more recommendations under the line of inquiry.
  • Some progress has been made in implementing action plan(s), but not enough to adequately address all recommendations under the line of inquiry, for example one or more recommendations achieved level 3, but work remains to be done.
  • Organization's response is supported with some documentation and/or testing results.
Unsatisfactory
  • Initial PPR resulted in one or more recommendations under the line of inquiry.
  • No or insignificant progress made to implement action plan(s).
  • Inadequate supporting documentation provided or testing does not demonstrate that positive results have been achieved.
  • Recommendation(s) not addressed and remain(s) outstanding.

Appendix 3

Table 4: Summary of recommendations & OPO detailed assessment
Recommendation OPO Assessment Rating
1. ESDC should implement measures to ensure that the existing guidance regarding the development of evaluation criteria is implemented by ESDC’s project authorities, and that ESDC’s Procurement Group exercises the necessary oversight to ensure that evaluation criteria are communicated in a clear, precise, and measurable manner.

ESDC has:

  • offered a bid evaluation refresher session in order to highlight key takeaways and lessons learned based solely on the findings and recommendations made in the initial report,
  • provided instructions and information on how to use existing guidance,
  • described what tools were available, discussed the revisions made to the Bid Evaluation Instructions and the Bid Evaluation Guide.

The mandatory session was made available to all Asset Management, Policy and Procurement employees to help with bid-evaluation questions and to discuss lessons learned. This one and a half hour bid criteria and evaluation refresher session took place once in June 2021. The refresher session aimed to ensure evaluation criteria were developed using existing guidance by ESDC’s project authorities. ESDC will continue to provide the refresher session to all new procurement hires on a yearly basis starting fiscal year of 2023-2024.

The PSPC “Bid Evaluation and Selection Methodologies” training course included in ESDC’s action plan was not available during the 2-year period following the initial review, and no alternative course was suggested for new hires.

Besides what is included in the action plan, ESDC will be implementing a Quality Assurance Framework to address the development of bid evaluation criteria.

In addition, ESDC implemented a Lessons Learned process to leverage the knowledge gained from previous procurement processes. Data collected are compiled and analyzed by ESDC to identify potential gaps and risks in the procurement process. The expanded version of this process will be released in the 2023-2024 fiscal year.

OPO noted that the implementation of the Lessons Learned process is exclusively for complex procurements or upon requests from Senior Managers, and the observations and knowledge obtained from this procedure are retrospective in nature. Therefore, ESDC should consider further developing the Lessons Learned process with additional tools such as an action plan, to extract insights from the process, and translate the collected data into specific steps to improve the fairness, openness and transparency of future procurements.

OPO finds it inadequate to provide the refresher session only once in 2 years, and the lack of alternative training courses for new hires fails to ensure continuous and sustainable improvement in the development of evaluation criteria in a clear, precise and measurable manner. Thus, ESDC should provide the refresher session and training to both procurement officers and new hires on a more regular basis.

ESDC is recommended to carry out additional analysis to ensure the attainment of positive outcomes through the implementation of the action plan.

 Level 3—Preparations for implementation
2. ESDC should implement measures to ensure that communications with suppliers support the preparation of responsive bids, including ensuring that suppliers be provided adequate time to prepare and submit tenders.

ESDC provided a bid evaluation refresher session to help ensure that communications with suppliers support the preparation of responsive bids.

OPO’s review found that the refresher session was made mandatory and available to all the Asset Management, Policy and Procurement (AMPP) employees, however, it was only delivered once in June 2021. ESDC will continue to provide the refresher session to all new procurement hires on a yearly basis starting in fiscal year 2023-2024.

According to the instructor notes, one of the best practices suggested in the information session is “to provide 3 days extension after any amendment”. However, OPO suggests a more flexible approach be adopted as the length of the extension of the bidding period should differ based on the complexity of the amendments and the questions being raised by suppliers. ESDC should consider a more flexible and documented approach concerning bid extension periods to support the preparation of responsive bids.

 Level 3—Preparations for implementation
3. ESDC should implement measures to ensure evaluations are carried out in accordance with the planned approach and are appropriately documented to support the transparency of the award process.

ESDC has:

  • revised the Bid Evaluation Guide to cover the updates and details of each step of the bid evaluation process. The updated version emphasized the Contracting Authorities’ role in overseeing the consensus meeting to ensure that evaluations are performed fairly and in accordance with the solicitation documents.
  • developed and offered a bid evaluation refresher session which discussed the revisions and updates in the guide and instruction document.
  • revised the Bid Evaluation Instructions document to include information on evaluation procedures and potential risks. The Bid Evaluation Instructions were expanded to include the mandatory requirement of having consensus meetings, which must be chaired by the Contracting Authority after individual evaluations are completed. A certifications section, in the form of a checklist, was added to the revised Bid Evaluation Instructions for evaluators to sign and check off before the bid evaluation process.
  • updated the Bid Evaluation Guide and the Bid Evaluation Instructions and shared them with all procurement officers via email, and posted them on ESDC’s intranet (iService).
  • advanced its electronic filing process to ensure proper documentation.

All together, these products partially address the second part of Recommendation 3, which involves ensuring evaluations are appropriately documented to support the transparency of the award process. However, OPO’s review found that existing products/ policies are insufficient in ensuring that evaluations are aligned with the planned approach.

As noted above regarding both LOE 1 and LOE 2, the bid evaluation refresher session was made mandatory and available to all the Asset Management, Policy and Procurement (AMPP) employees, however, it was only delivered once during the month of June 2021. ESDC will continue to provide the refresher session to all new procurement hires on a yearly basis starting fiscal year of 2023-2024.

Training is important for procurement and project authorities to develop and improve their procurement skills and expertise, and to ensure that procurement officers have the required knowledge to carry out evaluations in accordance with the planned approach. Failing to properly train employees can have a detrimental effect on the transparency of the award process. Therefore, providing only a one and a half hour refresher session every 2 years is not sufficient to ensure long-term and ongoing development and improvements in the evaluation process. In addition, the simple use of a checklist may not guarantee that the evaluation has been carried out in a thorough and consistent manner in accordance with the planned approach.

Electronic filing was advanced to both promote and secure appropriate documentation of evaluation records, which helps to support the transparency of the award process. Guidelines were updated to reflect changes in documentation processes and to provide an outline of how files should be structured. Additional guidance on how to maintain good document management practices was shared in February 2023 to further enhance the implementation of electronic filing.

ESDC is recommended to carry out additional analysis to ensure the attainment of positive outcomes through the implementation of the action plan.

 Level 3—Preparations for implementation
Date modified: