Procurement practice review: Review of bid evaluation processes

November 2016
Promoting Fairness, Openness and Transparency in Federal Procurement

Introduction

Context

1. Paragraph 22.1(3)(a) of the Department of Public Works and Government Services Act provides the Procurement Ombudsman with the authority to review the procurement practices of departments to assess their fairness, openness and transparency.
2. The Office of the Procurement Ombudsman (OPO) assesses all written complaints, as well as all issues brought to the OPO’s attention via procurement-related contacts, in order to determine whether they raise potential systemic issues or risks to fairness, openness and transparency. All issues are classified by the OPO using key elements of the procurement process.
3. An analysis of these issues, completed as part of the annual process to develop the Office’s Procurement Practice Review (PPR) Plan, identified several potential issues regarding bid evaluation processes used to acquire goods and services.
4. A risk assessment of the issues classified by the OPO under the bid evaluation process elements was conducted. The risk assessment considered the likelihood of the risk occurring and the impact on fairness, openness and transparency. The assessment of the potential issues within the bid evaluation process elements identified that the highest risks to fairness, openness and transparency were:
   • biased, restrictive and/or excessive criteria that may limit competition and create real or perceived instances of favouritism;
   • unclear, vague and/or inconsistent criteria that may cause confusion and uncertainty, which can result in an increased need for questions and answers, and which may limit competition; and
   • bids not evaluated according to evaluation and selection plans which may have resulted in unfair or subjective evaluation processes, and contracts being awarded to an incorrect or an unqualified supplier.

5. In addition, since 2011-12, the bid evaluation process has been among the top five issues raised by the OPO; in 2014-15, evaluation criteria and the evaluation of bids were the second and third top issues, respectively. The ongoing nature and volume of issues brought to the OPO’s attention suggests that current and potential suppliers may not always be treated fairly or equally throughout the bid evaluation process.
6. The analysis conducted suggested there may be issues with the bid evaluations processes used to acquire goods and services. Consequently, and as per Subsection 4(1) of the Regulations, the Procurement Ombudsman determined there were reasonable grounds to review bid evaluation processes.

Background

7. Section 2 of the Treasury Board (TB) Contracting Policy states that government contracting must be conducted in a manner that will “stand the test of public scrutiny in matters of prudence and probity, facilitate access, encourage competition, and reflect fairness in the spending of public funds.”
8. As this principle applies to the bid evaluation process, departments must establish the evaluation criteria to be used to select the successful bid prior to soliciting bids. These criteria must be recorded and applied equally to all bidders in order for an evaluation to be conducted in a fair, open and transparent manner.
9. For the purpose of this review, the bid evaluation process included: the development of evaluation criteria; selection of the evaluation rating system; and the contractor selection methods used to determine the successful bid.

Review objective and criteria

10. The objective of this procurement practice review was to determine whether departments’ bid evaluation processes were conducted in a manner consistent with applicable sections of the TB Contracting Policy, the Financial Administration Act and regulations made under it, and to support the principles of fairness, openness and transparency, which are defined by the OPO as follows:
    • Fairness: Providing equal treatment to all current and potential suppliers
    • Openness: Providing all potential suppliers with the opportunity to submit bids for government procurement
    • Transparency: Providing information to Canadians in a timely manner that facilitates public scrutiny of the decisions made and actions undertaken

11. The bid evaluation processes of select departments were reviewed using pre-established criteria. Departmental practices were assessed against these criteria which, when in place and operating as intended, should individually and collectively contribute to bid evaluation processes that respect the principles of fairness, openness and transparency:
    • Departments have guidance in place to support bid evaluation processes, consistent with TB Contracting Policy.
    • Bid evaluation criteria are established as per applicable policies and guidance.
    • Bid evaluations and supplier selection are conducted as per applicable policies and guidance.

Scope and methodology

12. The scope of the review included competitive contracts for goods and services up to $2 million from April 1, 2014, to November 30, 2015. Construction contracts and procurements undertaken using acquisition cards were excluded.
13. The following departments were included in the scope of the review as they were identified as a top 10 purchasing department, as per the Purchasing Activity Report for 2014, and were a top 10 department in terms of supplier contacts to the OPO specifically related to bid evaluation. In addition, these organizations had not been subject to an OPO practice review in the past five years:
    • Canada Border Services Agency (CBSA)
    • Parks Canada (Parks)
    • Royal Canadian Mounted Police (RCMP)
14. The OPO requested these three departments provide contracting data for all contracts issued within the scope of the review. The following is a summary of the data provided:

The OPO requested these three departments provide contracting data for all contracts issued within the scope of the review. The following is a summary of the data provided:

Department	Number of Contracts	Value of Contracts ($)
CBSA	404	15,786,643
Parks	9,159	143,081,088
RCMP	3,548	66,812,674
Totals	13,111	225,680,450

15. The review methodology included:
    • interviews with departmental staff;
    • review and analysis of documentation, such as internal policies, guidelines and directives and authority delegations; and
    • review and analysis of selected procurement files.

Why this is important

16. When tendering a contract, the Government of Canada has a duty to conduct a fair, open and transparent competition. Purchasing departments are expected “to treat all bidders fairly and equally to avoid any preferential or unfavourable treatment of any competing bidders” and to conduct their bid evaluation processes “in a manner that is consistent with the pre-established rules set out in the tender call.” ^{Footnote 1}
17. Section 10.7.27 of the TB Contracting Policy reinforces this principle and states “[t]he principle of applying bid criteria or requirements equally to all bidders is part of Canadian contract law and is applicable to both the public as well as the private sectors. Fairness to all prospective contractors and transparency in the award process are imperative.” The nature of the issues identified by the OPO suggests underlying risks to fairness, openness and transparency may exist in bid evaluation processes used by departments to acquire goods or services.

Observations

Bid evaluation guidance

18. The TB Contracting Policy establishes the framework within which contracting authorities must be exercised, including specific sections pertaining to the evaluation of bids. Departmental guidance and processes are to be consistent with the TB Contracting Policy and provide staff with additional direction to support fair, open and transparent bid evaluation processes.
19. The OPO asked departments if they had the following documents:

The OPO asked departments if they had the following documents:

	Departmental Procurement Policy	Bid Evaluation Guidelines/Instructions	Process Maps
Canadian Border Services Agency (CBSA)	✓	✓
Parks Canada (Parks)		✓	✓
Royal Canadian Mounted Police (RCMP)	✓	✓	✓

20. Departmental guidance documents and processes for the selected departments were reviewed and overall were determined to be consistent with applicable sections of the TB Contracting Policy as they relate to bid evaluation. The OPO noted references were made in support of fairness, openness and transparency in each department, for example:
    • CBSA’s Contracting Policy reiterates the TB Contracting Policy objective “to acquire goods and services in a manner that enhances access, competition and fairness” and states “it is the policy of the Agency to encourage transparency in contracting activities;”
    • Parks’ evaluation guide is in place to ensure “the selection process is carried out in a fair and transparent manner;”
    • RCMP’s Asset Management Manual defines principles that guide decisions with respect to the management of procurement services to ensure they are fair, transparent and accessible.

21. As the chart above indicates, CBSA and the RCMP already had formal procurement policies in place. The RCMP had a process in place for recording and updating policies on a yearly basis, or as required if changes necessitated immediate updating. The OPO noted that CBSA’s policy had not been updated since 2005, and was informed a project to update it was currently underway. Parks did not have an overarching departmental procurement policy, but did have other guidelines advising contracting and program authorities to use the Public Works and Government Services Canada (PWGSC) Supply Manual.
22. Bid evaluation guidelines and instructions were found in all three departments. These guidelines were generally clear and contained information to support fair evaluation processes, such as the identification of evaluation team members and conflict of interest declarations. The OPO identified areas for improvement in two departments reviewed: there were multiple versions of bid evaluation plans and scoring sheet templates at CBSA; Parks instructions were unclear as to when the use of an evaluation guide was required. Standardized templates and guidance help support consistent bid evaluation processes. CBSA reported it was undertaking a policy and procedure suite renewal, while Parks reported it would continue to update and make available procurement guidance and supporting documentation.
23. Process maps were found in two departments reviewed. RCMP process maps were included in their procurement policy and outlined decision points and potential additional policy requirements for all procurements. Parks had process maps in place for goods and services procurement processes, as well as architectural and engineering services; however, employees were generally not aware of their existence.
24. The OPO found guidance was provided to both procurement and program staff through various methods, such as staff meetings, communiqués, newsletters and departmental intranet sites. While CBSA reported issues with accessing information due to their intranet being rebuilt, making it unavailable for months, guidelines remained available to procurement staff on the shared drive and were communicated to clients through auto-generated email response messages.
25. The RCMP has a Procurement and Contracting GCpedia page, which has been highlighted as a best practice, and is available to all RCMP staff and other government departments, and includes links to templates and tools, policy guidance and important procurement notices. ^{Footnote 2}

Roles and responsibilities

26. Roles and responsibilities of bid evaluation team members, including both contracting and technical authorities, were documented in bid evaluation guidelines and delegated authorities charts, and were available to staff on intranet sites in all three departments. Evaluators were required to sign evaluation guidelines to acknowledge their responsibilities with regard to conflict of interest and acceptance of evaluation rules. The RCMP’s Asset Management Manual included a grid that clearly outlined the roles and responsibilities for various tasks relating to bid evaluation, ranging from Statement of Work (SOW) development to the evaluation of financial proposals.
27. While the selected departments had clear delegated contracting authorities, the OPO found that Parks had one contracting authority level. Section 11.2.10 of the TB Contracting Policy recommends “[c]ontracting authorities should establish some differentiation of authority levels within the internal delegation of authority. Higher expenditures, complex situations, or very sensitive contracts should receive approval by a more senior level.” In this regard, the horizontal delegated authority structure at Parks is not consistent with the best practices referenced in the TB Contracting Policy and is therefore not consistent with the risk management approach applied across government departments and agencies.

Monitoring and quality assurance

28. Section 5.1.1 of the TB Contracting Policy states it is “the responsibility of departments and agencies to ensure that adequate control frameworks for due diligence and effective stewardship of public funds are in place and working.” A control framework is a set of controls, procedures and systems that play a key role in providing assurance that departmental activities are sufficiently and appropriately risk managed and controlled. These controls include monitoring and quality assurance activities to ensure bid evaluation processes are conducted as per applicable guidelines and policies, and that corrective action is taken when required.
29. The OPO found that monitoring activities, such as peer reviews, were occurring within all departments. At Parks, peer reviews were informal and occurred on an ad hoc basis at the discretion of the contracting authority. At CBSA, peer reviews / self-verifications were encouraged but not mandatory, and had a corresponding checklist of assessment items indicating, for example, whether the selection methodology and criteria were clear and appropriate.
30. The RCMP had a fully implemented Contract Quality Control Program in place, which was recognized as a best practice on the basis of its risk-based approach to monitoring procurement activity, and which included several key components.
    • Contract Review Committee – with pre-established risk-ranked criteria to determine what files should be reviewed.
    • Contract Quality Assurance Evaluation reports – files reviewed by the quality control group to identify areas of concern that require attention. Pre- and Post-Contract reviews – to identify possible issues before solicitation and for follow-up to determine whether actions were taken to mitigate potential problems.

31. While not in place during the scope of the review, CBSA established a Contracting Review Committee during the OPO’s examination phase and began to implement a Contracting Quality Assurance program, which when fully implemented will include the following features:
    • Established criteria for file review, with increasing levels of review dependent on risk. For example, mandatory review required at the bid evaluation stage if approval is required at manager level or higher.
    • File close-out checklists.
    • Service standards for the Quality Assurance group, advising clients about the timing and duration of QA file review.

32. Parks had no contract review board in place. Some formal documented quality assurance activities were found for non-competitive services and construction services above specific thresholds, but not for the contracts within the scope of this review. Though not required, Section 11.1.1 of the TB Contracting Policy states “contracting authorities are encouraged to establish and maintain a formal challenge mechanism for all contractual proposals.” Contract review board mechanisms are considered to be a good practice by the OPO as “in many departments, the principles of fairness, openness and transparency are safeguarded through oversight review, and monitoring by a senior procurement review committee.” ^{Footnote 3}
33. The absence of a formal challenge mechanism, the horizontal delegation of contracting authority and the lack of monitoring are factors that increase the risk that Parks’ bid evaluation processes may not be conducted in a manner consistent with the TB Contracting Policy and that support the principles of fairness, openness and transparency.

File review

34. File review was designed to assess whether evaluation criteria were established and bid evaluations and selection were conducted as per applicable policies and guidelines. To perform the assessment, the OPO requested contracting data for all competitive contracts from April 1, 2014, to November 30, 2015, from the three selected departments for goods and services contracts valued at less than $2 million.
35. From this file population, 90 files were selected for review, 30 from each department. Files were randomly selected from two stratified populations (less than and greater than $25,000) using data analytical software, with consideration given to factors including regional representation, procurement strategy (traditional competitive or open bidding), and goods or services.

Evaluation criteria and selection methodology

36. Section 10.3.1 of the TB Contracting Policy states that “assessment and award criteria must be spelled out in the solicitation document.” Evaluation criteria are restrictive by nature, as their purpose is to identify the minimum requirements needed to perform the tasks defined in the Statement of Work (SOW), and to assess the competency and relative merit of a bid. This includes mandatory criteria, which are pass/fail oriented, and rated criteria that assign points as specified in the solicitation document.
37. The OPO reviewed files to assess whether:
    • mandatory criteria were aligned with requirements that were measurable and demonstrable, appropriate with regard to security requirements, and established prior to solicitation and evaluation; and
    • rated criteria were clearly marked and established prior to solicitation and evaluation.

38. In three of the 90 files, one from each department, the OPO was unable to determine whether mandatory or rated criteria were aligned with requirements because there was no solicitation document on file. The OPO was therefore unable to identify the criteria or requirements and could not determine whether they were aligned. One additional file, from the Royal Canadian Mounted Police, had a rated criterion that was not clearly marked in that it was unclear how evaluators should allocate points. Such instances increase the risk that a criterion will not be evaluated in a consistent manner.
39. In the remaining 86 files, criteria were assessed against requirements in the SOWs and were found to be measurable, demonstrable and aligned with requirements. The OPO found no instances of unnecessarily restrictive criteria, and security requirements were appropriate in that they were aligned with operational needs and were not restrictive (i.e. required by bid closing as opposed to contract award).
40. All three departments provided evidence of contracting officers giving advice and performing a challenge function with regard to the establishment of evaluation criteria. For example, a contracting officer questioned the number of years of experience being requested by the technical authority, as it appeared to be excessive for the level of resource required. The years of experience requested were subsequently reduced, thereby eliminating a potentially restrictive criterion that may have limited competition.

Bid evaluation

41. For a bid evaluation process to support the principles of fairness, openness and transparency, the OPO expected the following conditions to be met: (1) evaluations were performed by identified team members; (2) criteria and selection methodology corresponded with the solicitation document; (3) criteria and selection methodology were followed; and (4) all bids were evaluated in a consistent manner.
42. Detailed results of the OPO’s file review are outlined below.

(1) In 8 out of 90 files, the OPO was unable to identify all members of the evaluation team:

• Canadian Border Services Agency (CBSA) 1 out of 30
• Parks Canada (Parks) 4 out of 30
• Royal Canadian Mounted Police (RCMP) 3 out of 30

(2) The OPO was able to ascertain the selection methodology in all 90 files.

(3a) In 8 out of 90 files, the OPO was unable to determine whether the evaluations were conducted as per the evaluation criteria:

• CBSA 1 out of 30
• Parks 5 out of 30
• RCMP 2 out of 30

(3b) In 10 out of 90 files, the OPO was unable to determine whether the winning bidders were selected using the stated selection methodology:

• CBSA 2 out of 30
• Parks 6 out of 30
• RCMP 2 out of 30

(4a) In 10 out of 90 files, the OPO was unable to determine whether bids were evaluated in a consistent manner:

• CBSA 1 out of 30
• Parks 6 out of 30
• RCMP 3 out of 30

(4b) In 2 out of 90 files, the OPO determined that bids were not evaluated in a consistent manner:

• CBSA 1 out of 30
• RCMP 1 out of 30

43. Additionally, the OPO pre-defined six critical errors linked to applicable sections of the TB Contracting Policy. Specifically:
    • Award criteria not established prior to solicitation and evaluation;
    • Award criteria not assigned as mandatory or rated;
    • Selection methodology (e.g. lowest price; lowest price per unit; highest combined technical rating and price) not specified in the solicitation;
    • Bids not evaluated as per the pre-defined award criteria;
    • Sufficient documentation not on file to support decisions made and actions taken (e.g. signed evaluation sheets, emails); and
    • Bid evaluation not consistent with the principles of fairness, openness and transparency.

44. As each file could contain six possible critical errors, the total possible number of critical errors per department was 180, with a total possible number of critical errors for this file review of 540. Overall critical error rates are reported in Table 1.2 Critical Error Rates:

Critical error rates table 1.2

Department	Files with critical errors	Total critical errors
CBSA	10/30	23/180
Parks	10/30	28/180
RCMP	3/30	9/180
Total	23/90	60/540

45. Files with critical errors were pointed out by the OPO to the departments and they were given the opportunity to provide additional documentation. As a result of the additional documentation, the revised total critical error rates are reported in Table 1.3 – Revised Critical Error Rates:

Revised critical error rates table 1.3

Department	Files with critical errors	Total critical errors
CBSA	6/30	12/180
Parks	8/30	21/180
RCMP	3/30	9/180
Total	17/90	42/540

46. Critical errors were identified in 23 out of 90 (26%) files reviewed. Subsequently, and prior to finalizing the report, the errors were reduced to 17 out of 90 (19%) after additional information was provided by CBSA and Parks.
47. The OPO found that the RCMP provided no additional information because files were generally complete and all existing information was made available during the file review period.
48. Critical errors were primarily caused by insufficient documentation (11 out of 90 or 12%), such as bids or bid evaluations missing from the file.
    • CBSA – 3 out of 30 files (10%)
    • Parks – 6 out of 30 files (20%)
    • RCMP – 2 out of 30 files (6%)

49. Section 12.3.1 of the TB Contracting Policy states that “procurement files shall be established and structured to facilitate management oversight with a complete audit trail that contains contracting details related to relevant communications and decisions including the identification of involved officials and contracting approval authorities.” Departments must document procurement files in a manner that allows their bid evaluation processes to stand the test of public scrutiny and demonstrate that a fair process has been conducted.
50. Lack of documentation directly affects the transparency of every bid evaluation process. When departments are unable to provide documentary evidence of actions taken and decisions made, suppliers may question whether bid evaluations have been conducted in a fair manner. This uncertainty may dissuade suppliers from contracting with the federal government, which in turn could limit the number of potential suppliers. In essence, a lack of documentation undermines the competitive process and the objective of the TB Contracting Policy, which is to carry out contracting “in a manner that enhances access, competition and fairness and to obtain best value.”
51. While the majority of critical errors were triggered by a lack of documentation, the OPO noted that two files, one each at CBSA and Parks, where individual evaluations were not conducted by each member of the evaluation team.
52. In order to demonstrate that evaluation processes were conducted in a fair manner, individual evaluations should be conducted by all members of an evaluation team and these evaluations should be maintained as part of the procurement file in support of transparent decision-making.
53. The Canadian International Trade Tribunal (the Tribunal) has clarified in several cases that “[w]hile the Tribunal acknowledges that evaluators’ individual score sheets will not necessarily be relevant for every ground of complaint brought before the Tribunal, they nonetheless comprise an important component of the record of the solicitation process that must be retained in accordance with the requirements of [the North American Free Trade Agreement] NAFTA.” ^{Footnote 4} The Tribunal went on to state “individual scoring sheets constitute a check that may allow it to verify that the procurement was conducted in compliance with NAFTA, and they may provide unsuccessful bidders more insight into the evaluation process and thereby support their confidence that the procurement was carried out with integrity.” ^{Footnote 5}
54. Furthermore, critical errors due to fairness issues were identified in five files noted below.

    CBSA

    • One file where bids were evaluated unfairly due to the inconsistent application of a mandatory criterion regarding insurance certification.
    • One file where the financial bid that was part of a non-responsive bid was evaluated, which increased the risk of a non-responsive bidder being awarded a contract.

    Parks

    • One file where the solicitation process was arranged according to the incumbent supplier’s vacation schedule.
    • One file where the financial bid that was part of a non-responsive bid was evaluated, which increased the risk of a non-responsive bidder being awarded a contract.

    RCMP

    • One file where bids were evaluated inconsistently due to the unclear point allocation of a rated criterion.

55. As stated earlier, the TB Contracting Policy stresses that “fairness to all prospective contractors and transparency in the award process are imperative.” Examples as listed above indicate suppliers are not always provided equal treatment, resulting in inconsistency with the principles of fairness, openness and transparency and validating the recurring issues and complaints that the OPO receives every year regarding bid evaluation processes.

Conclusion

56. Overall, the reviewed departments had policies and guidelines in place to support bid evaluation processes; roles and responsibilities were clear and communicated; and monitoring and quality assurance activities were noted.
57. The bid evaluation processes for the majority of files reviewed were conducted in a manner consistent with applicable legislation and policies, and supported the principles of fairness, openness and transparency. However, inconsistencies and areas for improvement were noted, including the need for organizations to strengthen file documentation.

Recommendations

58. The Procurement Ombudsman recommends that:
    • Canadian Border Services Agency to complete the implementation of its Contracting Quality Assurance program and the updating of policies, templates and guidance; and that
    • Parks Canada to strengthen its oversight of bid evaluation processes through: the continued updating of procurement guidance; the development and implementation of formal quality assurance activities; establishment of an appropriate review mechanism, such as a contracting review board; and a review of its contracting authority delegations.

Departmental responses

59. In accordance with section 5 of the Regulations, the Procurement Ombudsman provided CBSA, Parks and the RCMP with the opportunity to comment on the proposed recommendations of this review and the reasons for them. Departments were given the opportunity to comment on the findings of the review, and comments were taken into consideration prior to the report being finalized and published. The following responses were provided by the departments.
60. Canadian Border Services Agency: Agreed. The Canada Border Services Agency is actively renewing its policy and procedure suites complemented and supported by process mapping and educational products for functional experts and cost centre managers. These actions will be completed by March 2018.
61. Parks Canada: Agreed. Parks Canada has some formal documented quality assurance activities such as for non-competitive services and construction services above specific thresholds. Parks Canada will continue to assess its contracting activities and implement quality assurance and monitoring appropriate to the associated risk of the Agency’s contracting activities.
    1. The TB Contracting Policy encourages establishment of a formal challenge mechanism which might range from a formal central review board to divisional or regional advisory groups, depending on the departmental organization and magnitude of contracting. Due to its decentralized and flat organizational structure, for procurements within its authority, a full formal central review board at Parks Canada would consist of the same executives that had recently approved the associated projects. As such, Parks Canada will continue to seek methods of ensuring that appropriate centralised review of the Agency’s procurement projects are conducted.
    2. Due to the relatively recent increase in contracting activity, procurement delegation of authorities and the complexity of its contracting activities, Parks Canada has been considering options for either differentiated delegation of authority or a differentiated review mechanism appropriate to the flat, yet geographically dispersed, organizational structure of the Agency. The TB contracting policy recommends approval by an appropriate authority, whereupon the actual contracts may be executed by the contracting authority.

62. Royal Canadian Mounted Police: The report does not contain any recommendation for the RCMP to implement at this time, and the content of the report is accurate and complete.

Date modified:

2018-11-01