Procurement practice review of Immigration, Refugees and Citizenship Canada

Mandatory criteria were inadequately defined, and were not limited to the essential qualifications

15. Of the 34 files that contained mandatory criteria, these criteria were generally well-defined and could be assessed objectively on a pass-fail basis. However, in 9 files, mandatory criteria were inadequately defined, and were not communicated in a clear, precise and measurable manner. There were an additional 4 files in which the mandatory criteria were not limited to the essential qualifications necessary to fulfill the requirement.

1. In 8 files mandatory criteria required bidders to demonstrate ‘experience’ but did not specify how much experience was required. As a result, any amount of experience would have resulted in the criteria being met. As an example, one solicitation that would provide permanent residence status to entrepreneurs who have the potential to build high-growth start-ups in Canada required the bidder to “have experience providing written recommendations relating to business and/or innovation to the Federal Government of Canada, including Crown Corporations, over the past two years from bid closing date.” However, the amount of experience required was not specified in the criterion, as a result, as little as one day of experience would have resulted in the criterion being met. In such circumstances, the requirement should be tailored to specify the amount of experience that was essential to perform the work.
2. In 1 file IRCC should have been more prescriptive in defining the mandatory criteria. For example, in a solicitation for reconfiguration and moving services, the criterion stated: "[t]he bidder must possess the equipment as follows: The Bidder must provide all the resources, tools, lifting equipment and supplies, necessary to perform all tasks properly, efficiently and safely, at no additional cost to [IRCC]. Example of what could be required…". The way in which this criterion is framed may create challenges for bidders to demonstrate their compliance and for evaluators to assess that the criterion has been satisfied by the bidder. IRCC should have been prescriptive in defining all of the necessary equipment so that the criterion could be assessed on a pass/fail basis.

16. There were 4 files where the mandatory criteria were not limited to the essential qualifications necessary to complete the requirement. In 1 of these files, the criterion was overly restrictive and may have reduced competition.

1. 2 files for translation, editing and review services included a mandatory criterion that required bidders to identify a single point of contact. Mandatory criteria must be limited to the essential qualifications necessary to conduct the work; identifying a single point of contact, while useful from an administrative standpoint, is unrelated to the bidder’s ability to perform the work and is not aligned with requirements, and could result in non-compliance by bidders due to an administrative oversight; this info could be sought differently, and still achieve the desired result
2. In 1 file the solicitation for loan collection services contained 1 mandatory criterion that required bidders to attest to their capacity to deliver services. While the intent of this criterion was to ask bidders to certify that they could deliver the services within the specified time period, its inclusion as a mandatory criterion is not essential as the contractor’s certification is achieved through their acceptance of the statement of work on signing the contract. Furthermore, a bidder’s attestation in their proposal does not demonstrate to evaluators that they have the capacity to deliver the services within the required timeframe. The inclusion of unnecessary evaluation criteria, particularly as mandatory criteria, unnecessarily complicates the procurement process by making it administratively burdensome on bidders to prepare their proposal. Further, it creates opportunities for bidders to make mistakes and for their proposal to be deemed non-responsive for failing to meet a mandatory criterion that was ultimately not required.
3. In 1 file for expert advice related to a transformation strategy that was subject to the North American Free Trade Agreement (NAFTA) and the World Trade Organization-Agreement on Government Procurement (WTO-AGP), a mandatory criterion required bidders to demonstrate experience in 4 transformation studies “where the Bidder was not involved with the implementation of the solution.” Both NAFTA and the WTO-AGP require that the conditions for participation by suppliers in tendering procedures are “limited to those that are essential to ensure the fulfilment of the contract in question.” In this case, it is unclear why a bidder’s involvement in the ‘implementation of the solution’ would be considered a necessary exclusion. IRCC received numerous comments from interested suppliers regarding the restrictive nature of the criteria. The criteria was not changed and the solicitation process resulted in only 1 compliant bid with the bidder receiving a perfect score in the technical evaluation.

Point-rated criteria and rating scales were not overly restrictive, were appropriate to the requirement, and reflected the relative importance of the criteria

17. Section 10.7.25 of the TBCP states that “criteria should identify accurately all the performance elements significant to the success of the project and should measure both the competence of the firm and the worth of its particular technical approach.”

18. In all 24 files that contained point-rated criteria, point-rated criteria were not overly restrictive, were appropriate to the requirement and reflected the relative importance of the criteria. However, in 2 files, the rated criteria or the scoring grid could be communicated in a more clear and precise manner to minimize subjectivity.

1. In 1 file for qualitative research in labour markets, the point-rated criteria did not specify how much experience would result in maximum points being awarded. Rather the criteria stated that the maximum allotment of points would be awarded if “[t]he proposal show[ed] an excellent match between knowledge and experience on the one hand and the research theme on the other.” Terminology such as “excellent match” is highly subjective and is open to multiple interpretations.
2. In 1 file for satisfaction surveys the scoring grid for a rated criterion could have been communicated in a clearer manner. For example, the criterion required bidders to submit a workplan and listed 4 criteria that “should at a minimum be included in the Bidder’s work plan.” The scoring grid states that 5 points shall be awarded for a workplan that is “well defined and includes all criteria” while 4 points shall be awarded for a workplan that is “clear and includes 4 of the criteria.” Since the criteria for awarding 4 and 5 points both require the bidder to include 4 of the criteria in their workplan, this could make it difficult for potential bidders to understand how evaluators would distinguish between a workplan that receives 5 points versus one that receives 4 points. This did not impact the outcome of the competitive process.

19. When evaluation criteria are communicated in a clear, precise and measurable manner, it enables bidders to know the requirements and the methods by which their proposals will be evaluated. Failure to adequately define evaluation criteria at the outset carries the additional risk that evaluators may struggle to interpret these criteria during the evaluation process. It can also be difficult to defend against external challenges, as it is more difficult to demonstrate that criteria have been strictly adhered to when the criteria are unclear and open to multiple interpretations.

Bidders were consistently advised of the manner in which the contract would be awarded

20. 33 of the 40 solicitations were carried out as requests for proposals. In all 33 files, the selection methodology was clearly communicated in the solicitation document and reflected the complexity of the requirement.

21. IRCC’s Procurement and Contracting Services group has implemented a number of preventative controls to ensure the overall consistency and compliance of the procurement process. One such control is the completion of the Contract Planning and Advance Approval (CPAA) document. This document is completed by the Contracting Officer following the identification of the requirement, and is also required for each contract amendment. The CPAA provides a comprehensive summary of all relevant information necessary to facilitate management oversight; including a summary of the requirement, estimated costs, related contract history, any contract amendments and the associated rationale, the proposed procurement strategy, and analysis of risks, and general remarks from the Contracting Officer. The CPAA is accompanied by a risk assessment that is completed using IRCC’s Procurement Risk Assessment Tool. The risk rating resulting from the completion of this assessment determines the required approval level for the procurement. High risk procurements, such as those involving an amendment that exceeds 50% of the original contract value are subject to approval by IRCC’s Procurement and Contracting Oversight Committee (PCOC).

22. IRCC’s procurement officers consistently completed the CPAA and provided the necessary context for key decisions, including the rationale for the respective selection methodology. This practice helps to ensure that selection methodologies are appropriate to the complexity of the requirement and that an adequate justification is on file to support decisions and to facilitate management oversight. In instances where common selection methodologies are not chosen, this practice may be useful should the process be subject to scrutiny. IRCC’s CPAA is an effective tool for documenting key decisions and is a good practice that other departments should emulate.

Immigration, Refugees and Citizenship Canada has established a standard evaluation process that is well-documented and was applied consistently across almost all of the files reviewed

34. Of the 33 files for which an evaluation was conducted, the evaluation grid contained criteria that were identical to those found in the solicitation document, and the number and identities of the evaluators had been documented in the file. In all but 1 file, IRCC ensured that the requisite number of evaluators participated in the process, and the evaluation followed the prescribed process (etc. kick off and consensus meetings were held).

35. IRCC has developed a robust suite of process documentation, including guidance for evaluators, to ensure that the bid evaluation process complies with key requirements of the TBCP. IRCC’s Bid Evaluation Procedure sets forth a standardized process for the evaluation of proposals, and provides guidance to contracting authorities on their responsibilities throughout the bid evaluation process. The Evaluation Team Guidelines provide evaluators with a detailed overview of the process that includes a comprehensive summary of their responsibilities, instructions to ensure proposals are evaluated in accordance with the bid evaluation procedures, and are appropriately documented, including the identification of strengths, weaknesses, missing information and are cross-referenced to the bid in the Evaluation Report Form. These instructions also include recommended procedures to mitigate risk during the process such as procedures to protect the confidentiality of the process, and guidance for seeking clarifications, as some examples. These guidelines contain a conflict of interest declaration and are required to be signed by each evaluation team member and returned to the Contracting Officer prior to the receipt of the proposals. In OPO’s review of the 33 files for which a technical evaluation was completed, a copy of these guidelines was consistently on file and completed by evaluators.

Some evaluations were incorrectly conducted

36. In order to ensure the transparency and defensibility of evaluation processes, section 10.7.27 of the TBCP requires that evaluation criteria must be adhered to strictly and applied equally to all bidders. Failure to ensure the consistent evaluation of proposals increases the risk that the contract will be wrongly awarded and may call into question the integrity of the procurement process.

37. IRCC has established a robust evaluation process; however, there were a few instances in which the evaluation was not carried out in accordance with the planned approach. In 2 files, the evaluation team incorrectly deemed proposals that did not meet certain mandatory criteria to be responsive, and in 2 additional files, the evaluation team awarded partial points, which deviated from the procedure identified in the solicitation.

1. In 1 file, 2 of the bidders failed to include mandatory information in their proposals yet their bids were incorrectly deemed responsive. The mandatory criterion required bidders to identify, among other things, the name of the organization in a reference letter. The criteria states “[i]f the Bidder cannot provide a reference letter that meets the requirements, [IRCC] will disqualify the Bidder.” One of the bidders did not provide the name of the organization and instead stated “a major investment company”. The bid was incorrectly evaluated as meeting this mandatory criterion. Similarly, the successful bidder failed to provide the job title for a reference. Emails on file indicate that the Contracting Officer deemed the bid non-responsive as a result of this omission; however, the bid was ultimately deemed to have met the criterion at the consensus evaluation.

   While these issues should not impact a bidder’s ability to fulfill the requirement, they are nevertheless mandatory elements of the evaluation criteria that must be adhered to strictly. While section 10.8.7 of TBCP leaves room for contracting authorities to consider proposals with minor aberrations, this is only the case where all mandatory requirements have been satisfied. Since the omissions are identified as mandatory requirements, evaluators have no discretion but to deem the bids non-responsive in accordance with section 10.8.6 of the TBCP.

   This draws attention to the consequence of including elements in mandatory criteria that are not limited to the essential qualifications necessary to meet the requirement and reflects a broader opportunity for simplification.

2. In 1 file for project management services, the proposed replacement resource, at the Task Authorization stage, must be evaluated in accordance with the requirements of the bid solicitation and must have the qualifications and experience that meet or exceed those obtained for the original resource. One of the mandatory criteria of the bid solicitation pertaining to experience was incorrectly accepted. The mandatory criterion required the resource to possess experience “overseeing at least two (2) projects within a Canadian federal government organizations and/or agencies”. The proposed resource demonstrated this experience using 2 projects, 1 of which was a project for the City of Calgary and did not satisfy the criterion. A review of the proposed resource’s resume demonstrated that this individual did not possess experience within the last five years working on two projects within a Canadian federal government organization or agency, and should not have been accepted as a resource.
3. In 2 files partial points were awarded by evaluators, which deviates from the planned approach. Details regarding how points will be awarded by evaluators should be disclosed to bidders and should be clear to evaluators to support consistency in the scoring approach. The Canadian International Trade Tribunal (CITT) has ruled that partial points may not be awarded unless specifically contemplated in the solicitation document; the evaluation must be conducted according to the methodology established in the RFP.

Overall file documentation was exceptionally good

38. File documentation was reviewed to determine whether a complete audit trail was retained to support consistent and transparent decision-making. Overall, IRCC maintained detailed records of all procurements.

39. IRCC was given an opportunity to provide OPO with additional documentation that may have been missing from the initial file disclosure. Of the 40 files that were reviewed, only 3 documentation issues could not be resolved. These included 1 instance where it was unclear that unsuccessful bidders had been notified; 1 instance where the invitation to tender was on file, but names of the suppliers were not visible (Blind carbon copy (BCC)); and 1 instance where the file did not contain the Centralized Professional Services System (CPSS) Client Module generated list of the selected suppliers.

40. IRCC has developed and has consistently implemented several tools to ensure that procurement files contain adequate records of decisions to facilitate management oversight and maintain an adequate audit trail. Some examples of note include:

1. Contract Planning and Advanced Approval (CPAA) document: This document provides a comprehensive record of all information pertinent to decision-making, and serves to document the approval of key decisions, such as the approval of procurement strategy and any contract amendments. The CPAA includes information such as a summary of the requirement, related contract history, any contract amendments, summary of risk as well as general remarks from the Contracting Officer.
2. Risk Assessment Tool: This tool is intended to assist Contracting Officers in the assessment of risks and to support the development of recommendations in the CPAA. The results of the risk assessment determine the level of approval required for the procurement. For example, a low to medium risk rating requires signature from the delegated contract authority, whereas a high risk rating prompts review and approval by IRCC’s Procurement and Contracting Oversight Committee (PCOC). Some examples where PCOC review is required include all sole-source contracts greater than 10k, unplanned contract amendments where the total value exceeds 50% of the planned contract, and where there is a second (and all subsequent) unplanned amendment.
3. Evaluation Summary Report: This report is completed by the Contracting Officer following the conclusion of the bid evaluation process. It provides a comprehensive record of the results of the evaluation including the time each bid was received, the results of the technical and financial evaluation, and the recommendation for contract award. This document is signed and saved to file by the Contracting Officer.
4. Procurement File Checklist: This checklist includes a detailed list of all steps and considerations for each procurement phase: pre-contracting, contracting, contract management and close out. The Checklist is saved to each procurement file and requires the Contracting Officer to complete each step by checking “Yes”, “No”, or “N/A” depending on the requirement and method of solicitation. A common element across all phases is the prompt to save all pertinent correspondence and decision making emails to file.