Procurement practice review of Transport Canada

Mandatory criteria

In 20 of the 30 applicable files, TC had mandatory criteria which were aligned with the requirement, not overly restrictive and did not unnecessarily favour or penalize any particular bidder or group of bidders. However, in 6 of the files, the mandatory criteria were not communicated in a clear, precise or measurable manner and in 2 of the files the criteria appeared to favour a bidder. There were also 2 files which had at least 1 unclear criterion and also appeared to favour a bidder. Details from OPO’s review of mandatory criteria are presented below.

17. Regarding the files where the mandatory criteria were unclear and/or unnecessarily restrictive—this can undermine the transparency and openness of the bid solicitation process and can cause bidders to submit non compliant or sub-optimal proposals because the requirements are not well understood. It also limits TC’s ability to effectively assess a supplier’s ability to meet an essential element of the work to be performed.

18. Examples of unclear mandatory criteria included the following:

1. In 3 files, the mandatory criteria required a proposed resource to have “relevant” or “acceptable” experience without defining what relevant/acceptable was. As an example, 1 solicitation for the operations of an airport in Quebec required bidders to demonstrate a proposed airport manager had a “… minimum 2 years of acceptable experience, within the last 5 years, in an airport environment”. It cannot be left up to bidders to guess what is acceptable nor for evaluators to determine if the response noted in the bid crosses each evaluator’s individual, subjective threshold to be considered “acceptable”.
2. In 4 files, the mandatory criteria did not require suppliers to provide any documentation which could be assessed on a pass/fail basis to prove the resource had the necessary certification/experience. As an example, 1 solicitation for training services required bidders to be able to provide training on locomotive emission control systems. Suppliers did not have to provide an explanation of how they were qualified to teach such topics. As such, how could TC be assured the supplier had the required experience to provide the training? Using such wording for mandatory criteria makes it more difficult for both bidders and evaluators to understand how these criteria are to be evaluated.
3. In 2 files, the bidder had to demonstrate something which would be difficult, if not impossible, to assess. In 1 case for a telecom system specialist, that resource had to demonstrate experience working with other departments, “…in particular the ability to liaise effectively”; in another, for a trip travelling to 6 airports in the James Bay area, the bidder had to have 2 “experienced…[f]lightcrew members familiar with the northern operation of planes on short and gravel runways”. It would be almost impossible for an evaluator to explain to an unsuccessful bidder how its resources did not liaise “effectively” or were not “familiar” enough with a plane’s operation on gravel runways.

19. There were also 4 files for which the mandatory criteria appeared to unnecessarily favour a bidder or group of bidders; included the following:

1. In 2 files, the mandatory criteria were specific enough to appear to favour a particular bidder/resource. As an example, 1 solicitation for training services required the proposed leadership trainer to have “…a minimum of [3] years of education in law, psychology or public administration”. There was no explanation of why the leadership trainer would need 3 years of education, but not a degree, diploma or formal certification in law, psychology or public administration.
2. In 2 files where TC contracts out the operations and maintenance of small airports, a mandatory criterion required proposed resources to have experience operating the vehicles at the airport. The listed vehicle descriptions included the make/model/year of the vehicle, and some were listed down to the Vehicle Identification Number. This means that no one other than the resources currently performing, or previously performed, the work could meet that criterion. This makes it exceedingly difficult for a non-incumbent to submit a compliant bid.

Point-rated criteria

Overall, point-rated criteria and rating scales were appropriate to the requirement, clearly communicated and not overly restrictive in almost all of the files reviewed. There were 3 files, however, where solicitations contained missing, incorrect or unclear information regarding point-rated criteria. Details from OPO’s review of point rated criteria are presented below.

20. In all 17 files, the criteria and applicable rating scales were appropriate, clearly communicated and, in 14 files, those rating scales were clearly defined. However, in 3 files, the rating scale was unclear, which could lead to confusion by both bidders and evaluators in how a bid was to address those criteria. An example of where it was unclear was for 2 criteria on 1 file where bidders received either 10/10 or 3/10 points depending if the bidder’s work plan either “clearly” (10/10) or “does not clearly” (3/10) demonstrate how it would meet the requirements listed in the specification.

21. There were also 2 files in which the rated point allocation scheme indicated the requirements were actually mandatory. It would be easier, and clearer to all, if the criterion was written as a mandatory one. As an example, in 1 solicitation for the economic assessment of a proposed airline acquisition, bidders were required to list their experience in the use of commonly-used statistical software, of which there are dozens. However, 2 points were awarded, per package, for experience in 3 specifically-identified software packages. As the minimum passing score for that criterion was 2/6, what it really means is that it was mandatory for bidders to have used 1 of those 3 point-getting software packages.

22. In 4 further cases, the scoring grid, while clearly stated, did not appear to reflect the importance of the criteria. As an example, in 1 solicitation for consulting services, a rated criterion assessed the bidder’s work plan for drafting a public risk management framework and the minimum pass mark was 14/20 points. The scoring grid, however, awarded 14 points for a proposed work plan which “…demonstrates some understanding of the requirement. The proposal has weaknesses, is not likely to meet the requirement or be effective and does not demonstrate good technical value to Canada”.

Selection methodology

Overall, in almost all of 30 applicable files, the selection methodology was accurate and aligned with the requirements as described in the solicitation document; however, 2 exceptions were noted:

23. In both cases, due to a lack of documentation, TC could not demonstrate it followed the required bid-solicitation process for files related to a PSPC Temporary Help Services (THS) SO, which is no longer an active PSPC method of supply. For THS SO files, suppliers were ranked according the their charge-out rates which departments had to follow when soliciting bids, i.e. the lowest-priced supplier had to be asked first if it had available resources, and if it did not, the second-lowest priced supplier was contacted, and so on. In these 2 files, there is nothing on file indicating TC followed the THS SO right-of-first-refusal methodology.

LOE 2: To determine whether solicitation documents and organizational practices during the bid solicitation period were consistent with applicable laws, regulations and policies

24. The TBCP sets out detailed procedures to ensure government contracting is carried out in a manner which enhances access, competition and fairness and results in best value. Section 10.7 of the TBCP includes the minimum requirements to be included in the solicitation document as well as mandatory elements related to the design and execution of the process.

25. Solicitation documents must contain work descriptions or specifications defined in terms of clear outputs or performance requirements. Solicitation documents should include: bidder instructions; bid preparation instructions; clear evaluation procedures; certification requirements; security and financial requirements; resulting contract clauses; and instructions informing bidders they may request information on the results of the requests for proposal (RFP) and how their bid was evaluated. Clarity of provided information is key in supporting the Government of Canada’s obligations laid out in the TBCP.

26. This LOE applied to 35 files; 30 files for which TC either used 1 of PSPC’s methods of supply or issued a competitive solicitation as well as 5 TAs against contracts. Solicitation documents (excluding evaluation criteria and selection plans that were assessed under LOE 1) were assessed to determine whether, among other things, they contained a clear description of the requirement and instructions necessary to prepare a compliant bid. The assessment of organizational practices included factors such as whether the solicitation was open to the appropriate number of bidders and for the required duration, and whether communications with suppliers supported the preparation of responsive bids.

Solicitation documents

Most of the files reviewed included clear and complete information and instructions for submitting compliant bids, though there were 6 files for which TC could not produce most, if not all, of the solicitation documents. Of the remaining 29 applicable files, most met the documentation and process requirements; however, there were some files which were found lacking in this area. OPO notes TC’s ability to access hard-copy files was impacted by COVID-19 pandemic-related office closures and that some of the missing documents may be located in the applicable TC office. Details from OPO’s review of solicitation documents are presented below.

27. When using RFPs, TC’s solicitation process requires bidders use a 2-envelope system. Envelope 1 contains only the technical proposal and bidders are informed it is to be in sufficient detail to allow for the evaluation as per the RFP’s Evaluation plan. Envelope 2 contains the financial information and is opened only after the technical evaluation is complete and only if the technical proposal met all mandatory requirements and met or exceeded any minimum score for the rated evaluation criteria. Otherwise, the financial envelope is to be returned, unopened, to the supplier. OPO considers this to be a positive practice, which should be carried forward in the context of electronic bidding provided technical and financial proposals are kept separate. It allows both evaluators and bidders to ensure the right information is contained in the right area of the bid. In addition, if the financial bids are returned unopened, bidders can be assured the bid prices did not influence the technical evaluation, e.g. if a less-expensive bid was improperly “passed” to take advantage of the lower price. Bidders can also be confident their prices have not been revealed to other bidders.

28. Regarding the 6 files for which TC was unable to provide a majority of the solicitation documents, they fell into 2 categories:

1. Competitive files—these are complete, discrete files and must, therefore, contain all documentation necessary to demonstrate all TBCP requirements were met in awarding the contract. In 2 cases, TC did not have the solicitation documents. Missing documentation will be addressed under LOE 3.
2. TAs—Some contracts are structured to allow Departments to identify a definable work package (the TA) for the contractor to complete. Unlike a competitive file, the contract provides the administrative structure under which the work will be done. The TA process is therefore streamlined, because it only deals with qualifying a resource to complete the work package. Another significant difference between the TA and competitive files is that, for TAs, there is not another supplier—if the proposed resource does not meet the TAs requirement, the process will be re-done and the company will have to find another resource. In 4 cases, however, the documentation on file did not demonstrate the contract-defined TA process had been followed.

29. For those files for which there was adequate solicitation documentation, there were 4 files where it was unclear if TC invited the correct number of suppliers. There were 2 cases where it was unknown if TC followed the correct “right-of-first-refusal” solicitation process or if TC bypassed certain suppliers and went directly to the contract awardees. In the third case, TC was required to invite 15 suppliers, but it appears TC only sent the solicitation to 10 suppliers. In the final case, TC did not have any record of how many suppliers were actually invited.

Communication with suppliers

In a majority of files reviewed, where applicable, there was evidence communication with suppliers during the solicitation period was appropriate. In 13 cases, however, communications with suppliers were not captured in the file. Details of OPO’s review are presented below.

30. Section 2.a of the TBCP states government contracting shall be conducted in a manner which will “…stand the test of public scrutiny in matters of prudence and probity, facilitate access, encourage competition, and reflect fairness in the spending of public funds”. Section 12.3.1 further states procurement files “…shall be established and structured to facilitate management oversight with a complete audit trail that contains details related to relevant communications and decisions…”.

31. These requirements apply to all aspects of the procurement process, including interactions with suppliers. During the bid solicitation process, suppliers may communicate with contracting authorities to obtain clarifications or explanations of the content of the solicitation. For procurements subject to the Canada Free Trade Agreement and the World Trade Organization’s Agreement on Government Procurement, section 12.3.2 of the TBCP requires contracting authorities to ensure all communications with bidders are supported by complete documentation and records to demonstrate that the procurement process was carried out in accordance with the agreements. Any significant information given by a contracting authority to a supplier with respect to a particular procurement must also be given simultaneously to all other interested suppliers.

32. In 13 cases, addenda responding to questions were issued, but for which there was no record of when the question was asked or by whom. Without knowing this, OPO is unable to determine if all suppliers’ questions were answered with the same speed and, therefore, that they all were treated equally in getting timely clarifications needed to assist them in crafting their bids.

33. After the solicitation process is completed and bids evaluated, TC informs bidders of the evaluation results via either a “regret” letter, advising the bidder it was not the successful bidder, or an award letter advising the bidder it won and that the contractual documents will follow shortly.

34. “Regret” letters are sent to unsuccessful bidders and should offer an explanation of why their bids were not selected. TC’s regret letters were inconsistent in the amount of information they provided. In 1 file, the letter was excellent and explained each score the bidder obtained for every rated criterion. In another file, however, the letter just stated the bidder would not be awarded the contract but gave no information about its bid; it did not advise if the bid was not compliant, compliant, lost on price, etc. In another file, the letter advised bidders to contact the technical authority, which differs from the normal practice of having the contracting authority manage the entire solicitation process. There were also 2 files in which different “regret” letters were sent to the bidders, i.e. the letter sent to some bidders stated the contract would not be placed with them; the letter sent to other bidders also explained how their proposal compared to the winning bidder. These last 2 examples were of particular concern, because it appears TC treated bidders unfairly—some received more relevant information than others.

LOE 3: To determine whether the evaluation of bids and contract award were conducted in accordance with the solicitation

35. In order to ensure the fairness and defensibility of evaluation processes, section 10.7.27 of the TBCP requires that evaluation criteria be adhered to strictly and applied equally to all bidders. Failure to ensure the consistent evaluation of proposals increases the risk ambiguities in the selection process result in the contract being wrongly awarded. OPO examined how the mandatory and rated criteria were evaluated as well as if other administrative requirements of a solicitation were met.

36. Of the 35 competitive solicitation processes reviewed, 30 files had a technical/financial evaluation leading to contract award and 5 were for TAs under an already-awarded contract. These 35 files were examined to determine whether a process had been established to ensure: the consistent evaluation of bids; that the evaluation of bids had been carried out in accordance with the planned approach; and that files were adequately documented.

Bid evaluation

In 17 files, the bids were missing complete records about either the individual or consensus evaluations. For the 18 files for which there were complete evaluation results, 11 showed inconsistencies in the evaluation of bids and deviations from the planned approach.

37. Prior to the evaluation of the technical component of bids, TC contracting authorities provide the technical evaluators with Evaluation Guidelines explaining how the evaluation process is to proceed. Included is a form to be signed by each evaluator before the technical bids are released for evaluation. Among other things, the forms state the evaluator agrees that:

• The bids shall be treated confidentially
• The evaluator is not in a conflict position
• Prior to consensus meetings, the evaluator shall not discuss the evaluation with other evaluators
• The evaluator does not know the prices of the bid

38. This signed form was not found on 15 of the 35 applicable files and in 2 cases, the identity of the evaluators was unknown.

39. OPO’s review found that, in 10 files, bids were not disqualified for missing either a mandatory or required administrative evaluation criterion and/or that the evaluation of the rated criteria was not done in accordance with the solicitation:

1. In 7 cases, information was either not provided by the bidder or the file does not indicate TC properly verified the information provided by the bidder:
   • In 2 cases, winning bidders did not provide mandatory curriculum vitae (CV) for their proposed resources. In 1 case for the management of an airport in Quebec, the evaluators noted a bid was missing a CV for 1 of the 7 proposed resources. The contracting authority advised them to not disqualify the bid solely on this basis because: (a) the solicitation did not require the CV to be a separate document; (b) there was no description of what information had to be in a CV, and (c) the solicitation did not require the CV to be evaluated in any way and that its real purpose was just to identify the proposed resource. It appears TC used the weaknesses in its own solicitation drafting to allow a bidder, which did provide 6 other CVs in that same bid, to stay in the competition. In this case, there was another compliant bidder who should have been considered for contract award.
   • In 2 other cases, the bid was missing required documentation to either prove its proposed resources had the necessary experience or that the company had the necessary corporate experience/equipment.
   • In 3 files, the documentation did not include anything confirming:
     • TC verified the winning bidder attended a mandatory site visit:
     • TC contacted references to confirm a resource’s language capabilities:
     • TC conducted an interview to confirm a resource’s language capabilities.
2. In 3 cases, administrative requirements, which do not relate to completing the work, must, nonetheless, be met by each bidder. In 2 cases, the bidder did not fill out their financial proposal properly and, in the third case, the bidder changed the standard terms and conditions of the solicitation document, which is not allowed. In all cases, TC continued with the evaluation.

40. TC solicitation documents all state that failing a mandatory criterion will result in the disqualification of the bid. As such, given the above and the lack of related disqualifications, TC staff should re-examine evaluation criteria to determine if they are, in fact, to be judged as mandatory to meeting the solicitation’s requirement.

41. There were also 4 cases where the scoring for the rated criteria did not reflect the listed scoring grid:

1. In 1 case, some of the rated criteria required bidders to provide information about the experience of its proposed resources in a certain format – the dates had to be couched in terms of years/months of engagement and the start and end dates of work. The (only) bidder did not do so, in many cases only the year was listed, i.e. “1993 to 1994”, which could indicate an engagement of between 2 days and 24 months, but TC scored it nonetheless.
2. In 1 other case, 1 rated criterion required bidders to demonstrate up to 4 different experience elements in their response. However, the TC evaluators’ notes state they decided to only award points for experience related to the first and fourth element of that criterion.
3. There were also 2 cases which were missing evaluation results and so OPO could not confirm how the scoring was achieved:
   • In 1 case, bidders were awarded 6 points per qualifying event—TC awarded the bidder 22.5 points even though the scoring grid did not allow for part marks:
   • In another case, only 1 of the 3 individual evaluations were on file. The consensus score for the only bid just met the minimum, 49/70, but the only individual score on the file was 28/70.

42. By not conducting the evaluation in the manner prescribed by the solicitation, TC has exposed itself to several risks including awarding the contract to either a non-compliant bidder or the wrong compliant bidder. It also means TC cannot defend and explain its decisions and actions and prove that they were made in accordance with the applicable laws, regulations, and policies.

Procurement documentation

In most files, the procurement file documentation was incomplete. OPO’s observations regarding file documentation are detailed below.

43. Section 12.3.1 of the TBCP requires that procurement files facilitate management oversight with a complete audit trail containing details related to relevant communications and decisions, including the identification of the involved officials and contracting approval authorities. The requirement to ensure adequate file documentation extends to the actions undertaken during the solicitation period as well as the evaluation of proposals.

44. File documentation was reviewed to determine whether a complete audit trail was retained to support consistent and transparent decision-making. In general, TC’s documentation was not complete, as 30 of 38 files were missing documents. However, it must be noted many of the “missing” documents may be located, in hardcopy form, in the applicable TC office. TC provided OPO staff with whatever relevant documentation it had from its electronic record keeping system. However, due to the COVID pandemic, TC staff could not return to their offices to search for some of the missing documents.

45. Incomplete procurement files resulted in inadequately supported procurement actions that risk undermining the integrity, fairness and transparency of the procurement process. Keeping complete and detailed evaluation records is crucial for demonstrating that evaluation criteria have been applied equally to all competing bids, and demonstrating that the procurement has been carried out in a manner consistent with TC’s obligations under the TBCP and applicable trade agreements.

Regional contracting

46. Within OPO’s random file selection, there were 2 files awarded by 1 of TC’s decentralized regional procurement units. Both files were for the same service, the salvage and disposal of boats which had sunk or been abandoned near or in Canada’s waterways.

47.In both cases, a TC employee, who is not a contracting officer, e-mailed an unknown number of suppliers a SOW and little other solicitation documentation, i.e., the bid selection methodology, what information had to be included with the bid (In 1 case, the entire winning bid was “Disposal/Demolition of [Ship Name] $108,000”) or a copy of the draft contract. It is unknown if any interaction took place between the invited suppliers and TC until that TC employee received the bids. That employee evaluated the bids and then advised another employee, who does appear to be a contracting officer, with the results of whatever evaluation took place. That second TC employee then drafted and signed the contracts after conducting an Integrity Regime verification for the winning bidders.

48.There was almost no formal documentation on the files and neither file provided assurance the solicitation processes were conducted in an open, fair or transparent manner. These files have been brought to the attention of TC’s Deputy Minister for further consideration as appropriate. As part of the practice review process, OPO follows up 2 years after the publication of the review report to track the progress TC has made on the recommendations in the report. OPO will also be following up on any outcome to the Deputy Minister’s inquiries into these 2 files.