Procurement practice review of Innovation, Science and Economic Development Canada
On this page
1. The Office of the Procurement Ombudsman (OPO) conducted a procurement practice review of Innovation, Science and Economic Development Canada (ISED).
2. In accordance with paragraph 22.1(3)(a) of the Department of Public Works and Government Services Act, the Procurement Ombudsman has the authority to review the procurement practices of departments to assess their fairness, openness and transparency.
3. OPO’s procurement practice reviews are based on issues and complaints brought to OPO’s attention by stakeholders, both in general and in regard to specific solicitations by various federal organizations. Based on this information, OPO has identified the 3 highest-risk procurement elements as: (1) the establishment of evaluation criteria and selection plans; (2) the bid solicitation process; and (3) the evaluation of bids and contract award. For the purposes of this review, these elements are defined as follows:
- Evaluation criteria and selection plans—the development of mandatory and point-rated evaluation criteria, and the identification of the selection method to determine the successful bid
- Solicitation—the design and execution of the solicitation process, including the clarity and completeness of solicitation documents
- Evaluation of bids and contract award—the establishment of a process to ensure the consistent evaluation of bids in accordance with the planned approach, including an evaluation plan and instructions to evaluators, and the adequacy of documentation to support the selection of the successful bidder
4. ISED was selected for review as one of the top 20 federal departments/agencies in terms of the value and volume of its annual procurement activity. OPO is conducting similar reviews of other top 20 departments/agencies over a five-year period ending in 2022-23.
5. ISED’s mandate and role is to help Canadian businesses grow, innovate and expand so they can create good-quality jobs and wealth for Canadians. It supports science research and the integration of scientific considerations into investment and policy choices. The Department helps small businesses grow through trade and innovation and promotes increased tourism in Canada. It works to position Canada as a global centre for innovation where investments support clean and inclusive growth, the middle class prospers through more job opportunities and companies become global leaders. ISED's efforts also focus on improving conditions for investment, supporting science, building capacity for clean and sustainable technologies and processes, increasing Canada’s share of global trade, promoting tourism, and building an efficient and competitive marketplace.
6. The Contracting and Materiel Management (CMM) Division of the Corporate Finance, Systems and Procurement Branch is the departmental procurement centre of expertise at ISED. In 2011, ISED established a Contracting Control Framework, which outlined roles and responsibilities, contracting processes and procedures and methods of monitoring and reporting. Over time, the Contracting Control Framework has been supplemented with updated guidance for departmental officials to navigate the contracting process. During the period of OPO’s review, ISED was in the process of updating its contracting guidance to align with the new Treasury Board Directive for the Management of Procurement.
7. According to information provided by ISED, it awarded 1,142 contracts worth over $217 million during OPO’s review period of October 1, 2019 to September 30, 2021 (excluding contracts valued below $25,000).
II. Objective and scope
8. This review was undertaken to determine whether ISED’s procurement practices pertaining to evaluation criteria and selection plans, solicitation documents, and evaluation of bids and contract award, supported the principles of fairness, openness and transparency. To make this determination OPO examined whether ISED’s procurement practices were consistent with Canada’s obligations under applicable national and international trade agreements, the Financial Administration Act and regulations made under it, the Treasury Board Contracting Policy (TBCP), and, when present, departmental guidelines.
9. The following three lines of enquiry (LOE) were used to assess the highest-risk procurement elements identified in paragraph 3 above:
- LOE 1: Evaluation criteria and selection plans were established in accordance with applicable laws, regulations and policies
- LOE 2: Solicitation documents and organizational practices during the bid solicitation period were consistent with applicable laws, regulations and policies
- LOE 3: Evaluation of bids and contract award were conducted in accordance with the solicitation
10. OPO’s review consisted of an assessment of procurement files for contracts awarded by ISED between October 1, 2019 and September 30, 2021. This review did not include construction contracts, non-competitive contracts, acquisition card activity, low-dollar value contracts (below $25,000 including taxes), or procurement activity for which ISED was not the contracting authority.
11. Based on contracting data provided by ISED, OPO selected 40 competitive procurement files for assessment. The judgmental sample was developed with consideration to factors including materiality and risk. The risk of selection bias was minimized through random selection of individual files meeting these pre-established factors.
12. The 40 files selected for review included 10 open and traditional competitive requests for proposals (RFP), 18 RFPs issued under PSPC supply arrangements, 3 requests for quotations (RFQ), 2 invitations to tender (ITT), 4 call-ups under Public Services and Procurement Canada (PSPC) standing offers and 2 call-ups under ISED standing offers. After receiving selected files from ISED, OPO discovered that one file selected for review was a directed contract issued under a supply arrangement. After excluding this file, the final sample was comprised of 39 files.
13. ISED’s procurement practices pertaining to evaluation criteria and selection plans, solicitation documents, and evaluation of bids and contract award were assessed against the three LOEs noted above. OPO made five recommendations to address the issues identified in the review, which are summarized in Annex I of this report. The recommendations are based on the analysis of information and documentation provided to OPO by ISED during the course of the review.
14. In instances throughout the report, multiple observations have been made regarding a single file. As a result, the number of observations may not always correspond to the number of files cited.
Line of enquiry 1: To determine whether evaluation criteria and selection plans were established in accordance with applicable laws, regulations and policies
15. Section 10.7.27 of the TBCP states that “competing firms should be told the measurement criteria and the weighting assigned to them. ...The courts have ruled that the factors and their weighting must be established beforehand and adhered to strictly. ...Fairness to all prospective contractors and transparency in the award process are imperative.” Using clear and precise language to define the evaluation criteria and selection methodology helps bidders prepare a responsive bid and evaluators to apply the same criteria equally to all bidders.
16. This LOE applied to 30 files where evaluation criteria were present. Of these, 24 contained both mandatory and point rated evaluation criteria, and 6 contained only mandatory criteria. All 30 applicable files were examined to determine if the evaluation criteria and selection methodologies were: aligned with the requirement; not overly restrictive; and clearly communicated in the solicitation. The method of allocating points to weighted criteria was also assessed to determine whether instructions were clearly communicated and reflected the relative importance of the criteria.
Overall, mandatory criteria were not overly restrictive and were aligned with requirements. There were, however, a number of files that did not define mandatory criteria in a clear, precise and measurable manner.
17. In 29 of the 30 applicable files, mandatory criteria were aligned with the requirements as described in the solicitation. The 1 exception was a solicitation for services to understand Canada’s strategic advantage within critical mineral supply chains. The solicitation, which was subject to several trade agreements, included a mandatory criterion requiring bidders to have a physical presence/location in Canada. This criterion was not aligned with the requirements and provided preferential treatment to domestic suppliers over suppliers from other nations with whom Canada has signed trade agreements.
18. Mandatory criteria were clear, measurable and demonstrable in 23 of the 30 applicable files. However, in 7 files mandatory criteria were not communicated in a clear, precise or measurable manner. Such evaluation criteria can cause bidders to submit non-compliant or sub-optimal proposals, result in unexpected or inconsistent evaluations between evaluators and generally undermine the transparency of the bid solicitation process. Examples of unclear mandatory criteria included the following:
- In a file for Information technology (IT) security vulnerability analysis specialist services, a mandatory criterion stated that the proposed resource must have “a university degree from a recognized university with acceptable specialization in computer science, IT security or in any other field relevant to the work to be performed or equivalent based on their education, training and/or experience.” There was, however, no indication as to what levels of education, training or experience would be required to be considered as “equivalent” to a university degree with acceptable specialization. OPO noted that ISED deemed that the eventual successful supplier met this mandatory criterion based on the undefined ‘equivalent to a university degree’ provision.
- In another file, a solicitation for economic analysis services, a mandatory criterion required bidders to demonstrate “significant experience delivering presentations of developed products.” While the criterion did clarify that these were to be presentations to Senior Officials (Deputy Minister level or higher) or Chief Executive Officer (CEO)s, it did not specify how much experience was required to be considered “significant”, meaning that delivering as few as 2 presentations to Senior Officials would meet the requirement.
- In a solicitation for protective apparel rental services, a mandatory criterion stated bidders “must supply all testing and safety information related to the fabric, construction, and design of apparel options” without specifying what was meant by “all testing and safety information.” In their consensus evaluation of the successful supplier’s bid, evaluators commented that it was “not clear if [there was] ARC protection for all items.” It should be noted that ARC protection refers to flash protective clothing.
- In a solicitation for economic analysis services, it was not clear what was required for a bidder to meet two of the nine mandatory criteria, requiring a subjective assessment by evaluators. For these two criteria, bidders were required to demonstrate that planned resources and the proposed team collectively were qualified for the level of work proposed. The criteria stated, “considered criteria include education, professional designations, and work experience.” However, there was no indication as to the level of education required, which professional designations would be accepted or what work experience was required to meet these mandatory criteria.
Point rated criteria
With limited exceptions, point rated criteria were not overly restrictive and rating scales reflected their relative importance. Overall, point rated criteria were clearly communicated in solicitation documents; however, issues were identified with the associated scoring grids in one-third of applicable files.
19. In addition to Section 10.7.27 noted above, Section 10.7.25 of the TBCP states that “criteria should identify accurately all the performance elements significant to the success of the project and should measure both the competence of the firm and the worth of its particular technical approach.”
20. In 22 of 24 applicable files, the point rated criteria were not overly restrictive and did not unnecessarily favour or penalize particular bidders. In one file, however, a solicitation for evaluation consultant services issued under the PSPC ProServices supply arrangement, a point rated criterion was designed to favour repeat ISED suppliers over other qualified suppliers that had not previously performed work for the department. This criterion awarded points for each completed evaluation project, up to five projects, that the proposed resource performed for ISED. In another file, for IT project executive services, 2 of the 10 point rated criteria related to experience managing sprints and sprint planning sessions (a sprint is a component of agile project management). These two criteria included weighting scales that awarded the maximum points available for 36+ months of experience, which differed greatly from other rated criteria that required 14+ years of experience to achieve full points. The one bid received included a proposed resource who had been working at ISED at the time of the solicitation. It indicated the resource had 3 years and 4 months experience with sprint giving the impression that the two sprint-related point rated criteria rating scales were designed to favor, or at least not penalize, this resource.
21. In 21 of applicable 24 files, point rated criteria were appropriate to the requirements and weighting factors reflected their relative importance. The remaining 3 files had point weighting schemes that could yield irrational results. In these files, for certain criteria the points increased rapidly bringing into question the relationship between years and months of experience and points to be awarded. For example, in a solicitation for IT Help Desk specialist services, a point rated criterion indicated a proposed resource with 5 to 5.5 years of experience providing IT user support in MS Windows 7 or later operating systems would receive only 4/20 points, while a proposed resource with greater than 7 years of experience would receive the full 20/20 points available. As a result, as little as 18 months experience beyond 5.5 years yielded 5 times the points.
22. In a similar case, in its determination in P&L Communications Inc. v. Department of Public Works and Government Services (PWGSC) (PR-2005-026), the Canadian International Trade Tribunal (the Tribunal) found that the government “unduly and inappropriately allocate[d] points to bidders with more experience.” In that case, the Tribunal noted that for multiple categories used in the RFP, points per year in rating scales were more than, or almost double, for bidders that scored in the highest-rated level, compared to those that just met the threshold. (In the ISED example cited above the difference was five-fold.) The Tribunal was of the view that PWGSC had “properly demonstrated the importance of establishing minimum levels of experience to ensure that the chosen supplier has a satisfactory level of stability and experience. However, it [was] of the view that PWGSC [had] failed to explain how an additional year or an additional project or two could warrant the award of the number of points contemplated in the RFP.” The Tribunal provided an example, similar to the ISED IT Help Desk specialist services example above, where the relationship between years and months of experience and the points to be awarded could “illustrate the unreasonable results that would flow from the application of the … point-allocation scheme.”
23. In 22 of 24 applicable files, the point rated criteria were clearly communicated in solicitation documents. In the other two files, point rated criteria used ambiguous, unclear or non-specific language and in one of those cases an element evaluated for a point rated criterion was not disclosed to bidders. For example:
- In a solicitation for the conduct of a price comparison study of telecommunication services, bidders were asked to "provide examples of projects for which the proposed project lead demonstrated professional work experience in producing analytical research reports on price comparisons of goods and services." Written comments from the bid evaluators indicated they only considered projects for which the proposed lead had also functioned as a project lead. That element, however, was not stated in the criterion, meaning the evaluators were applying an undisclosed criterion.
- In a solicitation for services related to intellectual property in innovation programming, there were two similarly worded point rated criteria, one of which clearly stated what was required while the other was open to interpretation. For both criteria, bidders were required to provide project examples to demonstrate professional work experience synthesizing information on public sector intellectual property policy. While one of the criteria specified that bidders were to provide examples of “foreign (non-Canadian)” projects, the other did not state a preference for domestic or foreign projects. Presumably ISED intended to assess experience with different types of projects in each of these criteria, for example one assessing foreign (non-Canadian) project experience and the other assessing domestic (Canadian) project experience; however, that was not clearly stated. Given the lack of precision in one of the criteria, a bidder with only foreign (non-Canadian) experience could have submitted the same foreign projects for both criteria.
24. Of the 24 files with point rated criteria, 10 had issues with respect to clarity of the scoring grids, for example, limited or no breakdown for some or all point rated criteria. Examples of scoring grids containing unclear information included the following:
- In a solicitation for spectrum auction verification services, it was not clear how the 30 possible points for a point rated criterion would be awarded. The criterion mentioned two elements: 1. demonstration of an understanding of the scope and objectives of the work; and 2. understanding of the clock auction format, clock round price incrementing rules and bid processing, and assignment stage winner and price determination. However, there was no indication how points were to be assigned for these two elements. The scoring grid stated that 27-30 points would be available if the bidder demonstrated an “excellent” understanding of the scope and objectives of the Work, 21-26 points for a “good” understanding or 0-20 points if the bidder “failed to demonstrate” an adequate understanding. However, it was unclear whether the maximum number of points available for each of the two elements was 15, or if a bidder could obtain as few as 0 points on one element and up to 30 points on the other. Details regarding how points will be awarded by evaluators should be disclosed to bidders in advance and should be clear to evaluators to support transparency and consistency in the evaluation process.
- In a solicitation for IT security analysis services, a point rated criterion listed four areas in which the bidder’s proposed resource should have experience, including identifying gaps in cybersecurity processes and preparing IT security risk briefings. This was to be demonstrated by the bidder providing 3 projects showing where experience in those four areas was obtained. The criterion stated, “points will be allocated as follows: up to 10 points for each area, up to a maximum of 40 points.” ISED, however, did not provide an explanation to assist bidders or evaluators in assessing how the points were to be awarded. For example, did a resource require experience in all 4 areas in all 3 projects to obtain the maximum 40 points, or was it possible to obtain 40 points by: (a) demonstrating experience in all 4 areas, but less than 3 projects; or (b) demonstrating experience in all 3 projects, but less than 4 areas? Without a clear explanation of how points were to be allocated, potential bidders would not know which projects/ areas to identify, and evaluators would not know how to award points.
- In a solicitation for spectrum auction services, bids were assessed against 8 point rated criteria labeled as RT1 – RT8. The first criterion (RT1) assessed past experience and was worth up to 40 points. For RT1, bidders were asked to provide information on three projects demonstrating relevant experience and a description of proposed team members’ experience producing written reports. From the RFP it was unclear how the maximum 40 points would be awarded. The remaining seven point rated criteria (RT2 – RT8) had maximum point values of either 5 points or 10 points each. There was no scoring grid in the RFP to indicate how points would be awarded for these criteria. The grid used by ISED bid evaluators, however, did provide greater clarity including a breakdown of the 40 points for RT1 showing 36 points for the three projects (maximum 12 points each) and 4 points for team members’ experience producing a written report. Further, the evaluation grid included a qualitative scale for each criterion that included “Excellent”, “Good”, or “Inadequate” ratings with accompanying descriptions and point ranges. For greater transparency, this information should have been provided to potential bidders in the solicitation document.
- In a solicitation for protective apparel rental services, a point rated criterion worth up to 50 points stated, “the Bidder should identify the type and the composition of material for each item being offered.” Up to 10 points was available in each of five categories: FR/ARC ratings, colors available, sizes available, thickness, and durability. There was no indication, however, as to what would merit 10 points for each category versus a lower score.
25. Clear, precise and measurable evaluation criteria and scoring grids enable bidders to know the requirements and the methods by which their proposals will be evaluated. Failure to adequately define evaluation criteria at the outset carries the risk that evaluators may struggle to interpret these criteria during the evaluation process and apply the same tests in assessing all bids. It can also be difficult to defend against external challenges to the evaluation process and demonstrate that criteria have been strictly adhered to when the criteria are unclear and open to multiple interpretations. Clear, precise and measurable evaluation criteria and scoring grids support fair and transparent procurement.
ISED should establish a mechanism to ensure mandatory criteria and scoring schemes for point rated criteria are clearly communicated in solicitation documents.
In all but one file reviewed, the selection methodology clearly communicated the manner in which the contract would be awarded. Similarly, with the exception of one file, the selection methodology also aligned with the requirements.
26. Common selection methodologies such as “highest responsive combined rating of technical merit and price” and “lowest price responsive bid” were used in almost all applicable files reviewed. In two files, both solicitations issued under the PSPC temporary help services (THS) supply arrangement, a less-common “right-fit” selection method was used. In both cases, use of the “right-fit” method was appropriate to the method of supply and respected the rules for these THS solicitations.
27. In one file, for research services to better understand Canada’s strategic advantage within critical mineral supply chains, the selection methodology was not clear in the original solicitation. It stated, “the contractor will be selected on the basis of highest responsive combined rating of technical merit and price.” This was partially clarified through a solicitation amendment that indicated 75% of the score would be based on technical merit and 25% would be based on price; however, the amendment did not explain how financial points would be calculated.
Line of enquiry 2: To determine whether solicitation documents and organizational practices during the bid solicitation period were consistent with applicable laws, regulations and policies
28. The TBCP sets out detailed procedures to ensure that government contracting is carried out in a manner that enhances access, competition and fairness and results in best value. Section 10.7 of the TBCP includes the minimum requirements to be included in the solicitation document as well as mandatory elements related to the design and execution of the process.
29. Solicitation documents must contain work descriptions or specifications defined in terms of clear outputs or performance requirements. Solicitation documents must also contain the objectives to be attained and timeframe for delivery and the assessment and award criteria.
30. For LOE 2, solicitation documents (excluding evaluation criteria and selection plans that were assessed under LOE 1) were assessed to determine whether they contained a clear description of the requirement and instructions necessary to prepare a compliant bid. The assessment of organizational practices included factors such as whether the solicitation was open to the appropriate number of bidders and for the required duration, and whether communications with suppliers supported the preparation of responsive bids.
Solicitation documents and processes
With minimal exceptions, solicitation documents provided a clear description of the requirements.
31. In 37 of 39 files reviewed, solicitation documents included a clear description of the requirements. In the other two files, there was insufficient documentation to conclude on clarity of the requirements. Details regarding the two files are provided below:
- In a file for mapping services, ISED requested bids from two suppliers over the phone. According to ISED, "the project had gone out to a company that could provide this type of service to see what is available and possible. Once we knew what was available, we were able to find a second company who could provide the same service to make this competitive. There are very few companies, that we were aware of, that could provide this sort of service." Without a properly documented file, OPO could not determine if a clear description of the requirements was provided to the 2 bidders.
- In a second file, ISED was required to issue an RFQ to seven pre-approved resellers listed in the applicable software licensing supply arrangement (SLSA). While there was an internal ISED email stating all resellers had been contacted, the actual solicitation email including a description of the requirements was not found in the file. Without the solicitation document OPO cannot conclude whether or not the description of the requirements was clear.
Solicitation documents from all applicable files included clear instructions for posing questions and submitting bids; however, certain solicitations included instructions to bidders for responding to mandatory and point rated criteria that conflicted with what was stated in the actual criteria.
32. In 10 files reviewed, solicitations were issued under the PSPC Task-Based Informatics Professional Services (TBIPS) supply arrangement. These followed a common structure and included mandatory and point-rated criteria. In all 10 files, the instructions preceding the mandatory technical criteria, stated:
Bidders must provide only the following information in their technical grid response to the mandatory resource criteria listed below. Failure to follow the instructions will render the proposal non-compliant.
- Project #,
- Client Organization/Project Name,
- Duration (month/year to month/year),
- Total level of work effort (# years/# months).
However, the mandatory instructions noted above were inconsistent with some of the mandatory criteria that followed, making it impossible for bidders to comply with both the instructions and the actual criteria themselves.
33. For example, an RFP for Senior Java Developer services under the Programmer/Analyst TBIPS category included eight mandatory criteria, two of which required the Bidder to provide details beyond the information listed in the instructions above. One mandatory criterion stated: “[T]he Bidder MUST indicate the resource category for which the proposed resource was contracted under.” If a bidder were to provide this information to comply with the mandatory criterion, it would simultaneously fail to follow the above-noted instructions and therefore be deemed non-compliant. Another mandatory criterion stated: “Bidders must include the brief client [project] description in the technical grid response or map the description in the proposed resource’s c.v.” Again, if a bidder were to include such a description, it would comply with the mandatory criterion but fail to follow the instructions.
34. Ensuring information is accurate, complete and consistent (i.e., not contradictory) eliminates ambiguities and helps ensure suppliers have the information they need to prepare and submit responsive bids. It also helps reduce the number of questions from suppliers regarding such discrepancies, which in turn reduces the effort required on the part of the department to respond to these questions during the solicitation period.
ISED should review its RFP templates to ensure clear and internally consistent instructions are communicated to bidders.
Overall, solicitations were open to an appropriate number of suppliers and for an appropriate period of time.
35. The duration, or period of time, a bid solicitation must remain open varies depending on the nature of the solicitation. When PSPC methods of supply are used, such as the TBIPS, ProServices, or THS supply arrangements, user departments must respect the bid solicitation rules published by PSPC. For example, for solicitations issued under the TBIPS supply arrangement, for requirements between $100,001 and $3.75 million, the minimum bidding period is 15 calendar days. User departments do have the option to extend the time beyond the minimum based upon a requirement's complexity. For solicitations not issued under a supply arrangement, but covered by one or more trade agreements, the rules of the applicable trade agreement(s) will apply. While exceptions exist, generally, trade agreements such as World Trade Organization-Agreement on Government Procurement (WTO-AGP), the Comprehensive Economic and Trade Agreement, and the Canada-Colombia Free Trade agreement require a 40-day bidding period.
36. In 31 of 33 applicable files, OPO was able to confirm that the duration of the bid solicitation period and the number of suppliers invited to bid was appropriate given the solicitation method used and any applicable trade agreements. For the two remaining files, there was insufficient information to support that finding. One file was a verbal solicitation, and the other was a solicitation issued under an SLSA where ISED was required to issue an RFQ to the 7 pre-approved resellers and provide at least 5 working days for a response. While there was an internal ISED email stating all resellers had been contacted, the number of days bidders were given to submit bids is not known as the actual correspondence was not found in the file.
Communication with suppliers
Communications with suppliers appeared to be appropriate and supported the preparation of responsive bids; however, documentation of communication was incomplete in many files.
37. Section 2 of the TBCP states: “Government contracting shall be conducted in a manner that will: (a) stand the test of public scrutiny in matters of prudence and probity, facilitate access, encourage competition, and reflect fairness in the spending of public funds.” Section 12.3.1 of the TBCP also states: “Procurement files shall be established and structured to facilitate management oversight with a complete audit trail that contains contracting details related to relevant communications and decisions…”. These requirements apply to all aspects of the procurement process, including interactions with suppliers.
38. In 22 of the 39 files reviewed, there was an indication of communications (i.e., questions and answers) with suppliers during the solicitation period. Overall, the responses provided by ISED appeared to adequately address supplier questions. Depending on the solicitation method used, responses were either posted to Buyandsell.gc.ca or provided by email to all invited suppliers. However, file records of communication with suppliers were incomplete. For example, 7 of the 22 files that included responses from ISED to supplier questions did not include the email in which the supplier posed the question. A complete file record would include all relevant correspondence from the email with the original supplier question through communication of the final response from ISED to suppliers. Without a complete record of communications, OPO is unable to determine if all suppliers’ questions were answered, answered in a timely manner, and whether all suppliers were treated equally and received the same information at the same time.
ISED should establish a mechanism to ensure that all relevant communications with suppliers are properly documented and retained on the procurement file.
Regret letters to unsuccessful bidders were generally adequate; however, award notices were not posted for a significant number of solicitations posted to Buyandsell.gc.ca.
39. In 22 of 32 files reviewed, there were multiple bidders that responded to the solicitation. In 19 of those 22 solicitations, the files contained regret letters or similar correspondence to unsuccessful bidders. In the remaining three files with multiple bidders, there was no indication that regret letters were sent to all unsuccessful bidders. Regret letters are an important tool as they alert bidders to the fact that they were unsuccessful in the selection process, should they wish to seek a detailed debriefing from the contracting department or avail themselves of potential recourse mechanisms such as OPO or the Canadian International Trade Tribunal. Regret letters also provide valuable information that may be useful to improve the bidders’ chances for success in future contracting opportunities.
40. In 17 of the 39 files reviewed, the solicitation or tender notice was posted publicly on Buyandsell.gc.ca. In only 2 of those 17 solicitations did ISED post an award notice within a reasonable timeframe to notify stakeholders that a contract had been awarded as a result of the procurement process. In the other 15 solicitations, 4 award notices were posted several months after the contracts were awarded and 11 solicitations had an expired contract status giving the false impression that a contract had not been awarded as a result of the solicitation process.
ISED should establish a process to ensure that applicable award notices are published within required timeframes following contract award.
Line of enquiry 3: To determine whether the evaluation of bids and contract award were conducted in accordance with the solicitation
41. In order to ensure the fairness and defensibility of evaluation processes, section 10.7.27 of the TBCP requires that evaluation criteria and their weighting be established beforehand, adhered to strictly and applied equally to all bidders. Failure to ensure the consistent evaluation of proposals increases the risk that ambiguities in the selection process result in the contract being wrongly awarded. Inconsistent evaluations may also call into question the integrity of the procurement process.
42. Of the 39 files reviewed, 30 had a bid evaluation process that included a technical and financial evaluation of bids, 3 had a bid evaluation process focussed on a financial evaluation of bids (i.e., RFQ—no technical evaluation criteria) and 6 were call-ups under standing offers. Excluding the 6 call-ups, these files were examined to determine whether a process had been established to ensure the consistent evaluation of bids, evaluations had been carried out in accordance with the planned approach, and results of evaluations were adequately documented.
Bid evaluation process
In nearly half of the applicable files, the bid evaluation procedures deviated from the procedures described in the solicitation document resulting in several instances where bidders were wrongly deemed non-compliant or the contract was awarded to the wrong bidder.
43. The requirements of the TBCP referenced above emphasize the importance of having a well-planned and documented process for bid evaluations. In nearly all of the 30 files reviewed that included a technical evaluation, OPO observed that evaluation grids were used with criteria that matched those found in the solicitation document, and the number and identities of the evaluators had been recorded in the file. The limited exceptions included two files that were missing documentation of the technical evaluation. One of those two files included an evaluation grid template; however, it did not include all mandatory criteria that appeared in the solicitation. For this file, OPO was unable to determine if all criteria were evaluated and if the contract was properly awarded, because the completed evaluations were not included in the file.
44. The evaluation of bids must be performed with integrity in a fair, honest and transparent manner. While the contracting authority is ultimately responsible for the process, the evaluation of the technical portion of bids is the responsibility of the evaluation team. Based on the files reviewed, ISED’s evaluation team is typically comprised of employees from the technical authority’s unit. Most often, ISED used a 3-person evaluation team, which included the technical authority and two other employees from that unit with subject matter knowledge.
45. Generally, while bid evaluation team members have the technical knowledge required to evaluate bids, they rely on guidance provided by the contracting authority to ensure procurement rules are followed. In 25 of the 30 applicable files, there were written instructions from the contracting authority to bid evaluators. A good practice observed in 19 files was the use of a “notice of non-disclosure” or equivalent guidance document. The notice of non-disclosure is a form that must be signed by each member of the ISED bid evaluation team prior to the start of the bid evaluation process. By signing the form, evaluators confirm that they would only divulge information to departmental or other officials authorized to participate in the bid evaluation process. The form also includes guidance to evaluators. This guidance stresses the importance of consistency in the process and full documentation of results on the evaluation grid. In 5 of the 30 files that included a technical evaluation, there was no record of instructions to the evaluators located in the file.
A significant number of files had issues with the bid evaluation procedures resulting in contracts being wrongly awarded or bidders incorrectly being found non-compliant.
46. In 18 of 33 files, file documentation demonstrated that technical and financial bids were evaluated in accordance with the planned approach. In 5 files, the absence of key documents including the successful supplier’s technical proposal, quotes in response to an RFQ, and completed bid evaluation grids meant the file did not contain sufficient information to demonstrate that evaluations followed the planned approach or that the successful supplier was selected as per the methodology specified in the solicitation. In the remaining 10 files, documentation showed the bid evaluation procedures that were applied had deviated from the procedures described in the solicitation document, which resulted in 4 contracts being awarded to the wrong bidder, and 6 bidders incorrectly deemed non-compliant. For example:
- In one file, a solicitation for a multi-year contract valued at over $3.3 million for IT help desk services with work authorized through a task authorization process, the contract was awarded to a non-responsive bidder. The basis of selection in the RFP was highest combined rating of technical merit and price. To be deemed responsive, a bid was required to comply with the requirements of the bid solicitation and meet all mandatory criteria. One such requirement stated, “any bidder proposing a total cost that falls between the “Median” minus 15% and the “Median” plus 15% will be considered financially responsive. Any bidder proposing a total cost that falls outside the “Median” minus 15% and the “Median” plus 15% will be considered non-compliant and no further consideration will be given to that bid.” There were 4 bids received, one was eliminated because it failed to meet all mandatory technical criteria and the other 3 were deemed responsive. All 3 responsive bids received full points for the point rated criteria, meaning the bid price was the determining factor in identifying the successful supplier. ISED did not apply the requirement for bids to be within 15% of the median price and awarded the contract to a non-compliant bidder with a bid price more than 15% below the median. Approximately two months after contract award, ISED discovered that it had wrongfully awarded the contract. Despite the fact the contract would be in place for 4 years with option periods and that the vast majority of resources who would eventually work on the contract had not yet been identified through the task authorization process, ISED decided to continue with the contract, rather than cancel it and re-solicit the requirement, citing a concern over interruption in service if it were cancelled. A note was added to the file to document the error.
- In two files, both solicitations for IT services issued under the TBIPS supply arrangement, bids were incorrectly rejected for not meeting all mandatory criteria due to ISED’s method of calculating experience of the proposed resource. In one file, an objection was received from a bidder that had been deemed non-compliant. The evaluators had found that 4 of the mandatory criteria (M2, M4, M5, and M6) were not met because they did not believe the proposed resource met the years of experience requirements. The RFP stated, "in situations in which a proposed resource worked at the same time on more than one project, the duration of any overlapping time period will be counted only once toward any requirements that relate to the individual’s length of experience.” The evaluators’ notes showed the decision was made because one project from the proposed resource's CV overlapped with other projects. However, project X was not cited by the bidder in the mandatory technical grid in response to the 4 above noted mandatory criteria and should not have factored into the evaluation of those criteria. Project X was only cited in the bid under one of the point rated criteria. There was no reason given to explain why evaluators considered experience from project X over projects that were cited by the bidder for the 4 criteria in the mandatory technical grid. Additionally, the Mandatory Evaluation Criteria instructions in the RFP stated, "only information cited in the Mandatory technical grid response will be evaluated." OPO cannot determine where the bid would have ranked if it had passed the mandatory criteria because ISED did not evaluate the point rated criteria; however, if it had passed the technical evaluation, it would have been the lowest priced compliant bid. In response to the bidder’s objection to being deemed non-compliant, ISED stated that they stood by their decision. According to ISED’s response, it was the supplier’s “failure to account for the overlap and to ensure an accurate level of work [that] rendered its bid non-compliant.” By not being able to demonstrate evaluations were conducted in accordance with the terms of the solicitation (i.e., "only information cited in the Mandatory technical grid response will be evaluated"), the integrity of the procurement process is brought in to question.
- In one file for IT Project Executive services, the contract valued at over $1.7 million was awarded to a bidder that should have been deemed non-responsive. The solicitation, which was issued under the TBIPS supply arrangement, included 8 mandatory criteria and 10 point rated criteria. Though the successful supplier’s bid was deemed to have met all mandatory criteria, the bid did not actually demonstrate that the proposed resource met the requirements of mandatory criteria M2 and M3. Criterion M2 stated, “the proposed resource must have greater than ten (10) years professional work experience with-in the last fifteen (15) years working as a project executive.” And M3 stated “the Bidder must provide two (2) client references … where the proposed resource was contracted as a Project Executive within the last five (5) years from the date of bid closing.” However, in response to another mandatory criterion (M1), pertaining to work conducted at ISED in the last 5 years, the bid stated the “resource category for which the proposed resource was contracted under [as] Project Leader”. Project Executive and Project Leader are separate categories under TBIPS, meaning the individual’s recent experience at ISED as a Project Leader should not have been considered towards meeting M2 and M3 requirements for Project Executive experience, resulting in the criteria not being met. There were no notes in the evaluation grid to explain or clarify why the bid was deemed to have met all mandatory criteria.
- In one file, for economic analysis services, the solicitation had 5 mandatory and 5 point rated criteria and a basis of selection of highest combined rating of technical merit and price. Two bids were received. A 3-person bid evaluation team conducted the technical evaluation and determined that both bids met all mandatory criteria and exceeded the minimum points requirement for point rated criteria. The contracting authority then completed the evaluation process by conducting the financial evaluation, which determined the lower ranked technical bid was ranked 1st overall based on its lower price and superior financial score. After the results were shared with the bid evaluation team, the technical authority said he was concerned that they may have overlooked something in the mandatory criteria and wanted to re-evaluate the 1st overall ranked bid. The contracting authority initially told the evaluators “unfortunately, since the financial evaluation has been completed and we had already come to a consensus on the technical evaluation we cannot go back now.” The response from one of the evaluators stated he would be “happy to delete the financial evaluation email.” Ultimately, ISED did revise its evaluation of the 1st ranked bidder and deemed it non-compliant for not meeting all mandatory criteria. While some aspects of the procurement process were well documented, there was a significant shortcoming with respect to documentation of the decision to allow the evaluators to change their evaluation of the 1st ranked bid after they learned the results of the process. The actions taken and ultimate results of this evaluation process leave ISED open to the perception that this contract was not awarded in a fair and transparent manner.
47. Bid evaluations not conducted consistently and in the manner prescribed by the solicitation call into question the integrity of the procurement process. Failure to strictly adhere to evaluation criteria increases the risk that non-compliant bids will be considered, and in some cases as seen above, awarded government contracts.
ISED should update its procurement guidance and training, and implement an oversight process and review mechanisms to ensure that evaluations are carried out in accordance with the planned approach specified in the solicitation, and that contracts are not awarded to non-compliant bidders.
Completed bid evaluation documentation was missing from a number of files. This was particularly evident among solicitations that pre-dated the COVID-19 pandemic.
48. Section 12.3.1 of the TBCP requires that procurement files facilitate management oversight with a complete audit trail containing details related to relevant communications and decisions, including the identification of the involved officials and contracting approval authorities. The requirement to ensure adequate file documentation extends to the actions undertaken during the solicitation period, the evaluation of bids, as well as the overall procurement file.
49. File documentation was reviewed to determine whether a complete audit trail was retained to support consistent and transparent decision-making. As was highlighted throughout this report, there were multiple files in which key documents, including correspondence with prospective bidders, instructions to bid evaluators, and an RFQ solicitation were missing. OPO recognizes that ISED’s ability to access hard-copy files may have been impacted by COVID-19 pandemic-related office closures and that some of the missing documentation may be located in the ISED offices.
50. Regarding technical evaluations, 7 of 30 files that included mandatory or point rated criteria were missing evaluation documents. Three of those files were missing completed individual and consensus evaluations, and the others did include completed consensus evaluations, but the individual evaluations were not present in the files. Additionally, one file was missing the technical proposal from the successful supplier.
51. OPO considers the lack of documents on file to support decisions to be a concern, particularly with respect to decisions regarding contract award. For files with incomplete records, ISED cannot fully explain its business decisions, inhibiting its ability to demonstrate good stewardship of Crown resources or that the procurements were conducted in a fair, open and transparent manner. Keeping complete and detailed evaluation records is crucial for demonstrating that evaluation criteria have been applied consistently to all competing bids, demonstrating that the procurement has been carried out in a manner consistent with ISED’s obligations under the TBCP and applicable trade agreements, and that the contract was awarded in accordance with the procedure set out in the solicitation documents and to the actual winning bidder.
52. It should be noted, however, that the scope period for this review was October 1, 2019 to September 30, 2021. In March 2020, employees at ISED were forced to pivot to a remote working model due to the COVID-19 pandemic. Six of the seven files with missing evaluation documents had solicitation periods that closed before March 2020, i.e., pre-pandemic and one closed in April 2020, shortly after the lockdown began. Prior to the pandemic, when employees were working from the office, paper-based evaluation processes were more commonly used. It is important that information management practices, whether paper-based or digital, are resilient to threats posed by dramatic events such as the COVID-19 pandemic as well as more predictable events such as the departure of contracting authorities.
53. OPO regularly hears from both Canadian businesses and federal officials who believe the contracting process is unnecessarily complex. In reviewing ISED’s procurement practices, OPO sought to identify opportunities to alleviate unnecessary administrative burdens placed on bidders and federal procurement officials and draw attention to good practices for simplifying the procurement process.
54. ISED frequently requires bidders to provide references, but rarely checks those references during the bid evaluation process. In 20 of the 28 files reviewed that included an RFP solicitation process, ISED asked bidders to provide references including contact information. In 19 of the 20 cases, references formed part of mandatory criteria, meaning if they were not provided as instructed in the solicitation the bid would be deemed non-responsive. Extensive reference requirements were noted in some files, for example in one file references were requested for two mandatory and one point-rated criteria. In another file, 4 mandatory criteria required references and in a third file, references were required to respond to 6 mandatory criteria.
55. When used appropriately and verified consistently, references can provide valuable information to support the bid evaluation process. OPO noted that solicitations, generally, also included a reference check clause stating it is Canada’s discretion whether or not to check references. However, in only 1 of the 20 files was there any indication that any references were checked as part of the bid evaluation process, thus placing an unnecessary burden on bidders.
Standardization of procurement documents
56. ISED RFPs reviewed for this review were broadly similar in their basic structure, with certain variances in layout and content that reduce uniformity and consistency. Most RFPs not issued under a PSPC supply arrangement, i.e., open and traditional competitive solicitations, followed a standardized layout and format. Most of these RFPs included ISED General Conditions, while some referenced general terms, conditions and clauses from the Standard Acquisition Clauses and Conditions (SACC) Manual. For RFPs issued under various PSPC supply arrangements, all but one file followed a standardized format appropriate to the particular supply arrangement. Consistent use of standardized procurement documents contributes to greater simplification by ensuring consistency and uniformity across procurement processes. It can also decrease the cost to bid for suppliers and potentially lower bid prices for departments.
Redundant technical criteria
57. All RFPs include terms and conditions that must be respected for a bid to be deemed responsive. For example, solicitations state the time and date by which bids must be received, the address (physical or electronic) to which bids must be delivered, format that must be followed (for example, technical bid and financial bid submitted as separate documents), and certifications required with the bid. Prior to conducting technical evaluation of mandatory and point rated criteria, contracting officials must verify that all terms and conditions of bid submission have been met. For RFP’s including mandatory or technical criteria, as seen with the ISED files reviewed, an evaluation team evaluates compliant bids against those criteria.
58. In eight files reviewed, RFPs included redundant mandatory criteria that simply restated certain requirements for bid submission. Most commonly, the mandatory criteria stated that financial bids must be submitted separately from technical bids. Including some, but not all, bid submission requirements in mandatory criteria could add confusion as to what conditions must be met at time of bid submission. Removal of these redundant requirements from the mandatory criteria would reduce the number of criteria listed in the RFP and simplify the process for the bid evaluation team.
59. ISED’s procurement practices pertaining to evaluation and selection plans, solicitation, and evaluation of bids and contract award were assessed for consistency with Canada’s obligations under sections of applicable national and international trade agreements, the Financial Administration Act and regulations made under it, the TBCP and departmental guidelines, and to determine if they supported the principles of fairness, openness and transparency.
60. Regarding LOE 1, overall OPO concluded that evaluation criteria and selection plans were established in accordance with applicable laws, regulations and policies. Mandatory criteria were, for the most part, not overly restrictive and were aligned with requirements of the solicitation. However, roughly a quarter of files did not define mandatory criteria in a clear, precise and measurable manner. Similarly, point rated criteria were also not overly restrictive; however, issues were identified with scoring grids for point rated criteria in one-third of applicable files. Finally, the selection methodology clearly communicated the manner in which the contract would be awarded and was aligned with the requirements in nearly all files reviewed.
61. Regarding LOE 2, with certain exceptions, OPO concluded that solicitation documents and organizational practices during the bid solicitation period were consistent with applicable laws, regulations and policies. OPO found that, with minimal exceptions, solicitations were open to an appropriate number of suppliers and for an appropriate period of time. Solicitation documents provided a clear description of the requirements. Further, solicitations included clear instructions for posing questions and submitting bids. However, certain solicitations included instructions to bidders for responding to mandatory and point rated criteria that conflicted with what was required to meet the actual criteria. Communications with suppliers during the solicitation period appeared to be appropriate and supportive of the preparation of responsive bids; however, documentation of communication was incomplete in many files. Incomplete documentation resulted in unsupported procurement actions that risk undermining the integrity, fairness and transparency of the procurement process. Finally, regret letters to unsuccessful bidders were generally adequate, however, award notices were not posted for a significant number of solicitations posted in a timely manner to Buyandsell.gc.ca thereby inhibiting the transparency of these processes.
62. Regarding LOE 3, OPO concluded that ISED’s bid evaluation processes did not consistently ensure contracts were awarded in accordance with the solicitation. OPO found that a process had been established to support the consistent evaluation of bids and most files reviewed included guidance for bid evaluators. However, in nearly half of applicable files, the bid evaluation procedures that were applied deviated from the procedures described in the solicitation document or there was insufficient documentation to demonstrate that planned procedures were followed. This led to non – compliant bids being considered for award and, in some cases, to contracts being wrongly awarded to non – responsive suppliers.
63. In order to address issues identified, OPO made five recommendations. These recommendations can be found in Annex I of this report.
VI. Organizational response
64. ISED would like to thank the Office of the Procurement Ombudsman (OPO) for the initial review of its findings. We are pleased that the OPO is in agreement that the fundamentals of procurement are well exercised at ISED and that there are no causes for major concern, with most findings appearing to be administrative in nature. The findings of the OPO provide the opportunity to strengthen business processes and implement best practices. ISED would note that the period sampled by the OPO falls directly within the outbreak of the COVID 19 pandemic. With the resulting need to abruptly shift from paper to electronic files, and with staff suddenly working from home, it is possible that some administrative or record keeping omission may have occurred that would not normally be the case.
65. We also observe that the review of ISED’s procurement functions has led to findings that are also in-line with the OPO’s review of other medium and large Departments across Government. Given these overall findings and in the spirit of continuous improvement, ISED is in agreement with the recommendations and is committed to implementing the responses below.
66. OPO wishes to express its appreciation to the staff of ISED’s Corporate Finance, Systems and Procurement Branch for the assistance and cooperation extended to the reviewers during this assessment.
Annex I. Organizational response and action plan
Procurement practice review of evaluation and selection plans, solicitation, and evaluation of bids and contract award at Innovation, Science and Economic Development Canada.
|Number||Recommendation||ISED response / Action plan||Timeline for implementation|
|1||ISED should establish a mechanism to ensure mandatory criteria and scoring schemes for point rated criteria are clearly communicated in solicitation documents.||The OPO found that ISED’s evaluation criteria and selection plans were established in accordance with applicable laws, regulations and policies and that mandatory and point-rated criteria are generally not overly restrictive and were aligned with requirements of the solicitation. The OPO also found that ISED did not clearly define mandatory criteria, fully document scoring grids for point rated criteria or clearly communicate the selection methodology in a minority of files reviewed. As such, ISED accepts the OPO’s recommendation that these items be addressed with greater consistency across all procurement files. As noted in the OPO’s report (page 3), in 29 of 30 applicable files, mandatory criteria were aligned with the requirements as described in the solicitation, and in 22 of 24 applicable files, the point rated criteria were clearly communicated in the solicitation documents (page 6). These point to ISED having existing strong controls and practices. However, to increase overall consistency, ISED will update its procedures and guidance documents to procurement staff, and also pilot a peer review process across all levels for the next 6 months to ensure clarity and consistency. At the end of the pilot period, all new entry level hires will have their work peer reviewed for a period of prescribed duration. The application of these standardized criteria and scoring will be monitored and reported on via ISED’s already established quality assurance review function to allow for ongoing improvement.||Q4 of 2022-23|
|2||ISED should review its RFP templates to ensure clear and internally consistent instructions are communicated to bidders.||
ISED thanks the OPO and agrees with the recommendation. As noted by the OPO, in 37 out of the 39 files reviewed, ISED was fully compliant and consistent in providing clear and internally consistent instructions to bidders. Further, as the OPO notes, 31 of 33 files had bid solicitation periods that were of an appropriate length to the OPO, that overall solicitations were open to an appropriate number of suppliers and for an appropriate period of time (page 10). In response to the OPO’s observation that “communications with suppliers appeared to be appropriate and supported the preparation of responsive bids, however documentation of communication was incomplete in many files” (page 11), ISED would note that as a result of the pandemic, and the abrupt transition from paper to electronic filing it caused, there may have been inadvertent inconsistencies in the completeness of files as they may have had both electronic and paper records, introducing a new challenge in maintaining one complete file and leading to the observations noted by the OPO.
In the spirit of continuous improvement, ISED performs quality assurance review on this procurement activity to ensure overall effectiveness and compliance. In this context, ISED agrees with the OPO and is committed to reviewing its RFP templates and administrative practices to ensure standard, clear and consistent messaging and tools. Further, as noted above, ISED will undertake a pilot peer review scheme as well as support ongoing reviews from its internal quality assurance group.
|Q4 of 2022-23|
|3||ISED should establish a mechanism to ensure that all relevant communications with suppliers are properly documented and retained on the procurement file.||
ISED notes this observation and potential to improve best practices, and highlights that the OPO found in 19 of 21 solicitations, the files contained the necessary documentation. ISED notes this observation and that the period of the OPO’s focus fell during the COVID crisis. As a result of the pandemic, and the abrupt transition from paper to electronic filing it caused, there may have been inadvertent inconsistencies in the completeness of files as they may have had both electronic and paper records, making it difficult to maintain one complete file and leading to the observations noted by the OPO.
Since this transition to fully electronic files in March 2020, ISED has implemented improved electronic document management practices and standards leveraging the departmental corporate Information Management tool. ISED’s established Quality Assurance function also conducts post monitoring of procurement files and communicates fundings and areas of improvements to ensure continuous optimization of processes, tools and guidelines.
ISED will continue to emphasize the importance of document management as part of its training and guidance to its procurement community.
|Q4 of 2022-23|
|4||ISED should establish a process to ensure that applicable award notices are published within required timeframes following contract award.||ISED fully supports this observation and recommendation. While the OPO report notes that ISED consistently ensures that award notices are published within the required timeframes, there are areas for improvement in the files selected. ISED also notes that the period of the OPO’s focus fell well within the COVID-19 period where files may have been exceptionally incomplete due to the rapid and unprecedented need to transition to working from home, and the consequent shift from paper based to electronic files which may have led to this observation. However, ISED is committed to adopting best practices and in future employee onboarding and training will further emphasize the importance of timely posting. In parallel, ISED is currently updating processes and tools for its procurement officers, which will provide instructions on procedures and service standards (timelines) for the posting of award notices and the issuance of regret letters, ISED will also ensure continuity of procurement files should a transition between procurement offices be required. Ongoing monitoring will occur via ISED’S established quality assurance function to ensure continuous improvement, including quarterly reports to senior management.||Q4 of 2022-23|
|5||ISED should update its procurement guidance and training, and implement an oversight process and review mechanisms to ensure that evaluations are carried out in accordance with the planned approach specified in the solicitation, and that contracts are not awarded to non-compliant bidders.||
ISED fully agrees with the need for consistency in training and application of procedures. The overall findings of the OPO report appear to support the conclusion that ISED executes the fundamentals of procurement with a good assurance. Nonetheless, there is an opportunity to strengthen practices and controls especially in the evaluation process, and so guidance and training for procurement specialists, as well as piloting a peer review process to further strengthen the evaluation process, will be revisited as appropriate. ISED is also developing a curriculum for Procurement Functional Specialist and updated training as part of its newly launched community development program for purchasing and supply group (PG)s.
Other findings in the report relate to inconsistencies in the evaluation process specifically related to Information technology contracts, where ISED’s current model is heavily reliant on non procurement subject-matter experts. These findings present ISED with an opportunity to more fundamentally restructure the manner in which Information Technology (IT) contracts are overseen. As a result, ISED will immediately supplement its existing oversight and review mechanisms by increasing the involvement of senior management. ISED will also undertake a review of the current operating structure and strengthening its governance model to ensure clear accountability, roles and responsibilities and improve overall compliance and IT procurement strategy. Further, ISED will ensure on-going monitoring via its established quality assurance function with increased rigour on IT related contracts and reporting to senior management.
|Q4 of 2022-23|
- Date modified: