Procurement Practices Review 2009-2010 - Chapter 3: Bid Evaluation and Selection Methods

Executive Summary

3.1 Public Works and Government Services Canada (PWGSC) and other government departments utilize procurement plans that address operational needs and proposed methods of solicitation, bid evaluation and supplier selection, as well as the identification of key risks and mitigation strategies to support the effective acquisition of goods and services.

3.2 A clearly defined procurement plan provides the foundation for the effective management of decisions that take place throughout the life of the procurement. It also provides the baseline against which results can be measured and improvement plans developed. Proper procurement plans are particularly important when setting up supply arrangements (SAs) that may cover several years and are used by departments to award multiple contracts to pre-qualified suppliers.

3.3 The Treasury Board (TB) Contracting Policy establishes the framework within which contracting authorities must be exercised. The Policy states: "The objective of government procurement contracting is to acquire goods and services and to carry out construction in a manner that enhances access, competition and fairness and results in best value or, if appropriate, the optimal balance of overall benefits to the Crown and the Canadian people."

3.4 The Policy defines "best value" as "the combination of price, technical merit, and quality, as determined by the contracting authority prior to the bid solicitation and set out in the bid solicitation evaluation criteria, and which forms the basis of evaluation and negotiation between buyers and sellers to arrive at an acceptable basis for a purchase and sale." It further states that factors such as all relevant costs over the useful life of the acquisition should be considered.

3.5 Some suppliers have expressed concerns about the complexity and ambiguity of bid evaluation and selection methods, which led them to question the fairness, openness and transparency of the government's procurement practices. The Association of Registered Graphic Designers of Ontario (RGD Ontario) approached the Office of the Procurement Ombudsman (OPO) with concerns about competitive bid evaluation and selection methods that raise questions about the following:

• Whether bid evaluation and selection methods took into account an appropriate balance between cost and quality or other factors that add value;
• Whether the bid solicitation methods were aligned with industry capacity and conditions;
• Whether the contracts awarded met the operational needs of purchasing organizations; and
• Whether the bidding requirements were fair.

3.6 The government's 2008 Speech from the Throne emphasized that ways needed to be found to make it easier for businesses to provide products and services to the government and deliver better results for Canadians.

3.7 In 2009-2010, the OPO chose to conduct a review of the policies and practices relating to competitive procurement planning and corresponding solicitation, bid evaluation and selection methods relevant to the purchase of graphic design services (GDS). Government departments have established a number of SAs to procure graphic design services. SAs are a method of supply used to procure goods and services from a list of pre-qualified suppliers. This list is established through a competitive process that measures both quality and price. This is the first stage of bidding. In the second stage of bidding, departments meet their specific needs by issuing a subsequent call for bids to suppliers on the list. This call for bids may be made to a single supplier, a number of suppliers or all suppliers on the list, depending on the details in the SA.

3.8 The practice review focussed on three SAs. Two of these were set up by PWGSC and one by Transport Canada (TC). The objective of the review was to determine whether the three SAs for GDS were based on adequate strategies and plans to support the development of effective and efficient bid solicitation, evaluation and selection methods and to determine whether these plans and their practical implementation.

• support operational needs;
• reflect best value considerations; and
• encourage both competition and quality.

3.9 The OPO selected a sample including the three SAs and 39 related files from contracting activity undertaken between June 2006 and January 2010. The sample was drawn from the contracting data in three of the departments identified as significant purchasers of GDS: Human Resources and Skills Development Canada (HRSDC), Natural Resources Canada (NRCan) and Transport Canada (TC). Public Works and Government Services Canada was the common service procurement provider for HRSDC and NRCan. We noted that the vast majority of contracts awarded under the SAs were for less than $25,000 and thus, in accordance with the Government Contracts Regulations, could have been awarded without any competition.

3.10 We reviewed policies, procedures and guidance in each department. In addition, interviews were conducted with procurement personnel, and in three departments, information was also obtained from program personnel. Practices were assessed against four criteria that support the development of an effective and efficient procurement strategy:

1. Departmental procurement policies and processes support the development of effective and efficient procurement plans and corresponding solicitation, bid evaluation and selection methods.
2. Procurement plans and practices ensure operational needs are supported and clearly defined.
3. Departmental procurement plans and practices reflect key best value considerations.
4. Departmental procurement plans and practices integrate the principles of fairness, openness and transparency, including encouraging competition and allowing innovation, while respecting the requirements of legislation, trade agreements, regulations and procurement policies.

3.11 We noted that all four departments had defined roles and responsibilities in terms of specific procurement activities. Departmental policies and procedures reflected the requirement to achieve best value as well as the principles of fairness, openness and transparency. The primary focus of their guidance was on ensuring the adequacy of statements of work and compliance with trade agreements and the Treasury Board (TB) policies. Competitive bidding was encouraged in all departments. Each department had established checklists to ensure that these key considerations were addressed.

3.12 At PWGSC, we observed requirements to obtain written confirmation of the client's agreement with the statement of work (SOW) and, for sensitive or high-risk procurements, with the evaluation and selection methodology. We believe that for such procurements where multiple organizations are involved, obtaining sign-offs is an effective means of minimizing risks of acquiring goods and services that do not meet operational needs.

3.13 In terms of the practical application of the principles of fairness, openness and transparency, PWGSC and HRSDC reminded personnel that competitive solicitation of bids for requirements valued below $25,000 should be pursued whenever it was cost-effective to do so. At TC, contracting procedures included a list of relevant considerations to assist contracting specialists in vetting the bid evaluation and selection methods to ensure that they were adequate to encourage competitive bidding.

3.14 We noted, however, that in some instances, departmental policies needed to be updated to remain in sync with changes to trade agreements and TB policies. For example, at HRSDC, the policy on intellectual property had not been updated to reflect new TB policy requirements.

3.15 Overall, the policies and guidance relating to the procurement planning of the SAs reviewed support the principles of fairness, openness and transparency; reflect the requirement to achieve best value; and allow for the development of effective and efficient procurement plans.

3.16 We reviewed the procurement planning for each of the SAs and expected to find detailed procurement plans that clearly showed the processes and procedures that would be followed by procurement personnel during the procurement. We looked to see if the decisions reflected in the plans were supported by documentary evidence of proper analysis in support of risk evaluation and the decisions made.

3.17 The review found that two of the three SAs were supported by procurement plans. PWGSC had prepared the plans for HRSDC and NRCan. We note that TC, which had issued its own SA, did not have a formal supporting procurement plan. We were informed that in that department, some planning activity was carried out by program officers and that it influenced the decisions reflected in the SA.

3.18 Both of the plans we reviewed contained high level descriptions of required services, estimated business volume, solicitation and evaluation methods, relevant trade agreements, and policies and milestones for establishing the SA. We further examined files and supporting documentation to understand the rationale and basic analysis for the bid evaluation and selection methods used for the three requests for supply arrangements (RFSAs) and the subsequent bidding processes conducted at stage 2. We found the documentation to be weak or missing in a number of areas, including identification of risks and corresponding mitigation strategies and documented analyses supporting the rationale for the use of an SA over other procurement options, for estimates of business volumes, and for the evaluation and selection methods to qualify suppliers for the SA and subsequent calls for bids.

3.19 The complexity and diversity of the bid evaluation and selection methodologies being used caused confusion for suppliers. The departments had similar requirements for GDS but used different detailed methods of evaluation and selection. For example, each of the three user departments had different thresholds and requirements for competitive bidding at stage 2. At HRSDC, competitive bidding was required for individual requirements valued in excess of $15,000. At NRCan, all purchases were to be competed. At TC, the competitive bidding threshold was set at greater than $25,000.

3.20 In addition, for the NRCan RFSA, we found the lack of clarity in the description of mandatory and rated requirements resulted in 35 questions from suppliers, creating an unnecessary level of effort on the part of departments as well as suppliers.

3.21 At TC, we noted a number of instances of confusing information contained in the solicitation document. For example, references to the basis of payment varied from "firm hourly rates" to "ceiling per diem" rates. Significant deficiencies in the methodology also prompted TC to replace all of the evaluation criteria.

3.22 In addition, we noted that RFSAs included more than 50 pages of relevant instructions, terms and conditions, including references to further clauses and general conditions that were published in on-line publications accessible to suppliers. Suppliers' comments received as part of our review raised concerns about the appropriateness of the level of effort to qualify for an SA. The majority of the suppliers who responded to our questionnaire recommended that the government standardize methods of procuring GDS. Suppliers further advised us that there is a significant level of effort and cost associated with preparing a bid in response to an RFSA. Estimated costs mentioned ranged from $2,000 to $5,000 to respond to an RFSA.

3.23 PWGSC has an initiative to develop a new method of supply for graphic, Web, and exhibit and display design services that would be available for use by all government departments and agencies. The objective of this initiative is to develop a standard method that would incorporate the results of a market analysis and the views of key stakeholders. This initiative is being led by PWGSC in cooperation with eight federal departments, the Office of Small and Medium Enterprises (OSME), the Office of Greening Government Operations and a former president of RGD Ontario. We believe PWGSC's decision to pursue a strategic commodity management approach is well considered and demonstrates leadership.

3.24 The OPO found that the requirements included in the RFSAs for GDS did ensure that the suppliers who received supply arrangements had the capacity to perform a range of graphic design services in accordance with established quality standards. The government supported quality and best value by ensuring that only suppliers with an established track record and capacity to perform the needed services were pre-qualified for the SA. For bid evaluation, the departments under review considered the suppliers' years of experience in the graphic design industry, the depth of experience, and the qualifications of its personnel and its capacity to carry out various types of graphic design work. In addition, samples of previous work were examined. These factors were individually rated and assigned points. Bidders were required to achieve an overall technical score of 75 to 85 percent. Qualified suppliers were also obligated to meet further quality requirements such as Government of Canada printing, editing and writing standards and green procurement requirements that were built into the SOWs for the SAs.

3.25 In the subsequent solicitations under the supply arrangements, the second stage of bidding, pre-qualified suppliers submitted bids for clearly defined requirements. At that stage, the financial considerations were much more important, and cost was usually the determining factor for selecting pre-qualified bidders. The quality and completeness of work was reviewed against the stated requirements. Given that most of the contracts were under $25,000, we concluded that this is a reasonable approach to obtaining best value.

3.26 Our review found that GDS were performed in accordance with the terms of the contracts, with clearly defined requirements and detailed lists of deliverables. Interviews with departmental personnel confirmed that the overall quality of the work was satisfactory. In the files reviewed, the findings indicated that operational requirements as defined were met.

3.27 Our review of the procurement of GDS confirms that the methodology used for awarding contracts is indeed heavily weighted toward quality over price. We are also supportive of the PWGSC initiative to strengthen the processes even further.

3.28 The three supply arrangements for GDS were based on adequate strategies and plans to support the development of effective and efficient bid solicitation, evaluation and selection methods. Overall, these plans and practices support operational needs, reflect best value considerations, and encourage competition and quality.

Recommendations

3.29 The OPO encourages PWGSC to include the following key areas in its development of a new method of supply for graphic, Web, and exhibit and display design services:

• Developing a streamlined standard method of supply, to the extent feasible, to purchase GDS within the government;
• Ensuring clearly defined bid evaluation and supplier selection methods; and
• Ensuring projected business volumes are supported by a proper analysis of departmental needs.

3.30 The OPO also recommends that there be adequate documentation in procurement files in support of decisions made. The level of documentation should be commensurate with the risks of the particular procurement. Ensuring that documentation is complete saves time and effort when procurement personnel respond to questions and inquiries about specific procurements. Complete documentation is a requirement of TB and PWGSC policies, and documentation is essential to prevent speculation about whether procurement is fair, open and transparent.

All departments involved in this review were given an opportunity to review this report, and their comments were taken into consideration when it was finalized.

Introduction

Context

3.31 The federal government purchases approximately $14 billion of goods and services annually. Deputy Ministers are accountable for ensuring value for money and maintaining the fairness, openness and transparency of the procurement process. They must also ensure that procurement decisions are aligned with a broad range of legislative, regulatory and policy requirements and support federal objectives.

3.32 A clearly defined procurement plan provides the foundation for the effective management of decisions that take place throughout the life of the procurement. It also provides the baseline against which results can be measured and improvement plans developed. Proper procurement plans are particularly important when setting up supply arrangements (SAs) that may cover several years and are used by departments to award multiple contracts to pre-qualified suppliers.

3.33 Normally, the procurement plan should include the following:

• Key objectives; and
• Proposed methods for meeting key objectives, including best value considerations.

3.34 The rationale for the proposed methods should include a discussion of known procurement risks, including those related to bid evaluation and selection methods, and balancing supplier access with administrative efficiency.

3.35 The Financial Administration Act confirms Canada's commitment to "taking appropriate measures to promote fairness, openness and transparency in the bidding process for contracts with Her Majesty for the performance of work, the supply of goods or the rendering of services."

3.36 The Treasury Board (TB) Contracting Policy establishes the framework within which contracting authorities must be exercised. The Policy states: "The objective of government procurement contracting is to acquire goods and services and to carry out construction in a manner that enhances access, competition and fairness and results in best value or, if appropriate, the optimal balance of overall benefits to the Crown and the Canadian people."

3.37 The Policy defines "best value" as "the combination of price, technical merit, and quality, as determined by the contracting authority prior to the bid solicitation and set out in the bid solicitation evaluation criteria, and which forms the basis of evaluation and negotiation between buyers and sellers to arrive at an acceptable basis for a purchase and sale." It further states that factors such as all relevant costs over the useful life of the acquisition should be considered.

3.38 The Policy further states that government contracting shall

• stand the test of public scrutiny in matters of prudence and probity;
• facilitate access, encourage competition and reflect fairness;
• ensure the pre-eminence of operational requirements;
• support national objectives; and
• comply with the trade agreements.

Why it Matters

3.39 Timely and cost-effective acquisition of the right goods and services directly affects the ability of federal departments to deliver services to Canadians.

3.40 Public Works and Government Services Canada (PWGSC) and many other government departments require the establishment of a procurement plan that addresses operational needs and proposed methods of solicitation, bid evaluation and supplier selection. Most departments also require the identification of key risks and mitigation strategies relating to the procurement.

3.41 Some government suppliers have expressed concerns about the complexity and ambiguity of bid evaluation and selection methods, which led them to question the fairness, openness and transparency of the government's procurement practices.

3.42 For example, the Association of Registered Graphic Designers of Ontario (RGD Ontario) has approached the Office of the Procurement Ombudsman (OPO) with concerns about competitive bid evaluation and selection methods that raise questions about the following:

• Whether bid evaluation and selection methods take into account an appropriate balance between cost and quality or other value-adding factors;
• Whether the bid solicitation methods are aligned with industry capacity and conditions;
• Whether the contracts awarded meet the operational needs of purchasing organizations; and
• Whether the bidding requirements are fair.

3.43 The government's 2008 Speech from the Throne emphasized that ways needed to be found to make it easier for businesses to provide products and services to the government and deliver better results for Canadians.

Focus of the Review

3.44 Government departments have established a number of supply arrangements (SAs) to procure graphic design services (GDS). In this review, we focussed on three SAs, two of which were set up by PWGSC and one by Transport Canada (TC).

Review Objectives

3.45 To determine whether the three SAs for graphic design services were based on adequate strategies and plans to support the development of effective and efficient bid solicitation, evaluation and selection methods and to determine whether these plans and their practical implementation

• support operational needs;
• reflect best value considerations; and
• encourage competition and quality.

Scope and Period of Review

3.46 This review focussed on the policies and practices relating to competitive procurement planning and corresponding solicitation, bid evaluation and selection methods relevant to the purchase of graphic design services (GDS). We selected a sample of three supply arrangements and 39 related files for review.

3.47 The sample was drawn from the contracting data in three of the departments identified as significant purchasers of GDS: Human Resources and Skills Development Canada (HRSDC), Natural Resources Canada (NRCan) and Transport Canada (TC). PWGSC was the common service procurement provider for HRSDC and NRCan.

3.48 The review did not assess the actual conduct of the bid evaluation and selection methods, nor did it directly assess the requirement definition, contract administration or post-contractual activities.

3.49 The review covered contracting activity undertaken in the period from June 2006 to January 2010.

Review Methodology

3.50 We reviewed departmental processes and controls to assess their adequacy in supporting the development of effective and efficient procurement strategies.

3.51 In our review, we considered concerns raised by RGD Ontario to the Office of the Procurement Ombudsman.

3.52 We solicited relevant input from nine suppliers that were providing GDS to at least two of the departments reviewed. We invited these suppliers to provide their views on the procurement methods used by the government, including the key quality factors that the government should consider when evaluating potential suppliers' proposals.

3.53 The judgmental sample of relevant procurement files was selected based on a review of departmental contracting activity reports covering three years between June 2006 and January 2010.

3.54 We sent questionnaires and conducted interviews with each department to ensure a clear understanding of the scope and nature of the department's GDS purchases and the information we had reviewed. These interviews covered procurement as well as program personnel and included questions regarding the effectiveness of the purchasing methods used.

3.55 We assessed practices against the following four criteria and related sub-criteria that support the development of an effective and efficient procurement strategy:

• Criterion 1: Departmental procurement policies and processes support the development of effective and efficient procurement plans and corresponding solicitation, bid evaluation and selection methods.
• Criterion 2: Procurement plans and practices ensure operational needs are supported and clearly defined.
• Criterion 3: Departmental procurement plans and practices reflect key best value considerations.
• Criterion 4: Departmental procurement plans and practices integrate the principles of fairness, openness and transparency, including encouraging competition and allowing innovation, while respecting the requirements of legislation, trade agreements, regulations and procurement policies.

Review Observations

Departmental Policy Frameworks Support Compliance with the Regulations and Meeting Operational Needs

3.56 For each of the departments we reviewed, we looked for policies, procedures and guidance that

• clearly defined roles and responsibilities of both the contract and program authorities;
• explained key factors for the achievement of best value and how they should be incorporated in procurement plans and practices, including bid evaluation and selection methods to achieve best value;
• explained the review and approval mechanisms or functions that had been established to manage relevant procurement risks; and
• supported the principles of fairness, openness and transparency.

3.57 We noted that all four departments had defined roles and responsibilities in terms of specific procurement activities. Departmental policies and procedures reflected the requirement to achieve best value as well as the principles of fairness, openness and transparency. The primary focus of their guidance was on ensuring the adequacy of statements of work and compliance with trade agreements and the Treasury Board (TB) policies. Competitive bidding was encouraged in all departments. Each department had established checklists to ensure that these key considerations were addressed.

3.58 At PWGSC, the checklist indicated that the client was to provide written confirmation that the statement of work (SOW) represented their requirements. For sensitive or high-risk procurements, the contracting officer was also required to obtain written confirmation that the client concurred with the evaluation and selection methodology, prior to its publication.

3.59 We believe that for procurements where multiple organizations are involved, the practice of confirming the client's agreement with the SOW and the bid evaluation and selection methodology is an effective means of minimizing risks of acquiring goods and services that do not meet operational needs. The departmental policies also promote administrative efficiencies through the use of standing offers, supply arrangements and acquisition cards.

3.60 In terms of the practical application of the principles of fairness, openness and transparency, PWGSC and HRSDC reminded personnel that competitive solicitation of bids for requirements valued below $25,000 should be pursued whenever it was cost-effective to do so. At TC, contracting procedures included a list of relevant considerations to assist contracting specialists in vetting the bid evaluation and selection methods to ensure that they were adequate to encourage competitive bidding.

3.61 We noted, however, that in some instances, departmental policies needed to be updated to remain in sync with changes to trade agreements and TB policies. For example, at HRSDC, the policy on intellectual property and the trade agreement thresholds were out of date. PWGSC's Guideline for Bid Evaluation Process - Contractor Selection Methods contains important information but has not been updated in a number of years.

3.62 At a high level, the frameworks we reviewed allow for the development of effective and efficient procurement plans. The departments also developed some useful tools for implementation. We encourage the departments to share good practices related to bid evaluation and selection methods and to keep their frameworks up to date with both TB policies and trade agreements.

Departmental Plans Identify SAs to Meet Their Needs for Graphic Design

3.63 During the course of our review, we noted that in the three departments selected (HRSDC, NRCan and TC), graphic design requirements were being met by the use of three distinct supply arrangements. Two of these, at HRSDC and NRCan, had been established by PWGSC, and the third by TC itself.

3.64 An SA is a non-binding arrangement between the government and a pre-qualified supplier that allows departments and agencies to award contracts and solicit bids from a pool of pre-qualified suppliers for specific requirements within the scope of the SA. Departments meet their specific needs by issuing another call for bids – a subsequent, second-stage solicitation – to one, some or all of the suppliers on the SA list, depending on the details in the SA.

3.65 The nature of the services covered under each of the SAs was very similar, with the exceptions that HRSDC had included related printing services valued up to $10,000 per contract and TC had limited its SA to electronically delivered GDS products.

3.66 Available contract data from departments revealed that the total volume of expenditures to date relating to the three supply arrangements averaged about $500,000 per year. We also noted that the vast majority of the contracts issued under these SAs were under $25,000 and thus, in accordance with the Government Contracts Regulations, could have been awarded without any competition.

3.67 For each of the SAs issued by PWGSC and TC, we expected to find detailed procurement plans that clearly described the proposed solicitation, bid evaluation and selection methods and SA implementation plans that were to be followed. We also expected to find that the decisions reflected in the plans were supported by documentary evidence of proper analysis in support of the decisions made, including those relating to the evaluation methodology and the means to achieve best value. We also expected to find evidence of how risks related to the bid evaluation and selection methods were being managed.

3.68 We observed that two of the three SAs were supported by procurement plans. PWGSC had prepared the plans for HRSDC and NRCan. We note that TC, which had issued its own SA, did not have a formal supporting procurement plan. We were informed that in that department, some planning activity was carried out by program officers and that it influenced the decisions reflected in the SA. We also noted that in all three departments, the SAs were a continuation of previous practices that influenced the decisions regarding the evaluation and selection methods.

3.69 We noted that these two procurement plans for the SAs contained high level descriptions of

• required services, which is the procurement objective;
• estimated business volume;
• solicitation, evaluation and selection methods; and
• relevant trade agreements and policies.

They also contained milestones for establishing the SA.

3.70 We further examined the Request for Supply Arrangement (RFSA) and contract files to understand the rationale for the bid evaluation and selection methods used for the three RFSAs and the subsequent bidding processes conducted at stage 2. In all four departments, we found the documentation to be weak or missing in one or more of the following areas:

Planning for the RFSAs

• Identification of risks and corresponding mitigation strategies;
• Analysis supporting the rationale for using an SA over other procurement options;
• Analysis to support estimates of business volumes for the SAs;
• Analysis supporting the rationale for the evaluation and selection methods to qualify suppliers under the SA and for subsequent calls for bids. This would include
  • the rationale for the number of suppliers qualified and also the rationale for how additional suppliers are qualified each year;
  • analysis supporting TC's decision to allow about 90% of the points for technical merit and 10% for cost in its evaluation methodology; and
  • analysis supporting PWGSC's methodology which essentially eliminates the highest- and lowest-cost proposals.
• A mechanism to monitor the implementation of the plan, with respect to achieving the plan's objectives, as well as the balance between the objectives of fairness, openness and transparency and operational efficiency.

For subsequent bidding processes (stage 2)

• Supporting rationale explaining why the particular method of supplier selection was used and whether it conforms to the provisions of the supply arrangement.

Recommendation 1

3.71 We continue to be concerned about the lack of adequate documentation in procurement files in support of decisions made. The level of documentation should be commensurate with the risks for the particular procurement. The lack of adequate documentation not only violates TB and PWGSC policy requirements, but also gives rise to speculation about fairness, openness and transparency of the procurement process. It also adds unnecessary work for departmental personnel when responding to questions and queries relating to specific procurements.

The Complexity and Diversity of the Bid Evaluation and Selection Methodologies Being used Caused Confusion for Suppliers

3.72 As noted earlier, many of the requirements of the SAs for GDS were of a similar nature, and yet the detailed methods of evaluation and selection varied. In some cases, this resulted in confusion and additional work on the part of suppliers. For example, each of the three user departments had different thresholds and requirements for competitive bidding at stage 2. At HRSDC, competitive bidding was required for individual requirements valued in excess of $15,000. At NRCan, all purchases were to be competed. At TC, the competitive bidding threshold was set at greater than $25,000. This variety of practices for the purchase of very similar requirements leads to confusion and also raises questions about the supporting rationale for the different thresholds applied.

3.73 Our review of RFSA procurement files confirmed that the bid evaluation and selection methodologies were complex and confusing. For example, for the NRCan RFSA, we found the lack of clarity in the description of mandatory and rated requirements resulted in 35 questions from suppliers, creating an unnecessary level of effort on the part of departments as well as suppliers.

3.74 At TC, we noted a number of instances of confusing information contained in the solicitation document. For example, references to the basis of payment varied from "firm hourly rates" to "ceiling per diem" rates. Significant deficiencies in the methodology also prompted TC to replace all of the evaluation criteria.

3.75 In addition, we noted that RFSAs included more than 50 pages of relevant instructions, terms and conditions, including references to further clauses and general conditions that were published in on-line publications accessible to suppliers. Suppliers' comments received as part of our review raised concerns about the appropriateness of the level of effort to qualify for an SA. The majority of the suppliers who responded to our questionnaire recommended that the government standardize methods of procuring GDS. Suppliers further advised us that there is a significant level of effort and cost associated with preparing a bid in response to an RFSA (first stage to qualify for an SA). Estimated costs mentioned ranged from $2,000 to $5,000 to respond to an RFSA.

3.76 PWGSC has an initiative to develop a new method of supply for graphic, Web, and exhibit and display design services that would be available for use by all government departments and agencies. The objective of this initiative is to develop a standard method that would incorporate the results of a market analysis and the views of key stakeholders. This initiative is being led by PWGSC in cooperation with eight federal departments, the Office of Small and Medium Enterprises (OSME), the Office of Greening Government Operations and a former president of RGD Ontario.

3.77 In June 2009, PWGSC posted a request for industry comments (RFIC) on MERX™ seeking comments on its proposed methods of supply. Resulting comments from ten graphic design firms and three graphic design associations were taken into consideration in the development of a solicitation document (RFSA/RFSO). At the time of the completion of this review, the new RFSA/RFSO was being reviewed by PWGSC before being submitted for publication on MERX™. We believe PWGSC's decision to pursue a strategic commodity management approach is well considered and demonstrates leadership.

Recommendation 2

3.78 We encourage PWGSC to include the following key areas in the initiative:

• Developing a streamlined standard method of supply, to the extent feasible, to purchase GDS within the government;
• Ensuring clearly defined bid evaluation and supplier selection methods; and
• Ensuring projected business volumes are supported by a proper analysis of departmental needs.

Procurement Plans and Processes Emphasize Quality and Best Value

3.79 Representatives of RGD Ontario have raised concerns to the OPO that the government does not adequately consider quality factors when contracting for GDS. They stated that, typically, lowest-cost proposals are chosen and as a result the quality of work suffers.

3.80 Our review of the competitive RFSA processes done by or on behalf of the three departments under review found that the following factors were considered key for quality and were included for bid evaluation purposes:

• Years of experience the firm has in the graphic design industry;
• Depth of experience and qualifications of the firms' personnel;
• Capacities to carry out various categories of graphic design work;
• Quality of examples of previous work carried out by the firm; and
• Ability to carry out the work in both official languages.

3.81 We found that these factors were individually rated and assigned points. The bidders' proposals then received an overall "technical" score by adding up all of the points awarded for the rated criteria. We noted that the minimum pass mark for the three RFSAs ranged from 75 to 85 percent. As a result, the quality standards that firms had to satisfy to qualify for each of the SAs were set at a high level.

3.82 There were also quality requirements built into the SOWs for each of the resulting SAs. For instance:

• Green procurement – use of recycled paper and vegetable-based ink;
• Government of Canada (GC) printing, editing and writing standards – common look and feel, Federal Identity Program (FIP); and
• GC and departmental standards for GDS.

3.83 Firms that qualified under an SA were obligated to meet these standards when performing a contract issued under the SA.

3.84 The financial proposals, on the other hand, are considered only for those that obtain the minimum technical score. The cost evaluation had minimal influence on the final results in qualifying for an SA. In almost every case, the reason that a particular supplier did not qualify for a supply arrangement was weakness in the technical qualifications rather than a price factor.

3.85 In our review of subsequent purchases conducted under the terms of the SA (second stage), we found that the financial considerations were much more important. Pre-qualified suppliers were invited to bid on GDS requirements based on detailed descriptions of the deliverables required (e.g. Web page design, annual report, CD cover). Bids were solicited from one or more bidders depending on the value of the requirement and the bidding requirements prescribed in that particular department's SA (i.e. number of firms to be invited depending on the dollar value). Where competitive bids were solicited, the contract was awarded to the lowest compliant bidder.

3.86 In our review, we looked for evidence that operational requirements (as defined in the statement of work (SOW) or terms of reference (TOR)) were supported by the procurement practices.

3.87 Contracts for GDS reflected clearly defined requirements with detailed lists of deliverables. File examinations showed no significant evidence of contract amendments that were attributable to poor contractor performance, and there were no incidents of payment disputes that might suggest non-performance on the part of the contractors. Our interviews with departmental personnel confirmed that the overall quality of the work was satisfactory. Overall, in the files we reviewed, we found that the procurement practices ensured that operational requirements as defined were met.

3.88 In summary, we found that the requirements contained in the RFSAs ensured that the suppliers that received supply arrangements had the capacity to perform a range of GDS in accordance with the established quality standards. The subsequent solicitations under the supply arrangements allowed suppliers to submit bids for specific requirements. The government ensures quality and best value, firstly, by ensuring that only suppliers with an established track record and capacity are pre-qualified for potential business opportunities. Secondly, at stage 2, departments meet their specific needs by obtaining bid(s) from one or more of the pre-qualified suppliers. Bids are then assessed against detailed statements of requirements for compliance, and cost is usually the determining factor for compliant bidders.

3.89 The quality and completeness of the work is subject to proof of concept and/or final inspections to ensure it met the stated requirements. In our view, given that most of these contracts are under $25,000, this appears to be a reasonable approach.

Conclusion

3.90 In its June 2009 report In Pursuit of Balance – Assisting Small and Medium Enterprises in Accessing Federal Government Procurement, the Government Standing Committee on Operations and Estimates recommended that the government ensure that innovation and quality are key determinants in the evaluation of bids and the awarding of contracts. The government responded that in its evaluation of bids and awarding of contracts, PWGSC is committed to ensuring that innovation and quality are considered, while ensuring maximum value for Canadians. It further stated that the government will explore how federal procurement can be less prescriptive and support innovation as part of its ongoing work, and will continue to use an evaluation methodology that heavily emphasizes quality over price, while ensuring compliance with relevant procurement policies and trade obligations.

3.91 Our review of the procurement of GDS confirms that the methodology used for awarding contracts is indeed heavily weighted toward quality over price. We are also supportive of the PWGSC initiative to strengthen the processes even further.

3.92 The three supply arrangements for GDS were based on adequate strategies and plans to support the development of effective and efficient procurement bid solicitation, evaluation and selection methods

3.93 Overall, these plans and practices support operational needs, reflect best value considerations, and encourage competition and quality.

All departments involved in this review were given an opportunity to review this report, and their comments were taken into consideration when it was finalized.

Date modified:

2018-11-01