Research on goods versus services contracts

Introduction

Public procurement is undertaken in a complex environment where political interests (e.g. regionalism, electioneering) intersect with spending and the market economy. As Worthington (2013) explains, "[…] the public procurement of goods, services and construction are highly regulated processes" (p. 508). Furthermore, public procurement "involves a wider set of choices about what to buy, from whom and how" (Hartley 1991, p. 45). The vast sets of rules and regulations in place to govern procurement are meant to sanitize the process and remove as many potential conflicts of interest as possible.

Public procurement is inherently different from private procurement with repercussions not for shareholders, but for taxpayers, citizens as well as the political livelihood of a government. Paul Emanuelli conveys the interplay of public procurement, politics and public trust well:

As the Pacific Scandal^{Footnote 1} illustrates, government procurement controversies can bring down the best of them. Even the Fathers of Confederation were not immune. The paramount importance of good government is firmly entrenched in our political fabric. Reactions to spending improprieties run as deep in our political psyche as the memories run long. Public officials who run on the wrong side of these rules do so at their peril. Today, computer and advertising contracts may have replaced railway construction deals as the subject of controversy, but accountability for proper government spending remains a matter of national importance (Emanuelli 2005, p. 1).

Given its political repercussions, it was little surprise that procurement was targeted when political scandal surrounding accountability surfaced in the mid-2000s in Canada. Part of the reaction was to enact legislation for a Procurement Ombudsman to receive procurement contract complaints and to review the procurement processes of federal organizations. Pursuant to the Federal Accountability Act that received Royal Assent in December 2006, the Procurement Ombudsman's mandate and enabling legislation were made by amendments to the Department of Public Works and Government Services Act (2006) (section 23.1).^{Footnote 2} Establishing the Procurement Ombudsman was one of "several elements of the government's comprehensive plan to strengthen the fairness, openness and transparency of federal procurement" (OPO Annual Report 2008-09, p. 10). While the Act establishes the role and responsibilities of the Procurement Ombudsman, the Procurement Ombudsman Regulations that came into effect in May 2008 operationalize the mandate.^{Footnote 3}

The Procurement Ombudsman's mandate has four basic tenets^{Footnote 4} (as defined in the Department of Public Works and Government Services Act (2006))^{Footnote 5}:

1. Ensure government departments procure goods and services in a fair, open and transparent manner, through a review of their practices, making recommendations for improvement when appropriate.
2. Review complaints for the contract award for goods below $25,000 and services below $100,000.
3. Regardless of dollar value, review complaints regarding the administration of a contract.
4. Ensure alternative dispute resolution services are available (both parties to the contract must agree to participate).

A plain read of the Procurement Ombudsman's mandate defined in the Department of Public Works and Government Services Act and the operationalization of their mandate through the Procurement Ombudsman's Regulations, appears to suggest that the Procurement Ombudsman can receive complaints once a contract has been awarded for matters related to the evaluation of bids, the contract award, as well as contract administration. There may be a risk of bias in the nature of complaints filed with the Procurement Ombudsman in relation to their mandate. However, given the perceived scope of the Procurement Ombudsman's mandate, there appear to be linkages between the legislation and regulations and the pre-contractual, contracting, and contract administration phases of the four phases of the contracting process (see Table 1). Only phase four, the post-contract phase, may be contestably part of the Procurement Ombudsman's mandate. For a discussion of how the Procurement Ombudsman operationalizes its mandate through the filing and evaluation of complaints, please see Annex 1.

The Government of Canada spends roughly 15-20 billion dollars in the procurement of goods and services every year through hundreds of thousands of contracts (Treasury Board of Canada Secretariat (TBS) 2015). Procurement practices at an aggregate level matter for politics and public finance because of the amount of money spent and the ways in which it is spent.

When contracts are assessed based on monetary value, service contracts cost consistently more than goods contracts since 1996 (Data tables 1 and Data table 2). In terms of volume, the number of goods contracts since 1996 is greater than the number of service contracts, with exception to 1997 and 1998 (see Data table 3 and Data table 4). Most contracts (90%) are worth less than $25,000 (i.e. low value) and represent less than 10% of the total value of contracts. While contracts worth $25,000 and above represent the largest portion of spending, they are far fewer in number (OPO Presentation 2013). Prior to the establishment of the OPO in 2008, suppliers for low-value contracts had no formal complaint process (vendors with higher value contracts have recourse with the Canadian International Trade Tribunal).

Table 1: Contract phases and descriptions
(Office of the Procurement Ombudsman Annual Report, 2008-09, p. 32)

Phase	Phase description (OPO Annual Report 2008-09, p. 32)	Description of complaints
1. Pre-contractual	Activities related to requirement definition and procurement planning.	Requirement understated. Statement of Work unclear or biased. Unfair or onerous requirements, e.g. security clearance level, liability insurance.
2. Contracting	All activities from bid solicitation to contract award.	Request for proposal (RFP) development, e.g. onerous terms and conditions. How the bidders are evaluated and whether it's fair. Evaluation criteria: mandatory or rated; complaints that they're too restrictive, e.g. 12 years at OPO instead of 12 years in Public Service (PS) generally. Mandatory criteria that is pass/fail (they either have it or not) is treated as rated where partial points may be awarded, allowing some vendors to move to the next level even if they should not have been allowed. Procurement strategy, e.g. contracting agencies not using the procurement tool accurately; perception of contract splitting offer (SO) for long-term/recurring procurement). Complaint that the contracting agency did not evaluate the bid the way they said they would, e.g. inaccurate application of the point system by the contracting agency (i.e. they give partial or half points), problematic for vendors who did not receive points. Wrong contractor selected. Note: Many evaluation issues are related to planning, i.e. accurately defining (and applying) the terms of assessment clearly and regularly from the outset.
3. Contract administration	Activities such as issuing contract amendments, monitoring progress, following up on delivery, payment action.	Contract amendments, e.g. change in scope, insufficient funding. Payment delays.
4. Post contract	Final action (e.g. client satisfaction, contractor agreement to final claim, final contract amendment, completion of financial audits, proof of delivery, return of performance bonds) and close-out (e.g. completeness and accuracy of file documentation and adherence to file presentation standards).	Performance unsatisfactory. Key documents not on file.

Data table 1: Contract values by type, 1996-2004

Year	Total value of yearly contracts	Total value of construction contracts (approx.)	Total value of services contracts (approx.)	Total value of goods contracts (approx.)
1996	8,703,172	1 M	3.5 M	4.5 M
1997	9,093,408	0.8 M	0	4.2 M
1998	13,732,457	0.6 M	0	5.1 M
1999	9,858,154	0.6 M	4.9 M	4.5 M
2000	9,360,172	0.7 M	5.1 M	3.6 M
2001	13,146,679	0.9 M	6.5 M	6.2 M
2002	12,822,851	1 M	7.7 M	4.4 M
2003	13,393,801	1 M	7.9 M	5.2 M
2004	19,063,453	1 M	11.9 M	6 M

Data table 2: Values of goods, services and construction contracts (yearly, 2005-13)

Year	Total value of contracts (Yearly)	Total value of construction contracts (approx.)	Total value of services contracts (approx.)	Total value of goods contracts (approx.)
2005	15,944,061	0.9 M	10.2 M	4.9 M
2006	12,153,012	0.9 M	7 M	4.1 M
2007	14,257,457	0.8 M	6.2 M	7.3 M
2008	16,439,171	0.9 M	6.8 M	8.7 M
2009	20,351,495	1.3 M	8.8 M	10.1 M
2010	16,203,482	1.3 M	9.2 M	5.4 M
2011	16,033,758	1.9 M	7.0 M	7.2 M
2012	15,149,203	1.3 M	7.1 M	7.1 M
2013	14,552,157	1.3 M	7.6 M	6.1 M

Data table 3: Percentage of goods, services and construction contracts based on total number of contracts (1996-2004)

Source: Treasury Board of Canada Secretariat, 2015

Year	% of Total number of contracts (Goods)	% of Total number of contracts (Services)	% of Total number of contracts (Construction)
1996	52	38	9.5
1997	47.5	48	9
1998	38	59	5
1999	71.5	38	1.4
2000	62	35	2
2001	61.5	35	2
2002	63.5	34	2
2003	63.5	33.5	1.4
2004	58	36	9

Data table 4: Percentage of goods, services and construction contracts based on total number of contracts (2005-13)

Source: Treasury Board of Canada Secretariat, 2015

Year	% of Total number of contracts (Goods)	% of Total number of contracts (Services)	% of Total number of contracts (Construction)
2005	55	39	6
2006	55	40	5
2007	57	40	2
2008	54.5	40.5	5
2009	52	45	3
2010	51	44	5
2011	58.5	38.5	3
2012	56	38.5	5.5
2013	55	39	6

Since the establishment of the office in 2008, the OPO observed inordinate numbers of service contract related complaints. While it may be suggested that the number of service contract complaints would naturally increase as more services were procured, the argument does not appear to hold based on Treasury Board of Canada Secretariat (TBS) procurement data. Despite increases in spending on service procurement over the last 10-15 years, the volume of goods procurement contracts remains higher than the volume of service contracts. Since there are a greater number of goods contracts, one could presume that a higher number of complaints should ensue, but this is not the case.

One could also suggest that the subjective nature of service makes it more difficult to assess or quantify its outcomes, thereby incurring higher than average numbers of complaints. However, analysis of OPO data challenged this reasoning since the majority of complaints filed were related to the contracting phase of the procurement process (i.e. the period in which the request for proposals is developed, bidders' submissions are evaluated) and the contract awarded, and not the contract or post-contract phases where the contracts are amended and performance are evaluated, respectively. At the inception of this research project, the subjectivity of service was hypothesized to have led to increased instances of complaints. However, definitions from half of the participating U.S. jurisdictions provide some indication that service is not so subjective that it cannot be defined.

If the volume of contracts cannot explain high numbers of complaints and if complaints are not related to the evaluation of service contract outcomes, then how can the inordinate number of service contract complaints be explained? What variables may be contributing to this result? Is the public procurement of services inherently different than that of goods? Is the OPO the only jurisdiction experiencing this tendency or is service procurement a challenge across jurisdictions?

The researchers were engaged by the OPO to attempt to answer these questions. To undertake this inquiry, a series of research questions was developed and agreed upon by the researchers and the OPO. Two working hypotheses were proposed and refined based on consultations with international public procurement experts. The consulted experts also verified the research design and methodology. This report's fundamental observation is that a structural incompatibility exists between the actual procurement system (designed for goods) and the increasing public procurement of services (still outnumbered by goods). In our consultations, there was consensus among experts around the term 'incompatible' to explain the current goods procurement system and its mismatch with service procurement.

From the experts' perspective, the fundamental incompatibility is one of human capacity (i.e. procurement officers do not have the education/training to execute service procurement in a complex environment). Building on this perspective, this report suggests that there are various sources of incompatibility including: strategy, people, processes and infrastructure that must be considered to evaluate the existing model for service procurement. While this finding is not definitive, it is a helpful advancement in the largely understudied area of public procurement, notably through the lens of contract complaints.

This report proceeds by establishing the project's research design, followed by its methodology and discussion and analysis of the findings and the comparative data. To conclude, the report suggests future research directions to continue to investigate this subject.

Research design

Context

Canada's modern procurement system was developed during wartime, as many of the historic foundations of government procurement tend to lie at critical moments, e.g. war, a major national undertaking such as space exploration, etc. (Keeney 2007). During World War I (WWI), military procurement in Canada was ad-hoc with the army, air force and navy purchasing independently. To correct deficiencies and limit profiteering by suppliers, the federal government began to coordinate procurement efforts (Plamondon 2010, p. 3). The historic procurement processes established in the mid-20th century to acquire materials tended to emphasize goods (for a short overview of key moments in Canada's procurement history, see).

The path dependent tendencies of any policy are inclined to lock-in and perpetuate over time (Pierson 2004; Peters 2012; Hall and Taylor 1996). With these precedents, change while not impossible can be difficult. Following the concept of path dependency, the procurement system (made up of strategies, people, processes and infrastructure) has a historical precedent designed to acquire goods that persists, that may impact public procurement today.

Key terms and definitions

For the purposes of this report, goods are defined as articles, commodities, equipment, materials or supplies (following the Government of Canada's Supply Manual Glossary). Service is defined as what is not a good. Only the distinction between goods and service contracts are of interest in this report. As such, all OPO data related to procurement contracts was coded as related to either goods, services, good and service (where the file included both aspects of goods and service) or not applicable (where the file was undefined or not relevant to the study). Construction was not considered an independent category and was coded based on the most closely related of the four categories.^{Footnote 6}

Neither the Treasury Board Contracting Policy (Chapter 16)^{Footnote 7} nor the Department of Public Works and Government Services' Supply Manual Glossary provide an explicit definition of what constitutes a service, yet definitions for goods and construction exist. In spite of not having a definition for service, the TBS distinguishes between goods, service and construction procurement in its procurement reporting data.^{Footnote 8}

Among public procurement officials, there is no single agreed upon definition of what constitutes a service. The National Association of State Procurement Officials (NASPO), an umbrella organization that represents the state chief procurement officers in the U.S., distinguishes between goods and service procurement. Definitions of service range from what is not a good to clear lists of what constitutes a service, as is the case in various U.S. states (see Table 2).

Goods procurement tends to be input focused where the request for proposals seeks to acquire an article, commodity, equipment, materials or supplies. By contrast, when procuring a service (which, for the purpose of this report is anything but a good), one is more likely than not seeking a solution to a problem or a particular outcome. When procuring a service, "[…] the solicitation should include a program narrative, answering the question, […]" What problem is the agency seeking to solve?"" (NASPO 2015 p. 163). As NASPO's handbook for public procurers explains,

In the purchase of services, the specifications or scopes of work ought to emphasize achieving a specific result, and not the rigid details of the process for getting to it. The goal of a competition is to ensure that the service provider's bid or proposal responds with a solution to a defined need (NASPO 2015, p. 162-163).

The crucial difference in goods versus service procurement comes down to a distinction in purpose. A good is acquired because an article that takes the form of equipment, materials or supplies is needed. When procuring a good, be it to fight a war, to create workspaces by buying desks and chairs, to purchase fencing for a federally sponsored event in the National Capital Region, etc., the item is acquired because it is needed as a tool or input to fulfill a function. By contrast, the procurement of services tends to solve a problem instead of solely acquiring a physical item. The way that service should be bought is different than buying a good. Rather than going to market to buy a specific item (as in goods procurement), one goes to tender seeking solutions to solve a problem.

Table 2: Jurisdictional definitions of service contracts^{Footnote 9}

Arizona

The furnishing of labor, time or effort by a contractor or subcontractor that does not involve the delivery of a specific end product other than required reports and performance. Does not include employment agreements or collective bargaining agreements.

Government of Canada

No definition exists.

Nebraska

That which directly engages in the time or effort of an independent contractor whose purpose is to perform an identifiable task, study, or report rather than to furnish an end item of supply, goods, equipment, or material (Statute 73-502).

New Jersey

A service contract primarily requires labor, time and/or effort to provide intangible general or professional assistance to accomplish an objective (construction and architecture are overseen by a separate division).

North Dakota

The furnishing of labor, time, or effort by a contractor, not involving the delivery of a specific end product other than reports that are merely incidental to their required performance. The term includes professional services.

Pennsylvania

The furnishing of labor, time or effort by a contractor not involving the delivery of a specific end product other than drawings, specifications, or reports which are merely incidental to the required performance. The term includes the routine operation or maintenance of existing structures, buildings, or real property. The term does not include employment agreements or collective bargaining agreements. The term includes utility services and those services formerly provided by public utilities such as electrical, telephone, water, and sewage service.

Given the inordinate numbers of service contract complaints OPO receives, there are likely explanatory variables that can account for the tendency. The researchers have found however, that the data collected by the OPO over its history is of insufficient depth to validate the observations presented in this report. However, leveraging the limited data and the expertise of a panel of international public procurement experts, the researchers have attempted to draw certain inferences as to the root causes of the growth in service complaints. Yet, in spite of consultations with several dozen U.S. jurisdictions and the Organisation for Economic Co-operation and Development (OECD) on behalf of its member countries, a dataset that would substantiate the professional opinions of the expert panel could not be identified. The observations in this report require further study to determine whether they in fact represent root causes.

Research directions in public procurement

While the practice of public procurement has been undertaken in many forms for over a century (see Kenney 2007), its study remains a relatively niche undertaking. In the literature on public procurement, the distinctions between public and private procurement (as well as their interplay) are discussed (Emanuelli 2005; Worthington 2013), the complexities of public procurement are studied (Hartley 1991; Emanuelli 2005; Worthington 2004 and 2013; Thai 2004; Thai et al. 2005; Thai et al. 2007), the roles of human resources and management capacity are assessed (McCue 2001; Snider 2006), as are new trends, e.g. green procurement (OECD 2003; OECD 2013), e-procurement (McCue and Alexandru 2012; Gardenal 2013), alternative service delivery (NASPO 2015), etc. The differences in the complexity of procuring services versus goods are recognized (California 2013; NASPO 2015), with special emphasis on the procurement of Information technology (IT) services (Sieverding 2008; Ancarani, Di Mauro and Giammanco 2009; California 2013). However, the implications of procuring services in systems that tend to be goods oriented, remains understudied, leaving a gap in the literature on public procurement. As governments around the world procure increasing numbers of services, the ways in which they are acquired merits closer attention.

Complaints can be used as a lens through which to study the existing procurement process. The number and nature of procurement related complaints serve as an entry point for testing whether the goods-oriented procurement process is contributing to inordinate numbers of service contract complaints. Based on data collected for this report, the majority of comparable jurisdictions have also experienced higher levels of service contract related complaints. However, no published research appears to have been undertaken to suggest explanations for this state of affairs. Literature reviews and expert consultations^{Footnote 10} confirmed that using complaints as a lens to evaluate the procurement process is a unique research angle.

This report confirms that there are higher numbers of service contract complaints versus goods contract complaints (despite the greater volume of goods procurement in the federal government). Through research and expert consultations, potential explanations (that require further research to confirm) include incompatibilities between the goods-oriented procurement system and the procurement of services as well as human capacity in the system to undertake service procurement. Without changing the underlying structure and without training staff to think differently about the task of service procurement, the Procurement Ombudsman may inevitably continue to receive inordinate numbers of service contract versus goods contract related complaints.

The content of this report is based on the OPO's internal data, data shared by comparable jurisdictions, expert consultations, Government of Canada data, as well as secondary sources (including scientific articles and expert reports). This report serves not only to fulfill the OPO's research request, but will also contribute to the void in the literature on the incompatibility between the use of a goods-oriented procurement process for the procurement of services.

Research questions

In consultation with the OPO, the following research questions were defined to guide the inquiry:

1. What inherent differences in contracting for services (as opposed to goods) may be contributing to the disproportionate number of complaints about service contracts compared to contracts for goods?
2. What issues, factors and/or reasons may be contributing to the disproportionate number of complaints related to services?
3. What conclusions may be drawn from the research?
4. Are comparable jurisdictions experiencing similar problems?

Hypotheses

Based on initial research, two hypotheses were developed to confirm the higher number of service contract complaints relative to goods contract complaints and to suggest potential explanations. Both hypotheses were confirmed through data collection and verified by expert consultations.^{Footnote 11}

This report's hypotheses are:

H1:

There are an inordinate number of service contract related complaints v. goods contract related complaints.

To validate H1, there should be higher numbers of service contract complaints than goods contract complaints. The data confirmed H1 with 79% of relevant complaints related to service contracts compared to 21% for goods contracts based on OPO's data since 2008.

H2:

Goods procurement is input focused, whereas service procurement is outcome focused. Therefore, there is a structural incompatibility of the existing procurement model (developed to procure goods) for service procurement.

If H2 holds, there should be higher numbers of complaints in the pre-contractual and contracting phases, suggesting that the procurement process may be oriented toward input-based goods instead of outcome oriented service procurement. The data confirmed H2 with 55% of filed complaints since 2008 related to the contracting phase. The researchers did not assess the complaints in the context of the Procurement Ombudsman's mandate but on which contract phase the complaint falls.

Confirmation of the two hypotheses point to possible root causes of difficulties with service procurement, rather than a definitive "theory."  To ensure that the results of this study are not overstated, it should be noted that there is broad agreement among experts that there is currently insufficient data and analysis in the literature to fully explain the endemic problems associated with service procurement—it is a work in progress.

To better establish the linkages made in this report, it may be useful to draw a statistically valid sample of the OPO files for further in-depth assessment. Identifying clearly the reason for the complaint and even interviewing the complainant can help to obtain more explicit conclusions about the variables contributing to higher numbers of service contract complaints. This supplementary focused researched can verify the report's observation of an incompatibility between the procurement process and the services being procured.

Methodology

For this project, a scoping exercise of OPO's complaints files was undertaken to assess the numbers of and differences between goods and service contract complaints. Given the confidential nature of OPO's files, approximately eight members of the office's staff reviewed all procurement complaint files since the office's opening in 2008 (the researchers were not allowed to handle the confidential files). While there may be inconsistencies in the subjective nature of data collection and recording given the involvement of eight staff, standardization was encouraged through the use of a pre-established template (see Annex 2 for the template and Table 1 for coding (i.e. OPO's contract phase descriptions)).

OPO staff reviewed complaint files and recorded relevant data based on the nature of the contract (i.e. good v. service), the type of complaint (i.e. related contract phase), the related department, as well as the actions taken or resolution obtained. Only complaints related to procurement contracts were reviewed and counted in this assessment. All other files (e.g. general inquiry calls to the office) were not counted. Data collected by OPO staff was anonymized with all indicators removed so that it could be made public. The researchers recorded and coded the anonymized data.

In total 668 complaint files were received from the OPO and coded by the researchers. Of the 668 entries, 480 were related to service contracts, 125 related to goods, 7 were classified as part goods contract and part service contract, and 56 were not applicable (N/A) as they were either undefined or irrelevant to the data set (e.g. question to the OPO) (see Table 3 and Data tables 5 and Data table 6).

Data table 5: Complaint types by number 2008-14, n=668

Complaint type	Number
Goods	125
Service	480
Goods and services	7
N/A	56

For the purpose of this final report, as well as for comparability with other jurisdictions, only files clearly defined as goods or service related were analyzed. This reduces the number of applicable files from 668 to 605.

Data table 6: Complaint types by percentage 2008-14, n=605

Complaint type	Percentage (%)
Goods contract	19
Service contract	72
Goods and services	1
N/A	8

This methodology was crucial for two reasons. First, for the researchers to test H1, all complaints and the split between goods and service had to be tabulated. Second, to test H2, it was necessary to link the complaint to a specific contract phase. OPO's pre-defined contract phases were adopted in this report and were used to code the "time" (i.e. related contract phase) of the complaint linking it to a set of steps and practices in the procurement process. See Table 1 for the phases and their descriptions.

When supported by a critical mass of data, identifying the related contract phase can suggest reasons for the higher number of service versus goods contract related complaints. For instance, complaints related to phases #1 and #2 (see Table 1 for definitions), may suggest pre-contract issues related to the formulation of the request for proposals as well as issues related to assessment and interpretation of terms in the evaluation process.

To test research findings, and to determine if similar jurisdictions were experiencing analogous complaint splits, a comparative assessment with data from comparable jurisdictions was undertaken. The 50 states of the United States were targeted for comparison, as many share geographical similarities with Canada and work with similar vendors. While not all states have the budgets of the Canadian federal government or similar populations to that of Canada in its entirety, many states have procurement-specific complaints processes embedded in their state system.

As a professional association, NASPO produces research, organizes education and training and convenes its members to address the challenges and practices of public procurement. In 2014, NASPO undertook a survey of "State Procurement Practices". Of its 50 members, 42 have legislation, rules or regulations related to procurement complaints. The researchers contacted these 42 states requesting data on the number of complaints their office received, the split between goods and service complaints, as well as their definitions (if they exist) for good and service (see Annex 3 for a sample data request chart and Annex 4 for a list of the contacted jurisdictions).

There were 16 participating jurisdictions that provided full or partial data (including a definition of "service") for this report. All of the jurisdictions were assured that the collected data was anonymized to ensure it could not be attributed to any specific jurisdiction, although some officials noted that their data is in fact publicly available.^{Footnote 12} There was significant interest among state procurement officials in the results of this study. To promote collaboration and interjurisdictional comparisons, anonymized data collected for this report will be shared in aggregate form with the participating jurisdictions.

The methodology for this report is rooted in inductive analysis. In conducting their enquiry, the researchers moved from a specific observation that OPO was experiencing inordinate numbers of service contract complaints to a discovery that most of the complaints link to the initial phases of the procurement process. While this report's dataset cannot explain why this pattern exists, inductive reasoning confirms that a pattern does exist.

Data from U.S. jurisdictions confirmed that OPO's experience is not unique. To suggest potential explanations and to establish future research directions, public procurement experts were consulted. The experts came to the conclusion that the current system is incompatible for the procurement of services and is exacerbated by a lack of human capacity in procurement, leading to higher numbers of service contract complaints. These potential explanations merit further research.

Findings

This section will discuss and analyze the report's results through the lenses of the two hypotheses and the results of their testing.

It was crucial to verify that there are consistently higher numbers of service contract complaints than goods contract complaints as the foundational premise of this report. The data confirmed H1 with 79% of relevant complaints related to service contracts compared to 21% for goods contracts based on data since 2008 (see Data table 7).

Data table 7: Goods v. service complaints (based on relevant complaints only)

Type of complaint	Percentage (%)
Service contract	79
Goods contract	21

With service complaints data significantly outnumbering those on goods, a time series of complaints since 2008 was developed to ensure that the discrepancy could not be attributed to anomalies or complaint "spikes" in particular years. As Data table 8 demonstrates, since 2008, service contract complaints have consistently outnumbered goods contract complaints, suggesting a relatively stable ratio (service: goods) over the study period.

The consistently higher numbers of service contract versus goods contract related complaints may suggest that a constant variable or set of variables is contributing to this trend.

The significantly higher numbers of service contract complaints were unsurprising to the panel of experts. For instance, NASPO indicated that service procurement, if not done properly, can be challenging. The organization is in the process of developing an educational platform for public procurement support for entry to senior-level state procurement personnel, which includes access to education and training on topics such as service level contracting methods.

Having established that service contract complaint numbers are much higher than goods contract complaints, the next section focuses on identifying potential variables to better understand the state of affairs.

Underlying H2 are assumptions about the procurement of goods and services. Working from the assumption that the procurement of goods and services should reflect their different goals, the second hypothesis seeks to test whether the way in which goods and services are procured can be a variable in the higher number of service contract complaints because of their different purposes. At an operational level, it is hypothesized that the distinctions manifest themselves as a structural incompatibility between the existing procurement model developed to procure input-oriented goods and the use of the model to procure outcome-oriented services. Can a model built to procure a good be applied to procure (what are often more complex) services?

Data table 8: Number of complaints received (per annum) by complaint type

Year	Number of goods contract complaints	Number of service contract complaints
2008-09	29	63
2009-10	8	26
2010-11	14	53
2011-12	22	85
2012-13	14	80
2013-14	20	78
2014-15	18	95

The majority of the experts consulted pointed to a problem in human capacity when it comes to service procurement (see Box 2). From the experts' perspective, since the procurement of services is undertaken for an outcome and not an input distinguishing it from goods, it requires different processes (e.g. evaluation) and a different organizational culture to manage these files.

As Professor Stephen Gordon of Old Dominion University indicated, public procurement is more than a process; it is part of a complex system of government that must respond to different pressures. Given the complex and dynamic environment of public procurement generally, the higher level of difficulty associated with the procurement of services as compared to the procurement of goods, the capability of individual procurement and contracting officials to execute service procurement as it should be executed is critical to the strategic success of public organizations.

The idea of focusing on individual procurers for change was reiterated by Professor Dermot Cahill, Bangor University Institute for Competition and Procurement Studies as well as NASPO that is providing procurement support and training. There is broad agreement among experts that change has to come from individual procurers who would adjust their analysis for the procurement of services on a case-by-case basis. The experts were clear that this type of change requires training and education programs as well as changes in organizational culture.

On the issue of human capacity, Professor Khi Thai of Florida Atlantic University specifically emphasized the importance of refining evaluation procedures to support individual procurers in assessing outcomes. Clear criteria for services do not always exist, thus procuring outcomes can be challenging. Often times, Dr. Thai indicated, procurement committees will default to price, treating lowest cost (i.e. best value for money) as the best option. This however, can be problematic because it may not fulfill the required outcome. Further investigation is required to address the question of "best value v. lowest cost" in public procurement, since how to benchmark quality against price differences remains to be studied.

Based on the researchers' consultations, there was consensus among experts around the term 'incompatible' to explain the actual goods procurement system and its mismatch with service procurement.

The data confirmed H2 with the highest numbers of complaints (55%) filed for the contracting phase (phase #2) since 2008. There were 668 complaint files received from the OPO and coded by the researchers. The related contract phases are depicted in Table 3 and their relative weights illustrated in Data table 9.

Table 3: Complaints and related contract phases

Phase	Total related complaints
1. Pre-contractual	106
2. Contracting	369
3. Contract administration	109
4. Post-contract	12
1&2	5
2&3	1
2&4	1
3&4	1
N/A	64
Total	668

Data table 9: Percentage of complaints by contract phase

Contract phase	Percentage (%)
Phase one (Pre-contractual)	16
Phase two (Contracting)	55
Phase three (Contract administration)	16
Phase 4 (Post-contract)	2
N/A (Undefined or unapplicable)	10
Combined phases	1

For the purpose of this report, only files clearly related to phases #1-4 were analyzed. As previously noted, combined classifications as well as files that were not applicable (N/A) were eliminated. This reduced the number of applicable files from 668 to 596. The number of applicable files was further reduced to 563 by only analyzing files that were clearly defined as being either goods or service related (eliminating, as was previously done the N/A and goods and service tags). The distribution of complaints by contract phase after reducing the sample is shown in Data table 10.

Data table 10: Complaints by contract phase using reduced sample, n=563 (2008-14)

Contract phase	Number of complaints
Phase 1	102
Phase 2	350
Phase 3	99
Phase 4	12

Data table 11: Service complaints only by contract phase (2008-14)

Contract phase	Number of complaints
Phase 1	77
Phase 2	281
Phase 3	79
Phase 4	8

Data table 12: Goods complaints only by contract phase (2008-14)

Contract phase	Number of complaints
Phase 1	25
Phase 2	69
Phase 3	20
Phase 4	4

When sorted based on contract phase, most complaints are related to phases #1, #2 and #3 (see Data table 10), with the vast majority related specifically to phase #2. This trend holds for both goods and service complaints when they are isolated, with both demonstrating higher instances of contracting phase #2 connections (see Data tables 11 and 12).

That the problems originate principally from contracting phase #2 suggests that it is not the subjective nature of evaluating a service that is at issue. Rather, since Phase #2 contains the steps from evaluation to contract award, it can be suggested that there is an incompatibility with the strategy, people, processes and infrastructure that inform the actual procurement system, potentially for both goods and service procurement. For instance, if a government agency is tendering for software, it may release a request for proposals that seeks labour at an hourly rate. The agent appears to be procuring a good (labour) when in fact, they should be looking for a solution to a problem and should formulate their request accordingly.

Since phase #2 complaints are highest for goods and services, further research is required to determine precisely which variables are informing the result. Are unsuccessful bidders filing complaints, thereby driving up numbers? Is the evaluation of bids a source of contention for vendors? Are the underlying requests for proposals unclear and inviting unsuitable vendors to bid? Can specific elements in the procurement system be isolated to explain the complaints discrepancy? Is it a problem of human capacity? Or is it a combination of the two?

While the results of this report are not definitive, the evidence in consultation with the panel of public procurement experts appears to suggest that there are problems with the formulation and tendering of requests for proposals and the evaluation of bids that lead to higher numbers of service contract complaints. As in the example of software acquisition above, public procurers may be at an inherent disadvantage by treating procurement as a linear process (i.e. checking boxes) that can be applied in full no matter the type of procurement at hand.

Comparative data

To verify the findings from OPO's data, the researchers collected data from comparative jurisdictions from Canadian provinces and U.S. states. All Canadian provinces and territories (with exception to the Northwest Territories, Nunavut and Prince Edward Island) have provincial ombudsmen (see Table 4). These offices tend to be general ombudsmen managing complaints related to provincial services e.g. health, and report directly to the legislature (unlike the federal OPO). There is no domestic provincial equivalent of a procurement ombudsman in Canada. The majority of provincial ombudsmen do receive procurement contract related complaints (Quebec only accepts complaints related to provincial access to services). Due to low frequency of complaints, most provincial offices do not collect data on procurement complaints. Anecdotally however, the participating provincial offices indicated higher numbers of service contract related complaints when procurement related complaints are received.

Table 4: Canadian provincial ombudsmen offices' anecdotal evidence on procurement contract related complaints

Jurisdiction and comments

• Alberta
  • Jurisdiction over procurement contract complaints; few complaints received.
  • Of few received, mainly dealt with services.
• British Columbia
  • Jurisdiction over procurement contract complaints.
  • 289 complaint files since 2008 (unable to review goods v. service split).
• Manitoba
  • Unresponsive
• New Brunswick
  • Unresponsive
• Newfoundland and Labrador
  • Jurisdiction over procurement contract complaints.
  • Approximately 16 complaints since 2002.
  • Majority related to service contracts.
• Nova Scotia
  • Unresponsive
• Ontario
  • Jurisdiction over procurement contract complaints.
  • Mostly related to services.
• Quebec
  • Not applicable. Only accepts complaints related to provincial access to services.
• Saskatchewan
  • Jurisdiction over procurement contract complaints.
  • Rarely receives such complaints.
• Yukon
  • Jurisdiction over procurement contract complaints. No data on complaints.

Note: Canadian provinces and territories (with exception to the Northwest Territories, Nunavut and Prince Edward Island) have provincial ombudsmen.

Although the Canadian data could not further substantiate the report's initial findings, U.S. states provided a fruitful ground for comparison. The 42 comparative jurisdictions derived from NASPO's 2014 survey were contacted to request procurement complaints data.

Data table 13: Number and relevant sample of participating U.S. jurisdictions

Type of respondents	Amount
Total sample population	42
Responsive jurisdictions	16
Participating jurisdictions	11
Applicable sample	5

As depicted in Data table 13, of the 42 contacted jurisdictions, 16 jurisdictions were responsive and provided some form of data and/or information including complaints data and/or definitions of goods and service. From the 16 jurisdictions, 11 specifically shared complaints data of which five tracked the difference between goods contract and service contract related complaints on an annual basis. These five jurisdictions serve as the comparative sample population for this report. Two other jurisdictions produced complaints data differentiating between goods and service but only in total/aggregate form and not on an annual basis. To ensure comparability with OPO's data and to verify complaint levels across the time series, only jurisdictions that tracked goods versus service contract complaint data annually were used.

Data tables 14-18 depict the complaint numbers based on complaint type for Jurisdictions 1-5 on an annual basis.

Data table 14: Jurisdiction 1, complaint type, annual basis

Year	Number of goods complaints	Number of service complaints
2008	6	9
2009	5	12
2010	11	11
2011	7	4
2012	5	11
2013	2	3
2014	1	5

Note: As of 2011, procurement of Information technology (IT) services for Jurisdiction 1 was moved to a separate administrative branch.

Data table 15: Jurisdiction 2, complaint type, annual basis

Year	Number of goods complaints	Number of service complaints
2008	25	16
2009	36	14
2010	24	17
2011	33	29
2012	30	18
2013	12	18
2014	10	15

Data table 16: Jurisdiction 3, complaint type, annual basis

Year	Number of goods complaints	Number of service complaints
2008	0	0
2009	0	2
2010	0	2
2011	3	2
2012	1	1
2013	11	11
2014	1	5

Note: Data not collected in Jurisdiction 3 prior to 2009.

Data table 17: Jurisdiction 4, complaint type, annual basis

Year	Number of goods complaints	Number of service complaints
2008	3	16
2009	7	12
2010	2	24
2011	0	16
2012	1	18
2013	4	9
2014	2	10

Data table 18: Jurisdiction 5, complaint type, annual basis

Note: Data not collected in Jurisdiction 5 prior to 2012.

Year	Goods complaints	Service complaints
2012	15	10
2013	14	9
2014	15	10

Three out of five jurisdictions indicate higher numbers of service contract versus goods contract complaints. In aggregate form (i.e. total numbers of goods and service complaint data since 2008) and with two additional jurisdictions that provided data on complaints but not on an annual basis, the collected data indicates a trend similar to OPO's where the U.S. jurisdictions experience higher numbers of service contract versus goods contract complaints since 2008.

Data table 19: Comparing total complaints by type and number of Jurisdictions 1-7 and OPO

Jurisdiction	Goods complaints	Service complaints
1	37	55
2	170	127
3	16	23
4	19	105
5	15	10
6	25	229
7	33	2
OPO	125	480

Given the trends in data from U.S. jurisdictions, OPO's experience of higher numbers of service contract complaints does not appear to be unique. The comparative research also highlighted that some jurisdictions make the distinction between of goods, services, IT and even construction materials. These distinctions have been recognized administratively by separating their procurement functions into different agencies.

In Mississippi for instance, there are four oversight agencies for procurement that have been established in recognition of their differences: one for Commodities (non-IT-related); one for Services; one for Construction and one for IT services. This differentiation underscores the distinct processes and expertise required for acquiring goods versus services. Pennsylvania has underscored the complexity of IT services procurement by delegating the responsibility to the Office of Administration in the Governor's Office, since 2011.

Having experienced difficulties in procuring major IT projects (e.g. canceling or suspending IT projects after years of development, projects exceeding cost and scheduling assessments), California struck a taskforce to study the situation and provide solutions named, the California Task Force on Reengineering IT Procurement for Success. Reporting in August 2013, the task force made various recommendations to improve IT procurement through a more in-depth pre-contract phase, human resources with procurement expertise, changes to bid evaluation, etc. The recommendations from California's taskforce appear to align with this report's suggestion of an incompatibility between the procurement of service and the existing procurement system, including strategy, people, processes and infrastructure.

Observations and next steps

The Procurement Ombudsman's request to assess inordinate numbers of service contract complaints through the lens of complaints data appears to be novel. The findings of this report may, in-fact, reflect the limits of potential analysis to test the two hypotheses, based on available data, a constructed comparative data set and expert consultations.

The following observations are derived from the data collected for this report:

1. OPO is not unique in experiencing higher instances of service contract complaints.
2. There is an incompatibility between the procurement of outcome-oriented services and the underlying goods-oriented procurement system.
3. Further research is required to isolate explanations to definitively conclude why OPO is experiencing these results and their causes.

While the numbers of service complaints in Canada have increased beyond their relative increase in volume, causal linkages cannot be established to explain the inordinate number of service contract related complaints filed with the OPO. TBS data indicates that the quantity of goods procurement still outnumber the number of service procurement contracts, yet service contracts continue to register higher numbers of complaints.

Without more information from OPO files or vendor interviews, OPO data is limited to showing higher instances of complaints for contract phase #2. When these results were discussed with the expert panel, there was virtual unison in their interpretation of the findings. Expert commentary coalesced around the idea that human capacity is the most significant driver of service contract based complaints. The procurement process underlying the staff is unsuitable for complex outcome-based service procurement. Political and bureaucratic incentives muddy the environment, leaving accountability for procurement nebulous.

The researchers were asked to better understand why service complaints outnumbered goods. Beyond the analysis undertaken in this report, the data cannot be extrapolated further to explain or prove why there are higher numbers of service complaints. Therefore, suggestions for future study are made based on expert consultations to frame the current findings and propose directions for next studies.

Data to explain why the numbers of service contract complaints are higher can be acquired in different ways, such as internal analysis and assessment because OPO staff has most direct access to files. Future research directions may include: comparative analysis of jurisdictions with high levels of service contract procurement complaints as well as jurisdictions that treat service procurement as a separate issue (e.g. by placing it in a separate department); interviewing current public procurers to assess whether service and goods procurement are viewed differently; and interviewing complainants to better understand the nature of their complaint.

Public procurement is a multifaceted undertaking in a complex environment of political incentives and rules. To procure today is doing more than acquiring an item. With the increasing procurement of various services and the different requirements for outcome or solution oriented service procurement versus input oriented goods procurement, a rethinking of the strategies, processes and human resources required to effectively procure a service may be necessary.

Annex 1: Operationalization of the Procurement Ombudsman's mandate

The Procurement Ombudsman operationalizes their mandate by evaluating the procurement practices of departments and receiving complaints from Canadian suppliers. Departmental procurement practices are reviewed for instance, to ensure that they align with Cabinet policies, that they respect the Financial Administration Act, that the practices adhere to previous reviews of the department's procurement practices, etc.

Procurement contract complaints, that are the focus of this report, tend to fall in two categories: first, to contest a contract award; and the second, is to contest the administration of a contract.

For both types of complaints, Canadian suppliers can file a complaint in writing with the Procurement Ombudsman (PWGSC Act, 22.2 (1)) within 30 days after a contract award has been announced (and up to 90 days after the announcement if an exception is granted by the OPO) (Procurement Ombudsman Regulations 7(1)). A complaint is considered filed when the Procurement Ombudsman receives the required information that includes documentation to establish the complaint, the facts and grounds of the complaint, the potential net profit the complainant could have made and the complainant's cost of submitting the bid (Procurement Ombudsman Regulations, 7(4)). Once the complaint is filed, the Procurement Ombudsman has 10 working days to notify the complainant and the relevant department whether the complaint will be reviewed (Procurement Ombudsman Regulations, 8).

Beyond meeting the minimum information requirements established in section 7 of the regulations, the facts and grounds for the complaint can never have been subject to court proceedings or have been a matter before the Canadian International Trade Tribunal (CITT) and cannot be exempted by any legislation (Procurement Ombudsman Regulations, 9(1)). At any time, a review by the Procurement Ombudsman can be ended for not meeting requirements, if the complainant withdraws the complaint, if the contract award is cancelled or if the Procurement Ombudsman "determines that the complaint is frivolous or vexatious" (Procurement Ombudsman Regulations, 10).

The Procurement Ombudsman typically reviews complaints "under a simple review process", (Procurement Ombudsman Regulations, 11(1)). Relevant departments are given 15 working days to provide comments to the complainant on the file. An expanded review process at the request of the complainant or the department or at the initiative of the Procurement Ombudsman is also possible. Involved parties must be notified immediately if the Procurement Ombudsman is undertaking an expanded review.

For an expanded review, the nature of the complaint, the value of the contract, the complexity of the bid requirements or procurement process, potential impacts that findings and any recommendations could have on government operations and resources must be considered (Procurement Ombudsman Regulations, 11(2)). During an expanded review, the Procurement Ombudsman reviews the complaint file and mediates as the department concerned and the complainant provide comments giving the other party an opportunity to respond in 10 working days (beginning with the department) (Procurement Ombudsman Regulations, 11(2)).

Regardless of whether a simple or expanded review is undertaken by the Procurement Ombudsman, they are required to report on their findings and recommendations no more than 120 working days after the complaint was filed (Procurement Ombudsman Regulations, 14) to the complainant, the minister of the department concerned and the Minister of PWGSC. The Procurement Ombudsman is restricted to recommending compensation if found to be reasonable^{Footnote 13}, since only the Governor in Council can impose fines (Public Works and Government Services Act, 23(2)).

When the Procurement Ombudsman reviews contract administration complaints, the same validation process as with contract award complaints is followed. The review process however, differs slightly. To review a contract administration complaint, the Procurement Ombudsman considers whether the administration was reasonable given the circumstances and whether the parties acted in bad faith (Procurement Ombudsman Regulations, 20). While the Procurement Ombudsman cannot make recommendations to alter the terms and conditions of the contract or to suggest a remedy other than those in the contract (Procurement Ombudsman Regulations, 21), there are alternate means for resolution, namely through alternative dispute resolution process (ADR).

ADR is an option to settle disputes if requested by a party to the contract related "to a dispute on the interpretation or application of contract's terms and conditions" (Procurement Ombudsman Regulations, 23(1)). In order to proceed with ADR, the Procurement Ombudsman must obtain agreement of all parties, with participants agreeing to bear the costs. ADR only proceeds if parties agree to the terms.

Annex 2: Data collection template

File/Vendor #	Nature of contract (G or S)	Industry	Vendor specs	Related department	Nature of complaint	Related phase (1-4)	Resolution? (Y/N)	Date of complaint (MM/YY)	Date of resolution (MM/YY)
2012/13-115	(S)	N/A	N/A	National Research Council Canada	Security clearance	1- Pre- contractual	Out of mandate; considered for future Procurement Practice Reviews (PPR)	12-Aug	Not indicated
2012/13-301	(S)	N/A	N/A	Canadian Nuclear Safety Commission (CNSC)	Kickbacks	1- Pre- contractual	Out of mandate; Privy Council Office (PCO) is conducting investigation centrally	13-Feb	Not indicated
2012/13-267	(S)	N/A	N/A	Government wide	Employer- employee relationship	1- Pre- contractual	Out of mandate; systemic issue, considered for future PPR	13-Jan	Not indicated
2014/15-159	(S)	N/A	N/A	PWGSC obo Health Canada	Security	1- Pre- contractual	Out of mandate; not contract holder	14-Aug	Not indicated
2014/15-511	(S)	N/A	N/A	N/A	Security	1- Pre- contractual	Provided Criminal Intelligence Service Canada (CISC) information	15-Mar	Not indicated
2014/15-002	(S)	N/A	N/A	Canada Revenue Agency (CRA)	Statement of work (SOW)	1- Pre- contractual	Did not provide permission to contact department	14-Apr	Not indicated
2014/15-491	(S)	N/A	N/A	PWGSC Library and Archives	Not getting work	1- Pre- contractual	Out of mandate; Supply Arrangement (SA)	15-Feb	Not indicated
2014/15-489	(S)	N/A	N/A	N/A	Evaluation and selection plan	1- Pre- contractual	Provided options to consider	15-Feb	Not indicated
2013/14-336	(S)	N/A	N/A	Canadian Human Rights Tribunal	Constructing directly with company's resource	1- Pre- contractual	Complainant did not come forward in writing	13-Dec	Not indicated
2012/13-303	(S)	N/A	N/A	Industry Canada	Restrictive mandatory criteria	1- Pre- contractual	Out of mandate; OPO was only cc'd	13-Feb	Not indicated
2011/12-221	(S)	N/A	N/A	Not indicated	Restrictive requirements	1- Pre- contractual	Withdrawn	11-Nov	Not indicated
2010/11-061	(S)	N/A	N/A	PWGSC	Selection criteria favoured one bidder	1- Pre- contractual	Out of mandate	10-Jun	Closed
2014/15-301	(S)	N/A	N/A	Unknown	SOW and evaluation and selection plan	1- Pre- contractual	Would be considered for future PPR	14-Nov	Not indicated

Annex 3: Data request chart

Jurisdiction name:

Total annual complaints
This table is only an example
and contains no data.

Year	Total number of complaints
2008
2009
2010
2011
2012
2013
2014

Goods v. service complaints
This table is only an example
and contains no data.

Year	Good complaints	Service complaints
2008
2009
2010
2011
2012
2013
2014

Definitions
This table is only an example
and contains no data.

Good	Service

Annex 4: Contacted jurisdictions

Works cited

• Ancarani, Alessandra, Mauro, Carmela, M. Daniela Di Giammanco. (2009). "The Purchase of Technology in Health Organisations: An Analysis of its Impact on Performance." Journal of Public Procurement 9(1): 109-131.
• Emanuelli, Paul. (2005). Government Procurement. Markham, ON: LexisNexis Butterworths.
• Gardenal, Francesco. (2013). "A Model to Measure e-Procurement Impacts on Organizational Performance." Journal of Public Procurement 13(2): 215-242.
• Government of Canada. (2008). Available at: Procurement Ombudsman Regulations. Ottawa: Minister of Justice.
• -------. (2006). Available at: Department of Public Works and Government Services Act. Ottawa: Minister of Justice.
• Hall, Peter A. and Rosemary Taylor. (1996). "Political Science and the Three New Institutionalisms." Political Studies XLIV: 936-957.
• Hartley, Keith. (1991). "Public Purchasing." Public Money and Management 11(1): 45-49.
• Keeney, Sandy. (2007). "The Foundations of Government Contracting." Journal of Contract Management (Summer 2007): 7-19.
• Library and Archives Canada. (2001). "ARCHIVED - Canadian Confederation" (accessed June 12, 2015).
• McCue, Clifford and Roman, Alexandru V. "E-Procurement: Myth or Reality?" Journal of Public Procurement 12(2): 221-248.
• McCue, Clifford P., Gianakis, Gerasimos A. (2001). "Public Purchasing: Who's Minding the Store?" Journal of Public Procurement 1(1): 71-95.
• National Association of State Procurement Officers (NASPO). NASPO State and Local Procurement: A Practical Guide, Second Edition. Washington, D.C.
• -------. (2014). 2014 Survey of State Procurement Practices. Washington, D.C.
• Website: Office of the Procurement Ombudsman. (2015).
• -------. (2013). Presentation to the Delegation from the Ministry of Supervision People's Republic of China.
• Organisation for Economic Co-operation and Development (OECD) (2013). Mapping out good practices for promoting green public procurement. Paris: OECD Publishing.
• -------. (2003). The Environmental Performance of Public Procurement: Issues of Policy Coherence. Paris: OECD Publishing.
• Pierson, Paul. (2004). Politics in Time: History, Institutions and Social Analysis. Princeton, NJ: Princeton University Press.
• Peters, Guy B. (2012). Institutionalist Theory in Political Science, 3rd Edition. New York: Continuum.
• Plamondon, Aaron. (2010). The Politics of Procurement. University of British Columbia (UBC) Press: Vancouver.
• Public Works and Government Services Canada. (2015). Available at: "1. Supply Manual Glossary".
• Sieverding, McLean. (2008). "Choice in Government Software Procurement: A Winning Strategy." Journal of Public Procurement 8(1): 70-97.
• Snider, Kieth F. (2006). "Procurement Leadership: From Means to Ends." Journal of Public Procurement 6(3): 274-294.
• State of California. (2013). Available at: "Recommendations to improve large information technology procurements: A road map for success in California" (PDF version 960Kb) (Help with alternative formats and plug-ins). Taskforce on Reengineering IT Procurement for Success.
• Thai, Khi. (2004). "Challenges in Public Procurement: An International Perspective—Symposium Introduction." Journal of Public Procurement 4(3): 312-318.
• Thai, Khi, C.M. Hartland, J. Telgen, L.A. Knight and G. Callendar. (2007). "Challenges Facing Public Procurement." In L.A. Knight et al. (Eds). Public Procurement: International Cases and Commentary. London, UK: Routledge.
• Thai, Khi et al. (2005). "Challenges in Public Procurement." K. Thai et al. (Eds). Challenges in Public Procurement: An International Perspective. Boca Raton, FL: PrAcademics Press.
• Treasury Board of Canada Secretariat (2015). "Publications - Contracting Data" (accessed June 16, 2015).
• -------. (2013), "Publications - Contracting Data" (accessed April 27, 2015).
• Worthington, Robert C. (2013). Desktop guide to procurement law. Markham, ON: LexisNexis.
• -------. (2004). The public purchasing law handbook. Markham, ON: LexisNexis.

Experts consulted

• Dermot Cahill, Professor, Bangor University Institute for Competition and Procurement Studies, United Kingdom.
• Paul Emanuelli, General Counsel and Managing Director, the Procurement Office, Toronto, Canada.
• Krista Ferrel (for NASPO), Director of Strategic Programs, National Association of State Procurement Officials, United States.
• Steven Graves, Professor, Massachusetts Institute of Technology (MIT), United States.
• Stephen B. Gordon, Professor of Practice, Public Administration, Old Dominion University, United States.
• Khai Thai, School of Public Administration, Florida Atlantic University, United States.

Date modified:

2018-11-01