Research on goods versus services contracts
- Research design
- Comparative data
- Observations and next steps
- Annex 1: Operationalization of the Procurement Ombudsman's mandate
- Annex 2: Data collection template
- Annex 3: Data request chart
- Annex 4: Contacted jurisdictions
- Works cited
An analysis of supplier contacts to the Office of the Procurement Ombudsman (OPO) since its establishment in 2008, revealed a disproportionate number of complaints related to service contracts (as oppose to contracts for goods). This despite the fact that, while there have been increases in spending on service procurement by the federal Government over the last 15 years, the number of goods contracts awarded remain higher than the number of service contracts. Since there are greater number of goods contracts awarded each year, once could presume that a higher number of goods complaints should ensue, but this is not the case.
To better understand this situation and why it is happening, OPO engaged University of Ottawa researchers to answer the following questions:
- What inherent differences in contracting for services (as opposed to goods) may be contributing to the disproportionate number of complaints about service contracts compared to contracts for goods?
- What issues, factors or reasons may be contributing to the disproportionate number of complaints related to services?
- What conclusions may be drawn from the research?
- Are comparable jurisdictions experiencing similar problems?
The University of Ottawa's report confirmed there are higher numbers of service contract complaints compared to goods contract complaints. Based on data collected for this report, the majority of comparable United States (U.S.) jurisdictions have also experienced higher levels of service contract related complaints, meaning that OPO is not unique in this area.
While the research was inconclusive with regards to causality, it nonetheless provides a potential reason for the disproportionality: the report's key observation is that a 'structural incompatibility' may exist between the actual procurement system (designed for goods) and the increasing number of service contracts, which are often viewed as more complex. This 'structural incompatibility' may be a result of the evolution of Canada's procurement system.
The report notes this area of public procurement (i.e. the difference between procuring goods compared to services) is largely understudied and suggests that more research is required.
OPO is pleased to make this report publicly available to encourage additional research in this area of public procurement.
The authors wish to acknowledge the following public procurement experts for their insightful comments throughout the development of this report:
- Dermot Cahill, Professor, Bangor University Institute for Competition and Procurement Studies, United Kingdom.
- Paul Emanuelli, General Counsel and Managing Director, the Procurement Office, Toronto, Canada.
- Krista Ferrel (for National Association of State Procurement Officials (NASPO)), Director of Strategic Programs, NASPO, United States.
- Steven Graves, Professor, Massachusetts Institute of Technology (MIT), United States.
- Stephen B. Gordon, Professor of Practice, Public Administration, Old Dominion University, United States.
- Khai Thai, Professor, School of Public Administration, Florida Atlantic University, United States.
Any errors or omissions in this report are the sole responsibility of the authors.
Public procurement is undertaken in a complex environment where political interests (e.g. regionalism, electioneering) intersect with spending and the market economy. As Worthington (2013) explains,
"[…] the public procurement of goods, services and construction are highly regulated processes" (p. 508). Furthermore, public procurement
"involves a wider set of choices about what to buy, from whom and how" (Hartley 1991, p. 45). The vast sets of rules and regulations in place to govern procurement are meant to sanitize the process and remove as many potential conflicts of interest as possible.
Public procurement is inherently different from private procurement with repercussions not for shareholders, but for taxpayers, citizens as well as the political livelihood of a government. Paul Emanuelli conveys the interplay of public procurement, politics and public trust well:
As the Pacific ScandalFootnote 1 illustrates, government procurement controversies can bring down the best of them. Even the Fathers of Confederation were not immune. The paramount importance of good government is firmly entrenched in our political fabric. Reactions to spending improprieties run as deep in our political psyche as the memories run long. Public officials who run on the wrong side of these rules do so at their peril. Today, computer and advertising contracts may have replaced railway construction deals as the subject of controversy, but accountability for proper government spending remains a matter of national importance (Emanuelli 2005, p. 1).
Given its political repercussions, it was little surprise that procurement was targeted when political scandal surrounding accountability surfaced in the mid-2000s in Canada. Part of the reaction was to enact legislation for a Procurement Ombudsman to receive procurement contract complaints and to review the procurement processes of federal organizations. Pursuant to the Federal Accountability Act that received Royal Assent in December 2006, the Procurement Ombudsman's mandate and enabling legislation were made by amendments to the Department of Public Works and Government Services Act (2006) (section 23.1).Footnote 2 Establishing the Procurement Ombudsman was one of "several elements of the government's comprehensive plan to strengthen the fairness, openness and transparency of federal procurement" (OPO Annual Report 2008-09, p. 10). While the Act establishes the role and responsibilities of the Procurement Ombudsman, the Procurement Ombudsman Regulations that came into effect in May 2008 operationalize the mandate.Footnote 3
A plain read of the Procurement Ombudsman's mandate defined in the Department of Public Works and Government Services Act and the operationalization of their mandate through the Procurement Ombudsman's Regulations, appears to suggest that the Procurement Ombudsman can receive complaints once a contract has been awarded for matters related to the evaluation of bids, the contract award, as well as contract administration. There may be a risk of bias in the nature of complaints filed with the Procurement Ombudsman in relation to their mandate. However, given the perceived scope of the Procurement Ombudsman's mandate, there appear to be linkages between the legislation and regulations and the pre-contractual, contracting, and contract administration phases of the four phases of the contracting process (see Table 1). Only phase four, the post-contract phase, may be contestably part of the Procurement Ombudsman's mandate. For a discussion of how the Procurement Ombudsman operationalizes its mandate through the filing and evaluation of complaints, please see Annex 1.
The Government of Canada spends roughly 15-20 billion dollars in the procurement of goods and services every year through hundreds of thousands of contracts (Treasury Board of Canada Secretariat (TBS) 2015). Procurement practices at an aggregate level matter for politics and public finance because of the amount of money spent and the ways in which it is spent.
When contracts are assessed based on monetary value, service contracts cost consistently more than goods contracts since 1996 (Data tables 1 and Data table 2). In terms of volume, the number of goods contracts since 1996 is greater than the number of service contracts, with exception to 1997 and 1998 (see Data table 3 and Data table 4). Most contracts (90%) are worth less than $25,000 (i.e. low value) and represent less than 10% of the total value of contracts. While contracts worth $25,000 and above represent the largest portion of spending, they are far fewer in number (OPO Presentation 2013). Prior to the establishment of the OPO in 2008, suppliers for low-value contracts had no formal complaint process (vendors with higher value contracts have recourse with the Canadian International Trade Tribunal).
(OPO Annual Report 2008-09, p. 32)
|Description of complaints|
Note: Many evaluation issues are related to planning, i.e. accurately defining (and applying) the terms of assessment clearly and regularly from the outset.
|3. Contract administration||
|4. Post contract||
|Year||Total value of yearly contracts||Total value of construction contracts (approx.)||Total value of services contracts (approx.)||Total value of goods contracts (approx.)|
|1996||8,703,172||1 M||3.5 M||4.5 M|
|1997||9,093,408||0.8 M||0||4.2 M|
|1998||13,732,457||0.6 M||0||5.1 M|
|1999||9,858,154||0.6 M||4.9 M||4.5 M|
|2000||9,360,172||0.7 M||5.1 M||3.6 M|
|2001||13,146,679||0.9 M||6.5 M||6.2 M|
|2002||12,822,851||1 M||7.7 M||4.4 M|
|2003||13,393,801||1 M||7.9 M||5.2 M|
|2004||19,063,453||1 M||11.9 M||6 M|
|Year||Total value of contracts (Yearly)||Total value of construction contracts (approx.)||Total value of services contracts (approx.)||Total value of goods contracts (approx.)|
|2005||15,944,061||0.9 M||10.2 M||4.9 M|
|2006||12,153,012||0.9 M||7 M||4.1 M|
|2007||14,257,457||0.8 M||6.2 M||7.3 M|
|2008||16,439,171||0.9 M||6.8 M||8.7 M|
|2009||20,351,495||1.3 M||8.8 M||10.1 M|
|2010||16,203,482||1.3 M||9.2 M||5.4 M|
|2011||16,033,758||1.9 M||7.0 M||7.2 M|
|2012||15,149,203||1.3 M||7.1 M||7.1 M|
|2013||14,552,157||1.3 M||7.6 M||6.1 M|
|Year||% of Total number of contracts (Goods)||% of Total number of contracts (Services)||% of Total number of contracts (Construction)|
|Year||% of Total number of contracts (Goods)||% of Total number of contracts (Services)||% of Total number of contracts (Construction)|
Since the establishment of the office in 2008, the OPO observed inordinate numbers of service contract related complaints. While it may be suggested that the number of service contract complaints would naturally increase as more services were procured, the argument does not appear to hold based on Treasury Board of Canada Secretariat (TBS) procurement data. Despite increases in spending on service procurement over the last 10-15 years, the volume of goods procurement contracts remains higher than the volume of service contracts. Since there are a greater number of goods contracts, one could presume that a higher number of complaints should ensue, but this is not the case.
One could also suggest that the subjective nature of service makes it more difficult to assess or quantify its outcomes, thereby incurring higher than average numbers of complaints. However, analysis of OPO data challenged this reasoning since the majority of complaints filed were related to the contracting phase of the procurement process (i.e. the period in which the request for proposals is developed, bidders' submissions are evaluated) and the contract awarded, and not the contract or post-contract phases where the contracts are amended and performance are evaluated, respectively. At the inception of this research project, the subjectivity of service was hypothesized to have led to increased instances of complaints. However, definitions from half of the participating U.S. jurisdictions provide some indication that service is not so subjective that it cannot be defined.
If the volume of contracts cannot explain high numbers of complaints and if complaints are not related to the evaluation of service contract outcomes, then how can the inordinate number of service contract complaints be explained? What variables may be contributing to this result? Is the public procurement of services inherently different than that of goods? Is the OPO the only jurisdiction experiencing this tendency or is service procurement a challenge across jurisdictions?
The researchers were engaged by the OPO to attempt to answer these questions. To undertake this inquiry, a series of research questions was developed and agreed upon by the researchers and the OPO. Two working hypotheses were proposed and refined based on consultations with international public procurement experts. The consulted experts also verified the research design and methodology. This report's fundamental observation is that a structural incompatibility exists between the actual procurement system (designed for goods) and the increasing public procurement of services (still outnumbered by goods). In our consultations, there was consensus among experts around the term 'incompatible' to explain the current goods procurement system and its mismatch with service procurement.
From the experts' perspective, the fundamental incompatibility is one of human capacity (i.e. procurement officers do not have the education/training to execute service procurement in a complex environment). Building on this perspective, this report suggests that there are various sources of incompatibility including: strategy, people, processes and infrastructure that must be considered to evaluate the existing model for service procurement. While this finding is not definitive, it is a helpful advancement in the largely understudied area of public procurement, notably through the lens of contract complaints.
This report proceeds by establishing the project's research design, followed by its methodology and discussion and analysis of the findings and the comparative data. To conclude, the report suggests future research directions to continue to investigate this subject.
Canada's modern procurement system was developed during wartime, as many of the historic foundations of government procurement tend to lie at critical moments, e.g. war, a major national undertaking such as space exploration, etc. (Keeney 2007). During World War I (WWI), military procurement in Canada was ad-hoc with the army, air force and navy purchasing independently. To correct deficiencies and limit profiteering by suppliers, the federal government began to coordinate procurement efforts (Plamondon 2010, p. 3). The historic procurement processes established in the mid-20th century to acquire materials tended to emphasize goods (for a short overview of key moments in Canada's procurement history, see).
Summary of Canada's procurement history
To coordinate procurement efforts the federal government established:
- Defence Purchasing Board (DPB) (operational July 1939).
- With increased demands of WWII, DPB replaced with War Supply Board (est. November 1939), which became the Department of Munitions and Supply.
- Non-military supplies at the time were procured through the Canadian Export Board (within the Department of Trade and Commerce).
- Post-WWII, military procurement was transferred to the Ministry of Trade and Commerce (February 1947), then managed through the Canadian Commercial Corporation.
- Government reorganization (1963, post-Glassco Commission) saw the establishment of the Canadian Government Purchasing, Supply and Repair Services.
- In 1993, Public Works and Government Services Canada (PWGSC) was created. PWGSC managed procurement for the entire public service including the military, no longer differentiating between military and non-military procurement.
- Departments are able to undertake their own procurement for contracts valued under $25,000.
The path dependent tendencies of any policy are inclined to lock-in and perpetuate over time (Pierson 2004; Peters 2012; Hall and Taylor 1996). With these precedents, change while not impossible can be difficult. Following the concept of path dependency, the procurement system (made up of strategies, people, processes and infrastructure) has a historical precedent designed to acquire goods that persists, that may impact public procurement today.
Key terms and definitions
For the purposes of this report, goods are defined as articles, commodities, equipment, materials or supplies (following the Government of Canada's Supply Manual Glossary). Service is defined as what is not a good. Only the distinction between goods and service contracts are of interest in this report. As such, all OPO data related to procurement contracts was coded as related to either goods, services, good and service (where the file included both aspects of goods and service) or not applicable (where the file was undefined or not relevant to the study). Construction was not considered an independent category and was coded based on the most closely related of the four categories.Footnote 6
Neither the Treasury Board Contracting Policy (Chapter 16)Footnote 7 nor the Department of Public Works and Government Services' Supply Manual Glossary provide an explicit definition of what constitutes a service, yet definitions for goods and construction exist. In spite of not having a definition for service, the TBS distinguishes between goods, service and construction procurement in its procurement reporting data.Footnote 8
Among public procurement officials, there is no single agreed upon definition of what constitutes a service. The National Association of State Procurement Officials (NASPO), an umbrella organization that represents the state chief procurement officers in the U.S., distinguishes between goods and service procurement. Definitions of service range from what is not a good to clear lists of what constitutes a service, as is the case in various U.S. states (see Table 2).
Goods procurement tends to be input focused where the request for proposals seeks to acquire an article, commodity, equipment, materials or supplies. By contrast, when procuring a service (which, for the purpose of this report is anything but a good), one is more likely than not seeking a solution to a problem or a particular outcome. When procuring a service,
"[…] the solicitation should include a program narrative, answering the question, […]" What problem is the agency seeking to solve?"" (NASPO 2015 p. 163). As NASPO's handbook for public procurers explains,
In the purchase of services, the specifications or scopes of work ought to emphasize achieving a specific result, and not the rigid details of the process for getting to it. The goal of a competition is to ensure that the service provider's bid or proposal responds with a solution to a defined need (NASPO 2015, p. 162-163).
The crucial difference in goods versus service procurement comes down to a distinction in purpose. A good is acquired because an article that takes the form of equipment, materials or supplies is needed. When procuring a good, be it to fight a war, to create workspaces by buying desks and chairs, to purchase fencing for a federally sponsored event in the National Capital Region, etc., the item is acquired because it is needed as a tool or input to fulfill a function. By contrast, the procurement of services tends to solve a problem instead of solely acquiring a physical item. The way that service should be bought is different than buying a good. Rather than going to market to buy a specific item (as in goods procurement), one goes to tender seeking solutions to solve a problem.
Table 2: Jurisdictional definitions of service contractsFootnote 9
The furnishing of labor, time or effort by a contractor or subcontractor that does not involve the delivery of a specific end product other than required reports and performance. Does not include employment agreements or collective bargaining agreements.
- Government of Canada
No definition exists.
That which directly engages in the time or effort of an independent contractor whose purpose is to perform an identifiable task, study, or report rather than to furnish an end item of supply, goods, equipment, or material (Statute 73-502).
- New Jersey
A service contract primarily requires labor, time and/or effort to provide intangible general or professional assistance to accomplish an objective (construction and architecture are overseen by a separate division).
- North Dakota
The furnishing of labor, time, or effort by a contractor, not involving the delivery of a specific end product other than reports that are merely incidental to their required performance. The term includes professional services.
The furnishing of labor, time or effort by a contractor not involving the delivery of a specific end product other than drawings, specifications, or reports which are merely incidental to the required performance. The term includes the routine operation or maintenance of existing structures, buildings, or real property. The term does not include employment agreements or collective bargaining agreements. The term includes utility services and those services formerly provided by public utilities such as electrical, telephone, water, and sewage service.
Given the inordinate numbers of service contract complaints OPO receives, there are likely explanatory variables that can account for the tendency. The researchers have found however, that the data collected by the OPO over its history is of insufficient depth to validate the observations presented in this report. However, leveraging the limited data and the expertise of a panel of international public procurement experts, the researchers have attempted to draw certain inferences as to the root causes of the growth in service complaints. Yet, in spite of consultations with several dozen U.S. jurisdictions and the Organisation for Economic Co-operation and Development (OECD) on behalf of its member countries, a dataset that would substantiate the professional opinions of the expert panel could not be identified. The observations in this report require further study to determine whether they in fact represent root causes.
Research directions in public procurement
While the practice of public procurement has been undertaken in many forms for over a century (see Kenney 2007), its study remains a relatively niche undertaking. In the literature on public procurement, the distinctions between public and private procurement (as well as their interplay) are discussed (Emanuelli 2005; Worthington 2013), the complexities of public procurement are studied (Hartley 1991; Emanuelli 2005; Worthington 2004 and 2013; Thai 2004; Thai et al. 2005; Thai et al. 2007), the roles of human resources and management capacity are assessed (McCue 2001; Snider 2006), as are new trends, e.g. green procurement (OECD 2003; OECD 2013), e-procurement (McCue and Alexandru 2012; Gardenal 2013), alternative service delivery (NASPO 2015), etc. The differences in the complexity of procuring services versus goods are recognized (California 2013; NASPO 2015), with special emphasis on the procurement of Information technology (IT) services (Sieverding 2008; Ancarani, Di Mauro and Giammanco 2009; California 2013). However, the implications of procuring services in systems that tend to be goods oriented, remains understudied, leaving a gap in the literature on public procurement. As governments around the world procure increasing numbers of services, the ways in which they are acquired merits closer attention.
Complaints can be used as a lens through which to study the existing procurement process. The number and nature of procurement related complaints serve as an entry point for testing whether the goods-oriented procurement process is contributing to inordinate numbers of service contract complaints. Based on data collected for this report, the majority of comparable jurisdictions have also experienced higher levels of service contract related complaints. However, no published research appears to have been undertaken to suggest explanations for this state of affairs. Literature reviews and expert consultationsFootnote 10 confirmed that using complaints as a lens to evaluate the procurement process is a unique research angle.
This report confirms that there are higher numbers of service contract complaints versus goods contract complaints (despite the greater volume of goods procurement in the federal government). Through research and expert consultations, potential explanations (that require further research to confirm) include incompatibilities between the goods-oriented procurement system and the procurement of services as well as human capacity in the system to undertake service procurement. Without changing the underlying structure and without training staff to think differently about the task of service procurement, the Procurement Ombudsman may inevitably continue to receive inordinate numbers of service contract versus goods contract related complaints.
The content of this report is based on the OPO's internal data, data shared by comparable jurisdictions, expert consultations, Government of Canada data, as well as secondary sources (including scientific articles and expert reports). This report serves not only to fulfill the OPO's research request, but will also contribute to the void in the literature on the incompatibility between the use of a goods-oriented procurement process for the procurement of services.
In consultation with the OPO, the following research questions were defined to guide the inquiry:
Based on initial research, two hypotheses were developed to confirm the higher number of service contract complaints relative to goods contract complaints and to suggest potential explanations. Both hypotheses were confirmed through data collection and verified by expert consultations.Footnote 11
This report's hypotheses are:
There are an inordinate number of service contract related complaints v. goods contract related complaints.
To validate H1, there should be higher numbers of service contract complaints than goods contract complaints. The data confirmed H1 with 79% of relevant complaints related to service contracts compared to 21% for goods contracts based on OPO's data since 2008.
Goods procurement is input focused, whereas service procurement is outcome focused. Therefore, there is a structural incompatibility of the existing procurement model (developed to procure goods) for service procurement.
If H2 holds, there should be higher numbers of complaints in the pre-contractual and contracting phases, suggesting that the procurement process may be oriented toward input-based goods instead of outcome oriented service procurement. The data confirmed H2 with 55% of filed complaints since 2008 related to the contracting phase. The researchers did not assess the complaints in the context of the Procurement Ombudsman's mandate but on which contract phase the complaint falls.
Confirmation of the two hypotheses point to possible root causes of difficulties with service procurement, rather than a definitive "theory." To ensure that the results of this study are not overstated, it should be noted that there is broad agreement among experts that there is currently insufficient data and analysis in the literature to fully explain the endemic problems associated with service procurement—it is a work in progress.
To better establish the linkages made in this report, it may be useful to draw a statistically valid sample of the OPO files for further in-depth assessment. Identifying clearly the reason for the complaint and even interviewing the complainant can help to obtain more explicit conclusions about the variables contributing to higher numbers of service contract complaints. This supplementary focused researched can verify the report's observation of an incompatibility between the procurement process and the services being procured.
For this project, a scoping exercise of OPO's complaints files was undertaken to assess the numbers of and differences between goods and service contract complaints. Given the confidential nature of OPO's files, approximately eight members of the office's staff reviewed all procurement complaint files since the office's opening in 2008 (the researchers were not allowed to handle the confidential files). While there may be inconsistencies in the subjective nature of data collection and recording given the involvement of eight staff, standardization was encouraged through the use of a pre-established template (see Annex 2 for the template and Table 1 for coding (i.e. OPO's contract phase descriptions)).
OPO staff reviewed complaint files and recorded relevant data based on the nature of the contract (i.e. good v. service), the type of complaint (i.e. related contract phase), the related department, as well as the actions taken or resolution obtained. Only complaints related to procurement contracts were reviewed and counted in this assessment. All other files (e.g. general inquiry calls to the office) were not counted. Data collected by OPO staff was anonymized with all indicators removed so that it could be made public. The researchers recorded and coded the anonymized data.
In total 668 complaint files were received from the OPO and coded by the researchers. Of the 668 entries, 480 were related to service contracts, 125 related to goods, 7 were classified as part goods contract and part service contract, and 56 were not applicable (N/A) as they were either undefined or irrelevant to the data set (e.g. question to the OPO) (see Table 3 and Data tables 5 and Data table 6).
|Goods and services||7|
For the purpose of this final report, as well as for comparability with other jurisdictions, only files clearly defined as goods or service related were analyzed. This reduces the number of applicable files from 668 to 605.
|Complaint type||Percentage (%)|
|Goods and services||1|
This methodology was crucial for two reasons. First, for the researchers to test H1, all complaints and the split between goods and service had to be tabulated. Second, to test H2, it was necessary to link the complaint to a specific contract phase. OPO's pre-defined contract phases were adopted in this report and were used to code the "time" (i.e. related contract phase) of the complaint linking it to a set of steps and practices in the procurement process. See Table 1 for the phases and their descriptions.
When supported by a critical mass of data, identifying the related contract phase can suggest reasons for the higher number of service versus goods contract related complaints. For instance, complaints related to phases #1 and #2 (see Table 1 for definitions), may suggest pre-contract issues related to the formulation of the request for proposals as well as issues related to assessment and interpretation of terms in the evaluation process.
To test research findings, and to determine if similar jurisdictions were experiencing analogous complaint splits, a comparative assessment with data from comparable jurisdictions was undertaken. The 50 states of the United States were targeted for comparison, as many share geographical similarities with Canada and work with similar vendors. While not all states have the budgets of the Canadian federal government or similar populations to that of Canada in its entirety, many states have procurement-specific complaints processes embedded in their state system.
As a professional association, NASPO produces research, organizes education and training and convenes its members to address the challenges and practices of public procurement. In 2014, NASPO undertook a survey of "State Procurement Practices". Of its 50 members, 42 have legislation, rules or regulations related to procurement complaints. The researchers contacted these 42 states requesting data on the number of complaints their office received, the split between goods and service complaints, as well as their definitions (if they exist) for good and service (see Annex 3 for a sample data request chart and Annex 4 for a list of the contacted jurisdictions).
There were 16 participating jurisdictions that provided full or partial data (including a definition of "service") for this report. All of the jurisdictions were assured that the collected data was anonymized to ensure it could not be attributed to any specific jurisdiction, although some officials noted that their data is in fact publicly available.Footnote 12 There was significant interest among state procurement officials in the results of this study. To promote collaboration and interjurisdictional comparisons, anonymized data collected for this report will be shared in aggregate form with the participating jurisdictions.
The methodology for this report is rooted in inductive analysis. In conducting their enquiry, the researchers moved from a specific observation that OPO was experiencing inordinate numbers of service contract complaints to a discovery that most of the complaints link to the initial phases of the procurement process. While this report's dataset cannot explain why this pattern exists, inductive reasoning confirms that a pattern does exist.
Data from U.S. jurisdictions confirmed that OPO's experience is not unique. To suggest potential explanations and to establish future research directions, public procurement experts were consulted. The experts came to the conclusion that the current system is incompatible for the procurement of services and is exacerbated by a lack of human capacity in procurement, leading to higher numbers of service contract complaints. These potential explanations merit further research.
This section will discuss and analyze the report's results through the lenses of the two hypotheses and the results of their testing.
It was crucial to verify that there are consistently higher numbers of service contract complaints than goods contract complaints as the foundational premise of this report. The data confirmed H1 with 79% of relevant complaints related to service contracts compared to 21% for goods contracts based on data since 2008 (see Data table 7).
|Type of complaint||Percentage (%)|
With service complaints data significantly outnumbering those on goods, a time series of complaints since 2008 was developed to ensure that the discrepancy could not be attributed to anomalies or complaint "spikes" in particular years. As Data table 8 demonstrates, since 2008, service contract complaints have consistently outnumbered goods contract complaints, suggesting a relatively stable ratio (service: goods) over the study period.
The consistently higher numbers of service contract versus goods contract related complaints may suggest that a constant variable or set of variables is contributing to this trend.
The significantly higher numbers of service contract complaints were unsurprising to the panel of experts. For instance, NASPO indicated that service procurement, if not done properly, can be challenging. The organization is in the process of developing an educational platform for public procurement support for entry to senior-level state procurement personnel, which includes access to education and training on topics such as service level contracting methods.
Having established that service contract complaint numbers are much higher than goods contract complaints, the next section focuses on identifying potential variables to better understand the state of affairs.
Underlying H2 are assumptions about the procurement of goods and services. Working from the assumption that the procurement of goods and services should reflect their different goals, the second hypothesis seeks to test whether the way in which goods and services are procured can be a variable in the higher number of service contract complaints because of their different purposes. At an operational level, it is hypothesized that the distinctions manifest themselves as a structural incompatibility between the existing procurement model developed to procure input-oriented goods and the use of the model to procure outcome-oriented services. Can a model built to procure a good be applied to procure (what are often more complex) services?
|Year||Number of goods contract complaints||Number of service contract complaints|
The majority of the experts consulted pointed to a problem in human capacity when it comes to service procurement (see Box 2). From the experts' perspective, since the procurement of services is undertaken for an outcome and not an input distinguishing it from goods, it requires different processes (e.g. evaluation) and a different organizational culture to manage these files.
Experts and the importance of human capacity
In order to investigate potential root causes of service complaints in the procurement system, an international panel of experts was retained based on:
- Their global public sector perspective and experiences.
- Their knowledge of research, academic literature and their own contributions to the public procurement field.
- Their own experiences consulting and assessing public procurement problems to identify and understand possible avenues for future study to assess the identified problems/weaknesses in the system.
The experts generally agreed that beyond the structural incompatibilities, procurement is an inherent human problem.
As Professor Stephen Gordon of Old Dominion University indicated, public procurement is more than a process; it is part of a complex system of government that must respond to different pressures. Given the complex and dynamic environment of public procurement generally, the higher level of difficulty associated with the procurement of services as compared to the procurement of goods, the capability of individual procurement and contracting officials to execute service procurement as it should be executed is critical to the strategic success of public organizations.
The idea of focusing on individual procurers for change was reiterated by Professor Dermot Cahill, Bangor University Institute for Competition and Procurement Studies as well as NASPO that is providing procurement support and training. There is broad agreement among experts that change has to come from individual procurers who would adjust their analysis for the procurement of services on a case-by-case basis. The experts were clear that this type of change requires training and education programs as well as changes in organizational culture.
On the issue of human capacity, Professor Khi Thai of Florida Atlantic University specifically emphasized the importance of refining evaluation procedures to support individual procurers in assessing outcomes. Clear criteria for services do not always exist, thus procuring outcomes can be challenging. Often times, Dr. Thai indicated, procurement committees will default to price, treating lowest cost (i.e. best value for money) as the best option. This however, can be problematic because it may not fulfill the required outcome. Further investigation is required to address the question of "best value v. lowest cost" in public procurement, since how to benchmark quality against price differences remains to be studied.
Based on the researchers' consultations, there was consensus among experts around the term 'incompatible' to explain the actual goods procurement system and its mismatch with service procurement.
The data confirmed H2 with the highest numbers of complaints (55%) filed for the contracting phase (phase #2) since 2008. There were 668 complaint files received from the OPO and coded by the researchers. The related contract phases are depicted in Table 3 and their relative weights illustrated in Data table 9.
|Phase||Total related complaints|
3. Contract administration
|Contract phase||Percentage (%)|
|Phase one (Pre-contractual)||16|
|Phase two (Contracting)||55|
|Phase three (Contract administration)||16|
|Phase 4 (Post-contract)||2|
|N/A (Undefined or unapplicable)||10|
For the purpose of this report, only files clearly related to phases #1-4 were analyzed. As previously noted, combined classifications as well as files that were not applicable (N/A) were eliminated. This reduced the number of applicable files from 668 to 596. The number of applicable files was further reduced to 563 by only analyzing files that were clearly defined as being either goods or service related (eliminating, as was previously done the N/A and goods and service tags). The distribution of complaints by contract phase after reducing the sample is shown in Data table 10.
|Contract phase||Number of complaints|
|Contract phase||Number of complaints|
|Contract phase||Number of complaints|
When sorted based on contract phase, most complaints are related to phases #1, #2 and #3 (see Data table 10), with the vast majority related specifically to phase #2. This trend holds for both goods and service complaints when they are isolated, with both demonstrating higher instances of contracting phase #2 connections (see Data tables 11 and 12).
That the problems originate principally from contracting phase #2 suggests that it is not the subjective nature of evaluating a service that is at issue. Rather, since Phase #2 contains the steps from evaluation to contract award, it can be suggested that there is an incompatibility with the strategy, people, processes and infrastructure that inform the actual procurement system, potentially for both goods and service procurement. For instance, if a government agency is tendering for software, it may release a request for proposals that seeks labour at an hourly rate. The agent appears to be procuring a good (labour) when in fact, they should be looking for a solution to a problem and should formulate their request accordingly.
Since phase #2 complaints are highest for goods and services, further research is required to determine precisely which variables are informing the result. Are unsuccessful bidders filing complaints, thereby driving up numbers? Is the evaluation of bids a source of contention for vendors? Are the underlying requests for proposals unclear and inviting unsuitable vendors to bid? Can specific elements in the procurement system be isolated to explain the complaints discrepancy? Is it a problem of human capacity? Or is it a combination of the two?
While the results of this report are not definitive, the evidence in consultation with the panel of public procurement experts appears to suggest that there are problems with the formulation and tendering of requests for proposals and the evaluation of bids that lead to higher numbers of service contract complaints. As in the example of software acquisition above, public procurers may be at an inherent disadvantage by treating procurement as a linear process (i.e. checking boxes) that can be applied in full no matter the type of procurement at hand.
To verify the findings from OPO's data, the researchers collected data from comparative jurisdictions from Canadian provinces and U.S. states. All Canadian provinces and territories (with exception to the Northwest Territories, Nunavut and Prince Edward Island) have provincial ombudsmen (see Table 4). These offices tend to be general ombudsmen managing complaints related to provincial services e.g. health, and report directly to the legislature (unlike the federal OPO). There is no domestic provincial equivalent of a procurement ombudsman in Canada. The majority of provincial ombudsmen do receive procurement contract related complaints (Quebec only accepts complaints related to provincial access to services). Due to low frequency of complaints, most provincial offices do not collect data on procurement complaints. Anecdotally however, the participating provincial offices indicated higher numbers of service contract related complaints when procurement related complaints are received.
Table 4: Canadian provincial ombudsmen offices' anecdotal evidence on procurement contract related complaints
Jurisdiction and comments
Note: Canadian provinces and territories (with exception to the Northwest Territories, Nunavut and Prince Edward Island) have provincial ombudsmen.
Although the Canadian data could not further substantiate the report's initial findings, U.S. states provided a fruitful ground for comparison. The 42 comparative jurisdictions derived from NASPO's 2014 survey were contacted to request procurement complaints data.
|Type of respondents||Amount|
|Total sample population||42|
As depicted in Data table 13, of the 42 contacted jurisdictions, 16 jurisdictions were responsive and provided some form of data and/or information including complaints data and/or definitions of goods and service. From the 16 jurisdictions, 11 specifically shared complaints data of which five tracked the difference between goods contract and service contract related complaints on an annual basis. These five jurisdictions serve as the comparative sample population for this report. Two other jurisdictions produced complaints data differentiating between goods and service but only in total/aggregate form and not on an annual basis. To ensure comparability with OPO's data and to verify complaint levels across the time series, only jurisdictions that tracked goods versus service contract complaint data annually were used.
Data tables 14-18 depict the complaint numbers based on complaint type for Jurisdictions 1-5 on an annual basis.
|Year||Number of goods complaints||Number of service complaints|
Note: As of 2011, procurement of Information technology (IT) services for Jurisdiction 1 was moved to a separate administrative branch.
|Year||Number of goods complaints||Number of service complaints|
|Year||Number of goods complaints||Number of service complaints|
Note: Data not collected in Jurisdiction 3 prior to 2009.
|Year||Number of goods complaints||Number of service complaints|
|Year||Goods complaints||Service complaints|
Three out of five jurisdictions indicate higher numbers of service contract versus goods contract complaints. In aggregate form (i.e. total numbers of goods and service complaint data since 2008) and with two additional jurisdictions that provided data on complaints but not on an annual basis, the collected data indicates a trend similar to OPO's where the U.S. jurisdictions experience higher numbers of service contract versus goods contract complaints since 2008.
|Jurisdiction||Goods complaints||Service complaints|
Given the trends in data from U.S. jurisdictions, OPO's experience of higher numbers of service contract complaints does not appear to be unique. The comparative research also highlighted that some jurisdictions make the distinction between of goods, services, IT and even construction materials. These distinctions have been recognized administratively by separating their procurement functions into different agencies.
In Mississippi for instance, there are four oversight agencies for procurement that have been established in recognition of their differences: one for Commodities (non-IT-related); one for Services; one for Construction and one for IT services. This differentiation underscores the distinct processes and expertise required for acquiring goods versus services. Pennsylvania has underscored the complexity of IT services procurement by delegating the responsibility to the Office of Administration in the Governor's Office, since 2011.
Having experienced difficulties in procuring major IT projects (e.g. canceling or suspending IT projects after years of development, projects exceeding cost and scheduling assessments), California struck a taskforce to study the situation and provide solutions named, the California Task Force on Reengineering IT Procurement for Success. Reporting in August 2013, the task force made various recommendations to improve IT procurement through a more in-depth pre-contract phase, human resources with procurement expertise, changes to bid evaluation, etc. The recommendations from California's taskforce appear to align with this report's suggestion of an incompatibility between the procurement of service and the existing procurement system, including strategy, people, processes and infrastructure.
Observations and next steps
The Procurement Ombudsman's request to assess inordinate numbers of service contract complaints through the lens of complaints data appears to be novel. The findings of this report may, in-fact, reflect the limits of potential analysis to test the two hypotheses, based on available data, a constructed comparative data set and expert consultations.
The following observations are derived from the data collected for this report:
While the numbers of service complaints in Canada have increased beyond their relative increase in volume, causal linkages cannot be established to explain the inordinate number of service contract related complaints filed with the OPO. TBS data indicates that the quantity of goods procurement still outnumber the number of service procurement contracts, yet service contracts continue to register higher numbers of complaints.
Without more information from OPO files or vendor interviews, OPO data is limited to showing higher instances of complaints for contract phase #2. When these results were discussed with the expert panel, there was virtual unison in their interpretation of the findings. Expert commentary coalesced around the idea that human capacity is the most significant driver of service contract based complaints. The procurement process underlying the staff is unsuitable for complex outcome-based service procurement. Political and bureaucratic incentives muddy the environment, leaving accountability for procurement nebulous.
The researchers were asked to better understand why service complaints outnumbered goods. Beyond the analysis undertaken in this report, the data cannot be extrapolated further to explain or prove why there are higher numbers of service complaints. Therefore, suggestions for future study are made based on expert consultations to frame the current findings and propose directions for next studies.
Data to explain why the numbers of service contract complaints are higher can be acquired in different ways, such as internal analysis and assessment because OPO staff has most direct access to files. Future research directions may include: comparative analysis of jurisdictions with high levels of service contract procurement complaints as well as jurisdictions that treat service procurement as a separate issue (e.g. by placing it in a separate department); interviewing current public procurers to assess whether service and goods procurement are viewed differently; and interviewing complainants to better understand the nature of their complaint.
Public procurement is a multifaceted undertaking in a complex environment of political incentives and rules. To procure today is doing more than acquiring an item. With the increasing procurement of various services and the different requirements for outcome or solution oriented service procurement versus input oriented goods procurement, a rethinking of the strategies, processes and human resources required to effectively procure a service may be necessary.
Annex 1: Operationalization of the Procurement Ombudsman's mandate
The Procurement Ombudsman operationalizes their mandate by evaluating the procurement practices of departments and receiving complaints from Canadian suppliers. Departmental procurement practices are reviewed for instance, to ensure that they align with Cabinet policies, that they respect the Financial Administration Act, that the practices adhere to previous reviews of the department's procurement practices, etc.
Procurement contract complaints, that are the focus of this report, tend to fall in two categories: first, to contest a contract award; and the second, is to contest the administration of a contract.
For both types of complaints, Canadian suppliers can file a complaint in writing with the Procurement Ombudsman (PWGSC Act, 22.2 (1)) within 30 days after a contract award has been announced (and up to 90 days after the announcement if an exception is granted by the OPO) (Procurement Ombudsman Regulations 7(1)). A complaint is considered filed when the Procurement Ombudsman receives the required information that includes documentation to establish the complaint, the facts and grounds of the complaint, the potential net profit the complainant could have made and the complainant's cost of submitting the bid (Procurement Ombudsman Regulations, 7(4)). Once the complaint is filed, the Procurement Ombudsman has 10 working days to notify the complainant and the relevant department whether the complaint will be reviewed (Procurement Ombudsman Regulations, 8).
Beyond meeting the minimum information requirements established in section 7 of the regulations, the facts and grounds for the complaint can never have been subject to court proceedings or have been a matter before the Canadian International Trade Tribunal (CITT) and cannot be exempted by any legislation (Procurement Ombudsman Regulations, 9(1)). At any time, a review by the Procurement Ombudsman can be ended for not meeting requirements, if the complainant withdraws the complaint, if the contract award is cancelled or if the Procurement Ombudsman "determines that the complaint is frivolous or vexatious" (Procurement Ombudsman Regulations, 10).
The Procurement Ombudsman typically reviews complaints "under a simple review process", (Procurement Ombudsman Regulations, 11(1)). Relevant departments are given 15 working days to provide comments to the complainant on the file. An expanded review process at the request of the complainant or the department or at the initiative of the Procurement Ombudsman is also possible. Involved parties must be notified immediately if the Procurement Ombudsman is undertaking an expanded review.
For an expanded review, the nature of the complaint, the value of the contract, the complexity of the bid requirements or procurement process, potential impacts that findings and any recommendations could have on government operations and resources must be considered (Procurement Ombudsman Regulations, 11(2)). During an expanded review, the Procurement Ombudsman reviews the complaint file and mediates as the department concerned and the complainant provide comments giving the other party an opportunity to respond in 10 working days (beginning with the department) (Procurement Ombudsman Regulations, 11(2)).
Regardless of whether a simple or expanded review is undertaken by the Procurement Ombudsman, they are required to report on their findings and recommendations no more than 120 working days after the complaint was filed (Procurement Ombudsman Regulations, 14) to the complainant, the minister of the department concerned and the Minister of PWGSC. The Procurement Ombudsman is restricted to recommending compensation if found to be reasonableFootnote 13, since only the Governor in Council can impose fines (Public Works and Government Services Act, 23(2)).
When the Procurement Ombudsman reviews contract administration complaints, the same validation process as with contract award complaints is followed. The review process however, differs slightly. To review a contract administration complaint, the Procurement Ombudsman considers whether the administration was reasonable given the circumstances and whether the parties acted in bad faith (Procurement Ombudsman Regulations, 20). While the Procurement Ombudsman cannot make recommendations to alter the terms and conditions of the contract or to suggest a remedy other than those in the contract (Procurement Ombudsman Regulations, 21), there are alternate means for resolution, namely through alternative dispute resolution process (ADR).
ADR is an option to settle disputes if requested by a party to the contract related "to a dispute on the interpretation or application of contract's terms and conditions" (Procurement Ombudsman Regulations, 23(1)). In order to proceed with ADR, the Procurement Ombudsman must obtain agreement of all parties, with participants agreeing to bear the costs. ADR only proceeds if parties agree to the terms.
Annex 2: Data collection template
|File/Vendor #||Nature of contract
(G or S)
|Industry||Vendor specs||Related department||Nature of complaint||Related phase (1-4)||Resolution? (Y/N)||Date of complaint (MM/YY)||Date of resolution (MM/YY)|
|2012/13-115||(S)||N/A||N/A||National Research Council Canada||Security clearance||1- Pre-
|Out of mandate; considered for future Procurement Practice Reviews (PPR)||12-Aug||Not indicated|
|2012/13-301||(S)||N/A||N/A||Canadian Nuclear Safety Commission (CNSC)||Kickbacks||1- Pre-
|Out of mandate; Privy Council Office (PCO) is conducting investigation centrally||13-Feb||Not indicated|
|Out of mandate; systemic issue, considered for future PPR||13-Jan||Not indicated|
|2014/15-159||(S)||N/A||N/A||PWGSC obo Health Canada||Security||1- Pre-
|Out of mandate; not contract holder||14-Aug||Not indicated|
|Provided Criminal Intelligence Service Canada (CISC) information||15-Mar||Not indicated|
|2014/15-002||(S)||N/A||N/A||Canada Revenue Agency (CRA)||Statement of work (SOW)||1- Pre-
|Did not provide permission to contact department||14-Apr||Not indicated|
|2014/15-491||(S)||N/A||N/A||PWGSC Library and Archives||Not getting work||1- Pre-
|Out of mandate; Supply Arrangement (SA)||15-Feb||Not indicated|
|2014/15-489||(S)||N/A||N/A||N/A||Evaluation and selection plan||1- Pre-
|Provided options to consider||15-Feb||Not indicated|
|2013/14-336||(S)||N/A||N/A||Canadian Human Rights Tribunal||Constructing directly with company's resource||1- Pre-
|Complainant did not come forward in writing||13-Dec||Not indicated|
|2012/13-303||(S)||N/A||N/A||Industry Canada||Restrictive mandatory criteria||1- Pre-
|Out of mandate; OPO was only cc'd||13-Feb||Not indicated|
|2011/12-221||(S)||N/A||N/A||Not indicated||Restrictive requirements||1- Pre-
|2010/11-061||(S)||N/A||N/A||PWGSC||Selection criteria favoured one bidder||1- Pre-
|Out of mandate||10-Jun||Closed|
|2014/15-301||(S)||N/A||N/A||Unknown||SOW and evaluation and selection plan||1- Pre-
|Would be considered for future PPR||14-Nov||Not indicated|
Annex 3: Data request chart
|Year||Total number of complaints|
|Year||Good complaints||Service complaints|
Annex 4: Contacted jurisdictions
- District of Columbia
- New Hampshire
- New Jersey
- New Mexico
- New York
- North Carolina
- North Dakota
- Rhode Island
- South Carolina
- West Virginia
- Footnote 1
In April 1873, it was discovered that then Prime Minister Sir John A. Macdonald was accepting large sums of money from Sir Hugh Allan. In exchange, Sir Allan was assured that he would be awarded the lucrative contract to build the transcontinental Canadian Pacific Railway. A Royal Commission was struck in August 1873 to investigate what came to be known as the Pacific Scandal. With evidence confirming the affair, Sir John A. Macdonald's government resigned in November 1873 and a general election ensued. In 1878, Sir John A. Macdonald's government was reelected and returned to power (Library and Archives Canada (2001), "ARCHIVED - Canadian Confederation", (accessed June 12, 2015).
- Footnote 2
See Government of Canada, (2015), Department of Public Works and Government Services Act.
- Footnote 3
See Government of Canada, (2015), Procurement Ombudsman Regulations.
- Footnote 4
The Governor in Council or the Minister of Public Works and Government Services can assign other duties or functions to the Procurement Ombudsman related to procurement practices of goods and services.
- Footnote 5
See also the Office of the Procurement Ombudsman's Website.
- Footnote 6
Given the complexity of service procurement, it is advisable that the Government of Canada develops a definition of service (or define what it is not).
- Footnote 7
Public Works and Government Services Canada (2015), "1. Supply Manual Glossary", (accessed April 27, 2015).
- Footnote 8
Treasury Board of Canada Secretariat (2013), "Publications – Contracting Data", (accessed April 27, 2015).
- Footnote 9
For jurisdictions other than Canada, definitions were obtained through email exchanges for data collection with state procurement offices.
- Footnote 10
Six consultations were undertaken with academics and practitioners in the field of procurement. For a list of the consulted experts, see the "Experts Consulted" section of the Works Cited page.
- Footnote 11
Adjustments were made to the original hypotheses being tested based on the availability of data and consultations with experts.
- Footnote 12
Jurisdictions that made their procurement complaints data public cited transparency and accountability as mechanisms to improve their quality of work. If the public can track complaints and their related categories, the system may be more inclined to undertake more efficient procurement practices.
- Footnote 13
As per the Procurement Ombudsman Regulations, compensation may be recommended by the Procurement Ombudsman in the following ways:
13. (1) Subject to subsection (2), the Procurement Ombudsman may recommend payment of:
- compensation in respect of the profit that the complainant would have realized if, but for the actions of the contracting department, it had been awarded the contract, which compensation may not exceed the lesser of:
- the net profit that the complainant would have realized, had it been awarded the contract, at the lower of the amount that it bid, if any, and the amount at which the contract was awarded, without including the value of any options or any extensions of the contract, less the cost of submitting the bid, if any; and
- 10% of the value of the contract awarded without including the value of any options or any extensions of the contract, less the cost of submitting the bid, if any; or
- compensation equal to the costs of submitting the bid but that compensation may not exceed 10% of the value of the contract awarded, without including the value of any options or any extensions of the contract.
- Date modified: