Improving Monitoring and Evaluation in the Civic Tech Ecosystem: Developing Theories of Change to Support Future Contribution Analysis

For nearly a decade, civic tech stakeholders have been creating technology supported solutions to civic challenges. Globally, the civic tech movement is rapidly professionalizing but has limited documented evidence of successes and challenges. Comprehensive monitoring and evaluation in the civic tech ecosystem are necessary to create a foundation of knowledge for future initiatives. Monitoring plays a key role in improving services, pivoting approaches, and guiding more efficient resource allocation. Evaluation highlights what is working, what is not working, and critically, why? In a sector that merges data, design, and technology with user-centred principles, monitoring and evaluation in the civic tech ecosystem has several inherent challenges. This paper suggests that a theory-based evaluation approach called Contribution Analysis has the necessary sophistication and agility to support comprehensive monitoring and evaluation to support the growth and sustainability of the movement. This paper applies the early steps of contribution analysis to two Canadian civic tech projects to demonstrate its feasibility.


Introduction
The rapid growth of global interest in civic-focused technology (civic tech) throughout the previous decade can be partially attributed to the convergence of increasing public interest in democratic processes, emerging digital technologies, and ongoing government reforms. Government and civil society efforts to increase transparency, accountability and participation through e-government, government 2.0, open data and open government movements provide a conducive environment for civic tech to flourish. Beginning as a series of volunteer, ad hoc efforts and hackathons in the United States and the United Kingdom, civic tech has evolved over the past decade into a global movement. Community civic tech groups modelled after the Code for America brigades across the United States are part of a larger ecosystem that includes government, community organizations, non-profit organizations, private and social enterprise, academia, and residents (McNutt, 2016). Actors within the civic tech ecosystem employ technology-supported products and processes to address civic issues of concern in communities. The driving goal from the beginning was, "to make government at all levels, more responsive to constituents, more judicious in its use of technology, and less apt to squander taxpayer dollars on failed procurement" (Harrell, 2018:1). Advocates suggest that the civic tech movement could lead to a 'revolution' in government accountability and transparency by changing the way that governments and residents co-produce solutions to civic challenges (McNutt, et al., 2016).
Given the cross-disciplinary nature of civic tech within academia, the scholarly research on the tangible benefits is often buried within diverse disciplines, using discipline-specific language leading to pervasive questions about how to comprehensively evaluate civic tech approaches. This highlights an opportunity to test new monitoring and evaluation approaches that assess how civic tech initiatives make a difference in public challenges. Noveck (2015:144) states, "the first step toward implementing smarter governance… is to develop an agenda for research and experimentation…" Civic tech at its core is about developing smarter governance through combining internal and external knowledge, skills, and experience to tackle complex challenges. Relevant literature strongly emphasize that evaluation needs to be a collaborative process and embedded within initiatives in the ideation stage rather than post-facto (Boyd et al, 2004;Patton, 2006;.
Several important questions can be answered by having comprehensive monitoring and evaluation embedded in civic tech initiatives: How well is a particular civic tech platform or approach working? Did the initiative have direct influence in the change occurring? Are the tangible changes made through the initiative sufficient given the resources invested? There are inherent challenges to monitoring and evaluating within the civic tech ecosystem and the approach needs to be sophisticated enough to adapt to mid-initiative iterations. Currently, these challenges often lead to a to focus on low-hanging fruit like gathering metrics for website hits, clicks, or other forms of digital interactions, and shy away from the complexity of more substantive evaluation. Improving monitoring and evaluation in the civic tech ecosystem is a part of a maturing process necessary to empower people and organizations to be ambitious, do their best work, and consistently strive to do better.
Theory-based approaches to monitoring and evaluation are adept at explaining impact mechanisms in situations where counterfactual-based methods are inappropriate or ill-suited (Delahais, & Toulemonde, 2012). This paper proposes a theory-based evaluation framework called Contribution Analysis as an adequately sophisticated approach to address the complexity of monitoring and evaluation in the civic tech ecosystem. Contribution analysis uses a theory of change that combines a results chain with assumptions about the necessary conditions for the results to be achieved. The goal of this paper is to explore the applicability of contribution analysis to civic tech and provide an example of applying its early stages to relevant projects in Canada.
In the following four subsections: the paper briefly explores the ongoing efforts to define civic tech globally; reviews existing literature on civic tech initiatives and discusses the challenges that limit effective monitoring and evaluation and the importance of addressing those challenges. The paper then introduces two civic tech initiatives in Canada and demonstrates how contribution analysis can be applied in distinct contexts. It continues with a discussion on how contribution analysis can address civic tech challenges through the lens of the previously mentioned examples and finally, provides concluding thoughts.

Defining Civic Tech
Civic tech lacks a universally accepted definition, but there are few points of contention within existing attempts and the divergence is largely found in perspective, scope, and stakeholders (Gilman, 2017). 1 Literature describes civic tech broadly as a "convergence of fields" or a "system" that includes community organizing, social networking, opening data, participatory or collaborative governance that makes use of emerging digital technologies, and resident-to-resident collaboration (Patel et al., 2013:6;McNutt et al., 2016). Its emergence has interdependencies with e-government (Armstrong, 2010), government 2.0 (Uppstrom, & Lonn, 2013), open government (McNutt et al., 2016), open data (Robinson & Johnson, 2016), smart cities (Chatwin & Arku, 2019), and evidencebased policy and practice reforms (McGann, Blomkamp, & Lewis, 2018). The interdependence of civic tech and these fields is reflected in the commonality of the language used to describe them. For example, the International Data Corporation defines civic tech in this way, "Using civic applications, open data platforms, and a range of other technologies, civic tech connects citizens, tourists, and businesses with government services and government workers to make civic engagement and government infrastructure more effective" (2014).
While some definitions primarily focus on technology; including open data, social media, apps, and websites, most recognize the social and relational aspects as intrinsic components of civic tech (Knight Foundation, 2015;Living City, 2012;Purpose, 2016). McNutt et al. (2016:154) state, "It is the social-technology dimension of civic technology that potentially represents the most significant change to established institutions and techniques of local governance." The civic tech ecosystem cannot be reduced to technology mediated interactions between government and public. Gilman (2017) aptly states, "Civic tech used for governance is less focused on finding the next "killer app" than on employing technology in order to achieve more responsive and inclusive governance." In this way, data, design, and technology are a means to an end, tools rather than strategies or endstates (Wilson, & Linders, 2011). At its core, it is a social intervention movement aimed at using data, design, and technology to reinvent government and community relations and rebuild trust (McNutt, et al., 2016). An emerging theme in civic tech is the equipping of individuals, groups, and organizations outside of government to re-establish the power of their collective voice in decision-making (Sifry, 2014). To this end, the oft repeated mantra is that civic tech stakeholders "build with, not for" communities to deliver more contextually appropriate and sustainable solutions (McCann, 2015).
The divergence in civic tech definitions is most evident when exploring its boundaries, including which stakeholders are included in the ecosystem. For example, a Microsoft vice president suggests, "Broadly defined, civic tech ranges from engagement between the city government and its population on social platforms, all the way to enterprise solutions that offer deep government IT problem-solving" (Microsoft, 2014, emphasis added). While there is broad agreement that civic tech includes technology mediated interaction between government and residents, Microsoft's inclusion of government technology and the upgrading of legacy systems in their definition is contentious. Many practitioners insist that legacy systems are 'gov tech' and suggest this is a distinct field from civic tech. For example, a report co-authored by The Knight Foundation and Rita Allen Foundation (2017), suggests that 'gov tech' only becomes civic tech when it is focused on the intended outcome of addressing civic challenges in collaboration with residents. In other words, government technology for the sake of internal efficiency is not civic tech, but technology employed by the government to engage residents would fall within this definition of civic tech.
A second point of contention is The Knight Foundation's infamous inclusion of for-profit entities in the sharing economy such as Airbnb and Lyft into their accounting for investment in civic tech in the United States (2013). The inclusion of private companies from the sharing economy aligns with The Knight Foundation's definition of using technology to enhance how people interact with each other and with the broad definition of civic tech proposed by the Civic Tech Field Guide where civic tech is using technology for the public good (The Knight Foundation, 2013;Civic Tech Field Guide, N.D.). In contrast, organizations like Women in Civic Tech, and the International Data Corporation and definitions in literature, suggest that the absence of a civic challenge in private entities in the sharing economy excludes them from the boundaries of the civic tech ecosystem (Harding et al., 2015;Rumbul, 2016;Gilman, 2017).
While there are inherent risks in permitting undefined parameters for a field of practice, it also allows for creativity and innovation. Articulating a definition is primarily beneficial as a means to an end; if it helps to connect to ideas, people, and resources. This paper proposes a definition of civic tech that includes an ecosystem of stakeholders and centres the use of boundary spanning data, design and technology to support the interaction of democratic institutions with their constituents (McNutt, et al., 2016;Gilman, 2017;Lukensmeyer, 2017). For the purposes of this paper, we define civic tech as a process that uses data, design, and technology to inform, engage and connect residents with government and/or each other to advance civic outcomes. The stakeholders in this ecosystem include public facing digital government institutions, academia and non-profit intermediaries, civic focused private institutions and start-ups and residents engaged in civic participation (See Figure  1). The primary purpose of this definition is to ground forthcoming discussions of monitoring and evaluation approaches in a common understanding of the civic tech ecosystem.

Civic Tech: A review of the monitoring, evaluation, and research literature
Though civic tech is now widely recognized in literature and practice as a distinct field, monitoring and evaluation in its ecosystem is still largely experimental and a relatively new area of study (McNutt, et al., 2016). While the field of civic tech has a lot to learn from adjacent disciplines, the literature reviewed for this paper was limited to scholarship that identified as a being a part of the civic tech ecosystem. Civic tech literature can be broadly grouped into three categories; analysis of secondary data to understand participant demographics of specific platforms, how civic tech intersects with other distinct fields of practice and conceptual frameworks, and the evaluating of the efficacy of civic tech interventions.
Much of the existing civic tech literature aims to answer research questions using secondary data rather than exploring the efficacy of programs through primary data. The distinction lies in the literature's focus on who participated in specific platforms, rather than what made these platforms or processes effective (or not). While there is overlap between research and evaluation, research asks different types of questions that may not relate to the understanding of outcomes or context. For example, in a post-hoc evaluation of a civic action site, E-Democracy.org, used external public data sets to supplement user demographic data to explore the participation of women and people of colour (Lopez, & Farzan, 2017). The study attempted to demonstrate how to overcome a lack of demographic data because many civic action sites do not collect information about gender and race. This approach answers important research questions, but it does not directly attempt to assess the efficacy of the platform.
The second predominant section of literature explores civic tech in relation to a diverse set of theoretical frameworks from different disciplines. As an example, Gilman (2017) discusses how civic tech fits into a paradigm of collaborative or networked governance. Rather than delve into civic tech initiatives, the paper predominantly focuses on how to leverage civic focused technology for collaborative government services and effect change on the traditional relationship between government and the public. Similarly, a study from a human-computer interaction (HCI) frame examined the development of trust and relationships between residents and civic authorities using an empirical participatory design process to provide pragmatic considerations for addressing mistrust inherent in the use of technology mediated crowdsourcing (Harding et al., 2015). There is a large quantum of civic tech related research that is buried within distinct disciplines that is not currently being accessed by practitioners.
The third, and most applicable, section of civic tech literature uses different tools and methodologies to examine the outcome and impact of interventions. This includes practitioner and philanthropic case studies and a limited number of scholarly contributions. For example, Code for America published a case study that documents the formation of Code for the Caribbean Fellowship program (Code for America, 2014). The case study is a comprehensive review of the process of establishing a Fellowship program but includes limited discussion on the approach to monitoring and evaluation. This case study is one example of many provided by the Code for All network, including Code for Canada's early reporting. 2 Similarly, in 2015 the Knight Foundation published a review of numerous civic tech case studies, identifying metrics and indicators used to evaluate success (Knight Foundation, 2015). While this is one of the most comprehensive reviews of civic tech projects, its metrics are primarily quantitative counts of qualitative features and largely absent of any analysis of behaviour change, capacity building or exploration of the necessary conditions that facilitated the desired change.
In contrast, a study on the impact of civic action sites (ie. UK-based mySociety) specifically assesses initiatives in the civic tech ecosystem (Cantijoch, & Galandini, 2016). The evaluators gather unique qualitative and quantitative data to address the question of whether the websites contribute to 'community efficacy'-defined as the belief that as a resident, one can make a difference in the area in which they live. This study takes an innovative approach by applying a mixed-method research design that studies individuals' civic and political activities inside and outside of the site over a 3month period (Cantijoch, & Galandini, 2016). A similar study explored how, and to what extent, tech solutions and the process that produced them, make relevant parts of the local government more responsive to participation by low-income young adults (Network Impact, 2016). An analysis of Community PlanIT and Public Agenda combined analysis of platform data, including user demographics, with interviews with game players and developers (Network Impact, 2016). The evaluation focused on whether their game facilitated meaningful engagement and how the processes in the game helped or hindered deliberation and decision-making. Additional literature highlights research on the impact of neighbourhood scale technology to support civic engagement (Taylor, et al., 2018), a sociotechnical exploration of how the failure to consider organizational forms and institutional arrangements leads to poor results in e-participation initiatives (Harding, et al., 2018), the challenges with crowd-sourcing civic solutions (Uppstrom, & Lonn, 2013;May, & Ross, 2018), and technology-enabled changes in political participation in the last decade (McNutt et al., 2016). Our paper aims to provide practitioners with the tools to contribute to this body of literature through the application of a theory-based approach to monitoring and evaluating civic tech initiatives.

What are the challenges to effective monitoring and evaluation in the civic tech ecosystem??
Demonstrating how interventions lead to the desired change, is difficult in any sector, and civic tech is no different. Evaluators suggest that social change work is akin to a complex and dynamic network of nonlinear and interdependent cause-effect relationships (White, 2010;Dybdal, et al., 2011;Chatwin, & Arku, 2019). The arguments regarding the attribution challenge are well-documented elsewhere 3 , but evaluators are beginning to acknowledge that despite the methodological strengths of counterfactual designs, holding them up as the gold standard in all circumstances is not appropriate or optimal (Cook et al., 2010; Deaton, 2019). Mayne agrees stating, "in complex systems, experimenting with exogenous variables is not possible or not practical: the counterfactual case cannot be established" (2011:4). This is especially true in the civic tech ecosystem where collaboration, iteration, and adaptation are intrinsic features of the sector.
The literature on the civic tech movement highlights several key challenges to effective monitoring and evaluation that stakeholders will need to address. The first oft cited challenge is a lack of resources and capacity; there are limited options to directly fund robust data collection and analysis of outcomes and impact (Dybdal, et al., 2011;The Knight Foundation and Rita Allen Foundation, 2017). Frequently, evaluation is not within the mandate of project leaders and initiatives are operating as start-ups with limited organizational capacity (Sturgill, 2019). Additionally, civic tech initiatives are faced with the challenge of emerging outcomes and undefined long-term changes. While the overarching goals are often clear from the beginning, the changes that result from initiatives are fluid and iterative in response to new learning, changing context and other external factors (Kane, et al., 2017). Further, the actions taken in the short-term may be in service to outcomes and changes that are imperceptible within the timeframe of a typical project, especially philanthropically funded initiatives (Kane, et al., 2017;Sturgill, 2019).
Civic tech is by nature a collaborative and user-centred process. It often involves product development and capacity building working in concert to address presenting challenges. The complexity of initiatives makes it difficult to segment what aspects are working and whether the efforts of individuals and organizations are the direct cause of the change (Koleros, & Mayne, 2019). The complexity of the challenges often requires multiple stakeholders working collaboratively on different components. This leads to difficulty in isolating the direct effect of a singular intervention or attributing change to any one partner within a multi-faceted program (Kane, et al., 2017). Further, interventions often target multiple beneficiaries with some being the recipient of capacity building and some benefiting from improved direct service. Within an intervention there are distinct activities aimed at different actors (Koleros, & Mayne, 2017). Those interventions aimed at end-users need to be sensitive about collecting and using demographic data in evaluations. Given the sensitivity of the users, often collecting this data is not appropriate and proxies must be used instead (Lopez, & Farzan, 2017). Finally, monitoring and evaluation in the civic tech ecosystem is challenged by a lack of trust amongst the stakeholders. This results in reduced transparency in outcome monitoring by government and apathy or adversarial approaches by the public (Harding, et al., 2015).

Why is robust monitoring and evaluation important in the civic tech ecosystem?
Civic tech as a sector has primarily focused on hoped-for-benefits rather than engaging in rigorous monitoring and evaluation (McNutt, et al., 2016). May and Ross (2018) suggest that the dearth of studies assessing the impact of civic tech, from a methodological and outcomes perspective, leads to unanswered questions about the immediate and longer-term impacts of civic tech. The potential of civic tech to improve the relationship between government and the public is compelling, but the shortcomings, failures of technology and overreach of government focused technology companies cannot be ignored. Sturgill (2019) argues, a more robust evidence base is required to demonstrate the cultural change that stakeholders desire to catalyze within government and communities. Organizations embarking on digital transformation of civic facing processes benefit from understanding where to prioritize investments and how to measure the true value of data, design, and technology (Fernandez-Olano, et al. 2015;Gottlieb, & Willmott, 2014). Further, clear evidence of 'success' over time is critical to attract and retain investors, clients, and government or philanthropic contributions. Ongoing budget constraints and scrutiny result in a desire for robust evaluation with a heightened focus on outcomes and impact. Impact Canada acknowledges, "We are aware that program spending is an investment that we are making on behalf of, and directly for Canadians, and we need to place greater emphasis on understanding what differences these investments make in improving the lives of citizens" (2019:iii).
While the importance of robust evaluation is evident for efforts external to government, there is an equivalent importance for civic facing technology within government. Innovations in governance are occurring rapidly resulting in a lack of accumulated knowledge about what works and what does not (Clarke, 2019). Governments are increasingly turning to Key Performance Indicators (KPIs) to 'measure' their digital work. While KPIs hold value for monitoring performance, without being embedded in a larger story that focuses on affecting the desired change, their utility is limited when trying to understand why initiatives are working or not (Kotarba, 2017). Similarly, public facing digital technologies are increasingly provided by organizations external to government. While this challenge extends beyond the civic tech ecosystem and into the 'gov tech' field, it highlights the necessity of comprehensive evaluation as a norm for government digital procurement. The knowledge gaps that lead to government looking for external solutions is making them vulnerable to poor services and getting locked into waterfall development processes and long-term legacy contracts (Clarke, 2019). Recently, governments have begun to test more agile procurement processes with smaller technology companies and civic tech communities, providing an opportunity for early-implemented and ongoing evaluation of externally created products 4 . Governments turning to external stakeholders must learn to be 'smart shoppers' and build their in-house knowledge through comprehensive evaluations of procured services (Morozov, 2016;Clarke, 2019).
Broadly, the importance of monitoring and evaluation for civic tech is in its ability to generate findings that can be used for learning and better decision-making by all stakeholders in the ecosystem and to demonstrate their contribution to improved civic outcomes.

What is Contribution Analysis?
Contribution analysis is a theory-based evaluation approach designed for complex initiatives. The evolution of the model has been outlined in numerous articles and working papers produced by the author of the approach, John Mayne (see: 2001Mayne (see: , 2008Mayne (see: , 2011Mayne (see: , 2012Mayne (see: , 2015Mayne (see: , 2017Mayne (see: , 2018Mayne (see: , 2019. There is a substantial amount of writing on its components and using it as a framework for strengthening the confidence and conclusiveness of theory-based evaluations (Rogers, 2008;White, 2009;Toulemonde et al., 2011;Delahais, & Toulemonde, 2012;Befani, & Mayne, 2014;Koleros, & Mayne, 2019).
Contribution analysis is an iterative approach that begins with what is known at the time and gathers evidence of what has changed, adapting, and evolving the theory of change to increase confidence that the initiative is creating the desired outcomes. In this way, contribution analysis can be compared to the hypothetico-deductive method employed by physicists; it is the making and checking of evidence-based predictions. Determining causation does not come from any singular study, it is through triangulation and the totality of the evidence that leads to confidence in a verdict of contribution or causality (Spears, Ban, & Cumming, 2020). Contribution analysis seeks to verify a model of how the intervention's activities are expected to lead to the desired aims of the initiative, its theory of change (ToC). The ToC shows the anticipated progression from activities to desired long-term change and outlines the causal links and contribution story or stories involved. That is, contribution analysis seeks to tell an evidence-based story about how changes have occurred and what role the intervention played in their realization.
While civic tech is predominantly known for its innovative uses of technology to address challenges, it is equally focused on building capacity and changing behaviours of all stakeholders. There is considerable research in evaluating behaviour change that can be utilized for civic tech. Capability: The individual's psychological and physical capacity to engage in the activity concerned. It includes having the necessary knowledge and skills.
Opportunity: All the factors that lie outside the individual which make the behaviour possible or prompt it.
Motivation: All those brain processes that energize and direct behaviour, not just goals and conscious decision-making. It includes habitual processes, emotional responding, as well as analytical decision-making. Mayne (2018) incorporated this model into contribution analysis, developing the COM-B ToC model (See Figure 2) and demonstrated its applicability to a wide variety of interventions. The approach centers on the development of a robust ToC that posits the pathways of contribution that provide varying levels of confidence that an initiative is leading to the desired results. To clearly understand the ToC used in contribution analysis requires a brief review of the terms and their definitions in this context (Mayne, 2015) 5 : • Impact pathways: Also referred to as a results chain, the impact pathways display the causal pathways showing the linkages between a sequence of steps in getting from Activities and Outputs to Capacity Changes, Behaviour Changes, Immediate Benefits, and the Long-Term Change (See the white boxes in Figure 2).
• Causal link assumptions: The events or conditions necessary or likely necessary for a particular causal link in a ToC pathway to be realized beginning with the Reach and Reaction Assumptions to the Capacity Change Assumptions, Behaviour Change Assumptions, Immediate Benefit Assumptions and the Long-Term Change Assumptions (See the shaded boxes in Figure 2).
• Theory of Change: A combination of the impact pathways and the causal link assumptions with the timeline and external influences identified.
Contribution analysis is designed to answer questions such as: Did the intervention(s) contribute to the desired change in organizational capacity and behaviour change over the intervention period? How did these changes occur? What external influences were involved in the change?
The general steps in Contribution Analysis are as follows (Mayne, 2011): • Step 1: Identify the problem or challenge, and articulate the desired results and contribution questions to be addressed This article focuses on completing steps 1 and 2 to set the stage for developing a contribution analysis monitoring and evaluation plan in the civic tech ecosystem. Initiatives of all scales within diverse contexts benefit from a well-crafted theory of change even if they do not apply the contribution analysis approach. This also creates a foundation for future research to assess the full application of contribution analysis in diverse contexts. The contribution analysis steps are set out in a linear sequence, but in practice there are numerous iterations among them. This is particularly true for complex interventions involving a number of actors and components, as is usually the case for civic tech projects. In these cases, developing a single ToC that tries to lay out the details of the intervention is likely not practical or evaluable. To clarify the interconnectedness and complexity, it is often useful to first set out an Overview ToC showing the big picture and the major pathways to longterm change and then to develop more evaluable nested ToCs showing how the various causal factors are working to bring about the intended impacts (Mayne 2015). These less complicated nested ToCs are considerably more amenable to a contribution analysis, clearly identifying the various contribution stories involved. They also then allow for more specific evaluation questions to be identified (back to Step 1).

Applying Contribution Analysis to Civic Tech
Due to capacity and resource constraints, stakeholders in the civic tech ecosystem have to make decisions about the level of resources to apply to monitoring and evaluation. For small-scale community-based initiatives without sufficient funding, the early development of a ToC provides numerous benefits and requires limited ongoing resources. First, creating a COM-B ToC compels stakeholders to grapple with their rationale, assumptions, and underlying beliefs, and creates a foundation that can be built on at later stages of program maturity or funding. Second, it provides rigour in thinking critically about the necessary conditions for core activities and their interactions with other influences within a complex and dynamic environment (Kane, et. al., 2017). These potential factors include organizations working towards the same goal, or barriers that emerge due to changing socio-political conditions As noted, one significant strength of the approach is its ability to untangle activities and results in a way that explicitly examines multiple actors and influences in change-making initiatives (Kane, et al., 2017). This untangling begins with the early iterations of a COM-B ToC.
The utility of contribution analysis to the civic tech ecosystem is best assessed by examining its application to existing projects. This paper selected two distinct projects out of the many digital change-making initiatives in Canada in an attempt to highlight its adaptability to different stakeholders, scale, and intent. The first example, a Fellowship program from Code for Canada, highlights a digital product-led capacity building intervention from an intermediary (See Figure 1) with numerous activities aimed at accelerating digital transformation within government. The second example, a web-based platform from BetaCity YEG, a community civic tech group in Edmonton, highlights a grassroots approach to ensuring people access the services they need while concurrently changing the way local government interacts with the tech community. Given the broad scope of initiatives that exist within the civic tech ecosystem, these examples are not considered fully representative. However, they serve as two distinct projects that can highlight the applicability of contribution analysis.

Example #1-Code for Canada Fellowship
Code for Canada (C4C) is a national non-profit that supports governments and communities in Canada to harness the power of digital technologies through connecting government innovators with people, knowledge and skills from the data, design, and technology sectors. Functionally, C4C acts as an intermediary in the space between government and the public to facilitate technology assisted solutions to usability and engagement challenges (See Figure 1). The flagship program is a Fellowship that embeds technology and design professionals within government teams at the municipal, provincial, and federal levels. It is a response to the gap between governments enthusiasm for digital innovation and their lack of capacity to lead these government reforms (Clarke, 2019).
The ongoing evaluation of the Fellowship aims to answer the following questions for Code for Canada, government partners, and other stakeholders: • Did the Fellowship contribute to changes in government partner capacity (capability, motivation, opportunity) to incorporate modern digital tools and processes, leading to more effective delivery in their service area?
• Did the increase in government partner capacity impact the ability of users to find/access the desired public service?
• What were the key factors that contributed to/limited these changes?
With government partners and fellows, C4C co-developed and applied a three-story nested program COM-B ToC, to guide the monitoring and evaluation of the effects of the Fellowship program on product delivery and capacity building (See Figure 3).

Figure 3: Code for Canada Fellowship Theory of Change Overview
Given the complexity of the Fellowship program, it was important to segment it into evaluable 'stories' or areas of focus. Figure 2 is a high-level look at how the nested ToC's, identified as Stories 1, 2, and 3, interact within the Fellowship program. Each story represents a detailed and evaluable ToC (described below and visually shown in Figures 3, 4, and 5). The approach of using nested ToC's segments the complexity of the Fellowship program and provides a clear evaluation focus for each stage and actor group involved. The interactions and dependencies between the stories are highlighted in red in Stories 2 and 3. Developing the nested ToC's provides a basis for agreeing on how the Fellowship program is expected to work and the various results and assumptions about the necessary conditions shown, which informs the development of a monitoring and evaluation plan. Given the resources required, it is unlikely that all results and assumptions shown would be monitored. A key component of developing the M&E plan for the Fellowship is making decisions about where to focus data collection efforts.

Story 1: C4C pre-fellowship preparation
The focus of Story 1 (Figure 4) is on the extensive preparation that C4C engages in to find and prepare appropriate government partners, scope the presenting challenge(s) into a feasible project and recruit, assess, hire and train highly qualified fellows. The results chain begins with C4C's ongoing communication strategy (Activities) that aims to engage with potential government partners and potential fellows who are external to government. Government partners (GP) who express interest in participating are assessed on their readiness and resources and supported to develop a clear articulation of their particular challenge. Code for Canada then recruits user-experience researchers, developers, and product managers from the private sector. The results chain ends at having competent and adequately trained Fellows and an adequately prepared government partner (Outputs). Progression on this results chain is largely dependent on C4C providing sufficient training to both the Fellows and the government staff, captured as Output Assumptions in the causal links.

Story 2: Product development and accelerated digital transformation in the government partner
The theme of Story 2 ( Figure 5) is twofold; building a product to address a presenting government challenge and using the product development process to accelerate the digital transformation of the GP. This is a product led process, meaning the capacity and behaviour change occurs through the co-development of a digital product. The development of a specific digital product is a means to achieve this end, but progress towards this change can be achieved without the successful launch of the digital product. The results chain begins with the outputs from Story 1 and progresses into the capabilities, opportunities, and motivations (Capacity Changes). Given the unique situation of each government partner, the capability changes are intentionally left vague. For one partner, user-experience research may be the capability that they need to unlock new behaviours. Alternatively, other partners may benefit from learning how to incorporate agile product development processes into the way they develop products. A feature of the COM-B model is the interplay of the capabilities, opportunities, and motivations. For example, often government reforms focus on building new skills or capabilities. The Fellowship ToC recognizes that if government staff do not have adequate opportunities and motivations to use the new skills, they will not turn into day to day behaviours. The unique characteristics of each context are determined and incorporated into the M&E plan. An example of a necessary condition, or assumption, is that the skills and experience of the fellows translates successfully into the government context. If this assumption does not hold true, there is little chance that the government partner will realize the capacity change desired. In Story 2, the early and ongoing beneficiaries are the staff in the government partner. The ToC is built on early increases in capacity, leading to intermediate changes in behaviour and a conducive environment for improved product development, with the aim of ultimately leading to improved user-experience in the Long-Term Change. This is an example of a singular initiative having multiple beneficiaries, whereby government staffers are the early and intermediary beneficiaries, with a long-term desired impact on residents.

Story 3: Long-term transformation in the government entity
The focus of Story 3 ( Figure 6) is on the proliferation of the digital and user-centred approaches, originating within the GP and spreading to the broader government entity (GE). As the GP builds its capacities and embeds new behaviours into the way it approaches challenges, they are sharing their approach beyond their immediate team and into the adjacent departments, ministries, and agencies. Story 3 begins at the Behaviour Changes stage of the ToC and is dependent on the behaviour change from Story 2 being sustained. The results chain in Story 3 moves beyond the immediate government partner and into the larger government entity within which they operate. While some of this change can and should occur during the Fellowship, the ToC highlights that the long-term change desired will require multiple Fellowships or interventions within the government entity. The necessary conditions represented in the Behaviour Change Assumptions are bi-directional. The original GP must proactively share their new practices outside of their immediate unit and the adjacent staff begin to observe with interest and pursue their own learning. As with Story 2, the early beneficiaries are internal to government and the intended Long-Term Change is the delivery of measurably better services and policy to the public.

Figure 6: Code for Canada Fellowship Theory of Change Story 3 (Long-term digital transformation in GE)
The stories within the ToC serve as a foundation for developing a monitoring and evaluation plan. Specifically, it ensures that all partners have a basic understanding of the goals for the Fellowship and what their respective responsibilities are. Discussions between the stakeholders can illuminate what data is high-priority and what, conditions should be closely monitored for.

Example #2-BetaCity YEG 'YouCanBenefit'
In 2016, the End Poverty Edmonton Task Force, in Edmonton Alberta, recommended that city staff investigate how people can be better supported to access available benefits across all levels of government. The task force recognized that one of the reasons people in need were not applying for services was simply that they did not know what they were eligible for. While there are existing eligibility tools that organizations have for analyzing service eligibility, there are very few organizations that look across different levels of government. BetaCity YEG, Edmonton's communitybased civic tech group, suggested to the city that they were well positioned to hire a developer to create a web-based application that equipped users to quickly search all services and resources across levels of government. E4c, a local non-profit organization that offered an annual service to its clients called 'Make Tax Time Pay', was used as a delivery agent for the web application. On its surface, 'YouCanBenefit' is the result of a civic tech organization, responding to a call from a city task force, working with a local non-profit, to develop an application that helps people in need, access services and benefits that they are eligible for. But there are alternative motivations and intentions as well. The City of Edmonton historically has not had a process to enter into contracts with community organizations like BetaCity YEG. This limits their ability to collaborate on innovative digital responses to meet the emerging needs of their residents. The additional desired benefit of the 'YouCanBenefit' project is to increase the city's ability to enter into new partnerships and create a pathway for small and agile contracts with local technology providers. For the purposes of this paper, with support from early actors in BetaCity YEG's contribution to the project and available documentation, a representative ToC for 'YouCanBenefit' (Figure 7) was developed. Moving forward, and based on the desires of the community, this ToC could be utilized to develop a retrospective evaluation of the pilot, monitoring and evaluation for the ongoing operation of the platform or as a basis for collaborative development of a nested ToC. The example shown is a composite ToC highlighting two actor groups, e4c and city staff, and could easily be segmented for additional clarity and M&E plan development. One of the key aspects of the results chain is the concurrent Capacity Changes within e4c as the user of the platform and the City of Edmonton as the stakeholder responsible for growing and sustaining it. As seen in the Fellowship example ( Figure  5), the early beneficiaries are the staff of e4c and the City of Edmonton. The desired Immediate Benefits include residents seeking benefits and the relationship between the City of Edmonton and smaller technology organizations. Another critical component of the causal links is the Behaviour Change Assumption that the handover of the system from the developer (BetaCity YEG) to the City of Edmonton is successful and the system is maintained by city staff. Success in this transition is not only necessary for the sustainability of the current project, it also impacts the trust the city will place in working with smaller and non-traditional groups for technology development, a stated desire in the Immediate Benefits.

Discussion and Implications for Practice
While unique in both their approach and implementation, the Fellowship and 'YouCanBenefit' programs face similar challenges in monitoring and evaluation. The challenges inherent in monitoring and evaluating these programs and civic tech more broadly are a feature, rather than a bug, and require an evaluative approach that identifies and adapts to, instead of attempting to isolate and control, its complexity. The C4C Fellowship and the 'YouCanBenefit' programs demonstrate what the early stages of contribution analysis look like, when applied to civic tech initiatives. A limitation of this paper, and an opportunity for future research, is the exclusive focus on early stages of developing COM-B ToC's for contribution analysis. However, the creation of robust ToC's provides a way to develop a common understanding of the intervention and insight into how contribution analysis can support organizations to overcome challenges inherent in monitoring and evaluation within civic tech initiatives. Future research is required to test the efficacy of the approach for evaluating its ability assess long-term change.
The Fellowship program aims to accelerate digital capacity in government partners through the process of co-developing a digital product to address a specific need the government has with their user (users can be both internal government staff or external residents and beneficiaries). Though there is an extensive scoping process that precedes embedding fellows in the government team; the extent of the challenge is not fully known until the user-research and product development begin. While the Fellowship is designed around the development of a digital product, it is fully possible for the product to be incomplete or a complete failure and the Fellowship is still an overall success. As the fellows and their government counterparts embark on the process, they will discover gaps in digital skills, policy and norms, and other barriers. The desired outcomes and impacts of the Fellowship are unique to the government partner context and may not fully emerge until the Fellowship is well underway. Only once the government partner is pushing up against system boundaries will the required changes to norms and policies become known. The iterative nature of the contribution analysis framework adapts to this reality of emerging outcomes. The ToC is established with the current information and adjusted when new information is learned. The iterations and changes to the context are all captured within the performance story (Step 6). This transparency allows for external audiences to have a full understanding of the conditions that were necessary for the changes to occur.
The 'YouCanBenefit' project emerged from a group of volunteers from Edmonton's BetaCity YEG civic tech community. The limited resources dedicated to the project were committed to the development of the online platform. One of the key considerations for the 'YouCanBenefit' project was whether it could be done primarily with volunteers and with a very limited budget. This meant that all available funding was directed at the developer with the work being supported by volunteers from the BetaCity YEG community. Evaluating impact was not a priority of the project throughout the inception and design phase. However, as the project was a response to a recommendation from a City of Edmonton task force, there was interest in seeing it continue successfully. Developing an evaluable ToC requires an upfront investment of time but uses limited resources. It can then help in designing the intervention, reaching a shared agreement on how change is expected to occur, and identifying what results would be useful to monitor. If the ToC is comprehensive, then even minimal collecting of early results provides an initial basis for assessing change and contribution.
The Fellowship program is a 10-month agreement between Code for Canada and a government partner. Behaviour change and shifts in norms and policies are not complete within that amount of time. The overarching goal is, sustained change in the relationship between government partners and their end-users. The contribution analysis framework allows for an initial observation of changes made, and exploration of the contributory causes while leaving behind a plausible explanation of how those changes will have a long-term impact on end-users. As Fellowships are completed for more government partners, the legacy theories of change allow for retrospective work at different intervals. Therefore, the ToC is used initially as a program design and observation tool and exists indefinitely as a retrospective tool for desired long-term outcomes and impact.
'YouCanBenefit' has an explicit goal to improve the wellbeing of Edmontonians who require access to government funded programs and services. The pilot project embarked on numerous activities to both address the presenting challenge and improve the way the stakeholders worked together. Initially, it required a volunteer from BetaCity YEG to build an adequate platform so that e4c was better able to serve their clients. This required a focus on capacity building so the volunteers from e4c could use the tool in their day-to-day interactions with beneficiaries. The contribution analysis framework makes explicit all of these distinct activities to enable monitoring (if they occur) and evaluation (what change was made). Further, and perhaps a more distinct advantage of contribution analysis demonstrated in this example is the potential to create nested theories of change that outline, for example, how the 'YouCanBenefit' project was used to build the capacity and change the behaviour of City of Edmonton staff in working with BetaCity YEG and other community entities, to better address community needs.
The Fellowship program operates within an ever-changing political and bureaucratic environment with shifting priorities, resources and supports. Although a frequent, and often fair, critique of government at all levels is that they operate in silos, it is not fully accurate. The government partners of the Fellowship have numerous external influences that impact their work, and their ability to affect change, both directly and indirectly. For example, one of the initial cohorts of the Code for Canada Fellowship was a project with Veterans Affairs Canada 6 . This project benefited from significant leadership and support from the outstanding public servants at the Canadian Digital Services (CDS) 7 . Since contribution analysis is not concerned with isolating and attributing causation to one source, the framework is adaptable to external contributory influences towards shared objectives and impacts. CDS is identified within the ToC as an ongoing external influence for all Fellowship cohorts within the Government of Canada.
Although beyond the scope of this article, the early stages of contribution analysis establish a foundation for determining the level of confidence in an interventions contribution to the desired change, by balancing the optimal methodologically design with the practical limitations and ethical responsibilities (Dybdal, Nielsen, & Lemire, 2011). There is a growing body of literature that focuses on increasing the level of confidence in the causal process through strengthening empirical evidence linking causes and outcomes (Lemire et al., 2012;Schmitt, & Beach, 2015). One frequent approach is using process tracing to study the crucial causal links empirically (Befani, & Mayne, 2015;Schmitt, & Beach, 2015). Process tracing is a method for analyzing empirical evidence within-case to increase the confidence in the existence or non-existence of the causal mechanism (Befani, & Mayne, 2015). In process tracing, evidence is a combination of ethnographic data and other factors surfaced during a contextual analysis. This process is not dissimilar to the weighing of evidence in law: observations, statements or artefacts are weighed against known factors of the circumstances, motives, history, and other contextual influences.
An additional method of increasing the level of confidence in the causal claims from contribution analysis is by using it as a part of a mixed-method approach to evaluation, such as when using comparative groups to assess how much change has occurred. For example, in a study on police reform in the Democratic Republic of the Congo, Koleros and Mayne (2019) demonstrate this approach by embedding a quasi-experimental difference-in-difference (DiD) approach within a contribution analysis framework. Nesting the DiD design within a theory-based evaluation is an approach that increases confidence in how interventions influence a desired outcome (Koleros, & Mayne, 2019).

Conclusion
As a nascent movement, with limited precedence for comprehensive monitoring and evaluation, it is unsurprising that the civic tech ecosystem, is still seeking appropriate methods. Currently, the stories of impact are largely normative ideals of the benefits of technology to address civic challenges, or quantitative measures, like datasets shared or daily active users. Normative statements or narrow quantitative measures are not conducive to making a compelling case for investment and support of civic tech and digital government initiatives. A more robust evidence base is required to demonstrate the digital, design and technology cultural change that civic tech stakeholders desire to facilitate within government and communities (Sturgill, 2019). Any method, whether an experimental methodology like randomized control trials or a purely qualitative design, has advantages and drawbacks that are more or less prominent depending on the circumstances within which they are applied (Deaton, 2019).
Contribution analysis is most effective when it is a collaborative process; the ToC is best developed by all relevant stakeholders, including potential beneficiaries when possible. Engaging key stakeholders in the problem identification and solution design is fundamental to the civic tech ethos. Collaborative design of the ToC can ensure a consistent understanding of the challenge to be addressed and help identify gaps and areas with insufficient evidence to validate the contribution claim. This does not mean the process is straightforward or easy. One member of a group from Civic Tech Toronto, in relation to developing a ToC, stated, "…the process was messy and sometimes confusing". 8 Beginning with a generic ToC model provides stakeholders with a place to begin dialogue on the desired results and existing assumptions. Within a complex, multi-actor environment, it is important to acknowledge the individual, organizational and coalition contributions to the outcomes. Contribution analysis provides a systematic consideration of the influence of each actor (Kane, et al., 2017). This paper proposes that contribution analysis is an approach that is sufficiently adaptable to address the inherent challenges and to accommodate the M&E needs of the civic tech ecosystem. However, that does not mean that it is without challenges or that it adequately addresses all of the problems inherent in comprehensive monitoring and evaluation. It is important to note that contribution analysis is not a quick and easy approach to addressing causality. Its advantages are that (1) it can be used to make causal inferences when experimental and quasi-experimental designs are not possible, (2) it explores why and how an intervention has influenced change, (3) it allows for making causal inferences about the intervention without necessarily examining external causal factors, and (4) it addresses cases where there are numerous causal factors at work by assessing contributory causes leading to credible contribution claims.
While the paper proposes contribution analysis as a suitable framework for civic tech, it is not without its challenges and limitations. First, building a comprehensive and plausible ToC requires insight, comfort in ambiguity, thoughtful deliberation, and patience. In complex settings such as many civic tech interventions, there is the need to find a way to unpack the program using nested ToCs to be able to highlight key stories about how interventions are affecting different stakeholders, beneficiaries and at what scale. Typically, there is a need for significant primary and secondary data collection. As discussed, collecting data outside of the partner organization to assess the programs impact on beneficiaries is resource intensive. A clear limitation of contribution analysis is that it does not endeavour to estimate the quantitative amount of change brought about by the initiative, rather it explains how the intervention contributes to impacts of interest. 8 See https://medium.com/@liamilito/clarifying-impact-lessons-learned-at-civic-tech-toronto-d3d3b85318d7#.6cdsnw297 Despite these challenges, contribution analysis is adaptable to the story that each individual or organization wants to tell, whether that is a story of scaling, degrees of change within a community, or comprehensively addressing one specific civic challenge. It assists in answering the questions; How do we know that civic tech initiatives are succeeding? Did this initiative contribute to the change? The value of contribution analysis is in its systematic and iterative approach to a complex causal problem. It allows initiatives to set broad and aspirational goals without the concern of what they can feasibly achieve in the short-term or prove causation for. It is also not tied to methodological prejudice, rather it provides a framework within which to gather evidence that improves confidence in an initiative's contribution to the desired impact. Contribution analysis adapts to context and with a plausible theory of change, evaluators can decide on which methods best suit their desired outcomes.