Figures
Abstract
Few studies have examined the sustainability of family planning program outcomes in the post-program period. This article presents the results of a natural experiment where the Nigerian Urban Reproductive Health Initiative Phase I programming ended in early 2015 and Phase II activities continued in a subset of cities. Using data collected in 2015 and 2017, we compare contraceptive ideation and modern family planning use in two cities: Ilorin where program activities concluded in 2015 and Kaduna where program activities continued. The results demonstrate that exposure to program activities decreased in Ilorin but for those individuals reporting continuing exposure, the effect size of exposure on modern family planning use remained the same and was not significantly different from Kaduna. Modern family planning use continued to increase in both cites but at a lower rate than during Phase I. The results are useful for designing family planning programs that sustain beyond the life of the program.
Citation: Speizer IS, Guilkey DK, Escamilla V, Lance PM, Calhoun LM, Ojogun OT, et al. (2019) On the sustainability of a family planning program in Nigeria when funding ends. PLoS ONE 14(9): e0222790. https://doi.org/10.1371/journal.pone.0222790
Editor: Philip Anglewicz, Johns Hopkins Bloomberg School of Public Health, UNITED STATES
Received: April 29, 2019; Accepted: September 7, 2019; Published: September 26, 2019
Copyright: © 2019 Speizer et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: The authors collected publicly available data from The Carolina Population Center. The dataset supporting the conclusions of this article can be requested at the following URL: https://data.cpc.unc.edu/projects/14/view. Users are asked to complete and sign a data use agreement that requests brief information about intent of use prior to receiving approval to download the data. The authors confirm they had no special privileges or access to the data.
Funding: The authors acknowledge funding from a grant awarded to the Carolina Population Center (CPC) at the University of North Carolina at Chapel Hill by the Bill & Melinda Gates Foundation. The authors also received general support from the Population Research Infrastructure Program through an award to the Carolina Population Center (P2C HD050924) at The University of North Carolina at Chapel Hill. The contents of this article are solely the responsibility of the authors and do not necessarily represent the official views of CPC or the Bill & Melinda Gates Foundation.
Competing interests: The authors have declared that no competing interests exist.
Introduction
There has been a growing emphasis on understanding the meaning and measurement of program sustainability in public health interventions [1–4]. The definition of sustainability varies across studies, but the focus is often on program components and outcomes that were developed or achieved in an earlier phase of a project and are maintained after the initial or focused funding period ends[5,6]. Sustainability is also called institutionalization, maintenance, durability, and continuation [5]and a number of studies have developed conceptual frameworks to inform the measurement of sustainability and/or scale-up of programs [1,3,7–9].
While much of the literature on the sustainability of health programs comes from developed countries[4,5], a growing body of work considers the sustainability of public health interventions in sub-Saharan Africa [1]and Asia [10–14]. A recent systematic review of the sustainability of health programs from sub-Saharan Africa found 41 studies that met the inclusion criteria that included: a) examining the status of the intervention during or after the initial funding ended; and b) examining continuation of the intervention. Many focused on communicable diseases with HIV and AIDS and malaria the most common; none examined the sustainability of a family planning (FP) program [1]. Studies of sustainability often use qualitative data and focus on systems-level outcomes and processes including whether program implementation continues at the school, health facility, or organizational level where it was initially introduced, and whether the program was implemented with fidelity [4,6,10,12,15,16]. A smaller number of studies examine whether individual-level behavioral outcomes are sustained and what factors lead to sustained behaviors.
Interestingly, Scheier’s systematic review on the sustainability of health-related programs in developed nations [5] indicated that about 80% of interventions that measured continuation demonstrated sustainability of at least one component in at least 60% of their sites after the program officially ended. Sustained interventions tended to have local ownership, were flexible to the program setting, and had visible benefits to the beneficiaries at all levels [5]. Reviews from developing countries have identified similar factors associated with successful scale-up and sustained delivery of programs at scale including factors relevant to ownership or leadership at the political, community, and organizational levels; plans for long-term financing; the program’s flexibility, adaptability and fit within the target organization; and the extent of service delivery capacity created for long-term implementation [1,7,17].
In the field of FP, one of the few examples of a rigorous examination of sustainability involved the Promoting Change in Reproductive Behavior (PRACHAR) program in Bihar, India that sought to delay marriage, delay first birth, and encourage contraceptive use within three months of a birth [18]. Evaluations of the PRACHAR program showed large impacts on key outcomes [18,19]. In 2013, Jeejebhoy and colleagues [20] used cross-sectional data to examine the sustainability of impacts in intervention and matched comparison sites four-years post-intervention. Because of differing study areas over time, the sustainability study used a post-test only design. It found significant differences between the intervention and comparison areas on awareness and use of FP, but no long-term effects were found on fertility or marriage outcomes. Interestingly, long-term effects on FP awareness and use were also found among women only indirectly exposed in the intervention communities [20], suggesting enduring spillover or diffusion effects to non-participants.
Another analysis of PRACHAR program sustainability examined whether young people who participated in a three-day training from intervention communities differed from matched young people from comparison communities four years after the training [21]. The authors demonstrated significantly higher awareness of sexual and reproductive health matters as well as better contraceptive practice among the intervention group exposed to the training as compared to the comparison group, however, this might relate to self-selection into the program as indicated by the authors [21]. Notably the primary outcomes of the program, including timing of marriage and timing of childbearing, did not differ across groups [21].
Another example of a FP program that had a follow-up evaluation of program sustainability is the Better Life Options (BLO) Program in India that examined intervention and comparison areas with another post-test only design [22]. The BLO program found that 1–4 years after program implementation, young women (ages 15–26) in the intervention community had more awareness of HIV, an older age at marriage, better school and employment outcomes, and higher contraceptive use, among other outcomes, than young women in comparison areas [22]. It is notable that the few sustainability studies of FP programs found were from India and tend to use post-test only and cross-sectional designs. There is a lack of rigorous evidence on FP program sustainability for sub-Saharan Africa.
This paper begins to fill this gap by examining the sustainability of the Nigerian Urban Reproductive Health Initiative (NURHI) program on FP attitudes and behaviors two-years after the initial program end date. It takes advantage of the fact that the program was completely terminated in one site and continued in a modified form in the other site. This study is unique not only because it contributes to the knowledge base about sustainability of FP programs but also because it uses novel data sources and methods to inform the key outcomes of ideation toward FP and modern contraceptive use. The results provide crucial evidence regarding the sustainability of the NURHI program on FP attitudes and behaviors as well as whether there are diminishing returns on the continuation of the same activities over the longer-term in the two settings: where the program continued and where the program terminated. The hope is that this study will be important for informing strategies to spread and scale-up FP programming in urban Nigeria and elsewhere in sub-Saharan Africa.
Methods
Data
The Nigerian Urban Reproductive Health Initiative (NURHI), funded by the Bill & Melinda Gates Foundation (BMGF) and implemented by the Johns Hopkins Center for Communications Programs, aimed to improve FP access and modern contraceptive use in select urban areas through comprehensive demand and supply side programming. The NURHI program utilized demand generation activities to encourage interpersonal discussion about FP, reduce barriers, myths, and social stigma, and increase approval of FP methods. Strategies included community-level outreach events, distribution of information, education, and communication (IEC) materials at public and private health facilities and in the communities, and through mass media, including television and radio programs. In addition, the NURHI program worked to improve the supply of FP services by training providers on provision of quality care and new methods; ensuring commodity supplies; and improving the overall clinic environment in target, high volume facilities. Phase I (2009–2014) of the NURHI FP program was implemented in six cities: Abuja, Benin City, Ibadan, Ilorin, Kaduna, and Zaria. For details of the NURHI program see Krenn et al. [23]. Following completion of Phase I activities, a subsequent Phase II of the NURHI project began in 2015 and continued in Oyo state (incorporating Ibadan) and Kaduna state (incorporating Kaduna city and Zaria) and expanded to Lagos state; program activities were ended in the other cities. Of note, Phase II of the NURHI program was less intense at the city level given lower funding levels and the focus on state-level programming rather than only cities.
The Measurement, Learning & Evaluation (MLE) project (also funded by BMGF), conducted the evaluation of the NURHI Phase I program with Data, Research and Mapping Consult (DRMC) and the National Population Commission as the local implementing partners [24]. The MLE project applied a hybrid design, collecting both longitudinal and cross-sectional survey data to evaluate the direct impact of NURHI on contraceptive use and behavioral and attitudinal norms in the six study cities. At endline (2015), a two-stage sampling design was used to identify a representative sample of women from each city. For the first stage selection of enumeration areas that could serve as primary sampling units, we used a sampling frame based on the 2006 Nigeria census that provided enumeration area (EA) population estimates that were the most accurate possible and thus allowed for the most efficient selection of representative samples of households. Census EAs were grouped into primary sampling units (PSUs) for random selection in the first stage of sampling. In the second stage of sampling, a household listing was undertaken and once complete, 41 households were randomly selected from selected PSUs. All women of reproductive age (15–49 years) living in the selected households were eligible to participate in the survey following informed consent. All women were asked to provide verbal consent to participate; study interviewers signed the consent forms following receipt of verbal consent. In 2015, we surveyed a total of 2,801 women from Kaduna, and 940 women from Ilorin.
In 2017, another cross-sectional representative survey was conducted by DRMC in both Kaduna and Ilorin to examine the sustainability of the NURHI program two-years after program operations terminated. We compare two cities: one where program activities concluded at the end of Phase I (Ilorin) and one where activities continued in Phase II of the NURHI program (Kaduna). All clusters surveyed in Kaduna and Ilorin in 2015 were included in the 2017 survey. As part of the 2017 data collection, we undertook a census of all households located in study clusters in Ilorin and Kaduna. All women ages 15–49 were eligible for interview after providing informed verbal consent. A total of 4,673 women from Kaduna and 2,209 women from Ilorin were surveyed in 2017. The study protocol and all consent procedures and consent forms for both the 2015 and 2017 surveys were approved by the Institutional Review Board at the University of North Carolina at Chapel Hill and by the National Health Research Ethics Committee of Nigeria (NHREC) in Nigeria.
The census of all households in study clusters in 2017 allowed us to match households and women from 2015 (that included a sample of households) to create a longitudinal sample of women from the two cities. During 2017 fieldwork, households that were interviewed in 2015 were identified and flagged for matching using information on the name of the household head and the household composition. In addition, a small number of households (n = 22) were matched after data collection using stringent criteria for matching (contact first author for details of these additional matches). Matching of women within matched households rested primarily on age and consistency of reported parity as of the 2015 round. A total of 1,535 (1,097 from Kaduna, 438 from Ilorin) women were matched across the two time periods for longitudinal analyses. It is worth noting that data are also available from the larger cross-sectional samples in 2015 and 2017 for the two cities (contact lead author for additional information). Multivariate analyses of the longitudinal sample are used to estimate NURHI’s continuing effects after Phase I activities formally concluded in 2015.
Variables
At both time periods, all surveyed women were asked questions on fertility and family planning behaviors as well as information on exposure to the NURHI program, among other things. Questionnaires used came from earlier rounds of the study (baseline and midterm) and most of the variables included in this paper were validated as part of the impact evaluation of the NURHI program [24]. The first outcome of interest is modern contraceptive use. For this study, modern use includes use of injectables, intrauterine devices (IUD), implants, pills (emergency and daily), condoms (female and male), and sterilization (female and male). We excluded lactational amenorrhea method (LAM) from our measure of modern use because it was not one of the methods promoted by NURHI, and it is often mis-reported on surveys [25].
The second outcome of interest is women’s FP ideation, a representation of knowledge, values, beliefs, and communication related to FP and modern contraceptive use. Because of the large number of factors that relate to ideation, principal components analysis (PCA) was used to construct a single, continuous ideation index. The index is based on the same set of underlying factors used by NURHI to examine ideation in an earlier analysis [26]. The NURHI program theory of change identifies changes in a person’s beliefs, ideas, and feelings (ideational factors) as a key intermediate step to influencing contraceptive use over time [23]. Therefore, ideation is used as an outcome as well as an explanatory variable in the analyses that follow. The following binary variables were used for the ideation PCA: knowledge of more than six (median) modern contraceptive methods; reject more than six (median) myths regarding risks associated with modern contraception; discussed family size with partner in the last six months; discussed FP with partner in the last six months; discussed FP with significant other besides partner in the last year; perceived at least one non-spouse significant other supports her use of FP; perceived that all or most women in her community use FP; self-efficacy score above 4 out of 6 (median) that captures a woman’s comfort with obtaining or discussing FP with her partner and others; reportedly does not need permission to use FP; would recommend FP to someone. To make the comparison of ideation over time meaningful, we first estimated the PCA for the full 2017 cross-sectional sample from Kaduna and Ilorin. We used the full cross-sectional sample because it is representative of the population. The factor loadings from the first principal component were then applied to the full 2015 cross-sectional data to generate the continuous ideation index values for Kaduna and Ilorin in 2015. The same factor loadings were used over time to standardize the measure of ideation (see S1 Table for the factor loadings). Besides being an outcome variable, the ideation index was also treated as an endogenous explanatory variable in the modern contraceptive use model.
The key explanatory variables for these analyses are exposure to NURHI program related factors. The NURHI program had many demand side activities and our analyses focus on indicators for eight of them. In earlier longitudinal impact evaluation analyses, we estimated the causal impact of each of these activities separately (i.e., at the end of the NURHI Phase I program [24]). However, the sample size was much larger in that analysis and was spread across six cities. In the current work, we have data from two cities, Kaduna and Ilorin, and a smaller sample size. In addition, we wanted to test whether exposure effects were different for the two cities and different across the two time periods. To do this, we would have needed to create many interaction terms which would make interpretation of the results quite difficult. Therefore, for our first set of multivariate analyses, we use principal components analysis to construct a single, continuous exposure index. The index represented reported exposure to the following: NURHI television dramas and advertisements promoting FP; NURHI radio dramas and commercials promoting FP; community outreach presentations on FP at familial events including naming ceremonies and weddings; received SMS promoting FP; and exposure to any of the following media with NURHI FP or general FP slogans including provider badge with NURHI messages, NURHI logo, NURHI message cards, and billboards. The same approach used for standardizing the PCA of ideation over time (described above) was used for standardizing the PCA of the exposure variables. In a second set of analyses presented below, we examine grouped exposures to determine the marginal effects for various program components separately on modern contraceptive use.
In the NURHI impact evaluation, supply-side variables were included [24]. In this analysis, the supply-side variables including distance to the nearest facility, as well as characteristics of facilities within 1km of a woman’s residence such as whether the facility had recent FP stockouts, was recently renovated, or participated in NURHI training, were tested and were not significant and therefore were dropped from the final models. This insignificance of supply side factors is a common problem when trying to measure supply side effects in urban environments where services are densely packed, and one does not know the actual facility that the respondent may choose to visit. In a companion facility level analysis, we found that NURHI facilities had better quality in 2017 than non-NURHI facilities and quality was related to the number of family planning clients that visited a facility [27].
A migration related variable was also included as an explanatory variable in the models. We had no a priori belief about the direction of the effect of this regressor—whether a woman travelled to another city or town in the last year could increase or reduce NURHI exposure and contraceptive use. Control variables included age, education, union status (married or living with partner), religion, primary language spoken at home, and a household asset score (see Table 1 for classifications of these variables). The household asset score represented the sum of reported household materials and assets including cement walls, electricity, piped water in the home/yard, flush toilet to septic/sewer system, refrigerator, vehicle, and own home structure.
Conceptual framework
Fig 1 provides the conceptual framework for how FP programming can be sustained beyond program termination. Prior research has demonstrated that Phase 1 of NURHI programming was related to improved ideation and modern contraceptive use [24,26]; this is demonstrated on the left side of the framework. Moreover, literature on the diffusion of family planning programming has demonstrated that program effects can spread beyond initial adopters to other individuals in the city through the process of social learning [28]; these diffusion effects are shown with dotted lines. Also shown on the figure is the Phase 2 programming that takes place only in Kaduna. In this case, ideation and use may be higher in Kaduna than Ilorin because of this additional program exposure. Our analyses can measure the levels and trends in ideation and modern contraceptive use under these scenarios–Ilorin, where the program stopped in 2015 but diffusion effects may result in continued increases in ideation and contraceptive use in 2017, and Kaduna, where diffusion and continued programming both influence these outcomes.
Statistical methods
We exploit the longitudinal sub-sample to estimate models that control for the potential endogeneity of exposure or ideation caused by correlation with unobservables at both the community and individual levels. We use two dependent variables: modern contraceptive use estimated using a logit framework and the ideation index using an extension of the linear regression model. We illustrate our methods using the more complicated, nonlinear logit model:
(1)
where the dependent variable is the log odds that woman i (i = 1,2,…,Nj) from community j (j = 1,2,…,M) at time t (t = 1,2) is a current user of modern contraception at the time of the survey. Cityij is a dummy variable indicating city of residence for woman i from community j, Eijt represents program exposure or the ideation index, Xijt represents individual level control variables such as age and education that are assumed to be exogenous, and μij represents time invariant unobservable individual and community level variables such as an individual’s level of motivation to limit family size or attitudes of community leaders towards modern contraceptive use. We do not specify separate time invariant errors for community and individual since, by design, we only have individuals from the 2015 survey who were found to still be living in the same community in 2017 and our estimation method is a variation of fixed effects because we suspect that program exposure is potentially correlated with this time invariant error.
The specific variant of fixed effects that we use is the correlated random effects model [29–31] which involves modeling the time invariant error as follows:
(2)
where
and
denote averages at the individual level of the two observations for the X and E variables respectively. We then substitute (2) into (1) to get:
(3)
In the linear case, it is well known that if we estimate (3) using random effects or feasible generalized least squares (GLS) that the estimates of α4 and α5 will be identical to the fixed effects estimates or within estimates [32] and a joint test that the λ's are equal to zero is a robust version of the Hausman test of the null hypothesis of no correlation between program exposure and the time invariant error term. In this logit form, we estimate Eq (3) by random effects logit. Note, unlike standard fixed effects methods, this hybrid approach to modeling that combines random and fixed effects permits us to obtain random effect estimates for regressors that do not vary with time such as the city dummy.
Results
Descriptive statistics
Table 1 provides the descriptive characteristics for the longitudinal data from 2015 and 2017. As a reminder, the city of Kaduna is where program activities continued after 2015 while program activities in Ilorin ceased in March of 2015. Table 1 demonstrates that, as expected, the women were somewhat older in 2017 than 2015 (i.e., mean age is two years higher in 2017). Further, in 2017, women from both cities were more likely to have traveled to another city than two years earlier, and the women were slightly more educated at the later date. Comparing across the cities, a slightly greater percentage of women in the sample from Ilorin were in union than in Kaduna. Further, women from Ilorin were more educated than women from Kaduna. Women from Kaduna were predominately Hausa speaking while women from Ilorin were predominately Yoruba speaking.
Table 2 presents descriptive statistics for the outcome variables. In Kaduna, modern contraceptive use significantly increased from 14.86% to 21.69% (p<0.001). In Ilorin, modern contraceptive use also increased from 22.89% to 27.07% (p = 0.049). As a percentage of modern method use, there was no significant change in method mix (long-acting methods vs. short acting methods) between 2015 and 2017.
Table 2 also includes descriptive statistics for ideation and exposure. In Kaduna, there was a significant increase in the ideation index from 0.13 to 0.46 (p<0.001) between 2015 and 2017. Certain ideation factors significantly increased including knowledge on FP; discussion of family size with the respondent’s partner in last 6 months; discussion of FP with a significant other besides the partner in last year; perceived support for FP; does not need permission to use FP; and would recommend FP to someone. The exception was for discussion of family planning in the last 6 months which significantly declined between 2015 and 2017 (p<0.001). The results for Ilorin show a different pattern. Overall, the ideation index in Ilorin did not change significantly between 2015 and 2017; however, a few of the items significantly improved including reject myths; discussion of family size with partner; does not need permission to use FP; and would recommend FP. Other items significantly declined over the period including discussion of FP in the last six months and perceived support for family planning. Notably, ideation was higher in Kaduna than in Ilorin in 2015.
The key independent variable of interest is exposure to NURHI demand generation activities. In Kaduna where the program continued, women generally reported significantly greater exposure to the NURHI activities in 2017 relative to 2015 (see Table 2). Conversely, while Ilorin had higher exposure in 2015 than Kaduna, over the follow-up period, exposure in Ilorin declined significantly; this is not surprising since the program ended in 2015. Overall, the mean exposure value in Kaduna goes up over time (from -0.34 to 0.27) while the mean value goes down in Ilorin (from 1.25 to -0.03). Finally, the bottom of Table 2 provides grouped exposure measures. We see that for Kaduna, the mass media (radio and television), print media (billboards, provider badge, and cards), and community (outreach and SMS) measures all significantly increased between 2015 and 2017 whereas in Ilorin, they all significantly declined in the same period.
Multivariate results
Table 3 presents results for three sets of models. For all three regression models, a joint test of the null hypothesis that the λ's for Eq (3) are equal to zero was strongly rejected at all standard significance levels. This means that the use of correlated random effects to control for the possibility of correlation between individual and community level unobservables and NURHI program related variables was necessary and standard methods would yield biased estimates for program exposure. This also justifies our use of the longitudinal sub-sample rather than the larger cross-sectional samples.
Model (a) examines the relationship between exposure to NURHI program components and ideation, controlling for demographic factors; model (b) examines how ideation is related to modern method use; and model (c) examines how NURHI exposure relates to modern method use. Table 4 provides the control variables from Model (c) which are similar across all three models. In preliminary models (S2 and S3 Tables), we used interaction terms to test for differing effects of exposure and ideation by city, by year, and by whether the respondent was Muslim. We also used three and four way interactions to capture differing effects with all possible combinations. Overall, the interaction terms were not significant, which was unexpected. We expected to see significant interactions between the 2017 dummy and living in Kaduna reflecting continued exposure to the NURHI program in this site. Based on the preliminary results that were generally not significant, we present below the results without the interactions.
The multivariate results (Table 3) for model (a) show a strongly positive and significant effect for the Kaduna dummy suggesting that all else being equal, ideation is higher in Kaduna, the city where the program continued. We also see that ideation continued to increase in 2017 and exposure to NURHI program variables had a strong positive effect on ideation.
In model (b) with modern contraceptive use as the dependent variable, we see that the sign of the Kaduna dummy is negative (p<0.10). Further, the effect of the ideation index is strongly significant and positive. From model (a) we know that the Kaduna dummy has a strong positive effect on ideation and thus the ideation effect is likely capturing some of this higher ideation in Kaduna. Modern contraceptive use continued to increase in 2017, and recent travel had a significantly negative effect on modern use. We have no a priori expectations for the sign of the travel variable since women who move between cities may be more likely to use contraception if they have greater access to methods from their travel or may have less need for contraception if they are moving around and are less likely to be having regular sex.
Finally, model (c) shows the direct effects of exposure on modern contraceptive use. The effects that are positive and significant are the time dummy and the exposure index where over time, use is higher and women who have greater exposure are more likely to use. The Kaduna dummy has a positive point estimate; however, it is far from significant. The effect of recent travel is basically the same as model (b).
Results for the control variables from the three sets of models are presented in Table 4. We present control variable results from model (c) which examines the relationship between NURHI exposure and modern contraceptive use since this is the primary model of interest. Generally, the control variables had the expected effect such that older women were more likely to use FP and Muslim women were less likely to use FP.
The major conclusion one can draw to this point is that exposure to NURHI programs has continued to have strong positive effects on both ideation and modern contraceptive use and that ideation has strong positive effects on modern contraceptive use and so it is a major pathway through which exposure has its effect. However, the exposure index, while it was quite useful when we tested for many interaction effects, does not allow us to examine each program component separately. Since we found almost no significant interaction effects and a joint test of the interaction terms failed to reject the null that the effects were jointly zero (see results including interactions in S2 and S3 Tables model (c)), we now examine the effects for groups of program exposure components as well as a measure of the effect of a small set of them simultaneously.
Marginal effects for program components
To examine specific marginal effects for each of the program components, we replaced the exposure index with seven exposure measures and then re-estimated model (c) (results not shown). Note that the logo exposure measure was dropped because it tended to be found as part of many of the other exposure measures. We then calculated marginal effects for each program component. Since all exposures are binary, the marginal effect is the difference between predicted modern contraceptive use with the exposure set to zero and then set to one and all other variables kept at their actual empirical values. The marginal effects for individual exposures were not significant probably due to collinearity. However, when we examine the effect of groups of similar exposures (mass media; print media; and outreach), we see some significant effects (Table 5). Print media, which captures exposure to the NURHI logo on one or more of the following: billboards, cards, or provider badge, has a positive effect on modern contraceptive use. Exposure to mass media which includes NURHI television and/or radio programs also has a positive effect on modern contraceptive use. Outreach, which includes receiving FP messages at life events or SMS promoting FP, was not significant. Of note, mass media and print media activities may be sustained after the program leaves by remaining available (e.g., billboards not taken down) or through television programming that spreads beyond the state-level setting (e.g., through national campaigns or national advertisements). We also calculated a marginal effect for exposure to both mass media (TV and/or radio) and print media (billboard and/or cards and/or badge) at once and we see a substantial impact on modern contraceptive use. However, it should be noted that in 2015, 78% of all respondents were exposed to either mass media or print media, while 33% were exposed to both. In 2017, the numbers were 87% and 30% respectively.
Finally, the marginal effect for the 2017 dummy variable is also presented in Table 5 and we see that it has a significant and positive effect on modern contraceptive use. This is after controlling for exposures and demographic characteristics. One possible explanation is diffusion, as hypothesized in the conceptual framework. An analysis of Nigeria data found significant effects of social learning on contraceptive use as measured by community level contraceptive use [33]. A similar, mechanism could be at work here, but we do not explicitly try to model social learning in this work.
Discussion
This paper takes advantage of a natural experiment that occurred in two cities in Nigeria where the NURHI FP program received funding to continue operations in one city and its surrounding state (Kaduna) and not in another (Ilorin). We use data gathered in 2015, which coincided with the termination of the program in Ilorin, and in 2017, two years after termination, to see how the two cities fared.
As we point out in the introduction, there have been very few studies that have examined the sustainability of FP programs anywhere in the world and none that we are aware of from sub-Saharan Africa that had data at the time of program termination and data from a later point in time. We not only have this type of data for two repeated cross-sections, but we also have a subset of the sample where we have the same individuals at two points in time and so we are able to use the powerful tools associated with longitudinal data in this analysis.
An interesting descriptive result is that respondents in Ilorin reported continued exposure to the different aspects of the program even though it was terminated but at a level that was reduced from the peak exposure experienced in 2015. In contrast, exposure continued to rise in Kaduna. Because of the large number of exposures, we used principal components analysis to develop an exposure index that then could be used to test for interactions of exposure with city, time, and religion. Surprisingly, we found little evidence of interaction effects. Thus, the effect size of exposure in Ilorin was not statistically different from the effect size in Kaduna. The time and religion interactions also turned up little of substance. It is also notable that ideation had a similar pattern to exposure such that it decreased over time in Ilorin while it increased in Kaduna. However, the effect of ideation on contraceptive use remained positive for both cities and exposure to NURHI program activities was significantly related to ideation for both cities as well.
The descriptive statistics provide some interesting perspectives on observed changes in the sample. Notably, in both Kaduna and Ilorin, modern contraceptive use increased significantly. Further, as mentioned above, there was significantly more exposure in Kaduna in 2017 relative to 2015 while exposure significantly dropped in Ilorin. The demographic factors worked in favor of increasing use in this longitudinal sample with more respondents “aging into” ages where we tend to see higher use and marital status having little change over time. This demonstrates the importance of undertaking the multivariate analyses.
The results of the analyses that teased apart the exposure items demonstrated that a number of the NURHI program activities, when grouped by similar exposure methods, were related to continued contraceptive use in the follow-up period. Mass media and print media had significant effects in the final analyses. Notably, mass media increased significantly in Kaduna but decreased in Ilorin over the 2015 to 2017 period. Similarly, exposure to print media significantly increased in Kaduna but significantly decreased in Ilorin. These types of activities may continue to have an effect in Ilorin through sustained effects or diffusion if, for example, the outreach workers continue to use messages learned earlier under the NURHI program or if the television shows or advertisements are seen through the national media or through local cable channels. In an analysis that used the 2015 Nigeria data, authors [33] demonstrated diffusion of program effects beyond the study cities into rural parts of the study states; this may be what is taking place in Ilorin to sustain the activities, but at a lower level.
Our results can be compared to the earlier studies on sustainability of FP programs in India and for other health outcomes in sub-Saharan Africa [1,21]. Like the PRACHAR program, we demonstrate that the intermediate outcome of ideation sustains over time; however, ideation is significantly higher in the city that continued compared to the city where the program was terminated [21]. That said, we do find that both ideation and exposure continue to be related to modern method use in both cities. Our study differs from many of the sub-Saharan African studies by including quantitative data and focusing on outcomes (ideation and contraceptive use) whereas many of the studies of sustainability of other health programs focus on the sustainability of program components (e.g., quality of care or program implementation) and are undertaken using qualitative methods [1,4,6,10,12,15,16]. Future studies examining family planning outcomes post-program termination are important for understanding the circumstances of program sustainability beyond the initial funding cycle.
This study is not without limitations. The first limitation is that it was difficult to examine the exposure effects with the granularity that we would have liked since there were a large number of program components and there was correlation across the NURHI program activities. With that in mind, we created an exposure index which demonstrated the importance of NURHI exposure in both Kaduna that continued activities as well as in Ilorin where activities ended; overall exposure declined in Ilorin over time. Second, the longitudinal sample used for the multivariate analysis, while providing a more rigorous analysis approach, was not large enough to allow us to stratify by city which would have been helpful for understanding city-level differences. Third, women are self-reporting their exposure to the program activities and while they were asked about exposure in the last year for many of the components, the radio and television measures were broader and may capture remembered exposure from earlier in the NURHI Phase I programming. With the data available, it is not possible to control for this potential retrospective reporting bias. Finally, discussions with the NURHI team have indicated that in NURHI Phase II, a focus of the program was to spread to the state-level. This meant that efforts were less intensive in the city of Kaduna in the two-year follow-up period. This may have attenuated the effect of program sustainability in Kaduna from what might have been expected if Kaduna city had remained the primary target area of the program.
This study also has important strengths. Data were collected to examine the sustainability of program activities after the program ended; these types of data are rarely available post-program funding. The results of this study demonstrate important results to inform future FP programming. In particular, the program effects sustained two-years out such that exposure to the program continued to be related to ideation, a key intermediate factor for the NURHI program, and modern contraceptive use. Further, there may be some diffusion effects such that women continue to discuss program-related factors (e.g., discuss family size; reject myths; etc.) and this discussion may help to spread the effects of the program post-implementation. Finally, this study showed that in Kaduna, the city where the program continued, there was less increase in modern method use than in the earlier phase. This may reflect diminishing returns of program activities, or alternatively may reflect the increased focus of program activities on other parts of the state.
To conclude, the effect of the NURHI Phase I program sustained and was associated with continued increases in modern contraceptive use, even in the city where the program ended. It is notable that the NURHI Phase I program activities did not have an explicit plan for sustainability and therefore, sustained exposure, whether through diffusion or remembering earlier messages on the radio and television, is promising. That said, exposure in Ilorin did decline significantly, indicating a risk that with more time, the effect of NURHI programming on ideation and modern contraceptive use may fade. As part of NURHI Phase II, greater attention is focused on identifying strategies that are sustainable and engaging the government for long-term support. That said, in this analysis, the exposure effects in Kaduna were no greater than those in Ilorin; this might reflect the program focus in NURHI Phase II on spreading to rural areas and thus not continued attention to urban areas. Funders and program planners need to consider undertaking activities that can be sustained from the launch; this likely involves engaging the government and public sector throughout program design and implementation, as done under NURHI Phase I and more strongly done under NURHI Phase II. That said, there is also a need to identify strategies to insert new energy into FP programming to help affect greater increases in modern method use in a place like Kaduna that has had FP programming for a number of years. This study demonstrated sustainability of program activities; however, questions remain on what will happen over time given observed declines in exposure in Ilorin. Future studies are needed to examine the long-term impact of these declines on modern family planning use.
Supporting information
S1 Table. Principal components analysis factor loadings used to generate ideation and exposure variables.
Factor loadings obtained using full representative cross-sectional sample for Kaduna and Ilorin in 2017 were used to calculate 2015 values.
https://doi.org/10.1371/journal.pone.0222790.s001
(DOCX)
S2 Table. Correlated random effects results with interactions for the 2015 and 2017 longitudinal sample–three sets of results defined by the outcome and main predictors.
https://doi.org/10.1371/journal.pone.0222790.s002
(DOCX)
S3 Table. Controls for correlated random effects model [S2 Table, model (c)] Modern contraceptive use and exposure index for the 2015 and 2017 longitudinal sample.
https://doi.org/10.1371/journal.pone.0222790.s003
(DOCX)
S1 File. Nigeria 2015 Endline Woman Questionnaire.pdf–woman’s questionnaire used in 2015 survey.
https://doi.org/10.1371/journal.pone.0222790.s004
(PDF)
S2 File. Nigeria 2017 Sustainability Woman Questionnaire–woman’s questionnaire used in 2017 survey.
https://doi.org/10.1371/journal.pone.0222790.s005
(PDF)
Acknowledgments
The authors acknowledge those who supported this work at various phases of project design, implementation, and analysis. This includes: the Nigerian Urban Reproductive Health Initiative (NURHI) Team, data collectors, and study participants.
References
- 1. Iwelunmor J, Blackstone S, Veira D, Nwaozuru U, Airhihenbuwa C, Munodawafa D, Kalipeni E, Jutal A, Shelley D, Ogedegbe G. Toward the sustainability of health interventions implemented in sub-Saharan Africa: a systematic review and conceptual framework. Implement. Sci. 2016; 11: 43. pmid:27005280
- 2. Schell SF, Luke DA, Schooley MW, Elliott MB, Herbers SH, Mueller NB, Bunger AC. Public health program capacity for sustainability: a new framework. Implement. Sci. 2013; 8: 1.
- 3. Stephenson R, Tsui AO, Knight R. Measuring family planning sustainability at the outcome and programme levels. Health Policy Plan. 2004; 19: 88–100. pmid:14982887
- 4. Stirman SW, Kimberly J, Cook N, Calloway A, Castro F, Charns M. The sustainability of new programs and innovations: a review of the empirical literature and recommendations for future research. Implement. Sci. 2012; 7: 17. pmid:22417162
- 5. Scheirer MA. Is sustainability possible? A review and commentary on empirical studies of program sustainability. Am. J. Eval. 2005; 26: 320–347.
- 6. Tibbits MK, Bumbarger BK, Kyler SJ, Perkins DF. Sustaining evidence-based interventions under real-world conditions: Results from a large-scale diffusion project. Prev. Sci. 2010; 11: 252–262. pmid:20229358
- 7. Bao J, Rodriguez DC, Paina L, Ozawa S, Bennett S. Monitoring and evaluating the transition of large-scale programs in global health. Glob. Health Sci. Pract. 2015; 3: 591–605. pmid:26681706
- 8. Helzner JF. Transforming family planning services in the Latin American and Caribbean region. Stud. Fam. Plann. 2002; 33: 49–60. pmid:11974419
- 9. Scheirer MA, Dearing JW. An agenda for research on the sustainability of public health programs. Am. J. Public Health 2011; 101: 2059–2067. pmid:21940916
- 10. Bennett S, Ozawa S, Rodriguez D, Paul A, Singh K, Singh S. Monitoring and evaluating transition and sustainability of donor-funded programs: Reflections on the Avahan experience. Eval. Program Plann. 2015; 52: 148–158. pmid:26099560
- 11. Eder C, Khatiwada LK, Schooley J. Sustainability of a community-based CHOICE program to improve the health and nutrition status of mothers and infants in Indonesia. Matern. Child Health J. 2018; 22: 903–912. pmid:29429132
- 12. Ozawa S, Singh S, Singh K, Chhabra V, Bennett S. The Avahan transition: effects of transition readiness on program institutionalization and sustained outcomes. PloS One 2016; 11: e0158659. pmid:27434542
- 13. Sarriot EG, Jahan S, Kouletio M, Sardar M, Ali KL, Saha S, Rasul I. The end of magical thinking: Sustainability evaluation three years after the end of the Saidpur and Parbatipur urban health project. Final Rep. 2008. https://www.measureevaluation.org/resources/networks/datausenet/dashboards-and-data-use-forum-may-2010/end-magical-thinking.pdf (accessed April 26, 2019)
- 14. Thilakavathi S, Boopathi K, Kumar CG, Santhakumar A, Senthilkumar R, Eswaramurthy C, Bharathy VI, Ramakrishnan L, Thongamba G, Adhikary R. Assessment of the scale, coverage and outcomes of the Avahan HIV prevention program for female sex workers in Tamil Nadu, India: is there evidence of an effect? BMC Public Health 2011; 11: S3.
- 15. Savaya R, Spiro S, Elran-Barak R. Sustainability of social programs: a comparative case study analysis. Am. J. Eval. 2008; 29: 478–493.
- 16. Zulu JM, Goicolea I, Kinsman J, Sandøy IF, Blystad A, Mulubwa C, Makasa MC, Michelo C, Musonda P, Hurtig AK. Community based interventions for strengthening adolescent sexual reproductive health and rights: how can they be integrated and sustained? A realist evaluation protocol from Zambia. Reprod. Health. 2018; 15: 145. pmid:30153839
- 17. Hirschhorn LR, Talbot JR, Irwin AC, May MA, Dhavan N, Shady R, Ellner AL, Weintraub RL. From scaling up to sustainability in HIV: potential lessons for moving forward. Glob. Health 2013; 9: 57.
- 18. Subramanian L, Simon C, Daniel EE. Increasing contraceptive use among young married couples in Bihar, India: Evidence from a decade of implementation of the PRACHAR Project. Glob. Health Sci. Pract. 2018; 6: 330–344. pmid:29959273
- 19. Daniel EE, Masilamani R, Rahman M. The effect of community-based reproductive health communication interventions on contraceptive use among young married couples in Bihar, India. Int. Fam. Plan. Perspect. 2008; 189–197. pmid:19201679
- 20. Jejeebhoy SJ, Prakash R, Acharya R, Singh SK, Daniel E. Meeting contraceptive needs: Long-term associations of the PRACHAR Project with married women’s awareness and behavior in Bihar. Int. Perspect. Sex. Reprod. Health. 2015; 41: 115–125. pmid:26600565
- 21. Pandey N, Jejeebhoy SJ, Acharya R, Singh SK, Srinivas M. Effects of the PRACHAR Project’s reproductive health training programme for adolescents: Findings from a longitudinal study. Population Council. 2016.
- 22.
Levitt-Dayal M, Motihar R. Adolescent girls in India choose a better future: an impact assessment. In: Towards adulthood: exploring the sexual and reproductive health of adolescents in South Asia, edited by Bott S, Jejeebhoy S, Shah I, Puri C. Geneva, Switzerland, World Health Organization [WHO], Department of Reproductive Health and Research. 2003.
- 23. Krenn S, Cobb L, Babalola S, Odeku M, Kusemiju B. Using behavior change communication to lead a comprehensive family planning program: the Nigerian Urban Reproductive Health Initiative. Glob. Health Sci. Pract. 2014; 2: 427–443. pmid:25611477
- 24. Measurement, Learning & Evaluation (MLE) Project Nigeria Team. Evaluation of the Nigerian Urban Reproductive Health Initiative (NURHI) Program. Stud. Fam. Plann. 2017; 448: 253–268. https://doi.org/doi:10.1111/sifp.12027
- 25. Fabic MS, Choi Y. Assessing the quality of data regarding use of the lactational amenorrhea method. Stud. Fam. Plann. 2013; 44: 205–221. pmid:23720003
- 26. Babalola S. Changes in ideational profiles of women of reproductive age in urban Nigeria: The role of health communication. Health Educ. Behav. 2017; 44: 907–917. pmid:28387572
- 27. Speizer IS, Calhoun LM, McGuire C, Lance PM, Heller C, Guilkey DK. Assessing sustainability of N Urban Reproductive Health Initiative programming: longitudinal analysis of service quality. BMC Health Services Research 2019: 559. pmid:31399085
- 28. Montgomery MR, Kiros GE, Agyeman D, Casterline JB, Aglobitse P, Hewett PC. Social networks and contraceptive dynamics in Southern Ghana. Policy Res. Div. Work. Pap. 153. 2001.
- 29.
Wooldridge JM. Econometrics: Panel Data Methods, in: Complex systems in finance and econometrics. Springer, pp. 215–237. 2009.
- 30. Schunck R. Within and between estimates in random-effects models: Advantages and drawbacks of correlated random effects and hybrid models. Stata J. 2013; 13: 65–76.
- 31. Schunck R, Perales F. Within-and between-cluster effects in generalized linear mixed models: A discussion of approaches and the xthybrid command. Stata J. 2017; 17: 89–115.
- 32. Mundlak Y. On the pooling of time series and cross section data. Econom. J. Econom. Soc. 1978: 69–85.
- 33. Guilkey DK, Escamilla V, Calhoun LM, Speizer IS. The Examination of Diffusion Effects on Modern Contraceptive Use in Nigeria. Technical Working Paper, Chapel Hill, NC. 2019.