Investigating complementarities in subscription software usage using advertising experiments

Abstract

In this study, we examine complementarities in usage across a set of related software products from a multi-product firm. We employ a novel experimental approach to causally estimate complementarities, leveraging rich usage data and advertising experiments that directly affect the usage of only one product at a time to measure complementarities based on consumption rather than purchase. Our approach is particularly useful as digital contexts are characterized by the simultaneous presence of both substitutability and complementarity between products. They also have scant price variation, bundled pricing plans, and infrequent purchase or subscription renewal decisions, often making typical cross-price elasticity measures for complementarities infeasible. We apply our approach to data from a software company with a suite of related products and find evidence for varying degrees of complementarity across both user groups and products. We show that accounting for complementarities significantly affects the measurement of ad effectiveness and may impact ad targeting decisions by the firm. We explore heterogeneity in complementarities, finding that they are larger for users who have used the products heavily in the past, but small or zero for those who have not. Ours is one of the first studies to causally examine complementarity in usage in the context of subscription products, and our identification strategy can be applied to a variety of contexts.

A. Appendix

The appendix contains the following sets of additional analysis:

• 1. Full Campaign Details & Descriptions, Tables 4and  5: statistics and descriptions of the full set of campaigns for both app A and app B, taken from the firm’s campaign management platform.

• 2. Randomization Checks, Table 6: randomization checks from a subset of demographic variables for the users in each campaign. The variables are user account tenure, number of previously-canceled subscriptions, and the percent of users on promotions. We only include the p-value for a difference in means/proportions test to protect confidentiality. Other demographic variables were deemed too firm-specific to be made public. 4 of 144 (2.7%) variables are significant at the 1% level, and 13 of 144 (9%) at a 5% level, indicating slight randomization failures, but we do not view this as a major problem

• 3. App Usage Across Campaigns, Figs. 8 and 9

• 4. Individual App A Campaigns, Binary Specification, campaigns with non-significant first stage results, Table 7

• 5. Robustness Checks: Individual App B Campaigns, Binary Specification, Tables 8 and 9, Figs. 10 and  11

• 6. Robustness Checks: Cross-product Ad Elasticity Measures, Table 10

• 7. Robustness Checks: Individual App A Campaigns, Session Count Specification, Tables 11and 12, Figs. 12 and  13

• 8. Robustness Checks: Volume Specification Active User Subpopulations, App A Tables  13, 14, 15

Table 4 Campaign Statistics: All Campaigns

The supplementary analysis, located here, contains additional analyses that are not included here for brevity. The authors intend to host this on their own sites. They are the following:

• 1. Full Pooled Regression Results with Campaign Dummies, Tables 1and 2

• 2. App B Session Count Regressions, Tables 3and 4

• 3. App B Volume Specification Active User Subpopulations Session Count Regressions, Tables 5and 6

• 4. Heterogeneity with Product Experience Regressions, Tables 7-18

Table 5 Campaign Descriptions: All App A Campaigns

Table 6 Randomization checks: demographic variables

Fig. 8

This plot and Fig. 9 show the average daily usage likelihood across apps for users in the app A and B campaigns. Usage patterns are heterogeneous, indicating that a variety of user groups are contained in these populations

Fig. 9

See Fig. 8

Table 7 Campaign Regression Comparison: Individual App A Campaigns, 7-day Window (Part 2/2)

Table 8 Campaign Regression Comparison: Individual App B Campaigns, 7-day Window (Part 1/2)

Table 9 Campaign Regression Comparison: Individual App B Campaigns, 7-day Window (Part 2/2)

Fig. 10

App B first stage results: coefficients on the X-axis and log-transformed F-statistics on the Y-axis. Points above the blue line at $\log (10)$ have a strong first stage. Over 40% of campaigns have a strong first stage and all but one have positive effects on application usage

Fig. 11

Second stage app B results: each row is one campaign, and the colored bands represent 95% confidence intervals (Bonferroni adjusted) for each app. Campaigns are ordered by first stage F-stat, and campaigns above the dashed line have an F-stat exceeding 10. App B usage rarely leads to increased usage of other applications

Table 10 Cross-product Ad elasticity results: App A campaigns

Table 11 Campaign Regression Comparison: Individual App A Campaigns, 7-day Window, Session Count (Part 1/2)

Table 12 Campaign Regression Comparison: Individual App A Campaigns, 7-day Window, Session Count (Part 2/2)

Fig. 12

App A volume specification first stage results: slightly fewer campaigns are significant in the first stage, which makes sense as advertising is more likely to remind inactive users than increase the intensity of usage

Fig. 13

App A volume specification second stage results, Bonferroni-adjusted. Campaigns are ordered by first stage F-stat, and campaigns above the dashed line have an F-stat exceeding 10. We see significant results in the second stage roughly as frequently, indicating the presence of complementarity net of reminder-type effects

Table 13 Campaign Regression Comparison: Volume Specification Analysis - App A Campaigns (Part 1/3)

Table 14 Campaign Regression Comparison: Volume Specification Analysis - App A Campaigns (Part 2/3)

Table 15 Campaign Regression Comparison: Volume Specification Analysis - App A Campaigns (Part 3/3)