Object Oriented (Dynamic) Programming: Closing the “Structural” Estimation Coding Gap

Abstract

This paper discusses how to design, solve and estimate dynamic programming models using the open source package niqlow. Reasons are given for why such a package has not appeared earlier and why the object-oriented approach followed by niqlow seems essential. An example is followed that starts with basic coding then expands the model and applies different solution methods to finally estimate parameters from data. The niqlow approach is used to organize the empirical DP literature differently from traditional surveys which may make it more accessible to new researchers. Features for efficiency and customization are also discussed.

Appendices

Appendix A. Example of PP vs. OOP Coding

To illustrate differences between OOP and PP, consider a package written by a programmer to be used by economists (users): first using the PP paradigm only and then using OOP. The package, named Marshall, solves for Marshallian demand for a consumer with utility U(x) defined on a vector x and a given price vector p and income m : 

$$\begin{aligned} x^{\star} (p,m;U)\quad \equiv \quad {\arg \max }_{\ x: px\le m\quad } U(x). \end{aligned}$$

Most readers have probably coded an objective and then called a built-in optimization procedure to optimize that function. Marshall is a specialized version of that general problem.

The PP package documentation explains how users should code U() in order to interact with tools in the package. The user codes u(x), and sends it to a built-in procedure of the form demand(u,p,m). That procedure uses algorithms to compute $x^{\star} (p,m).$ Suppose the user wants to use the Cobb-Douglas function, $U(x) = {\sum } \ln x_i.$ Using "pseudo-code" the key parts of the user’s program might look like:

Now consider the OOP version of Marshall. The programmer might define a class for a consumer:

The syntax is pseudo code similar to actual OOP languages, including Ox. The method u(x) belongs to the Consumer class and does the work of computing utility. The budget parameters are stored as members of the class. These will be set by passing them to the budget() method. The method demand() is the same as the PP procedure above, but it will get the information it needs from the data members rather than from arguments. It stores the result in the member xstar.

The package comes with the Cobb-Douglas function set as the a default to demonstrate the package without any coding. Now u() is coded as a method of the Consumer class:

Unlike an ordinary function, the code for u() has the prefix of the class it belongs to. Code for the other methods would also be part of the package. User code to create a Consumer object, set the budget to already-defined values, and solve for $x^{\star}$ might look like this:

Here the new operator makes an object from the Consumer template and stores it in the variable agent. The code above would use the built-in utility and compute quantity demanded at the prices and income sent to the budget operator. The syntax object -> method() is a common way to invoke a method for a particular object. That is, instead of sending u to demand() in the PP approach, the data specific to agent is automatically available to the demand() method belonging to agent.

The code so far uses a built-in utility. To use, say, a CES utility the user creates a class derived from Consumer.

The first line shows that CES is a child of Consumer. The new class does not declare its own demand() method, because CES inherits the version from Consumer. The user also provides a method that is called to create a new object. This constructor has the same name as the class in the pseudo code. The CES parameter is passed to it and stored in CES.a, ready to be used by CES::u().

When demand() is invoked, the user-provided CES::u() is called instead of the default version written by the programmer. The user has not changed, and perhaps cannot even see, the original code for demand(). This is because the programmer marked Consumer::u() as virtual. By doing so the programmer gives the user a controlled ability to change the underlying code through insertion a replacement function. In the PP packaged this injection of code was accomplished by passing u to the demand function. When many functions need to be replaced in many algorithms the PP framework can become unwieldily and unreliable. The OOP approach scales more efficiently for both the programmer and the user with problem complexity. With OOP it is easier to ensure the right data and the right functions are being used within the package.

The programmer can create a taxonomy of classes for the user to choose from. In this simple case, Marshall might not just have a single Consumer class. It might have child classes for different classes of utility. The user can then start with one of those classes to specialize or extend it for their model. An OOP package can provide the user with a menu of options, an important part of the niqlow approach to empirical DP.

Appendix B: Code Segments and Additional Explanation of niqlow Features.

1.1 CV() and AV() in Code Segment E

If the action counter M were an ordinary programming counter taking on values between 0 and 39 then the statement E.1 would not require sending it to CV(). A simple assignment D=M would set D to the current value of M. However, M is an object of a class so it is not the same as its value. Instead, the current or counter value of M is a member of the object. Classes in niqlow that represent DP variables have a member v that holds the counter or current value of the variable. The internal code sets the value of v to correspond to the current state $\theta$ before code such as Utility() is called. The function CV(M) returns M.v. So either D=M.v; or D=CV(M); is how user code would set D to the value of M at the current state.

Recall that utility is treated as a vector valued function corresponding to the feasible set $A(\theta ).$ Since m is an action variable its current value is not a scalar at $\theta .$ In this simple one-choice model the current value is always the same: $CV(m) = \left(\begin{array}{l}0\\ 1\end{array}\right).$ If other actions were added, or if constraints on feasible actions were imposed on choice, the current value of m would be a different length because the number of distinct actions $\alpha$ would change.

Since M is a simple count variable, it takes on values like a loop counter or index. The earnings shock e, is also like a loop counter, so its Its counter value ranges from 0 to 14. However, e is a discretized normal random variable and the counting values are associated with both positive and negative real numbers, i.e. quantiles of the standard normal distribution, such as $-1.282,$ the 10th percentile of N(0, 1). The user’s code can carry out these transformations of the integer value e.v, but niqlow can track actual values of variables for the user.

The actual values of an object are stored as vector member actual. Thus the actual value at any point is actual[v]. The current value is an index into the actual vector. The function AV() function retrieves this value, so when $CV(e)=3$ AV(e) might equal -1.282. For M the actual vector is (0 1 ...39) and actual[v]=v. That is, the default is that AV(s)=CV(s). Only in the case like a discretized normal will there be a difference. The user can set actual values for their state and action values and can make them dependent on structural parameters.

1.2 Semi-Exogenous State Variables

An example of an IID process that could be sorted into $\eta$ but not $\epsilon$ is a wage offer h with on-the-job search. If the offer is accepted it determines earnings this period also the existing wage next period. The existing wage is then a state variable placed in $\theta$ because its transition depends on choices. Next period a new IID outside shock h is realized as well, but current h affected the transition beyond its influence on the action $\alpha$ so it must be placed in $\eta$ not $\epsilon .$

1.3 Additional Actions and Restricted Choices

Starting with the basic labor supply model suppose the user wants to add a choice to attend school or not (s) and a state variable to track accumulated schooling: $S^\prime = S + s.$ The agent cannot attend school and work in the same period, so the choice vector and feasible set are now

$$\begin{aligned} \alpha = \left(\begin{array}{ll}s&a\end{array}\right) \in A(\theta ) \equiv \left\{ \alpha :\ s*a = 0\right\} . \end{aligned}$$

Although initialization and space creation can only occur once, in between new variables can be added to the vectors more than once. So the extended build is simply:

The base version added m to the action vector. This adds s to it. S is an action counter like M but limited to 8 years of additional schooling to reduce the size of the state space.

The user must tell niqlow to impose the condition that the agent can either work or study but not both. This creates an additional trimming of unreachable states: $M+S\le t.$ In this approach the agent has to impose this extra condition on reachable states. The user replaces two built-in virtual methods with their versions:

The first returns a vector of ones and zeros that indicates whether an action $\alpha$ is feasible at the current state $\theta .$ It says the product of each row of the action vector must be 0. Ox syntax allows the expression to closely match the definition of $A(\theta ).$ The second returns a scalar 0 or 1 to indicate whether the current state is reachable from initial conditions. It needs to know what the current value of t is which up until now was not required. Since the clock is stored internally niqlow places its current value in the I class, so I::t is always available as well as other indices of the current state.

1.4 Augmented State Variables

The current library of pre-defined state variable types in niqlow include counters, accumulators, lagged values, durations, and discrete jump processes. A user can also provide a Markov transition matrix for an arbitrary process. Many estimated DP models contain state variables that customize these basic versions. For example, some models may "freeze" a state variable at it’s current value after some period $t^{\star} .$ In some models a state variable stops being relevant to the agent’s problem at some point. For example, a model of schooling and work might track credits earned while still in school, but once out in the labor market credits no longer matter and tracking them is inefficient. These are called augmented state variables in niqlow.

Here are 3 augmented state variables:

The first one starts with the schooling variable added to the labor supply model. Instead of adding it directly it is augmented to freeze at its value from $t=15$ and forward. The base variable x is is created and sent to Freeze which then wraps the augmented transition rules around the base transition.

State variable y augments a base state variable b (not shown here) so that its value resets to 0 whenever the agent sets the action variable a to 1 (also defined elsewhere). This is a special case of the general Triggered() augmentation. Several triggers besides the simple reset are already coded. Finally, z is a double augmentation of a state variable d and another state variable tvar. First, when tvar=1 z will reset to 5. From $t=20$ and onward the value of z is not tracked because of the ForgetAtT augmentation. Forgetting a sate variable means its value is simply $CV(z)=0$ from then on, which avoids expanding the state space unnecessarily.

1.5 Complete Code and Output for the Labor Supply Example

This Ox code is available in the examples folder in the niqlow download. A few lines are different than the code in the main body to account for the reservation wage extensions (Fig. 5).

Fig. 5

Predictions for the Labor Supply Model

How can we confirm that these results are correct? First, it is eaasier to check the CCP and EV values at each state instead of these averaged predictions. However in general the best way to confirm results is to set parameters of the problem so that output can be compared to known true values.

In this model we can push the smoothing parameter from 1.0 to near 0 and to very large:

Make the agent myopic so that only current utility matters. CCP’s are easy to compute at any state:

noindent Make the environment static by eliminating the effect of endogenous states:

Once the extremes are confirmed we can deform the problems slightly and see that the output moves in the right direction.

Unlike purpose-built code, in niqlow these kinds of tests use the same underlining code for all models. In addition, the behavior of a state variable class, such as ActionCounter can be confirmed in a small test program. Then it is very likely it will perform correctly in any other problem.

Certainly not all bugs have been discovered let alone fixed in the current niqlow code. And varying parameters of a single model will not reveal all errors. However, as changes are made that might break code that used to work, a suite of test programs are run to make sure that output is still correct. In that sense the changes to experiments on the labor supply model are important for a user to run in order to overcome healthy skepticism. However, the kinds of errors that might uncover have been squashed by checking test program output that include many other features than the labor supply model but in smaller spaces. There are also debugging features that can be turned on to trace output when a bug has been discovered.

Appendix C: Solution Algorithms

These algorithms are explained in Sect. 4.2. Aguirregabiria and Mira (2010) review several methods in more detail.

Appendix D: Reservation Values

For a binary choice a, there is a single $z^{\star}$ at each (implicit) state that solves $v(1,z^{\star} )-v(0,z^{\star} ) =0.$ Let $EV_i$ denote $EV\left(\theta ^{\,\prime }\,|\,a=i\right)$. After some rearranging $z^{\star}$ satisfies

$$\begin{aligned} U(1,z^{\star} )-U(0,z^{\star} ) = \delta \left[ EV_0-EV_1\right]. \end{aligned}$$

(35)

The differences between current and future values balance, and (35) are solved as a non-linear equation. The condition that future expected values cannot depend on current z,  avoids a more complicated condition. Each additional choice beyond binary adds another equation between adjacent values to be solved simultaneously.

Once $z^{\star}$ has been found, Bellman iteration requires calculation of the expected value of arriving at the (now explicit) state $\theta :$

$$\begin{aligned} EV(\theta )= & {} G(z^{\star} )\left( E\left[U|a=0,z< z^{\star} \right] + \delta EV_0 \right)\nonumber \\&+ \left(1-G(z^{\star} )\right)\left( E\left[U|a=1,z\ge z^{\star} \right] + \delta EV_1 \right). \end{aligned}$$

(36)

These conditions are illustrated in Fig. 6. Conditions (35), (36) include three additional objects that the user’s code must provide. The first condition for $z^{\star}$ needs $U\left(A(\theta ),z\right),$ a vector of utilities for candidate values of z while solving for $z^{\star} .$ The second includes $F(z^{\star} )$ and the vector of expected conditional utilities, $E\left[U\left(0\right)| z< z^{\star} \right]$ and $E\left[U\left(1\right)|z\ge z^{\star} \right].$

Fig. 6

Conditions for Reservation Values

The model must satisfy several conditions described there to be eligible for this solution algorithms. These are enforced by allowing only models derived from the OneDimensionalChoice class to use the ReservationValues method.

• 1. $\alpha$ must be one-dimensional (only one variable added to the action vector);

• 2. No smoothing shock $\zeta$ is included (because Z plays that role);

• 3. No state variables are placed in $\epsilon$ and $\eta$ vectors (because reservation values must be stored using $\theta$);

• 4. Choice values $v\left(\alpha ,\theta \right)$ must the single-crossing property.

The user must parameterized the model to enforce the last condition. Any IID state variables must stay in $\theta$ so that the value of $z^{\star}$ can be stored conditional on their values as well.

OneDimentionsalChoice adds two virtual functions that other Bellman classes do not. The two functions are Uz() and EUtility(). At $z^{\star}$ the difference in current utility equals the discounted difference in future values. The user codes Uz(z) to return utility of all choices at z. The reservation value solution method uses that to solve for $z^{\star} .$ Backward induction requires the expected utility of each option conditional given the optimal value $z^{\star}$, hence the need to provide EUtility().

1.1 Code for the Reservation Value Labor Supply Model

For the basic labor supply model, the earnings shock e is a discretized normal. Change that assumption so it is a continuous standard normal random variable, re-labeled z. The class created for this version will be named LSz. Earnings are now written

$$\begin{aligned} E = \exp \{\beta _0+\beta _1M +\beta _2M^2 + \beta _3 z\}. \end{aligned}$$

Since E exceeds the value of not working ($\pi$) for large enough values of z there is a reservation value for the choice $m=1.$ Since LSz must be a one-dimensional choice model it cannot be derived from LS as was done earlier with LSext.

However, code from the basic model can be reused because most elements of LS are static. The only non-static member was Utility(). By using the static elements a change to LS will still be reflected automatically in LSz. In particular, LSz::Uz() can use LS::Earn() to utility at z. First set the value of LS::e to z then calls LS::Earn().

In the model expected utility of not working is simply $\pi$. For working, expected utility involves $E[e^{\beta _3z}|z>z^{\star} ]$. The Mills-ratio formula for log-normality gives expected utility conditional of acceptance as

$$\begin{aligned} E[U|m=1,z>z^{\star} ] = \exp \left\{ \beta _0+\beta _1M+\beta _2M^2+{\beta _3^2\over 2}\right\} {\Phi \left(z^{\star} /\beta _3 - \beta _3\right)\over \Phi \left(z^{\star} \right)}. \end{aligned}$$

(37)

The constant factor includes $\beta _3^2/2$, because it is coefficient on the normal random variable z (and hence the variance of the model shock). The ratio of $\Phi ()$ values is new to the continuous specification and can’t be borrowed from the discretized LS.

However, note that only the constant term includes the Mincer equation, and it is the same as the original earnings function if $e=\beta _3/2.$ So again the base LS::Earn() function can be used by setting the value of e first. Thus, even though LS was coded for a completely different approach its specification is still synchronized with the reservation wage version. Only elements specific to the new version need to be coded.

This code segment converts the labor supply model to a continuous choice reservation value problem.

Appendix E: Estimation on Simulated Labor Supply Data

Code segment K estimated the labor supply model from external data. To use simulated data the code is modified slightly:

The second line simulates 1000 observations over full 40 year lifetimes. The simulated data could be printed to a file and then read in as segment M does. In this case the simulated data is already contained in the object so it is ready to be used for estimation.

Abbreviated output is below. The code detects that none of the observations are full CCP because e is unobserved. Since all actions and endogenous states $\theta$ are observed all paths are categorized as IID. This means niqlow will automatically sum over $\epsilon$ to compute the likelihood but it is unnecessary to use the backward Algorithm 6. The measurement error on noisy earnings is fixed at its true value. The result is convergence after 5 BHHH iterations with weak convergence. Because by default niqlow scales starting parameters the reported standard errors would need to be re-scaled, or recomputed with scaling and constraining turned off (details available in the documentation). Having started at the true parameter values, the sample likelihood is within .01% of the initial value with some differences in parameter estimates, particularly the coefficient on experience (M) (Fig. 7).

Fig. 7

Output of MLE Estimates