“Econometrics,” “policy and program analysis,” “pre-post comparisons,” and “confounding variables”—these are all terms which may have felt unfamiliar to attendees at Free the Facts’ third and final Summer Series event last Wednesday. But by the end of the evening, Dr. Risha Gidwani had explained how they are all important components of the policy evaluation process. 

As a senior policy researcher at the RAND Corporation and an adjunct professor at the UCLA Fielding School of Public Health, Gidwani uses quantitative analysis methods to compare the quality and costs of health care systems in the United States. Over the course of Wednesday’s event, she walked attendees through the processes and methods she uses to conduct her research.

‍

Gidwani began by describing the difference between a “policy analysis” and a “program analysis.” Policy analyses evaluate the effect a policy has on pre-specified outcomes, and are usually commissioned by the government or a sponsor of the policy. The findings from such an analysis may not end up affecting the policy directly. 

For example, imagine you conducted research on the financial impact of high drug prices on Medicare recipients. Even if your policy analysis found that regulating drug prices would improve financial outcomes for recipients, nothing about Medicare would change until new legislation was written and passed.

On the other hand, Gidwani explained, program analyses evaluate existing programs and are usually commissioned by the organization funding the program. Their findings tend to have a more direct impact, since the organization relies on them to improve the program’s outcomes. For example, imagine you were researching how to reduce the cost of care for patients with multiple chronic illnesses. The findings from your program analysis could directly change the methods that hospitals use to care for these types of patients.

‍

Both forms of analysis have stringent research methods that must be followed, Gidwani went on to say. For example, researchers must use a comparison group of people who are completely unaffected by the policy or program being studied. However, since public policies are usually rolled out uniformly across an entire population, it can be difficult to find people who have never been affected by them.

As a workaround, researchers often conduct studies using “pre-post comparisons.” This method compares what happened before and after a policy or program was implemented. Gidwani noted that this is not always ideal, because outside factors—like an unexpected global pandemic—may change people’s behavior and lead policy researchers to record false outcomes.

In this example, the pandemic is a “confounding variable,” or an unforeseen third factor that changes the relationship between the two variables being compared. So how do policy experts make sure that their research controls for confounding variables? This is where “econometrics” comes in handy.

‍

“That is really just a fancy term for statistics that is used in economics,” Gidwani explained.

Randomization is necessary for a study to avoid confounding variables, but it rarely ever happens in economics. For example, the entire population is usually affected by a tax policy or an economic shock, leaving little room for randomization. To combat this, economists often adopt statistical methods such as regressions and difference-in-difference modeling, which yield statistically significant results and explicitly deal with confounding variables.

‍

“All of these are advanced statistical topics...” Gidwani said, “Each one of these could be at least one lecture in and of itself—probably more—and any good graduate school program that has a robust quantitative analysis path is going to cover these topics.”

‍

This wasn’t the only mention of graduate school during the event. Attendees used the Q&A session to ask Gidwani about pursuing a PhD and to express their own interest in conducting policy research.

Gidwani encouraged students to follow this career path. She happily noted that her work can be fun and challenging, often at the same time. “It is certainly a job that requires a lot of training, and it is not a ‘phone-it-in’ kind of job,” she said. The stakes can be high when thousands of people’s health and well-being are on the line, but that’s also what makes the field extremely rewarding.

When asked if she had any final advice for those interested in applying to grad school for policy research, Gidwani explained that students should look for universities with a strong research presence. By considering both the quality and quantity of research a university produces, students can ensure that they are exposed to a wide variety of projects and derive the most benefit from their time in a graduate program.