----------------------------------------------------------------------
This is the API documentation for the specification_curve library.
----------------------------------------------------------------------
## Classes
The main entry point. Describe the specifications, estimate them, and plot the curve.
SpecificationCurve(data: pandas.DataFrame, formula: Optional[str] = None, *, y: Union[str, Sequence[str], NoneType] = None, x: Union[str, Sequence[str], NoneType] = None, controls: Union[str, Sequence[str], NoneType] = None, always: Union[str, Sequence[str], NoneType] = None, exclude: Optional[Sequence[Sequence[str]]] = None, min_controls: int = 0, max_controls: Optional[int] = None, fe: Union[str, Sequence[Optional[str]], NoneType] = None, iv: Union[str, Sequence[Optional[str]], NoneType] = None, model: Union[str, Sequence[str]] = 'feols', family: Union[str, Sequence[str], NoneType] = None, quantile: Union[float, Sequence[float], NoneType] = None, vcov: Union[str, dict[str, str], NoneType, Sequence[Union[str, dict[str, str], NoneType]]] = None, weights: Union[str, Sequence[Optional[str]], NoneType] = None, subsets: Optional[Mapping[str, Optional[str]]] = None, common_sample: bool = True, skip_invalid: bool = False) -> None
Runs and plots every variant of a regression specification.
A specification curve shows how the estimate of one coefficient (the focal
regressor, ``x``) changes across many defensible ways of specifying a model.
The choices that can be varied are the outcome, the focal regressor, the
control variables, the fixed effects, instruments, the estimator, the
standard errors, the weights and the sample. All estimation is done by
pyfixest, and every term is written in pyfixest's formula syntax, so
transformations such as ``log(wage)`` and interactions such as ``i(year)``
can be used anywhere.
Specifications are described either with keyword arguments or with a
pyfixest formula. In a formula, the multiple-estimation operators define the
choices: several outcomes (``y1 + y2 ~``), ``sw()``, ``sw0()``, ``csw()``,
``csw0()``, and two operators added by this package, ``comb()`` (every
non-empty subset of its arguments) and ``comb0()`` (every subset, including
none of them).
Args:
data (pd.DataFrame): The data.
formula (Optional[str]): pyfixest formula describing the outcomes,
regressors, fixed effects and instruments, for example
``"y1 + y2 ~ x + comb0(a, b) | sw0(firm, firm + year)"``. Use this or
``y``, not both.
y (Union[str, Sequence[str], None]): Outcome term(s).
x (Union[str, Sequence[str], None]): Focal regressor(s): the coefficient
that is plotted, named as pyfixest reports it. Required. With several,
each specification uses one of them. With instruments, this is usually
the endogenous variable.
controls (Union[str, Sequence[str], None]): Terms included in every
combination.
always (Union[str, Sequence[str], None]): Terms included in every
specification.
exclude (Optional[Sequence[Sequence[str]]]): Groups of controls that never
appear together in a specification.
min_controls (int): Smallest number of controls in a specification.
max_controls (Optional[int]): Largest number of controls in a specification.
fe (Union[str, Sequence[Optional[str]], None]): Fixed-effect sets, each
written as in a pyfixest formula (``"firm + year"``). ``None`` in the
list means no fixed effects.
iv (Union[str, Sequence[Optional[str]], None]): First stage(s), for example
``"x ~ z"``. ``None`` in the list means no instruments.
model (Union[str, Sequence[str]]): Estimator(s): "feols", "fepois", "feglm"
or "quantreg".
family (Union[str, Sequence[str], None]): Family or families for "feglm".
quantile (Union[float, Sequence[float], None]): Quantile(s) for "quantreg".
Defaults to 0.5.
vcov (Union[VcovSpec, Sequence[VcovSpec]]): Variance-covariance type(s), as
accepted by pyfixest, for example ``"hetero"`` or ``{"CRV1": "firm"}``.
``None`` uses pyfixest's default.
weights (Union[str, Sequence[Optional[str]], None]): Weight column(s).
``None`` in a list means unweighted.
subsets (Optional[Mapping[str, Optional[str]]]): Named samples, each a
``DataFrame.query`` string or ``None`` for all rows.
common_sample (bool): Within each subset, estimate every specification on
the rows that are complete for all variables used by any of them, so
that estimates are comparable. If False, each specification uses all
the rows it can.
skip_invalid (bool): Leave out combinations that an estimator cannot run
(for example quantile regression with fixed effects) instead of raising.
Attributes:
grid (pd.DataFrame): One row per specification, available before fitting.
results (pd.DataFrame): The grid with estimates, available after ``fit``.
models (dict[int, Any]): Fitted pyfixest models keyed by ``spec_id``, if
``fit(keep_models=True)`` was used.
null_summary (pd.DataFrame): Tests of the curve against the null, after
``fit_null``.
null_curves (pd.DataFrame): Quantiles of the curve under the null, after
``fit_null``.
null_draws (pd.DataFrame): Estimates in every resample, after
``fit_null(keep_draws=True)``.
Examples:
>>> import specification_curve as sc
>>> df = sc.load_example_data2()
>>> sco = sc.SpecificationCurve(df, y="y", x="x_1", controls=["x_2", "x_3"])
>>> sco.n_specs
4
>>> len(sco.fit().results)
4
## Functions
Plot a results table directly, and select subsets of specifications.
plot_curve(results: pandas.DataFrame, null_curves: Optional[pandas.DataFrame] = None, *, alpha: float = 0.05, ci_levels: Optional[Sequence[float]] = None, sort_by: Union[str, Sequence[str]] = 'estimate', color_by: str = 'significance', panels: Optional[Sequence[str]] = None, labels: Optional[Mapping[str, str]] = None, highlight: Union[int, str, Mapping[str, Any], Sequence[Union[int, str, Mapping[str, Any]]], NoneType] = None, show_median: bool = True, show_nobs: bool = False, colors: Union[Mapping[str, Any], Sequence[Any], NoneType] = None, figsize: Optional[tuple[float, float]] = None, fig: Union[matplotlib.figure.Figure, matplotlib.figure.SubFigure, NoneType] = None, style: bool = True) -> specification_curve.plot.CurvePlot
Draws a specification curve.
The top panel shows the focal estimate of every specification with its
confidence interval(s). Below it, one panel per dimension that varies shows
which choice each specification makes. Specifications that could not be
estimated are left out.
The function works on any data frame with the columns of
``SpecificationCurve.results``, so results that were saved, filtered or
combined from several curves can be drawn too. It does not call
``plt.show()`` and does not change matplotlib's global settings.
Args:
results (pd.DataFrame): Results as produced by ``SpecificationCurve.fit``.
null_curves (Optional[pd.DataFrame]): Quantiles of the curve under the
null, as produced by ``SpecificationCurve.fit_null``. Can only be
shown when sorting by estimate.
alpha (float): Significance level used when colouring by significance.
ci_levels (Optional[Sequence[float]]): Which of the stored confidence
levels to draw. By default, all of them.
sort_by (Union[str, Sequence[str]]): Order of the specifications:
"estimate", another column of the results, or a list of columns.
Ties are ordered by estimate.
color_by (str): "significance" colours estimates by sign and
significance. A dimension ("y", "x", "fe", "iv", "vcov", "model",
"weights" or "subset") colours by the value of that dimension and
adds a legend.
panels (Optional[Sequence[str]]): Which dimensions to show under the
curve, in order. By default, every dimension that varies.
labels (Optional[Mapping[str, str]]): Replacement display text for panel
titles (keyed by dimension, e.g. ``{"fe": "Fixed effects"}``) and for
values (keyed by the value, e.g. ``{"firm + year": "Firm and year"}``).
highlight (Union[Selector, Sequence[Selector], None]): A preferred
specification to mark and annotate, given as a ``spec_id``, a dict of
column values (e.g. ``{"y": "log(wage)", "controls": ["age"], "fe":
None}``) or a query string. It must match exactly one specification.
Pass a list to mark several.
show_median (bool): Draw a line at the median estimate.
show_nobs (bool): Add a panel with the number of observations used by
each specification.
colors (ColorSpec): Colours to use. When colouring by significance, a
dict with any of the keys "positive", "negative", "insignificant".
When colouring by a dimension, a list of colours or a dict keyed by
the dimension's values.
figsize (Optional[tuple[float, float]]): Figure size in inches. By
default, chosen from the number of panels.
fig (Union[Figure, SubFigure, None]): An existing figure or subfigure to
draw into. By default a new figure is created.
style (bool): Use this package's fonts and sizes for the figure.
Returns:
CurvePlot: The figure and its axes keyed by panel name: "curve",
then the dimensions shown, then "nobs" if requested. It unpacks
as ``fig, axes``.
Raises:
ValueError: If no specification was estimated successfully, or an
argument does not fit the results.
subsets_by(data: pandas.DataFrame, column: str, include_all: bool = False) -> dict[str, typing.Optional[str]]
Builds a ``subsets`` argument with one sample per level of a column.
Args:
data (pd.DataFrame): The data.
column (str): Column whose levels define the samples.
include_all (bool): Also include the full sample, named "all".
Returns:
dict[str, Optional[str]]: Maps a name for each sample to its query.
Examples:
>>> import pandas as pd
>>> subsets_by(pd.DataFrame({"g": ["a", "b", "a"]}), "g")
{'g = a': "`g` == 'a'", 'g = b': "`g` == 'b'"}
## Example data
Data sets used in the documentation and tests.
load_example_panel(n_firms: int = 300, n_years: int = 8) -> pandas.DataFrame
Generates a synthetic firm-year panel.
The true effect of ``treated`` on ``log(sales)`` is 0.3. Treatment is more
likely for firms with a high unobserved firm effect, so specifications
without firm fixed effects overstate it. ``eligible`` is a randomly assigned
instrument for ``treated``.
Columns: ``firm``, ``year``, ``industry``, ``state`` (identifiers);
``treated`` (0/1), ``eligible`` (0/1); ``size``, ``age``, ``leverage``,
``rd`` (controls; ``age`` and ``leverage`` have a few missing values);
``sales`` (positive), ``patents`` (count), ``exporter`` (0/1) (outcomes);
``weight``. Note that ``age`` rises by one each year, so it cannot be
separated from firm and year fixed effects used together.
Args:
n_firms (int): Number of firms.
n_years (int): Number of years, starting in 2010.
Returns:
pd.DataFrame: One row per firm and year.
load_example_data1() -> pandas.DataFrame
Retrieves example data from a file included with the package.
Returns:
pd.DataFrame: Example data suitable for regression.
load_example_data2() -> pandas.DataFrame
Generates example data with one outcome and four regressors.
Returns:
pd.DataFrame: Example data suitable for regression.
load_example_data3() -> pandas.DataFrame
Generates example data with two outcomes, where the true effect of x1 is 0.4.
Returns:
pd.DataFrame: Example data suitable for regression.
----------------------------------------------------------------------
This is the User Guide documentation for the package.
----------------------------------------------------------------------
### Specification Curve
```{=html}
Specification curve analysis for Python
Specification Curve
Run every defensible regression and see all of the estimates in one chart.
```
Specification Curve is a Python package that performs specification curve analysis; it helps with the problem of the "garden of forking paths" (many defensible choices) when doing analysis by running many regressions and summarising the effects in an easy to digest chart. Estimation is done by [pyfixest](https://pyfixest.org/), so specifications can vary the outcome, the regressor of interest, controls, fixed effects, instruments, standard errors, weights, the sample and the estimator (OLS, Poisson, logit, probit and quantile regression).
::: {.sc-badges}
[](https://pypi.org/project/specification_curve/)
[](https://pypi.org/project/specification_curve)
[](https://opensource.org/licenses/MIT)
[](https://github.com/aeturrell/specification_curve/actions?workflow=Tests)
[](https://codecov.io/gh/aeturrell/specification_curve)
[](https://doi.org/10.5281/zenodo.17523510)
[](https://pepy.tech/projects/Specification_curve)
:::
## Introduction
When specifying a causal model, modellers have a number of options. These can be informed by field intelligence, priors, and even misguided attempts to find a significant result. Even with the best of intentions, research teams can reach entirely different conclusions using the same, or similar, data because of different choices made in preparing data or in modelling it.
Typically this happens when there isn't a clear way to do "feature engineering" on the data or the data are imperfect in some way. For example, you have a high frequency time series which needs to be aggregated to a lower frequency: you could take the maximum, the minimum, or the average over each high frequency time period. A different choice may be appropriate in different settings.
There's formal evidence that researchers really do make different decisions; one study [@silberzahn2018many] gave the same research question---whether soccer referees are more likely to give red cards to players with dark skin tones than to players with light skin tones---to 29 different teams. From the abstract of that paper:
> Analytic approaches varied widely across the teams, and the estimated effect sizes ranged from 0.89 to 2.93 (Mdn = 1.31) in odds-ratio units. Twenty teams (69%) found a statistically significant positive effect, and 9 teams (31%) did not observe a significant relationship. Overall, the 29 different analyses used 21 unique combinations of covariates. Neither analysts’ prior beliefs about the effect of interest nor their level of expertise readily explained the variation in the outcomes of the analyses. Peer ratings of the quality of the analyses also did not account for the variability.
So not only can different decisions made, but there seems to be no clearly identifiable reason for them!
There have since been a number of papers that find similar conclusions [@huntington2021influence].
Specification curves have been invented [@simonsohn2020specification] as a way to better grasp the garden of forking paths that analysts face, and help them show how sensitive their results are to alternative specifications.
More than one piece of software [@masur2019specr] has appeared to help researchers and analysts use this technique: **specification curve** is one such package!
## Quickstart
Here's an example of using **Specification Curve**. The example data are a made-up panel of firms, in which being `treated` raises log sales by 0.3, but firms that would have had high sales anyway are more likely to be treated.
We ask how the estimated effect of treatment depends on which controls are included, whether firm and year fixed effects are used, and how the standard errors are computed.
```{python}
#| echo: false
import matplotlib_inline.backend_inline
matplotlib_inline.backend_inline.set_matplotlib_formats("svg")
```
```{python}
import specification_curve as sc
df = sc.load_example_panel()
df.head()
```
```{python}
sco = sc.SpecificationCurve(
df,
y="log(sales)",
x="treated",
controls=["size", "leverage", "rd"],
fe=[None, "firm", "firm + year"],
vcov=["hetero", {"CRV1": "firm"}],
)
sco.fit()
sco.plot()
```
Each column is one specification. The top panel shows the estimated coefficient on `treated` with its confidence interval, and the panels underneath show which choices that specification makes. Markers are coloured blue when the coefficient is positive and significant (at the 0.05 level), red when it is negative and significant, and black otherwise.
Here, the specifications without firm fixed effects overstate the effect, and those with them are close to the true value.
You can also describe the specifications with a [pyfixest formula](https://pyfixest.org/quickstart.html). Its multiple-estimation operators define the choices: several outcomes (`y1 + y2 ~ ...`), `sw()` (one of these terms at a time) and `sw0()` (one at a time, or none). **Specification Curve** adds `comb()` and `comb0()`, which stand for every combination of their terms (with `comb0()` including none of them). The `x` argument names the coefficient to plot.
This formula describes the same specifications as above:
```{python}
sco = sc.SpecificationCurve(
df,
"log(sales) ~ treated + comb0(size, leverage, rd) | sw0(firm, firm + year)",
x="treated",
vcov=["hetero", {"CRV1": "firm"}],
)
sco.fit()
sco.plot()
```
The [features page](user-guide/features.qmd) shows everything else: other estimators, instrumental variables, samples, customising the chart, and statistical inference on the curve as a whole.
## Installation
You can install **Specification Curve** via uv:
```bash
$ uv add specification-curve
```
To install the development version from git, use:
```bash
$ pip install git+https://github.com/aeturrell/specification_curve.git
```
## License
Distributed under the terms of the [MIT license](https://opensource.org/licenses/MIT).
## Citing Specification Curve
You can find full citation information at the following link: [https://zenodo.org/badge/latestdoi/282989537](https://zenodo.org/badge/latestdoi/282989537).
Using **Specification Curve** in your paper? Let us know by raising an issue beginning with "citation".
## Similar Packages
In RStats, there is [specr](https://github.com/masurp/specr) (which inspired many design choices in this package) and [spec_chart](https://github.com/ArielOrtizBobea/spec_chart). Some of the example data in this package is the same as in specr, but please note that results have not been cross-checked across packages.
## References
::: {#refs}
:::
### Features
Let's see **Specification Curve**'s features in action.
```{python}
#| echo: false
import matplotlib_inline.backend_inline
matplotlib_inline.backend_inline.set_matplotlib_formats("svg")
```
## Basic use
We'll use an example data set that comes with the package: a synthetic panel of firms observed over several years. Some firms are `treated` in some years, and the question is what effect treatment has on `sales`. Because the data are made up, we know the answer: treatment raises log sales by 0.3. But firms with a high unobserved firm-level effect are more likely to be treated, so analyses that ignore this will overstate the effect.
```{python}
import specification_curve as sc
df = sc.load_example_panel()
df.head()
```
To create a specification curve, say which outcome (`y`) and which regressor of interest (`x`) you care about, and which choices should vary. Below, each of three control variables is switched in and out, giving eight specifications. `fit()` estimates them and `plot()` draws the curve.
```{python}
sco = sc.SpecificationCurve(
df,
y="log(sales)",
x="treated",
controls=["size", "age", "leverage"],
)
sco.fit()
sco.plot()
```
Each column is one specification. The top panel shows the estimated coefficient on `treated` with its 95% confidence interval, sorted from smallest to largest. The panel underneath shows which controls each specification includes. Markers are coloured blue when the coefficient is positive and significant (at the 0.05 level), red when it is negative and significant, and black otherwise.
All estimation is done by [pyfixest](https://pyfixest.org/), and every term is written in its formula syntax. That means transformations such as `log(sales)`, interactions such as `size:leverage`, and categorical terms such as `i(industry)` or `C(industry)` can be used anywhere a variable name can.
## Choosing controls
`controls` are included in every combination. Three more arguments shape which combinations are run:
- `always` names terms that are in every specification;
- `exclude` lists groups of controls that should never appear together (for example, two measures of the same thing);
- `min_controls` and `max_controls` limit how many controls a specification has.
```{python}
sco = sc.SpecificationCurve(
df,
y="log(sales)",
x="treated",
controls=["size", "age", "log(age)", "leverage"],
always="i(industry)",
exclude=[["age", "log(age)"]],
min_controls=1,
)
sco.fit()
sco.plot()
```
## Fixed effects
Fixed effects are given with `fe`, as a list of alternatives. Each is written as in a pyfixest formula, and `None` means no fixed effects. pyfixest absorbs fixed effects rather than estimating a coefficient for each one, so this stays fast even with many firms.
Here the specifications without firm fixed effects overstate the effect, as we expected, while those with them recover something close to the true value of 0.3. Passing `color_by="fe"` colours the curve by this choice to make that easy to see.
```{python}
sco = sc.SpecificationCurve(
df,
y="log(sales)",
x="treated",
controls=["size", "leverage", "rd"],
fe=[None, "firm", "firm + year"],
)
sco.fit()
sco.plot(color_by="fe")
```
## Standard errors
How standard errors are computed is often as contestable as what is in the model. `vcov` takes a list of alternatives, each in the form pyfixest accepts: for example `"iid"`, `"hetero"`, or `{"CRV1": "firm"}` for standard errors clustered by firm. Leaving it out uses pyfixest's default.
Specifications that differ only in their standard errors share a single model fit, so adding alternatives costs very little.
```{python}
sco = sc.SpecificationCurve(
df,
y="log(sales)",
x="treated",
controls=["size", "leverage"],
fe=["firm", "firm + year"],
vcov=["iid", "hetero", {"CRV1": "firm"}, {"CRV1": "state"}],
)
sco.fit()
sco.plot()
```
## Describing specifications with a formula
Instead of keyword arguments you can describe the specifications with a single pyfixest formula, using its multiple-estimation syntax to define the choices:
- several outcomes: `y1 + y2 ~ ...`
- `sw(a, b)`: `a` or `b`, one at a time
- `sw0(a, b)`: as `sw`, or neither
- `csw(a, b)`: `a`, then `a + b` (cumulative); `csw0` also includes neither
**Specification Curve** adds two operators of its own for the most common need in a specification curve:
- `comb(a, b)`: every combination of `a` and `b`
- `comb0(a, b)`: every combination, including neither
These work for fixed effects too, after the `|`. The `x` argument is still needed to say which coefficient to plot. The formula below describes the same specifications as the fixed effects example above.
```{python}
sco = sc.SpecificationCurve(
df,
"log(sales) ~ treated + comb0(size, leverage, rd) | sw0(firm, firm + year)",
x="treated",
)
sco.fit()
sco.plot()
```
The other arguments (`vcov`, `model`, `weights`, `subsets`, `exclude` and so on) can be combined with a formula.
## Several outcomes and regressors of interest
`y` and `x` take lists too. With several `x`, each specification uses one of them, and the coefficient plotted is the one on that variable. Below, the outcome is the log of sales or the log of sales plus one, and the regressor of interest is either treatment or eligibility for it.
```{python}
sco = sc.SpecificationCurve(
df,
y=["log(sales)", "log(sales + 1)"],
x=["treated", "eligible"],
controls=["size", "leverage"],
fe="firm + year",
)
sco.fit()
sco.plot()
```
Take care when the alternatives are on different scales: the coefficients of a model for `sales` and one for `log(sales)` are not comparable, and putting them on one chart would be misleading.
## Different samples
`subsets` runs every specification on several samples. It maps a name for each sample to a query (as used by `DataFrame.query`), with `None` meaning all rows.
```{python}
sco = sc.SpecificationCurve(
df,
y="log(sales)",
x="treated",
controls=["size", "leverage"],
fe="firm + year",
subsets={
"all": None,
"from 2012": "year >= 2012",
"industries 0 to 2": "industry <= 2",
},
)
sco.fit()
sco.plot()
```
The helper `subsets_by` builds one sample per level of a column, for the common case of running the analysis separately for each group:
```{python}
sc.subsets_by(df, "industry", include_all=True)
```
### Missing values
In the example data, `age` and `leverage` have some missing values. (Firm `age` rises by one each year, so it should not be combined with firm and year fixed effects together: pyfixest would drop it as collinear.) By default, within each sample every specification is estimated on the same rows: those that are complete for all the variables used by any specification. That way differences between estimates come from the specification and not from a changing sample. Passing `common_sample=False` lets each specification use every row it can instead. The number of observations each one used is in the results, and `show_nobs=True` adds it to the chart.
```{python}
sco = sc.SpecificationCurve(
df,
y="log(sales)",
x="treated",
controls=["size", "age", "leverage"],
common_sample=False,
)
sco.fit()
sco.plot(show_nobs=True)
```
## Other estimators
The default estimator is OLS (pyfixest's `feols`). The `model` argument selects others:
- `"fepois"`: Poisson regression, for counts and other non-negative outcomes;
- `"feglm"`: generalised linear models, with `family` set to `"logit"`, `"probit"`, `"gaussian"` or `"poisson"`;
- `"quantreg"`: quantile regression, with `quantile` (the median by default).
Here is a Poisson specification curve for the number of patents, with and without firm fixed effects:
```{python}
sco = sc.SpecificationCurve(
df,
y="patents",
x="treated",
controls=["size", "leverage"],
fe=[None, "firm", "firm + year"],
model="fepois",
vcov={"CRV1": "firm"},
)
sco.fit()
sco.plot()
```
`model`, `family` and `quantile` accept lists as well, so the estimator can itself be one of the choices. Below, whether a firm exports is modelled with a linear probability model, a logit and a probit. (The coefficients of these three are on different scales, so compare signs and significance, not sizes.)
```{python}
sco = sc.SpecificationCurve(
df,
y="exporter",
x="treated",
controls=["size", "leverage"],
model=["feols", "feglm"],
family=["logit", "probit"],
vcov="hetero",
)
sco.fit()
sco.plot()
```
And quantile regressions at three quantiles:
```{python}
sco = sc.SpecificationCurve(
df,
y="log(sales)",
x="treated",
controls=["size", "leverage"],
model="quantreg",
quantile=[0.25, 0.5, 0.75],
)
sco.fit()
sco.plot()
```
Not every estimator supports every option: quantile regression cannot be combined with fixed effects, weights or instruments, and instruments are only available with `"feols"`. If the choices you ask for include a combination that cannot be estimated you get an error that says which. Passing `skip_invalid=True` leaves those combinations out instead.
## Instrumental variables
`iv` gives the first stage in pyfixest's syntax, `"endogenous ~ instruments"`. As with the other arguments it can be a list, and `None` means no instruments. In the example data, `eligible` was assigned at random and makes treatment more likely, so it can be used as an instrument for `treated`.
Without firm fixed effects, OLS overstates the effect. The instrumental variable estimates are closer to the true value of 0.3 whether or not fixed effects are included, though they are much less precise.
```{python}
sco = sc.SpecificationCurve(
df,
y="log(sales)",
x="treated",
controls=["size", "leverage"],
fe=[None, "firm + year"],
iv=[None, "treated ~ eligible"],
vcov="hetero",
)
sco.fit()
sco.plot()
```
## Weights
`weights` names a column of weights, or a list of alternatives in which `None` means unweighted.
```{python}
sco = sc.SpecificationCurve(
df,
y="log(sales)",
x="treated",
controls=["size", "leverage"],
fe="firm + year",
weights=[None, "weight"],
)
sco.fit()
sco.plot()
```
## Working with the results
Before fitting, `grid` shows the specifications that will be run. After fitting, `results` has the same rows with the estimates added: the coefficient on the regressor of interest, its standard error, test statistic, p-value and confidence interval, and the number of observations.
```{python}
sco.results.drop(columns=["model", "family", "quantile", "iv", "subset"])
```
This is an ordinary data frame, so it can be filtered, summarised or saved (for example with `sco.results.to_csv("results.csv")`). Printing the object gives a summary:
```{python}
sco
```
A few options to `fit()` are useful here:
- `ci_levels=(0.9, 0.95)` stores (and plots) confidence intervals at several levels;
- `keep_models=True` keeps the fitted pyfixest models in `sco.models`, keyed by `spec_id`, for when you want a regression table or further tests for particular specifications;
- other keyword arguments, such as `ssc` or `fixef_rm`, are passed on to pyfixest.
If a specification cannot be estimated, it stays in the results with the reason in its `error` column and is left out of the chart. Pass `on_error="raise"` to stop at the first failure instead.
## Large numbers of specifications
The number of specifications multiplies quickly. Eight controls, two outcomes and three sets of fixed effects already give 1,536. `n_specs` tells you how many there are before you fit anything.
```{python}
sco = sc.SpecificationCurve(
df,
y=["log(sales)", "log(sales + 1)"],
x="treated",
controls=[
"size",
"leverage",
"rd",
"size:leverage",
"size:rd",
"exporter",
"patents",
"eligible",
],
fe=[None, "firm", "firm + year"],
)
sco.n_specs
```
There are two ways to keep this manageable. `n_jobs` estimates specifications in parallel, and `max_specs` estimates a random sample of them (with `seed` to make the sample reproducible). The chart switches to a more compact style when there are many specifications.
```{python}
sco.fit(max_specs=500, seed=0, n_jobs=2)
sco.plot()
```
```{python}
#| echo: false
#| output: asis
# The timings come from the most recent run of benchmarks/benchmark.py, which
# appends each run to benchmarks/results.csv. Re-run that script to update them.
from pathlib import Path
import pandas as pd
def _find(relative: str) -> Path:
for folder in [Path.cwd(), *Path.cwd().parents]:
if (folder / relative).exists():
return folder / relative
raise FileNotFoundError(relative)
_bench = pd.read_csv(_find("benchmarks/results.csv"))
_latest = _bench[_bench["run_at"] == _bench["run_at"].max()]
_sizes = sorted(_latest["n_rows"].unique())
_specs = _latest.groupby("scenario")["n_specs"].first()
_rows = [
("feols_batched", "`fit()`"),
("feols_batched_jobs4", "`fit(n_jobs=4)`"),
(
"feols_3vcov_batched",
f"`fit()` with three kinds of standard errors ({_specs['feols_3vcov_batched']:,} specifications)",
),
(
"fit_null_50",
f"`fit_null(n_boot=50)` for a curve of {_specs['fit_null_50']:,} specifications",
),
]
_meta = _latest.iloc[0]
_when = pd.Timestamp(_meta["run_at"]).strftime("%B %Y")
print(
f"As a guide to speed, here are timings for a curve of {_specs['feols_batched']:,} OLS "
"specifications with up to two sets of fixed effects. They come from "
"`benchmarks/benchmark.py` in the repository, last run in "
f"{_when} with version {_meta['package_version']} (pyfixest "
f"{_meta['pyfixest_version']}, Python {_meta['python_version']}) on {_meta['cpu']} "
f"with {_meta['cpu_count']} cores. Every run is logged in `benchmarks/results.csv`.\n"
)
print("| | " + " | ".join(f"{n:,} rows" for n in _sizes) + " |")
print("|---|" + "---|" * len(_sizes))
for _scenario, _label in _rows:
_times = _latest[_latest["scenario"] == _scenario].set_index("n_rows")["seconds"]
print(f"| {_label} | " + " | ".join(f"{_times[n]:.1f} s" for n in _sizes) + " |")
```
## Customising the chart
`plot()` has options for the most common changes:
- `sort_by` orders the specifications by a column of the results, or by several;
- `color_by` colours them by one of the choices instead of by significance;
- `panels` says which choices are shown underneath, and in what order;
- `labels` replaces names with nicer ones for display;
- `highlight` flags a preferred specification;
- `ci_levels` and `alpha` choose the confidence intervals drawn and the significance level used for colouring.
Often, in practice, you will have a *preferred* specification that you use as your main estimate. `highlight` takes the values that identify it (or its `spec_id`, or a query string).
```{python}
sco = sc.SpecificationCurve(
df,
y="log(sales)",
x="treated",
controls=["size", "leverage", "rd"],
fe=[None, "firm", "firm + year"],
vcov=["hetero", {"CRV1": "firm"}],
)
sco.fit(ci_levels=(0.9, 0.99))
sco.plot(
highlight={
"controls": ["size", "leverage", "rd"],
"fe": "firm + year",
"vcov": "CRV1: firm",
},
labels={
"fe": "Fixed effects",
"none": "None",
"firm": "Firm",
"firm + year": "Firm and year",
"vcov": "Standard errors",
"hetero": "Robust",
"CRV1: firm": "Clustered by firm",
},
)
```
```{python}
sco.plot(sort_by=["fe", "vcov"], color_by="vcov", panels=["fe", "controls"])
```
`plot()` returns the matplotlib figure and a dictionary of its axes (`"curve"` for the top panel, then one per panel underneath), so anything else can be changed afterwards. It never changes matplotlib's global settings, and `style=False` turns off the package's own fonts and sizes so that your settings apply.
To put a curve inside a figure of your own, pass a figure or subfigure as `fig`. The function `sc.plot_curve` does the same as the `plot` method for any results data frame, which is handy for results you saved earlier or combined from several curves.
```{python}
import matplotlib.pyplot as plt
fig = plt.figure(figsize=(14, 6))
left, right = fig.subfigures(1, 2)
results = sco.results
_, axes = sc.plot_curve(results[results["fe"].isna()], fig=left, panels=["controls"])
axes["curve"].set_title("No fixed effects")
_, axes = sc.plot_curve(results[results["fe"].notna()], fig=right, panels=["controls"])
axes["curve"].set_title("With fixed effects")
```
To save a chart, pass a path to `plot()`; the format is taken from the file extension.
```python
sco.plot("specification_curve.pdf")
```
## Inference under the null
A specification curve shows how much an estimate varies, but it is natural to also ask: taking all of these specifications together, how inconsistent are the results with there being no effect at all? Individual p-values cannot answer that, because the specifications are far from independent. [@simonsohn2020specification] propose answering it by resampling: re-estimate the whole curve many times on data in which the null hypothesis is true, and see how unusual the observed curve is by comparison.
`fit_null()` does this. By default it uses a bootstrap in which each specification's estimated effect is first removed from the outcome, so that the null holds, and then rows are resampled with replacement. Because this is panel data we resample whole firms, with `cluster="firm"`.
You should use a large number of resamples for results you rely on (500, say), but be warned that each one re-estimates every specification; `n_jobs` runs them in parallel. We use a small number here to keep things quick.
```{python}
sco = sc.SpecificationCurve(
df,
y="log(sales)",
x="treated",
controls=["size", "leverage"],
fe=["firm", "firm + year"],
vcov={"CRV1": "firm"},
)
sco.fit()
sco.fit_null(n_boot=50, cluster="firm", seed=0)
sco.null_summary
```
The summary has three tests. Each compares a feature of the observed curve with its distribution across the resamples: the median estimate, the share of estimates that are statistically significant, and Stouffer's Z, which combines the evidence of all the specifications. The p-value is the share of resamples at least as extreme as what was observed. Here all three are as small as they can be with this many resamples, as they should be: the effect is real.
The curve expected under the null, with a band covering 95% of the resamples, can be added to the chart:
```{python}
sco.plot(show_null_curve=True)
```
### Choosing a resampling method
There are three ways of generating data in which the null is true, chosen with `method`:
- **`"bootstrap"`** (the default for linear models). Each specification's estimated effect is removed from its outcome, $y^* = y - \hat\beta x$, so that the regressor of interest has no effect on $y^*$ by construction, and then rows (or whole clusters) are drawn with replacement.
- **`"bayesian-bootstrap"`**, the Bayesian bootstrap [@rubin1981bayesian; @andrews2026praise]. The null is imposed in the same way, but instead of drawing rows with replacement every row (or cluster) is kept and given a random weight from a flat Dirichlet distribution, scaled so that the weights average one. Where the ordinary bootstrap gives each row a whole number of copies, possibly zero, the Bayesian bootstrap gives it a positive, continuously varying weight.
- **`"permutation"`**. The regressor of interest is shuffled between rows (or clusters), which breaks any link between it and the outcome.
Which methods you can use depends on the estimators and designs in the curve:
| Specifications | `"bootstrap"` | `"bayesian-bootstrap"` | `"permutation"` |
| --- | :-: | :-: | :-: |
| Linear (`model="feols"`) | ✓ | ✓ | ✓ † |
| Instrumental variables (`feols` with `iv`) | ✓ | ✓ | ✗ |
| Poisson (`"fepois"`) and other GLMs (`"feglm"`) | ✗ | ✗ | ✓ † |
| Quantile regression (`"quantreg"`) | ✗ | ✗ | ✓ † |
| With `weights` | ✓ | ✓ ‡ | ✓ † |
| With `cluster` | ✓ (resamples clusters) | ✓ (one weight per cluster) | ✓ † (the regressor must be constant within clusters) |
| With `strata` | ✓ | ✓ | ✓ † |
† Only valid if the regressor of interest is as good as randomly assigned (within `strata`, if given).
‡ The random weights multiply your own; your weights must be analytic weights (pyfixest's default, `weights_type="aweights"`).
The two bootstraps need every specification to be linear because removing $\hat\beta x$ from the outcome only imposes the null in a linear model: in a Poisson model or a logit it does not. If the curve mixes linear and non-linear estimators, `method="auto"` raises an error and asks you to choose. Permutation is not available for instrumental-variable specifications, because shuffling the endogenous regressor but not its instruments breaks the first stage.
### The Bayesian bootstrap
The two bootstraps give very similar answers in large samples. The Bayesian bootstrap does better when resampling can produce data in which a specification cannot be estimated. With the ordinary bootstrap a resample may, by chance, contain no treated units, no rows of some fixed-effect level, or too few clusters; the specifications that cannot be estimated are missing from that resample, and a resample in which fewer than `min_valid` of them could be estimated is left out (with a warning). Because the Bayesian bootstrap never leaves a row out, that does not happen. Its null distributions are also smoother, which helps with few clusters.
Here is an extreme case: eight clusters, only one of them treated. Every ordinary bootstrap resample that misses the treated cluster has no variation in the regressor of interest.
```{python}
import warnings
import numpy as np
import pandas as pd
rng = np.random.default_rng(0)
small = pd.DataFrame({"group": np.repeat(np.arange(8), 5), "a": rng.normal(size=40)})
small["x"] = (small["group"] == 0).astype(float)
small["y"] = small["a"] + 0.5 * small["x"] + rng.normal(size=40)
sco_small = sc.SpecificationCurve(small, y="y", x="x", controls=["a"]).fit()
for method in ["bootstrap", "bayesian-bootstrap"]:
with warnings.catch_warnings(record=True) as caught:
warnings.simplefilter("always")
sco_small.fit_null(
n_boot=200, method=method, cluster="group", seed=0, progress=False
)
left_out = [str(w.message) for w in caught if "left out" in str(w.message)]
print(f"{method}: {left_out[0] if left_out else 'no resamples were left out.'}")
```
On the panel data from above, it is used like this:
```{python}
sco = sc.SpecificationCurve(
df,
y="log(sales)",
x="treated",
controls=["size", "leverage"],
fe=["firm", "firm + year"],
vcov={"CRV1": "firm"},
)
sco.fit()
sco.fit_null(n_boot=50, method="bayesian-bootstrap", cluster="firm", seed=0)
sco.null_summary
```
Although it has "Bayesian" in its name, the result here is the same kind of frequentist p-value as with the other methods: the Bayesian bootstrap is simply used as a way of generating resamples in which the null holds.
### Permutation
For non-linear estimators, permutation is the option. In the example data `eligible` was assigned at random and has no effect of its own on patents once treatment is controlled for, so here we expect large p-values:
```{python}
sco = sc.SpecificationCurve(
df,
y="patents",
x="eligible",
controls=["size", "leverage"],
always="treated",
model="fepois",
)
sco.fit()
sco.fit_null(n_boot=50, method="permutation", seed=0)
sco.null_summary
```
### Other options
Other options: `alternative="greater"` or `"less"` gives one-sided tests, `alpha` sets the significance level used for the share of significant estimates, and `keep_draws=True` keeps the estimates from every resample in `sco.null_draws`.
### Contributing
Thank you for your interest in improving this project. This project is
open-source under the [MIT license](https://opensource.org/licenses/MIT)
and welcomes contributions in the form of bug reports, feature requests,
and pull requests.
Here is a list of important resources for contributors:
- [Source Code](https://github.com/aeturrell/specification_curve)
- [Documentation](https://aeturrell.github.io/specification_curve/)
- [Issue Tracker](https://github.com/aeturrell/specification_curve/issues)
## How to report a bug
Report bugs on the [Issue Tracker](https://github.com/aeturrell/specification_curve/issues).
When filing an issue, make sure to answer these questions:
- Which operating system and Python version are you using?
- Which version of this project are you using?
- What did you do?
- What did you expect to see?
- What did you see instead?
The best way to get your bug fixed is to provide a test case, and/or
steps to reproduce the issue.
## How to request a feature
Request features on the [Issue Tracker](https://github.com/aeturrell/specification_curve/issues).
## How to set up your development environment
You need Python and the following tools:
- [uv](https://docs.astral.sh/uv/)
- [Nox](https://nox.thea.codes/)
- [Make](https://www.gnu.org/software/make/)
- [Quarto](https://quarto.org/)
Install the package with the existing development requirements:
```bash
$ uv sync --frozen
```
To also update packages, do not use the `--frozen` flag.
To build the documentation locally, you will also need [Make](https://www.gnu.org/software/make/) and [Quarto](https://quarto.org/) (these are non-Python dependencies).
The documentation is built with [Great Docs](https://posit-dev.github.io/great-docs/). Its configuration is in `great-docs.yml`, the pages are in `docs/user_guide/`, and the site's theme is in `docs/theme/indigo.css`. You can build the docs locally with `make`, which runs `uv run great-docs build` and writes the website to `great-docs/_site/`. Run `make preview` to serve it locally.
The timings table on the features page is built from `benchmarks/results.csv`. To update it, run `uv run python benchmarks/benchmark.py` from the repository root and commit the updated file. Each run is appended to that file along with the date, the package and pyfixest versions and a description of the machine, so it also serves as a performance log. The docs show the most recent run.
To publish new docs to GitHub Pages (where the documentation is displayed as web pages), it's `make publish`—but only devs with admin rights will be able to execute this.
## How to test the project
Run the full test suite:
```bash
$ uv run nox
```
List the available Nox sessions:
```bash
$ uv run nox --list-sessions
```
You can also run a specific Nox session. For example, invoke the unit
test suite like this:
```bash
$ uv run nox --session=tests
```
Unit tests are located in the `tests` directory, and are written using
the [pytest](https://pytest.readthedocs.io/) testing framework.
You may need to use, for example, `uv run nox` to ensure that the
tests are run in the right environment.
For the pre-commit checks, use
```bash
$ uv run pre-commit run --all-files
```
## How to submit changes
Open a [pull request](https://github.com/aeturrell/specification_curve/pulls) to
submit changes to this project.
Your pull request needs to meet the following guidelines for acceptance:
- The Nox test suite must pass without errors and warnings.
- Include unit tests. This project aims to maintain 98% code
coverage.
- If your changes add functionality, update the documentation
accordingly.
- Run make to generate the new documentation.
- Run the pre-commit suite before committing.
Feel free to submit early, though---we can always iterate on this.
To run linting and code formatting checks before committing your change,
you need to run the following
command:
```bash
$ uv run nox --session=pre-commit -- install
```
It is recommended to open an issue before starting work on anything.
This will allow a chance to talk it over with the owners and validate
your approach.
## How to create a package release
- Open a new branch with the version name
- Change the version in pyproject.toml using `uv version --bump `
- Commit the change with a new version label as the commit message (checking the tests pass)
- Head to GitHub and merge into main (again, if the CI works)
- The release GitHub Action then sees that the version has changed. It tags the merge commit with the new version, publishes the package to PyPI, publishes the release notes that Release Drafter has been collecting from merged pull requests, and publishes the documentation. Don't create the tag yourself: if the tag already exists, the action treats the push as an ordinary one.
Every other push to main publishes a developmental release (the next patch version with a `.dev` suffix) to TestPyPI and checks that the documentation builds.
If you ever need distributable files, you can use the `uv build` command locally.
## How to build the documentation manually and locally
You shouldn't need to publish the documentation because there's a GitHub action that covers it automatically whenever there's a new release. But to upload the documentation manually, it's
- Run `make` to build the documentation
- Run `make publish` to publish the documentation