Skip to contents

Refit continuous and categorical alternatives inside common folds, learn quantile boundaries on training observations only, and compare paired prediction losses. An advanced callback supports other estimators.

Usage

bin_validate(data, outcome, variables = NULL, schemes = NULL,
  controls = NULL, interaction = length(variables) == 2L,
  model = c("linear", "logit"), folds = 5L, group = NULL,
  fold_id = NULL, cluster = group, seed = 1L,
  fit_predict = NULL, reps = 499L, level = .95,
  spline_df = NULL, reference = NULL)

Arguments

data

Data frame. Original rows and columns are not modified.

outcome

Quoted numeric outcome column. Logit requires values zero and one.

variables

One or two quoted numeric focal columns for the built-in lm/glm workflow.

schemes

Named candidates. Each maps focal column names to recipe lists. A recipe supplies probs starting at zero and ending at one, or prespecified breaks, and optional separate_zero = TRUE. Full-data bin_diagnosis objects are deliberately not accepted. When spline_df is supplied, NULL or an empty list compares only the linear and spline models. The names continuous and, when enabled, spline are reserved.

controls

Optional quoted additive control columns for the built-in workflow.

interaction

Include the two focal variables' main effects and product. Requires two focal variables when TRUE.

model

"linear" reports RMSE and MAE; "logit" reports Brier score and log loss. Also selects lm or binomial glm for the built-in workflow.

folds

Number of folds, at least two. Assignment balances numbers of groups, not numbers of rows. Ignored when fold_id supplies the folds.

group

Optional column identifying units that must remain together. For dyadic data, construct an unordered country-pair identifier if both directions should remain together.

fold_id

Optional nonmissing fold label for every original row. At least two populated folds are needed. Each fold is tested using all other folds for training. This is not a forward-only temporal validation interface.

cluster

Zero, one, or two quoted columns used to resample paired held-out losses for score intervals. Defaults to group. NULL resamples rows. Clustering here does not change how a model is fitted.

seed

Nonnegative integer controlling folds and bootstrap draws. The call restores the user's random-number state.

fit_predict

Advanced function taking train and test data frames. Return a numeric matrix/data frame with one prediction per test row and at least two named model columns. The first model is the default reference. Alternatively return a list with predictions, optional details, and optional messages named by model. Names and order must agree across folds. The callback owns training-only preprocessing, model fitting, and failure diagnostics. Retain test-row order and use NA for unavailable predictions. The automatic focal/recipe/control arguments are unused with this callback.

reps

Number of paired bootstrap draws, zero to omit intervals or at least 99. Default 499. Increase for more stable interval endpoints.

level

Confidence level strictly between zero and one.

spline_df

NULL or an integer of at least 2. Adds a natural cubic spline comparator with this many degrees of freedom per focal variable. With two focal variables, interaction = TRUE includes all products of their spline basis terms plus main effects; FALSE uses additive splines. Controls are unchanged. Knots and boundary knots are estimated only on each training fold and reused for its test observations. Cannot be combined with fit_predict; that callback must implement its own spline if desired.

reference

NULL uses the first returned model (continuous in the built-in workflow). A model name, such as spline, selects the reference for paired losses and validation plots. All scores still use the same common observations.

Details

Training-only recipes. Quantiles are recalculated within each training fold. Tied cutpoints are collapsed; the first and last interval extend to negative and positive infinity so an unseen extreme receives a tail category. Prespecified breaks are honored without extension. Missing categories in test data produce unavailable predictions; uncovered training values stop that fit. Separate-zero recipes require nonnegative values and learn quantiles from positive training values. The built-in models use linear terms or factors, optional two-variable interactions, additive controls, and an optional natural cubic spline comparator. Spline degrees of freedom are fixed before validation; they are not tuned automatically. Natural splines extrapolate linearly beyond the training boundary knots on the link scale. Per-fold spline definitions are retained under $fold_details[[k]]$spline. More complex continuous models can be supplied through a callback. Using many spline terms with sparse support may produce fitting failures; the requested complexity is never silently reduced.

Common scoring. Every model is evaluated on the intersection of rows with finite predictions from every candidate. For logit, predictions outside [0,1] are unavailable. Coverage and per-fold failures are reported. Log loss clips probabilities at 1e-15 and 1-1e-15 solely to avoid infinite logarithms. Losses average over observations, not equally over groups. Folds are never selected according to their outcomes. Built-in missing model inputs are excluded once; callbacks receive all rows with finite outcomes and handle their own inputs.

Intervals. The same bootstrap weights apply to all candidate losses, allowing paired differences. One clustering column resamples whole clusters. Two columns independently resample each dimension and multiply their weights, the pigeonhole bootstrap. Repeated rows within a cluster intersection remain together. Percentile intervals use the requested confidence level. Predictions and fold assignments are held fixed: these intervals describe evaluation-sample uncertainty, not the full uncertainty from refitting models, learning cutpoints, or choosing splits. This approximation can be conservative for crossed data. They are not confidence intervals for individual predictions or causal effects.

No best binning scheme is selected automatically. A positive prediction gain does not establish a true threshold. Selecting and reporting a winner using the same cross-validation results requires nested validation or a new untouched test set for an unbiased final evaluation. Grouped folds are not future-year forecasts and do not ensure independence across groups sharing another unit.

Value

A bin_validation list with scores, paired differences, fold_scores, out-of-fold predictions, fold_details, diagnostics, excluded_rows, and a plain-language assessment. Negative loss differences favor the candidate over the selected reference. Settings include reference and spline_df.

References

Owen, A. B. (2007). The pigeonhole bootstrap. The Annals of Applied Statistics 1(2), 386–411. https://arxiv.org/abs/0712.1111

Examples

set.seed(12)
d <- data.frame(x = runif(400, -2, 2))
d$y <- ifelse(d$x <= 0, -1, 1) + rnorm(400, sd = .5)
v <- bin_validate(d, "y", "x", model = "linear", reps = 0, spline_df = 3,
  reference = "spline",
  schemes = list(boundary_zero = list(x = list(breaks = c(-Inf, 0, Inf)))))
print(v)
#> Held-out model comparison
#>          model metric     score lower upper
#>         spline   RMSE 0.5633603    NA    NA
#>     continuous   RMSE 0.6749570    NA    NA
#>  boundary_zero   RMSE 0.4784533    NA    NA
#>         spline    MAE 0.4438434    NA    NA
#>     continuous    MAE 0.5460606    NA    NA
#>  boundary_zero    MAE 0.3703798    NA    NA
#> 
#> Every scored prediction comes from a model fitted without that row.
#> All candidates are scored on the same 400 of 400 usable observations (100.0% coverage).
#> Lower prediction losses are better. Negative paired differences favor the candidate over spline.
#> continuous versus spline: RMSE difference 0.1116.
#> boundary_zero versus spline: RMSE difference -0.084907.
#> continuous versus spline: MAE difference 0.10222.
#> boundary_zero versus spline: MAE difference -0.073464.
#> The spline comparator uses 3 degrees of freedom per focal variable; knots and boundaries are learned separately in each training fold. Inspect $fold_details. Better scores than a linear model alone do not establish that bins beat this flexible comparator.
#> Score intervals resample the held-out losses with predictions and folds fixed; they do not include refitting or split-selection uncertainty.
#> Prediction gains do not prove causal thresholds or invalidate categories used for interpretation. Repeatedly choosing bins from these scores needs nested validation or a new test set.