Skip to content

Various Conformal Prediction methods implemented from scratch in pure NumPy for an educational purpose.

Notifications You must be signed in to change notification settings

joneswack/conformal-predictions-from-scratch

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

19 Commits
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Conformal Predictions from Scratch

The aim of this repository is to implement some popular Conformal Prediction methods from scratch using pure NumPy. The correctness is then verified with reference implementations from libraries like MAPIE. All of this is done with illustrative examples and thorough explanation of the code.

Why do we reimplement things from scratch? Because we want to understand the methods from ground up in order to use them with confidence in future projects where stakes may be high. This understanding is also needed for researchers who may think about coming up with novel methods on their own. Therefore, the purpose of this repository is educational.

🕵️‍♀️ What are Conformal Predictions?

Conformal Predictions have become a very popular way of equipping machine learning models with predictive uncertainty estimates. One reason for this is that they are model-agnostic meaning that previously trained (or even deployed) models can be enhanced with predictive uncertainties post hoc. Another advantage is that Conformal Predictions come with theoretical guarantees on how much of the future data to be predicted will actually fall inside the predictive bands (generally assuming exchangeability of the data). This may seem to good to be true, but makes a lot of sense when understanding how the predictions are "conformalized".

A great list of references and tutorials on the topic can be found here.

📝 The Structure of this Repository

The repository contains one Jupyter notebook per conformal prediction method, i.e., one way to obtain predictive error bands. All notebooks are located in the notebooks folder and new notebooks will be added over time.

📈 Regression

Notebook Title (Method Name) Description Academic Reference Reference Implementation
Split Conformal Prediction Conformal Predictions with fixed prediction bands obtained via (the default) absolute residual scores. Distribution-Free Predictive Inference for Regression (Section 2.2) MAPIE Regressor with AbsoluteConformityScore
Locally Weighted Conformal Prediction Conformal Predictions with variable prediction bands obtained via residual normalised scores. Distribution-Free Predictive Inference for Regression (Section 5.2) MAPIE Regressor with ResidualNormalisedScore
Conformalized Quantile Regression (CQR) Conformal Predictions with variable prediction bands obtained via quantile regression. Conformalized Quantile Regression MAPIE QuantileRegressor
Conformal Prediction Beyond Exchangeability Analysis of Conformal Predictions when the exchangeability assumption is violated. Conformal prediction beyond exchangeability Paper Code ZIP-File

About

Various Conformal Prediction methods implemented from scratch in pure NumPy for an educational purpose.

Topics

Resources

Stars

Watchers

Forks