Conformalizing a Deep Learning Image Classifier in Julia: A Step-by-Step Guide
Modifying Deep Image Classifiers through Conformal Prediction
In this article, we will guide you through the process of conformalizing a deep learning image classifier in Julia, specifically for predicting handwritten digits from the MNIST dataset.
Step 1: Model Training and Predictions (Python)
First, train your model using Keras (Python), as follows:
```python
from tensorflow.keras.models import Sequential from tensorflow.keras.layers import Dense, Flatten
model = Sequential([ Flatten(input_shape=(28, 28)), Dense(128, activation='relu'), Dense(64, activation='relu'), Dense(10) ])
model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy']) model.fit(x_train, y_train, epochs=10)
calibration_preds = model.predict(x_val) test_preds = model.predict(x_test) ```
Step 2: Save Predictions & Calibration Labels
Save the softmax outputs and true labels to disk, e.g., as files:
```python import numpy as np
np.save("calibration_predictions.npy", calibration_preds) np.save("calibration_labels.npy", y_val) np.save("test_predictions.npy", test_preds) np.save("test_labels.npy", y_test) ```
Step 3: Load Predictions and Labels in Julia
In Julia, use or to load the numpy saved files:
Step 4: Using ConformalPrediction.jl
Assuming your softmax outputs are the model's predicted class probabilities, use conformity scores based on those.
Here is a rough outline of how to do it:
```julia using ConformalPrediction
calib_labels_int = Int.(calib_labels) .+ 1 test_labels_int = Int.(test_labels) .+ 1
conformity_scores = [1 - calib_preds[i, calib_labels_int[i]] for i in 1:length(calib_labels_int)]
alpha = 0.1
qhat = quantile(conformity_scores, 1 - alpha)
pred_sets = [findall(p -> p >= 1 - qhat, test_preds[i, :]) for i in 1:size(test_preds, 1)]
coverage = mean([test_labels_int[i] in pred_sets[i] for i in 1:length(test_labels_int)])
println("Empirical coverage: ", coverage) ```
Additional Notes:
- Calibration set: The calibration (validation) set independent from training data is essential for valid conformal prediction.
- Alpha: The error level parameter controls the size of the prediction sets.
- Prediction sets: The output is a set of labels with guaranteed coverage at least (1 - \alpha).
Summary
- Train in Keras; save softmax outputs and true labels on a calibration set + test set.
- Load predictions in Julia using .
- Calculate conformity scores and quantile threshold using calibration set.
- Use thresholding on test set predictions to get conformal prediction sets with guarantees.
- Evaluate coverage.
For a more Julia-native workflow, you could re-implement or retrain your model in Julia, but working across Python and Julia this way is common too. If you need assistance with a specific part of the pipeline or a more complete Julia example, feel free to ask!
This article is inspired by the paper titled "A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification" by Angelopoulos and Bates (2021). The spread is wider for the adaptive approach, reflecting that it effectively distinguishes between easy and hard inputs.
Data-and-cloud-computing allows for the storage and computation of large datasets like the MNIST dataset used in the guide, facilitating the training of deep learning image classifiers like the one described.
The use of artificial-intelligence, specifically conformal prediction, provides the ability to generate sets of labels with guaranteed coverage at least (1 - \alpha), enhancing the reliability and accuracy of predictions in deep learning image classifiers.
