Guidelines for Confirming a Deep Neural Network Picture Classifier
In this article, we will guide you through the process of conformalizing a deep learning image classifier using TensorFlow and the Julia package, . This approach is particularly useful for handling samples with high aleatoric uncertainty, such as handwritten digits.
Prerequisites
Before diving into the steps, ensure that you have the following prerequisites:
- A trained TensorFlow deep learning model for classifying handwritten digits.
- The Julia language installed on your machine.
- The Julia packages , , and installed.
Steps to Conformalize Your TensorFlow Model
1. Export your TensorFlow Model
Save your trained TensorFlow model in a format accessible from Julia, such as TensorFlow SavedModel. Alternatively, you can export predictions on a calibration dataset saved as arrays (e.g., CSV, HDF5).
2. Load Model Output or Produce Predictions in Julia
Use Julia packages like or to invoke the TensorFlow model inside Julia for new image data. Or, load stored predictions on calibration and test datasets.
3. Apply Conformal Prediction in Julia
a. Define a Nonconformity Score Function
Create a function that calculates the nonconformity score based on your classifier’s output. Common choices include 1 minus the predicted probability of the true class or margins between top class probabilities.
b. Calculate Nonconformity Scores
Compute nonconformity scores on a calibration set (a holdout portion of labeled data).
c. Create a Conformal Predictor Object
Use these scores to determine quantiles that will be used to make prediction sets with guaranteed coverage. Then, apply the conformal predictor to new samples to obtain prediction sets (possibly multiple classes) rather than single point predictions, with a set level of confidence.
d. Make Predictions
For a new sample with predicted probabilities from TensorFlow model:
will contain a subset of possible classes guaranteed to contain the true label with 90% confidence.
Wrap-up
In this article, we have provided a guide on how to conformalize a deep learning image classifier in Julia using the package. This approach can be applied to TensorFlow models, and we have provided a pseudo-code example to help you get started.
For a more detailed, step-by-step tutorial, including data handling, TensorFlow model saving/loading in Julia, or example code, please let us know. Additionally, you can build the MLP in TensorFlow.js using a macro to make the transition from Julia to TensorFlow.js as seamless as possible.
Data and cloud computing technology, such as Julia, enables us to apply artificial intelligence techniques like conformal prediction to TensorFlow models. In our guide, we utilize Julia's packages to conformalize a deep learning image classifier, defining a nonconformity score function, calculating nonconformity scores, creating a conformal predictor object, and making predictions with guaranteed coverage.
