SciKeras Benchmarks¶

SciKeras wraps Keras Models, but does not alter their performance since all of the heavy lifting still happens within Keras/Tensorflow. In this notebook, we compare the performance and accuracy of a pure-Keras Model to the same model wrapped in SciKeras.

Table of contents¶

1. Setup
2. Dataset
3. Define Keras Model
4. Keras benchmarks
5. SciKeras benchmark

1. Setup¶

[1]:

try:
    import scikeras
except ImportError:
    !python -m pip install scikeras[tensorflow]

Silence TensorFlow logging to keep output succinct.

[2]:

import warnings
from tensorflow import get_logger
get_logger().setLevel('ERROR')
warnings.filterwarnings("ignore", message="Setting the random state for TF")

[3]:

import numpy as np
from scikeras.wrappers import KerasClassifier, KerasRegressor
import keras

2. Dataset¶

We will be using the MNIST dataset available within Keras.

[4]:

(x_train, y_train), (x_test, y_test) = keras.datasets.mnist.load_data()
# Scale images to the [0, 1] range
x_train = x_train.astype("float32") / 255
x_test = x_test.astype("float32") / 255
# Make sure images have shape (28, 28, 1)
x_train = np.expand_dims(x_train, -1)
x_test = np.expand_dims(x_test, -1)
# Reduce dataset size for faster benchmarks
x_train, y_train = x_train[:2000], y_train[:2000]
x_test, y_test = x_test[:500], y_test[:500]

Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz

 0/11490434 ━━━━━━━━━━━━━━━━━━━━ 0s 0s/step: </pre>
textbf{ 0/11490434} textcolor{ansi-white}{━━━━━━━━━━━━━━━━━━━━} textbf{0s} 0s/step: end{sphinxVerbatim}

[1m 0/11490434[0m [37m━━━━━━━━━━━━━━━━━━━━[0m [1m0s[0m 0s/step

 8257536/11490434 ━━━━━━━━━━━━━━━━━━━━ 0s 0us/step: </pre>
textbf{ 8257536/11490434} textcolor{ansi-green}{━━━━━━━━━━━━━━}textcolor{ansi-white}{━━━━━━} textbf{0s} 0us/step: end{sphinxVerbatim}

[1m 8257536/11490434[0m [32m━━━━━━━━━━━━━━[0m[37m━━━━━━[0m [1m0s[0m 0us/step

11490434/11490434 ━━━━━━━━━━━━━━━━━━━━ 0s 0us/step: </pre>
textbf{11490434/11490434} textcolor{ansi-green}{━━━━━━━━━━━━━━━━━━━━} textbf{0s} 0us/step: end{sphinxVerbatim}

[1m11490434/11490434[0m [32m━━━━━━━━━━━━━━━━━━━━[0m[37m[0m [1m0s[0m 0us/step

3. Define Keras Model¶

Next we will define our Keras model (adapted from keras.io):

[5]:

num_classes = 10
input_shape = (28, 28, 1)


def get_model():
    model = keras.Sequential(
        [
            keras.Input(input_shape),
            keras.layers.Conv2D(32, kernel_size=(3, 3), activation="relu"),
            keras.layers.MaxPooling2D(pool_size=(2, 2)),
            keras.layers.Conv2D(64, kernel_size=(3, 3), activation="relu"),
            keras.layers.MaxPooling2D(pool_size=(2, 2)),
            keras.layers.Flatten(),
            keras.layers.Dropout(0.5),
            keras.layers.Dense(num_classes, activation="softmax"),
        ]
    )
    model.compile(
        loss="sparse_categorical_crossentropy", optimizer="adam"
    )
    return model

4. Keras benchmarks¶

[6]:

fit_kwargs = {"batch_size": 128, "validation_split": 0.1, "verbose": 0, "epochs": 5}

[7]:

from sklearn.metrics import accuracy_score
from scikeras.utils.random_state import tensorflow_random_state

[8]:

from time import time

with tensorflow_random_state(seed=0):  # we force a TF random state to be able to compare accuracy
    model = get_model()
    start = time()
    model.fit(x_train, y_train, **fit_kwargs)
    print(f"Training time: {time()-start:.2f}")
    y_pred = np.argmax(model.predict(x_test), axis=1)
print(f"Accuracy: {accuracy_score(y_test, y_pred)}")

Training time: 3.56

 1/16 ━━━━━━━━━━━━━━━━━━━━ 0s 48ms/step: </pre>
textbf{ 1/16} textcolor{ansi-green}{━}textcolor{ansi-white}{━━━━━━━━━━━━━━━━━━━} textbf{0s} 48ms/step: end{sphinxVerbatim}

[1m 1/16[0m [32m━[0m[37m━━━━━━━━━━━━━━━━━━━[0m [1m0s[0m 48ms/step

16/16 ━━━━━━━━━━━━━━━━━━━━ 0s 5ms/step: </pre>
textbf{16/16} textcolor{ansi-green}{━━━━━━━━━━━━━━━━━━━━} textbf{0s} 5ms/step: end{sphinxVerbatim}

[1m16/16[0m [32m━━━━━━━━━━━━━━━━━━━━[0m[37m[0m [1m0s[0m 5ms/step

16/16 ━━━━━━━━━━━━━━━━━━━━ 0s 5ms/step: </pre>
textbf{16/16} textcolor{ansi-green}{━━━━━━━━━━━━━━━━━━━━} textbf{0s} 5ms/step: end{sphinxVerbatim}

[1m16/16[0m [32m━━━━━━━━━━━━━━━━━━━━[0m[37m[0m [1m0s[0m 5ms/step

Accuracy: 0.89

5. SciKeras benchmark¶

[9]:

clf = KerasClassifier(
    model=get_model,
    random_state=0,
    **fit_kwargs
)

[10]:

start = time()
clf.fit(x_train, y_train)
print(f"Training time: {time()-start:.2f}")
y_pred = clf.predict(x_test)
print(f"Accuracy: {accuracy_score(y_test, y_pred)}")

Training time: 3.79
Accuracy: 0.89

As you can see, the overhead for SciKeras is <1 sec, and the accuracy is identical.