{
 "cells": [
  {
   "cell_type": "raw",
   "id": "a9ca2758",
   "metadata": {},
   "source": [
    "<a href=\"https://colab.research.google.com/github/adriangb/scikeras/blob/docs-deploy/refs/heads/master/notebooks/AutoEncoders.ipynb\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\">Run in Google Colab</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bff57800",
   "metadata": {},
   "source": [
    "# Autoencoders in SciKeras\n",
    "\n",
    "Autencoders are an approach to use nearual networks to distill data into it's most important features, thereby compressing the data.\n",
    "We will be following the [Keras tutorial](https://blog.keras.io/building-autoencoders-in-keras.html) on the topic, which goes much more in depth and breadth than we will here.\n",
    "You are highly encouraged to check out that tutorial if you want to learn about autoencoders in the general sense.\n",
    "\n",
    "## Table of contents\n",
    "\n",
    "* [1. Setup](#1.-Setup)\n",
    "* [2. Data](#2.-Data)\n",
    "* [3. Define Keras Model](#3.-Define-Keras-Model)\n",
    "* [4. Training](#4.-Training)\n",
    "* [5. Explore Results](#5.-Explore-Results)\n",
    "* [6. Deep AutoEncoder](#6.-Deep-AutoEncoder)\n",
    "\n",
    "## 1. Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "256653ba",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-04-11T22:25:59.911293Z",
     "iopub.status.busy": "2024-04-11T22:25:59.910992Z",
     "iopub.status.idle": "2024-04-11T22:26:02.198202Z",
     "shell.execute_reply": "2024-04-11T22:26:02.197373Z"
    }
   },
   "outputs": [],
   "source": [
    "try:\n",
    "    import scikeras\n",
    "except ImportError:\n",
    "    !python -m pip install scikeras"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9258cff8",
   "metadata": {},
   "source": [
    "Silence TensorFlow logging to keep output succinct."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "0805ff36",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-04-11T22:26:02.201668Z",
     "iopub.status.busy": "2024-04-11T22:26:02.200858Z",
     "iopub.status.idle": "2024-04-11T22:26:02.205681Z",
     "shell.execute_reply": "2024-04-11T22:26:02.204963Z"
    }
   },
   "outputs": [],
   "source": [
    "import warnings\n",
    "from tensorflow import get_logger\n",
    "get_logger().setLevel('ERROR')\n",
    "warnings.filterwarnings(\"ignore\", message=\"Setting the random state for TF\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "6d40044c",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-04-11T22:26:02.208418Z",
     "iopub.status.busy": "2024-04-11T22:26:02.207957Z",
     "iopub.status.idle": "2024-04-11T22:26:02.524940Z",
     "shell.execute_reply": "2024-04-11T22:26:02.524301Z"
    }
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from scikeras.wrappers import KerasClassifier, KerasRegressor\n",
    "import keras"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a6b7c526",
   "metadata": {},
   "source": [
    "## 2. Data\n",
    "\n",
    "We load the dataset from the Keras tutorial. The dataset consists of images of cats and dogs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "0b28adac",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-04-11T22:26:02.528383Z",
     "iopub.status.busy": "2024-04-11T22:26:02.528036Z",
     "iopub.status.idle": "2024-04-11T22:26:02.779117Z",
     "shell.execute_reply": "2024-04-11T22:26:02.778465Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(60000, 784)\n",
      "(10000, 784)\n"
     ]
    }
   ],
   "source": [
    "from keras.datasets import mnist\n",
    "import numpy as np\n",
    "\n",
    "\n",
    "(x_train, _), (x_test, _) = mnist.load_data()\n",
    "x_train = x_train.astype('float32') / 255.\n",
    "x_test = x_test.astype('float32') / 255.\n",
    "x_train = x_train.reshape((len(x_train), np.prod(x_train.shape[1:])))\n",
    "x_test = x_test.reshape((len(x_test), np.prod(x_test.shape[1:])))\n",
    "print(x_train.shape)\n",
    "print(x_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b871a2bb",
   "metadata": {},
   "source": [
    "## 3. Define Keras Model\n",
    "\n",
    "We will be defining a very simple autencoder. We define _three_ model architectures:\n",
    "\n",
    "1. An encoder: a series of densly connected layers culminating in an \"output\" layer that determines the encoding dimensions.\n",
    "2. A decoder: takes the output of the encoder as it's input and reconstructs the original data.\n",
    "3. An autoencoder: a chain of the encoder and decoder that directly connects them for training purposes.\n",
    "\n",
    "The only variable we give our model is the encoding dimensions, which will be a hyperparemter of our final transformer.\n",
    "\n",
    "The encoder and decoder are views to the first/last layers of the autoencoder model.\n",
    "They'll be directly used in `transform` and `inverse_transform`, so we'll create some SciKeras models with those layers\n",
    "and save them as in `encoder_model_` and `decoder_model_`. All three models are created within `_keras_build_fn`.\n",
    "\n",
    "For a background on chaining Functional Models like this, see [All models are callable](https://keras.io/guides/functional_api/#all-models-are-callable-just-like-layers) in the Keras docs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "425cde73",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-04-11T22:26:02.781990Z",
     "iopub.status.busy": "2024-04-11T22:26:02.781734Z",
     "iopub.status.idle": "2024-04-11T22:26:02.791253Z",
     "shell.execute_reply": "2024-04-11T22:26:02.790636Z"
    }
   },
   "outputs": [],
   "source": [
    "from typing import Dict, Any\n",
    "\n",
    "from sklearn.base import TransformerMixin\n",
    "from sklearn.metrics import mean_squared_error\n",
    "from scikeras.wrappers import BaseWrapper\n",
    "\n",
    "\n",
    "class AutoEncoder(BaseWrapper, TransformerMixin):\n",
    "    \"\"\"A class that enables transform and fit_transform.\n",
    "    \"\"\"\n",
    "\n",
    "    encoder_model_: BaseWrapper\n",
    "    decoder_model_: BaseWrapper\n",
    "    \n",
    "    def _keras_build_fn(self, encoding_dim: int, meta: Dict[str, Any]):\n",
    "        n_features_in = meta[\"n_features_in_\"]\n",
    "\n",
    "        encoder_input = keras.Input(shape=(n_features_in,))\n",
    "        encoder_output = keras.layers.Dense(encoding_dim, activation='relu')(encoder_input)\n",
    "        encoder_model = keras.Model(encoder_input, encoder_output)\n",
    "\n",
    "        decoder_input = keras.Input(shape=(encoding_dim,))\n",
    "        decoder_output = keras.layers.Dense(n_features_in, activation='sigmoid', name=\"decoder\")(decoder_input)\n",
    "        decoder_model = keras.Model(decoder_input, decoder_output)\n",
    "        \n",
    "        autoencoder_input = keras.Input(shape=(n_features_in,))\n",
    "        encoded_img = encoder_model(autoencoder_input)\n",
    "        reconstructed_img = decoder_model(encoded_img)\n",
    "\n",
    "        autoencoder_model = keras.Model(autoencoder_input, reconstructed_img)\n",
    "\n",
    "        self.encoder_model_ = BaseWrapper(encoder_model, verbose=self.verbose)\n",
    "        self.decoder_model_ = BaseWrapper(decoder_model, verbose=self.verbose)\n",
    "\n",
    "        return autoencoder_model\n",
    "    \n",
    "    def _initialize(self, X, y=None):\n",
    "        X, _ = super()._initialize(X=X, y=y)\n",
    "        # since encoder_model_ and decoder_model_ share layers (and their weights)\n",
    "        # X_tf here come from random weights, but we only use it to initialize our models\n",
    "        X_tf = self.encoder_model_.initialize(X).predict(X)\n",
    "        self.decoder_model_.initialize(X_tf)\n",
    "        return X, X\n",
    "\n",
    "    def initialize(self, X):\n",
    "        self._initialize(X=X, y=X)\n",
    "        return self\n",
    "\n",
    "    def fit(self, X, *, sample_weight=None) -> \"AutoEncoder\":\n",
    "        super().fit(X=X, y=X, sample_weight=sample_weight)\n",
    "        # at this point, encoder_model_ and decoder_model_\n",
    "        # are both \"fitted\" because they share layers w/ model_\n",
    "        # which is fit in the above call\n",
    "        return self\n",
    "\n",
    "    def score(self, X) -> float:\n",
    "        # Note: we use 1-MSE as the score\n",
    "        # With MSE, \"larger is better\", but Scikit-Learn\n",
    "        # always maximizes the score (e.g. in GridSearch)\n",
    "        return 1 - mean_squared_error(self.predict(X), X)\n",
    "\n",
    "    def transform(self, X) -> np.ndarray:\n",
    "        X: np.ndarray = self.feature_encoder_.transform(X)\n",
    "        return self.encoder_model_.predict(X)\n",
    "\n",
    "    def inverse_transform(self, X_tf: np.ndarray):\n",
    "        X: np.ndarray = self.decoder_model_.predict(X_tf)\n",
    "        return self.feature_encoder_.inverse_transform(X)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "62e46a22",
   "metadata": {},
   "source": [
    "Next, we wrap the Keras Model with Scikeras. Note that for our encoder/decoder estimators, we do not need to provide a loss function since no training will be done.\n",
    "We do however need to have the `fit_model` and `encoding_dim` so that these will be settable by `BaseWrapper.set_params`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "7b5efc10",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-04-11T22:26:02.793637Z",
     "iopub.status.busy": "2024-04-11T22:26:02.793352Z",
     "iopub.status.idle": "2024-04-11T22:26:02.797210Z",
     "shell.execute_reply": "2024-04-11T22:26:02.796670Z"
    }
   },
   "outputs": [],
   "source": [
    "autoencoder = AutoEncoder(\n",
    "    loss=\"binary_crossentropy\",\n",
    "    encoding_dim=32,\n",
    "    random_state=0,\n",
    "    epochs=5,\n",
    "    verbose=False,\n",
    "    optimizer=\"adam\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2d7d1215",
   "metadata": {},
   "source": [
    "## 4. Training\n",
    "\n",
    "To train the model, we pass the input images as both the features and the target.\n",
    "This will train the layers to compress the data as accurately as possible between the encoder and decoder.\n",
    "Note that we only pass the `X` parameter, since we defined the mapping `y=X` in `KerasTransformer.fit` above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "36b9f2b6",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-04-11T22:26:02.799987Z",
     "iopub.status.busy": "2024-04-11T22:26:02.799756Z",
     "iopub.status.idle": "2024-04-11T22:26:20.841354Z",
     "shell.execute_reply": "2024-04-11T22:26:20.840686Z"
    }
   },
   "outputs": [],
   "source": [
    "_ = autoencoder.fit(X=x_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4f39666c",
   "metadata": {},
   "source": [
    "Next, we round trip the test dataset and explore the performance of the autoencoder."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "f20faea6",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-04-11T22:26:20.844730Z",
     "iopub.status.busy": "2024-04-11T22:26:20.844262Z",
     "iopub.status.idle": "2024-04-11T22:26:21.655626Z",
     "shell.execute_reply": "2024-04-11T22:26:21.654935Z"
    }
   },
   "outputs": [],
   "source": [
    "roundtrip_imgs = autoencoder.inverse_transform(autoencoder.transform(x_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "38f796b8",
   "metadata": {},
   "source": [
    "## 5. Explore Results\n",
    "\n",
    "Let's compare our inputs to lossy decoded outputs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "82d3dc2e",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-04-11T22:26:21.658975Z",
     "iopub.status.busy": "2024-04-11T22:26:21.658528Z",
     "iopub.status.idle": "2024-04-11T22:26:23.531507Z",
     "shell.execute_reply": "2024-04-11T22:26:23.530851Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAABiEAAAE/CAYAAAAg+mBzAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8fJSN1AAAACXBIWXMAAA9hAAAPYQGoP6dpAABNpklEQVR4nO3dabhcVZk37h0gBBIChJAAEQKEGZkhCAoIgooICoqM+tog4gAOrYID2CK02I02iraKtK8KiIpMigiRQWRQEMOkzBCGEBJIICEDSSCE/D/1/3Wv58GzOald5yS57+viw3quVfusVK1ae1ctav8GLFq0aFEFAAAAAADQYcv19QAAAAAAAIClk00IAAAAAACgFTYhAAAAAACAVtiEAAAAAAAAWmETAgAAAAAAaIVNCAAAAAAAoBU2IQAAAAAAgFbYhAAAAAAAAFqxQpNOr7zySjV58uRq6NCh1YABA9oeE/3YokWLqtmzZ1ejRo2qlluu3T0s847/1a15Z87xj8w7us05lr5graPbrHX0BWsdfcG8o9ucY+kLTeddo02IyZMnV+utt17HBseS78knn6zWXXfdVv+GeUep7XlnzpEx7+g251j6grWObrPW0ResdfQF845uc46lL/Q07xptiw0dOrRjA2Lp0I05Yd5RantOmHNkzDu6zTmWvmCto9usdfQFax19wbyj25xj6Qs9zYlGmxB+VkOpG3PCvKPU9pww58iYd3Sbcyx9wVpHt1nr6AvWOvqCeUe3OcfSF3qaE4KpAQAAAACAVtiEAAAAAAAAWmETAgAAAAAAaIVNCAAAAAAAoBU2IQAAAAAAgFbYhAAAAAAAAFphEwIAAAAAAGiFTQgAAAAAAKAVNiEAAAAAAIBW2IQAAAAAAABasUJfDwCWVp/73OdCbeWVVw61bbbZptY++OCDGx3/Bz/4Qa19yy23hD7nn39+o2MBAAAAALTBLyEAAAAAAIBW2IQAAAAAAABaYRMCAAAAAABohU0IAAAAAACgFYKpoQMuvPDCUGsaMF165ZVXGvX7yEc+Umvvs88+oc8NN9wQahMnTuzVuKC06aabhtoDDzwQap/61KdC7bvf/W4rY6L/GjJkSK39jW98I/Qp17Wqqqrbb7+91n7f+94X+jzxxBOLOToAAGBZNWzYsFAbPXp0r46VfTb513/911r7nnvuCX0eeuihULv77rt7NQboj/wSAgAAAAAAaIVNCAAAAAAAoBU2IQAAAAAAgFbYhAAAAAAAAFohmBp6oQyi7m0IdVXFIN/f//73oc+YMWNC7YADDqi1N9poo9DnyCOPDLWvf/3rr3WIkNp+++1DLQtWnzRpUjeGQz+3zjrr1Nof/vCHQ59s/uy444619v777x/6fO9731vM0bGk2WGHHULt0ksvDbUNNtigC6P55972trfV2vfff3/o8+STT3ZrOCwhyuu8qqqqyy+/PNSOP/74UDv77LNr7YULF3ZuYLRm5MiRofarX/0q1P785z+H2jnnnFNrP/744x0bVyetttpqobbHHnvU2uPGjQt9FixY0NqYgKXfO9/5zlr7Xe96V+iz5557htrGG2/cq7+XBUyvv/76tfagQYMaHWv55Zfv1RigP/JLCAAAAAAAoBU2IQAAAAAAgFbYhAAAAAAAAFohEwJ6sNNOO4XaQQcd1OPj7r333lDL7j347LPP1tpz5swJfVZcccVQu/XWW2vtbbfdNvQZPnx4j+OE3tpuu+1C7YUXXgi1yy67rAujoT8ZMWJEqJ177rl9MBKWVm9/+9tDrem9dbutvLf/0UcfHfocdthh3RoO/VR5zfb973+/0eP++7//O9R+/OMf19rz5s3r/cBozbBhw2rt7LNDlqHwzDPPhFp/zIDIxn777beHWnnNUGZBVVVVPfLII50bGK/ZqquuGmplzuBWW20V+uyzzz6hJt+DxVHmYB533HGhT5Y7t/LKK9faAwYM6OzACptuummrx4cllV9CAAAAAAAArbAJAQAAAAAAtMImBAAAAAAA0AqbEAAAAAAAQCv6bTD1wQcfHGpZwMzkyZNr7fnz54c+F1xwQag9/fTToSbwisw666wTamWQURYkl4VmTpkypVdj+OxnPxtqW265ZY+P+93vftervweZMnDu+OOPD33OP//8bg2HfuKTn/xkqB144IGhtvPOO3fk7+2xxx6httxy8f+puPvuu0Ptxhtv7MgY6K4VVoiXq/vtt18fjKR3yiDWz3zmM6HPkCFDQu2FF15obUz0P+Xatu666zZ63C9+8YtQyz4P0bfWXHPNULvwwgtr7TXWWCP0yQLKP/GJT3RuYC06+eSTQ23DDTcMtY985CO1ts/kfevII48Mta997Wuhtt566/V4rCzQ+rnnnuvdwKCK58ZPfepTfTSS/+eBBx4Itez7IZYeG2+8cahl5/mDDjqo1t5zzz1Dn1deeSXUzj777FD705/+VGsvqedKv4QAAAAAAABaYRMCAAAAAABohU0IAAAAAACgFTYhAAAAAACAVvTbYOozzjgj1DbYYINeHasMu6qqqpo9e3ao9cfwmEmTJoVa9tyMHz++G8NZJv32t78NtTKIJptP06dP79gYDjvssFAbOHBgx44PTWy++ea1dhakWoYssvT71re+FWpZwFanvOc972lUe+KJJ0Lt0EMPrbXLwGD6p7322ivUdt1111DLro/6g2HDhtXaW265ZegzePDgUBNMvfQaNGhQqJ100km9Otb5558faosWLerVsWjPDjvsEGpZQGXp1FNPbWE07Xj9619fa3/2s58NfS677LJQc+3Yd8qQ36qqqm9/+9uhNnz48FBrss5897vfDbXjjz++1u7kZ2b6pzKwNwuTLkN3q6qqxo0bF2ovvvhirT1z5szQJ7t+Kj+3Xn311aHPPffcE2p/+ctfQu3OO++stefNm9doDCwZttpqq1Ar163ss2cWTN1bb3jDG0Lt5ZdfrrUffPDB0Ofmm28OtfL99tJLLy3m6BaPX0IAAAAAAACtsAkBAAAAAAC0wiYEAAAAAADQin6bCfHhD3841LbZZptQu//++2vtLbbYIvRpeg/OXXbZpdZ+8sknQ5/11lsv1Joo799VVVU1bdq0UFtnnXV6PNbEiRNDTSZEd2X3Gu+UE044IdQ23XTTHh+X3a8wq0FvnXjiibV29j6wFi3drrzyylBbbrl2/3+G5557rtaeM2dO6LP++uuH2oYbbhhqt912W629/PLLL+boaEN5L9Zf/OIXoc+ECRNC7fTTT29tTIvj3e9+d18PgX5m6623DrUdd9yxx8dlnyeuuuqqjoyJzhk5cmSovfe97+3xcR/60IdCLfu82B+U+Q9VVVXXXnttj4/LMiGybD2643Of+1yorbHGGh07fpnFVVVVte+++9baX/va10KfLEuir+9jTjNZZmCZv7DtttuGPgcddFCj49966621dvZd3+OPPx5qo0ePrrWz7NU2M+3oe9n3yccdd1yoZevWqquu2uPxn3rqqVC76aabau3HHnss9Cm/Y6mqPLdw5513rrWztXq//fYLtbvvvrvWPvvss0OfbvJLCAAAAAAAoBU2IQAAAAAAgFbYhAAAAAAAAFphEwIAAAAAAGhFvw2mvu666xrVSuPGjWt0/GHDhoXadtttV2tnYSBjx45tdPzS/PnzQ+2hhx4KtTJoOwsbycIYWXLtv//+tfapp54a+qy44oqhNnXq1Fr7i1/8Yugzd+7cxRwdy6oNNtgg1HbaaadaO1vDXnjhhbaGRB9485vfXGtvttlmoU8W4tbbYLcsKKsMs5s5c2bo85a3vCXUTjrppB7/3sc+9rFQ+8EPftDj42jXySefXGtnIYdlsGVV5aHl3ZZdt5XvI8GHNAkpzpTrIf3Tf/3Xf4Xa+9///lArP2tedNFFrY2p03bfffdQW2uttWrtn/70p6HPz372s7aGRAPrr79+rX3UUUc1etzf/va3UHvmmWdq7X322afRsVZbbbVaOwvHvuCCC0Lt6aefbnR8uif7juLnP/95qJVB1Keffnro0yTYPpOFUGcmTpzYq+Oz5PrhD39Ya2fh52uuuWajY5XfRf/9738Pfb70pS+FWvY9cOmNb3xjqGWfUX/84x/X2uX311UV1+Wqqqrvfe97tfYll1wS+kybNq2nYXaMX0IAAAAAAACtsAkBAAAAAAC0wiYEAAAAAADQCpsQAAAAAABAK/ptMHXbZsyYEWrXX399j49rEo7dVBZKVwZmZ4EnF154YcfGQN8rw36zgKdMOQ9uuOGGjo0JyiDVTDcDjGhfFkb+y1/+stZuGt6VeeKJJ2rtLBTrq1/9aqjNnTv3NR+7qqrq2GOPDbURI0bU2meccUbos9JKK4Xaf//3f9faCxYs6HFMNHPwwQeH2n777VdrP/LII6HP+PHjWxvT4sgC0csg6j/+8Y+hz/PPP9/SiOiP9thjjx77vPTSS6GWzS/6n0WLFoVaFkg/efLkWjt7zbtt5ZVXDrUsbPPjH/94qJX/7qOPPrpzA6MjyiDToUOHhj433XRTqGWfC8rrpcMPPzz0yebORhttVGuvvfbaoc9vfvObUHvHO94RatOnTw812rPKKqvU2l/84hdDn/333z/Unn322Vr7m9/8ZujT5Hofqir/rHbiiSeG2jHHHFNrDxgwIPTJvs/4wQ9+EGrf+MY3au0XXnihx3E2NXz48FBbfvnlQ+2UU06ptceNGxf6rL/++h0bV1v8EgIAAAAAAGiFTQgAAAAAAKAVNiEAAAAAAIBW2IQAAAAAAABascwGU3fbyJEjQ+373/9+qC23XH1f6NRTTw19BDAtuX7961+H2tve9rYeH3feeeeF2sknn9yJIUFq66237rFPFurLkmuFFeIlQW+DqG+44YZQO+yww2rtMqRucWTB1F//+tdD7cwzz6y1Bw8eHPpk8/ryyy+vtSdMmPBah8ireN/73hdq5euSXS/1B1mY+5FHHhlqCxcurLX//d//PfQRdr70euMb39ioVspCD++6665ODIl+4p3vfGetffXVV4c+WWh9FprZW2Xg8J577hn67LLLLo2OdfHFF3diSLRo0KBBtXYWov6tb32r0bHmz59fa//kJz8JfbJz/JgxY3o8dhZS3B+C25d1Bx54YK39hS98IfSZOHFiqO2+++619syZMzs6LpYt2XnqhBNOCLUyiPqpp54Kfd773veG2m233db7wRXKgOn11lsv9Mm+67vyyitDbdiwYT3+vSx8+/zzz6+1s+uKbvJLCAAAAAAAoBU2IQAAAAAAgFbYhAAAAAAAAFohE6JLjjvuuFAbMWJEqM2YMaPWfvDBB1sbE+1aZ511Qi27B3B5b87sPunZ/aPnzJmzGKOD/ye71+9RRx0VanfeeWetfc0117Q2JpYc48ePD7Wjjz461DqZAdFEmeNQVfF+/WPHju3WcKiqarXVVgu1Jvca7+T9zzvp2GOPDbUsR+X++++vta+//vrWxkT/09t1pr/Oe3p21llnhdpee+0VaqNGjaq199hjj9Anu7/zu971rsUY3T8/fpYRkHn00UdD7Utf+lJHxkR7Dj/88B77lFklVZXnGjax00479epxt956a6j57Nv3muQZlZ8Xq6qqJk2a1MZwWEaVOQtVFfPXMi+//HKoveENbwi1gw8+ONQ233zzHo8/b968UNtiiy3+abuq8s/Ia621Vo9/L/PMM8+EWvldYl/n0PklBAAAAAAA0AqbEAAAAAAAQCtsQgAAAAAAAK2wCQEAAAAAALRCMHUL3vSmN4XaF77whUaPPfDAA2vte+65pxNDog9ccskloTZ8+PAeH/ezn/0s1CZMmNCRMUFmn332CbU11lgj1MaNG1drz58/v7Ux0T8st1zP/69CFujVH2RhnuW/p8m/r6qq6pRTTqm1P/CBD/R6XMuyQYMGhdrrXve6UPvFL37RjeEsto022qhRP9dyy7amwazPP/98rS2Yesl1++23h9o222wTatttt12tve+++4Y+J5xwQqhNmzYt1M4999zXMML/5/zzz6+177777kaP+/Of/xxqPq/0f+X5NQs5Hzt2bKhloaxbb711rX3QQQeFPsOGDQu1cq3L+nz4wx8OtXKuVlVV3XfffaFGe7LA3lK2jn3lK1+ptX/zm9+EPnfddVevx8Wy5Q9/+EOoXX/99aFWfscxevTo0Oc73/lOqC1atKjHMWRB2FlgdhNNQ6hfeeWVWvuyyy4LfT75yU+G2pQpU3o1rrb4JQQAAAAAANAKmxAAAAAAAEArbEIAAAAAAACtsAkBAAAAAAC0QjB1C/bbb79QGzhwYKhdd911oXbLLbe0MibalYV67bDDDo0e+8c//rHWLoOboG3bbrttqGWBTBdffHE3hkMf+ehHPxpqZQDWkuSAAw4Ite23377Wzv59Wa0MpqZ3Zs+eHWpZEGEZ4LrGGmuEPtOnT+/YuJoYOXJkqDUJaKyqqrr55ps7PRz6sd12263WPuKIIxo9bubMmbX2pEmTOjYm+t6MGTNCrQzSzII1P//5z7c2pqqqqjFjxtTaAwYMCH2ydfpzn/tcW0OiRddee22tXa47VRUDp6sqD4BuEt5a/r2qqqrjjjuu1r7iiitCn0022STUssDV7NqV9owYMaLWzq6ZBw0aFGr/9m//VmuffPLJoc/ZZ58darfeemuoleHCjzzySOhz7733hlrp9a9/fahl38U5F/c/8+bNC7WDDjoo1FZfffVa+wtf+ELo86Y3vSnUnnvuuVCbOHFirZ3N8+w7lZ133jnUeuucc86ptb/0pS+FPs8//3zH/l5b/BICAAAAAABohU0IAAAAAACgFTYhAAAAAACAVsiE6ICVV1651t53331Dn5deeinUsnv/L1iwoHMDozXDhw+vtbP7sWU5IJnyPqtz5szp9bigibXXXrvW3n333UOfBx98MNQuu+yy1sZE38syFPqj8n60VVVVW265Zahl63IT06ZNCzXn5s7I7uE6YcKEUHvve99ba//ud78Lfc4888yOjWurrbYKtfI+6RtssEHo0+R+2FW1ZGer8NqV14jLLdfs//m65ppr2hgO/FPlvdqzdS3LpcjOlfR/ZZ7SIYccEvpkGXCrrbZaj8f+7ne/G2rZ3Jk/f36tfemll4Y+2b3b3/72t4faRhttVGtn1xR0zje/+c1a+zOf+UyvjpOdFz/+8Y83qrUpW9fK/M6qqqrDDjusC6NhcZX5CNm60knnnXdeqDXJhMgy87L31k9/+tNae+HChc0H14/4JQQAAAAAANAKmxAAAAAAAEArbEIAAAAAAACtsAkBAAAAAAC0QjB1B5xwwgm19vbbbx/6jBs3LtT+/Oc/tzYm2vXZz3621h47dmyjx/36178OtSygHNr0L//yL7X2yJEjQ5+rrrqqS6OB1+akk04KteOOO65Xx3r88cdD7YMf/GCoTZw4sVfHp2fZOXDAgAG19jvf+c7Q5xe/+EXHxvDss8+GWhnOuuaaa/b6+GWQHEu3gw8+uMc+ZVhiVVXVD3/4wxZGA//P+973vlD7P//n/9TaWUDmc88919qY6FvXXnttqGVr2BFHHBFq5TpWhpxXVQyhzpx22mmhtsUWW4Tau971rlAr/2Z2DUfnlMG+F154Yejz85//PNRWWKH+teN6660X+mRh1d02YsSIUMveDyeffHKt/e///u+tjYn+6cQTTwy13gaWf/SjHw21Tn7O6W/6/p0OAAAAAAAslWxCAAAAAAAArbAJAQAAAAAAtMImBAAAAAAA0ArB1K9RFo745S9/udaeNWtW6HPqqae2Nia67zOf+UyvHnf88ceH2pw5cxZ3OPCarL/++j32mTFjRhdGAj278sora+3NNtusY8e+7777Qu3mm2/u2PHp2QMPPBBqhxxySK293XbbhT4bb7xxx8Zw8cUX99jn3HPPDbUjjzyy0fHnzZv3msfEkmHdddcNtSzAtTRp0qRQGz9+fEfGBK/mHe94R499rrjiilC744472hgO/VQWVp3VOiU7R2aBx1kw9V577VVrr7HGGqHP9OnTF2N0/KOFCxfW2tl5a9NNN+3xOHvvvXeoDRw4MNROOeWUUBs7dmyPx++kAQMGhNqOO+7Y1THQ94455phauwwnr6oYwJ659957Q+3SSy/t/cCWQH4JAQAAAAAAtMImBAAAAAAA0AqbEAAAAAAAQCtsQgAAAAAAAK0QTP1PDB8+PNS+853vhNryyy9fa5chmlVVVbfeemvnBsYSKwvLWrBgQUeOPXPmzEbHzkKfVltttR6Pv/rqq4dabwO6y1Crqqqqz3/+87X23Llze3Vserb//vv32Oe3v/1tF0ZCf5IFry23XM//r0KToMuqqqpzzjmn1h41alSjx5VjeOWVVxo9rokDDjigY8eiPXfddVejWpseffTRXj92q622qrXvueeexR0O/cQb3/jGUGuybv76179uYTTwz2Xn6xdeeKHW/q//+q9uDQde1a9+9atQy4KpDz300Fr7+OOPD31OPfXUzg2Mjrjuuusa9dtuu+1CrQymfvnll0Ofn/zkJ6H2P//zP7X2pz/96dDniCOOaDQulm4777xzqJXnxlVWWaXRsebMmVNrf/SjHw19XnzxxdcwuiWfX0IAAAAAAACtsAkBAAAAAAC0wiYEAAAAAADQCpkQ/6DMdhg3blzos+GGG4bahAkTau0vf/nLnR0YS42//e1vrR37oosuCrUpU6aE2lprrRVq5f00+8LTTz9da3/ta1/ro5EsXXbbbbdQW3vttftgJPR3P/jBD0LtjDPO6PFxV1xxRag1yW3obbbD4mRCnH322b1+LMu2LDMlq2VkQCy9svy40rPPPhtqZ511VhvDgf9fdt/p7DPA1KlTa+077rijtTFBU9m1XnZN+u53v7vW/spXvhL6/PKXvwy1hx56aDFGR7dcffXVoVZ+R7DCCvErzQ9/+MOhtvHGG9fae+65Z6/HNWnSpF4/lv4vywwcOnRoj48rM5aqKmbZ/OlPf+r9wJYSfgkBAAAAAAC0wiYEAAAAAADQCpsQAAAAAABAK2xCAAAAAAAArRBM/Q822mijWnvHHXds9LjPfOYztXYZVM3S58orr6y1y1CsvvC+972vY8d6+eWXQ61JGOzll18eauPHj2/0N2+66aZG/XhtDjrooFBbfvnla+0777wz9LnxxhtbGxP906WXXhpqJ5xwQq09YsSIbg3nVU2bNi3U7r///lA79thjQ23KlCmtjIml36JFixrVWLa8/e1v77HPxIkTQ23mzJltDAf+f1kwdbZm/e53v+vxWFkg57Bhw0Itm+vQKXfddVeo/du//Vut/Y1vfCP0Of3000PtAx/4QK09b968xRscrciu73/1q1/V2occckijY+2111499lm4cGGoZWvkF77whUZ/k/4vO7+deOKJvTrWBRdcEGp//OMfe3WspZlfQgAAAAAAAK2wCQEAAAAAALTCJgQAAAAAANAKmxAAAAAAAEArltlg6vXXXz/Urr766h4fV4Z0VlVVXXHFFR0ZE0uO97znPbV2Fl4zcODAXh379a9/fagdeuihvTrWj3/841B7/PHHe3zcJZdcEmoPPPBAr8ZA9wwePDjU9ttvvx4fd/HFF4daFszF0u2JJ54ItcMOO6zWPvDAA0OfT33qU20NKfW1r30t1L73ve91dQwse1ZaaaVG/YRbLr2y67qNNtqox8fNnz8/1BYsWNCRMcHiKq/3jjzyyNDnX//1X0Pt3nvvDbUPfvCDnRsYNHDeeefV2h/5yEdCn/Jze1VV1amnnlpr/+1vf+vswOiI7Jrq05/+dK29yiqrhD477bRTqI0cObLWzr4TOf/880PtlFNO+eeDZImRzZX77rsv1Jp8j5etGeXcJOeXEAAAAAAAQCtsQgAAAAAAAK2wCQEAAAAAALRimc2EOPbYY0Nt9OjRPT7uhhtuCLVFixZ1ZEwsuc4444xWj3/EEUe0enyWDtk9pmfMmBFql19+ea191llntTYmlmw33njjP21XVZ6nlJ1jDzjggFq7nIdVVVXnnHNOqA0YMKDWzu7dCW076qijQu35558PtdNOO60Lo6EvvPLKK6E2fvz4UNtqq61q7UceeaS1McHiOuaYY2rtD33oQ6HP//2//zfUrHX0B9OmTau199lnn9Anu/f/5z//+Vo7y0Khf3rmmWdq7fLzRVVV1Qc+8IFQ22WXXWrtr371q6HP1KlTF3N09GdvectbQm3dddcNtSbf72ZZSVkGGJFfQgAAAAAAAK2wCQEAAAAAALTCJgQAAAAAANAKmxAAAAAAAEArlolg6t122y3UPvGJT/TBSADakwVTv/GNb+yDkbAsGTduXKMaLMn++te/htqZZ54Zatdff303hkMfWLhwYaiddNJJoVYGGt5+++2tjQlezfHHHx9qp556aqjdeOONtfYPfvCD0GfGjBmh9tJLLy3G6KAdEydODLVrr7021N71rnfV2ltuuWXoc99993VuYHTV+eef36jGsuW0004LtSYh1FVVVd/4xjdqbdf7veeXEAAAAAAAQCtsQgAAAAAAAK2wCQEAAAAAALTCJgQAAAAAANCKZSKYevfddw+1VVZZpcfHTZgwIdTmzJnTkTEBALBkOOCAA/p6CPRDkydPDrWjjz66D0YCdTfffHOoveUtb+mDkUDfOvjgg0Pt7rvvrrU33njj0EcwNSxd1lhjjVAbMGBAqE2dOjXUvv3tb7cxpGWSX0IAAAAAAACtsAkBAAAAAAC0wiYEAAAAAADQCpsQAAAAAABAK5aJYOqmyoCivffeO/SZPn16t4YDAAAAQC/MmjUr1DbccMM+GAnQl84888xGtdNOOy3UpkyZ0sqYlkV+CQEAAAAAALTCJgQAAAAAANAKmxAAAAAAAEArlolMiK9//euNagAAAAAALB2+9a1vNarRLr+EAAAAAAAAWmETAgAAAAAAaIVNCAAAAAAAoBWNNiEWLVrU9jhYwnRjTph3lNqeE+YcGfOObnOOpS9Y6+g2ax19wVpHXzDv6DbnWPpCT3Oi0SbE7NmzOzIYlh7dmBPmHaW254Q5R8a8o9ucY+kL1jq6zVpHX7DW0RfMO7rNOZa+0NOcGLCowdbVK6+8Uk2ePLkaOnRoNWDAgI4NjiXPokWLqtmzZ1ejRo2qlluu3bt5mXf8r27NO3OOf2Te0W3OsfQFax3dZq2jL1jr6AvmHd3mHEtfaDrvGm1CAAAAAAAAvFaCqQEAAAAAgFbYhAAAAAAAAFphEwIAAAAAAGiFTQgAAAAAAKAVNiEAAAAAAIBW2IQAAAAAAABaYRMCAAAAAABohU0IAAAAAACgFTYhAAAAAACAVtiEAAAAAAAAWmETAgAAAAAAaIVNCAAAAAAAoBU2IQAAAAAAgFbYhAAAAAAAAFphEwIAAAAAAGiFTQgAAAAAAKAVNiEAAAAAAIBW2IQAAAAAAABaYRMCAAAAAABohU0IAAAAAACgFTYhAAAAAACAVtiEAAAAAAAAWmETAgAAAAAAaIVNCAAAAAAAoBU2IQAAAAAAgFbYhAAAAAAAAFphEwIAAAAAAGiFTQgAAAAAAKAVNiEAAAAAAIBW2IQAAAAAAABaYRMCAAAAAABohU0IAAAAAACgFTYhAAAAAACAVqzQpNMrr7xSTZ48uRo6dGg1YMCAtsdEP7Zo0aJq9uzZ1ahRo6rllmt3D8u84391a96Zc/wj845uc46lL1jr6DZrHX3BWkdfMO/oNudY+kLTeddoE2Ly5MnVeuut17HBseR78sknq3XXXbfVv2HeUWp73plzZMw7us05lr5graPbrHX0BWsdfcG8o9ucY+kLPc27RttiQ4cO7diAWDp0Y06Yd5TanhPmHBnzjm5zjqUvWOvoNmsdfcFaR18w7+g251j6Qk9zotEmhJ/VUOrGnDDvKLU9J8w5MuYd3eYcS1+w1tFt1jr6grWOvmDe0W3OsfSFnuaEYGoAAAAAAKAVNiEAAAAAAIBW2IQAAAAAAABaYRMCAAAAAABohU0IAAAAAACgFSv09QAAWLoNGDAg1BYtWtQHIwEAAACg2/wSAgAAAAAAaIVNCAAAAAAAoBU2IQAAAAAAgFbYhAAAAAAAAFohmBp6sNxyca9u8ODBtfYBBxwQ+nz6058OtdGjR4fa8ssvX2s/+eSToc/UqVND7a9//Wut/cgjj4Q+48aNC7UZM2aE2sKFC2vtLDS4t0HCQomXHuVrWc7dqsrfL6+88kqovfzyy50bGEuNbE4NHDiw1i7Xq6qqqgULFrQ2JgAAXrvsc2BZy/o0PVb5GSP7zAGdVs7F7PNvpvwOpJPfucCSwi8hAAAAAACAVtiEAAAAAAAAWmETAgAAAAAAaIVNCAAAAAAAoBWCqVmmlaFCK6wQ3xKjRo0KtY9+9KO19hFHHBH6jBgxItQGDRoUamWQUfa4LIj1jW98Y639wAMPhD5ZWPXtt98eamVIcCcDkZqGjQlh6jvZa5TN1de97nW19s477xz6DB8+PNTKEPWqqqr77ruv1n7hhRdCH+FyS4Zs/mQB0yNHjqy1P/jBD4Y+W265ZaiV69Pll18e+lx55ZWh9uKLL8bBskRqEmz5Wmql7BzbJuc7snk5cODAWnvVVVcNfbJz8/PPPx9q8+bNq7WdT5cMWbhpVsvWkPI17q/rTHZ9UMrma3/99/DPNQne7e25u6ri+6O3nzsFBFNV+Xq78sorh9rWW29da6+33nqhz7Bhw0LtrrvuqrWffPLJ0GfmzJmhNnfu3FCDJZVfQgAAAAAAAK2wCQEAAAAAALTCJgQAAAAAANAKmxAAAAAAAEArBFOzTCvDq7LAv3XWWSfUhgwZUmtnoYDZsbJaGbqaBWplAatlkO+f/vSn0CcLO3rppZdCrc3grSxcrmloGH0ne93Keb/JJpv02Keq8tD03gbJ0V3l65K9TiusEC8lRo0aFWqf+MQnau1DDjkk9MmCWMv19cYbbwx9hK4uXcrg0iwUcLXVVgu1rF95jn3uuedCnzLEt6qahbxm74cs1LB8j2TzNQvH7nZgNt2TzZ2RI0fW2oceemjos8EGG4TalVdeGWo33HBDrZ3NcaGr3ZUFMg8fPrzWHjNmTOiTnWOzzx2TJ0+utWfNmhX6lOthp5X/xqFDh4Y+2RyeMmVKrZ2t09l6aA53T5Pw6N6GOy9OKHQ5hux9ln3+XrBgQY/Hzs7n2edo16DdVb4ugwcPDn222267UDvqqKNq7e233z70GTFiRKituOKKoTZw4MCehpnOn/Iz8Q9/+MPQ56abbgq1xx9/PNTaXs+hLX4JAQAAAAAAtMImBAAAAAAA0AqbEAAAAAAAQCv6bSZEdu+17J6Y5T3+svsVzp8/P9Sye+z3h3vvNrm3Ip1T3sMxmxePPfZYqJX333300UdDn/L+plVVVQ8//HCP/bJ76m+++eahtscee9Ta2f1+s/dR9h4xz/q/3mYm9PZerNl6OHfu3Fo7u/96tk4/++yzoVauy+6n2j+Vc6PpWrHrrruG2nvf+95aO8vbyZT3VH3LW94S+lx++eWhNn369FCz1vU/TXJG1l9//dBnl112CbXsXtA333xzrT116tTQJ1t/ersmZWNYe+21a+2VVlop9MmuGcp7upu/7cnu3VxanPuWN1HmLB133HGhzxprrBFqWQbP+PHja+3sGpH2ZPcLLzM/qipmI+22226hT5b/cO2114ZauYZkczpbn5qc57NadvzVV1+91h47dmzok32mufXWW2vtMvfu1Wq0I7uWz3KYsvvnl5544olQ6+R6VM7N7Nyd5T80mfdyDfte9nyX328cffTRoc9Xv/rVUCvncNPXspPn+R122KHWPvHEE0Ofv//97x37e/S9Jrk12fm0SX7ckvq5wC8hAAAAAACAVtiEAAAAAAAAWmETAgAAAAAAaIVNCAAAAAAAoBV9EkydhcAMHTq01t53331DnzKwrapi2F8WlHr33XeH2p133hlqTz75ZK09Z86c0CcLaiprWbhxJgsmfN3rXtfjsR588MFQEzjXGS+//HKoZfPggQceqLWzwOmnnnoq1LLXswyUaRoc/bGPfazWHjNmTOiThbKfddZZoVYGDtO3+mvo2Zprrllrb7zxxqFPFqY1YcKEUHvppZdq7SU1WGlZk83N4cOHh1q5PlVVVa277rq1dhbClR1/8ODBtfb+++/fY5+qqqpPf/rToTZ58uRaOwtgp7uy13yttdaqtT/1qU+FPuutt16oXXrppaH27LPP1trZebG3IdRNA1w322yzWnvvvfcOfX75y1+GWrmeZuGa/HPZ/GpaK3XyPJWtf2VI8ejRo0OfbJzZOlZeuzrHtqt8XcqA5qqqqve85z2hdsQRR9Ta5XVWVVXV73//+1C75ZZbQq0Msu+L81s5/ne9612hTxmgXVVxnW7yeYnOyL7b2GmnnULtK1/5SqiVIbtZePgFF1wQauVn0eeeey706e3rnT2ut+fOpp/JaU8W4rvVVlvV2oceemjoM2TIkB6Pnb2W2bo5bdq0UCs/x5bfZVZVPn/Kx/31r38NfR566KFQy76jojOaXhOWa2UZdF5VVfXmN7851I499thae4sttgh9snl+zz33hNrFF19ca2fXB08//XSolWtg088vbfFLCAAAAAAAoBU2IQAAAAAAgFbYhAAAAAAAAFphEwIAAAAAAGhF68HUWahHFta13Xbb1dpjx44NfbbccstQK8MLs6C3DTfcMNT222+/UCuDCUeNGtXoWOW/8Yknngh9sjDpLICkPH4ZollVVfWtb30r1K6//vpaW9hm72SBLFk42jPPPFNrlyFDVdXZ1+Dwww8PtT322KPWzsa+8847h1oWnFQGm7cdTCPo65/LnovyOWv7OVxxxRVDrVw3y6Dhqqqqc889N9SyoDqv95IpO8dmIdS77LJLqGXnvN5YeeWVQ61cD6uqqi666KJQu/rqq2vtMhyxqqrq+eefDzXn1PZkc6oMbctCMssg06qqqrvvvjvUpk+fXmtnIdSdXE+zY5X/nm222Sb0ya73HnjggVpbMPVr1/R1bBJM3UmrrLJKqJXn2GzNzObAeeedF2pZADudka1ZZS0LrNxoo41CbdCgQbX2I488Evp8+9vfDrWJEyeGWra2ldoOYN9kk01q7a233rrRGMp1Wghre8rnvwz5raqquuSSS0JtnXXW6fFY2bVSdo1YhgafccYZoU/5Wbuqms3NTn6+8FmlZ9n7OVsjeyv7PFqeB7Nw3vK7jaqqqoEDB9bajz32WOhz5ZVXhto111wTauVcHzZsWOhTfr9ZVVW10kor1drXXXdd6DNnzpxQo3fK+VnOgarKz9fZa7fvvvvW2m9729tCn+w8X/7NbF3JznnZGMpz6lFHHRX6jB8/PtTKz7vZNUQ2hibXFb3hlxAAAAAAAEArbEIAAAAAAACtsAkBAAAAAAC0ovVMiOyeV9k92sp7sv3hD38IfWbPnh1qY8aM6XEM2f0Js3vVlfkS2T3osseV98rK7ied3Zs1u89XeY/17J6xm222WajdcMMNtbb7V3dONofLDIhOPt/ZPQWz+2mW95GdMmVK6PPLX/4y1GbMmLEYo+uMtu4vtyxpev/qJvczzY61xhprhNqee+5Zaw8ePDj0mTp1aq/G0FST+4xmf899XTtj1113DbUTTzwx1LJ7bpaye09m58qylj0uO1dmOVLrr79+rb3mmmuGPllORHmfbufYzskyPg477LBaO3ud7r///lCbMGFCqJWvVW/vX9w0S2LVVVcNte23377WzrKZsvv/dypHhbrenhd7mx2SPW6DDTYItWyel+bOnRtqt956a6/Glelt/tSydI5t8m9dYYX4EXvkyJGhVj6/N954Y+iTrWu9vY7uZNZNttYdfPDBtXb2PPztb38LtTI7bFmaT91Wnn9+8pOfhD5ZLmamfJ2yHMXse5999tmn1s6y47JxZfcxL8/x5k53Zdf7TebPtGnTQi37DJDlbpbX5FluTlZrIvuMkZ13S7NmzQq1SZMmhVp5zZu9P3xP0jvZNXM5F9/97neHPgcddFCPj6uqeM7LPjtkr+dzzz1Xa2cZcNn3c9l3gmXuUvZZd/To0aFWzs8sYzj7Drut6z2/hAAAAAAAAFphEwIAAAAAAGiFTQgAAAAAAKAVNiEAAAAAAIBWtB5MnclCi8rglqeffjr0uf3220OtDOfNwmQyTYKp11lnndAnC5K76aabau0nn3wy9MnCu7LApfL4WRh3dnwBNu1pO4CvDNH5/ve/H/qMGDEi1J599tla+4gjjgh9srDCbK4I8VryNA2mbiJbD7faaqtQK4OOyjlYVXlonPm15CoDDC+88MLQpzwPv5oyPPD6668PfbLzfBlimYV7ZoHZe+yxR6itvvrqtfbuu+8e+mThcqeddlqtnYUo0jtZgNrrX//6WjsLJrzllltCLbtmKmVr54orrhhq2d9scqwyhLqq4jx76qmnQp8swDC7XqY7mgZTN3lsFs677777hlo5D7NzZ7Zuzpw5s9G4eqO3YdxLsyafC7L1IwukLz300EOhVp47F0cW3NnkM2Q29uxz7G677VZrjxs3LvS56qqrQq3Jestrl71/99xzz1p78803b3SsBQsWhNodd9xRa//2t78NfXbeeedQ22abbWrtQw89NPTJwlsvvfTSUCs/i3Ty/UJUriEbbrhh6HPccceF2r333ltr//znPw99mn6flYX/lpqcu7Jzc9PvEpvI5qLPD52Rvb7Zd7dnnXVWrZ19NizDwqsqv64qz8/nnntu6PPwww+H2t///vdaOwtgX2mllULtjDPOCLVtt9021EpZWPzGG29caze9FmjrGtAvIQAAAAAAgFbYhAAAAAAAAFphEwIAAAAAAGiFTQgAAAAAAKAVfRJMnYVZlCEwWZBLFoLa27+XKcOwyzDMVztWkwCkLHArCyApQ0Kyx5XBPlUlmLpNbQdTjx07ttZ+xzveEfpkc+z000+vtbOQTvNi6dEkELO3r3cWYLTXXnuF2iqrrFJr/+Uvfwl9mgTDNtXk39z2+3NZkj3f//mf/1lrZ6FfmWzN+uEPf1hrl2tYVeXnvCZBcnfddVeojRgxItTK9XbMmDGhz3777Rdq5fMgWK53llsu/r8vZUBlVcW1ZsqUKaHPxRdfHGpZcGY5f7J5ns27Jtd2WbDb4YcfHmprrrlmrZ0FU996662hlv176I5OhvENGTIk1N797neHWvn+yILJzznnnFDrZJBmKbuucI6NyvkyePDg0Gf11VfvsVaeo6qqqn7zm9+EWjY3yjFkYdLZXJwxY0aPfc4888xQy8LVy3Fdcskloc+kSZNCzZxqR3bOfdOb3tTj45577rlQ++hHPxpqv//972vtbO7ssMMOoVaeE7NzcBaYnQXI0l0jR46stb/xjW+EPjvuuGOoletTJ88t2Txvoi/Ob9a63innz4orrhj6fPKTnwy18nu17DuPLIT62muvDbVTTz211n7yySdDn+y6vcn3M9nniV133TXUsn93kzFMnjy51p4zZ07o083vDf0SAgAAAAAAaIVNCAAAAAAAoBU2IQAAAAAAgFbYhAAAAAAAAFrRJ8HUTfRF2GiTEMLeWn/99UNtk002CbUyWOfxxx8PfbKARiE37enkczts2LBQ+853vlNrDxo0KPR5+OGHQ+1Xv/pVrS2EeulWzsPFeb2bBCjuueeeoVaukdddd13ok4XL9WZMr1YrnwdrX+dk69P73//+Wjt7TbK5WIZQV1VVffrTn661szDVLFyu/JvZa54FbP3sZz8LtTIgMQtRzMK3y+emDPKkmez8tuWWW4ZaGdB29dVXhz5ZcGaT9aCT58pRo0aF2i677NLj37z99ttDn2eeeSbUrG/dUz7X2frU9PUo16wxY8aEPltssUWPx8mu98ePH9/rcZWy9bxTx17WlOeulVZaqdHjyvDoffbZJ/R56KGHQm369OmhVn6u3GCDDUKfLEjzscceq7WPPvro0Gf33XcPtWz+/OUvf6m1s+vENoPUqVthhfhVzyqrrFJrP/HEE6HPaaedFmq//vWvQ608t2Vr3Vvf+tZQKz93ZN/BZJ9N+uL7oWVZ9h4vg82z8NzstSsDrcsQ+9eiU5+Js3lnPi0Zhg8fHmplCHVVxSDn7PWdNWtWqF144YWhVl6TZfOnyfcZWQj1xz72sVDLrhObXLfNnz8/1Mpzc1+fh/0SAgAAAAAAaIVNCAAAAAAAoBU2IQAAAAAAgFb020yIJVl2n68TTzwx1FZbbbVQK+8zfe6554Y+8+bNW4zR0YbsPubZ65vdJ72839vMmTNDn/PPPz/Unn/++R7H5T77S49OvkblHFhvvfVCn9GjR4fa5MmTa+0//OEPoU/b9xg0NzsjWwfe/OY3h1p5D//s+c8ya0444YRQW7BgQY/j6m020wsvvBBqd955Z49jyJ6HLLdgxIgRtfajjz76Woe4TCqf3/Je1FVVVWuttVaoPf3007X2NddcE/p0+36m2bXdQQcdFGrZv2fixIm19kUXXRT6LM79kVl8ncwKKefK2972ttAny6Mp17/LL7889Gly7bc4nGN71uQ6+tlnnw19JkyYEGrlPfSznJmmnyHL+/9nY5g6dWqolXMxy+kZOHBgqM2ePTvUyuyn7NxM95T3Q6+qmAFy2WWXhT7XXnttqGVrQ3nv//POOy/0KXNPsmNl57/yOuDVxtDkHun0Tvbc7rXXXrX2qquuGvpk34usvPLKPfbprTZzkegfyteqzBipqmZrTXatl82f7Lqt/JtZLkW2lpWfGQ855JDQJ6tlmT6l7LPQLbfcEmq33nprrZ09D02ubTrFLyEAAAAAAIBW2IQAAAAAAABaYRMCAAAAAABohU0IAAAAAACgFctEMHUWspEFDJaahh6Wx89CCffYY49Qy4I+rr/++n/afrXH0V3la54FxJ1xxhmh9ta3vjXUynl2wQUXhD5ZMHWT+ZmFPmXhci+99FKtbY4t3cqgo2233Tb0yebAbbfdVmtPmjSp0eN6yzxsT3YOPPDAA3t83Jw5c0Lt85//fKjNnTu3V+PqrWyurLnmmqFWBuNljyvXw6pqFg5Gz7LQ7+y5nTVrVq2dBaC2rTx/Zuf5d7zjHaGWnWPLtTMLTbfeLZmyzxhlAPub3vSmRscq5/nZZ58d+nQykN2c653seStDHqdNmxb6/OQnPwm1Mrh5gw02CH2ya/kscLg81hVXXBH6ZCGyH/nIR2rtbE4vWLAg1K677rpQu//++2vtTga+889lr1t2zi1fyxkzZoQ+G220Uaitv/76ofblL3+51t58881Dn+z9Us7VLAh7+vTpobbrrruG2pVXXllr98X1wrIku0YuZXNxm222qbWHDRsW+kydOjXUOnmeavL9X9Nw3oULF3ZkTPRO9jnzkUceCbXyeiyTXbe/733vC7Wjjz66x2Nlc6X8nJP9vew832TtzL4r/tznPhdqU6ZMqbWz+ZuNoa157pcQAAAAAABAK2xCAAAAAAAArbAJAQAAAAAAtMImBAAAAAAA0IqlLmmxaQj1SiutFGpl8EYWptUkbOTQQw8NfcowzKqqqvvuuy/UyiCR+fPnhz70vfI1/8AHPhD6vPvd7w61LEjugQceqLWzIMIs4K5J2FvTcDnBcUuvbP0bMWJErZ0FEmdBRGXIYZNwsleTzc2S0Mz2ZGvR2muvHWrlOSgLjbv99ttDrWmwW6dk4cYf//jHQ60892fjzIK5yiCwbv/7lhbZc5sF7Zbrzw477BD6TJo0KdSyoLryWE1DXsu5cvDBB4c+WUhmpnyPuLZbemTn2HK+7rzzzo2OVQaWT5w4MfSxzvRP5XX0iy++GPqMHz8+1O65555aO5tP2bVWdo1WrivZtf3w4cNDbbfddqu1s2uBLOz3P/7jP0It+4xB3xk8eHCoDR06tNbOzmPZdxlZaHp5rGx9KgNRq6qqfvSjH9Xad911V+jzjne8I9T222+/UJs5c2atfc0114Q+QoR7J7vWnTNnTq2dPbfZNXm59vz0pz8Nff70pz+FWvn6VlU8Vz799NOhz4Ybbhhq5fo6ZMiQ0Od1r3tdqN12222hVq7n2bUsnVOuLU8++WToc9xxx4XauuuuW2tvtNFGoU9W23vvvUNtk002qbUHDRoU+mTn8PJzR9Og8+y8e8YZZ9Ta//M//xP6zJo1K9TK64FsDN38PtAvIQAAAAAAgFbYhAAAAAAAAFphEwIAAAAAAGhFv8mEaPP+4Nn9rebNm9fj8bO/l42zvHdclg/w/PPPh9onPvGJUJs8eXKo9Yb7VXdO9lxuvfXWtXZ2D7rsHtPZfTFPOumkWvuxxx4LfZrcyzIbZ/Y486B/6eR7tWkmzsYbb1xrb7vttqFPdm/L+++/v9bu5FwyL7trlVVWCbVhw4aFWpP7WDa9D2o5P7N782fKc3g2p9///veH2lvf+tZQK+9Tm/17Hn300VB74oknenwcPcuyEB5++OFQW2+99Wrtww8/PPTZfvvtQy177crzYHYP1zLzo6pi/smHPvSh0Ce733aWS/H444/X2uZP97R9js2u9d75znfW2tnamo3hN7/5Ta2d5Qo0VY7VnGtX+fxm19/Z2pDVejr24sjW4NVWW63WbpIJVlX5ffx7e09p83XxZetT9nqU11CjR48OfUaNGhVq2VpX5gOMGzcu9MmyDstczPJzSVVV1etf//pQGzNmTKh9+MMfrrX//Oc/hz7ZPdJ7a1n6jiX7d91888219jHHHBP6rLrqqqFWXvPvtddeoU923d5kXjd9/svv/7LHZdeDV111VaiV15tZfufSOi/6QvlcZueyhx56qMfa9ddfH/pkn0dPOeWUUNtss81q7S9/+cuhTzaHy7UzW5eznLtPfepTofaHP/yh1l6c68S+5JcQAAAAAABAK2xCAAAAAAAArbAJAQAAAAAAtMImBAAAAAAA0Io+CabOAmaaBFNnmoRJdzIUZuWVVw61b37zm7X2hhtuGPpcd911odbJQK9SFrCSPQ+d+ntLsyzEsgyiLkM0qyoPdjv33HNDrZwbCxYsaDSuNsPcmyrnWRYYmz0P5l33ZPOkDHYbOnRo6HPHHXeE2vTp02vtpvOrt2u+QK/2NH1NytdgyJAhoc9GG20UamWob3aspq9vuQaXoa9VVVXf+c53Qi0baylbb0866aRQe/7553s8FlH5GmfPYxYSV87FPffcM/TJQg3Hjh0bamV4XRZe/etf/zrUyiC5JsHtVZWfB5uEIdIZvf080VtZOPmb3/zmWjubEzNmzAi1MlC16bVSt//N9Ky/vse33nrrUNthhx1q7SyY9fTTTw+13gZimq/tyJ7X7DUqQ3bL81NVVdWIESNC7e9//3uolWvW008/3WgM5blz9uzZoU/T9a88N6+xxhqhT3b8/voe7U+y5+jWW2+ttbO1YZ999gm18nXaeeedQ5/sXJkp5082zpdffjnUyvN19p5ZZZVVQm2//fYLtYsuuqjWzr7ry8ZAZ/T2/dv0+9Dstfvb3/5Wa//sZz8LfbbddttQW3fddWvtbM0966yzQi2bUy+99FKoLYn8EgIAAAAAAGiFTQgAAAAAAKAVNiEAAAAAAIBW2IQAAAAAAABasUQFU3c7QGjgwIGhdvjhh4fa3nvvXWtn4yzDa6qq94FemTKgJwtLzAKC6dlOO+0Uavvvv3+tnc2V7Pm+6aabeuzX9P2xwgr1t28WqpONIZuf5fHLY1dVHtRUBhpnAY2TJk0KtSzsp6z949iX1PCwbofvZc9TthaUYWArrbRS6HPvvfeGWhZW2ET2PDQJFbNmtSd7j2evQRkSl82nzTffPNRuu+22UCvf49mxyjDgqqqqD3zgA7X2f/7nf4Y+2fqUzbtynSxDFauqqm644YZQW1LXoP4mCwK/5557Qq08b1x55ZWhzyabbBJqq666aqiVAXAPPfRQ6PPwww+HWnl+e+aZZxqNIXtvZaHWLL7+EHC75pprhtro0aNr7Wycjz/+eKhNmDChY+Mq/2aT9ZB2dXK+NjknDRo0KNQ+//nP99gvO38/9dRTr2F0/1yTzyG8dtnzmgUylwHTd911V6NjNflMmQULZ+tM+bjss+L5558fasccc0yolf/G7Hqwyfci5mDUZE796Ec/Cn0uueSSUBs5cmStfeihh4Y+Rx55ZKhl3y2U37tk113Z9eaQIUNq7aZB2FlQeznWG2+8MfQRTL10KT+jZufTMoS6quI8e+SRR0Kfn//856G2tIRQZ/wSAgAAAAAAaIVNCAAAAAAAoBU2IQAAAAAAgFbYhAAAAAAAAFrRJ8HUWUBRFhjUbeUYdt1119DnpJNOCrUypOR3v/td6HPVVVeFWpthNU0DienZBhtsEGplYGXTYKN11lkn1FZbbbVaOwsJzsKxd9hhh1o7C5K7++67Qy17/5XBsv/yL/8S+uy4446hVgYVX3zxxaHP1VdfHWpZ+OzTTz9da2chZUuaLOSszfdh9vfWWGONUNt3331r7SxY/f777w+1LOSriSb/5qaB7NaxzpgxY0aoZeG85dqQBex+8pOfDLUsdLV8j+++++6hz+GHHx5q5bk4C6DLZOfBMlzzE5/4ROjz4osvNjo+r132/p0/f36olSHQ06ZNC32y+drkXJwFvWVzpbxGy/7eG97whkbHGj58eK2dXfMKCX7tmgSJdvKckf29LbbYItTKa8Tstb3hhhtCrQz8bDr2JkG/zrH9U7kWZGtYtqZktfL1LINgq6qqttpqqx4fd9NNN4U+nfzMms278nloEmZMXfb8ZNcznXwey9ctuz5rcs7NArQvuuiiHh9XVVW15ZZb1trZ5/bsmrS89siel6a1ZUn53pw3b17ok9WmTJlSa997772hz5lnnhlqm266aah95StfqbXL70SqqqpWXnnlUGv6fU0T5Xm+bFdVfn27rM+fJcWgQYNC7fLLL6+1x44dG/pk57dyHmSfPbPP5Uuzvv/mHwAAAAAAWCrZhAAAAAAAAFphEwIAAAAAAGhFn2RCZPrDvXBXX331WvuLX/xi6DNixIhQe+SRR2rtz372s6FPdq/DTt4Trj88f0ur7D7Qc+fOrbWz+8Zl99n/+te/Hmqf+cxnau0111wz9MmyJMp7zmX3X5w1a1aoZXkMQ4YM6bFPdl/RqVOn1trZfa6z98zzzz8fakvjvfCavMfbvDdrVeX32B89enStnd2z8r777gu1Tq4z5b/bfVe7K3vNf/vb34ba3nvvXWsPHjw49Cnvx/tqxyrXrGy+Zutm+bhsXmT3q37wwQdD7aijjqq1s3lu3vW98jXo7T3Rs2M1VR4/yynKcnKydbI8x8qE6Ixun2Oz+0kfeOCBoVbeFz27firvLVxV+ZzuFPfY73tNrnOyc1nT16lc/8aMGRP6ZJ8xynmXfe7p5NraJBMiYw6/dm1n4pQ5hk0zKZt8BpgzZ06oZXOzzIBYa621ehxnVcX3WjZ2c65zyuctu37K8r+y7zfKc2r2vUX2mjeRvb7ZWMvv9rIxyF1aMmTf45133nmhts8++9Ta2eubrSNf/vKXa+2//vWvr3WISx2/hAAAAAAAAFphEwIAAAAAAGiFTQgAAAAAAKAVNiEAAAAAAIBW9Jtg6m7LwuUOOuigWnvXXXcNfbIwz1NOOaXWzsILhdAsubJw01/96le19mGHHRb6ZCE3a6+9dqitu+66tXY2N5vIAl2HDh0aak1CtrIQxSeeeCLUfvnLX9baV155Zejz9NNP9/j3qiqGfS8Nuv2+z0KD3/ve94ZaGQA4c+bM0CcLf+ut7HkQwtq3suf/uuuuC7Wrrrqq1j7yyCNDnzKE9dVqnZKdh2+++eZQ++QnPxlqDz30UK1tHi5d2gwhzs6n2bly+vTpoXbnnXfW2q4JO6Pbz2N2nbXZZpuFWjmuWbNmhT733Xdf5waWaBL8St/r5DmoDEbNzoFZWGs5hk022ST0yT7TZOficp5lwZ1NwlqbhBnTsybPddYnk13XlXMnC/Dt5Ov20ksvhVr5+TEbw+qrrx5qZTD1Cy+80GgMwoa7Kwv6LV/PbF1rsoaUc6Cq8jk2fvz4UDvnnHNq7eeee67RGErmU3dl37MdffTRofae97wn1MrXKnudskDrb33rWz0+blnjlxAAAAAAAEArbEIAAAAAAACtsAkBAAAAAAC0wiYEAAAAAADQimU2mLoMA66qqjr++ONr7TK8taqq6g9/+EOoXXHFFbW2sJGlSxYoeOKJJ9baP/rRj0KfbbfdNtTe+c53htqmm25aa2dzMwtcKudnFtw0b968UMuCtm+55ZZa+9577w19brvttlB7/PHHe/x7Td8P3jevTRZkNXz48FArgwqrKoZnnXvuuaFPFlbdSV7v/ufZZ58NtXKt23jjjUOfnXfeOdSyeVfK5kAWEvfMM8/U2ieddFLoc/HFF4fa0hh2T/esssoqtXYWQv3EE0+E2u9///tQe/jhh2vtpkGg9C9ZuOnKK68cauXa8+c//7nHPp3mHLt0yz6jrrXWWrX2Flts0ehx5VwZM2ZM6DNkyJBQy9bEJvOuSRi3+dsZWQhr+dxmfbJ5kilDfDsZtD5w4MBQy8LQH3vssVr7da97Xeiz5pprhlr5+T4LphaQ3veyoPH/+I//qLVPOOGE0Cc7X5cmTJgQalkI9Y9//ONQKz9LZ9/DNJkr5lO7yuvtDTfcMPQ5+eSTQ63J2nnNNdeEPsccc0yodXJdXFr4JQQAAAAAANAKmxAAAAAAAEArbEIAAAAAAACtWCYyIbJ7Cp5++umhtuWWW9ba5X0Oq6qqfve734Vadq86lh7ZfdzmzJlTa99xxx2hT1b7yU9+0qsxZPfmLHMihg4dGvpkczi7n2bWr5Q9D+5j2Heye4pna1E258rsiCwTIrs3P0u37P08bdq0WnufffYJfbL7a44dOzbUynvsl/fJr6qqeuSRR0Jt+vTptXa5/laV+20uKbJ1q7f5CJ18zbNzbHmv//L+v1VVVXfddVeolVlJVRXvFdz0ftv0new1yu4rnr3e5WPvvPPORscv3wuusXg1TTKVnnrqqdAny3sojzV48ODQJ8uma7J2m8P9T/m9SJbhlX2eyD4rdur1bfI5t6qqatCgQaFW5hFm83f27NmhVl5DZMfOPjP7fNRd2fN99dVX19p/+ctfQp8sx6a8z382z8vPHFWVZ1763LFkWGGF+tfdxx57bOiTfYeWzbtJkybV2ocffnjoY14041MQAAAAAADQCpsQAAAAAABAK2xCAAAAAAAArbAJAQAAAAAAtGKpC6bOgo1OPPHEUDvkkENCrQwuyQJJHnzwwcUYHfROFnIzd+7cf9pm6ZbNiSlTpoTaOeecE2plkFy21gkTJPPiiy+G2gMPPNCoBr3VF0FvM2bMqLUvu+yy0Of6668PtalTp4ZaGYop2LL/y+bchAkTQu20004LtTLA+v777w992gx5ZemXzZXnnnuu1v7tb38b+qy77rqh9uyzz9ba5557bo99qqqqFi5c2OM46VtNzp3ZWpQF9mbHanPNyj7XZteWZQB7GR5bVfn8nTVrVq3ts9CSo5yf2eub1Vj2rLfeerX2m9/85tBnxRVXDLVsPfj9739fa8+ZM2cxR7fs8ksIAAAAAACgFTYhAAAAAACAVtiEAAAAAAAAWmETAgAAAAAAaMUSH0w9YMCAWnv06NGhz0c+8pFQK0OoqyqGD2XBRn//+99f6xABuiILjcsC5wD6QtOQx26HQWZr5wsvvFBrP/roo6FPeQ3a9PjCLpdM5Zyoqqq64447enxcXwSrs+yZP39+rf2jH/0o9Bk3blyoldeJZdBvVeVBxfQv2XklO0c1+VzQ7TUrG/uLL77YqFauy2VAe1VV1XLLxf/vtgyetU7Dkm2llVYKtS233LLW3nDDDUOf7Hvh2bNnh9p3v/vdWtt5sff8EgIAAAAAAGiFTQgAAAAAAKAVNiEAAAAAAIBW2IQAAAAAAABascQHUy+//PK19uqrrx76zJw5M9TWWWedUCtDkb7zne+EPlkgEgAAr92SEtIstJKMeUF/NXfu3FB76KGHQq0ML15S1mR61l/Xp3LOZQHavZXN3zK0var673MD9E4WFL3aaqvV2tOmTWt0rGuuuSbUHn300VrbubL3/BICAAAAAABohU0IAAAAAACgFTYhAAAAAACAVizxmRAvv/xyrX3XXXeFPttvv32oDRkypMdjvfDCC4s3OAAAAOiH3Neabivn3OLMwTLbofw+B1g2LFy4MNQuuOCCf9qmb/glBAAAAAAA0AqbEAAAAAAAQCtsQgAAAAAAAK1olAmxpN8rMht/0xq5bjxXXg9Kbc8Jc46MeUe3OcfSF6x1dJu1jr5graMvmHd0m3MsfaGnOdHolxCzZ8/uyGD6ysKFC8N/s2bNCv/NnTu39h+vrhtzYkmfd3Re23PCnCNj3tFtzrH0BWsd3Watoy9Y6+gL5h3d5hxLX+hpTgxY1GDr6pVXXqkmT55cDR06tBowYEDHBseSZ9GiRdXs2bOrUaNGVcst1+7dvMw7/le35p05xz8y7+g251j6grWObrPW0ResdfQF845uc46lLzSdd402IQAAAAAAAF4rwdQAAAAAAEArbEIAAAAAAACtsAkBAAAAAAC0wiYEAAAAAADQCpsQAAAAAABAK2xCAAAAAAAArbAJAQAAAAAAtOL/A4B0JcanDucQAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 2000x400 with 20 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "\n",
    "n = 10  # How many digits we will display\n",
    "plt.figure(figsize=(20, 4))\n",
    "for i in range(n):\n",
    "    # Display original\n",
    "    ax = plt.subplot(2, n, i + 1)\n",
    "    plt.imshow(x_test[i].reshape(28, 28))\n",
    "    plt.gray()\n",
    "    ax.get_xaxis().set_visible(False)\n",
    "    ax.get_yaxis().set_visible(False)\n",
    "\n",
    "    # Display reconstruction\n",
    "    ax = plt.subplot(2, n, i + 1 + n)\n",
    "    plt.imshow(roundtrip_imgs[i].reshape(28, 28))\n",
    "    plt.gray()\n",
    "    ax.get_xaxis().set_visible(False)\n",
    "    ax.get_yaxis().set_visible(False)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3ff5fcd2",
   "metadata": {},
   "source": [
    "What about the compression? Let's check the sizes of the arrays."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "f8addbcd",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-04-11T22:26:23.535569Z",
     "iopub.status.busy": "2024-04-11T22:26:23.534690Z",
     "iopub.status.idle": "2024-04-11T22:26:23.955186Z",
     "shell.execute_reply": "2024-04-11T22:26:23.954518Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x_test size (in MB): 29.91\n",
      "encoded_imgs size (in MB): 1.22\n",
      "Compression ratio: 1/25\n"
     ]
    }
   ],
   "source": [
    "encoded_imgs = autoencoder.transform(x_test)\n",
    "print(f\"x_test size (in MB): {x_test.nbytes/1024**2:.2f}\")\n",
    "print(f\"encoded_imgs size (in MB): {encoded_imgs.nbytes/1024**2:.2f}\")\n",
    "cr = round((encoded_imgs.nbytes/x_test.nbytes), 2)\n",
    "print(f\"Compression ratio: 1/{1/cr:.0f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e2f4cabb",
   "metadata": {},
   "source": [
    "## 6. Deep AutoEncoder"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5f542eb9",
   "metadata": {},
   "source": [
    "We can easily expand our model to be a deep autoencoder by adding some hidden layers. All we have to do is add a parameter `hidden_layer_sizes` and use it in `_keras_build_fn` to build hidden layers.\n",
    "For simplicity, we use a single `hidden_layer_sizes` parameter and mirror it across the encoding layers and decoding layers, but there is nothing forcing us to build symetrical models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "f5f28b5f",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-04-11T22:26:23.958298Z",
     "iopub.status.busy": "2024-04-11T22:26:23.958062Z",
     "iopub.status.idle": "2024-04-11T22:26:23.964255Z",
     "shell.execute_reply": "2024-04-11T22:26:23.963638Z"
    }
   },
   "outputs": [],
   "source": [
    "from typing import List\n",
    "\n",
    "\n",
    "class DeepAutoEncoder(AutoEncoder):\n",
    "    \"\"\"A class that enables transform and fit_transform.\n",
    "    \"\"\"\n",
    "    \n",
    "    def _keras_build_fn(self, encoding_dim: int, hidden_layer_sizes: List[str], meta: Dict[str, Any]):\n",
    "        n_features_in = meta[\"n_features_in_\"]\n",
    "\n",
    "        encoder_input = keras.Input(shape=(n_features_in,))\n",
    "        x = encoder_input\n",
    "        for layer_size in hidden_layer_sizes:\n",
    "            x = keras.layers.Dense(layer_size, activation='relu')(x)\n",
    "        encoder_output = keras.layers.Dense(encoding_dim, activation='relu')(x)\n",
    "        encoder_model = keras.Model(encoder_input, encoder_output)\n",
    "\n",
    "        decoder_input = keras.Input(shape=(encoding_dim,))\n",
    "        x = decoder_input\n",
    "        for layer_size in reversed(hidden_layer_sizes):\n",
    "            x = keras.layers.Dense(layer_size, activation='relu')(x)\n",
    "        decoder_output = keras.layers.Dense(n_features_in, activation='sigmoid', name=\"decoder\")(x)\n",
    "        decoder_model = keras.Model(decoder_input, decoder_output)\n",
    "\n",
    "        autoencoder_input = keras.Input(shape=(n_features_in,))\n",
    "        encoded_img = encoder_model(autoencoder_input)\n",
    "        reconstructed_img = decoder_model(encoded_img)\n",
    "\n",
    "        autoencoder_model = keras.Model(autoencoder_input, reconstructed_img)\n",
    "\n",
    "        self.encoder_model_ = BaseWrapper(encoder_model, verbose=self.verbose)\n",
    "        self.decoder_model_ = BaseWrapper(decoder_model, verbose=self.verbose)\n",
    "\n",
    "        return autoencoder_model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "d5b93874",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-04-11T22:26:23.966386Z",
     "iopub.status.busy": "2024-04-11T22:26:23.966191Z",
     "iopub.status.idle": "2024-04-11T22:27:00.678710Z",
     "shell.execute_reply": "2024-04-11T22:27:00.677799Z"
    }
   },
   "outputs": [],
   "source": [
    "deep = DeepAutoEncoder(\n",
    "    loss=\"binary_crossentropy\",\n",
    "    encoding_dim=32,\n",
    "    hidden_layer_sizes=[128],\n",
    "    random_state=0,\n",
    "    epochs=5,\n",
    "    verbose=False,\n",
    "    optimizer=\"adam\",\n",
    ")\n",
    "_ = deep.fit(X=x_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "0d094eb7",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-04-11T22:27:00.684700Z",
     "iopub.status.busy": "2024-04-11T22:27:00.682387Z",
     "iopub.status.idle": "2024-04-11T22:27:02.686895Z",
     "shell.execute_reply": "2024-04-11T22:27:02.686116Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1-MSE for training set (higher is better)\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "AutoEncoder: 0.9899\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Deep AutoEncoder: 0.9913\n"
     ]
    }
   ],
   "source": [
    "print(\"1-MSE for training set (higher is better)\\n\")\n",
    "score = autoencoder.score(X=x_test)\n",
    "print(f\"AutoEncoder: {score:.4f}\")\n",
    "\n",
    "score = deep.score(X=x_test)\n",
    "print(f\"Deep AutoEncoder: {score:.4f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "49fe812f",
   "metadata": {},
   "source": [
    "Suprisingly, our score got worse. It's possible that that because of the extra trainable variables, our deep model trains slower than our simple model.\n",
    "\n",
    "Check out the [Keras tutorial](https://blog.keras.io/building-autoencoders-in-keras.html) to see the difference after 100 epochs of training, as well as more architectures and applications for AutoEncoders!"
   ]
  }
 ],
 "metadata": {
  "jupytext": {
   "formats": "ipynb,md"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}