{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Quantifying Classification Uncertainty in Deep Neural Networks" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The purpose of this page is to provide an easy-to-run demo with low computational requirements for the ideas proposed in the paper _Evidential Deep Learning to Quantify Classification Uncertainty_. Using MNIST dataset, I demonstrate how to create neural networks that are able to quantify classification uncertainty. The paper can be accesed over http://arxiv.org/abs/1806.01768\n", "\n", "You can run this notebook in Colab using the colab icon below: \n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\"Open" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The notebook can also be downloaded using https://muratsensoy.github.io/uncertainty.ipynb" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Neural Networks Trained with Softmax Cross Entropy Loss" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The following lines of codes demonstrate how softmax based Deep Neural Networks fail when they encounter out-of-sample queries." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# use this while running this notebook in Colab\n", "%tensorflow_version 1.x" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "#import necessary libraries\n", "import tensorflow as tf\n", "import numpy as np\n", "from matplotlib import pyplot as plt\n", "import scipy.ndimage as nd\n", "\n", "%matplotlib inline\n", "import pylab as pl\n", "from IPython import display\n", "\n", "from tensorflow.examples.tutorials.mnist import input_data" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "colab": { "autoexec": { "startup": false, "wait_interval": 0 }, "base_uri": "https://localhost:8080/", "height": 431 }, "colab_type": "code", "executionInfo": { "elapsed": 14869, "status": "ok", "timestamp": 1527923826590, "user": { "displayName": "Murat Sensoy", "photoUrl": "https://lh3.googleusercontent.com/a/default-user=s128", "userId": "102692943223630372304" }, "user_tz": -180 }, "id": "MROFqBJQ1naS", "outputId": "56a3f37d-9b80-400b-c6da-ec68bcaa5cd8" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Extracting MNIST_data/train-images-idx3-ubyte.gz\n", "Extracting MNIST_data/train-labels-idx1-ubyte.gz\n", "Extracting MNIST_data/t10k-images-idx3-ubyte.gz\n", "Extracting MNIST_data/t10k-labels-idx1-ubyte.gz\n" ] } ], "source": [ "# Download MNIST dataset\n", "mnist = input_data.read_data_sets('MNIST_data', one_hot=True)\n", "\n", "K= 10 # number of classes" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAD8CAYAAAC4nHJkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAADDxJREFUeJzt3V2sHPV5x/Hvg3NspwZSSKhjHFKniCBRq3XQkVsJ0lJRIuJGNdzQuFLqSqhOpVA1EhdF5KJcoqpJFFURlQlWnIoCkQjCkWgaatEi1BZxsBxeQhIomMausYMcFRMV45enF2ccncDZOce7sy/Hz/cjrXZ3ntmZR2P/zszO7O4/MhNJ9Zwz7gYkjYfhl4oy/FJRhl8qyvBLRRl+qSjDLxVl+KWiDL9U1HtGubLlsSJXsmqUq5RKeYuf8XYei8XMO1D4I+J64CvAMuBrmXln2/wrWcVvxbWDrFJSiydz96Ln7fuwPyKWAV8FPglcAWyJiCv6XZ6k0RrkPf9G4KXMfDkz3wbuBzZ305akYRsk/GuBH895vr+Z9gsiYltEzETEzHGODbA6SV0a+tn+zNyemdOZOT3FimGvTtIiDRL+A8Alc55/qJkmaQkYJPxPAZdFxEciYjnwaWBXN21JGra+L/Vl5omIuAX4Z2Yv9e3IzOc760zSUA10nT8zHwEe6agXSSPkx3ulogy/VJThl4oy/FJRhl8qyvBLRRl+qSjDLxVl+KWiDL9UlOGXijL8UlGGXyrK8EtFGX6pKMMvFWX4paIMv1SU4ZeKMvxSUYZfKmqkQ3Rr6Xnl/t9orT9x1V2t9T/+k7/oWVv22J6+elI33PNLRRl+qSjDLxVl+KWiDL9UlOGXijL8UlEDXeePiH3AUeAkcCIzp7toSpMj/3tVa/39H39va/3I5St61i56rK+W1JEuPuTze5n5egfLkTRCHvZLRQ0a/gS+GxFPR8S2LhqSNBqDHvZfnZkHIuJXgEcj4geZ+fjcGZo/CtsAVvJLA65OUlcG2vNn5oHm/jDwELBxnnm2Z+Z0Zk5P0fvkj6TR6jv8EbEqIs47/Rj4BPBcV41JGq5BDvtXAw9FxOnl/GNmfqeTriQNXd/hz8yXgd/ssBdNoFX7Y6DXf/CPXu1ZO/n3Ay1aA/JSn1SU4ZeKMvxSUYZfKsrwS0UZfqkof7pbQ/V/J6Z61paPsA+9m3t+qSjDLxVl+KWiDL9UlOGXijL8UlGGXyrK6/xqdf4fHBzo9f/74MU9axfR++u+Gj73/FJRhl8qyvBLRRl+qSjDLxVl+KWiDL9UlNf5izt5zZWt9W//+ldb63vfXtZaX31v73FcTrW+UsPmnl8qyvBLRRl+qSjDLxVl+KWiDL9UlOGXilrwOn9E7AA+BRzOzPXNtAuBB4B1wD7gpsz86fDa1LCcXNH+9//cWNFaP57ZWj919OgZ96TRWMye/+vA9e+YdhuwOzMvA3Y3zyUtIQuGPzMfB468Y/JmYGfzeCdwQ8d9SRqyft/zr87M07/v9BqwuqN+JI3IwCf8MjOBnm/8ImJbRMxExMxxjg26Okkd6Tf8hyJiDUBzf7jXjJm5PTOnM3N6ivaTR5JGp9/w7wK2No+3Ag93046kUVkw/BFxH/AfwOURsT8ibgbuBK6LiBeB32+eS1pCFrzOn5lbepSu7bgXjcG+G/2cV1X+y0tFGX6pKMMvFWX4paIMv1SU4ZeK8qe7izvvg37ltir3/FJRhl8qyvBLRRl+qSjDLxVl+KWiDL9UlOGXijL8UlGGXyrK8EtFGX6pKMMvFWX4paIMv1SU3+c/y52zcmVr/eq1rwy0/LsP/+4Cc7w50PI1PO75paIMv1SU4ZeKMvxSUYZfKsrwS0UZfqmoBa/zR8QO4FPA4cxc30y7A/gz4CfNbLdn5iPDalL9O+eX39da/7uL/2mg5f/bE+tb65fynwMtX8OzmD3/14Hr55n+5czc0NwMvrTELBj+zHwcODKCXiSN0CDv+W+JiGciYkdEXNBZR5JGot/w3wVcCmwADgJf7DVjRGyLiJmImDnOsT5XJ6lrfYU/Mw9l5snMPAXcDWxsmXd7Zk5n5vQUK/rtU1LH+gp/RKyZ8/RG4Llu2pE0Kou51HcfcA3wgYjYD/w1cE1EbAAS2Ad8dog9ShqCBcOfmVvmmXzPEHrREJxYt3qoy//wd44PdfkaHj/hJxVl+KWiDL9UlOGXijL8UlGGXyrKn+4+y73+hbcGev2mH/xha335v36vtZ4DrV3D5J5fKsrwS0UZfqkowy8VZfilogy/VJThl4ryOv9Z7q719y4wx7LW6v+8cX5r/eIT+8+wI00K9/xSUYZfKsrwS0UZfqkowy8VZfilogy/VJTX+c8C71n34Z618+LfW1+7LKa6bkdLhHt+qSjDLxVl+KWiDL9UlOGXijL8UlGGXypqwev8EXEJ8A1gNbM/w749M78SERcCDwDrgH3ATZn50+G1ql7e+lrv2kenVra+9mSeaq2f+8327/Nr6VrMnv8EcGtmXgH8NvC5iLgCuA3YnZmXAbub55KWiAXDn5kHM3NP8/go8AKwFtgM7Gxm2wncMKwmJXXvjN7zR8Q64GPAk8DqzDzYlF5j9m2BpCVi0eGPiHOBB4HPZ+Ybc2uZmfQYli0itkXETETMHOfYQM1K6s6iwh8RU8wG/97M/FYz+VBErGnqa4DD8702M7dn5nRmTk+xooueJXVgwfBHRAD3AC9k5pfmlHYBW5vHW4GHu29P0rAs5iu9VwGfAZ6NiL3NtNuBO4FvRsTNwKvATcNpUcs+emlr/dZ1u/pe9pZXrmutn3//k30vW5NtwfBn5hNA9Chf2207kkbFT/hJRRl+qSjDLxVl+KWiDL9UlOGXivKnu5eAt9e+r7V+7Xv7/9j0jx64vLW+Ott/+ltLl3t+qSjDLxVl+KWiDL9UlOGXijL8UlGGXyrK6/xnuT/f//HW+sX3/bC1frLLZjRR3PNLRRl+qSjDLxVl+KWiDL9UlOGXijL8UlFe518Clj22p7W+ae2VLdWfLbD0heo6W7nnl4oy/FJRhl8qyvBLRRl+qSjDLxVl+KWiFgx/RFwSEY9FxPcj4vmI+Mtm+h0RcSAi9ja3TcNvV1JXFvMhnxPArZm5JyLOA56OiEeb2pcz82+H156kYVkw/Jl5EDjYPD4aES8Aa4fdmKThOqP3/BGxDvgY8GQz6ZaIeCYidkTEBT1esy0iZiJi5jj9DyslqVuLDn9EnAs8CHw+M98A7gIuBTYwe2Twxflel5nbM3M6M6enWNFBy5K6sKjwR8QUs8G/NzO/BZCZhzLzZGaeAu4GNg6vTUldW8zZ/gDuAV7IzC/Nmb5mzmw3As91356kYVnM2f6rgM8Az0bE3mba7cCWiNgAJLAP+OxQOpQ0FIs52/8EEPOUHum+HUmj4if8pKIMv1SU4ZeKMvxSUYZfKsrwS0UZfqkowy8VZfilogy/VJThl4oy/FJRhl8qyvBLRUVmjm5lET8BXp0z6QPA6yNr4MxMam+T2hfYW7+67O1XM/Oixcw40vC/a+URM5k5PbYGWkxqb5PaF9hbv8bVm4f9UlGGXypq3OHfPub1t5nU3ia1L7C3fo2lt7G+55c0PuPe80sak7GEPyKuj4gfRsRLEXHbOHroJSL2RcSzzcjDM2PuZUdEHI6I5+ZMuzAiHo2IF5v7eYdJG1NvEzFyc8vI0mPddpM24vXID/sjYhnwI+A6YD/wFLAlM78/0kZ6iIh9wHRmjv2acET8DvAm8I3MXN9M+xvgSGbe2fzhvCAz/2pCersDeHPcIzc3A8qsmTuyNHAD8KeMcdu19HUTY9hu49jzbwReysyXM/Nt4H5g8xj6mHiZ+Thw5B2TNwM7m8c7mf3PM3I9epsImXkwM/c0j48Cp0eWHuu2a+lrLMYR/rXAj+c8389kDfmdwHcj4umI2DbuZuaxuhk2HeA1YPU4m5nHgiM3j9I7RpaemG3Xz4jXXfOE37tdnZlXAp8EPtcc3k6knH3PNkmXaxY1cvOozDOy9M+Nc9v1O+J118YR/gPAJXOef6iZNhEy80Bzfxh4iMkbffjQ6UFSm/vDY+7n5yZp5Ob5RpZmArbdJI14PY7wPwVcFhEfiYjlwKeBXWPo410iYlVzIoaIWAV8gskbfXgXsLV5vBV4eIy9/IJJGbm518jSjHnbTdyI15k58huwidkz/v8FfGEcPfTo69eA7zW358fdG3Afs4eBx5k9N3Iz8H5gN/Ai8C/AhRPU2z8AzwLPMBu0NWPq7WpmD+mfAfY2t03j3nYtfY1lu/kJP6koT/hJRRl+qSjDLxVl+KWiDL9UlOGXijL8UlGGXyrq/wFWuLZ5z/d+bwAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "digit_one = mnist.train.images[4].copy()\n", "plt.imshow(digit_one.reshape(28,28)) \n", "plt.show()" ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "collapsed": true }, "outputs": [], "source": [ "# define some utility functions\n", "def var(name, shape, init=None):\n", " if init is None:\n", " init = tf.truncated_normal_initializer(stddev=(2/shape[0])**0.5)\n", " return tf.get_variable(name=name, shape=shape, dtype=tf.float32,\n", " initializer=init)\n", "\n", "def conv(Xin, f, strides=[1, 1, 1, 1], padding='SAME'):\n", " return tf.nn.conv2d(Xin, f, strides, padding)\n", "\n", "def max_pool(Xin, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='SAME'):\n", " return tf.nn.max_pool(Xin, ksize, strides, padding)\n", "\n", "def rotate_img(x, deg):\n", " import scipy.ndimage as nd\n", " return nd.rotate(x.reshape(28,28),deg,reshape=False).ravel()" ] }, { "cell_type": "code", "execution_count": 92, "metadata": { "collapsed": true }, "outputs": [], "source": [ "# Create a LeNet network with softmax cross entropy loss function\n", "def LeNet_softmax(lmb=0.005): \n", " g = tf.Graph()\n", " with g.as_default():\n", " X = tf.placeholder(shape=[None,28*28], dtype=tf.float32)\n", " Y = tf.placeholder(shape=[None,10], dtype=tf.float32)\n", " keep_prob = tf.placeholder(dtype=tf.float32)\n", " \n", " # first hidden layer - conv\n", " W1 = var('W1', [5,5,1,20])\n", " b1 = var('b1', [20])\n", " out1 = max_pool(tf.nn.relu(conv(tf.reshape(X, [-1, 28,28, 1]), \n", " W1, strides=[1, 1, 1, 1]) + b1))\n", " # second hidden layer - conv\n", " W2 = var('W2', [5,5,20,50])\n", " b2 = var('b2', [50])\n", " out2 = max_pool(tf.nn.relu(conv(out1, W2, strides=[1, 1, 1, 1]) + b2))\n", " # flatten the output\n", " Xflat = tf.contrib.layers.flatten(out2)\n", " # third hidden layer - fully connected\n", " W3 = var('W3', [Xflat.get_shape()[1].value, 500])\n", " b3 = var('b3', [500]) \n", " out3 = tf.nn.relu(tf.matmul(Xflat, W3) + b3)\n", " out3 = tf.nn.dropout(out3, keep_prob=keep_prob)\n", " #output layer\n", " W4 = var('W4', [500,10])\n", " b4 = var('b4',[10])\n", " logits = tf.matmul(out3, W4) + b4\n", " \n", " prob = tf.nn.softmax(logits=logits) \n", " \n", " loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=logits, labels=Y))\n", " l2_loss = (tf.nn.l2_loss(W3)+tf.nn.l2_loss(W4)) * lmb\n", " \n", " step = tf.train.AdamOptimizer().minimize(loss + l2_loss)\n", " \n", " # Calculate accuracy\n", " pred = tf.argmax(logits, 1)\n", " truth = tf.argmax(Y, 1)\n", " acc = tf.reduce_mean(tf.cast(tf.equal(pred, truth), tf.float32))\n", " \n", " return g, step, X, Y, keep_prob, prob, acc, loss" ] }, { "cell_type": "code", "execution_count": 93, "metadata": { "collapsed": true }, "outputs": [], "source": [ "# get the LeNet network\n", "g1, step1, X1, Y1, keep_prob1, prob1, acc1, loss1 = LeNet_softmax()" ] }, { "cell_type": "code", "execution_count": 169, "metadata": { "cellView": "code", "colab": { "autoexec": { "startup": false, "wait_interval": 0 } }, "colab_type": "code", "collapsed": true, "id": "xTVDRunk1Wsq", "scrolled": true }, "outputs": [], "source": [ "sess1 = tf.Session(graph=g1)\n", "with g1.as_default(): \n", " sess1.run(tf.global_variables_initializer())" ] }, { "cell_type": "code", "execution_count": 170, "metadata": { "colab": { "autoexec": { "startup": false, "wait_interval": 0 } }, "colab_type": "code", "id": "QwYkUP6L2q-n", "scrolled": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "epoch 1 - 100%) training accuracy: 0.9235 \t testing accuracy: 0.9255\n", "epoch 2 - 100%) training accuracy: 0.9514 \t testing accuracy: 0.9510\n", "epoch 3 - 100%) training accuracy: 0.9648 \t testing accuracy: 0.9644\n", "epoch 4 - 100%) training accuracy: 0.9699 \t testing accuracy: 0.9685\n", "epoch 5 - 100%) training accuracy: 0.9765 \t testing accuracy: 0.9732\n", "epoch 6 - 100%) training accuracy: 0.9796 \t testing accuracy: 0.9744\n", "epoch 7 - 100%) training accuracy: 0.9800 \t testing accuracy: 0.9766\n", "epoch 8 - 100%) training accuracy: 0.9816 \t testing accuracy: 0.9767\n", "epoch 9 - 100%) training accuracy: 0.9824 \t testing accuracy: 0.9787\n", "epoch 10 - 100%) training accuracy: 0.9863 \t testing accuracy: 0.9812\n", "epoch 11 - 100%) training accuracy: 0.9862 \t testing accuracy: 0.9822\n", "epoch 12 - 100%) training accuracy: 0.9873 \t testing accuracy: 0.9831\n", "epoch 13 - 100%) training accuracy: 0.9866 \t testing accuracy: 0.9816\n", "epoch 14 - 100%) training accuracy: 0.9875 \t testing accuracy: 0.9826\n", "epoch 15 - 100%) training accuracy: 0.9883 \t testing accuracy: 0.9837\n", "epoch 16 - 100%) training accuracy: 0.9895 \t testing accuracy: 0.9850\n", "epoch 17 - 100%) training accuracy: 0.9892 \t testing accuracy: 0.9838\n", "epoch 18 - 100%) training accuracy: 0.9877 \t testing accuracy: 0.9838\n", "epoch 19 - 100%) training accuracy: 0.9899 \t testing accuracy: 0.9851\n", "epoch 20 - 100%) training accuracy: 0.9903 \t testing accuracy: 0.9866\n", "epoch 21 - 100%) training accuracy: 0.9901 \t testing accuracy: 0.9868\n", "epoch 22 - 100%) training accuracy: 0.9909 \t testing accuracy: 0.9864\n", "epoch 23 - 100%) training accuracy: 0.9915 \t testing accuracy: 0.9855\n", "epoch 24 - 100%) training accuracy: 0.9914 \t testing accuracy: 0.9856\n", "epoch 25 - 100%) training accuracy: 0.9904 \t testing accuracy: 0.9837\n", "epoch 26 - 100%) training accuracy: 0.9913 \t testing accuracy: 0.9873\n", "epoch 27 - 100%) training accuracy: 0.9931 \t testing accuracy: 0.9892\n", "epoch 28 - 100%) training accuracy: 0.9910 \t testing accuracy: 0.9874\n", "epoch 29 - 100%) training accuracy: 0.9920 \t testing accuracy: 0.9877\n", "epoch 30 - 100%) training accuracy: 0.9923 \t testing accuracy: 0.9881\n", "epoch 31 - 100%) training accuracy: 0.9922 \t testing accuracy: 0.9870\n", "epoch 32 - 100%) training accuracy: 0.9934 \t testing accuracy: 0.9886\n", "epoch 33 - 100%) training accuracy: 0.9937 \t testing accuracy: 0.9893\n", "epoch 34 - 100%) training accuracy: 0.9946 \t testing accuracy: 0.9904\n", "epoch 35 - 100%) training accuracy: 0.9937 \t testing accuracy: 0.9890\n", "epoch 36 - 100%) training accuracy: 0.9933 \t testing accuracy: 0.9874\n", "epoch 37 - 100%) training accuracy: 0.9944 \t testing accuracy: 0.9883\n", "epoch 38 - 100%) training accuracy: 0.9944 \t testing accuracy: 0.9890\n", "epoch 39 - 100%) training accuracy: 0.9948 \t testing accuracy: 0.9895\n", "epoch 40 - 100%) training accuracy: 0.9947 \t testing accuracy: 0.9896\n", "epoch 41 - 100%) training accuracy: 0.9935 \t testing accuracy: 0.9893\n", "epoch 42 - 100%) training accuracy: 0.9943 \t testing accuracy: 0.9898\n", "epoch 43 - 100%) training accuracy: 0.9936 \t testing accuracy: 0.9884\n", "epoch 44 - 100%) training accuracy: 0.9945 \t testing accuracy: 0.9901\n", "epoch 45 - 100%) training accuracy: 0.9945 \t testing accuracy: 0.9887\n", "epoch 46 - 100%) training accuracy: 0.9949 \t testing accuracy: 0.9908\n", "epoch 47 - 100%) training accuracy: 0.9950 \t testing accuracy: 0.9893\n", "epoch 48 - 100%) training accuracy: 0.9949 \t testing accuracy: 0.9896\n", "epoch 49 - 100%) training accuracy: 0.9940 \t testing accuracy: 0.9891\n", "epoch 50 - 100%) training accuracy: 0.9949 \t testing accuracy: 0.9886\n" ] } ], "source": [ "bsize = 1000 #batch size\n", "n_batches = mnist.train.num_examples // bsize\n", "for epoch in range(50): \n", " for i in range(n_batches):\n", " data, label = mnist.train.next_batch(bsize)\n", " feed_dict={X1:data, Y1:label, keep_prob1:.5}\n", " sess1.run(step1,feed_dict)\n", " print('epoch %d - %d%%) '% (epoch+1, (100*(i+1))//n_batches), end='\\r' if i= threshold\n", " ldeg.append(deg) \n", " lp.append(p_pred_t[0])\n", " \n", " labels = np.arange(10)[scores[0].astype(bool)]\n", " lp = np.array(lp)[:,labels]\n", " c = ['black','blue','red','brown','purple','cyan']\n", " marker = ['s','^','o']*2\n", " labels = labels.tolist()\n", " for i in range(len(labels)):\n", " plt.plot(ldeg,lp[:,i],marker=marker[i],c=c[i])\n", " \n", " if uncertainty is not None:\n", " labels += ['uncertainty']\n", " plt.plot(ldeg,lu,marker='<',c='red')\n", " \n", " plt.legend(labels)\n", " \n", " plt.xlim([0,Mdeg]) \n", " plt.xlabel('Rotation Degree')\n", " plt.ylabel('Classification Probability')\n", " plt.show()\n", "\n", " plt.figure(figsize=[6.2,100])\n", " plt.imshow(1-rimgs,cmap='gray')\n", " plt.axis('off')\n", " plt.show()" ] }, { "cell_type": "code", "execution_count": 323, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEKCAYAAADjDHn2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzsnXl4XGXZ/z93tqbpvmZtmzatINBSmkxZVFZFQYQX9AVqkZ0KyvtjUVFBkMUq4oLoC0oFobUV8GWtiKCyI1tbCmUpbdomLVmadN/bNMn9++M5p5kks29nZvJ8rutcZ+bMmWfunCTnO89zb6KqWCwWi8USihyvDbBYLBZL+mPFwmKxWCxhsWJhsVgslrBYsbBYLBZLWKxYWCwWiyUsViwsFovFEhYrFhaLxWIJixULi8VisYTFioXFYrFYwpLntQHRMnLkSK2srPTaDIvFYskolixZslFVR8X6/owTi8rKShYvXuy1GRaLxZJRiMjaeN5vl6EsFovFEhYrFhaLxWIJixULi8VisYTFioXFYrFYwmLFwmKxWCxhSZpYiMifRKRVRD4I8rqIyG9FZJWILBORaZGMu2TJEkQEEaGkpCRqu0pKSg6833+LdqxEjWOxWLKDdLy3+I8FVEc9gB+SrE55InIssBOYp6qHBXj9VOB/gFOBI4G7VPXICMbtZnC09jsXLSDRjJWoccD8QltaWnodLy4uZv369VGNZbFYIieR/3vpeG/pOZaqBh88DEnLs1DVV0SkMsQpZ2CERIE3RWSoiJSqanM0n/PjH/840GdHM8QBbr75ZqDrAvtf6EDHgvHggw9SUFBAv379KCgoCLq5rwf6YwWCHrdYLIkhkv+9jo4O9u7dG3YLxX333dfr3hHoXhLu/jJnzhxUlc7OTlS129bzWGdnZ8ixoiVpMwsARyyeDjKzeBq4XVVfc54/D3xfVXtl3InILGCW8zSiqVSwi55pPcebmpooLS312gyLJSsJdXMeOHAge/fupb29PYUWJZe0nFkkElWdA8yB5C9DueP5j9vzmKpSUFAQdJy6ujra2trYt28fbW1tYbdLL7006FhlZWWUl5fj8/moqak5sB8+fHhUP7fFEg/NzXDuufDII9BX3HKzZs2isLAw4m369OlBx1q3bl2354HuW+6xUOWMGhoayMnJ6eXP6HnMfT548ODYfvgAeDmzuBd4SVUfcp6vAI4PtwyVjT6LUGPdeeedLFq0iEWLFlFbW3vgeFVV1QHx8Pl8TJs2jYkTJ1rfhyUpfOtbcO+9cPnlcPfdXlsTP21tbdx222385Cc/CXpONt5b4plZ9Fr3SuQGVAIfBHnty8A/AAGOAt6OcEx1t+LiYo2W4uJi9R8j1rESNY6qBhzH3fzZsmWL/vvf/9af/exn+tWvflXHjh174DxHRCMax2KJhqYm1fx8VVDNzVV9+GHV/fu9tip2PvjgA502bVrI/5dY/mfS8d7ScyyN534ez5tDDgwPAc3AfqABuAS4HLjceV2Au4HVwPtATSTjVldXR33B0p14/jhaWlr073//u958881WLCxJYcYMc6fw30aNUr3iCtWXXlJtb/fawsjo6OjQX/3qV9qvXz8dOXKkPv744wm9Mac7wGKN456e1GWoZFBTU6O26mxgEjl9tVgAGhth7FjwD6wpKIAvfhGefx5274bSUvjv/zY+jaOOgggCBlPO2rVrueCCC3j55Zf5yle+wh//+EeKi4u9NiuliMgSVa2J9f02g9tisQTlnHO6C4XLmDHQ2goPP2wE4t574ZhjoLISvvc9WLzYzEG8RlV58MEHmTx5MkuWLOH+++/nqaee6nNCkQisWFgsloCsXg2vv977eFubOT5ggBGTxx83wjFvHkyeDHfdBT4fTJoEN9wAy5Z1F47mZjjuOEh2zEVraytnnXUWF110EUcccQTLli3j4osvjihXytIbKxZZRLBvS/ZblCVaOjvhsstg0CD45JOeHgtYurT7+YMHwze+AU8/bUTgvvtgwgT4+c/h8MPhkEPg5pth+XK47TZ47TWzTxZPPfUUhx12GM888wy//OUveeGFFxg/fnzyPrAvEI/Dw4stGx3cyWDhwoUK6Guvvea1KZYM5Pe/N7Lwxz/GN05rqxnruONURcyY7r5/f9Xm5oSYe4Bt27bpRRddpIBOnTpV33///cR+QAZDnA5uO7PIUqqrTaK7DQawRMvatcbv8PnPwyWXxDfWqFEmN+Oll6ChAT7zma4lqY6OxM4uXn75ZaZMmcLcuXO5/vrreeuttzjssF4pXpYYyYgMbkv0lJWVUVJSwpIlS7w2xZJBqJrlJ1X44x8TG9kkAv5/jm1t8MADcOON0WWFByv+BzBx4kRee+01jj766DittfTEziyymJqaGisWlqj405/gX/+CO+4wkU2J5LbbekdWxTK7CFVg891337VCkSSsWGQx1dXVLF++nJ07d3ptiiUDaGiAa6+F4483S0eJ5o03zGzCHzeyKlEMGDAgcYNZumHFIouprq5GVXn33Xe9NsWS5qjCN78J7e0mkiknCXeGpUvN5/z61+b5pk2BI6ss6YkViyzGdXLbpShLOObPh2eegZ/+FKqqkvtZZWVm3xxV5xqL11ixyGLKysooLS21EVGWkDQ3w1VXmQzsK69M/ue5YtHUlPzPsiQOKxZZTnV1tZ1ZWIKiasqP79ljnNu5ucn/TLeXV6xiYZNPvcGKRZZTU1PDxx9/bJ3cloA88gg8+STceiscdFBqPtMVi1iXodavX4+q8pnPfIYTTjjhQNKY7dmSXKxYZDnWyW0JRmurWXaaPt1EQaWKAQNgyJD4l6Gampooc9e0LEnHikWWYzO5U8CCBSYpISfH7Bcs8NqiiLjyStixwyTGpWL5yZ/S0vjEQlWtWKQYm8Gd5ZSWllJaWmr9FsliwQKYNcs0dgBTK2PWLPN45kzv7ArDY4/B//0fzJ5tivylmrKy+MRiy5Yt7Nu3z4pFCrEziz6AzeROIjfc0CUULrt3m+NpyqZNxqk9bZqpAeUFZWXxhc42NjY641ixSBVWLPoA1dXVfPzxx+zYscNrU7KPdeuiO54GXHUVbN5sop/y872xwZ1ZxNogqcmZlpSXlyfQKksorFj0AayTO4mMGRPdcY/529/MytkNN5g+E15RWmpKfWzeHNv7XbGwM4vUYcWiD2AzuZNITZCWxl44AsKwZYsp6TFlClx/vbe2xJvF7YpFqRuHa0k6Viz6AKWlpZSVldmIqETz9NOmp+jnPgdjx5oa3OPGwRe/CM8+a9Z50ohrrzXhsg88AAUF3toSbxZ3Y2Mjw4cPp7CwMHFGWUJio6H6CDaTO8GsWgXnnWe8xM89B/37d73W3g6nnmpKtx50kOn44zHPPgsPPmhmFNOmeW1N/FncTU1N1l+RYuzMoo9QU1PDihUrrJM7EezaBWedZZITHnusu1AA5OWZ1Ohx48x5Hju7t283DY0+/Wm46SZPTTlAIsTC+itSixWLPoLr5F5q60HHh9tK7oMP4KGHgncIGjbMeJP37oXTTzcC4wHNzWZy09holp/69fPEjF4UFcHQofH5LKxYpBYrFn0E6+ROEHfdZURi9mw4+eTQ5x58MDz8MLz/PlxwQe82cSngm9+E9euNU/vII1P+8SGJNTGvo6OD9evXW7FIMVYs+gglJSWUl5dbsYiHl1+G734XzjwTfvCDyN5zyimmR+ljj0XfPzRO1q0zPniAlSuNaKQTsZb8aG1tpaOjw/osUowViz6EdXLHQWMjnH02TJxoPMUikb/32mvNzOLmm41opIjvf78r6S2WXtfJJtaZhc2x8AYrFn2I6upq6+SOhX374GtfM2U8Hn8cBg+O7v0i8Ic/wFFHwfnnQwqSI5ubjakubW3GZ5FOswu35Ee0WdxWLLzBikUfoqamxjq5Y+Gaa+DNN83dNtZku8JCeOIJGD7cOLxbWhJrYw9uu83MJvxJt9lFWRns329qVUWDFQtvsGLRh7BO7hh44AH4/e/huuvM7CIeSkrgqadg40YTUrtvX2JsDMAbb/QWi7Y2eP31pH1k1MQaPtvY2EhOTo7tjJdikioWIvIlEVkhIqtEpJdHUETGisiLIrJURJaJyKnJtKevU1xcTHl5uc3kjpQlS+CKK+Ckk0z0UyKYNs34PF5/3ZR+jbWSXhiWLoVzzoGqKvMR7pZOk8pYS340NTVRXFxMXp7NKU4lSRMLEckF7gZOAQ4BZohIzzn8j4C/quoRwLnAPcmyx2Kw5cojZONG+OpXYfRoEyqbyBvT2WfDj35kyoHcdVfixu1BfT2MH5+04eMm1pIfNsfCG5I5s5gOrFLVNaraBjwMnNHjHAVcb+EQIM5Gi5ZwVFdXs3LlSrZv3+61KelLRwfMmGG8wY8/DqNGJf4zbrnFhOB+5zumXEgSqKsLnjOYDsS6DGXFwhvCioWIfEVEYhGVcuATv+cNzjF/bgbOE5EG4Bngf2L4HEsU2EzuCPjRj+Df/4Z77gleVTZecnJg3jw47DCzXrRiRUKH373bFA1MZ7EoLDSJ7rH4LKxYpJ5IROAcoFZE7hCRgxP8+TOAB1W1AjgV+HMgYRKRWSKyWEQWb9iwIcEm9C2skzsMjz8Ot99uWqNefHFyP2vgQOPwzs83EVJbtyZs6LVrzT6dxQKi75i3b98+Nm7caBPyPCCsWKjqecARwGrgQRF5w7l5Dwrz1kbAvwNMhXPMn0uAvzqf8wZQCIwMYMMcVa1R1ZpRyVgS6EMUFxdTUVFhxSIQH39skuemT4ff/jY1n1lZaRL11qyBc881FWsTQF2d2aezzwKiT8xb7ySK2JlF6oloeUlVtwOPYvwOpcCZwDsiEmrZaBEwSUTGi0gBxoG9sMc564CTAETk0xixsFOHJFNdXW0jonqyY4fxIfTvb27eqay4d+yxZsnruedMiG4CqK83+3SfWURb8sPmWHhHJD6LM0TkCeAlIB+YrqqnAIcD3wn2PlVtB64EngOWY6KePhSRW0XkdOe07wCXich7wEPAhapJiiW0HMA6uXugChdeCLW1prR4RUXqbbjsMvif/4E774SRI41Po7LS9ECNgfp6o3clJQm1MuGUlZk4gkhrLFqx8I5I4gHPAu5U1Vf8D6rqbhG5JNQbVfUZjOPa/9hNfo8/ArzvDNPHqHGctkuXLuW4447z2BoPcZtRuwv8X/86nHCCd/b4fEYk3JTmtWuN7wRg5syohqqrM+00ctI87dY/izuSFebGxkbnfVYsUk0kf0rrewqFiPwcQFWfT4pVlqRindwYoZg1q0soAJ58MuZv8gnhxht7f8XevdsIWpTU16f/EhREHz7b1NREfn4+I0f2cm1akkwkYvGFAMdOSbQhltQxevRoKioq+rbf4oYbzI3YnxhvzAkjWEe9GDrtZYpYRJuY5+ZYSDRVfy0JIahYiMgVIvI+cLBTisPd6oBlqTPRkgz6fCZ3Am/MCWPs2OiOB2HnTpOAnu6RUBB9yQ+bkOcdoWYWfwG+Ajzl7N2t2gmntWQwfd7JnaAbc0KZPdv0G/WnqCjqulSZEgkF0S9D2YQ87wglFqqq9cC3gR1+GyIyPPmmWZKJ67d45513PLbEI2bP7h0eG8ONOaHMnAlz5nR93R4xwjyP0rmdSWLRr5+p2h7NMpRNyPOGcDMLgCXAYme/xO+5JYPp807umTNNXgWY5kTjxsV0Y06KXfX1kJsLl18ekz2ZkpDnEmkW986dO9m+fbudWXhE0NBZVT3N2WfIn5wlGkaPHs2YMWP6rliAKbcxciSkWwmZ/HyzHLZ6dUxvr683dZdGj06sWcki0izuZkdRrFh4Q1CxEJFpod6oqn10/SJ76POZ3LW1MGmS11YEpqoqLrGorIyuTbiXlJXBRx+FP88m5HlLqKS8X4V4TYETE2yLJcXU1NTw5JNPsm3bNoYMGeK1OamnthY+/3mvrQjMxInw17/G9NZ072PRk9LSrizuUEmEbkKe9Vl4Q6hlKA9TWS2pwPVbLF26lOOPP95bY1LNrl1m7eNTn/LaksBUVcHmzaYS7dChUb21rg6OPDJJdiWBsjJTP3HjxtBLZ3Zm4S2h8ixOdPZnBdpSZ6IlWbhi0SeXolatMvt0XoaCqJeitm2DLVsyIxLKJdLEvKamJgYMGMCgQeEKXluSQahlqOOAFzC5FT1R4PGkWGRJGaNGjWLs2LF908ldW2v26S4Wq1aBI+qR4FYvybRlKDBiMXVq8PNs9ra3hFqG+rGzvyh15lhSTXV1dd8Wi4kTvbUjGBMmmH2UMws3bDYbZxY2Ic9bIilRPkJEfisi74jIEhG5S0RGpMI4S/Kprq6mtraWbdu2eW1KaqmtNfW703VJY+BAKC6OWiwyKSHPxS2jHi7XwibkeUskhQQfxjQk+irwNefxI8k0ypI63HLlfS6TO53DZl0mToxJLIqKTPpIptCvn7E31MxCVW1dKI+JRCxKVfU2Va1ztp8Axck2zJIa+mwmdyaIRQy5FnV1xl+Racv64Trmbd26lb1791qx8JBIxOKfInKuiOQ429mY7neWLGDkyJF9z8m9fTu0tGSGWDQ0wJ49Eb8lU0qT9yRcyQ8bNus9oUJnd4jIduAyTJ2oNmd7GJiVGvMsqaDPZXKneySUixsR5XqtIyCTxSLUzMIm5HlPULFQ1UGqOtjZ56hqnrPlqOrgVBppSS41NTWsWrWq7zi5XbFI14Q8lyhzLbZsMXkWmRQ26+JmcXd0BH7dziy8J6IOvSIyTESmi8ix7pZswyypo8+VK3fFwr0ZpytRikUmRkK5lJUZoQhW09EVi1I3KcOSciIJnb0UeAXjp7jF2d+cXLMsqaTPZXLX1kJFRe9GQ+nGyJEweHCfEQsI7rdoampi2LBh9O/fP3VGWboRycziKsAHrHXqRR0BbE2qVZaUMnLkSMaNG9d3nNyZEAkFJqQpioioTOtj4U+4xDybkOc9kYjFXlXdCyAi/VT1Y+Cg5JplSTV9KpM7U8QCohKL+nqTYzhsWHJNSgbh2qvahDzviUQsGkRkKPAk8C8ReQpYm1yzLKmmurqaVatWsXVrlk8at2yBTZsySyzq6oJ7fv3ItD4W/oTL4rYJed4TVixU9UxV3aqqNwM3AvcD/5Vswyyppc9kcmdK2KxLVRXs3w+ffBL21EzrY+FPQQGMGhV4ZtHZ2Ulzc7MVC4+JNBpqmoj8P2AK0KCqbck1y5Jq+kwm98qVZp9JYgFhl6JUzQQkE53bLsGyuDds2EBHR4cVC4+JJBrqJmAuMAIYCTwgIj9KtmGW1DJixIi+4eSurTXrNG5V13THrYobRiw2b4adOzNbLIIl5tmEvPQgVD8Ll5nA4X5O7tuBd4GfJNMwS+qpqanJ/vDZ2loYNw4KC722JDLKy80aTRixcMNmM3UZCoxYLFvW+7hNyEsPIlmGagL8/7P6AY3JMcfiJdXV1axevTq7ndyZFAkFkJtrFCCMWGRiH4uelJUFzuK2YpEehKoN9TsR+S2wDfhQRB4UkQeAD7B5FllJ1mdyq2aeWIDxW7htYIOQyQl5LqWl0NkJra3djzc1NSEiFBfbYtdeEmoZyl2PWAI84Xf8paRZY/EU/0zuE0880WNrksDGjaZ4UiaKxSuvGLELEhdbXw9Dh5otU/HP4vav6tHY2EhxcTH5+fneGObH/v37aWhoYO/evV6bEpTCwkIqKioSfr1CtVWd6z4WkQLArbq2QlX3RzK4iHwJuAvIBe5T1dsDnHM2pnyIAu+p6tcjtt6SUEaMGEFlZWX2OrkzLWzWparKeK83bIDRowOekumRUNA9i3vatK7j6ZRj0dDQwKBBg6isrEzLXuCqyqZNm2hoaGB8gh1YkURDHQ/UAncD9wArIykkKCK5zntOAQ4BZojIIT3OmQT8EPiMqh4KXB3tD2BJLFmdyZ2pYhFBRFSmlib3J1jJj3QSi7179zJixIi0FAoAEWHEiBFJmflE4uD+FXCyqh6nqscCXwTujOB904FVqrrGyct4GDijxzmXAXer6hYAVe2xWmlJNTU1NaxevZotW7Z4bUriqa3tchhnEmFyLVSzQyyKi80qWzqLBZC2QuGSLPsiEYt8VV3hPlHVlUAki2HlgH/aaYNzzJ9PAZ8Skf+IyJvOslUvRGSWiCwWkcUbgtUwtiSErHZy19aaO2oarH1HhdsnNYhYbNgAu3dnngb2JD/fZHH7l/xoa2ujtbU1rcTCay6++GJGjx7NYYcdltLPjUQsFovIfSJyvLP9kS7nd7zkAZOA44EZwB+dOlTdUNU5qlqjqjWjRo1K0EdbAjHNWSxO56Wo5mY47jgTZhkVK1dm3hIUQL9+pqR6kIiobIiEcumZmLfe+SXbhLwuLrzwQp599tmUf24kYnEF8BHw/5ztI+dYOBqBMX7PK+idn9EALFTV/apaB6zEiIfFI1wndzon5912G7z2mtlHjBs2m+7d8YIRovpsNolFz5IfmZxjUVJSgoj02krcqokxcuyxxzJ8+PAEWRk5IcXCcVL/SVV/rapnOdudqrovgrEXAZNEZLwTTXUusLDHOU9iZhWIyEjMstSaaH8IS2KpqalJ25lFczM88ICJx3/ggShmF+vXw65dmTmzgJBikQ0JeS49ZxaZLBYtLS1RHU93Qpb7UNUOERknIgXRFg9U1XYRuRLTWc8VnQ9F5FZgsaoudF47WUQ+AjqA76nqpth+FEuiqK6u5tFHH2XLli0MS7PmCDffDPucryodHWZ2cffdEbwxUyOhXCZONNlqO3aYphV+1NfD8OGmqV6mU1Zmfsz2dsjLS2+xuPrqq3n33Xdjeu/xxx8f8PjUqVP5zW9+E4dVySOSZag1wH9E5EYRudbdIhlcVZ9R1U+papWqznaO3eQIBWq4VlUPUdXJqvpw7D+KJVGkq5O7qQnuv9+sKAG0tUUxu8h0sXAjotb0nnhnQySUS1lZ9yzuxsZG8vPzGTlypLeGWSIqJLja2XKAQWHOtWQB/pncJ510ksfWdHHuub3rBkU8u6itNeE2Y8cmzb6k4h8+e/jh3V6qq4MUB8YkDf+OeWZJqonS0lJyciLqppBSws0AQoWwvvTSSwm2JvmEFAsRGQX8HZMvYetB9RGGDx/O+PHj08pv8eqrZutJWxu8/noEA9TWmrLkeZF8P0pDXLHoERGlCmvXwmmneWBTEvAv+QHpl2ORDsyYMYOXXnqJjRs3UlFRwS233MIll1yS9M8N+p8jIpcCP8XMKsaLyCx3+ciS/aRTJvcnn8DXvgYHHQRvvQVDhpjQ2fZ2+M9/IhwkEwsI+jNkCIwY0cvJ3dICe/dm1zIUdDm5m5qaOPjgg70zKA6Ki4sDOrPjLYj40EMPxfX+WAk1t7saOFRVjwaOwZTlsPQRampqWLNmDZs3b/bUjj174Mwzzf7JJ809E2DyZHj//S7/RUg6O8038kwWCzBO7h5ikQ19LPzpmcXd2NiYsTOL9evXo6q9tvVRJwilB6HEok1VNwCo6hpMHwtLHyEdnNyqcPnlsGQJLFgA/l8wJ082gUFr10YwUGOjUZtMF4sA4bPZFDYLZpVw9GgjFrt27WLbtm02IS9NCCUWFSLyW3cL8NySxaRDJvddd8G8eXDrrfCVr3R/bcoUs3///QgGciOhMjUhz6WqCtatM44aB3dmMW6cNyYlg7Iy47NodhwXmTqzyDZCefu+1+N5eixgW1LC8OHDmTBhgmeZ3M8/D9/9rlmCuuGG3q+70T/LlvUWkl5ketisS1WVWVKrrz8gfPX1pp7SwIGeWpZQ3MS8dM6x6ItE1M/C0jeprq72RCzq6uCcc4xDe+5cCBQ1OWiQWaePeGZRWGjqK2Uy/uGzjlhkQx+LnpSWwuLFVizSjfQLXrakDdXV1dTV1aXUyb1rl5lNdHTAU0/1SlbuhuvkDkttrbnRpmGsflQEKFWeTQl5Lm4W97p1RiyszyI9yNCgc0uyKSkpORD2N2LEiAPHi4uLkxbNoQqXXGKWlp55pqvnTzAmT4a//92U/+gXKvyittZMUzKdkhIoKjogFp2dxsF/5pke25VgysrM38KqVTsoKipicDbUMUkglZWVDBo0iNzcXPLy8lI2+7diYQmIF0XQfvELeOQRuP12+FLAzibdmTLFzECWL4epU4Oc1NFhbq7ZkLUm0i0iqrnZ+LqzcWYBUF/fRllZWdo3G/KCF198MeUlUCJpqzpKRK4XkTki8id3S4Vxlr7Ds8/CD35gfBXXXRfZeyZPNvtly0Kc9Mkn5o6a6c5tFz+xyKbS5P64JT8aGzUr/BUx919JMyJZxH0KGAL8G1P6w90sloSwahXMmGFmCvffb75AR8KkSWb5KaTfIlsioVxcsejszLqEPBdXH1pbc7PCXxFT/5UQiAgnn3wy1dXVzJkzJzGDRkAky1BFqvr9pFti6ZPs2AH/9V+mNfaTT8KAAZG/Ny8PDjkkjFisXGn22SQW+/ZBUxN1dSa6K5tyLMAk5eXkKFu3FqX1zOLqqyFchfJ9++Dtt41/6Q9/gKVLoaAg+PlTp0K4CuWvvfYa5eXltLa28oUvfIGDDz6YY489NvofIEoimVk8LSKnJt0SS5+jsxMuuAA+/tj4KmJZTpk8OcwyVG2tcQqn8U0nKvwiourrTXmM/v09tSjh5OXBqFFKe/uotBaLSFi7tqskjVv0MV7c2dbo0aM588wzefvtt+MfNAIimVlcBVwvIm3AfueYqqoNUchiklUEzZ+f/hSeeALuvBNirYQ+ZYrJ8t64EQL6+9wCgtniJHVDxFavpr7+uKzzV7iMGLGPlpZSysrSN9sw3AygudkUOvYXiy1b4OGHTWBbLOzatYvOzk4GDRrErl27+Oc//8lNN90U22BREnZmoaqDVDVHVQudx4OsUGQ//kXQDjnkEE477bSEFkH729/gppvgG9+Aq66KfRzXyR10KSrTq832ZOxY89V79Wrq6rLPX+EyePBOoCyjZxa33WZmz/64/VdipaWlhc9+9rMcfvjhTJ8+nS9/+ct8KZLQwQQQUeisiJwOuItiL6nq08kzyZJu+Hw+nn32WVQ1IWGMH38M550H06bBvffG96Xfv0bUCSf0eLG93aQ4f+1rsX9AupHMASq6AAAgAElEQVSXB+PG0blqNevWwdlne21Qcigs3AyUUV6+02tTYuaNN7qV8QKi6L8ShAkTJvDee+/FZ1iMRBI6eztmKeojZ7tKRH6WbMMs6YPP56OlpYWGhoa4xmluhs98xqQ89OtnlqDiXW8vLjbLTwFnFvX1RjCyaWYBUFVF+/JVtLdnX9isS05OK1DMyJGlXpsSM0uXmqWnntvSpV5bFhuRzCxOBaaqaieAiMwFlmL7W/QZfD4fAIsWLWLMmDExj3PrreZblQi89BLEMdQBRMzsIqCTO9vCZl2qqpDXjVMzW5ehOjvNF5MdO4oO9DCxeEukxXKG+j22v7o+xuGHH05+fj6LFi2KeYzmZrjvPvM4Ly+x1cInT4YPPui9PpzNYpG/cyvD2Jy1M4t9+0yjDrcJksV7IhGLnwFLReRBZ1axBJidXLMs6US/fv2YMmVKXGJx9dVmRQjMbCBRCUpgxGL3blizpscLtbWmEuHo0Yn7sHTAiYiqYjVjx3psS5LYsWMFYMUinYgkGuoh4CjgceAx4GhVfSTZhlnSC5/Px6JFi+js9fU9PM3N8NhjXc/b2uCBBxJX/iBoI6RsC5t1cXItaoatprDQY1uSxObNHwJWLNKJoGIhIgc7+2lAKdDgbGXOMUsfwufzsX37dmrdpZ0ouO46EzLoT7whhP4ceqjRg15isXJl5nfHC8SECQBMHbQ6zImZSWdnJy0t7yPSidMsz5IGhJpZXOvsfxVg+2WS7bKkGf5O7mh57rnex+INIfSnqMiszHRzcre1mXTZbPNXABQV0ZJbysF5q7y2JCls3LiRjo42Bg/ebWcWPVixYgVTp049sA0ePJjfhMsOTBChOuXNch6eoqp7/V8TkSyd/FqC8elPf5qioiIWLVrEeeedF/H7tm6FPXtg5kyYPz959vVqhLRmjfF4Z6FYtLdDbUcV4/Zn58zC7ZA3cuR+KxY9OOigg3jXKUjV0dFBeXk5Z6aooUkkDu5A3/8S9J3Qkink5eUxbdq0qGcWf/gD7NwJ3+vZ0T3BTJ5sXBS7dzsHsjUSCmhogFVUMWpHdopFY2MjYEqVZ/wy1IIFJhkmJ8fsFyxI2NDPP/88VVVVjEtRJclQPosSEakG+ovIESIyzdmOB4pSYp0lrfD5fCxdupT9+/eHPxnYuxfuugtOPhkOPzy5tk2ZYhKePvrIOZDFYlFfD6uYyICtTWbalmW4M4tx4/Ize2axYAHMmtVVTXDtWvM8QYLx8MMPM2PGjISMFQmhkvK+CFwIVAC/9ju+A7g+iTZZ0hSfz8edd97Jhx9+yNSgrem6mD/fRDwlc/nJxb8RUk0NRiyGDQO/lrDZQn09rMapPrtmjfHwZxGuWEyY0J8NG4z7KVRZb88IV6P8zTdNjXJ/du82vYP/+MfA74mkRjnQ1tbGwoUL+dnPUldMI+jMQlXnquoJwIWqeoLfdrqqPp4yCy1pQzRO7s5O0yZ12jQ48cRkW2YChIqK/PwW2VZA0I+6OlhDV6nybKOpqYnRo0czZkwukMEd5noKRbjjUfCPf/yDadOmJbQKdDjClvtQ1cdE5MvAoUCh3/Fbw71XRL4E3AXkAvep6u1Bzvsq8CjgU9XUdB+3RE1VVRXDhg1j0aJFXHbZZSHPXbjQRK4+/HBq0hxyc80X7G5i8bnPJf+DPaC+HnaXVkEzps1gltHY2Eh5efmBFiTNzaRn8mG4GUBlZeAGFuPGmXo3cfDQQw+ldAkKIisk+AfgHOB/AAH+GwjrURGRXOBu4BTgEGCGiBwS4LxBmEKFb0VluSXliAg1NTURzSzuuMPULfrqV1NgmMOBRkh79pje21k6s6ivh2FVw2HIkKydWZSVlR0Qi4z1W8yebaa7/hQVmeNxsGvXLv71r39x1llnxTVOtEQSDXWMqp4PbFHVW4CjgUgynaYDq1R1jaq2AQ8DZwQ47zbg58DeAK9Z0gyfz8f777/PnhCO1f/8x5RnvvZaUwcqVUyZAhs2wMa3VhuHYjYm5GGWoSrHS1c/7izDFYvSUve5t/bEzMyZMGeOmUmImP2cOeZ4HAwYMIBNmzYxJMUVFiMRC/eusFtEyjDd8iKpG1wOfOL3vME5dgAnE3yMqv49gvEsaYDP56Ojo+NArHcg7rjD+JUvuiiFhtHl5G54MXsjodraoLHRKU0+cWLWicX+/ftpbW2lrKyMUaPM8mJGh8/OnGmmgp2dZh+nUHhJpD24hwK/AN4B6oGH4v1gEcnBRFl9J4JzZ4nIYhFZvGHDhng/2hIH4Zzcy5cbf8WVV8KAAam0rEssti/JXrFoaDD3ncpKzMzC7dmRJbS0tKCqlJeXk5tr2o9m7Mwiy4ikkOBtqrpVVR/D+CoOVtUbIxi7EfDvWFDhHHMZBBwGvCQi9ZhihQtFpCaADXNUtUZVa0aNGhXBR1uSRXl5OaWlpUHF4pe/NA2Nvv3tFBsGjBplbi66stY8ycJGCHWmcrfpY1FVZYTik09CvieTcBPy3HaqZWVWLNKFSBzc33ZmFqjqPiBHRL4VwdiLgEkiMl5ECoBzgYXui6q6TVVHqmqlqlYCbwKn22io9MetQNuTpib485/h4ovNvdoLpkyBAc3ZGzZbX2/2B2YWkFURUW6OhSsWpaXpJxaq6rUJIUmWfZEsQ12mqlv9DNkChI6bNOe1A1cCzwHLgb+q6ocicqvT09uSofh8PlasWMG2bdu6Hb/rLlNN9tprg7wxBUyeDGU7a+msyl6xyMmBigq6xCKL/BY9xaKsLL18FoWFhWzatCltBUNV2bRpE4VJqF0fSaxKroiIOlfHCYmNKJ9SVZ8Bnulx7KYg5x4fyZgW73H9FkuWLOFEJ+Nu+3ZTB+q///tABW1POOJTuyijiQ3DJ5GNC5b19aYdbX4+UF5umplnmVjk5eXhLjeXlcHGjSaPrV8/j40DKioqaGhoIJ19p4WFhVRUVCR83EjE4lngERG513n+TeeYpY9SU2PcSosWLTogFnPmGMFIdsHAcFQPMUsyqyQ7xaKujq5Wqjk5RpmzSCwaGxspLS0lJ8cserjhs+vXm8hTr8nPz2d8tjY+D0Mky1DfB14ErnC254HrkmmUJb0ZMWIEEyZMOOC3aGuDO+80ZT2qq721bUKHiYRaujN7l6G69d3OslwLN8fCxT+L2+ItkZT76AR+72wWCwDTp0/ndad70V/+YpyQf/qTx0YBBWuNWLzSPIlIojAyiX37zHXu9sW2qgpefNEkIWZB+9impiY+5ZdMmfFZ3FlEqBLlf3X274vIsp5b6ky0pCM+n49169axfn0rv/iFiUI6+WSvrQJWrmRzYSmLlg/02pKEs26d0YReM4tdu6ClxSuzEkqwmYUVC+8JNbO42tmflgpDLJmF6+T+/e/X8tFHo5k/P02+2NbWsqNkEmvWwI4dMGiQ1wYljm5hsy7+EVElJSm2KLHs2bOHLVu2UF7eVehh5EhTMsaKhfeE8lk87ex/oqpre26pMM6SvkybNo2cnBwefHA0Y8fC2Wd7bZFDbS1MNP6KDz/02JYE44pFt2WoiRPNPgv8Fj3DZsH48EtKrM8iHQg1sygQka8Dx4hIr/KGtqdF32bAgAFUVp7LmjXj+M1vnFBOr9m+HVpbGTRtEvzbVKA96iivjUocdXXmW7bfvbSrZWeWioV5bmcW6UAosbgcmAkMBb7S4zUFrFj0cTo6rkFkCxdfPBRTvd5jnFaqQ32TGDTIr7dFluDmWHSr5FtQYA5msViUlmbFj5fxBBULVX0NeE1EFqvq/Sm0yZIBrFwJ69ZVozqbzZu/waBBaRAE74hFzkGTOOyw7BSLgCH+WRI+G2pm8eqrXlhk8SdUNJTbDHOLiJzVc0uRfZY05Ve/gvx8BX7H22+/7bU5BkcsqKo60AgpTasyxES3hDx/qqqyoj5UY2Mj/fv3Z+jQod2Ol5XB5s0J6UZqiYNQDu7jnP1XAmw2QqoPs349zJ0L55+vFBRsjahzXkqorTVFk4qKmDIFtmzJnrXuPXvMdQ8qFhs3Gp9NBuOGzUqPsDqbmJcehFqG+rGzT3ELG0u687vfmazt667L5b33Dk8vsXASutzeFsuWmRJKmc66dWYfcBnKPyLqiCNSZlOi6Zlj4eLfMS+gWFpSQiQlyq8SkcFiuE9E3hGRdEi/snjAzp1wzz1w1lmmCrjP52PJkiV0dnZ6bZpxpDilyV2xyBa/hdvHIujMAjLebxFMLOzMIj2IpDbUxaq6HTgZGAF8A7g9qVZZ0pb77oOtW7sKBvp8Pnbs2MGKFSu8NWzzZrM5YjFsmFmRyhaxCJiQ55IFYqGqNDY2dkvIc7FZ3OlBJGLhLiCeCsxT1Q9JizhJS6rZvx9+/Ws49lg48khzLFyb1ZRR27uVquvkzgbq600uS4Av3iZNfdSojHZyb9++nd27dwecWYwYYbO404FIxGKJiPwTIxbPicggIA3WHCyp5pFHTAfP6/xqDh988MEMGDAgLcViyhTTE3z/fo9sSiB1daZEd06w/9gMD58NFjYL5mdOx455fY1IxOIS4AeAT1V3A/mAdXr3MVThjjvg0EPhlFO6jufm5lJdXZ0eYuH2d3CYPNkIhdcrZImgV2nynmSxWJjj1mfhNZGIxdHAClXdKiLnAT8CtoV5jyXLeO45s/7/ve/1/nbr8/l49913aWtr88Y4MGIxdmy3dmpTpph9NvgtworFxIlm2pehyQiRiIWdWXhLJGLxe2C3iBwOfAdYDcxLqlWWtOOOO0wI6owZvV/z+Xzs27ePDz74IPWGudTWdluCAjjoILPWnelisXs3tLYGCZt1qaoy0z/XE55hNDY2AsHFwi5DeU8kYtHu9N8+A/hfVb0byKLCz5ZwLF5s+utcc40pRdQTz53cqgHFoqAAPv3pzHdyh4yEcsnwiKimpiaGDBnCgAEDAr5eVmaSLPfuTbFhlgNEIhY7ROSHwHnA30UkB+O3sPQBmpvh1FNh4EC47LLA54wfP54RI0Z4JxYbN8K2bQcS8vyZPDnzZxZRiUWGRkQFy7FwsbkW3hOJWJwD7AMuUdX1QAXwi6RaZUkbvvtd2LDBfGkfPDjwOSJCTU2Nd2KxcqXZT+rdd3vyZJP9vHVrim1KIAH7WPRk9GgYMCCjZxaRiIVdivKOsGKhqutV9deq+qrzfJ2qWp9FH6C52YTLgglBXb8++Lk+n48PP/yQ3bt3p8Y4fwKEzbq4Tm4v3SnxUldn/PbFxSFOEsnoiKhgCXku/iU/LN4QSbmPo0RkkYjsFJE2EekQERsN1Qe4+Wbo6DCPOzvhttuCn+vz+ejo6GDp0qUpsa0btbWQmxtwnca/RlSmUl8fJsfCZeLEjBSLzs5Ompub7TJUmhPJMtT/AjOAWqA/cClwTzKNsnhPczM88EDX87Y28zzY7MJTJ3dtrVmjCdCur6IChg7NbL9F0D4WPamqMtOQdKjTFQWbNm1i//79IcVixAjz67UzC++IRCxQ1VVArqp2qOoDwJeSa5bFa267Ddrbux/r6Ag+uygtLaW8vNw7sQiwBAVmdSbTndxB+1j0pKrK5Fk4YaiZQrgcCzC/Rxs+6y2RiMVuESkA3hWRO0TkmgjfZ8lgXn21d+OgtjZ4/fXg7/H5fKkXiyBhs/64YpGJjZB27IBNm6IQC8i4iChXLEL5LMBmcXtNJDf9bwC5wJXALmAM8NVkGmXxnksvNXv3JutuoVwSPp+P2tpatqYy9Gj9eti1K6RYTJli+gK5PSEyibVrzT7iZSjIOL9FuIQ8F5vF7S2RREOtVdU9qrpdVW9R1WudZSlLFjNvHkybBocdFvl7pk+fDsDixYuTZFUAQkRCuWSykztkH4uejBljUtYzTCzcmUVJSUnI86xYeEuoHtzvi8iyYFsqjbSklg8+gHfegQsuiO59NTU1QIqd3K5YBEjIc3EFLxP9FhEl5Lnk5ZkpSAaKxahRoygIVB7Aj9JSky+zZ0+KDLN0I2hbVWyf7T7LvHnmvhOoDlQohg4dyqRJk1IvFgUFpohgEAYPNjfbTBWL/v1Nzl1EZGCuRbiEPBf/8Fm/4sKWFBFqGSofqHCWoQ5smAzuUCJzABH5koisEJFVIvKDAK9fKyIfObOV50VkXGw/hiVRdHTA/PmmxMeoUdG/P+VO7pUrzZ0jNzfkaZnaCMmNhJJI2425YpFB3vxwCXkuNovbW0KJxW+A7QGOb3deC4mI5AJ3A6cAhwAzROSQHqctBWpUdQrwKHBHJEZbksfzz5tvbuefH9v7fT4fDQ0NrA+V7p1IwkRCuUyZYvpaZFoF77ClyXtSVWXqZG3alCSLEk+kMwubxe0tocSiWFV7TdydY5URjD0dWKWqa1S1DXgYU7nWf6wXnYZKAG9iZi0WD5k71/SvPi3GRciUJud1dpow0QjEYvJkM2tavjz5ZiWSiBPyXDIsIqq9vZ2Wlpaol6EsqSeUWAwN8Vr/CMYuBz7xe97gHAvGJcA/Ar0gIrNEZLGILN6wYUMEH22Jhe3b4Ykn4Nxzu/UQioojjjiC3Nzc1IhFY6OpWR3hzAIyy2+xbZspyx31zAIyRixaWlpQ1YjEYvhw456yMwtvCCUWi0WkV1FqEbkUWJJII5wOfDUEqWarqnNUtUZVa0bFspBuiYhHHzWRJrEuQQEUFRVx6KGH8vbbbyfOsGBEEDbrMmmSEcBMEouoIqFcXM9vhohFpAl5YPw2NnzWO0I5qq8GnhCRmXSJQw1QAJwZwdiNmAQ+lwrnWDdE5PPADcBxqpphK8rZxbx55qZ65JHxjePz+XjiiSdQVSRiz2wMRCEWeXmZ1wgpJrHo39+0NMwQsYg0Ic/FlvzwjqAzC1VtUdVjgFuAeme7RVWPdvpahGMRMElExjvlQs4FFvqfICJHAPcCp6tqa2w/giUR1NfDyy+b3Ip47+8+n4/NmzdT52aUJYvaWigsNNUCI2DKlMycWUTls4CMCp+NpC6UP7bkh3dEksH9oqr+ztleiHRgVW3HlAh5DlgO/FVVPxSRW0XkdOe0XwADgf8TkXdFZGGQ4SxJ5s9/Nvvzzot/rJQ5uWtrTVnusLW7DZMnm2+lmRIoVFdn+hmNGBHlG6uqMqY+VFNTE7m5uUS6vGyXobwjqQUBVfUZVf2Uqlap6mzn2E2qutB5/HlVLVbVqc52eugRLclA1SxBHX+86ZsQL5MnT6Zfv36pEYsIlqBcMs3J7YbNRj3Tq6rqqpmV5jQ1NVFSUkJumDwZl7Iy4/jPgB8t67DVYy28+ab5IhpteY9g5OfnM3Xq1OSKRUeHWWqJQiwyrUZU1GGzLhMnmv2aNYk0JylEmpDn4uZa2KWo1GPFwsLcucYv+tUE1hL2+XwsWbKEDrfVXqJZt87UTI9CLEpKYOTIzJhZqEbRx6InGRQ+G2lCnovNtfAOKxZ9nL17TZ/ts86CQYMSN67P52PXrl18/PHHiRvUnygioVwyqRHS1q0m78WKRXdsyQ/vsGLRx3n6aXNjStQSlEvSndwxiAUYsfjgg/TvPBpzJBSYFPxhw9JeLPbu3cvmzZujEgtb8sM7rFj0cebONd/WTjwxseMedNBBDBo0KLliMWBA190jQqZMMc7RZEf1xktUfSwCkQERUc3OWlI0Pothw0xypV2GSj1WLPowra3wj3+YcNkIg1EiJicnh+rq6uSKxcSJUYcKZYqTO6aEPH8yINci2oQ8sFncXmLFog/z0EMmqCie8h6h8Pl8vPfee7S1tSV+8CjDZl0OPdTccNLdb1Ffb/pwDBsW4wATJ5qerPv3J9KshBJtQp6LFQtvsGLRh5k7F6qrzQ00Gfh8Ptra2liW6K/x7e1mnSZEd7xgDBhgvnSnu1hE3ceiJ1VV5ptAGjcej1UsbMkPb7Bi0Ud5/31YujR5swpIkpN7wQKTOdjeDvfea55HSSY0Qoq6j0VPMiAiqqmpicLCQoZFOX2yJT+8wYpFHyXW1qnRMG7cOEaOHJk4sViwAGbN6vpauWmTeR6lYEyZYny/u3eHP9cLVONIyHPJALFobGykrKws6mKTZWUmrHjnziQZZgmIFYs+SHt7fK1TI0VEEttm9frre9/hd++GG26IapjJk03o7EcfJcasRLNpk7kRxjWzKC01RRbTOCIq2hwLF5vF7Q1WLPogzz9vSgclcwnKxefz8dFHH7ErnmI+27fD738ffP09ynX5dK8RFXckFJjiihMmpPXMIlaxsFnc3mDFog8Sb+vUaPD5fHR2dvLOO+9E/+YlS8wyU1kZfOtbkJ8f+LyxY6MadsIEU94kXcVi6VKzjzujfuLErBYL6+ROLVYs+hiJaJ0aDVE7uXftgvvuA58PamrMetk558Bbb8EDD0BRUffzi4pg9uyobMrNNRFg6erkfuABs3/ooTgHqqoyxQRV47Yp0ezYsYOdO3dGlZDnYsXCG6xY9DEefdTUg0p0eY9gFBcXM2bMmPBisWwZfPvb5k5w2WXGyN/9ztwR7r8fpk+HmTNhzhwTDSVi9nPmmONRkq6NkJqbwe1I+5e/mOXCmNm82fh0cnPNmlYMkWPJIpaEPJchQ4w7xopFarFi0ceYO9ekJ0yfnrrPnD59OiUvvGBuWDk5XTeuPXuMQcccA4cfbkThjDPgtdeMeFx5JQwd2n2wmTPNon5np9nHIBRgnNytrdDSEucPl2AuuMCkR4DZ33ZbjAMtWAAPP2weq5oEvRgix5JFrDkW0JXFbX0WqcWKRR+irg5eecU4tpPZGrsn5+fl8ZPWVnPDcm9cF15oWsBdeKH5BvzrX0Njo4np/cxnkm5gujm5VeEHP4B//avrWFubWZKKaXZxww2wr0dL+xgix5JFPGJh3mdnFqnGikUfYv58s//GN1L7uV948UUG9DzY3m4E4cUXYflyuOaaGPqHxk461Yhqa4OLLoKf/7x3h9iYZxcJihxLFvGKhc3iTj1WLPoIbuvUE06IOngodrZtg/nzKWxtDfz6nj2ml2sqpzkOo0ZBcbH3M4vNm+Hkk81qXElJ79LpbW3w+usxDBzslxzjzTnRNDY2MnjwYAYOHBjT++0yVOqxYtFHeOMNk5+V9NyKLVvgwQdNXO7o0fCNbyDBStqmTLUC47WTu7YWjjrK/G4WLDA3P9XemxtKGxWzZ/eOHAOT7ReT+iSWWMNmXcrKYMcOs1lSgxWLPsK8eebekcjWqQfYuNE4p085xQjERReZu/CVV8Ibb/CtoiJ6puTtAq7YsiUJxkTO5Mnw4YddDuVU8uqrRii2bIEXXoCvfz3BHxAocuxnPzN9ZY8/Hu65x9OQ2kSIBdjZRUpR1Yzaqqur1RIde/aoDh2qet55CRy0pUX13ntVP/951dxc8yV4wgTV665Tfftt1c7OA6cCOgO0DrTD2c8ANX9+3vHgg8bsmhrV5ubUfe68ear5+aoHHaS6alXqPldVVbdsUT31VPODX3ih6u7dKTbAUFlZqefF8Qf573+bH+HFFxNnU7YDLNY47r12ZtEH+NvfTOvUqJegFizoHu56zz1mO/FE42H85jdNZNP3vw/vvGPWuX7+c5NQ18MP8RAwHsh19vHmmyUC18m9ZEkcIapRoAo33WR+D5/9rFl+cuv9pYyhQ80fxI9/bJYLP/c58ztMIapKU1NTTAl5LnZm4QHxKI0Xm51ZRM+Xv6xaXq7a3h7Fm+bPVy0qCrSErnrwwao33qj63nvdZhDBwJlFBNq8ZM2arh+pf//kzi727FGdMcN81sUXq+7bl7zPipiFC1UHD1YdMcJ8VU8RGzZsUEDvuuuumMfYutVcy1/+MoGGZTnYmYUlFC0t8OyzMbROve66wDW8S0tNqOuttxoPcZyRTG+88UZc74+HX/yiy/w9e+Ckk+DNNxO/lL9hgxn7oYfg9ttNNZOCgsR+Rkx85SuwaJEJCzv5ZHNBEv3DByDesFkwXQSLimz4bCqxYpHlRNU6VdV4Xk8/Pfh/YVz1J3pzzDHHcNppp/Huu+8mdNxwNDebhDf/e+NHH8HRR8PBBxtfcEND/J/z8cfGkf3OO/B//2dW7DyIFA7Opz5lFPKss8wXhHPOSXqjiESIhYjNtUg1ViyynLlzTT2+Qw4JcVJHhykadfTRcOyxJrRyyJDA58YQ7lpcXBzw+OjRo/npT3/Kf/7zH4444gjOPvtsli9fHvX4sXDbbb1zGgoKTB5KSYlpnTF2LHzxi0Zw9+yJ/jNeeMFc0l274OWX4WtfS4ztCWfQIPjrX42/6bHHjLrV1ibt41yxiMdnATbXIuXEs4blxWZ9FpHz3ntmXfe3vw1ywq5dqnffrVpVZU6sqlK95x5zPJDPoqjIHE8wW7Zs0RtvvFEHDhyoOTk5ev755+vq1asT/jn+TJ0a2B0zdap5fdUq1ZtuUh03zhwfPFh11izV11+PyE2j99+vmpeneuihqvX1Sf1REss//6k6fLjqkCGqf/tbUj7i1ltvVUD37t0b1zjnnKM6aVKCjOoDEKfPwvObf7SbFYvI+e53zQ2rtbXHC62t5k44YoT5EzjySNVHH+3tAZ8/39wtRcw+CULR3axW/c53vqOFhYWal5enl19+uTY0NCT1M8PR0aH6wguq55/fpZ2TJqnOnq26bl3XeU1Nqsceq9rYqPr975vzTj7ZOGIzjro61SOOMD/Ej39sLkICKC4uDhhCXVxcHP1g8+frxoHjtAPR/RXjkv63mQ1YsbAEZP9+1ZIS1TPO8Du4cqXq5ZerFhaaX/3pp6u+8kpkX5VTSENDg15xxRWan5+v/fr102uuuUZbeyle6tm+XfWBB1SPO85cPhHVL3xBdcEC1UsvVc3JMakmYC7z/v1eWxwHu3cbhXSnWxUVcX9pmAG6s8dUbqcjGFGRwllvNhGvWIgZIzmIyJeAuzDh9fep6u09Xu8HzAOqgU3AOapaH2rMmpoaXbx4cXIMzgYWLKD9+zeQ07iOdYyl9arZTD9ngol0efJJ023u/JLPQgoAAA/tSURBVPPhO98xntw0pq6ujltvvZV58+bRv39/gIDtWYuLi1mfYMd7ONasMVnxc+d2tUF1uflmk0+RVo7sWFA12fhz53Y/XlgIN95oQrz27Yto69y7lx0//SmBPGE7gEHf/GbXBQu093/84IMB63x0lJSRU7cGKUxBV68MRESWqGpNzAPEozShNoxArAYmAAXAe8AhPc75FvAH5/G5wCPhxq2G+Ked8+fr/vIETGETNU6ixgrwjaszJ8c8HjZM9YYbUpuqnCCWL1+uZ599dkLzNYqLiwOOE+2SyOjRJQpPKOx3Lvk+hf+NaWklUTYlcqx17t9PnFsHaGeQ1zpBdfRos40aZbaRI80y6YgRxocyfLj5Gx42LOTntJOjjQXj9P3ik/TtabN08Tl36PLZj+nGF97Tzh07A/+Q6fZ/nCSbqs3/SfrNLETkaOBmVf2i8/yHjjj9zO+c55xz3hCRPGA9MEpDGFUjoovBBFnH0iVtwQLTBMY/hyCWsRI1TrCx+veHX/0KPfmLtG3ZRdvmnbRv3cn+rbvo2LaTjm076dy5C92+E925E3buouKFeeTt6/3Nm2HDTGnqGCt8pgsS4qv6CSecwODBgxkyZMiBvf/jnvtx48YFHSua/wmRUmAN0N/v6G5gAqrRzXZC/XzR/p9GOpaq0tnZGXQbNHRowJDJTuCuk06iYcMG1rW00LBhA7s7O9kH7APagOGlpZRWVlI2fjxjxo/nktmzqQwwVj1QGcXP115RSV5j76zz3f1H8OIh36Z/4yqGbVnNmH2rGMmmbudszCth49AqdpdOhKoqimU9Zc/8Cdm3t+ukLL231ACLVWOe7yZTLL4GfElVL3WefwM4UlWv9DvnA+ecBuf5auecjcHGPSAWwH7yWFcwMSq7xratIp/2XsejHSvYOO3k0ZQ/FkERNV/oxH8LcGx4xwZy6ew1VqTspj87GcgoNhDwL0Gkd5xoBhLqBvjZz36W7du3s23bNrZt28b27dvpjPFnHj169IHPEpFuW89j69Z9H7gYKPQbYS9wPxMm/Dqqz12zZk3Q1yoqKiL+BtjZ2cnWrVuDjpWXl3dADMJRB0Fv8J+rqKCysjLgNmbMGAp6ZB5+XYQ/QrfeJruAy4C/RHEf+tPnF3DO87MYQNfNdBdF/PXzc7joX10307Y2WLdsK83/Wc2OpavpWLGKgk9WM2zTKsr2rqaCxqCf0YmwWwagkoMidJJDJzmo5AR8DEJJewN5Qe4tnxT0ruuigf9bQ96j1hZMCnVpejGurfbAWPGKRV6sb0wlIjILmAXGueGSRzsto6dENdaEho8DHo92rGDj5NLO2rKjAUFFzB6z5npg7x539iesmhNwLAWePONBdMBAGDAAGTQQGTSQ3MEDyBk8kNwhA8kbXEThgFz69YNBp1XSv7X3N6728rGZ8YuOg1dffbXbc1Vl165dBwSk5/7SSy8NOtaZZ555YAz/LdCxuXOPprtQ4Dw/hmOOeTOqnyGUWHzhC1/oJVyBtpycHESE3/3ud0HHuu6668jJyem25ebm9jqWk5PD9ddcE/AGfz3wySefRPXzvVBczGUtLfwUGAusc8Z5IUgeTjB+t2km/wZ+yg2Mxfjmrmc2yzfO5CK/8woKYGLNUCbWVNP9zgH798OaFbsZP3kgQm+hEpRXDpqF0Il0dpq9KqLmcY52gpq9OFv52j8HtDePdtYXT+1xNLg4Tvgk+D1qw+jDgr4vEFUNCcxbimcNK9QGHA085/f8h8APe5zzHHC08zgP2Igz2wm2VfutT+6vGBfNyp2qqu4vHxdwrTPasRI1TiLHuv+k+bqT7j6LnRTpnz6fHVEiJNBnkaix0tGmRI4F6VkxOFGk4/9xsmyK12eRzAzuRcAkERkvIgUYB/bCHucsBC5wHn8NeEE1svnoLor488GzozZq3sGz2UX3pjCxjJWocRI51u82zeQy5lDPODoR6hnHZczhtxujXOdMU4Jlggc7bkkM6VgxOFGk4/9xsm2KmXiUJtwGnAqsxERF3eAcuxU43XlcCPwfsAp4G5gQbsxq0DrG6QzmH8i2jYapU1VnMF/rMJEGsY6VqHESPZYlMhIVLZSOEUyJHCuRNqUj6fh/nCyb0jYaKlnYPAuLxWKJnnjzLGwhQYvFYrGExYqFxWKxWMJixcJisVgsYbFiYbFYLJawWLGwWCwWS1gyLhpKRHYAK7y2owcjMQmF6UQ62gTpaZe1KTKsTZGTjnYdpKqDYn1zJlaBWBFP+FcyEJHF1qbISEe7rE2RYW2KnHS0S0Tiyjmwy1AWi8ViCYsVC4vFYrGEJRPFInCJVm+xNkVOOtplbYoMa1PkpKNdcdmUcQ5ui8VisaSeTJxZWCwWiyXFZJRYiMiXRGSFiKwSkR94ZMMYEXlRRD4SkQ9F5Crn+HAR+ZeI1Dr7YR7YlisiS0Xkaef5eBF5y7lejzil4lNpz1AReVREPhaR5SJytNfXSUSucX5vH4jIQyJS6MV1EpE/iUir0y3SPRbw2ojht459y0RkWgpt+oXz+1smIk+IyFC/137o2LRCRL6YKpv8XvuOiKiIjHSee3adnOP/41yrD0XkDr/jSb9OwewSkaki8qaIvCsii0VkunM8+msVT8naVG6YcvqrgQlAAfAecIgHdpQC05zHgzAl2A8B7gB+4Bz/AfBzD2y7FvgL8LTz/K/Auc7jPwBXpNieucClzuMCYKiX1wkox3QK7e93fS704joBxwLTgA/8jgW8NphS//8ABDgKeCuFNp0M5DmPf+5n0yHO/2A/TJuL1UBuKmxyjo/BNE9bC4xMg+t0AvBvoJ/zfHQqr1MIu/4JnOJ3fV6K9Vpl0sxiOrBKVdeoahvwMHBGqo1Q1WZVfcd5vANYjrkJnYG5OeLs/yuVdolIBfBl4D7nuQAnAo96YZOIDMH88d4PoKptqroVj68TJreov4jkAUVAMx5cJ1V9Bdjc43Cwa3MGME8NbwJDRaQ0FTap6j9V1W0I/SZQ4WfTw6q6T1XrMD1ppqfCJoc7gevo3p/Us+sEXAHcrqr7nHNa/WxK+nUKYZcCg53HQ4AmP7uiulaZJBblgH/T3wbnmGeISCVwBPAWUKyqzc5L64FUt2/7Deafp9N5PgLY6vePnurrNR7YADzgLI3dJyID8PA6qWoj8EtM++dmYBuwBG+vkz/Brk26/O1fjPk2Ch7aJCJnAI2q+l6Pl7y8Tp8CPucsZ74sIr40sAngauAXIvIJ5m//h7HalUlikVaIyEDgMeBqVd3u/5qaeV7KwsxE5DSgVVWXpOozIyAPMyX+vaoeAezCLK0cwIPrNAzzjWo8UAYMAL6Uqs+PhlRfm3CIyA1AO7DAYzuKgOuBm7y0IwB5wHDMks73gL86s3uvuQK4RlXHANfgzPRjIZPEohGzTulS4RxLOSKSjxGKBar6uHO4xZ3GOfvWYO9PAp8BTheReszy3InAXZippVvSJdXXqwFoUNW3nOePYsTDy+v0eaBOVTeo6n7gccy18/I6+RPs2nj6ty8iFwKnATMdEfPSpiqM2L/n/L1XAO+ISImHNoH5e3/cWdZ5GzPDH+mxTQAXYP7OwbSwdpfAorYrk8RiETDJiVwpAM4FFqbaCOfbwv3AclX9td9LCzG/GJz9U6mySVV/qKoVqlqJuS4vqOpM4EXgax7ZtB74REQOcg6dBHyEh9cJs/x0lIgUOb9H1ybPrlMPgl2bhcD5TgTLUcA2v+WqpCIiX8Isb56uqrt72HquiPQTkfHAJODtZNujqu+r6mhVrXT+3hswASfr8fA6AU9inNyIyKcwAR0b8eg6+dEEHOc8PhGodR5Hf62S4ZVP1obx4K/ERBTc4JENn8UsDywD3nW2UzE+guedX8a/geEe2Xc8XdFQEzB/mKsw3yr6pdiWqcBi51o9CQzz+joBtwAfAx8Af8ZEqaT8OgEPYfwm+zE3vEuCXRtMxMrdzt/9+0BNCm1ahVnbdv/W/+B3/g2OTStwIm5SYVOP1+vpioby8joVAPOdv6t3gBNTeZ1C2PVZjF/uPYxvtTrWa2UzuC0Wi8USlkxahrJYLBaLR1ixsFgsFktYrFhYLBaLJSxWLCwWi8USFisWFovFYgmLFQtLRiEiHU4FzQ9E5G/+VVCDnF8pIl+PYNxu54lIjYj8NkE214vI+872kYj8REQKEzG2xZIqrFhYMo09qjpVVQ/DFE37dpjzK4GwYtHzPFVdrKr/L1YjA3CCqk7GZNBOAO5NxKB+mecWS1KxYmHJZN7AKX7mZKL+wplxvC8i5zjn3I4p8PaumF4WlSLyqoi842zHBDnveOnqCzJcRJ506v6/KSJTnOM3Oz0EXhKRNSISVlxUdSdwOfBfIjLcGed7IrLIGf8W91wRuVFMD4TXxPTe+K5z/CUR+Y2ILAauEpFRIvKYM8YiEfmMc94Ax763nWKOKa/SbMke7LcSS0YiIrmYch1uYbSzMBnjh2Nq8iwSkVcwxQu/q6qnOe8rAr6gqntFZBIm67UmwHnH+33cLcBSVf0vETkRmOd8FsDBmDIPg4AVIvJ7NXWngqKq20WkDlO+ZgimBMR0TFbtQhE5FtgDfNX5efIxWcH+hSILVLXGsfUvwJ2q+pqIjMX0efg0JnP4BVW92Fmue1tE/n97d+8aRRSFcfj3LgoKiRaijWBvqlgICiklnZWFiB+FhV2ColgLKfwHVNBCtBIL8QvBRFFMEEGL6DaitaARRdSoSbEei3PXDItxsgqahfeBhdlh5s7dgZ1z71w4525EfKm9wWYdHCys16yW9JScUTwH7pT9Q8CliGiRCfkeAFuBTx3nrwROSRoEWmRq6TpD5IObiLgnaZ2kdo2AW5E1DOYlvSXTir9aQpvtjKTD5TNdvveRwaMfuB4Rc8CcpJsd51+ubO8ABrSQ5HSNMivyMJlg8ljZvwrYRN43s644WFiv+RYRg2WGME6uWXSzEH0EmCFH7A1g7i/7M1/ZbrGE/5SkfnKN5CUZNE5GxNmOYw7XNFOdHTSAbSWwVNsQsCsiXtT1yayO1yysJ0VmQB0FjpZF3ilgt7IO+XqySt9j4DM5Sm9bC7yOiO/AfrJcL784rmoK2As/X0+9i44aJktVRvxngGsR8YEMeAfLfiRtlLQBeAjsVNYI7yNThC9mAhipXKP9imwcGClBA0lb/qTPZuCZhfWwiJiW1AT2kBk/t5PZNQM4HhFvJL0HWpKeARfIB/UVSQeA2yyM0Jsdx01XLnUCOF+u9ZWFNOLduF8e2g3gKjBWfsOEpM3Ao/JMnwX2RcQTSTdKv2bIzKAfF2l7FDhd+rcCmCQX0cfICopNSQ2y/vjvgo7Zopx11myZktQXEbPlldskcChK/Xezf80zC7Pl65ykAXJh+qIDhf1PnlmYmVktL3CbmVktBwszM6vlYGFmZrUcLMzMrJaDhZmZ1XKwMDOzWj8AW5CYnZY0meQAAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX8AAAA1CAYAAACp8OvZAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAADTxJREFUeJzt3V1MHGUXB/Czy7J8G1pWQitWUk3TIIloGkIoNpC0EmKkkqgtCQWJRiBsmxKCXJDQxA80VWsN0WqRi4oGmoJKiL1zbdwUTWmxSmi1dRvshu62tWLBrmVndv7vBZl5l9qPmWcQoXN+SS+62Tl75tln/vOx06kNADHGGLMW+3/dAGOMsYXH4c8YYxbE4c8YYxbE4c8YYxbE4c8YYxbE4c8YYxbE4c8YYxbE4c8YYxbE4c8YYxbE4c8YYxbkWODPW5BnSSiKQna7uf2aLMvkcJgbnkgkQjExMaZqMMbuHpIkUWxs7L/9MTY9b7orj/ztdjtNTk6aquFwOCgUCpmqMR/BH4lETNdgjJknSZLpGgsQ/LotuvB/7bXX6ODBg6ZqnDx5kjZt2kTHjx8XDs9QKERJSUm0a9cu4T52795NNpuNjh8/LlwjJiaG9u7dS+Pj40QktjMIh8NzlpVlWbgfFT8QkC0ERVH+8zpXrlyhF198kS5evCj82cFgkGpra2nTpk00MzMj3Munn35q+qqGatGF/wMPPEAHDhwwVSMrK4tOnDhBfX19QiEFgBITEyk7O5va29uFelAUhZ599lmqq6ujhoYGCgaDQr1cvHiRDh8+TBUVFfThhx8KnU04nU4iInrllVdoeHiYHA6H0MYgSZK2A7HZbLwDYP8KRVG0uWW3202Fpd/vp8cff1woMNW53traSl1dXfT7778L9aAoCvX09NCXX35JpaWl2vZoFAA6cOAArVy5Umj5mxZcwD93VF5ejt7eXj1vvS0igsfjMVWjrq4Os0MkLhQKgYjw1VdfIRwOC9eprq6Gw+EAAMzMzAjVaGtrg8vlMjUuRIS4uDiUlpYCgOF1UnsfGRnBt99+K9wHm1+SJCESiUBRFMPLRiIRRCIR7e+nTp3C0aNH57ymtw4AjI+Po7i4GE6nE42NjYbrALPrk5WVBSJCRUWF4eUVRUEoFMILL7wAIsJDDz2kvW5UOBwGESE5OdnwstG2bdsGIkIwGLzTW3Xl8aIK/5GRETidToRCoTuPxC0oioLLly+DiDA6OgpZloXqyLKMiooKU+GvThQiQk1NjdDEUZfp7OwU7kUdg8uXL+O9997DM888A0VRhHZGg4ODyM3NndOLJEmGaoTDYSQmJuKee+5BX1+f9pqRfhRFgSRJCIVC8Pv92vpIkiQ0zkuFum6yLCMSiRge++g6N4b29PS0qd6++OILNDU1weVyGZ6r6nr19/ejqKgINHtzCM6cOSPUy6FDh0BEcLlcOHv2rOHlJUmC1+tFTk4OiouL4fP5hOfVxMQEiAh5eXnC3xcwG/42m03PW5de+Dc3N+tdudsaGxsDEaGlpUX4KBkAurq6kJiYiNHRUVNf2tNPP236DEKWZfT09ICI8NZbbwGA8JGVekQ0PDwsVAcAent7QUSIj4/H1atXARg7Krp69SrOnDkDm82m7Ry//vpr3ctLkoSamhosX74caWlpICK43W54vV5cunRJe5+edTtx4gR6e3uxefPmeTnrvJmpqSn8+eef/3j9dv2pO+3oOXzu3DmcP38eo6OjePvtt/HRRx+hs7MTIyMjuvpQa4VCIRw5cgT79u3DihUrtLCNj4/HH3/8ccc66vbg9/vhdru1A4LMzExs375dd+Cq6//DDz+gvb0ddrsdRARJkhAOh4XmZn5+PogI9fX1AIwfrUciEUxNTYGIkJCQgOvXrxvuQf3ciYkJlJWVoba2FoFAQGh9VDab7e4N/8rKynkJ/7Nnz4KIUFJSInypJRKJoLu7G6tWrTK9E6mrq0NSUhI6OzuFawDA8PCwtpGqPYrYu3cv0tPTkZqaKtxLOBzWjtBWr16NgYEB3ctGn411dHRg3bp12np1d3djampK187W5/NhZGQEHo8HVVVVWo2YmBh0dHRgYmJCVz8JCQnahmWz2ZCenq792bBhA44dO6b7DHJ4eBgtLS3weDwYHR1FS0sLSkpKUFhYiNraWnR2dqKhoQH9/f0IBAK6agLQahUUFCA/Px+5ubnIyMgAESElJUW7HHCzOaGGX/RZZGFhoTZeashlZGTA4/EYOlvesGGDVuPJJ5+cs+PVQ/2eCwsLkZCQgPLycuzevRuA8UuKADAwMAAiQmxsrPCZTDgcRnd395xLRiJH/bIso6OjAytXrjS0fdzKXR3+69evNx3+0ZdaCgsLTdUCgKqqKhQUFMypLdJPQ0MDHnvsMRw7dkyoD3WDbGpqAhFhcHBQqI5qcnISDocD7e3tAMR3JKdPn0ZycjKICL/++qtwrcHBQezcuVMLEq/XCwC3DSL1GrU6xn///TfefPNN1NbWanWKioqwZcsWQ718//33aG5uRmVlpTYn1TMUm82G7u5u7Wwn2tTUFHJyclBfX4+tW7fi4Ycf1sL6qaee0kI7Pj4eSUlJc3bkN65nMBiEw+HQ3pOYmIhVq1ahqqoKra2t+Oyzz7B//354vV50dXXhwoULt5yfgUAAhw8fRk1NDYgISUlJWLduneE5FN2j2+0GEWHt2rXw+Xza60YCe2ZmBoFAAGvXrtUOsgDjlxHVZVpaWkBEePnllwGIzUO1fyLCzp07DS+vUr+LmJgYOJ3OOa+JstlsqKys1PPWpRf+BvZst6ROUCJCbW2tqVoA0Nraivz8fNN1BgcHQUQ4cuSIcNBKkgSfz4fU1FS43W4AYhNcHaOysjKkpaXh6NGjQrXUybxnzx7ExsaivLwchw4dMtxP9EZRUVGBhIQE5OTk6Plh65YGBgbm7AT27Nlj6EhbNTMzg7GxMWzduhWpqanaHL3xwCL68sXk5CSA2d9YxsbG5lwC+fnnnzE9PY33338fdXV1aGxs/MdnXrt2DXl5ecjOzobb7UZFRQW6urrQ3d1tuH8AyMvLQ0JCAogITqcTdXV1+OabbwD8f+zvFLjq+3788UdtLjudTnzwwQe6lr+VJ554QrtEMzk5KVRHDez4+HgQEU6fPi3US3QtswdY6noQEZYtWwbg9gcyenD430H0oO/atcvUNTYAGBoawurVqwGI77mjz0b27dtnuqf+/n7k5uZiaGjI1I5kZmYGdrsdJSUlwr2o4+31ehEbG4s1a9YAMD5W0Rt9W1sbiAjV1dXw+/2611FRFMiyrH12IBBAXl4ekpOTkZ6ejpdeeslQTzcKBoMoKyvTPU+jx0DtLdqNP7hG27JlC4aGhm56/V0dK7Xe7ULll19+ARFh/fr1ePXVV+HxeLRLmCJB++CDD8LhcCAzMxM9PT1CdSRJwrVr1+DxeEBEyMrKwvj4uOFegLljrJ7p3fi6UeodeleuXBGuEb0TycnJATA/4d/c3KznrUsz/Ldt26Zn5W7rp59+QkpKClpbW00P+P79+5GRkQFA/NKIioiwYsUKUzWia6WlpZmu8/zzz4OI8PHHH5uu9fnnn8PlcqG8vBznz58XqqFutD6fTztqF92Qo++Geeedd7Bx40YAYteS/wvRl7REybKMc+fOafVE5/D169dRWVmJxsZG7TKPmd7uvfdeuFwu+P1+4Rrq55eWloKI0Nvba+q7VRQFfX19yMrKQkdHh3AdVTAYBBHh4MGDpmuFQiE4nU69P+wvrfBXB2o+wn96ehp2ux0NDQ2mawUCARCRqeBXA0g99Y5+zSh1cmdmZpq+g0i9NTI3NxfFxcVaSIjWmp6e1n4Efv31102FNgDU19ebHi+VoijCd20sVWbH7EbvvvuudknFzPYgyzIcDgfa2tpM1wKAuLg45Obmmqqh9pCdnY24uDhTtYDZ+dbU1ISqqirTB6DA7N115eXlet++tML/vvvuw8aNG+flHu3nnnsO999/P0ZHR01PrK6uLqSkpJjuCYB2d8b09LTpvrZv3w4iwqlTp0z35ff7QUTYsWOHqTpq2HR0dODkyZOm7pBii0t0gM3HNvrdd9/NSy1ZluflBghVYmLivPxW6Pf7UVBQoPsW3DspLS3FJ598ovftuvJ40Tze4cKFC1RdXU02m64H0t2Soii0fPlyevTRR8nlcpl+DkZ2djb99ddfpmoQzf5T8c2bNxMRUXJysqm+IpEI7dixg9asWWP6AXaKolBmZiY1NjbSG2+8YeoZKOpTUN1uNz3yyCPC/4ydLT7RjxUxu42Gw2HKz8+fl1oxMTHk9XqpqKhoXp4DJMsypaSkmK6TlpY2r/O/urqafvvtt3mrR0RkAxb0+SwL8mE+n48yMzMpLi7OdC0AdOnSJUpPTzc9USORCE1NTdGyZctM96Waj8dXM8Zmt3Wz2/gioWsl7srwX+z4Of+M3d0AkCRJ/9XZL4c/Y4xZkK7wX+j/yeuuOKdijLGlji8WM8aYBXH4M8aYBXH4M8aYBXH4M8aYBXH4M8aYBXH4M8aYBXH4M8aYBXH4M8aYBXH4M8aYBXH4M8aYBXH4M8aYBXH4M8aYBXH4M8aYBXH4M8aYBXH4M8aYBXH4M8aYBXH4M8aYBXH4M8aYBXH4M8aYBXH4M8aYBXH4M8aYBXH4M8aYBXH4M8aYBf0PwIACOHRGfV4AAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "rotating_image_classification(digit_one, sess1, prob1, X1, keep_prob1)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As shown above, a neural network trained to generate softmax probabilities fails significantly when it encounters a sample that is different from the training examples. The softmax forces neural network to pick one class, even though the object belongs to an unknown category. This is demonstrated when we rotate the digit one between 60 and 130 degrees. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Classification with Evidential Deep Learning" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In the following sections, we train the same neural network using the loss functions introduced in the paper." ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "## Using the Expected Mean Square Error (Eq. 5)" ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "As described in the paper, a neural network can be trained to learn parameters of a Dirichlet distribution, instead of softmax probabilities. Dirichlet distributions with parameters $\\alpha \\geq 1$ behaves like a generative model for softmax probabilities (categorical distributions). It associates a likelihood value with each categorical distribution." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Some functions to convert logits to evidence" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "# This function to generate evidence is used for the first example\n", "def relu_evidence(logits):\n", " return tf.nn.relu(logits)\n", "\n", "# This one usually works better and used for the second and third examples\n", "# For general settings and different datasets, you may try this one first\n", "def exp_evidence(logits): \n", " return tf.exp(tf.clip_by_value(logits/10,-10,10))\n", "\n", "# This one is another alternative and \n", "# usually behaves better than the relu_evidence \n", "def softplus_evidence(logits):\n", " return tf.nn.softplus(logits)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Define the loss function" ] }, { "cell_type": "code", "execution_count": 201, "metadata": { "collapsed": true }, "outputs": [], "source": [ "def KL(alpha):\n", " beta=tf.constant(np.ones((1,K)),dtype=tf.float32)\n", " S_alpha = tf.reduce_sum(alpha,axis=1,keep_dims=True)\n", " S_beta = tf.reduce_sum(beta,axis=1,keep_dims=True)\n", " lnB = tf.lgamma(S_alpha) - tf.reduce_sum(tf.lgamma(alpha),axis=1,keep_dims=True)\n", " lnB_uni = tf.reduce_sum(tf.lgamma(beta),axis=1,keep_dims=True) - tf.lgamma(S_beta)\n", " \n", " dg0 = tf.digamma(S_alpha)\n", " dg1 = tf.digamma(alpha)\n", " \n", " kl = tf.reduce_sum((alpha - beta)*(dg1-dg0),axis=1,keep_dims=True) + lnB + lnB_uni\n", " return kl\n", "\n", "def mse_loss(p, alpha, global_step, annealing_step): \n", " S = tf.reduce_sum(alpha, axis=1, keep_dims=True) \n", " E = alpha - 1\n", " m = alpha / S\n", " \n", " A = tf.reduce_sum((p-m)**2, axis=1, keep_dims=True) \n", " B = tf.reduce_sum(alpha*(S-alpha)/(S*S*(S+1)), axis=1, keep_dims=True) \n", " \n", " annealing_coef = tf.minimum(1.0,tf.cast(global_step/annealing_step,tf.float32))\n", " \n", " alp = E*(1-p) + 1 \n", " C = annealing_coef * KL(alp)\n", " return (A + B) + C" ] }, { "cell_type": "code", "execution_count": 199, "metadata": { "collapsed": true }, "outputs": [], "source": [ "# train LeNet network with expected mean square error loss\n", "def LeNet_EDL(logits2evidence=relu_evidence,loss_function=mse_loss, lmb=0.005):\n", " g = tf.Graph()\n", " with g.as_default():\n", " X = tf.placeholder(shape=[None,28*28], dtype=tf.float32)\n", " Y = tf.placeholder(shape=[None,10], dtype=tf.float32)\n", " keep_prob = tf.placeholder(dtype=tf.float32)\n", " global_step = tf.Variable(initial_value=0, name='global_step', trainable=False)\n", " annealing_step = tf.placeholder(dtype=tf.int32) \n", " \n", " # first hidden layer - conv\n", " W1 = var('W1', [5,5,1,20])\n", " b1 = var('b1', [20])\n", " out1 = max_pool(tf.nn.relu(conv(tf.reshape(X, [-1, 28,28, 1]), \n", " W1, strides=[1, 1, 1, 1]) + b1))\n", " # second hidden layer - conv\n", " W2 = var('W2', [5,5,20,50])\n", " b2 = var('b2', [50])\n", " out2 = max_pool(tf.nn.relu(conv(out1, W2, strides=[1, 1, 1, 1]) + b2))\n", " # flatten the output\n", " Xflat = tf.contrib.layers.flatten(out2)\n", " # third hidden layer - fully connected\n", " W3 = var('W3', [Xflat.get_shape()[1].value, 500])\n", " b3 = var('b3', [500]) \n", " out3 = tf.nn.relu(tf.matmul(Xflat, W3) + b3)\n", " out3 = tf.nn.dropout(out3, keep_prob=keep_prob)\n", " #output layer\n", " W4 = var('W4', [500,10])\n", " b4 = var('b4',[10])\n", " logits = tf.matmul(out3, W4) + b4\n", " \n", " evidence = logits2evidence(logits)\n", " alpha = evidence + 1\n", " \n", " u = K / tf.reduce_sum(alpha, axis=1, keep_dims=True) #uncertainty\n", " \n", " prob = alpha/tf.reduce_sum(alpha, 1, keepdims=True) \n", " \n", " loss = tf.reduce_mean(loss_function(Y, alpha, global_step, annealing_step))\n", " l2_loss = (tf.nn.l2_loss(W3)+tf.nn.l2_loss(W4)) * lmb\n", " \n", " step = tf.train.AdamOptimizer().minimize(loss + l2_loss, global_step=global_step)\n", " \n", " # Calculate accuracy\n", " pred = tf.argmax(logits, 1)\n", " truth = tf.argmax(Y, 1)\n", " match = tf.reshape(tf.cast(tf.equal(pred, truth), tf.float32),(-1,1))\n", " acc = tf.reduce_mean(match)\n", " \n", " total_evidence = tf.reduce_sum(evidence,1, keepdims=True) \n", " mean_ev = tf.reduce_mean(total_evidence)\n", " mean_ev_succ = tf.reduce_sum(tf.reduce_sum(evidence,1, keepdims=True)*match) / tf.reduce_sum(match+1e-20)\n", " mean_ev_fail = tf.reduce_sum(tf.reduce_sum(evidence,1, keepdims=True)*(1-match)) / (tf.reduce_sum(tf.abs(1-match))+1e-20) \n", " \n", " return g, step, X, Y, annealing_step, keep_prob, prob, acc, loss, u, evidence, mean_ev, mean_ev_succ, mean_ev_fail" ] }, { "cell_type": "code", "execution_count": 156, "metadata": { "collapsed": true, "scrolled": true }, "outputs": [], "source": [ "g2, step2, X2, Y2, annealing_step, keep_prob2, prob2, acc2, loss2, u, evidence, \\\n", " mean_ev, mean_ev_succ, mean_ev_fail= LeNet_EDL()" ] }, { "cell_type": "code", "execution_count": 172, "metadata": { "collapsed": true, "scrolled": false }, "outputs": [], "source": [ "sess2 = tf.Session(graph=g2)\n", "with g2.as_default():\n", " sess2.run(tf.global_variables_initializer())" ] }, { "cell_type": "code", "execution_count": 173, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "epoch 1 - 100%) training: 0.9470 (29.9496 - 6.0470) \t testing: 0.9525 (30.3331 - 6.2809)\n", "epoch 2 - 100%) training: 0.9674 (33.9131 - 6.2303) \t testing: 0.9695 (34.4653 - 7.0255)\n", "epoch 3 - 100%) training: 0.9756 (33.1459 - 4.3576) \t testing: 0.9762 (33.7443 - 4.1752)\n", "epoch 4 - 100%) training: 0.9745 (33.9431 - 3.5602) \t testing: 0.9749 (34.4402 - 3.7496)\n", "epoch 5 - 100%) training: 0.9807 (36.8166 - 4.0170) \t testing: 0.9789 (37.4320 - 4.1132)\n", "epoch 6 - 100%) training: 0.9791 (36.6413 - 3.0833) \t testing: 0.9803 (37.2622 - 3.1119)\n", "epoch 7 - 100%) training: 0.9782 (39.2723 - 3.2930) \t testing: 0.9778 (40.0590 - 3.1863)\n", "epoch 8 - 100%) training: 0.9808 (37.4109 - 1.9068) \t testing: 0.9800 (38.1145 - 2.2643)\n", "epoch 9 - 100%) training: 0.9815 (39.3951 - 2.7377) \t testing: 0.9805 (40.1664 - 2.9485)\n", "epoch 10 - 100%) training: 0.9831 (40.2205 - 2.0345) \t testing: 0.9830 (40.8734 - 2.4930)\n", "epoch 11 - 100%) training: 0.9839 (40.3787 - 1.5182) \t testing: 0.9840 (41.1746 - 1.7158)\n", "epoch 12 - 100%) training: 0.9833 (39.9269 - 1.6233) \t testing: 0.9836 (40.7200 - 2.0022)\n", "epoch 13 - 100%) training: 0.9844 (42.0865 - 1.8232) \t testing: 0.9835 (42.9469 - 2.2165)\n", "epoch 14 - 100%) training: 0.9816 (40.6318 - 1.4549) \t testing: 0.9811 (41.4050 - 2.1860)\n", "epoch 15 - 100%) training: 0.9851 (44.5367 - 2.1988) \t testing: 0.9842 (45.3868 - 3.0271)\n", "epoch 16 - 100%) training: 0.9837 (44.3451 - 1.9175) \t testing: 0.9829 (45.0711 - 3.2015)\n", "epoch 17 - 100%) training: 0.9839 (47.7866 - 2.7627) \t testing: 0.9824 (48.6517 - 3.8829)\n", "epoch 18 - 100%) training: 0.9856 (44.8029 - 1.7993) \t testing: 0.9855 (45.7763 - 3.0277)\n", "epoch 19 - 100%) training: 0.9841 (44.6586 - 1.8779) \t testing: 0.9834 (45.5720 - 3.4869)\n", "epoch 20 - 100%) training: 0.9875 (45.7646 - 1.8881) \t testing: 0.9877 (46.7228 - 3.3131)\n", "epoch 21 - 100%) training: 0.9866 (46.3577 - 1.9462) \t testing: 0.9861 (47.1481 - 3.0685)\n", "epoch 22 - 100%) training: 0.9861 (46.5912 - 1.9597) \t testing: 0.9863 (47.3737 - 2.7471)\n", "epoch 23 - 100%) training: 0.9869 (48.9247 - 2.2133) \t testing: 0.9867 (49.9383 - 2.7514)\n", "epoch 24 - 100%) training: 0.9870 (46.9884 - 2.0665) \t testing: 0.9855 (48.0439 - 2.1623)\n", "epoch 25 - 100%) training: 0.9873 (50.8303 - 2.4647) \t testing: 0.9854 (51.8676 - 3.1680)\n", "epoch 26 - 100%) training: 0.9880 (49.6770 - 2.3419) \t testing: 0.9879 (50.5636 - 3.7460)\n", "epoch 27 - 100%) training: 0.9871 (49.9567 - 2.2482) \t testing: 0.9862 (51.1154 - 3.4443)\n", "epoch 28 - 100%) training: 0.9877 (50.9868 - 2.4529) \t testing: 0.9869 (52.1459 - 3.8382)\n", "epoch 29 - 100%) training: 0.9883 (46.9654 - 1.3623) \t testing: 0.9873 (47.9654 - 2.1033)\n", "epoch 30 - 100%) training: 0.9882 (51.7587 - 2.4527) \t testing: 0.9871 (52.7797 - 4.4101)\n", "epoch 31 - 100%) training: 0.9883 (52.9645 - 3.1079) \t testing: 0.9873 (54.1205 - 3.6993)\n", "epoch 32 - 100%) training: 0.9881 (51.1556 - 2.3109) \t testing: 0.9875 (52.0979 - 4.0251)\n", "epoch 33 - 100%) training: 0.9880 (48.3095 - 1.5548) \t testing: 0.9874 (49.4096 - 2.0053)\n", "epoch 34 - 100%) training: 0.9885 (51.3298 - 2.3731) \t testing: 0.9864 (52.2786 - 3.8313)\n", "epoch 35 - 100%) training: 0.9892 (53.0268 - 2.6064) \t testing: 0.9875 (54.1511 - 3.8138)\n", "epoch 36 - 100%) training: 0.9902 (50.2909 - 1.8811) \t testing: 0.9884 (51.5063 - 2.5316)\n", "epoch 37 - 100%) training: 0.9885 (51.8024 - 2.0432) \t testing: 0.9879 (52.7857 - 2.8535)\n", "epoch 38 - 100%) training: 0.9888 (51.6903 - 1.5005) \t testing: 0.9881 (52.7587 - 2.3378)\n", "epoch 39 - 100%) training: 0.9896 (54.4732 - 1.9507) \t testing: 0.9876 (55.7110 - 2.9229)\n", "epoch 40 - 100%) training: 0.9885 (49.8664 - 1.7354) \t testing: 0.9866 (50.7666 - 2.3627)\n", "epoch 41 - 100%) training: 0.9893 (54.7296 - 2.3521) \t testing: 0.9878 (55.9593 - 3.4784)\n", "epoch 42 - 100%) training: 0.9888 (55.0189 - 2.9684) \t testing: 0.9884 (55.9647 - 3.5898)\n", "epoch 43 - 100%) training: 0.9907 (54.7551 - 2.2131) \t testing: 0.9887 (55.9601 - 4.9516)\n", "epoch 44 - 100%) training: 0.9889 (55.8486 - 2.6489) \t testing: 0.9880 (57.0468 - 4.3089)\n", "epoch 45 - 100%) training: 0.9895 (56.3373 - 2.6319) \t testing: 0.9888 (57.3626 - 4.7427)\n", "epoch 46 - 100%) training: 0.9886 (52.5418 - 1.4541) \t testing: 0.9882 (53.6143 - 2.9547)\n", "epoch 47 - 100%) training: 0.9908 (53.2042 - 1.7521) \t testing: 0.9876 (54.3501 - 2.6650)\n", "epoch 48 - 100%) training: 0.9893 (57.3922 - 2.6419) \t testing: 0.9877 (58.5497 - 4.1028)\n", "epoch 49 - 100%) training: 0.9889 (55.2722 - 2.0515) \t testing: 0.9876 (56.4727 - 2.7934)\n", "epoch 50 - 100%) training: 0.9887 (55.1349 - 2.0851) \t testing: 0.9880 (56.2424 - 2.6495)\n" ] } ], "source": [ "bsize = 1000 #batch size\n", "n_batches = mnist.train.num_examples // bsize\n", "L_train_acc1=[]\n", "L_train_ev_s=[]\n", "L_train_ev_f=[]\n", "L_test_acc1=[]\n", "L_test_ev_s=[]\n", "L_test_ev_f=[]\n", "for epoch in range(50): \n", " for i in range(n_batches):\n", " data, label = mnist.train.next_batch(bsize)\n", " feed_dict={X2:data, Y2:label, keep_prob2:.5, annealing_step:10*n_batches}\n", " sess2.run(step2,feed_dict)\n", " print('epoch %d - %d%%) '% (epoch+1, (100*(i+1))//n_batches), end='\\r' if i" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "draw_EDL_results(L_train_acc1, L_train_ev_s, L_train_ev_f, L_test_acc1, L_test_ev_s, L_test_ev_f)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The figure above indicates that the proposed approach generates much smaller amount of evidence for the misclassified samples than the correctly classified ones. The uncertainty of the misclassified samples are around 0.8, while it is around 0.1 for the correctly classified ones, both for training and testing sets. This means that the neural network is very uncertain for the misclassified samples and provides certain predictions only for the correctly classified ones. In other words, the neural network also predicts when it fails by assigning high uncertainty to its wrong predictions." ] }, { "cell_type": "code", "execution_count": 174, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEKCAYAAADjDHn2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xl8VPW5+PHPQwJhJ0LYSVgSN7QsEi3uuOBWq7fFW6Eu1wX9qdVKra143WtfVattrdZe3LDqdaHaulzFjSqKVsWgoIggSCYQCPu+BpLn98f3DAxJZuZMkjOTmXner9d5TebMme95OCTzzPd7vouoKsYYY0wsrVIdgDHGmJbPkoUxxpi4LFkYY4yJy5KFMcaYuCxZGGOMicuShTHGmLgsWRhjjInLkoUxxpi4LFkYY4yJKzfVASSqoKBABwwYkOowjDEmrcyaNWuNqnZv7PvTLlkMGDCAsrKyVIdhjDFpRUQqmvJ+a4YyxhgTlyULY4wxcVmyMMYYE5clC2OMMXFZsjDGGBNXYMlCRCaLyCoRmRvldRGRB0RkkYh8KSKHBRVL1qqqgquuguHDM7Msiyn5ZWV6TCY6VQ1kA44DDgPmRnn9DOANQICRwKd+yh0xYoSaOJYvV73yStW2bVXbtFGFzCrLYkp+WZkeUxYAyrQJn+mBjbNQ1Q9EZECMQ84GnvL+EZ+ISL6I9FbVqqBiynhVVXDnnfDEE7BrF9TU7H3t1lsTK2vzZpgxA+bMAdWWUZbFlPyy0i0mExjRANfg9pLFa6p6aAOvvQbcraofes//BdygqvVG3InI5cDlAEVFRSMqKpo0tiRzHX88fPgh1NbWf00ksbJi/V6kqiyLKfllpVtMAX6epTsRmaWqpY19f1rc4FbVR1S1VFVLu3dv9Gj1zDdlClxxBbRuXf+12trEtnA7cLt20KZNyyjLYrJ/X2Q5bdvW/z03wWlKG1a8DRhA9HsWDwPjIp4vAHrHK9PuWfgwfrxqbq5qu3ZNb8utqlK96qqWVZbFlPyyWmJM5eXuvbm5ds/CB5p4zyKVyeIH7HuDe6afMi1Z+HD22aqHHrr3j3LYsKaX2RLLspiSX1ZLi6lHD9Xzzmu+mDJYU5NFYPcsROQ5YBRQAKwEbgNae7WZSSIiwF+A04BtwMXawP2KukpLS9UmEoxj+HDo2xdeey3VkRgTrNJS6N4d3ngj1ZG0eE29ZxFkb6hxcV5X4GdBnT+rhUJw9NGpjsKY4BUWwsKFqY4iK6TFDW6TgI0bYcMGsDU/TDYoKoIlS1IdRVawZJFpwt2K+/dPbRzGJENhoRt3sXFjqiPJeJYsMk0o5B6tZmGyQWGhe1y6NLVxZAFLFpnGahYmmxQVuUdrigqcJYtMEwq5QU82eNFkA6tZJI0li0wTCrkmqESnYjAmHfXuDTk5liySwJJFpqmosCYokz1yctyYImuGCpwli0wTrlkYky0KC61mkQSWLDLJli2wdq0lC5NdLFkkhSWLTGI9oUw2KipyyaKhqflNs7FkkUlsjIXJRoWFUF0Nq1enOpKMZskik4SThdUsTDax7rNJYckik1RUQF4e9OyZ6kiMSZ5wsrAeUYGyZJFJQiFXq2hl/60mi4RHcVvNIlD2qZJJbIyFyUbdurklVi1ZBMqSRSaxMRYmG4m4pihrhgqUJYtMsW0brFplycJkp3D3WRMYSxaZIvytypqhTDaygXmBS7tkMWvWLEQEEaFXr16pDqflsDEWJpsVFsLy5bBrV6ojyVhplywirVy5MuH39OrVa0+yidzSPvFYsjDZrKgIVF3CMIHITXUATXXdddeRl5dHmzZtaNOmzZ6f6z6Gf46WYBqTeFqUigpo3dpN2WxMtokcmGdNsYFI+2Tx6KOPUl1dTXV1dapDSa1QyH27sjEWJhvZwLzApX2y2Lx5MwCqyq5du9i5c+ee5BH+OfLx6KOPTnHEAQkPyDMmG9mUH4FL+2QRJiJ7mpsa68orr+SOO+6gR48ezRhZklRUwOmnpzoKY1KjUyfIz7dkEaC0brPo2cxzID322GOUlJRwzz33sGPHjmYtO1A7dkBVld3cNtnNBubVE9mhBxjRlLLSLlmMGDECVUVVWbFiRcLvj5Zgevbsydy5cznhhBOYOHEiBx10EFOmTEFVmxpy8MLfpqwZymSzFA/Ma4k9LZuz407aJYumWrFixZ5kE7mtWLGCAw88kFdeeYV//etf5OfnM3bsWI466ig++eSTVIcdm3WbNSblA/NaWk/Lbdu2NWt5WZcs/DjxxBOZNWsWkydPJhQKceSRRzJu3DhC4Q/llsaShTEuWaxd66a+yTKqysKFC3n66af52c9+xogRI+jcuXOznsOSRRQ5OTlcfPHFLFy4kFtvvZVXXnmFgw46iBtvvJFNmzalOrx9VVRATg706ZPqSIxJnRY8Vfnw4cMZN24cd9xxB1OmTGH27NlRv/n7ac7atGkT06ZN47e//S1nnnkm3bt354ADDuDCCy/k6aefZr/99mPixInN+m+QtGiTj1BaWqplZWVJP+/SpUu56aabePrpp+nevTs7d+5sMGn07NmzUfdSmuT88+Gjj6C8PLnnNaYlef99GDUK3nkHTj45qafevHlzzG/yp59+OvPnzycUCu1zH7R///4ceOCBHHTQQXu2E088MWo548eP5+OPP2bevHl7yhk8eDAjR47kyCOPZOTIkRx88MHk5OQAhG9s76GqUq9QnyxZJKisrIzrrruOGTNmRD0m6df0mGPc6O333kvueY1pSRYvhuJimDwZLr44aaddt24dp59+OjNnzox6TPgzYfv27SxcuJAFCxYwf/585s+fv+fnrVu3xj1X165dGTly5J7t8MMPJz8/P+rxvXr12ueeSVOSRcaMs0iW0tJS3n//fVq1pJHSFRVw0kmpjsKY1Orb161tkcTus1VVVZxyyiksXLiQ/Px8NmzYUO+YyB6Y7dq1Y8iQIQwZMmSfY1SVZcuWsWDBAk6OUStas2ZNvdpCLJGtHCIyy/cbG2DJohES+c8KXHU1LFtmN7eNCa8/n6R7FhUVFZx88slUVVXx+uuvc1ITvrCJCP369aNfv35xj0uVuF+PReSHItKCvkabfSxd6mbbtDEWxiSt++yCBQs45phjWLNmDe+8806TEkW68JMEzgUWisjvReSgoAMyCaqocI9WszAmKaO4Z8+ezbHHHkt1dTXTp0/nyCOPbNbyYw0cTqW4yUJVzweGA98BfxORj0XkchHpFO+9InKaiCwQkUUiUq8fl4gUich7IvKFiHwpImc06l+RAtH+43Jycpp9MExMNsbCmL3Co7gD6mTy8ccfM2rUKPLy8vjggw8YOnRos58j1sDhVPLVvKSqm4AXgeeB3sCPgM9F5Jpo7xGRHOAh4HRgMDBORAbXOexm4O+qOhwYC/w14X9BijT0H/r2229TW1vLVVddlbweURUVblryOG2dxmSFwkLYuhUauNHcVNOmTWP06NF0796dDz/8kAMPPLDZz9GS+blncbaIvARMB1oDR6jq6cBQ4Jcx3noEsEhVF6tqNS7RnF3nGAXCnZO7AGm9zNXo0aO59dZbefLJJ5k8eXJyThoKuV4grVsn53zGtGQBrWvxyiuv8IMf/ICBAwcyY8YM+mfhPUI/NYsfA39S1e+p6r2qugpAVbcBl8Z4X18g8k5Tpbcv0u3A+SJSCUwFGqypeM1eZSJStnr1ah8hp84tt9zCySefzNVXX82cOXOCP2EoZE1QxoQFMIr7mWeeYcyYMQwbNoz3338//ZdgbiQ/yWKFqn4QuUNE7gFQ1X818fzjgL+paj/gDODphnpeqeojqlqqqqXdu3dv4imDlZOTwzPPPEPXrl0555xz2LhxY7AnrKiwnlDGhDXzIkiTJk3iggsu4Nhjj2XatGl07dq1WcpNR36SxegG9vlZZWcZUBjxvJ+3L9KlwN8BVPVjoC1Q4KPsFq1Hjx5MmTKF8vJyxo8fH9z9i927obLSahbGhPXsCbm5zdIMdc8993DllVdyxhlnMHXqVDp1itunJ6NFTRYicqWIfAUc5PVUCm/lwJc+yv4M2F9EBopIG9wN7FfrHLMEOMk738G4ZNGy25l8OuaYY7jrrrt48cUXefDBB4M5SWUl1NRYzcKYsJwc19mjCTULVeWmm25i4sSJnHvuubz00ku0a9euGYNMUw110fK+CXcBBgDPAf0jtq7R3tNAGWcA3+K63d7k7fsNcJb382DgI2AOMBs4JV6ZI0aM0HRRW1urZ511lrZu3Vo/+eST5j/B9OmqoPrOO81ftjHp6thjVY87rlFvramp0auvvloBHT9+vO7evbuZg0sdoEx9fnY3tEWdSFBEOqvqJhFpsJFOVdc1R7JKVKonEkzU+vXrOeyww6ipqeGLL76gW7duzVf4k0/CRRfBwoVQUtJ85RqTzs47D/79b1+zMNedaC+sffv2bNmypWVN7dNEIjJLVUsb+/5Y9yye9R5nAWXe46yI58aH/fbbjxdffJGVK1dywQUXUFtb23yFhwfkFRbGPMyYrFJU5OZL8/G3Fm0Vu23btmVUomgOUZOFqp7pPQ5U1UHeY3gblLwQ09+IESO4//77eeONN7j77rubr+CKCrfgUV5e85VpTLorLIRduyBFy5lmqqizzorIYbHeqKqfN384meuKK65gxowZ3HLLLRx11FGMGjWq6YXaGAtj6oscmNe7d2pjySCxpij/Q4zXFIi+nJOpR0R4+OGH+fzzzxk7diyzZ89u+uCeUAhGjmyW+IzJGJED877//dTGkkGiJgtVPSGZgWSDTp068eKLL3LEEUcwbtw43nnnHXJzG7mkSE2N+2MYO7Z5gzQm3TXzwDzjxBpncaL3+OOGtuSFmFkOPfRQJk2axPTp07ntttsaX9Dy5W5QnjVDGbOv/faD9u19DcxrqdOBt0SxekMd7z3+sIHtzIDjymgXXngh48eP53e/+x1Tp05tXCHhdSxsQJ4x+xLZO1V5HCtWrOD1118H4KOPPmox04G3RLGaoW7zHpO38nkWeeCBBygrK+OCCy7giy++oCjczuqXrWNhTHQJrJgX8v6WBg4cGGBA6c/PFOXdROQBEflcRGaJyJ9FpBlHlmWndu3a8cILL7Br1y5+8pOfUF1dnVgB4WSRaJIxJhsksGJeeXk5eXl51vQUh5+JBJ/Hzdc0BjjH+3lKkEFli5KSEp544gk+/fRT8vLyEJF9tpi9pSoq3KRpNmeNMfUVFblxFj6+hIVCIQYMGECrVr7Wgstafq5Ob1W9U1XLve23gKXgZjJmzJior0UbXQrYGAtjYiksdEurLqs70XV95eXlDLC/pbj8JIu3RWSsiLTytp8AbwUdmIkjFLKb28ZEk8CKeeXl5Xa/wodYXWc3i8gm4DLcPFHV3vY8cHlywjMNqq11fwT2bciYhvkca7Fp0ybWrVtnycKHWL2hsnulj5ZsxQrXFmvJwpiG+UwW4Z5Q1gwVn6/hwyKyH7A/bnEiALTOUqsmicI9oawZypiGdegAXbvGbYaybrP+xU0WIjIeuBa3LOpsYCTwMTY3VLPp2bNngzezo3blCw/Is29DxkTnY6xFubfmhdUs4vNzg/ta4HCgwpsvajiwIdCossyKFStQVV544QUAZs+eHXsUqdUsjInPxyjuUChEhw4dKCgoSFJQ6ctPstihqjsARCRPVecDBwYbVnYqLi4GYNGiRbEPrKiAggJX1TbGNMxnzWLAgAG20JEPfu5ZVIpIPvAy8I6IrAcqgg0rO4WTxXfffRf7QBtjYUx8hYWwfj1s2QIdOzZ4iHWb9S9uzUJVf6SqG1T1duAW4HHgP4IOLBt17tyZgoICf8nCmqCMiS1yXYsGqCqhUMiShU++xreLyGEi8nNgCFCpqglOZGT8Ki4ujp0sVF0zlNUsjIktTvfZ9evXs2nTJru57ZOfiQRvBZ4EugEFwBMicnPQgWWruMli1SrYscOShTHxxBnFbd1mE+OnZnEecLiq3uZNWz4SuCDYsLJXcXExS5YsiT4LrfWEMsafvn3d2hZRahbWbTYxfpLFciIG4wF5QPzZuUyjlJSUUFtbu+dbTz02xsIYf1q3ht69oyYLq1kkJmpvKBF5EFBgI/C1iLzjPR8NzExOeNknskfUAQccUP8Aq1kY41+MdS3Ky8vJz88nPz8/yUGlp1hdZ8u8x1nASxH7pwcWjYnffTYUcmsMd+6cvKCMSVdFRTBnToMv2dTkiYk1keCT4Z9FpA0Q/pq7QFV3BR1YturZsycdOnSIniysJ5Qx/hUWwmuvuV6EdQbehUIhDjzQxhf75ac31ChgIfAQ8FfgWxE5LuC4spaIMGjQoNg1C2uCMsafwkLYvh3Wrt1nd3iMhdUs/PNzg/sPwCmqeryqHgecCvwp2LCyW9TuszbGwpjERBmYt3r1arZt22Y3txPgJ1m0VtUF4Seq+i3QOriQTElJCYsXL6a2tnbfF9auha1bLVkY41eUgXnWbTZxfpJFmYg8JiKjvO1R9t78NgEoLi5mx44dLF++fN8XrCeUMYmJMjAvnCysZuGfn2RxJTAP+Lm3zfP2mYBE7RFlYyyMSUyPHtCmTb2aha2Ql7iYs86KSA4wWVXPA/6YnJBMZLI4/vjj974QrlnYL7gx/rRqBf36NdgMVVBQQMcos9Ga+mImC1WtEZH+ItLGJg9MnqKiInJzc+vXLEIhN74iwwcR7dq1i8rKSnbs2JHqULJC27Zt6devH61bZ+ityAYG5tlss4nzs57FYuAjEXkV2BreqapxaxoichrwZyAHeExV727gmJ8At+NGh89R1Z/6Cz1z5ebm0r9//4abobKgVlFZWUmnTp1sUZokUFXWrl1LZWVl5n54FhbCBx/ss6u8vJxhw4alKKD05OeexXfAa96xnSK2mLwmrIeA04HBwDgRGVznmP2BG4GjVfUQYEJC0WewBrvPZskYix07dtCtWzdLFEkgInTr1i2za3FFRbBsGdTUAFBbW0tFRUXmJseAxLtn0R14HVikqomuu32E977FXlnPA2fjbpCHXQY8pKrrAVR1VYLnyFglJSU8++yze3eoumQxalSqQkoqSxTJk/HXurDQJYqqKujXj6qqKqqrq+3mdoKi1ixEZDzwNfAgMF9Ezkqw7L5A5F2lSm9fpAOAA0TkIxH5xGu2MriaxYYNG1i3bp3bsWEDbN6cFc1QLcEll1xCjx49OPTQQ1MdimmqOmMtrNts48RqhpoAHKKqRwJH4ZqLmlsusD8wChgHPOqt970PEblcRMpEpGz16tUBhNHy1Os+a2MsGtSrVy9EpN7Wq1evJpV70UUX8eabbzZTlCal6ozitqnJGydWsqhW1dUAXlNSXoJlLwMKI573o/46GJXAq6q6S1XLgW9xyWMfqvqIqpaqamn37t0TDCM91UsWNsaiQStXrkxov1/HHXccXbt2bVIZpoWoMzAvXLPob1+8EhLrnkU/EXkg2nNV/Xmcsj8D9heRgbgkMRao29PpZVyN4gkRKcA1Sy32G3wmGzRoENBAzSLLksWECROYPXt2o947Ksr9nWHDhnH//fc3ISqTVrp0gY4d96lZ9O7dm7Zt28Z5o4kUK1n8qs7zWYkUrKq7ReRq4C1c19nJqvq1iPwGKFPVV73XThGReUAN8CtVXRu91OzRvn17evfuvW+y6NAB7NuuMYkRcU1REfcs7OZ24nytZ9FYqjoVmFpn360RPytwnbeZOvbpPhseY5HpPVfqiFcDiNWTZ/r06c0cjUlbEQPzysvLOeqoo1IcUPrxM87CpEhJSQmLFi1yT0KhrGuCMqbZFBbC0qXs3r2bpUuXWs2iESxZtGDFxcUsX76c7du3Z82AvET17Nkzof1+jRs3jiOPPJIFCxbQr18/Hn/88SaVZ1KsqAhWrWLZd99RU1NjPaEawc90HyZFwj2iQnPmcPCGDVazaMCKFSsCKfe5554LpFyTIl6PqBVlbnUFSxaJi5ssvFHclwEDIo9X1UuCC8vA3mSxcuZMDgarWRjTWF6yWDdnDmBTkzeGn5rFK8AMYBqux5JJknCy2Oj9glvNwphG8gbmbf/2W1q1akVhYWGcN5i6/CSL9qp6Q+CRmHq6du1Kly5dqF640O2wZGFM4/TrB0BtKETfvn1p06ZNigNKP35ucL8mImcEHompR0QoLi6m1ZIl0K4dZMnodWOaXbt2UFBA65Ur7X5FI/lJFtfiEsYOEdnsbZuCDsw4JSUldFyzxt2vyLIxFsY0q6IiOm3YYMmikeImC1XtpKqtVLWt93MnVe2cjOCMu29RsHUrtXZz25gmqenbl+47dtjN7UbyNc5CRM4Skfu87cyggzJ7FRcX0x/YYtN8ZLSXX36ZefPmxT1u0qRJPPXUUzGPmT17NlOnTo15TDbanJ9PEdZttrHiJgsRuRvXFDXP264VkbuCDsw4B/TpQwGwsn37VIeSHqqq4KqrYPjwVEfi2+7du30niyuuuIILL7ww5jGWLBq2Ki+PLkCx3ftrFD81izOA0ao6WVUnA6cBPwg2LBNW0ro1ABUpjqPFCyeJQYPg8cehkTPVhoVCoX0WPrrvvvu4/fbbGTVqFDfccANHHHEEBxxwADNmzACgpqaG66+/nkMPPZQhQ4bw4IMPAjBr1iyOP/54RowYwamnnkpVVRXgZsSdMGECpaWl3HPPPbz66qv86le/YtiwYXz33Xc8+uijHH744QwdOpQxY8awbds2AG6//Xbuu+++PWXUjaW6uppbb72VKVOmMGzYMKZMmcL+++9PeB2Y2tpaSkpKyJZ1YSKFV2Irtp5QjeJ3BHc+4C3ZRpeAYjEN6Ll9OwDzt2/n5BTHkhITJsT+4N+5002yuHKlW3pWde9r0ZagHTYMmjBF+e7du5k5cyZTp07ljjvuYNq0aTzyyCOEQiFmz55Nbm4u69atY9euXVxzzTW88sordO/enSlTpnDTTTcxefJkAKqrqynzRhQvXLiQM888k3POOQeA/Px8LrvsMgBuvvlmHn/8ca655hpfsfzmN7+hrKyMv/zlLwDMnz+fZ555hgkTJjBt2jSGDh1KtqwLE2nhzp2cBPTI5PXGA+QnWdwFfCEi7wECHAdMDDQqs0crb1rlL9avT3EkLdQ338DGjUk95Y9//GMARowYsWfVtWnTpnHFFVeQm+v+pLp27crcuXOZO3cuo0ePBlzto3fv3nvKOffcc6OeY+7cudx8881s2LCBLVu2cOqpp/qOpa5LLrmEs88+mwkTJjB58mQuvvjihP69mWLuhg0A5CxfnuJI0lPcZKGqz4nIdOBwb9cNqhrMhDymvlCIXa1aMauyMtWRpEa8GsCKFXDnnfDEE1BTA9XVe19rwhTlubm51NbW7nm+I+LbaF6eWzQyJyeH3bt3Ry1DVTnkkEP4+OOPG3y9Q4cOUd970UUX8fLLLzN06FD+9re/RZ1u3U8shYWF9OzZk3fffZeZM2fyzDPPRD1vJpu9ahU1QM7SpXGPNfVFvWchIgd5j4cBvXFLoFYCfbx9JhlCIdZ37syixYvRyCYW4/TqBQ89BIsXw/jxbvBVM7RJ9+zZk1WrVrF27Vp27tzJa6+9FvP40aNH8/DDD+/5wF63bh0HHnggq1ev3pMsdu3axddff93g+zt16sTmzZv3PN+8eTO9e/dm165dCX+41y0LYPz48Zx//vn853/+Jzk5OQmVlykWhUJs7NBhz7oWJjGxbnCHFyT6QwPbfQHHZcIqKtjRsydbt25l1apVqY6m5aqbNIYNa1JxrVu35tZbb+WII45g9OjRHHTQQTGPHz9+PEVFRQwZMoShQ4fy7LPP0qZNG1588UVuuOEGhg4dyrBhw/j3v//d4PvHjh3Lvffey/Dhw/nuu++48847+f73v8/RRx8d99x1nXDCCcybN2/PDW6As846iy1btmRtE9T27dtZuXIl27p23bNinkmMxPu2KiJtVXVHvH3JUlpaquGbglmhZ0+WDB9O/7fe4qOPPsqKFb6++eYbDj744FSHkVHKysr4xS9+saf3Vl2Zfs2/+eYbBg8eTPnIkQxYswbC861lERGZpaqljX2/n66zDX0VavjrkWle27bBqlW08/6I9yyxakwC7r77bsaMGcNdd2Xv8Kjy8nIAWg8c6GoW1qSbsKg3uEWkF9AXaCciw3E9oQA6AzZCLBm8ttX8oUMRkb1LrBqTgIkTJzJxYnZ3YAz3FOt48MGuu/Xq1dCjR2qDSjOxekOdClwE9AP+GLF/M/DfAcZkwrxf8NYlJRQWFlrNwphGKi8vJy8vj06HHOJ2LF1qySJBUZOFqj4JPCkiY1T1H0mMyYRVeOO2BwygpKQkq5KFqiI2y25SZEMvu1AoxIABA2gVnpBzyRIYMSK1QaUZP+Ms/iEiPwAOAdpG7P9NkIEZXM0iNxd696a4uJiXX3451RElRdu2bVm7di3dunWzhBEwVWXt2rW0bds2/sFprLy83M02G14hz3pEJczPGtyTcPcoTgAeA84BZgYclwGXLIqKICeH4uJiVq9ezebNm+nUqVOqIwtUv379qKyszMr5i1Khbdu29PNWkstU5eXlHH744W4Bsbw8SxaN4Ge6j6NUdYiIfKmqd4jIH4A3gg7M4JqhvLn3w+txf/fddwxr4hiClq5169Y2jbRpNps2bWLdunWuZiHiahc2MC9hfrrObvcet4lIH2AXbkS3CVoo5FbIY99kYYzxL9wTas8XkMJCq1k0gt81uPOBe4HPgRDwXJBBGWDHDjftdp2ahXWfNSYx4WSxZ4W8oiJLFo3g5wb3nd6P/xCR14C2qprcaT6zUfiX2fsF79y5M927d7eahTEJCg/I26dmsXw57N7tOpAYX/yslPczr2aBqu4EWonIVYFHlu3C001HrL1dXFxsycKYBIVCITp06EBBQYHbUVgItbUuYRjf/DRDXaaqG8JPVHU9cFlwIRlgb7KIWFzekoUxiQt3m93TDbuoyD1aU1RC/CSLHIno7C4iOYCtSxi0igrIyYG+fffsKi4uZunSpVRHrtlgjImpvLx839514bEW1iMqIX6SxZvAFBE5SUROwt3cfjPYsAyhEPTrt0+banFxMbW1tVFXRDPG7EsDu/fMAAAa5ElEQVRV94ze3sMG5jWKn2RxA/AecKW3/Qv4dZBBGfYZYxFmPaKMScz69evZtGnTvjWLzp2hSxdLFgny0xuqFvgfbzPJEgrBiSfus8vGWhiTmHpjLMJsYF7CYk1R/ndV/YmIfAXUm2lMVYcEGlk2q66GZcvq1Sx69uxJhw4dLFkY41O42+yAOn9LNjAvcbFqFhO8xzMbW7iInAb8GcgBHlPVu6McNwZ4EThcVbNoGbwoKivd4ix1fsFFxHpEGZOAmDWLzz5LfkBpLNY9i/AK9b9V1Yq6W7yCvV5TDwGnA4OBcSIyuIHjOgHXAp8mHn6GamCMRZglC2P8Ky8vp0uXLuTn5+/7QlERrFnjVqM0vsRKFm1E5KfAUSLy47qbj7KPABap6mJVrQaeB85u4Lg7gXuAlKzp3SI1MMYirLi4mMWLF1NbW5vUkIxJR/W6zYaFe0RVViY3oDQWK1lcARwL5AM/rLP5aZrqC0Q2ClZ6+/YQkcOAQlV9PYGYM19FBbRq5brO1lFcXMzOnTtZbqNPjYmrXrfZMOs+m7BYK+V9CHwoImWq+nhzn1hEWuGWa73Ix7GXA5cDFIVHX2ayUAj69IE29cc+RnafzfQ1CIxpivAYi1NPPbX+i+HPEesR5VvUmoWIhPttrm9kM9QyoDDieT9vX1gn4FBguoiEgJHAqyJSWrcgVX1EVUtVtbR79+4+Tp3mQqEGm6DAus8a49fq1avZtm1bw81Q4S9aVrPwLVZvqOOBd3HNTnUp8M84ZX8G7C8iA3FJYizw0z0FuJlrC8LPRWQ6cL31hsI1Qx1zTIMvFRUVkZuba8nCmDiidpsFt1pejx6WLBIQqxnqNu/x4sYUrKq7ReRq4C1c19nJqvq1iPwGKFPVVxtTbsbbvdvddItSs8jNzWXAgAGWLIyJo97U5HXZuhYJ8bMG97XAE8Bm4FHgMGCiqr4d772qOhWYWmffrVGOHeUj3sy3bBnU1ERNFmDdZ43xo96iR3UVFsL8+UmLJ935mRvqElXdBJwCdAMuABocXGeaQYwxFmHFxcUsWrQI1XoD640xnvLycgoKCujYsWPDB4RHcdvfkS9+kkV4evIzgKdU9euIfaa5xRhjEVZcXMzGjRtZt25dUkIyJh1F7TYbVlQEW7bARlv40w8/yWKWiLyNSxZveSOubURYUCq8wfExughbjyhj4os6IC/M1rVIiJ9kcSkwETdv0zagNdCom97Gh1AIevd2vTWisGRhTGy1tbVUVFT4SxZ2k9sXP8niSGCBqm4QkfOBmwGrtwUlxhiLsEGDBgGWLIyJpqqqiurq6vjNUGDJwic/yeJ/gG0iMhT4JfAd8FSgUWWzBhY9qqt9+/b06dPHkoUxUcTtNgvQq5dbidKaoXzxkyx2q+t2czbwF1V9CDf62jS3mhr3ixujJ1SYdZ81Jrq43WbBrXHfp4/VLHzykyw2i8iNwPnA696cTq2DDStLLV/uBuXFqVmAJQtjYok5ejuSDczzzU+yOBfYCVyqqitwczzdG2hU2SrcE8pnzWL58uVss/n4jaknFArRu3dv2rZtG/tAW17Vt7jJQlVXqOofVXWG93yJqto9iyD4GGMRFu4RtXjx4uDiMSZNlZeXx69VgEsWlZVg68PEFTdZiMhIEflMRLaISLWI1IiI9YYKgo/R22HWfdaY6OKOsQgrKoJdu2DVquCDSnN+mqH+AowDFgLtgPHAX4MMKmtVVLiZMDdsgKuuguHDox5aUlICWLIwpq7du3ezdOlS/zULsKYoH+JOJAigqotEJEdVa4AnROQL4MZgQ8tCCxa4x0GDXLW4ujrqoV27diU/P9+ShTF1VFZWUlNT469mETkw74gjgg0szflJFttEpA0wW0R+D1Thr0Zi/Kqqgttugxkz3HKqPttPrUeUMfX56jYbZqO4ffPzoX8Bbj2Kq4GtuNXvxgQZVNYZOxYefdT9nMCNtvDss8aYvXwNyAvr1g3atbNmKB/89IaqUNXtqrpJVe9Q1etU1T6hmtO117rHnJwG192Opri4mIqKCnbv3h1QYMakn1AoRKtWrSgsLIx/sMjeqcpNTLHW4P5KRL6MtiUzyIy2eTP88pdQUgLffgvjx7tvOj6SRnFxMbt372aJfSsyZo/y8nL69u1LG79fvCxZ+BLrnsWZSYsim113nasCz5jhbmw/9BDccgvceSf8+98x3xrZIyo8uaAx2c53t9mwoiJ4883gAsoQsZqhWgP9vGaoPRtuBLevXlQmjtdfh8ceg1//Go46au/+Xr1c0vjii5hvt7EWxtQXd9GjugoLYcWKmL0PTexkcT+wqYH9m7zXTFOsXeuanIYMgdtvb1QRffr0IS8vz5KFMZ6dO3eybNmyxGoWhYVuadXly4MLLAPEShY9VfWruju9fQMCiygbqMKVV7qE8dRTMRc6iqVVq1YMGjTIkoUxnqVLl6KqidUswuta2L2/mGI1J+XHeK1dcweSVZ5/Hl54AX73Oxg6tElFWfdZY/ZKqNtsmI218CVWzaJMRC6ru1NExgOzggspwy1fDj/7GYwcCb/6VZOLKy4uZvHixbglR4zJbpYsghOrZjEBeElEzmNvcigF2gA/CjqwjKQKl14KO3e65qfcpvcTKC4uZuvWraxcuZJevXo1Q5DGpK9QKERubi59+/b1/6aOHWG//VwzVFWV64n48cdxO5hkm6ifVqq6EjhKRE4ADvV2v66q7yYlskz0yCOui95f/gL7798sRUZ2n7VkYbJdeXk5RUVF5OTkJPbGXr1g6lR44om487Jlq7hfbVX1PeC9JMSS2b77zg2+O/lkd3O7mUR2nz366KObrVxj0lHC3WbDNYkFC2xNizhsQsBkqKmBiy5yzU6TJ7vJApvJgAEDaNWqlfWIMoZGDMgbOxYeftgShQ+WLJLhj3+EDz+EBx/cezOtmbRp04bCwkLrEWWy3vbt21m5cmViNYspU+CKK9wUO61bBxZbJrBkEbS5c+Hmm+FHP4Lzzw/kFDZVuTF7pyZPqGYRni1h8WK47LJ952TbvLl5A0xzliyCVF0NF14IXbq4qq5IIKexZGFMI7vNhoWTRkUFnHaa2/fDH8K2bc0YYXqzZBGkO+903e8efRS6dw/sNMXFxaxZs4ZNmxqancWY7JDQokfR9OoFb7wBzz4LH3wAY8a4ru7GkkVgZs6Eu+6C//ovOPvsQE9l63Eb42oWeXl5zdOFfNw49yXvzTfdz7ZmjCWLQGzb5pqf+vSBP/858NPZ7LPGuJpF//79adVcvQ0vvdT9/b70kuvNmOU9pmyq8SDceKPrtz1tmrtfETBLFsY0otusHz//OWzdCv/939ChA0yaFNi9x5bOkkVze/ddeOABuOYaOOmkpJyyU6dOdO/e3brPmqwWCoU4/PDDm7/gG2+ELVvcxJ/t27uu8FmYMAJthhKR00RkgYgsEpGJDbx+nYjM85Zq/ZeI9A8ynsBt3AgXXwwHHAB3353UU1uPKJPNNm/ezNq1a5t2czuW3/7W1TLuvx9uuy2Yc7RwgdUsRCQHeAgYDVQCn4nIq6o6L+KwL4BSVd0mIlcCvwfODSqmwE2YAJWVbjnU9u2Teuri4mJmzJiR1HMa01I0qdusHyLwpz+5Jqk773RNUjfcEMy5WqggaxZHAItUdbGqVgPPA/t0C1LV91Q13JH5E9ySrenp1Vfhb39zVdbvfz/ppy8uLmbp0qXstG5+Jgs1S7fZeFq1cuOlxo2DiRPdhKBZJMhk0ReInCC+0tsXzaXAGwHGE5zVq+GSS6BbN3jttZSEUFJSgqru+aMxJpsEXrMIy8mBJ5903eGvucbNUpslWkTXWRE5H7dWxr1RXr9cRMpEpGz16tXJDS6e5ctdTWLtWti0CebMSUkY1iPKZLNQKET79u0pKCgI/mStW7s5pU45BcaPdz8HqaoKrroKhg8P9jxxBJkslgGRs+b18/btQ0ROBm4CzlLVBttQVPURVS1V1dLuAY6ETkhVFfy//wf9+4P3rYZdu1IWTjhZWI8ok43C3WYlWb2U8vLc+Iujj3Zzvv3f/zX/OcJJYtAgePxxmD27+c+RgCC7zn4G7C8iA3FJYizw08gDRGQ48DBwmqquCjCW5nfaafDll6mOYo8ePXrQoUMHq1mYrBTIGIt42rd3zc4nnwznnAOvv+5+bqzqajc+a8aMvclB1W0tQGDJQlV3i8jVwFtADjBZVb8Wkd8AZar6Kq7ZqSPwgveNYImqnhVUTM1izRq4/nqXKDp12jtvTIpX1hIR6z5rslL4Xt1xxx2X/JN37uymBDnhBHcf46234Jhj3GvRlmitqXGtEXPn7rstWBB/WpHDDoPjjoPjj4djj4VkNLt5Ah2Up6pTgal19t0a8XMT0nCSqbp1s3/5Szee4sYb3dTjmza5X4gnnnC/BClMGsXFxXzzzTcpO78xqbB+/Xo2bdqU/JpFWNeu8Pbb7gP8Bz9w9zBefXXfJVrvu29vUpg3D7Zv3/v+gQPh0EPhrLPcY69e8Pe/u8+bup8p+fmuR1Z4GqHBg915jzvObX36BPfvVNW02kaMGKFJN3++6qhRrkJ41FGqX31V/5iqKtWrrlIdNiz58Xmuv/56zcvL05qampTFYEyyzZo1SwH9xz/+kdpAyspUO3Z0nxMi4QakvVvv3qqjR6v+4heqjz+u+umnqps3Ry8v/JnSrp1qmzauDFXVnTtVP/pI9Xe/Uz3ttL3nBNWSEtVLLlF98knV8vK9ZS1frgNhlTbhszflH/6JbklNFjt2qN5+u/uP6tJF9eGHVVvwB/GkSZMU0CVLlqQ6FGOS5sUXX1RAP//889QGctxxDScJUF2zpvHlxvsiumuX6mefqf7hD6pnnaW63357z9u3r+r++6u2bq2HQa024bO3RXSdbZGmT4chQ+D2292c9vPnw+WXN+v62c3Nus+abJSUAXl+TJkCV17plmiNXHEP3BisxgovzBR53yNSbi6UlsJ118Err7j7ql9+6ZZx3rkTFi6EXbsQaFJXsZb7yZcqa9a4+Z1OOMF1hX3zTbcQSnPMkR8w6z5rslF5eTldunRhv/32S20gkUu0jh/fcNJIhlat4Hvfg6uvhq++cgksLw+FJnWrsmQRpupGZh50EPzv/7rh/HPnwqmnpjoy3woLC8nNzbWahckqKek2G0vdpDFsWGpj+etfIRRiHaxpSlE2RTm4LmtXXOGano480vU2+N73Uh1VwnJzcxkwYIAlC5NVQqEQBxxwQKrDqC+cNFqCXr0IwZKmFJGeNYvmGv6+c6cbMzF4sFtvd9Ik+PDDtEwUYTbWwmQTVTfGokXVLDJU+tUslixxw9/D/ZejUXUDXHbuhB073GPkNnu2m1J840bXxldb66bvSHPFxcV88sknqGrypj4wJkVWr17Ntm3bUn9zOwukX7KoO5Hg9763NwHUTQrq835OBq2tW1JSwsaNG1m3bh3dmtIDw5g0kLTZZk0aJou69t/fTeoV3tq23fd53e2ee9w9Cr+JJM1E9oiyZGEyXThZWM0ieOmXLET2/aD/5z8Te/9pp7WY6TmCEDnW4vspWITJmGQKj7GwmkXw0u8Gd0FB0/ovt5S+0AEZNGgQYAPzTHYoLy+noKCAjh07pjqUjJd+yaKoqHn6L7ekvtDNqF27dvTp08eShckKoVDImqCSJP2aoaB5+y+3pL7QzcS6z5psUV5ezrAM+aLX0qVfzcJE1atXL0SEGTNm8OGHHyIiiAi9GjFVSbisulsqy7KYkl9WS49p4cKFvPDCC42Oyfgnmma9gkpLS7WsrCzVYbRIscZVJPr/3BLLspiSX1amx5RNRGSWqpY29v3p2QxlEnbIIYdkdFkWU/LLaokxmeBYssgSgwcPTuj4efPmtbiyLKbkl5VuMZngWDNUBmmJTQbNWZbFlPyyMj2mbNLUZii7wW2MMSYuSxYZpGfPngntT7eyLKbkl5XpMRn/rBnKGGOygDVDGWOMCZwlC2OMMXFZsjDGGBOXJQtjjDFxWbIwxhgTV9r1hhKRzcCCVMdRRwGwJtVB1NESY4KWGZfF5I/F5F9LjOtAVe3U2Den43QfC5rS/SsIIlJmMfnTEuOymPyxmPxriXGJSJPGHFgzlDHGmLgsWRhjjIkrHZPFI6kOoAEWk38tMS6LyR+Lyb+WGFeTYkq7G9zGGGOSLx1rFsYYY5IsrZKFiJwmIgtEZJGITExRDIUi8p6IzBORr0XkWm9/VxF5R0QWeo/7pSC2HBH5QkRe854PFJFPves1RUTaJDmefBF5UUTmi8g3InJkqq+TiPzC+3+bKyLPiUjbVFwnEZksIqtEZG7EvgavjTgPePF9KSKHJTGme73/vy9F5CURyY947UYvpgUicmqyYop47ZcioiJS4D1P2XXy9l/jXauvReT3EfsDv07R4hKRYSLyiYjMFpEyETnC25/4tVLVtNiAHOA7YBDQBpgDDE5BHL2Bw7yfOwHfAoOB3wMTvf0TgXtSENt1wLPAa97zvwNjvZ8nAVcmOZ4ngfHez22A/FReJ6AvUA60i7g+F6XiOgHHAYcBcyP2NXhtgDOANwABRgKfJjGmU4Bc7+d7ImIa7P0N5gEDvb/NnGTE5O0vBN4CKoCCFnCdTgCmAXne8x7JvE4x4nobOD3i+kxv7LVKp5rFEcAiVV2sqtXA88DZyQ5CVatU9XPv583AN7gPobNxH454j/+RzLhEpB/wA+Ax77kAJwIvpiImEemC++V9HEBVq1V1Aym+TrixRe1EJBdoD1SRguukqh8A6+rsjnZtzgaeUucTIF9EeicjJlV9W1V3e08/AfpFxPS8qu5U1XJgEe5vNPCYPH8Cfg1E3nRN2XUCrgTuVtWd3jGrImIK/DrFiEuBzt7PXYDlEXEldK3SKVn0BZZGPK/09qWMiAwAhgOfAj1Vtcp7aQWQ7JVY7sf98dR6z7sBGyL+0JN9vQYCq4EnvKaxx0SkAym8Tqq6DLgPWIJLEhuBWaT2OkWKdm1ayu/+Jbhvo5DCmETkbGCZqs6p81Iqr9MBwLFec+b7InJ4C4gJYAJwr4gsxf3u39jYuNIpWbQoItIR+AcwQVU3Rb6mrp6XtG5mInImsEpVZyXrnD7k4qrE/6Oqw4GtuKaVPVJwnfbDfaMaCPQBOgCnJev8iUj2tYlHRG4CdgPPpDiO9sB/A7emMo4G5AJdcU06vwL+7tXuU+1K4BeqWgj8Aq+m3xjplCyW4dopw/p5+5JORFrjEsUzqvpPb/fKcDXOe1wV7f0BOBo4S0RCuOa5E4E/46qW4Sldkn29KoFKVf3Ue/4iLnmk8jqdDJSr6mpV3QX8E3ftUnmdIkW7Nin93ReRi4AzgfO8JJbKmIpxyX6O9/veD/hcRHqlMCZwv+//9Jp1ZuJq+AUpjgngv3C/5wAvsLcJLOG40ilZfAbs7/VcaQOMBV5NdhDet4XHgW9U9Y8RL72K+4/Be3wlWTGp6o2q2k9VB+Cuy7uqeh7wHnBOimJaASwVkQO9XScB80jhdcI1P40Ukfbe/2M4ppRdpzqiXZtXgQu9HiwjgY0RzVWBEpHTcM2bZ6nqtjqxjhWRPBEZCOwPzAw6HlX9SlV7qOoA7/e9EtfhZAUpvE7Ay7ib3IjIAbgOHWtI0XWKsBw43vv5RGCh93Pi1yqIu/JBbbg7+N/iehTclKIYjsE1D3wJzPa2M3D3CP7l/WdMA7qmKL5R7O0NNQj3i7kI960iL8mxDAPKvGv1MrBfqq8TcAcwH5gLPI3rpZL06wQ8h7tvsgv3gXdptGuD67HykPd7/xVQmsSYFuHatsO/65Mijr/Ji2kBXo+bZMRU5/UQe3tDpfI6tQH+1/u9+hw4MZnXKUZcx+Duy83B3Vsd0dhrZSO4jTHGxJVOzVDGGGNSxJKFMcaYuCxZGGOMicuShTHGmLgsWRhjjInLkoVJKyJS482gOVdE/i9yFtQoxw8QkZ/6KHef40SkVEQeaKaYQyLylbfNE5Hfikjb5ijbmGSxZGHSzXZVHaaqh+ImTftZnOMHAHGTRd3jVLVMVX/e2CAbcIKqfg83gnYQ8HBzFBox8tyYQFmyMOnsY7zJz7yRqPd6NY6vRORc75i7cRO8zRa3lsUAEZkhIp9721FRjhsle9cF6SoiL3vz/n8iIkO8/bd7awhMF5HFIhI3uajqFuAK4D9EpKtXzq9E5DOv/DvCx4rILeLWQPhQ3Nob13v7p4vI/SJSBlwrIt1F5B9eGZ+JyNHecR28+GZ6kzkmfZZmkznsW4lJSyKSg5uuIzwx2o9xI8aH4ubk+UxEPsBNXni9qp7pva89MFpVd4jI/rhRr6UNHDcq4nR3AF+o6n+IyInAU965AA7CTfPQCVggIv+jbt6pqFR1k4iU46av6YKbAuII3KjaV0XkOGA7MMb797TGjQqOnCiyjaqWerE+C/xJVT8UkSLcOg8H40YOv6uql3jNdTNFZJqqbo17gY2pw5KFSTftRGQ2rkbxDfCOt/8Y4DlVrcFNyPc+cDiwqc77WwN/EZFhQA1uaul4jsF9cKOq74pINxEJrxHwuro1DHaKyCrctOKVPsoMz0h6ird94T3viEsenYBXVHUHsENE/q/O+6dE/HwyMFj2TnLaWdysyKfgJpi83tvfFijCXTdjEmLJwqSb7ao6zKshvIW7Z5HIjehfACtx39hbATuaGM/OiJ9r8PE3JSKdcPdIvsUljbtU9eE6x0yIU0xk7aAVMNJLLJFlCDBGVRfEi8mYeOyehUlL6mZA/TnwS+8m7wzgXHHrkHfHrdI3E9iM+5Ye1gWoUtVa4ALccr00cFykGcB5sKd5ao3WWcPEL+8b/1+Bl1V1PS7hXeLtR0T6ikgP4CPgh+LWCO+ImyI8mreBayLOEW4iewu4xksaiMjwxsRsDFjNwqQxVf1CRL4ExuFm/DwSN7umAr9W1RUishaoEZE5wN9wH9T/EJELgTfZ+w39yzrHfRFxqtuByd65trF3GvFEvOd9aLcCXgLu9P4Nb4vIwcDH3mf6FuB8Vf1MRF714lqJmxl0Y5Syfw485MWXC3yAu4l+J24FxS9FpBVu/fFYSceYqGzWWWNaKBHpqKpbvCa3D4DL1Vv/3Zhks5qFMS3XIyIyGHdj+klLFCaVrGZhjDEmLrvBbYwxJi5LFsYYY+KyZGGMMSYuSxbGGGPismRhjDEmLksWxhhj4vr/oyhN99m1GUIAAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX8AAAA1CAYAAACp8OvZAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAADTxJREFUeJzt3V1MHGUXB/Czy7J8G1pWQitWUk3TIIloGkIoNpC0EmKkkqgtCQWJRiBsmxKCXJDQxA80VWsN0WqRi4oGmoJKiL1zbdwUTWmxSmi1dRvshu62tWLBrmVndv7vBZl5l9qPmWcQoXN+SS+62Tl75tln/vOx06kNADHGGLMW+3/dAGOMsYXH4c8YYxbE4c8YYxbE4c8YYxbE4c8YYxbE4c8YYxbE4c8YYxbE4c8YYxbE4c8YYxbE4c8YYxbkWODPW5BnSSiKQna7uf2aLMvkcJgbnkgkQjExMaZqMMbuHpIkUWxs7L/9MTY9b7orj/ztdjtNTk6aquFwOCgUCpmqMR/BH4lETNdgjJknSZLpGgsQ/LotuvB/7bXX6ODBg6ZqnDx5kjZt2kTHjx8XDs9QKERJSUm0a9cu4T52795NNpuNjh8/LlwjJiaG9u7dS+Pj40QktjMIh8NzlpVlWbgfFT8QkC0ERVH+8zpXrlyhF198kS5evCj82cFgkGpra2nTpk00MzMj3Munn35q+qqGatGF/wMPPEAHDhwwVSMrK4tOnDhBfX19QiEFgBITEyk7O5va29uFelAUhZ599lmqq6ujhoYGCgaDQr1cvHiRDh8+TBUVFfThhx8KnU04nU4iInrllVdoeHiYHA6H0MYgSZK2A7HZbLwDYP8KRVG0uWW3202Fpd/vp8cff1woMNW53traSl1dXfT7778L9aAoCvX09NCXX35JpaWl2vZoFAA6cOAArVy5Umj5mxZcwD93VF5ejt7eXj1vvS0igsfjMVWjrq4Os0MkLhQKgYjw1VdfIRwOC9eprq6Gw+EAAMzMzAjVaGtrg8vlMjUuRIS4uDiUlpYCgOF1UnsfGRnBt99+K9wHm1+SJCESiUBRFMPLRiIRRCIR7e+nTp3C0aNH57ymtw4AjI+Po7i4GE6nE42NjYbrALPrk5WVBSJCRUWF4eUVRUEoFMILL7wAIsJDDz2kvW5UOBwGESE5OdnwstG2bdsGIkIwGLzTW3Xl8aIK/5GRETidToRCoTuPxC0oioLLly+DiDA6OgpZloXqyLKMiooKU+GvThQiQk1NjdDEUZfp7OwU7kUdg8uXL+O9997DM888A0VRhHZGg4ODyM3NndOLJEmGaoTDYSQmJuKee+5BX1+f9pqRfhRFgSRJCIVC8Pv92vpIkiQ0zkuFum6yLCMSiRge++g6N4b29PS0qd6++OILNDU1weVyGZ6r6nr19/ejqKgINHtzCM6cOSPUy6FDh0BEcLlcOHv2rOHlJUmC1+tFTk4OiouL4fP5hOfVxMQEiAh5eXnC3xcwG/42m03PW5de+Dc3N+tdudsaGxsDEaGlpUX4KBkAurq6kJiYiNHRUVNf2tNPP236DEKWZfT09ICI8NZbbwGA8JGVekQ0PDwsVAcAent7QUSIj4/H1atXARg7Krp69SrOnDkDm82m7Ry//vpr3ctLkoSamhosX74caWlpICK43W54vV5cunRJe5+edTtx4gR6e3uxefPmeTnrvJmpqSn8+eef/3j9dv2pO+3oOXzu3DmcP38eo6OjePvtt/HRRx+hs7MTIyMjuvpQa4VCIRw5cgT79u3DihUrtLCNj4/HH3/8ccc66vbg9/vhdru1A4LMzExs375dd+Cq6//DDz+gvb0ddrsdRARJkhAOh4XmZn5+PogI9fX1AIwfrUciEUxNTYGIkJCQgOvXrxvuQf3ciYkJlJWVoba2FoFAQGh9VDab7e4N/8rKynkJ/7Nnz4KIUFJSInypJRKJoLu7G6tWrTK9E6mrq0NSUhI6OzuFawDA8PCwtpGqPYrYu3cv0tPTkZqaKtxLOBzWjtBWr16NgYEB3ctGn411dHRg3bp12np1d3djampK187W5/NhZGQEHo8HVVVVWo2YmBh0dHRgYmJCVz8JCQnahmWz2ZCenq792bBhA44dO6b7DHJ4eBgtLS3weDwYHR1FS0sLSkpKUFhYiNraWnR2dqKhoQH9/f0IBAK6agLQahUUFCA/Px+5ubnIyMgAESElJUW7HHCzOaGGX/RZZGFhoTZeashlZGTA4/EYOlvesGGDVuPJJ5+cs+PVQ/2eCwsLkZCQgPLycuzevRuA8UuKADAwMAAiQmxsrPCZTDgcRnd395xLRiJH/bIso6OjAytXrjS0fdzKXR3+69evNx3+0ZdaCgsLTdUCgKqqKhQUFMypLdJPQ0MDHnvsMRw7dkyoD3WDbGpqAhFhcHBQqI5qcnISDocD7e3tAMR3JKdPn0ZycjKICL/++qtwrcHBQezcuVMLEq/XCwC3DSL1GrU6xn///TfefPNN1NbWanWKioqwZcsWQ718//33aG5uRmVlpTYn1TMUm82G7u5u7Wwn2tTUFHJyclBfX4+tW7fi4Ycf1sL6qaee0kI7Pj4eSUlJc3bkN65nMBiEw+HQ3pOYmIhVq1ahqqoKra2t+Oyzz7B//354vV50dXXhwoULt5yfgUAAhw8fRk1NDYgISUlJWLduneE5FN2j2+0GEWHt2rXw+Xza60YCe2ZmBoFAAGvXrtUOsgDjlxHVZVpaWkBEePnllwGIzUO1fyLCzp07DS+vUr+LmJgYOJ3OOa+JstlsqKys1PPWpRf+BvZst6ROUCJCbW2tqVoA0Nraivz8fNN1BgcHQUQ4cuSIcNBKkgSfz4fU1FS43W4AYhNcHaOysjKkpaXh6NGjQrXUybxnzx7ExsaivLwchw4dMtxP9EZRUVGBhIQE5OTk6Plh65YGBgbm7AT27Nlj6EhbNTMzg7GxMWzduhWpqanaHL3xwCL68sXk5CSA2d9YxsbG5lwC+fnnnzE9PY33338fdXV1aGxs/MdnXrt2DXl5ecjOzobb7UZFRQW6urrQ3d1tuH8AyMvLQ0JCAogITqcTdXV1+OabbwD8f+zvFLjq+3788UdtLjudTnzwwQe6lr+VJ554QrtEMzk5KVRHDez4+HgQEU6fPi3US3QtswdY6noQEZYtWwbg9gcyenD430H0oO/atcvUNTYAGBoawurVqwGI77mjz0b27dtnuqf+/n7k5uZiaGjI1I5kZmYGdrsdJSUlwr2o4+31ehEbG4s1a9YAMD5W0Rt9W1sbiAjV1dXw+/2611FRFMiyrH12IBBAXl4ekpOTkZ6ejpdeeslQTzcKBoMoKyvTPU+jx0DtLdqNP7hG27JlC4aGhm56/V0dK7Xe7ULll19+ARFh/fr1ePXVV+HxeLRLmCJB++CDD8LhcCAzMxM9PT1CdSRJwrVr1+DxeEBEyMrKwvj4uOFegLljrJ7p3fi6UeodeleuXBGuEb0TycnJATA/4d/c3KznrUsz/Ldt26Zn5W7rp59+QkpKClpbW00P+P79+5GRkQFA/NKIioiwYsUKUzWia6WlpZmu8/zzz4OI8PHHH5uu9fnnn8PlcqG8vBznz58XqqFutD6fTztqF92Qo++Geeedd7Bx40YAYteS/wvRl7REybKMc+fOafVE5/D169dRWVmJxsZG7TKPmd7uvfdeuFwu+P1+4Rrq55eWloKI0Nvba+q7VRQFfX19yMrKQkdHh3AdVTAYBBHh4MGDpmuFQiE4nU69P+wvrfBXB2o+wn96ehp2ux0NDQ2mawUCARCRqeBXA0g99Y5+zSh1cmdmZpq+g0i9NTI3NxfFxcVaSIjWmp6e1n4Efv31102FNgDU19ebHi+VoijCd20sVWbH7EbvvvuudknFzPYgyzIcDgfa2tpM1wKAuLg45Obmmqqh9pCdnY24uDhTtYDZ+dbU1ISqqirTB6DA7N115eXlet++tML/vvvuw8aNG+flHu3nnnsO999/P0ZHR01PrK6uLqSkpJjuCYB2d8b09LTpvrZv3w4iwqlTp0z35ff7QUTYsWOHqTpq2HR0dODkyZOm7pBii0t0gM3HNvrdd9/NSy1ZluflBghVYmLivPxW6Pf7UVBQoPsW3DspLS3FJ598ovftuvJ40Tze4cKFC1RdXU02m64H0t2Soii0fPlyevTRR8nlcpl+DkZ2djb99ddfpmoQzf5T8c2bNxMRUXJysqm+IpEI7dixg9asWWP6AXaKolBmZiY1NjbSG2+8YeoZKOpTUN1uNz3yyCPC/4ydLT7RjxUxu42Gw2HKz8+fl1oxMTHk9XqpqKhoXp4DJMsypaSkmK6TlpY2r/O/urqafvvtt3mrR0RkAxb0+SwL8mE+n48yMzMpLi7OdC0AdOnSJUpPTzc9USORCE1NTdGyZctM96Waj8dXM8Zmt3Wz2/gioWsl7srwX+z4Of+M3d0AkCRJ/9XZL4c/Y4xZkK7wX+j/yeuuOKdijLGlji8WM8aYBXH4M8aYBXH4M8aYBXH4M8aYBXH4M8aYBXH4M8aYBXH4M8aYBXH4M8aYBXH4M8aYBXH4M8aYBXH4M8aYBXH4M8aYBXH4M8aYBXH4M8aYBXH4M8aYBXH4M8aYBXH4M8aYBXH4M8aYBXH4M8aYBXH4M8aYBXH4M8aYBXH4M8aYBXH4M8aYBf0PwIACOHRGfV4AAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "rotating_image_classification(digit_one, sess2, prob2, X2, keep_prob2, u)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Using the Expected Cross Entropy (Eq. 4)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In this section, we train neural network using the loss function described in Eq. 4 in the paper. This loss function is derived using the expected value of the cross entropy loss over the predicted Dirichlet distribution." ] }, { "cell_type": "code", "execution_count": 219, "metadata": { "colab": { "autoexec": { "startup": false, "wait_interval": 0 } }, "colab_type": "code", "collapsed": true, "id": "pzgZFuUx2vM-" }, "outputs": [], "source": [ "def loss_EDL(func=tf.digamma):\n", " def loss_func(p, alpha, global_step, annealing_step): \n", " S = tf.reduce_sum(alpha, axis=1, keep_dims=True) \n", " E = alpha - 1\n", " \n", " A = tf.reduce_sum(p * (func(S) - func(alpha)), axis=1, keepdims=True)\n", " \n", " annealing_coef = tf.minimum(1.0, tf.cast(global_step/annealing_step,tf.float32))\n", " \n", " alp = E*(1-p) + 1 \n", " B = annealing_coef * KL(alp)\n", " \n", " return (A + B)\n", " return loss_func" ] }, { "cell_type": "code", "execution_count": 210, "metadata": { "collapsed": true }, "outputs": [], "source": [ "g3, step3, X3, Y3, annealing_step3, keep_prob3, prob3, acc3, loss3, u3, evidence3, \\\n", " mean_ev3, mean_ev_succ3, mean_ev_fail3 = LeNet_EDL(exp_evidence, loss_EDL(tf.digamma), lmb=0.001)" ] }, { "cell_type": "code", "execution_count": 211, "metadata": { "colab": { "autoexec": { "startup": false, "wait_interval": 0 } }, "colab_type": "code", "collapsed": true, "id": "PQ2AqucY20f2" }, "outputs": [], "source": [ "sess3 = tf.Session(graph=g3)\n", "with g3.as_default():\n", " sess3.run(tf.global_variables_initializer())" ] }, { "cell_type": "code", "execution_count": 212, "metadata": { "colab": { "autoexec": { "startup": false, "wait_interval": 0 }, "base_uri": "https://localhost:8080/", "height": 179 }, "colab_type": "code", "executionInfo": { "elapsed": 2222, "status": "ok", "timestamp": 1527923836119, "user": { "displayName": "Murat Sensoy", "photoUrl": "https://lh3.googleusercontent.com/a/default-user=s128", "userId": "102692943223630372304" }, "user_tz": -180 }, "id": "FDhvxNKN25VE", "outputId": "b96b936e-955f-4338-faa3-ccd2f0a7e36b" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "epoch 1 - 100%) training: 0.8303 (42.6223 - 16.4772) \t testing: 0.8412 (42.8285 - 16.6934)\n", "epoch 2 - 100%) training: 0.8992 (123.3957 - 11.1870) \t testing: 0.9062 (122.5605 - 11.3092)\n", "epoch 3 - 100%) training: 0.9206 (208.4110 - 9.0025) \t testing: 0.9272 (203.0646 - 9.2725)\n", "epoch 4 - 100%) training: 0.9413 (264.3557 - 8.2034) \t testing: 0.9446 (259.2374 - 8.0735)\n", "epoch 5 - 100%) training: 0.9485 (298.0337 - 6.7932) \t testing: 0.9540 (297.0534 - 6.4097)\n", "epoch 6 - 100%) training: 0.9526 (300.2455 - 5.8476) \t testing: 0.9573 (301.2290 - 5.1179)\n", "epoch 7 - 100%) training: 0.9595 (536.3224 - 6.1392) \t testing: 0.9631 (548.7296 - 5.8329)\n", "epoch 8 - 100%) training: 0.9632 (616.2153 - 5.7967) \t testing: 0.9669 (642.6508 - 5.4167)\n", "epoch 9 - 100%) training: 0.9664 (691.5225 - 5.5155) \t testing: 0.9705 (711.5176 - 4.6776)\n", "epoch 10 - 100%) training: 0.9671 (743.4854 - 4.3620) \t testing: 0.9693 (765.5163 - 4.1878)\n", "epoch 11 - 100%) training: 0.9695 (1386.9512 - 6.1451) \t testing: 0.9735 (1369.8014 - 5.8578)\n", "epoch 12 - 100%) training: 0.9723 (1498.5973 - 6.5162) \t testing: 0.9747 (1543.9032 - 6.2582)\n", "epoch 13 - 100%) training: 0.9717 (1548.7797 - 4.6337) \t testing: 0.9743 (1684.6056 - 4.7657)\n", "epoch 14 - 100%) training: 0.9735 (1498.0391 - 5.4553) \t testing: 0.9757 (1613.8787 - 5.6297)\n", "epoch 15 - 100%) training: 0.9759 (1493.4908 - 5.2220) \t testing: 0.9779 (1563.8809 - 4.5270)\n", "epoch 16 - 100%) training: 0.9764 (2248.6760 - 5.7876) \t testing: 0.9777 (2307.0652 - 5.2200)\n", "epoch 17 - 100%) training: 0.9769 (1995.6049 - 4.3072) \t testing: 0.9804 (2252.7351 - 4.1795)\n", "epoch 18 - 100%) training: 0.9765 (2430.6807 - 4.4716) \t testing: 0.9795 (2640.8728 - 4.7834)\n", "epoch 19 - 100%) training: 0.9776 (3172.2893 - 5.9542) \t testing: 0.9792 (3272.4224 - 5.0041)\n", "epoch 20 - 100%) training: 0.9788 (3570.1475 - 4.9005) \t testing: 0.9812 (4264.9502 - 5.8081)\n", "epoch 21 - 100%) training: 0.9787 (4876.1646 - 6.1165) \t testing: 0.9808 (5475.2358 - 5.8976)\n", "epoch 22 - 100%) training: 0.9799 (3180.2634 - 3.9057) \t testing: 0.9813 (3317.7156 - 3.9012)\n", "epoch 23 - 100%) training: 0.9806 (2328.6206 - 3.7536) \t testing: 0.9826 (2397.2944 - 3.6987)\n", "epoch 24 - 100%) training: 0.9798 (4677.1987 - 4.4783) \t testing: 0.9823 (5023.0850 - 4.6507)\n", "epoch 25 - 100%) training: 0.9811 (3927.5339 - 4.8530) \t testing: 0.9830 (4024.1538 - 5.2015)\n", "epoch 26 - 100%) training: 0.9798 (4298.3862 - 5.0999) \t testing: 0.9831 (4730.0220 - 4.9904)\n", "epoch 27 - 100%) training: 0.9811 (4639.9775 - 5.3606) \t testing: 0.9831 (4931.9014 - 5.6236)\n", "epoch 28 - 100%) training: 0.9825 (4483.1514 - 4.3366) \t testing: 0.9844 (4708.0806 - 4.3383)\n", "epoch 29 - 100%) training: 0.9829 (4930.3579 - 4.2995) \t testing: 0.9836 (5377.6284 - 4.9327)\n", "epoch 30 - 100%) training: 0.9829 (6808.6553 - 3.7501) \t testing: 0.9822 (7229.0522 - 3.9000)\n", "epoch 31 - 100%) training: 0.9829 (6299.5278 - 5.7005) \t testing: 0.9845 (7308.3398 - 5.1458)\n", "epoch 32 - 100%) training: 0.9829 (6325.0591 - 5.7004) \t testing: 0.9854 (6570.1626 - 6.2992)\n", "epoch 33 - 100%) training: 0.9834 (5909.4917 - 4.6970) \t testing: 0.9843 (6029.9980 - 5.9591)\n", "epoch 34 - 100%) training: 0.9841 (7869.9731 - 5.8773) \t testing: 0.9849 (8628.2305 - 5.6427)\n", "epoch 35 - 100%) training: 0.9836 (8436.3779 - 4.7038) \t testing: 0.9850 (9239.3271 - 6.6275)\n", "epoch 36 - 100%) training: 0.9841 (5976.3433 - 3.9183) \t testing: 0.9854 (6832.0786 - 4.6664)\n", "epoch 37 - 100%) training: 0.9835 (7943.0410 - 4.1992) \t testing: 0.9848 (9035.4375 - 5.9112)\n", "epoch 38 - 100%) training: 0.9856 (7578.3315 - 4.6529) \t testing: 0.9869 (8479.9932 - 5.4112)\n", "epoch 39 - 100%) training: 0.9851 (6393.1548 - 4.1753) \t testing: 0.9859 (7391.0679 - 4.8497)\n", "epoch 40 - 100%) training: 0.9861 (7625.8906 - 4.5776) \t testing: 0.9862 (8205.8027 - 5.7368)\n", "epoch 41 - 100%) training: 0.9859 (8331.8799 - 3.7653) \t testing: 0.9870 (9274.6816 - 4.7081)\n", "epoch 42 - 100%) training: 0.9859 (13754.2227 - 4.9862) \t testing: 0.9875 (14675.8350 - 5.6302)\n", "epoch 43 - 100%) training: 0.9855 (15424.7334 - 6.8568) \t testing: 0.9864 (16225.4473 - 7.8045)\n", "epoch 44 - 100%) training: 0.9862 (12365.5068 - 4.6459) \t testing: 0.9866 (13331.6621 - 5.5387)\n", "epoch 45 - 100%) training: 0.9861 (12153.3447 - 4.4586) \t testing: 0.9876 (14052.3027 - 6.7540)\n", "epoch 46 - 100%) training: 0.9865 (17505.6484 - 4.8752) \t testing: 0.9870 (18024.4199 - 6.1390)\n", "epoch 47 - 100%) training: 0.9859 (15554.4268 - 4.5205) \t testing: 0.9860 (17220.3281 - 5.6510)\n", "epoch 48 - 100%) training: 0.9870 (12240.6279 - 3.7974) \t testing: 0.9873 (14007.5693 - 5.2733)\n", "epoch 49 - 100%) training: 0.9867 (13552.8066 - 3.8983) \t testing: 0.9875 (15117.2031 - 4.9112)\n", "epoch 50 - 100%) training: 0.9858 (13994.8203 - 3.8285) \t testing: 0.9854 (14965.4561 - 4.7617)\n" ] } ], "source": [ "bsize = 1000 #batch size\n", "n_batches = mnist.train.num_examples // bsize\n", "L3_train_acc1=[]\n", "L3_train_ev_s=[]\n", "L3_train_ev_f=[]\n", "L3_test_acc1=[]\n", "L3_test_ev_s=[]\n", "L3_test_ev_f=[]\n", "for epoch in range(50): \n", " for i in range(n_batches):\n", " data, label = mnist.train.next_batch(bsize)\n", " feed_dict={X3:data, Y3:label, keep_prob3:.5, annealing_step3:10*n_batches}\n", " sess3.run(step3,feed_dict)\n", " print('epoch %d - %d%%) '% (epoch+1, (100*(i+1))//n_batches), end='\\r' if i" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "draw_EDL_results(L3_train_acc1, L3_train_ev_s, L3_train_ev_f, L3_test_acc1, L3_test_ev_s, L3_test_ev_f)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The figure above indicates that the neural network generates much more evidence for the correctly classified samples. As a result, it has a very low uncertainty (around zero) for the correctly classified samples, while the uncertainty is very high (around 0.7) for the misclassified samples." ] }, { "cell_type": "code", "execution_count": 214, "metadata": { "colab": { "autoexec": { "startup": false, "wait_interval": 0 } }, "colab_type": "code", "id": "yrZTPQ563PlZ" }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEKCAYAAADjDHn2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xl8VPXV+PHPycIedkiQfRlcggKCuNR9XyiCS8XtccM+amvFLo9YrXVpf9rW9rGttlartD5usWqRCm4Ut7oCKphhkQQQIgSQHWLIdn5/fO+QScgsSWbNnPfrdV8zc+fOvSc3yT1z7/fc71dUFWOMMSacrGQHYIwxJvVZsjDGGBORJQtjjDERWbIwxhgTkSULY4wxEVmyMMYYE5ElC2OMMRFZsjDGGBORJQtjjDER5SQ7gObq3bu3DhkyJNlhGGNMWlm0aNHXqtqnpZ9Pu2QxZMgQFi5cmOwwjDEmrYjIl635vF2GMsYYE5ElC2OMMRFZsjDGGBNR3JKFiDwuIptEpDjE+yIifxCREhFZIiKHxysWY4wxrRPPM4u/AWeGef8swOdN3wX+HMdYjDHGtELckoWqvgNsDbPIucAT6nwIdBeRfpHWu2jRIkQEEaGgoCBW4RpjjAkjmW0W/YF1Qa/LvHlR27hxY7M3WlBQsC/ZBE+WeIwxJrS0uM9CRL6Lu1S1nzlz5tClSxfy8vIaPHbu3JmsrP1zYagE09zEU1BQ0ORn8vPzKS8vb9a6jDEmHhodp8a1Zl3JTBZfAQODXg/w5u1HVR8BHgEQkQaDhk+cODHkBjp37rxfEgnn8ccf35dogh+Dn+fm5gKxSzrGGBMvsTweJTNZzAa+LyLPAkcCO1R1Q3NX8tFHH7Fr1y52794d1WM411xzTcTt5ebmRkw6xhiTbGvWrInp+kRVIy/VkhWLPAOcCPQGNgI/B3IBVPVhERHgQVzFVAVwlapG7Mej8ZlFc+N3m23amjVr2L17N3v27GH37t0Nnjee9+CDD4ZcT7z2qTHGhFJdXc17773H3LlzmTNnDkuXLt1vGVUNfQCMIG7JIl6Ck0VL2gfCJYvm7ItYrccYY1pq48aNvPLKK8yZM4fXX3+dnTt3kpubywknnMDZZ5/ND3/4wwbLZ1SyGD9+vLamI8FYNUxbsjDGxEO4Y9T69etZuHDhvrOHwLHwgAMO4Oyzz+acc87hlFNOIS8vD9j/OGXJIglC/UKzs7PZtm3bvl+WMcY0R7gvon379mXTpk2ICEcddRTnnHMO55xzDqNHj27yc42PU61JFtY3VAuVl5ejqg2m+fPno6p897vftbOLdLNhA9xwA4wdm+xITAarrKwM+/6pp57Kk08+yaZNm3j//fe57bbbGDNmTMgEE3ycAha1Jra0uM8iXZx00kncc8893HbbbRx33HHccMMNyQ7JhLN3L7z/Pjz+ODz/PNTVQVVVsqPKHBs2wD33wAcfwKefJjuaVmvOJe6amhpKSkooLi5uMK1cuTLsNp566qmYxtwcdhkqxurq6vj2t7/NvHnzeO+99xg/fnyyQzI1NbByJfj9UFzsHj/7DEpKml7+5ZfhiCOgb9/ExpkpAkli5sz6BJ1mx6GmhLt89K9//atBUli2bBlV3heTrKwsRowYwahRoxg1ahR33313yPW05ngtIotUtcUHJEsWcbBlyxYOP/xwsrKy+OSTT+jRo0eyQ2qbGn8zrauD1atdQggkheJiWLGi/oxBBEaMgK1b3RTu73/QIJc0AtO4cdCtW/NiMvU2bIA773RJQtUl8YA0Ow41JVyyCBg0aNC+pBCYDjroIDp27BjVepKZLOwyVBz06tWL5557juOOO44rrriCWbNmNdn1iGmhDRvgf/4HnnsOamvdNH48LF0K33xTv9zgwTBqFJx9NhQWuucHHQQdO0J5ef2329rahpef3n4bFiyon154of69Aw9smEDGjHHra+rbsqm3ZQscfTR8GWJkT1WXyNuo999/n0MOOYRukb5s4C5bhbqclVSNG2lTfRo3bpymi9///vcK6K9+9atkh9I2rF+vesUVqllZqu7wUj+dfrrqzTerPvaY6ocfqu7cGd06N2xQveEG1Y4dVdu1c+tq7OuvVV99VfWee1QnTVItKKjfbna2aq9e7jE7u2FMRnXFCtXrrnP7F1QHDnT7OSen4b4aPlz1jjvc8mkKCDmlAmChtuLYm/SDf3OndEoWdXV1euGFF2p2dra+/fbbyQ4n/RUW7p8kYnVgDiSNMWMiL1tXp7puneqLL7qDX7xiSld1dapvvaX67W+rirjkcM01qsXF7v2mEvSpp7plQXXCBNXf/15148bk/hzNZMkixaZ0Shaqqjt27FCfz6f9+vXT8vLyZIeTnnbudAcbcN/i27evP8gk+8AcfOBr/G1527bkxZUMVVWqTz2lOm6c+/l793ZnC6H+7hsn6LIy1fvvd68DZ21nneXWuXt34n6OFurZs2eTiSI/Pz/ZoamqJYu0sHjxYu3QoYOefPLJWlNTk+xw0st//qM6bJj71jljhurevdFdOkq0QEwdOtRfjiooUP3HP9w37bZs2zbVX/9adcAA93MfeKDqX/6iWlHR8nUWF6veeqvqoEFunZ07q152mbscWF3dcNn161Wvvz66s8I4uvnmm7Vdu3a6LUW/JFiySBOPPfaYAvqzn/0s2aGkh717VX/6U9c+MWSI6rvv7r9Mcy4dJUogppEjVceOdf9ikyaprl2b7Mhib9Uq1ZtuUu3Sxf2cJ52k+vLLqrW1sdtGba3q22+rXnutavfubjv5+W67r7zi2kM6dEj6l4ba2lrt37+/Tpo0KWkxRGLJIo1ceeWVCugrr7yS7FBSm99ff6C9+mrVHTuSHVHLVFe7yyodO7oD6h//qJruZ5br16tOmeIO3FlZ7tLbZZepLloU/21XVrp2orPOarrIIYnJ4p133lFAn3rqqaTFEIklizSyZ88ePfTQQ7VXr166ti1+02yt2lrVBx5wbRK9e6v+85/Jjig2Vq1y1VqgeuSRqkuWJDui5lu71v0MgUZoUL3lFtfQn2jHH59yyeJ73/ueduzYUXft2pW0GCKxZJFmVqxYoXl5eXrUUUfp3r17kx1O9OJ9XXjdOtVTTnF/khMnhm4UTVd1dapPPumSYE6Ou8T2zTfJjiq8ujrXRnDooSl1YG7QZpUCMVVXV2vfvn31ggsuSMr2o2XJIg0999xzCuj06dOTHUpkgSQRz+vCTz/tLmt07qz6yCNtu0H466/dvSKg6vOpzp+f7Ij2t3q16i9/qXrwwU0niWQni4ANG1S7dnUFBUlss5g3b54C+o9//CMp24+WJYs0deONNyqgzz//fLJDaVogSbRvv39JaKxs3ao6dapb51FHqa5cGbt1p7o33nBVXqB61VWqW7YkN56tW12iPu64+t/zcce5Kqerr25YeZYqyULVFQ+MHJnUQodrr71Wu3Tponv27EnK9qNlySJN7d27VydMmKBdu3bVlal4kDz++IbXp4Onyy93B5FXXnG18S05E3jjDdX+/V0i+sUv9i+HzAR79rjr/tnZqn37qj7zTMN9Ge9Lf4EG4/POq08EBx7ofh+rVjVcNhXLlVVdeW1OjqueS4Kqqirt2bOnXnLJJUnZfnNYskhja9as0R49eujo0aO1ojU16bFWV6f64x+7Pw8R1dzchsmif/+Gr3v0cMnle99Tffhh1ffeC13BVFGhOm2a+1z79qoLFyb2Z0tFn32mesQRbp+cdZbqxx/H/tJfcOJ5913V//5v93sDl6huusn9LiIl/lQrV37ySfczBO4OT7C5c+cqoC+99FJStt8clizS3Msvvxyyi4Ck3PlZUaF60UW67wxi9eqmv1Fu2eLq3x980B14jjlGNS+vYRIZPNg1Vt96q2uXeOqp+lr5QDWLcWpqVO+6q/6SX+Nqn1mzXBcan37qfidbt0ZXhht8OTG476pOnVQvvdSdHabzWd2nn7qfp6goKZu/4oortFu3blpZWZmU7TdHa5OFdVGeAlJmPO/16+Hcc2HRIrj3XtezayC2QC+t778fuuttVVi7FpYsgc8/r5+WL3c9u4b6jHFOOAHefbd5+yQvD7p3d12nBz/m5Lh9v2SJ2/fB63ziCZg82X023VVWQufOcPvtcNddCd50Jfn5+Zx33nnMnDkzodtuCeui3MTGwoUuUezYAbNmwaRJDd8vKICHHgq/DhHXLfjgwfDtb9fPP+44eO89SwyRFBWF7jZ90SLYvt39fpp6DDxfvx6WLXNJO3i8iGCXX56YnycROnSA4cPd2CUJ9tprr7Fz504uuuiihG87KVpzWpKMqa1dhlLVkJehSNRlmqIid3180CDVxYtjv/6mGkdTqaIm1cSiMXnDBnf5KRP2+eTJqgcdlPDNXnzxxdqrVy+tqqpK+LZbglZehrIReTJZXZ0bueyii9wocAsWwGGHxX47gbOSVatg2jQ3WFC7drHfTlvReH+NGdOydfzpT5mxzwsL3bC5e/cmbJMVFRXMnj2b888/n9zc3IRtN5ksWWSqigqYOtVd573iCvj3v+M/5nQsDoKZJLC/WjM8aybs88JCd9nuiy8Stsk5c+awZ8+ezLkEhbVZpISED6NYVubaJz79FH7zG/jRjxI7pGU07R8mttryPi8sdI9+Pxx6aEI2WVRURH5+PieccEJCtpcK7MwiBZSXl6OqvPzyywC89957qCrl5eWx39jHH8OECe5b2OzZ8OMft+mxj00GOPBAyM5OWCP3rl27mDNnDhdeeCHZ2dkJ2WYqsGSRQnw+HwArV66MzwaeecaVZ7ZvDx98ABMnxmc7xiRS+/YwYkTCksXs2bOprKzMqEtQYMkipQwdOpTs7OzYJ4u6OvjZz+CSS+CII9zZxahRsd2GMclUWJiwZFFUVMSAAQM45phjErK9VGHJIoXk5uYyZMiQ2CaLPXvgwgvhF7+Aa66BefOgT5/Yrd+YVFBYCCUl7ia9ONq2bRuvvvoq3/nOd8jKyqzDZ2b9tGnA5/PFLlksXOhukHvxRfjd7+DRR9tu+aTJbIWF7gx6xYq4bmbWrFlUV1dn3CUoiHOyEJEzRWSFiJSIyIwm3h8kIm+KyKciskREzo5nPOkgkCy0NXc7b9gA55/vLjlt2eLm3XyzNWSbtiu4IiqOioqKGDp0KEcccURct5OK4pYsRCQbeAg4CzgEuFhEDmm02O3Ac6o6FpgK/Cle8aQLn8/H7t27myyljWjDBrjhBhg61J1NGJMpRo50/WHFMVls3ryZefPmcdFFF4Xtz62tiueZxQSgRFVXqWoV8CxwbqNlFOjqPe8GrI9jPGmhVRVRU6fCX/6S0DtZjUkJ7dqBzxfXZPHiiy9SW1ubkZegIL7Joj+wLuh1mTcv2J3AZSJSBswFboxjPGmhVcmiqAiuu859wzIm08S5IqqoqIgDDzyQ0aNHx20bqSxishCRb4tIvJLKxcDfVHUAcDbwf01tS0S+KyILRWTh5s2b4xRKahg8eDA5OTktSxaBu3TPOst1Vd2W+wMyprHCQigthW++ifmqN2zYwFtvvZWxl6AgujOLi4CVIvJrETmoGev+ChgY9HqANy/YNcBzAKr6AdAB6N14Rar6iKqOV9Xxfdp42WdOTg7Dhg1rXUVUWRkcc0zb7g/ImMYKC12/usuXx3zVzz//PKqasZegIIpkoaqXAWOBUuBvIvKB900/0sgpCwCfiAwVkXa4BuzZjZZZC5wCICIH45JF2z51iEKrymdV3ber4cNj0xGdMekijhVRRUVFHHrooRxySOMancwR1eUlVd0JPI9rpO4HTAE+EZGQbQyqWgN8H3gNWIarevKLyN0iEhhZ50fAtSKyGHgGuFJbVTPaNvh8PkpKSlpWPvv117Bzp+v+wJhM4vNBbm7Mk8W6det47733MvqsAqLodVZEzgWuBEYATwATVHWTiHQClgJ/DPVZVZ2La7gOnndH0POlwLdaFHkb5vP5qKioYP369fTv37gmIIKSEvdoycJkmtxcV0Ib42Tx3HPPAViyiGKZ84D/VdV3gmeqaoWIXBOfsDJbcEWUJQtjmqGw0PVcEENFRUWMGzeOERn+PxXNZajyxolCRH4FoKr/jktUGa5V5bMlJZCVBUOGxDYoY9JBYSGsXu0G94qB0tJSFixYkPFnFRBdsjitiXlnxToQU2/gwIG0a9eu5cli0CDXbbMxmSZQEbVsWUxWF7gE9Z3vfCcm60tnIZOFiFwvIp8DB3n9NgWm1cCSxIWYebKzsxk+fHjLk0WGny6bDBbjiqiioiKOPvpoBg8eHJP1pbNwbRZPA68A9wLBnQDuUtWtcY3KtLx8tqQE7FuQyVQjRrgbUWOQLJYvX87ixYt54IEHYhBY+gt3GUpVdQ3wPWBX0ISI9Ix/aJnN5/NRWlpKXV1d9B/atg22brUzC5O5cnLcMKsxSBZFRUWICBdeeGEMAkt/kc4sJgKLcB3+Bd/jrsCwOMaV8Xw+H5WVlZSVlTFo0KDoPlRa6h6HD49fYMakusJC+PDDVq1CVXn22Wc5/vjjOeCAA2IUWHoLeWahqhO9x6GqOsx7DEyWKOKsRRVRVjZrDBxyCKxZA7t3t3gVn3/+OcuXL7cqqCAhzyxE5PBwH1TVT2IfjgkIThannHJKdB8KJIthlstNBgs0ci9b5gYAa4GioiKysrI4//zzYxhYegt3Geq3Yd5T4OQYx2KC9O/fnw4dOjT/zKJ/f+jUKX6BGZPqgiuiWpAsApegTjnlFPr27Rvj4NJXyGShqiclMhDTUFZWFiNGjGh+srBLUCbTDR/eqoqoRYsWsWrVKn7605/GOLD0Fu4y1MmqOl9EzmvqfVW1cTvjzOfzsaw5NxeVlMDEifELyJh0kJMDBx3U4mRRVFRETk4OU6ZMiXFg6S3cZagTgPnAt5t4TwFLFnHm8/mYM2cOtbW1ZGdnh1941y7YuNHOLIwBdynqvfea/bG6ujqKioo444wz6NnT7hAIFu4y1M+9x6sSF44J5vP5qKqqYu3atQwdOjT8woGyWUsWxrhk8cwz7ktUXqShd+p9+OGHrFu3jl/+8pdxDC49RTOsai8R+YOIfCIii0Tk9yLSKxHBZbpmlc9asjCmXqCRe+nSZn2sqKiI9u3bc+6558YhqPQWTUeCz+JGrzsfuMB7XhTPoIzTrGQRKJu1G/KMaVEfUbW1tTz33HOcffbZdO3aNU6Bpa9oxrPop6r3BL3+hYjYnSoJ0K9fPzp37hx9sujbt1mn3Ma0WcOGQYcOzUoW7777LuXl5XYjXgjRnFm8LiJTRSTLm76DGyrVxJmIRF8+a2WzxtTLzm52RVRRURGdOnViolUUNilc6ewu6vuEmg486b2VBewGfhz36Aw+n4/FixdHXrCkBE62+ySN2aewEN5+O+wiBQUFbNy4scG8Ll26kJ+fT3l5eTyjSzvh+obKU9Wu3mOWquZ4U5aq2gW9BPH5fKxevZqamprQC33zDZSV2ZmFMcEKC93/xc6dIRdpnCgizc9k0VyGQkR6iMgEETk+MMU7MOP4fD5qampYs2ZN6IVWrXKPliyMqdfCiijTtGhKZ6cB7+DaKe7yHu+Mb1gmIKqKKOtt1pj9xXjUvEwXzZnFTcARwJdef1Fjge1xjcrsY8nCmBYaOhQ6drRkESPRJItKVa0EEJH2qrocODC+YZmAvn37kpeXFz5ZlJZCz57Qo0fiAjMm1WVlwcEHW7KIkWiSRZmIdAdmAW+IyEvAl/ENywSISOTxuEtK7GY8Y5pSWBg2WeTn5zdrfiaLmCxUdYqqblfVO4GfAY8Bk+MdmKkXVbKwS1DG7K+wEL76CrY3feW8vLycX/ziFwDs3LkTVUVVrWy2CdFWQx0uIj8ADgPKVLUqvmGZYD6fjzVr1lBV1cRur6qCL7+0ZGFMU6KoiPL7/QwePJg86/0grGiqoe4A/g70AnoDM0Xk9ngHZur5fD7q6upYvXr1/m+uWQN1dZYsjGlKFBVRfr+fwsByJqRoziwuBY5Q1Z973ZYfBVwe37BMsLAVUVYJZUxogwe7YYZDJIuamhqWL19uySIK0SSL9UCHoNftga/iE45piiULY1ooKwsOOSRksigpKaGqqsqSRRTC9Q31R1zfUDsAv4i84b0+Dfg4MeEZgF69etG9e/fQySIvD/r0SXxgxqSDwkJ4/fUm3/J7SWTUqFGJjCgthTuzWAgsAv4J/BR4E3gLuA14KZqVi8iZIrJCREpEZEaIZb4jIktFxC8iTzcr+gwRtnw2UAklkvjAjEkHhYWwYQNs27bfW36/HxHh4IMPTkJg6SXcsKp/DzwXkXbASO/lClWtjrRiEckGHsKdiZQBC0RktqouDVrGB9wKfEtVt4lI35b9GG2fz+fjvabGFC4thcMOS3xAxqSL4EbuY49t8Jbf72fo0KF06tQpCYGll2iqoU4EVuIO/H8CvoiyI8EJQImqrvJKbZ8FGo9VeC3wkKpuA1DVTc2IPaP4fD7Wrl1LZWVl/cyaGli92torjAknTEVUcXGxtVdEKZoG7t8Cp6vqCap6PHAG8L9RfK4/sC7odZk3L9hIYKSIvCciH4rImdEEnYl8Ph+qyqpAD7MA69ZBdbUlC2PCGTQIunTZL1lUVVXxxRdfWLKIUjTJIldVVwReqOoXQG6Mtp8D+IATgYuBR72uRRoQke+KyEIRWbh58+YYbTq9NFkRZZVQxkQm0mRF1MqVK6mpqbHG7ShFkywWishfReREb3oU1/gdyVfAwKDXA9i/5LYMmK2q1aq6GvgClzwaUNVHVHW8qo7vk6FVP5YsjGmFJvqIClRC2ZlFdKJJFtcDS4EfeNNSb14kCwCfiAz1GsinArMbLTMLd1aBiPTGXZZahdlPjx496N279/7JomNH6NcveYEZkw4KC2HjRtiyZd8sv99PVlYWBx10UBIDSx8hq6FgX0XT46p6KfC75qxYVWtE5Pu4wZIC6/GLyN3AQlWd7b13uogsBWqBn6jqltBrzWz7lc8GepvNiqqLL2MyV3Aj9/GuPqe4uJjhw4fToUOHMB80AWGTharWishgEWnXks4DVXUuMLfRvDuCnivwQ28yEfh8PubPn18/o6QERo4M/YE0VV1dTVlZWcPKLxM3HTp0YMCAAeTmxqopMgU1kSz8fr+1VzRD2GThWQW8JyKzgT2BmararDMN03o+n48nnniCiooKOnXo4O6xOPvsZIcVc2VlZeTl5TFkyBDEbjaMK1Vly5YtlJWVMXTo0GSHEz8DBkDXrvvaLfbu3UtJSQkXXnhhkgNLH9FcvygFXvaWzQuaTIIFGrlLS0th/XrYu7dNDnpUWVlJr169LFEkgIjQq1evtn8W16giasWKFdTW1lrjdjNEarPoA8zB3Vxn424nWXBF1KE9e7qZbbQSyhJF4mTMvi4shNmuxqa4uNibZckiWiHPLERkGuAH/ggsF5FJCYvKNKlB+ayVzcbV1VdfTd++fe2adltSWAibN8Pmzfj9frKzsxnZBtv84iXcZajpQKGqHg0cg+vDySRRXl4e+fn59ckiNxcGDoz8wTasoKAAEdlvKigoaNV6r7zySl599dUYRWlSQlAjt9/vZ+TIkbRv3z65MaWRcMmiSlU3A6jqKtw4FibJ9pXPlpTAsGGQnZ3skJJq48aNzZofreOPP56egUt9pm1olCzsElTzhGuzGCAifwj1WlV/EL+wTCg+n899492xIyMuQU2fPp3PPvusRZ898cQTm5w/ZswYHnjggVZEZdLSAQdAt27ULF5MaWkpl156abIjSivhksVPGr1eFM9ATHR8Ph8zZ85Ed+5EQhwMjTFNEIHCQr5ZtAhVtTOLZopqPAuTOnw+H/mA7NmTEWcWkc4AwlXyvPXWWzGOxqS9wkJyn3kGsNHxmsv6iUgzPp+PfSkiA5KFMTFVWEiH3bvpn5PDCPv/aRZLFmlmxIgR7LsNrw3ekNdc+fn5zZofrYsvvpijjz6aFStWMGDAAB577LFWrc+kCO/S0xltvXuTOIimuw+TQjp37szYLl2o3bOH7MGDkx1O0pWXl8dlvc94lypMG+Mli2N79EhyIOknYrLw7uK+FhgSvLyqXh2/sEw4h3bsSHl1Nf3btUt2KMakld1dulAFHJbhJectEc2ZxUvAu8A8XDfiJsmGq/JFXd1+Y9QaY8Jbtnw5lcDBe/ZEXNY0FE2y6KSqt8Q9EhMdVQp272ZOdTWH79hBt27dkh2RMWnD7/dTCRyzfj2ounJaE5VoGrhfFpG21w92utq6lQ6VlZTQaIhVY0xEfr+fFTk5ZO/YAXFq72qrokkWN+ESRqWI7PKmnfEOzITgdSBoycKY5isuLmb3oEHuRaMxuU14EZOFquapapaqdvCe56lq10QEZ5pgycKYFvP7/eSMHh14kdxg0kxU91mIyCQRud+bJsY7KBNGSQmIUDNggCWLNmTWrFksXbo04nIPP/wwTzzxRNhlPvvsM+bOnRt2mUy0c+dO1q1bx6Dx46FXL0sWzRQxWYjIfbhLUUu96SYRuTfegZkQSkthwAAGjRxpyaIpGzbADTfA2LHJjiRqNTU1USeL6667jv/6r/8Ku4wli6YF9m/hqFHufgtLFs0SzZnF2cBpqvq4qj4OnAmcE9+wTEglJTBiRH1X5cYJJIlhw+Cxx6CFPdUGrFmzpkHfQffffz933nknJ554IrfccgsTJkxg5MiRvPvuuwDU1tby4x//mFGjRnHYYYfxxz/+EYBFixZxwgknMG7cOM444ww2bNgAuB5xp0+fzvjx4/nVr37F7Nmz+clPfsKYMWMoLS3l0Ucf5YgjjmD06NGcf/75VFRUAHDnnXdy//3371tH41iqqqq44447KCoqYsyYMRQVFeHz+di8eTMAdXV1jBgxYt/rTNJgdLxAslBNclTpI9o7uLsDW73nVquZTCUlMHkyPp+PrVu3snXr1rY97sL06eEP/Hv3wpdfwsaN7h8/+J8/VK+8Y8ZAK7oor6mp4eOPP2bu3LncddddzJs3j0ceeYQ1a9bw2WefkZOTw9atW6murubGG2/kpZdeok+fPhQVFXHbbbfx+OOPA1BVVcXChQsB1/40ceJELrjgAgCo3h4GAAAgAElEQVS6d+/OtddeC8Dtt9/OY489xo033hhVLHfffTcLFy7kwQcfBGD58uU89dRTTJ8+nXnz5jF69Gj69OnT4p8/Xfn9fjp27MjQoUNdstixw41l39/uWIpGNMniXuBTEXkTEOB4YEZcozJN27HDDQvpnVmAO8gceeSRSQ4siZYtc/slgc477zwAxo0bx5o1awCYN28e1113HTk57l+qZ8+eFBcXU1xczGmnnQa4s49+/frtW89FF10UchvFxcXcfvvtbN++nd27d3PGGWdEHUtjV199Neeeey7Tp0/n8ccf56qrrmrWz9tW+P1+DjnkELKyshoMhGTJIjoRk4WqPiMibwFHeLNuUVUrUE6G0lL3mEnJItIZQHk53HMPzJwJtbVQVVX/Xiu6KM/JyaGurm7f68rKyn3PA0NxZmdnU1NTE3IdgTETPvjggybf79y5c8jPXnnllcyaNYvRo0fzt7/9LWR369HEMnDgQPLz85k/fz4ff/wxTz31VMjttmV+v59TTz3VvQhOFqefnryg0kjINgsROch7PBzoB5R50wHePJNoXtksI0YwbNgwsrKyrN2ioAAeeghWrYJp06BjR4hBn1n5+fls2rSJLVu2sHfvXl5++eWwy5922mn85S9/2XfA3rp1KwceeCCbN2/elyyqq6vxh2hUzcvLY9euXfte79q1i379+lFdXd3sg3vjdQFMmzaNyy67jAsvvJDsDOwXadu2baxfv75+wKM+fdxkjdxRC9fA/UPv8bdNTPfHOS7TlECyGD6c9u3bM2jQIEsWAY2TxpgxrVpdbm4ud9xxBxMmTOC0007joIMOCrv8tGnTGDRoEIcddhijR4/m6aefpl27djz//PPccsstjB49mjFjxvD+++83+fmpU6fym9/8hrFjx1JaWso999zDkUceybe+9a2I227spJNOYunSpfsauAEmTZrE7t27M/oSFNBwdDyriGoeVQ07AR2imZeoady4cZqxrrpKtV+/fS9PO+00HT9+fBIDio+lS5cmO4Q2Z8GCBXrssceGfL+t7/OHH35YAV29enX9zO99T7VrV9W6uqTFlUjAQm3FsTea0tmmvgo1/fXIxFdpaYMBjwLls2rlfyaM++67j/PPP597783c26P8fj9dunRhUKCrD3BnFjt3QllZ8gJLI+HaLApEZBzQUUTGisjh3nQi0ClhEZp63j0WAT6fjx07dvD1118nMSiT6mbMmMGXX37Jsccem+xQkqZBJVRAcCO3iShcNdQZwJXAAOB3QfN3AT+NY0ymKXv2uJrwRskCXEVUJtbNGxOt4uJizjmn0b3EwcnizDMTH1SaCZksVPXvwN9F5HxVfSGBMZmmrFrlHkMki2OOOSYZUcWNqiI21kBCtPXLmF9//TWbNm1q2LgNrn+o/Hw7s4hSNPdZvCAi5wCFQIeg+XfHMzDTSFDZbMDQoUPJzs5ucxVRHTp0YMuWLfTq1csSRpypKlu2bKFDhw6RF05TgUqo4O5b9rGKqKhFMwb3w7g2ipOAvwIXAB9Hs3IRORP4PZAN/FVV7wux3PnA88ARqrowutAzTFDZbEBubi5Dhgxpc8liwIABlJWVZWT/RcnQoUMHBgwYkOww4qbJstmAwkJ3Q6eNmhdRNN19HKOqh4nIElW9S0R+C7wS6UMikg08BJyGu5lvgYjMVtWljZbLw/Vq+1Hzw88gJSXQuzd0795gdlvsUDA3N9f132NMDPj9frp27Ur/prr1KCyE3bth7VoYPDjxwaWRaEpnv/EeK0TkAKAad0d3JBOAElVdpapVwLPAuU0sdw/wK6CyifdMQKNKqAArnzUmvOLiYgoLC5u+pGkVUVGLdgzu7sBvgE+ANcAzUXyuP7Au6HWZN28fr9uQgao6J9yKROS7IrJQRBZm7KWJMMli9+7dbNy4MQlBGZPaVBW/39/0JSiwZNEM0Qyreo+qbvcqogYDB6nqz1q7YRHJwpXk/iiKGB5R1fGqOj4jS0T37oV16xq0VwQEV0QZYxoK9O/VZOM2QI8e0K+fJYsoRDNS3ve8MwtUdS+QJSI3RLHur4CBQa8HePMC8oBRwFsisgY4CpgtIuOjjD1zrF7tGuBCnFmAJQtjmhK2cTvAKqKiEs1lqGtVdXvghapuA66N4nMLAJ+IDBWRdsBUYHbQenaoam9VHaKqQ4APgUlWDdWEJspmAwYPHkxOTo4lC2Oa0GB0vFAKC2HpUgjqkt7sL5pkkS1BLUNelVPEPqBVtQb4PvAasAx4TlX9InK3iExqacAZKUyyyMnJYdiwYZYsjGmC3++nR48eFBQUhF6osBAqKtyIiyakaEpnXwWKROQv3uv/9uZFpKpzgbmN5t0RYtkTo1lnRiopgW7d3B2nTWiL5bPGxILf72fUqFHhb+4MbuS2ku2QojmzuAV4E7jem/4N/E88gzKNBCqhQvzB+3w+SkpKrHzWmCARK6ECDjnEPfr9sGED3HADjB0b/wDTTDTdfdQBf/YmkwwlJTA+dLu/z+ejoqKC9evXN33jkTEZaMOGDWzfvj1ysuje3Q2e9fjjcOedru0ieHheA4Tvovw57/FzEVnSeEpciBmuuhrWrGmyvSLAKqKM2V9UjduBM4lNm+CLL6Cy0hJFCOHOLKZ7jxMTEYgJ4csvobY26mRx4oknJigwY1JbVGWzU6fCf/5jlVBRCNdmERih/heq+mXjKRHBGdzoeNDkDXkBAwcOpF27dnZmYUwQv99Pnz596Nu3b+iFiorguuugffvEBZamwiWLdiJyCXCMiJzXeEpUgBkvTNlsQHZ2NsOHD7dkYUyQqBq3CwrgoYfcpd7u3SErC9pFvDMgI4VLFtcBxwHdgW83muzSVKKUlECnTu6POgwrnzWmXtSVUAEFBXD99e75ZZfBmDHxCy5NhRsp7z/Af0Rkoao+lsCYTLAIZbMBPp+P119/nbq6uobjDBuTgdatW8euXbuiTxYAU6bAvffCCSfAY3bIayxcNdTJ3tNtdhkqiUL0NtuYz+ejsrKSsrKyBARlTGoLOzpeKOPHQ//+MGtWnKJKb+G+gp7gPTa+BGWXoRKlttaNvR1lsgArnzUGoqyEakwEJk+GV1913X+YBkImC1X9ufd4VRPT1YkLMYOVlbma7yiSxciRIwFLFsaASxYFBQX07NmzeR+cMgW++QbeeCM+gaWxaLoov0lEuorzVxH5REROT0RwGS+KSqiAAw44gI4dO1qyMIb60fGa7fjj3RgX//xn7INKc9G0hF6tqjuB04FewOXAfXGNyjiBZBHmHouArKwsRowYYcnCZLy6ujqWLl3asmSRmwsTJ8K//gU1NbEPLo1FkywCZThnA0+oqj9onomn0lJ3s9CAAVEtbuWzxsCXX35JRUVF8xq3g02eDFu3wrvvxjawNBdNslgkIq/jksVrIpIH2L3xiVBSAsOGuRuFouDz+Vi1ahW1tbVxDsyY1NWixu1gZ5wBHTpYVVQj0RyFrgFmAEeoagWQC1wV16iME2XZbIDP56Oqqoq1a9fGMShjUlugA8FDAl2PN1fnznD66S5ZWLf/+0STLI4GVqjqdhG5DLgd2BHfsAyqLUoWYBVRJrP5/X769+9P9+7dW76SKVNg7Vr49NPYBZbmokkWfwYqRGQ08COgFHgirlEZ13XyN99YsjCmmQKj47XKxInu8q9VRe0TTbKoUTcE27nAg6r6EJAX37BMc8pmAwoKCujSpYslC5OxamtrWbZsWcvbKwJ693ZltNZusU80yWKXiNwKXAbMEZEsXLuFiacWJAsRsfJZk9FWr15NZWVl65MFuKqo4uL6/8UMF02yuAjYC1yjquXAAOA3cY3KuD/QnBwYNKhZH7PyWZPJohodL1qTJ7tHO7sAokgWqlquqr9T1Xe912tV1dos4q2kBIYMcQmjGXw+H6tXr6bGbigyGShQNtviSqhggwfD2LHWbuGJpruPo0RkgYjsFpEqEakVEauGirfS0mZdggrw+XzU1NSwZs2a2MdkTIrz+/0MHjyYvLwYNatOmQIffADl5bFZXxqL5jLUg8DFwEqgIzAN+FM8g8p4LSibDbCKKJPJmjXgUTQmT3b/j7Nnx26daSqqW4NVtQTIVtVaVZ0JnBnfsDLc11/Dzp2WLIxphpqaGpYvXx7bZDFqlOubzdotokoWFSLSDvhMRH4tIjdH+TnTUi2ohAro06cPXbt2tWRhMk5JSQlVVVWxTRYi7lLUv//tvsBlsGgO+pcD2cD3gT3AQOD8eAaV8VqRLETEKqJMRmrR6HjRmDzZjSszd25s15tmIpbaqOqX3tNvgLviG44BXLLIynLVUM1QUFDAxo0b970Wb9zu/Px8yq2BzrRxfr8fEeHggw+O7YqPOgry892lqKlTY7vuNBIyWYjI50DIXrRU9bC4RGRcshg0yHVP3gzBiSKa+ca0JX6/n6FDh9KpU6fYrjg7GyZNgmefhb17m/1/2VaEO7OwcbaTpYWVUMZkshaPjheNKVPg0Udh/nw466z4bCPFhWuzyAUGqOqXwRPuDu7m3SlmmqekJKrR8YwxTlVVFV988UX8ksXJJ0NeXkbfoBcuWTwANNX8v9N7LyIROVNEVohIiYjMaOL9H4rIUhFZIiL/FpHB0YXdhm3b5kbpsjMLY6K2cuVKampqYt+4HdC+PZx9Nrz0EmTo4GLhkkW+qn7eeKY3b0ikFYtINvAQcBZwCHCxiDS+B/9TYLzX/vE88Oso4267SkvdoyULY6LW6tHxojF5MmzaBB9+GL9tpLBwySLcyCEdo1j3BKBEVVepahXwLK6b831U9U1v9D2AD3GXuDJbK8pm8/Pzm5yfnZ3Nnj17WhOVMSmtuLiYrKwsDjrooPht5OyzITc3Y2/QC5csForItY1nisg0YFEU6+4PrAt6XebNC+Ua4JWm3hCR74rIQhFZuHnz5ig2ncYCyWLYsGZ/tLy8HFVtMM2fP5+6ujqmT58e40CNSR1+v5/hw4fToUOH+G2ka1c45RTXbpGBw62GSxbTgatE5C0R+a03vY07qN8UyyC84VrHE6Lrc1V9RFXHq+r4Pn36xHLTqaekBPr3hxiV/5100knMmDGDv/71rzz//PMxWacxqSYmo+NFY8oUd6nYu+yVSUImC1XdqKrH4G7EW+NNd6nq0d64FpF8hbvbO2CAN68BETkVuA2YpKp7ow+9jYpD2exdd93FEUccwbXXXsu6desif8CYNLJ3715KSkri214RMGmS6wIkA6uiohnP4k1V/aM3zW/GuhcAPhEZ6vUtNRVo0HWjiIwF/oJLFJuaE3ibFYdkkZuby9NPP01NTQ2XXXYZtRlazWHaphUrVlBbW5uYZFFQAEcfnZHtFnHrEFBVa3D9Sb0GLAOeU1W/iNwtIpO8xX4DdAH+ISKfiUhm9wO8axds3BiXeyxGjBjBQw89xDvvvMO9994b8/UbkywxHR0vGpMnwyefwJdfRl62DYlr77GqOldVR6rqcFX9pTfvDlWd7T0/VVXzVXWMN00Kv8Y2btUq9xinstnLL7+cSy65hDvvvJMPPvggLtswJtH8fj/Z2dmMHDkyMRsMDLf60kuJ2V6KsK7GU0krymajISL86U9/YuDAgVxyySXs2GEDHpr05/f7GTlyJO0T1WeTzweFhRnXbmHJIpUEkkXnznDDDW783xjr1q0bTz/9NOvWreOGG25AM7AE0LQtMR8dLxpTpsA778CWLYndbhJZskglS5ZAhw4wejQ89hh89llcNnP00Udz55138vTTT/Pkk0/GZRvGJEJFRQWlpaXJSRZ1dfCvfyV2u0lkySIVbNjgziQCXSBXVrrBVuLo1ltv5bjjjuOGG26gNNDFiDFpZvny5ahq4pPF2LFuGIEMqoqyZJEKpk6Fhx9231QSdFkoOzubJ598kpycHC655BKqq6sTsl1jYiluo+NFIuIaul97DTKkKx1LFqngmWegXz/3PDc3YZsdNGgQjz76KB9//DE///nPE7ZdY2LF7/eTm5vLiGR0vDl5srsK8Nprid92EliySAVPPw3r18PvfgfXXgsdO0K7dgnZ9AUXXMC0adO47777ePPNNxOyTWNixe/3c+CBB5KbwC9Z+xx3HPTsmTGXoiTdqmHGjx+vCxcuTHYYsbN4MRxxBEycCC+84E5vy8vhnnvg/ffh00/jHsKePXs4/PDD2bNnD4sXL6ZXr15x36YxLdV4rPmApIw1f+WV7n6LTZsSelWgJURkkaqOb+nn7cwimSor4dJLoVcveOQRlyjAdSnw0EMJSRQAnTt35plnnmHTpk1MmzbNymlNSkupseanTIHt210ZbRtnySKZbr3V9V45cyb07p3UUA4//HDuu+8+Zs2axSOPPJLUWIxJG6ed5i4bZ8ANepYskuWNN+CBB+D734czz0x2NABMnz6d008/nZtvvpmlS5cmOxxj9tm2bRvPPvssl19+ebJDaahTJ/f/O2tWmx/jwtoskmHLFjjsMOjWDRYujNnYFbFQXl7OYYcdRr9+/fjoo4/iO5iMMSGoKsXFxcydO5c5c+bw/vvvU1tbS69evdgS5q7ppBzPnngCrrgCPv7YtT+mKGuzSDeqcN11sHkzPPlkSiUKcI2HM2fOZMmSJcyYMSPZ4cRNQUEBIrLfVFBQkOzQ2qxI+7yiooKXX36Z66+/niFDhnDYYYcxY8YMdu3axYwZM3j//feT0y4RycSJkJ3d9quiGg/DmerTuHHjNK39/e+qoHrvvcmOJKxOnTopsN+Un5+ftJjy8/NjFlNT6wlMyYopFSVqn5955pnavn17BbRz5846efJkffTRR7WsrCyuMcXMySerHnxw8rYfBWChtuLYmxO3LGT2t3q1a6M47jj4yU+SHU1YFRUVTc5vyTe7WJU6hquC+fzzz6mpqaG6upqampp9U/Dr4OfhvPDCC7Rr147c3Nx9U/Dr4OexrMyJZUloIvZ5JNXV1ezevZtdu3axa9eusMuWlpZy/fXXc84553DccceF7UE24eWx0ZgyBW68EVasgAMPTHY0cWFtFolSWwsnnACff+7urRgyJNkRhSWBMt4mXHHFFXTt2nW/qVu3bk3ObxfmBsPPP/+cbdu2sXXrVrZt29bgeeN5K1eujMePGjdHHnkk3bt3p0ePHnTv3n3fFOp13759Q66ruf+n4X5/gXWpKtXV1VRUVPDNN9/sm4Jfnxmm+OKKK65okAwaP9+7N/pRktPtOLSfdetcX1H33Qe33JLsaJrU2jYLO7NIlF//Gt57zzWGpXiiiOStt95i586d7Nixg7q6ulat69BDD91vnojQo0ePfVPPnj0ZMmRI2GTxj3/8g9zcXHJycsjJyQn5PPDa5/OFXNfixYupqqqiurp63xTq9bRp00Kup1u3bmzdupVVq1axfft2tm3bFvGsJpRRo0aRlZWFiDR4bGpeuEQB0KdPn33JoDW/v/nz55OXl0deXh5dunQhPz9/3/PA/ODXF110UYu3lfIGDoTx4127RYomi9ayM4tEWLQIjjoKzjvP9Swb4Z85FUT7zbSiooKdO3c2mHbs2NHg9c9+9rOQ6yoqKqJnz54NEkPXrl3Jytq/9iKamKIVq3U1Zz2B/bV9+/Z907Zt2/Y9v/HGG0Ou67zzzkNVqaur2++xqXnhum65/vrr6dix476pU6dODV4Hz/vWt74V9c8XSSx/fynpl7+E22+Hr76CAw5IdjT7ae2ZhSWLeKuogMMPh9273XgVPXsmO6KopPuBOZJYXdNPxf0Uy3Wl4j5PWUuXuhH0/vxnV/GYYqx0NtX95Ceu0evvf0+bRAHuH7g58xMhljGVl5c3WfHR3INWKu6nWErFfZ6yDj4YRo6sv5s7ME5NHEa8TAZrs4inuXPhT3+Cm2+GU05JdjTNEst/4Pz8/JDfKJsjFQ8qqbifYrmuVNznKSswxsVvfwvXXON6k66ri/tAZoliySJeNm+Gq6+GUaPg//2/ZEeTVHbAiU4s95Pt8yTYsAFWrnSVjzNnNuz+o7LSDZmcxixZxIOqG5di2zZ4/fW0/yMxxkRh6lT4z3/c88btOXl57jLVmDHustSYMW7q0SPyejdscEMWfPBBwnqibooli3h47DHXx/3997s+oIwxbV9RkTuoz5zpzi6CLz/dcos70P/73/B//1c/f/Dghglk7FhXhitSnyRmzkyJy1lWDRVrJSXulz5hAsybB02UgBpj2rDA4GXBSSP4OLtpE3z2mUsegccvvqhfpnt312fcxo0uaQTfm9OK47WVzqaSmho49lhX/bRkifuGYIzJTM0Z8XLPHte7w6efwt13u882ZeNGCHOnfzhWOptKfvlL+OgjePhhSxTGZLrmjHjZubO7cff6693yN9zgBlVqPFRrv36u26A//MF1MZJAlixi5aOP3LeISy+FttytgTEmvgJJZtUqVyjTsSME+le7/XY3Hs5NN7m+qCZMcP1RffFF3MOyy1CxsHs3HHoofP01DB3qLkEZY0wsNHU5a8UKd/Pfiy/CggVu3qhRrkuh885zhTWN7r63NovWam1Z2oYN7oa7ZcsgJ8e1W6TZPjXGpLG1a+sTx7vvuuPP8OH1iWPCBNi4kWEHHLB5lWrLGjxI12Txr3+1vu64qbK0TZtg1y437d5d/7yp1xs3wiefwJdfus8HS7N9aoxpIzZuhNmz4YUXXJluTY2rrNq7l3G1tbpItcVND3FNFiJyJvB7IBv4q6re1+j99sATwDhgC3CRqq4Jt87xffvqwl276g/wmze7g/fOnQ2nUPM2b3ZnAZs3t+yg3q4ddOniqhdC9ddvycIYk2zbt7tG8xUrABgPLFRtcZfXcbspT0SygYeA04AyYIGIzFbVpUGLXQNsU9URIjIV+BUQvnV48+aGr/v0iRSIu3syLw+6doWyMpc0mvKHP9QvG5i6dGn4OtDQ1FQttTHGpIru3eGtt9xx6vHH0crKVn2LjduZhYgcDdypqmd4r28FUNV7g5Z5zVvmAxHJAcqBPhomqPEi2qDF4ve/d0kgeAokhq5dXUla8I1x4Q7yLdkXkW7AMcaYZCsvZ2i/fptXp2KbhYhcAJypqtO815cDR6rq94OWKfaWKfNel3rLfN1oXd8FvgtwOIxbFPxmS+OP9UG+OTfgGGNMgmXETXmq+oiqjlfV8dKnT8O645YKrmWeNs110RGL9VmiMMa0QfFMFl8BwbcxD/DmNbmMdxmqG66hO7RBg2J3gAc7yBtjTBTi2evsAsAnIkNxSWEqcEmjZWYDVwAfABcA88O1V+wTOMAbY4xJiLglC1WtEZHvA6/hSmcfV1W/iNwNLFTV2cBjwP+JSAmwFZdQjDHGpJi4jmehqnOBuY3m3RH0vBK4MJ4xGGOMab20aOA2xhiTXJYsjDHGRGTJwhhjTERp15GgiOwCViQ7jkZ6A19HXCqxUjEmSM24LKboWEzRS8W4DlTVvJZ+OK4N3HGyojV3IcaDiCy0mKKTinFZTNGxmKKXinGJSKvGdrDLUMYYYyKyZGGMMSaidEwWjyQ7gCZYTNFLxbgspuhYTNFLxbhaFVPaNXAbY4xJvHQ8szDGGJNgaZUsRORMEVkhIiUiMiNJMQwUkTdFZKmI+EXkJm9+TxF5Q0RWeo89khBbtoh8KiIve6+HishH3v4qEpFW9uve7Hi6i8jzIrJcRJaJyNHJ3k8icrP3eysWkWdEpEMy9pOIPC4im7wxXQLzmtw34vzBi2+JiByewJh+4/3+lojIP0Wke9B7t3oxrRCRMxIVU9B7PxIRFZHe3uuk7Sdv/o3evvKLyK+D5sd9P4WKS0TGiMiHIvKZiCwUkQne/ObvK1VNiwnXGWEpMAxoBywGDklCHP2Aw73necAXwCHAr4EZ3vwZwK+SENsPgaeBl73XzwFTvecPA9cnOJ6/A9O85+2A7sncT0B/YDXQMWj/XJmM/QQcDxwOFAfNa3LfAGcDrwACHAV8lMCYTgdyvOe/CorpEO9/sD0w1PvfzE5ETN78gbhOSr8EeqfAfjoJmAe09173TeR+ChPX68BZQfvnrZbuq3Q6s5gAlKjqKlWtAp4Fzk10EKq6QVU/8Z7vApbhDkLn4g6OeI+TExmXiAwAzgH+6r0W4GTg+WTEJCLdcH+8jwGoapWqbifJ+wl3b1FHceOndAI2kIT9pKrv4HpaDhZq35wLPKHOh0B3EemXiJhU9XVVrfFefogblyYQ07OquldVVwMluP/RuMfk+V/gf4DgRtek7SfgeuA+Vd3rLbMpKKa476cwcSnQ1XveDVgfFFez9lU6JYv+wLqg12XevKQRkSHAWOAjIF9VN3hvlQP5CQ7nAdw/T533uhewPegfPdH7ayiwGZjpXRr7q4h0Jon7SVW/Au4H1uKSxA5gEcndT8FC7ZtU+du/GvdtFJIYk4icC3ylqosbvZXM/TQSOM67nPm2iByRAjEBTAd+IyLrcH/7t7Y0rnRKFilFRLoALwDTVXVn8HvqzvMSVmYmIhOBTaq6KOLCiZODOyX+s6qOBfbgLq3sk4T91AP3jWoocADQGTgzUdtvjkTvm0hE5DagBngqyXF0An4K3BFp2QTLAXriLun8BHjOO7tPtuuBm1V1IHAz3pl+S6RTsohmmNaEEJFcXKJ4SlVf9GZvDJzGeY+bQn0+Dr4FTBKRNbjLcycDv8edWga6dEn0/ioDylT1I+/187jkkcz9dCqwWlU3q2o18CJu3yVzPwULtW+S+rcvIlcCE4FLvSSWzJiG45L9Yu/vfQDwiYgUJDEmcH/vL3qXdT7GneH3TnJM4EYiDRyj/kH9JbBmx5VOyWLfMK1etcpU3LCsCeV9W3gMWKaqvwt6KzBELN7jS4mKSVVvVdUBqjoEt1/mq+qlwJu44WqTEVM5sE5EDvRmnQIsJYn7CXf56SgR6eT9HgMxJW0/NRJq38wG/surYDkK2BF0uSquRORM3OXNSapa0faShsIAAATFSURBVCjWqSLSXtzQyT7g43jHo6qfq2pfVR3i/b2X4QpOyknifgJm4Rq5EZGRuIKOr0nSfgqyHjjBe34ysNJ73vx9FY9W+XhNuBb8L3AVBbclKYZjcZcHlgCfedPZuDaCf3u/jHlAzyTFdyL11VDDcH+YJbhvFe0THMsYYKG3r2YBPZK9n4C7gOVAMfB/uCqVhO8n4Blcu0k17oB3Tah9g6tYecj7u/8cGJ/AmEpw17YDf+sPBy1/mxfTCryKm0TE1Oj9NdRXQyVzP7UDnvT+rj4BTk7kfgoT17G4drnFuLbVcS3dV3YHtzHGmIjS6TKUMcaYJLFkYYwxJiJLFsYYYyKyZGGMMSYiSxbGGGMismRh0oqI1Ho9aBaLyL+Ce0ENsfwQEbkkivU2WE5ExovIH2IU8xoR+dyblorIL0SkQyzWbUyiWLIw6eYbVR2jqqNwnaZ9L8LyQ4CIyaLxcqq6UFV/0NIgm3CSqh6Ku4N2GPCXWKw06M5zY+LKkoVJZx/gdX7m3Yn6G++M43MRuchb5j5cB2+fiRvLYoiIvCsin3jTMSGWO1HqxwXpKSKzvH7/PxSRw7z5d3pjCLwlIqtEJGJyUdXdwHXAZBHp6a3nJyKywFv/XYFlReRn4sZA+I+4sTd+7M1/S0QeEJGFwE0i0kdEXvDWsUBEvuUt19mL72OvM8eE99Js2g77VmLSkohk47rrCHSMdh7ujvHRuD55FojIO7jOC3+sqhO9z3UCTlPVShHx4e56Hd/EcicGbe4u4FNVnSwiJwNPeNsCOAjXzUMesEJE/qyu36mQVHWniKzGdV/TDdcFxATcXbWzReR44BvgfO/nycXdFRzcUWQ7VR3vxfo08L+q+h8RGYQb5+Fg3J3D81X1au9y3cciMk9V90TcwcY0YsnCpJuOIvIZ7oxiGfCGN/9Y4BlVrcV1yPc2cASws9Hnc4EHRWQMUIvrWjqSY3EHblR1voj0EpHAGAFz1I1hsFdENuG6FS+LYp2BHklP96ZPvdddcMkjD3hJVSuBShH5V6PPFwU9PxU4ROo7Oe0qrlfk03EdTP7Ym98BGITbb8Y0iyULk26+UdUx3hnCa7g2i+Y0RN8MbMR9Y88CKlsZz96g57VE8T8lInm4NpIvcEnjXlX9S6NlpkdYTfDZQRZwlJdYgtchwPmquiJSTMZEYm0WJi2p6wH1B8CPvEbed4GLxI1D3gc3St/HwC7ct/SAbsAGVa0DLscN10sTywV7F7gU9l2e+lobjWESLe8b/5+AWaq6DZfwrvbmIyL9RaQv8B7wbXFjhHfBdREeyuvAjUHbCFwiew240UsaiMjYlsRsDNiZhUljqvqpiCwBLsb1+Hk0rndNBf5HVctFZAtQKyKLgb/hDtQviMh/Aa9S/w19SaPlPg3a1J3A4962KqjvRrw53vQO2lnAP4F7vJ/hdRE5GPjAO6bvBi5T1QUiMtuLayOuZ9AdIdb9A+AhL74c4B1cI/o9uBEUl4hIFm788XBJx5iQrNdZY1KUiHRR1d3eJbd3gO+qN/67MYlmZxbGpK5HROQQXMP03y1RmGSyMwtjjDERWQO3McaYiCxZGGOMiciShTHGmIgsWRhjjInIkoUxxpiILFkYY4yJ6P8DCoZAO5KGtcgAAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX8AAAA1CAYAAACp8OvZAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAADTxJREFUeJzt3V1MHGUXB/Czy7J8G1pWQitWUk3TIIloGkIoNpC0EmKkkqgtCQWJRiBsmxKCXJDQxA80VWsN0WqRi4oGmoJKiL1zbdwUTWmxSmi1dRvshu62tWLBrmVndv7vBZl5l9qPmWcQoXN+SS+62Tl75tln/vOx06kNADHGGLMW+3/dAGOMsYXH4c8YYxbE4c8YYxbE4c8YYxbE4c8YYxbE4c8YYxbE4c8YYxbE4c8YYxbE4c8YYxbE4c8YYxbkWODPW5BnSSiKQna7uf2aLMvkcJgbnkgkQjExMaZqMMbuHpIkUWxs7L/9MTY9b7orj/ztdjtNTk6aquFwOCgUCpmqMR/BH4lETNdgjJknSZLpGgsQ/LotuvB/7bXX6ODBg6ZqnDx5kjZt2kTHjx8XDs9QKERJSUm0a9cu4T52795NNpuNjh8/LlwjJiaG9u7dS+Pj40QktjMIh8NzlpVlWbgfFT8QkC0ERVH+8zpXrlyhF198kS5evCj82cFgkGpra2nTpk00MzMj3Munn35q+qqGatGF/wMPPEAHDhwwVSMrK4tOnDhBfX19QiEFgBITEyk7O5va29uFelAUhZ599lmqq6ujhoYGCgaDQr1cvHiRDh8+TBUVFfThhx8KnU04nU4iInrllVdoeHiYHA6H0MYgSZK2A7HZbLwDYP8KRVG0uWW3202Fpd/vp8cff1woMNW53traSl1dXfT7778L9aAoCvX09NCXX35JpaWl2vZoFAA6cOAArVy5Umj5mxZcwD93VF5ejt7eXj1vvS0igsfjMVWjrq4Os0MkLhQKgYjw1VdfIRwOC9eprq6Gw+EAAMzMzAjVaGtrg8vlMjUuRIS4uDiUlpYCgOF1UnsfGRnBt99+K9wHm1+SJCESiUBRFMPLRiIRRCIR7e+nTp3C0aNH57ymtw4AjI+Po7i4GE6nE42NjYbrALPrk5WVBSJCRUWF4eUVRUEoFMILL7wAIsJDDz2kvW5UOBwGESE5OdnwstG2bdsGIkIwGLzTW3Xl8aIK/5GRETidToRCoTuPxC0oioLLly+DiDA6OgpZloXqyLKMiooKU+GvThQiQk1NjdDEUZfp7OwU7kUdg8uXL+O9997DM888A0VRhHZGg4ODyM3NndOLJEmGaoTDYSQmJuKee+5BX1+f9pqRfhRFgSRJCIVC8Pv92vpIkiQ0zkuFum6yLCMSiRge++g6N4b29PS0qd6++OILNDU1weVyGZ6r6nr19/ejqKgINHtzCM6cOSPUy6FDh0BEcLlcOHv2rOHlJUmC1+tFTk4OiouL4fP5hOfVxMQEiAh5eXnC3xcwG/42m03PW5de+Dc3N+tdudsaGxsDEaGlpUX4KBkAurq6kJiYiNHRUVNf2tNPP236DEKWZfT09ICI8NZbbwGA8JGVekQ0PDwsVAcAent7QUSIj4/H1atXARg7Krp69SrOnDkDm82m7Ry//vpr3ctLkoSamhosX74caWlpICK43W54vV5cunRJe5+edTtx4gR6e3uxefPmeTnrvJmpqSn8+eef/3j9dv2pO+3oOXzu3DmcP38eo6OjePvtt/HRRx+hs7MTIyMjuvpQa4VCIRw5cgT79u3DihUrtLCNj4/HH3/8ccc66vbg9/vhdru1A4LMzExs375dd+Cq6//DDz+gvb0ddrsdRARJkhAOh4XmZn5+PogI9fX1AIwfrUciEUxNTYGIkJCQgOvXrxvuQf3ciYkJlJWVoba2FoFAQGh9VDab7e4N/8rKynkJ/7Nnz4KIUFJSInypJRKJoLu7G6tWrTK9E6mrq0NSUhI6OzuFawDA8PCwtpGqPYrYu3cv0tPTkZqaKtxLOBzWjtBWr16NgYEB3ctGn411dHRg3bp12np1d3djampK187W5/NhZGQEHo8HVVVVWo2YmBh0dHRgYmJCVz8JCQnahmWz2ZCenq792bBhA44dO6b7DHJ4eBgtLS3weDwYHR1FS0sLSkpKUFhYiNraWnR2dqKhoQH9/f0IBAK6agLQahUUFCA/Px+5ubnIyMgAESElJUW7HHCzOaGGX/RZZGFhoTZeashlZGTA4/EYOlvesGGDVuPJJ5+cs+PVQ/2eCwsLkZCQgPLycuzevRuA8UuKADAwMAAiQmxsrPCZTDgcRnd395xLRiJH/bIso6OjAytXrjS0fdzKXR3+69evNx3+0ZdaCgsLTdUCgKqqKhQUFMypLdJPQ0MDHnvsMRw7dkyoD3WDbGpqAhFhcHBQqI5qcnISDocD7e3tAMR3JKdPn0ZycjKICL/++qtwrcHBQezcuVMLEq/XCwC3DSL1GrU6xn///TfefPNN1NbWanWKioqwZcsWQ718//33aG5uRmVlpTYn1TMUm82G7u5u7Wwn2tTUFHJyclBfX4+tW7fi4Ycf1sL6qaee0kI7Pj4eSUlJc3bkN65nMBiEw+HQ3pOYmIhVq1ahqqoKra2t+Oyzz7B//354vV50dXXhwoULt5yfgUAAhw8fRk1NDYgISUlJWLduneE5FN2j2+0GEWHt2rXw+Xza60YCe2ZmBoFAAGvXrtUOsgDjlxHVZVpaWkBEePnllwGIzUO1fyLCzp07DS+vUr+LmJgYOJ3OOa+JstlsqKys1PPWpRf+BvZst6ROUCJCbW2tqVoA0Nraivz8fNN1BgcHQUQ4cuSIcNBKkgSfz4fU1FS43W4AYhNcHaOysjKkpaXh6NGjQrXUybxnzx7ExsaivLwchw4dMtxP9EZRUVGBhIQE5OTk6Plh65YGBgbm7AT27Nlj6EhbNTMzg7GxMWzduhWpqanaHL3xwCL68sXk5CSA2d9YxsbG5lwC+fnnnzE9PY33338fdXV1aGxs/MdnXrt2DXl5ecjOzobb7UZFRQW6urrQ3d1tuH8AyMvLQ0JCAogITqcTdXV1+OabbwD8f+zvFLjq+3788UdtLjudTnzwwQe6lr+VJ554QrtEMzk5KVRHDez4+HgQEU6fPi3US3QtswdY6noQEZYtWwbg9gcyenD430H0oO/atcvUNTYAGBoawurVqwGI77mjz0b27dtnuqf+/n7k5uZiaGjI1I5kZmYGdrsdJSUlwr2o4+31ehEbG4s1a9YAMD5W0Rt9W1sbiAjV1dXw+/2611FRFMiyrH12IBBAXl4ekpOTkZ6ejpdeeslQTzcKBoMoKyvTPU+jx0DtLdqNP7hG27JlC4aGhm56/V0dK7Xe7ULll19+ARFh/fr1ePXVV+HxeLRLmCJB++CDD8LhcCAzMxM9PT1CdSRJwrVr1+DxeEBEyMrKwvj4uOFegLljrJ7p3fi6UeodeleuXBGuEb0TycnJATA/4d/c3KznrUsz/Ldt26Zn5W7rp59+QkpKClpbW00P+P79+5GRkQFA/NKIioiwYsUKUzWia6WlpZmu8/zzz4OI8PHHH5uu9fnnn8PlcqG8vBznz58XqqFutD6fTztqF92Qo++Geeedd7Bx40YAYteS/wvRl7REybKMc+fOafVE5/D169dRWVmJxsZG7TKPmd7uvfdeuFwu+P1+4Rrq55eWloKI0Nvba+q7VRQFfX19yMrKQkdHh3AdVTAYBBHh4MGDpmuFQiE4nU69P+wvrfBXB2o+wn96ehp2ux0NDQ2mawUCARCRqeBXA0g99Y5+zSh1cmdmZpq+g0i9NTI3NxfFxcVaSIjWmp6e1n4Efv31102FNgDU19ebHi+VoijCd20sVWbH7EbvvvuudknFzPYgyzIcDgfa2tpM1wKAuLg45Obmmqqh9pCdnY24uDhTtYDZ+dbU1ISqqirTB6DA7N115eXlet++tML/vvvuw8aNG+flHu3nnnsO999/P0ZHR01PrK6uLqSkpJjuCYB2d8b09LTpvrZv3w4iwqlTp0z35ff7QUTYsWOHqTpq2HR0dODkyZOm7pBii0t0gM3HNvrdd9/NSy1ZluflBghVYmLivPxW6Pf7UVBQoPsW3DspLS3FJ598ovftuvJ40Tze4cKFC1RdXU02m64H0t2Soii0fPlyevTRR8nlcpl+DkZ2djb99ddfpmoQzf5T8c2bNxMRUXJysqm+IpEI7dixg9asWWP6AXaKolBmZiY1NjbSG2+8YeoZKOpTUN1uNz3yyCPC/4ydLT7RjxUxu42Gw2HKz8+fl1oxMTHk9XqpqKhoXp4DJMsypaSkmK6TlpY2r/O/urqafvvtt3mrR0RkAxb0+SwL8mE+n48yMzMpLi7OdC0AdOnSJUpPTzc9USORCE1NTdGyZctM96Waj8dXM8Zmt3Wz2/gioWsl7srwX+z4Of+M3d0AkCRJ/9XZL4c/Y4xZkK7wX+j/yeuuOKdijLGlji8WM8aYBXH4M8aYBXH4M8aYBXH4M8aYBXH4M8aYBXH4M8aYBXH4M8aYBXH4M8aYBXH4M8aYBXH4M8aYBXH4M8aYBXH4M8aYBXH4M8aYBXH4M8aYBXH4M8aYBXH4M8aYBXH4M8aYBXH4M8aYBXH4M8aYBXH4M8aYBXH4M8aYBXH4M8aYBXH4M8aYBf0PwIACOHRGfV4AAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "rotating_image_classification(digit_one, sess3, prob3, X3, keep_prob3, u3)" ] }, { "cell_type": "markdown", "metadata": { "colab": { "autoexec": { "startup": false, "wait_interval": 0 } }, "colab_type": "code", "collapsed": true, "id": "x1G0RMxw3SWj" }, "source": [ "## Using Negative Log of the Expected Likelihood (Eq. 3)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In this section, we repeat our experiments using the loss function based on Eq. 3 in the paper." ] }, { "cell_type": "code", "execution_count": 221, "metadata": { "colab": { "autoexec": { "startup": false, "wait_interval": 0 }, "base_uri": "https://localhost:8080/", "height": 454 }, "colab_type": "code", "collapsed": true, "id": "qZeZ8M2-3U2o", "outputId": "ba07af58-3193-44a1-bcfe-e0a699affbdc" }, "outputs": [], "source": [ "g4, step4, X4, Y4, annealing_step4, keep_prob4, prob4, acc4, loss4, u4, evidence4, \\\n", " mean_ev4, mean_ev_succ4, mean_ev_fail4 = LeNet_EDL(exp_evidence, loss_EDL(tf.log), lmb=0.001)" ] }, { "cell_type": "code", "execution_count": 225, "metadata": { "colab": { "autoexec": { "startup": false, "wait_interval": 0 }, "base_uri": "https://localhost:8080/", "height": 1291 }, "colab_type": "code", "collapsed": true, "executionInfo": { "elapsed": 460, "status": "ok", "timestamp": 1527923232289, "user": { "displayName": "Murat Sensoy", "photoUrl": "https://lh3.googleusercontent.com/a/default-user=s128", "userId": "102692943223630372304" }, "user_tz": -180 }, "id": "aVltdhRR5dNG", "outputId": "fab9365b-f64b-4e47-de90-e48e6eda4aab" }, "outputs": [], "source": [ "sess4 = tf.Session(graph=g4)\n", "with g4.as_default():\n", " sess4.run(tf.global_variables_initializer())" ] }, { "cell_type": "code", "execution_count": 226, "metadata": { "colab": { "autoexec": { "startup": false, "wait_interval": 0 } }, "colab_type": "code", "id": "ZnmB0--c351F" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "epoch 1 - 100%) training: 0.8389 (47.7661 - 14.2396) \t testing: 0.8481 (48.1106 - 14.5359)\n", "epoch 2 - 100%) training: 0.9120 (98.9960 - 8.6988) \t testing: 0.9194 (99.6993 - 8.8808)\n", "epoch 3 - 100%) training: 0.9304 (180.1434 - 7.9925) \t testing: 0.9390 (182.9537 - 7.6880)\n", "epoch 4 - 100%) training: 0.9454 (357.4596 - 8.9753) \t testing: 0.9515 (353.7061 - 7.7063)\n", "epoch 5 - 100%) training: 0.9546 (374.0661 - 6.4429) \t testing: 0.9585 (369.2362 - 6.1300)\n", "epoch 6 - 100%) training: 0.9579 (717.8140 - 7.7555) \t testing: 0.9627 (698.8102 - 7.6518)\n", "epoch 7 - 100%) training: 0.9617 (820.1666 - 5.8483) \t testing: 0.9646 (819.3505 - 5.0063)\n", "epoch 8 - 100%) training: 0.9646 (685.4586 - 4.7092) \t testing: 0.9682 (700.0871 - 4.2215)\n", "epoch 9 - 100%) training: 0.9676 (864.0211 - 4.7806) \t testing: 0.9700 (882.0850 - 4.2597)\n", "epoch 10 - 100%) training: 0.9699 (1145.9790 - 4.9029) \t testing: 0.9728 (1166.5842 - 4.3104)\n", "epoch 11 - 100%) training: 0.9698 (1265.3774 - 3.9921) \t testing: 0.9730 (1349.9486 - 3.4476)\n", "epoch 12 - 100%) training: 0.9730 (1586.4016 - 5.1049) \t testing: 0.9747 (1696.7997 - 4.9364)\n", "epoch 13 - 100%) training: 0.9735 (2014.4788 - 5.1080) \t testing: 0.9763 (2115.8186 - 4.7999)\n", "epoch 14 - 100%) training: 0.9735 (2741.4673 - 4.3296) \t testing: 0.9752 (2957.9802 - 3.9254)\n", "epoch 15 - 100%) training: 0.9752 (2673.9426 - 4.4678) \t testing: 0.9772 (2692.9707 - 4.9921)\n", "epoch 16 - 100%) training: 0.9768 (2388.1035 - 4.0882) \t testing: 0.9781 (2634.2166 - 3.7764)\n", "epoch 17 - 100%) training: 0.9764 (2701.2002 - 4.6162) \t testing: 0.9791 (3023.1316 - 5.1292)\n", "epoch 18 - 100%) training: 0.9773 (2878.8640 - 4.0546) \t testing: 0.9792 (3105.1746 - 3.6160)\n", "epoch 19 - 100%) training: 0.9768 (3326.2048 - 4.1775) \t testing: 0.9787 (3591.4688 - 4.3881)\n", "epoch 20 - 100%) training: 0.9788 (3257.1694 - 4.1882) \t testing: 0.9791 (3451.4497 - 3.3292)\n", "epoch 21 - 100%) training: 0.9788 (4058.1035 - 4.2671) \t testing: 0.9801 (4339.0176 - 4.3674)\n", "epoch 22 - 100%) training: 0.9795 (4905.1646 - 5.0746) \t testing: 0.9806 (5299.4648 - 4.9685)\n", "epoch 23 - 100%) training: 0.9789 (4146.0679 - 4.8996) \t testing: 0.9794 (4280.2456 - 4.3978)\n", "epoch 24 - 100%) training: 0.9789 (5102.7090 - 5.0765) \t testing: 0.9811 (5573.3687 - 4.9651)\n", "epoch 25 - 100%) training: 0.9797 (4721.9268 - 3.4639) \t testing: 0.9824 (4908.5527 - 3.7816)\n", "epoch 26 - 100%) training: 0.9794 (4624.8179 - 3.5023) \t testing: 0.9810 (4622.0835 - 4.1510)\n", "epoch 27 - 100%) training: 0.9803 (7247.1953 - 4.8597) \t testing: 0.9823 (7369.5410 - 5.0344)\n", "epoch 28 - 100%) training: 0.9820 (6480.2974 - 4.0996) \t testing: 0.9825 (7157.7944 - 4.7370)\n", "epoch 29 - 100%) training: 0.9813 (7673.8716 - 4.6725) \t testing: 0.9817 (7892.9888 - 4.8215)\n", "epoch 30 - 100%) training: 0.9819 (7318.6362 - 4.3854) \t testing: 0.9841 (7933.3677 - 4.6528)\n", "epoch 31 - 100%) training: 0.9825 (8063.1187 - 4.8635) \t testing: 0.9827 (8506.7451 - 4.2365)\n", "epoch 32 - 100%) training: 0.9816 (6621.0513 - 3.1235) \t testing: 0.9819 (7588.6938 - 3.7367)\n", "epoch 33 - 100%) training: 0.9826 (10533.4658 - 4.9056) \t testing: 0.9843 (11875.2090 - 5.4827)\n", "epoch 34 - 100%) training: 0.9833 (10538.7793 - 4.8260) \t testing: 0.9830 (11016.7715 - 4.3061)\n", "epoch 35 - 100%) training: 0.9830 (9445.5898 - 4.3326) \t testing: 0.9830 (10311.8994 - 4.2545)\n", "epoch 36 - 100%) training: 0.9824 (9012.2568 - 3.6567) \t testing: 0.9831 (10201.7939 - 3.7273)\n", "epoch 37 - 100%) training: 0.9838 (8231.3916 - 3.4212) \t testing: 0.9848 (9213.5693 - 4.3222)\n", "epoch 38 - 100%) training: 0.9835 (9698.7676 - 3.6955) \t testing: 0.9844 (12237.3604 - 4.4046)\n", "epoch 39 - 100%) training: 0.9813 (12806.3682 - 3.5367) \t testing: 0.9833 (13110.8662 - 4.3908)\n", "epoch 40 - 100%) training: 0.9834 (12758.0078 - 3.7434) \t testing: 0.9852 (14355.3516 - 5.7558)\n", "epoch 41 - 100%) training: 0.9841 (18760.6660 - 4.6124) \t testing: 0.9847 (18847.6367 - 4.9546)\n", "epoch 42 - 100%) training: 0.9844 (14055.1133 - 4.2950) \t testing: 0.9850 (15240.1768 - 5.1640)\n", "epoch 43 - 100%) training: 0.9839 (17531.1875 - 4.8606) \t testing: 0.9856 (20936.6172 - 6.0384)\n", "epoch 44 - 100%) training: 0.9842 (11528.1709 - 4.0646) \t testing: 0.9857 (12673.5449 - 4.8653)\n", "epoch 45 - 100%) training: 0.9838 (13236.9697 - 4.4013) \t testing: 0.9848 (14658.7588 - 6.3498)\n", "epoch 46 - 100%) training: 0.9841 (16241.7793 - 3.5962) \t testing: 0.9871 (18014.9746 - 4.5276)\n", "epoch 47 - 100%) training: 0.9851 (15670.2637 - 5.8087) \t testing: 0.9855 (17604.8535 - 5.6529)\n", "epoch 48 - 100%) training: 0.9855 (14058.3115 - 4.6745) \t testing: 0.9865 (15463.1846 - 5.6835)\n", "epoch 49 - 100%) training: 0.9854 (16627.9102 - 4.3638) \t testing: 0.9851 (18772.2539 - 4.5671)\n", "epoch 50 - 100%) training: 0.9860 (18039.0078 - 3.9769) \t testing: 0.9875 (20124.0410 - 4.8621)\n" ] } ], "source": [ "bsize = 1000 #batch size\n", "n_batches = mnist.train.num_examples // bsize\n", "L4_train_acc1=[]\n", "L4_train_ev_s=[]\n", "L4_train_ev_f=[]\n", "L4_test_acc1=[]\n", "L4_test_ev_s=[]\n", "L4_test_ev_f=[]\n", "for epoch in range(50): \n", " for i in range(n_batches):\n", " data, label = mnist.train.next_batch(bsize)\n", " feed_dict={X4:data, Y4:label, keep_prob4:.5, annealing_step4:10*n_batches}\n", " sess4.run(step4,feed_dict)\n", " print('epoch %d - %d%%) '% (epoch+1, (100*(i+1))//n_batches), end='\\r' if i" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "draw_EDL_results(L4_train_acc1, L4_train_ev_s, L4_train_ev_f, L4_test_acc1, L4_test_ev_s, L4_test_ev_f)" ] }, { "cell_type": "code", "execution_count": 228, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEKCAYAAADjDHn2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJztnXl4nFX1xz8ne7rvSTeaNA1CUSi0AhWEgoLsIKAUFAUsyE5ZBFREoC5gyyIIKrsgmyxCLYvAj0UQEFooBcqSSZt0S7rvtM12fn/cd5pJmpl5J5k1cz7Pc593v++ZOzPv9733nnuuqCqGYRiGEYmcVBtgGIZhpD8mFoZhGEZUTCwMwzCMqJhYGIZhGFExsTAMwzCiYmJhGIZhRMXEwjAMw4iKiYVhGIYRFRMLwzAMIyp5qTYgVgYNGqRlZWWpNsMwDCOjmDNnzipVHdzZ6zNOLMrKypg9e3aqzTAMw8goRKS2K9dbM5RhGIYRFRMLwzAMIyomFoZhGEZUTCwMwzCMqCRMLETkXhFZISIfhzkuInKriAREZJ6I7JUoWwzDMIyukciaxf3AYRGOHw5Ueuks4M9+Mp0zZw4igohQWlraZSMNwzCM6CRMLFT1P8CaCKccCzygjneAfiIyNJZ7LF++vCsmGoZhpB2lpaXbX4hDU2dejkPzAsZ3xa5UjrMYDiwO2V7i7auLJZO6ujoGDx5MXp6/j1JaWtqhyJSUlFBfXx/LrQ3DMOJOuJfgSC/HqkpDQwPbtm1rk+L5Qp0Rg/JE5CxcU9UODBs2DBFh8ODBlJSUUFpa2mEKHuvMF9ERJjqGYYTSlWdCQ0MDixcvprY28ri5sWPH7iAI27Zto6GhoUu2+yGVYrEUGBmyPcLbtwOqeidwJ4CIaOixP//5z9TX17dJX3zxBfX19Wzbti0mg6677jr69u0bMRUUFACdU3/DMLovkZ4JW7duZdGiRdTW1lJTU0NNTc329draWpYuXYqqdnh9KLvtthuFhYW+049//OO4fT7xY2CnMxcpA2ap6lc7OHYkcD5wBLAPcKuq7u0jzzYGh7NfVVm/fj3Lly9vIyRTp06N/YOEUFRURN++faNWCQ3DyC68fgFf5ObmMnLkSMrKyhg1ahRlZWXb1w8++OCw18X6bGlvk6r6N7J9Xol6sInII8AkYBCwHPg1kA+gqn8R9yn+hPOY+hI4XVWjBn0KFYvONPlE+kIbGxvZsGED69evj5ruvPPOsPmYWBhG9hHp2XLddde1EYZhw4aF7WeNlE+3FItEMWHCBO1KIMF4fRHx/EKNTlJXB9OmwdtvwwcfpNoaI8uJ1zMhnv2h7fPqilhk3QjukpKSmPYbaUhdHZx7LoweDffcA3Pnxi/PPffsel6GP7pRmcfzBbG+vh5V3SF1xnEmNC9gTlfsyghvqHgSL0+lkpKSsOpvRKGzNYJFi+CXv4THH4fmZmhqaj0WCEBRERQXty5zfLwLBW257z5oaYGuepVYbSc68S7zNODRRx8Ne6y7PBOyTiziRXvRuf/++zn99NO56667UmRRBhDuIbF1K9TXw7Jl7pxgar+9cmX4vCsrd9xXUNAqHKEiUlwMIrB4MSxZ4s5taWm97oknYMAAlwYOdMsePdw1sX42o5VAAH7yE3jjDSfkzc2ptigu1NfXc/7557PPPvvw3//+l9zc3FSblBCyrs8iUTQ2NrLLLrvQv39/3nvvvZg8I7otqrB2LcyeDTffDP/3f+5BGvqQ6N/fndOe3FwoLYWhQ1tT794ur7ffdnk3Nrae/8ADTnS2bHEpuN5+GVx/913YsMH/Zyko2FFABgxw+z/4wCXVtrWdDPtvJYyqKrjjDrjttvACkaFlpaocf/zxPP/888ydO5dddtkl1SaFRUTmqOqEzl5vNYs4kZ+fzy9+8QumTJnC888/zxFHHJFqk+JLR80rLS1uf21tx2nRIti0KXK+p5zSVhCCafDg8M1I9fWtb/HNze4t/tRTY/s8HeURZN48WLPGpdWrW9dDtxcuhDlzYOnSjH3QJZTmZnjhBfjTn9wyLw+OPtrVzl54AbZta1uby1AeffRRnn76af7whz+ktVDEhY46UtI5jR8/XtOVhoYGHTVqlO6zzz7a0tKSanPiw2efqR59tGp+vmpuriqoTpqkOnq02+cela1pwADVPfdUPe441YsuUr3pJtW771Y98UTVoiLVgoK253eFujrVc89VHTeu63kUF7faFuv1Z53lPluwfOLx2TKV1atVZ8xwvw9QHTpU9dprVZctaz2nrk717LPbllcGUldXpwMGDNB9991Xm5qaUm1OVIDZ2oVnb8of/rGmdBYLVdW//vWvCugLL7yQalO6xvz5quPH7ygGoDpxourkyapXXKF6xx2qzz6r+vHHqhs3Rs6zqw/mRNJV4QleX1jYWk733RdXE9OaDz5QnTLFfbeg+s1vqj72mGpDQ/hrampUR4xw5//zn8mzNQ60tLTocccdp4WFhfrpp5+m2hxfmFikGdu2bdORI0fqxIkTM7N2sXq16lVX7fiWHM835njUCNKVujrV009X7dXLldVPf6q6dWuqrUoM27apPvqo6v77u89aXOxqWR9+6D+PjRtV99nHvTy88kribI0zDz30kAI6ffr0VJviGxOLNOSOO+5QQF966aVUm+Kf1atVf/lL1d693c/iyCNVv//9trWADG4ySDqNjaqXX+7Ka599VBcvTrVF8WHZMtUf/Ui1tNQ1MYFqRYVrblyzpnN5rl6tuttuTmDfey++9iaAYPPTxIkTM6L5KYiJRRqydetWHT58uO6///7pX7tYtUr1F79oFYkTT1SdN6/1eDo3HWUCTzzhHoKDB2fUm/MOLF3q+qFCa5xHHKH63HOqzc3xyb+sTHXgQNcEmqa0tLToscceq0VFRfrZZ5+l2pyYMLFIU2677TYF9JV0fUCsXKn685+7B5mI6ve+11Yk2tOdm44Szaefqu6yi2pOjur06arp/gIRyiefqO63n/uNxLs5sj1VVaolJa4fo7Y2/vnHgb///e8K6IwZM1JtSsyYWKQpW7Zs0aFDh+qkSZNSbUpbVq5UvfJK1Z493QPg+99X/eijVFvV/dmwQfWEE1prbxs2pNqi8LS0uFrQKad0LBKJbI6cO1e1b1/VnXdWXb48MffoJMuWLdP+/ftnXPNTEBOLNOaWW25RQF9//fVUm6K6YoXzXgqKxOTJzoPJSB4tLa5mkZOjuuuursaRTtTVqV5/veqYMe7R0K+f66xPdt/Vm2+6++21l+r69Ym7Twy0tLToMccck5HNT0FMLNKYzZs3a0lJiX7rW99KjQHLlqmedprqkCGtInHyya5pwUgdr7zi+jB691Z98snU2tLU5Podvvtd1bw890g44ADVBx9U/fLL1vOS3Xf13HPOngMPbGtHisjk5qcgJhZpzowZMxTQN998M3k3DYpE8M8PrkkhjTsOs45Fi1T33tt9N5df7rynkkltreqvf606cqSzYfBg1csuc4MwI5HMvquHH3YvOEcfHXm8RoIJNj994xvfyMjmpyAmFmnOpk2bdPDgwXrooYcm/mbLlqmec07bgWHm7pq+bN3qxmGA6sEHu6bCRLFsmbtXWZnq4Ye7h7CI6qGHqj7+uBszkY7ccYcrn1NPjY/XVYyENj99/vnnSb9/PDGxyABuuOEGBfTtt99O7I0OOMC1hyerM9KID/fe6wR+5EjV//2vdX9Q/LvyFr9smatVhtYyhw9X/dWvVBcu7LLpSWHaNGf3hRcm3ZPswQcfVEBvvPHGpN43EXRVLCzqbBLYtGkTZWVl7LPPPjz77LOJu1EwON6dd7aNfgoW7C7def99OP54F5jxN79xgQpDw523tLiIuRs2RE8bN7rfwvvvu4CO7b/7xkYX2C9TUIVLL3WRi6+9Fq6+Oim3raurY7fddmPs2LG8/vrrGR96vKtRZ1NeU4g1ZWLNQlX1d7/7nQL67rvvJv5mRx+t2qePDaTLND7+uDVWUvsUKfxKaMrPVx00yAU2TKbLa6Jpblb98Y+d/bfdlvDbtbS06NFHH90tmp+C0MWaRQa9XmQ25513HtOnT2fatGnMnDkzsTdbuhQmToT773c1jbfeSuz9jPhw7rluwqeOuOIK6NMneiosdOdHCsGeieTkwN13w7p1cMEFbi6RU05J2O3+/ve/869//YubbrqJnXfeOWH3ySi6ojSpSJlas1BVnTZtmgI6Z86cxN2kpcUNajrvvMTdw0gMHbmndrU20N3CtWzZ4kLk5+Wpzprl9sWjbyeEpUuXar9+/XS//fbLaO+n9mAd3JnDunXrtF+/fnrccccl7iYrV7qv9aabEncPI7Ek4gHfncK1rF/vwucXFqoee2zbeVK6SEtLix511FFaXFysX3zxRRyMTR+6KhY+ZrQ34kXfvn2ZOnUqTz/9NB9++GFibhIIuOWYMYnJ30g8paVw++2wYAFMmQLjxsUvz+Ash5nM5s3wta+5prVnnnHT5Mapme3BBx9k1qxZ/O53v6Oyo3ndsxjzhkoya9eupaysjEMOOYQnnngi/jd46CH44Q9h/nzYddf4528YqebAA+HNNzueljXG51lpaSnLly/fYX9JSQn19fWdtTAt6ao3lNUskkz//v258MILefLJJ/n444/jf4NAwM1zXF4e/7wNIx147DE4+2woKupyVh0JRaT92YyJRQq4+OKL6dWrF9OmTYt/5oEAjBgRlz+SYaQlwSa1hQuhoAByc93SSCgmFilgwIABXHDBBTz++OPMnz8/vplXV1t/hZEdlJbC7rvD/vvHr2/HCIuJRYq45JJL6NGjB7/5zW/im3EgABUV8c3TMNKVigpYsqT7dN6nMSYWKWLQoEGcd955PProo3z22WfxyXTDBli50moWRvZQUeFCmrQPb2PEHROLFHLppZdSXFzMb3/72/hkWF3tliYWRrZQUeGEYtGiTl1eUlIS0/5sxsQihQwZMoRzzjmHhx9+mKqqqq5nGBxjYc1QRrYQ/K0HX5RipL6+nn/9618AvP3229sHoHU3t9l4kFCxEJHDRORzEQmIyJUdHN9JRF4VkQ9EZJ6IHJFIe9KRyy67jIKCAn73u991PTMTCyPbGD3aLTspFsD2FzUbhBeZhImFiOQCtwOHA2OBk0VkbLvTrgL+oap7ApOBOxJlT7pSWlrK2WefzYMPPkh1F37wgPvDlJRA797xMc4w0p3hw13wxC78dwKBAP369WPAgAFxNKz7kciaxd5AQFUXqGoD8ChwbLtzFOjjrfcFwoTc7N489NBDNDc3M2bMGERkeyotLY0tI/OEMrKNnBw3AHXBgk5nUVVVtf2/Z4QnqliIyNEi0hlRGQ4sDtle4u0L5RrghyKyBHgOuCCMDWeJyGwRmb1y5cpOmJLehPtMMY8itTEWRjZSUdHlmoU1QUXHjwicBFSJyB9EZJc43/9k4H5VHQEcATzYkTCp6p2qOkFVJwwePDjOJnQTtmxx/uYmFka2ERSLTsS5a2hooLa2ljH2v4lKVLFQ1R8CewLVwP0i8rb3ph+tYXwpMDJke4S3L5SfAP/w7vM2UAQM8mm7EUqwGm7NUEa2UVEBmza5MUYxsnDhQlpaWqxm4QNfzUuqugF4AtfvMBT4LvC+iHTYbOTxHlApIuUiUoDrwG4/Rdwi4FsAIrIrTiy6XztTMrAxFka20gX32YDnQWg1i+j46bM4VkT+CbwG5AN7q+rhwB7ApeGuU9Um4Hzg38CnOK+nT0TkOhE5xjvtUuBMEfkQeAQ4TTMtZnq6YPNYGNlKF8Qi6DZrYhEdP3NwHw/crKr/Cd2pql+KyE8iXaiqz+E6rkP3XR2yPh/Yz7+53ZOSkpKwMfV9EwhAv35ubmLDyCbKy11Y/k7WLPr27cugQdb6HQ0/zVD17YVCRG4AUNX/S4hVWUZ9fT2qSnBSpyeffDL2UaTmCWVkK4WFLix/J8XC3Gb94UcsDulg3+HxNsRoHUHaqdAfgYCJhZG9dNJ9tqqqyjq3fRJWLETkHBH5CNjFC8URTAuBeckzMXvo06cPQ4YMiV0sGhtd5E3zhDKyldGjYxaLhoYGampqrL/CJ5H6LB4Gngd+D4TGddqoqmsSalUWU1lZGbtY1NZCc7PVLIzspaICli93LrS9evm6pKamhpaWFhMLn0RqhlJVrQHOAzaGJETEelETxJgxY2IXC/OEMrKdYK06hrAfQbdZa4byRySxeNhbzgFme8s5IdtGAqisrKSuro7Nmzf7v8iizRrZThfEwmoW/gjbDKWqR3nL8uSZYwTfcgKBAHvssYe/i6qroUcPNyexYWQjnRhrUVVVRZ8+fbAQQv4IKxYislekC1X1/fibY4R6RPkWi2C0WXP/M7KV/v1dikEszG02NiJ1cN8Y4ZgCB8fZFoPWKnFM/RaBAOy6a4IsMowMIUb32aqqKsaPH59Ag7oXkZqhDkqmIYajd+/elJaW+heL5mbXTnv00Yk1zDDSnYoKmO2vO7WxsZGamhomT56cYKO6D5GaoQ5W1VdE5PiOjqvqU4kzK7uJyX126VJoaLDObcOoqIAnn4SmJsiLHMmotrZ2+4Rjhj8ileiBwCtAR6+sCphYJIjKykqeffZZfyeb26xhOCoqnFAsWtQ6N3cYLIBg7ERqhvq1tzw9eeYY4MRi+fLlbNiwgT59+kQ+2UKTG4Yj1CMqiljYGIvY8ROifKCI3Coi74vIHBH5o4gMTIZx2Uqo+2xUAgHIz3eB1AwjmwkKhI9O7qqqKnr16sWQIUMSbFT3wU8gwUdxExKdAJzorT+WSKOynWDV2JdYBN+icnMTbJVhpDnDh7sItD7EIjjvtrnN+sePWAxV1WmqutBLvwFimGjBiJWY3Gct2qxhOHJy3NwWPsXC+itiw49YvCgik0Ukx0vfx81+ZySInj17MmzYsOhiodo6IM8wDPdfiBLyo6mpiYULF5pYxEgk19mNOK8nAaYCf/cO5QCbgMsSbl0W48t9dsUK2LzZahaGEaSiAl5/3b1IhWliqq2tpampyTq3YyRszUJVe6tqH2+Zo6p5XspR1SguOkZX8SUW5jZrGG2pqHBhyleuDHuKuc12Dj9zcCMi/YFKoCi4r/1Uq0Z8qaysZOXKlaxfv56+fft2fJJFmzWMtoS6z4bxdDK32c7hx3V2CvAfXD/Ftd7ymsSaZfiaYrW62nXqlZUlxyjDSHd8RJ8NBAL07NmTkhLz04kFPx3cFwFfB2q9eFF7AusSapXhTywCARg1CgoKkmSVYaQ55eWuryKCWFRVVVm02U7gRyy2qupWABEpVNXPgK8k1iyjwntDiioW1gRlGK0UFroBqlFqFtYEFTt+xGKJiPQDngZeEpFngNrEmmUUFxczcuTIyAPzqqutc9sw2hMhVLm5zXaeqB3cqvpdb/UaEXkV6Au8kFCrDCDKfNxr1rhkP3rDaEtFBcya1eGhRYsW0djYaDWLTuCnZoGI7CUiFwK7A0tUtSGxZhkQxX02+OZkzVCG0ZbRo2H5cudC2w6bd7vz+PGGuhr4GzAQGATcJyJXJdoww4nF6tWrWbt27Y4HLdqsYXRM8AWqg5HcNsai8/ipWfwA+Lqq/toLW74vcGpizTIgikdUsC8jSihmw8g6IrjPBgIBevTowdChQ5NsVObjRyyWETIYDygElibGHCOUqGIxbBj06JFkqwwjzYlQswgGEDS32diJFBvqNlxsqPXAJyLykrd9CPBucszLbkaPHo2IdCwW5gllGB3Tv79LHdQsqqqq2G233VJgVOYTyRsqOPP5HOCfIftfS5g1RhuKiorYaaedwtcsjjgi+UYZRibQgftsc3MzCxYs4LjjjkuRUZlNpGlV/xZcF5ECYGdv83NVbfSTuYgcBvwRyAXuVtXrOzjn+7jwIQp8qKqn+LY+C+jQI2rzZqivN08owwhHRQXMnt1mV9Bt1jq3O4cfb6hJQBVwO3AH8IWIHODjulzvmsOBscDJIjK23TmVwM+B/VR1N1wodCOEysrKHQfmmSeUYUSmogJqa6GpafsuCyDYNfx0cN8IHKqqB6rqAcB3gJt9XLc3EFDVBd64jEeBY9udcyZwu6quBVDVFf5Nzw4qKytZu3Ytq1evbt1pockNIzIVFU4oFi3avsvGWHQNP2KRr6qfBzdU9Qsg38d1w4HFIdtLvH2h7AzsLCL/FZF3vGarHRCRs0RktojMXhkhTn13pMMpVm1AnmFEpgP32aqqKoqLi81ttpP4EYvZInK3iEzy0l20dn53lTzcPBmTgJOBu7w4VG1Q1TtVdYKqThg8eHCcbp0ZdOg+GwjAoEEQbp4Lw8h2OhCLoNtsTo6vwBVGO/yU2jnAfOBCL8339kVjKTAyZHsEO47PWALMVNVGVV0IfIETD8Nj9OjR5OTk7CgWVpU2jPAMG+Yi0LarWVgTVOeJKBZeJ/W9qnqTqh7vpZtVdZuPvN8DKkWk3POmmgzMbHfO07haBSIyCNcsFXm29SyjoKCAUaNG7dgMZU1QhhGenBw3t4UnFkG3Wevc7jwRo86qarOIjBKRgliDB6pqk4icj5tZLyg6n4jIdcBsVZ3pHTtUROYDzcDPVHV1+Fyzkzbus9u2uU67bvyG1NjYyJIlS9i6dWuqTckKioqKGDFiBPn5froiM4iQsRZLliyhoaHBahZdwM8c3AuA/4rITGBzcKeq3hTtQlV9Dniu3b6rQ9YVuMRLRhgqKyt55513UFVk4UJQ7dZisWTJEnr37k1ZWZmFZUgwqsrq1atZsmQJ5eXlqTYnvlRUwOuvg6oFEIwDfvosqoFZ3rm9Q5KRJCorK9mwYQMrV67MCk+orVu3MnDgQBOKJCAiDBw4sHvW4ioqXJjylSttjEUciFizEJHBwLO48RI273aKCPWIGpIlYyxMKJJHty3rEI+oqqoqioqKGDZsWGptymDC1ixEZArwCXAb8JmIHJM0q4w2BMUiEAg4T6jevZ3rrJEwzjjjDIYMGcJXv/rVVJtidJYQsTC32a4TqeSmArup6kTgG7iwHEYKKCsrIzc317W7BqPNdte3wRgpLS1FRHZIpaWlXcr3tNNO44UXbPbgjKa83P1PQsTC6DyRxKJBVVcCqOoC3DwWRgrIz8+nvLzciYWNsWjD8uXLY9rvlwMOOIABAwZ0KQ8jxRQWwogRaCBAdXW1iUUXidRnMUJEbg23raoXJs4soz1jxoxhwRdfwMKFcMIJqTYnaUydOpW5c+d26tpJkyZ1uH/cuHHccsstXbDKyBgqKmj49FO2bdtmndtdJJJY/Kzd9pxEGmJEprKyksX/+Y8LjmZvSIbhj4oKeOopwNxmu4qv+SyM1FNZWcmnX37pNrLoRx+tBhDJk+e1116LszVGxjF6NIVr19ITc5vtKuYakCFUVlayXSK68RgLw4gr3n9ll/x8hg9vH/TaiAUTiwyhsrKSCqApP98FSTMAKCkpiWm/X04++WQmTpzI559/zogRI7jnnnu6lJ+RIjyx+EZJibnNdhE/4T6MNGDUqFFUirCqb19K7Ue/nfr6+oTk+8gjjyQkXyPJeGKxp4Xz7zJRxcIbxX0mUBZ6vqqekTizjPbk5eWxS34+i/Pz6doIAsPIHlr69mUd8JU8ey/uKn5K8BngDeBlXGRYIxW0tDCqqYl/NDXx9VTbYhgZwtKlS6kHRjbEFDTb6AA/YtFDVa9IuCVGZOrqKGpp4f316zlVtfvG8zGMOBIIBFgO7LZ+fapNyXj8NH7PEpEjEm6JERkv2uz8hgbq6upSbIxhZAZVVVVUA8UrVrgxSkan8SMWF+EEY6uIbPTShkQbZrTDizYboN183IZhhCUQCLAoLw9panKThhmdJqpYqGpvVc1R1SJvvbeq9kmGcUYIgQCal8ciTCwMwy9VVVVsCbqah8zHbcSOLx9METlGRGZ46ahEG2V0QHU1lJWRW1BgYtENefrpp5k/f37U8/7yl7/wwAMPRDxn7ty5PPfccxHPyRYCgQC5O+/sNkwsukRUsRCR63FNUfO9dJGI/D7RhhntCASQMWMYPXq0iUUk6urg3HNhzz1TbYlvmpqafIvF2WefzY9+9KOI55hYOFpaWqiurmbg177mItCaWHQJPzWLI4BDVPVeVb0XOAw4MrFmGW1Q3R6avLKy0sSiI4IiMXo03HMPdDJSbZCampo2Ex/NmDGDa665hkmTJnHFFVew9957s/POO/PGG28A0NzczGWXXcZXv/pVdt99d2677TYA5syZw4EHHsj48eP5zne+s905YdKkSUydOpUJEyZwww03MHPmTH72s58xbtw4qqurueuuu/j617/OHnvswQknnMCXXlywa665hhkzZmzPo70tDQ0NXH311Tz22GOMGzeOxx57jMrKSjclL+4BOmbMmO3b3Zlly5axZcsWKior3dwWJhZdwu9IlX7AGm/dhkImm9WrYcMGqKigsqCAl19+mZaWluwIXzB1auQH/7ZtUFsLy5c7UVVtPRYmRDnjxkEXQpQ3NTXx7rvv8txzz3Httdfy8ssvc+edd1JTU8PcuXPJy8tjzZo1NDY2csEFF/DMM88wePBgHnvsMX75y19y7733AtDQ0MDs2bMB17Z+1FFHceKJJwLQr18/zjzzTACuuuoq7rnnHi644AJftlx33XXMnj2bP/3pTwB89tlnPPTQQ0ydOpWXX36ZPfbYg8GDB3f682cKbebdrqgwsegifsTi98AHIvIqIMABwJUJtcpoS8i825VFRWzZsoVly5YxYsSI1NqVDnz6KSTZh/74448HYPz48dTU1ADw8ssvc/bZZ5PnjRQeMGAAH3/8MR9//DGHHHII4GofQ4cO3Z7PSSedFPYeH3/8MVdddRXr1q1j06ZNfOc73/FtS3vOOOMMjj32WKZOncq9997L6aefHtPnzVSCYjFmzBgnFq+95l4mbIxSp4gqFqr6iIi8BtsHDl+hqokJyGN0TKhYFBcD7k00K8QiWg2gvh6mTYP77oPmZggdqduFEOV5eXm0tLRs3966dev29cJCN2lkbm4uTRF891WV3XbbjbfffrvD4z179gx77WmnncbTTz/NHnvswf333x823LofW0aOHElJSQmvvPIK7777Lg+edAiAAAAgAElEQVQ99FDY+3YnqqqqKCgoYOTIkU4sNm+GlSthyJBUm5aRhG3HEJFdvOVewFBgiZeGefuMZFFd7d6Gysu3x+S3fguP0lK4/XZYsACmTIHiYigo6HK2JSUlrFixgtWrV7Nt2zZmzZoV8fxDDjmEv/71r9sf2GvWrOErX/kKK1eu3C4WjY2NfPLJJx1e37t3bzZu3Lh9e+PGjQwdOpTGxsaYH+7t8wKYMmUKP/zhD/ne975Hbm5uTPllKoFAgNGjR7vPGwzrb01RnSZSo/cl3vLGDtKMBNtlhBIIwMiRUFjIiBEjKCwsNLFoT3vRGDeuS9nl5+dz9dVXs/fee3PIIYewyy67RDx/ypQp7LTTTuy+++7ssccePPzwwxQUFPDEE09wxRVXsMceezBu3DjeeuutDq+fPHky06dPZ88996S6uppp06axzz77sN9++0W9d3sOOugg5s+fv72DG+CYY45h06ZNWdMEBe6FavvseCYWXUdVIyagyM++ZKXx48dr1rHvvqoHH7x9c+zYsXrsscem0KDEMn/+/FSb0O147733dP/99w97vLuVeUtLixYXF+vFF1/sdmzdqiqies01qTUshQCztQvPXj/uNB29CnX8emQkhurqNrPjmfusEQvXX389J5xwAr//ffYMj6qrq2PLli2tNYvCQhgxwmoWXSBSn0WpiIwHikVkTxHZy0uTgB5JszDb2bDBdcqFzLtdWVlJdXV1mw5YwwjHlVdeSW1tLfvvv3+qTUkawZepMaHz1Zv7bJeI5A31HeA0YARwU8j+jcAvEmiTEUrwx91OLLZt28bixYsZNWpUigwzjPSlzRiLIBUVEMVRwQhPWLFQ1b8BfxORE1T1ySTaZIQSdJtt1wwF7u2pu4qF2pwdSUNDBzJ2E6qqqsjPz3dus0FGj3aDNzdtgl69UmdchuIn6uyTInKkiFwuIlcHk5/MReQwEflcRAIiEnYgn4icICIqIhNiMT4riCAWwben7kZRURGrV6/ulg+xdENVWb16NUVFRak2Ja4EAgHKy8u3D5IEWv9DCxakxqgMx88c3H/B9VEcBNwNnAi86+O6XOB24BDc+Iz3RGSmqs5vd15vXKDC/8VsfTZQXe3cQkPehIYNG0ZxcXG37eQeMWIES5YsyYr4RelAUVFRtxvgGQgE2jZBQVv32d13T75RGY6fcB/fUNXdRWSeql4rIjcCz/u4bm8goKoLAETkUeBYXOTaUKYBNwA/i8Hu7CEQaFOrAMjJyWHMmDHdVizy8/MpLy9PtRlGhqKqBAIBJrWPDWZjLbqEH9fZLd7ySxEZBjTiRnRHYziwOGR7ibdvO95I8JGq+qyP/LITL9pse8x91jA6pr6+ns2bN+9Ys+jf3yVrhuoUfufg7gdMB94HaoBHunpjEcnBeVld6uPcs0RktojMzqqmiS1bYOnSHWoW4MRiwYIFNDc3p8Aww0hfOnSbDWLus53GTwf3NFVd53lEjQJ2UdVf+ch7KRDiisAIb1+Q3sBXgddEpAbYF5jZUSe3qt6pqhNUdUI2hFbeTvANqIMf/ZgxY2hoaGCRzStsGG1oE222PSYWncbPTHnneTULVHUbkCMi5/rI+z2gUkTKRaQAmAzMDB5U1fWqOkhVy1S1DHgHOEZVZ3fmg3RLQqLNtscCChpGxwQCAfLy8jp2K6+ocPOfRIgWbHSMn2aoM1V1XXBDVdcCZ0a7SFWbgPOBfwOfAv9Q1U9E5DoROaazBmcVwTegMM1QYGJhGO2pqqra0W02SEWFEwqrkceMH2+oXBERLxBV0CXWVwxoVX0OeK7dvg7HaKjqJD95ZhWBgOuQGzBgh0NDhw6lZ8+eJhaG0Y4O3WaDhHpEjR6dPKO6AX5qFi8Aj4nIt0TkW7jO7RcSa5YBhPWEAhARxowZ020H5hlGZ1DVtqHJ22Pus53GT83iCuCnwDne9ku4wXlGoqmuhr33Dnu4srKSefPmJdEgw0hvli9fzubNm8OLxbBhLgKtiUXM+PGGalHVP6vqiV76q6qav2aiaWiAmpqwNQtodZ+NNLWnYWQTHQYQDCUnB8rLTSw6QdiahYj8Q1W/LyIfATsE6VFVGy+fSGproaUlqlg0NTVRW1tLRQed4IaRbUQcYxHE3Gc7RaRmqKne8qhkGGK0I4InVJBQjygTC8NodZstKysLf1JFBbz2Gqi6ue0NX0RqhgoGfv+Nqta2T8kwLquJMMYiiLnPGkZbqqqqKCsr69htNkhFBWzeDCtWJM+wbkCkmkWBiJwCfENEjm9/UFWfSpxZBoEA9OwJJSVhTxkyZAi9evUysTAMj0AgELkJCtqGKo/w/zLaEkkszgZ+APQDjm53TAETi0QSnHc7QjVZRCygoGF4BKPNRp0+NtR9duLExBvWTYg0U96bwJsiMltV70miTQa4msXYsVFPq6ysZM6cOUkwyDDSmxUrVrBx48boNYvycvcSZp3cMRG2z0JEDvZW14rI8e1TkuzLTpqbXRU52o8eJxY1NTU0NjYmwTDDSF+ius0GKSyEESNMLGIkUjPUgcAr7NgEBdYMlViWLnXjLHx4OFVWVtLc3ExNTU30P4lhdGN8uc0GMffZmInUDPVrb3l68swxAF+eUEFCPaJMLIxsJhAIkJubG9ltNkhFBcyaFf08Yzt+QpRfJCJ9xHG3iLwvIocmw7isJfjGE6NYGEY2EwgEKCsrIz8/P/rJFRWwfDls2pR4w7oJfgIJnqGqG4BDgYHAqcD1CbUq2wkEoKAAhg+PeuqgQYPo27eviYWR9UQMINieYMRZm2LVN37EIui7eQTwgKp+ErLPSASBgPsx5+ZGPdXcZw2j1W3Wt1hY9NmY8SMWc0TkRZxY/FtEegMtiTUry6mu9tUEFcTEwsh2Vq1axYYNG/z325lYxIwfsfgJcCXwdVX9EsgHrNM7Uai6mkUMsZ7GjBlDbW0tDQ0NCTTMMNKXmDyhwE0q1r+/iUUM+BGLicDnqrpORH4IXAWsT6xZWczy5S5uTYw1i5aWFhZY+6uRpfgeYxFKRYX1WcSAH7H4M/CliOwBXApUAw8k1KpsJgZPqCDBP4jNmmdkK1VVVeTk5Phzmw1iYy1iwo9YNHnzbx8L/ElVbwd6J9asLCb4wI+hGcrcZ41sJxAIMGrUKAoKCvxfVFHh5o2xycN84UcsNorIz4EfAs+KSA6u38JIBIGA84IaNcr3JQMHDqR///4mFkbWEggEYh+UWlHhhGLRosQY1c3wIxYnAduAn6hqPTACmJ5Qq7KZ6monFLG8IWEeUUb2oqqxjbEIYh5RMeFnDu56Vb1JVd/wthepqvVZJIoYPaGCmFgY2crq1atZv35952oWYGLhEz/hPvYVkfdEZJOINIhIs4iYN1SiCARi6twOUllZyaJFi9i6dWsCjDKM9CVmt9kgw4a5CLQmFr7w0wz1J+BkoAooBqYAdyTSqKxlzRpYu7bTYqGq5j5rZB1BL8CYxSInx81tYWLhCz9igaoGgFxVbVbV+4DDEmtWlhL80XayGQrMI8rIPgKBADk5OZSXl8d+sbnP+ibSfBZBvhSRAmCuiPwBqMOnyBgx0okxFkGCb1UmFka2UVVVxU477URhYWHsF1dUwGuvucgJEaYwNvw99E8FcoHzgc3ASOCERBqVtQTHWBQXw7nnwp57+r60f//+DBw40MTCyDo65TYbpKLCRUxYsSK+RnVDotYsVLXWW90CXJtYc7KcefOgZ0/YbTdoaXGz5cVAZWWljeI2soqg2+zJJ5/cuQxCPaJKSuJnWDckrFiIyEe46VM7RFV3T4hF2UhdHUybBk8+6arDGrbYI1JZWclrr70WX9sMI41Zs2YN69ati71zO0hQLBYsgG98I36GdUMi1SyOSpoV2c7kyfDmm6420QUqKyt58MEH2bJlC8XFxXEyzjDSl04FEAylvNz1VVgnd1Qi9VnkAyNUtTY04UZw++kYR0QOE5HPRSQgIld2cPwSEZkvIvNE5P9ExH+Mi+7EY4/B17/u1v1MCRmG4B+m2n74RpbQ6TEWQQoLYcQIEwsfRBKLW4ANHezf4B2LiIjkArcDhwNjgZNFZGy70z4AJnhNWk8Af/BjdLejXz+oqYGDDoIzz3Qd3DGG+wBznzWyj0AggIh0zm02iLnP+iKSWJSo6kftd3r7ynzkvTcQUNUFqtoAPIqLXBua16vehEoA7+BqLdnHgw+6eSyuugpuv921n06ZAuPGxZSNiYWRbQTdZouKijqfiYmFLyI1J/WLcMxPg/hwYHHI9hJgnwjn/wR43ke+3YuWFpgxA8aPdzULgNJSJxox0qdPH4YMGWJiYWQNMc27HY6KCveytmkT9OoVH8O6IZHEYraInKmqd4XuFJEpwJx4GuHNwDcBODDM8bOAswB22mmneN469cycCV984fot4jAoyAIKGt2d0tJSli9f3mafiFBSUkJ9fX3sGY4e7ZYLFsDu5uQZjkhiMRX4p4j8gFZxmAAUAN/1kfdS3AC+ICO8fW0QkW8DvwQOVNVtHWWkqncCdwJMmDChc36l6Ygq3HCD88g4/vi4ZFlZWcmLL74Yl7wMIx1pLxTR9kcldKyFiUVYwvZZqOpyVf0GbiBejZeuVdWJ3rwW0XgPqBSRci9cyGRgZugJIrIn8FfgGFXNviGU//0vvPMOXHop5PlyMIvKmDFjWLZsGZs3b45LfobR7QkVi7q6mKMnZAt+RnC/Crwaa8aq2iQi5wP/xoULuVdVPxGR64DZqjoTN4lSL+BxcU0wi1T1mFjvlbFMnw4DB8Lpp8cty1D32d3tLckwotO/P/TtC/fdB7/6VaeiJ2QD8XmdDYOqPgc8127f1SHr307k/dOaTz91/RXXXAM9esQt21CPKBMLw4hCMHrChg0udTJ6QjaQULEwIjBjhhtPcd55cc3Wos8a3ZnHH388vhkGoyeYSETFQo2ngmXL3NiKM86AQYPimnXv3r0pLS01sTC6HXfeeScnnXQS+WGiHJR0JhDgY4/B2WdDV8ZpZAkmFqng1luhuRkuuSQh2Zv7rNGdUFWuv/56fvrTn3L44Yezbt06VHWH1Cm32eCYpoULWwMJxsnZpLthpZJsNmyAP/8ZTjyx1b87TrT3P/ecBjrvf24YKUZVufzyy5kxYwannHIK999/f9iaRZcoLYXXX3eus4sWwdj2kYkMq1kkmzvvdILxs5/FPeu4+58bRgppampiypQpzJgxg/PPP58HH3wwMUIRJC8P7rnHTYZ0yCGJu0+GYmKRTBoa4JZb4OCDYcKEVFtjGGnL1q1bOemkk7j33nu5+uqrufXWW8nJScLjauJEF8zzllvcZGTGdkwskskjj8DSpXD55Um/9dq1a5N+T8PoDBs3buTII4/kqaee4o9//CPXXnvt9ibVpHD99W7sxdlnd3mOme6EiUWyaGlxg/B23x0OPTTpty8vL2fatGls2NBR1HnDSA9Wr17Nt771LV5//XUeeOABLrzwwuQbMWCAc21/+23XLGUAJhbJ4/nn4ZNPXF9FMt+SPCZNmsTVV19NeXk5N9xwg4UDMdKOJUuW8M1vfpN58+bxz3/+k1NPPTV1xvzoR3DggXDFFbByZersSCNMLJLF9OkwciScdFLCbhHOz7ykpISnn36a2bNns++++3LllVcyevRobr75ZrZs2ZIwewzDL1988QX77bcfS5cu5d///jdHH310ag0SgTvugI0bE+KMkomYWCSD//3PueVdfHGXpk2NRn19fUT/8/Hjx/Pss8/y1ltvsfvuu3PJJZdQUVHB7bffzrZtHQb8NYyE88EHH7D//vuzZcsWXn31VQ48sMOZCpLP2LFOKP72N/f/zXJMLJLB9Olu6tQpU1JtCQATJ07kpZde4rXXXmPMmDGcf/75VFZWctddd9HY2Jhq84ws4o033mDSpEkUFxfz5ptvstdee6XapLZcdRWUlcE552R9cEETi0QTCMBTT7mwx717p9qaNhx44IG8/vrrvPjiiwwbNoyzzjqLXXbZhb/97W+UlpYiIjuk0tLSlNkbT5vS8fN1dzoq8wMOOIDNmzfz5ptvsvPOO6faxB3p0QP+9CcX+POmm1JtTWrpqNkindP48eM1ozj7bNXCQtW6ulRbEpGWlhadNWuW7rXXXgpETLFSUlLSYT4lJSUx5RNPm+KVV7w+W7oSz88Xz+8v6Xz3u6rFxaoLFqTakk6Dmxqi089eC/eRSJYvdzHyf/QjF04gjRERjjzySI444gieeeYZvvvd8JMh3n333eTl5ZGbmxtxGVyPNLL8v//9Ly0tLTskVd1hXySeeuqp7b74HdUY2h+LxBtvvEFOTs4OSUR22BfPUfMdTRcKnQvXEq+8Yv18zc3NbNq0iY0bN7Jx40Y2bNiwfZnR/PGPsOuucMEF8K9/pcSjMdWIZlho3gkTJujs2bNTbYY/fvUr+O1vXRX2K19JtTUxkdRBUN2Ur33tawwaNMhX6tGjR8Qyj/V/Gi2v5uZmvvzyS7788ks2b968fT00bd68mdNOOy1sPgcddNAOgtBZl+yMeA7deCNcdplrVo7wMpWuiMgcVe106AirWSSKTZtcNMvjjss4oYjG4sWLaWpqoqmpiebm5qjLb387/BxXL774oq83+JycHMaPHx82n7lz5wLhm1XbH5s4cWLYvF5++eUOazsd1XhOiuAKXVFRwapVq/joo49YtWoVq1evDvtQLC4uDpsPwFFHHbX93qHLcPsiUVhYSEMcOmsbGxsZPHgwFRUV9O7dmz59+kRcjhs3rsv3TCkXXug8oy680MWO6tUr1RYlFatZJIpbb4WLLoK33nLxZjKMZL7lZotNzc3NrFu3jlWrVnWYZsyYETavvfbaq42QBpvTwu175ZVXwuZ1xRVX0KNHD3r27EmPHj06TMFjFcH5qX18vmjE8/tLGW+9BfvtB5de6kZ5ZxBdrVmYWCSCxkYYMwZ22gneeCPV1nSKeLafx+shkY5t+ukoYPHMK542xfP7Sylnnun6It9/34XvyRC6KhbmOpsIHn/cxcRPQcDAeBFtgF8sRBpZniqb4pVXvD5buhLPzxfP7y+lZGmgQROLeKPqBuHtuisceWSqrUkLus1DogPSUVTjmVd3/u46zcCB7j/+9ttw772ptiZpmFjEm5dfhrlznddEMuLvG92GdKw5GWH48Y/hgANc60GWBBq0p1m8+cMfYOhQ+MEPUm2JYRiJIjTQYAY3N8eCiUU8+eADV7OYOhUKC1NtjWEYiWS33VwLwv33w3/+k2prEo6JRTyZPt3Ff/rpT1NtiWEYyeBXv4JRo7Ii0KCJRbxYuBD+8Q8nFH37ptoawzCSQTDQ4Pz53T7QoIlFvLj5ZtehfdFFqbbEMIxkctRRLlLDdddBTU2qrUkYJhbxYPVqN1fvD34AI0ak2hrDMJLNH//oXhYvuMC5z3dDTCziwR13wJdfwpYtsOeeqbbGMIxks9NOcM01MGsWPPOM21dX5+ax6SbPBAv3UVcH06a5ATYffBD79QsWOK+IhgbIy3PLDCtTwzDiQGMjjB8Pq1bBQQe56LQtLWnzTLCos50lKBL33df6ha5b15rWro28vmIFVFW57SDd3BvCMIwIrFrlYsJ99BE8/HDbY+eeCxUVLo0e7ZY9e/rLt6svtHEioWIhIocBfwRygbtV9fp2xwuBB4DxwGrgJFWtiZqx38JraID6eli6FJYtc8svvoCXXnLTnaq2Vfz+/SN9GOfl1K+fO2/BAli/PqqphmFkCZMnw5tvdnzskUfcS2YoJSU7CkgwDRninl3tX2hTSMKaoUQkF/gCOARYArwHnKyq80POORfYXVXPFpHJwHdVNfwEAcCEIUN09saNrYU3a1ZbMVi2rHW9o2H4IuGrhDfe6ISgX79WUQiu9+nTNnxH6BfZ3Nz2i0yDKqdhGEkm2jNhzRqornYvmtXVbdeXLGn73MjLc3kErw1y7rlQVORSYaH/9Y0bGX3ggSsXqA7p7MdLpFhMBK5R1e942z8HUNXfh5zzb++ct0UkD6gHBmsEoyaIaIc9FiJOjYcNg+HD2y5D1xsb3ex18XrId/QDMbEwjOylM8+ErVud2+2CBW5yperqjs8bONCdu3Vrq5j4ZDzoHNVOOzUlUixOBA5T1Sne9qnAPqp6fsg5H3vnLPG2q71zVoXLdwexeOcdJwKlpZCf79/AeD/kg/m99VZK2xUNw0gTOvtM8Ntq0dQE27Y54Qgu269feCF88gmoMgGYrdrp+ZIzQixE5CzgLIC9YPyc0INdtd8e8oZhpCPxeKENyWP8li1pW7NITDPUkCE6e9Mma/IxDCM7iMcLbX095UOHrlzYhT6LRA7Kew+oFJFyESkAJgMz250zE/ixt34i8EokoQDc4JcFC2DKFMj0CeANwzCiUVoKt9/etZaP0lJqYFFXzEiY66yqNonI+cC/ca6z96rqJyJyHTBbVWcC9wAPikgAWIMTlOgEC88wDMNICgkdZ6GqzwHPtdt3dcj6VuB7ibTBMAzD6DoWG8owDMOIiomFYRiGERUTC8MwDCMqJhaGYRhGVDIuRLmIbAQ+T7Ud7RgEhB11niLS0SZIT7vMJn+YTf5JR7u+oqq9O3txJoYo/7wrMdkTgYjMNpv8kY52mU3+MJv8k452iUiXJgKyZijDMAwjKiYWhmEYRlQyUSzuTLUBHWA2+Scd7TKb/GE2+Scd7eqSTRnXwW0YhmEkn0ysWRiGYRhJJqPEQkQOE5HPRSQgIlemyIaRIvKqiMwXkU9E5CJv/wAReUlEqrxlhAm9E2Zbroh8ICKzvO1yEfmfV16PedF/k2lPPxF5QkQ+E5FPRWRiqstJRC72vrePReQRESlKRTmJyL0issKb0yW4r8OyEcetnn3zRGSvJNo03fv+5onIP0WkX8ixn3s2fS4i30mWTSHHLhURFZFB3nbKysnbf4FXVp+IyB9C9ie8nMLZJSLjROQdEZkrIrNFZG9vf+xlpaoZkXCRa6uB0UAB8CEwNgV2DAX28tZ74+YZHwv8AbjS238lcEMKbLsEeBiY5W3/A5jsrf8FOCfJ9vwNmOKtFwD9UllOwHBgIVAcUj6npaKcgAOAvYCPQ/Z1WDbAEcDzgAD7Av9Lok2HAnne+g0hNo31/oOFQLn338xNhk3e/pG4iNa1wKA0KKeDgJeBQm97SDLLKYJdLwKHh5TPa50tq0yqWewNBFR1gao2AI8CxybbCFWtU9X3vfWNwKe4h9CxuIcj3vK4ZNolIiOAI4G7vW0BDgaeSIVNItIX9+O9B0BVG1R1HSkuJ9zYomJxk231AOpIQTmp6n9wYflDCVc2xwIPqOMdoJ+IDE2GTar6oqo2eZvvACNCbHpUVbep6kIggPuPJtwmj5uBy4HQTteUlRNwDnC9qm7zzlkRYlPCyymCXQr08db7AstC7IqprDJJLIYDi0O2l3j7UoaIlAF7Av8DSlS1zjtUD5Qk2ZxbcH+eFm97ILAu5I+e7PIqB1YC93lNY3eLSE9SWE6quhSYgZsEpg5YD8whteUUSriySZff/hm4t1FIoU0iciywVFU/bHcoleW0M/BNrznzdRH5ehrYBDAVmC4ii3G//Z931q5MEou0QkR6AU8CU1V1Q+gxdfW8pLmZichRwApVnRP15OSRh6sS/1lV9wQ245pWtpOCcuqPe6MqB4YBPYHDknX/WEh22URDRH4JNAEPpdiOHsAvgKujnZtk8oABuCadnwH/8Gr3qeYc4GJVHQlcjFfT7wyZJBZLce2UQUZ4+5KOiOTjhOIhVX3K2708WI3zlivCXZ8A9gOOEZEaXPPcwcAfcVXLYEiXZJfXEmCJqv7P234CJx6pLKdvAwtVdaWqNgJP4couleUUSriySelvX0ROA44CfuCJWCptqsCJ/Yfe730E8L6IlKbQJnC/96e8Zp13cTX8QSm2Cdy01cFn1OO0NoHFbFcmiYWfOb0Tjve2cA/wqareFHIodD7xHwPPJMsmVf25qo5Q1TJcubyiqj8AXsXNbZ4Km+qBxSLyFW/Xt4D5pLCccM1P+4pID+97DNqUsnJqR7iymQn8yPNg2RdYH9JclVBE5DBc8+YxqvplO1sni0ihiJQDlcC7ibZHVT9S1SGqWub93pfgHE7qSWE5AU/jOrkRkZ1xDh2rSFE5hbAMONBbPxio8tZjL6tE9MonKuF68L/AeRT8MkU27I9rHpgHzPXSEbg+gv/zvoyXgQEpsm8Srd5Qo3E/zADuraIwybaMA2Z7ZfU00D/V5QRcC3wGfAw8iPNSSXo5AY/g+k0acQ+8n4QrG5zHyu3e7/4jYEISbQrg2raDv/W/hJz/S8+mz/E8bpJhU7vjNbR6Q6WynAqAv3u/q/eBg5NZThHs2h/XL/chrm91fGfLykZwG4ZhGFHJpGYowzAMI0WYWBiGYRhRMbEwDMMwomJiYRiGYUTFxMIwDMOIiomFkVGISLMXQfNjEflXaBTUMOeXicgpPvJtc56ITBCRW+Nkc42IfOSl+SLyGxEpikfehpEsTCyMTGOLqo5T1a/igqadF+X8MiCqWLQ/T1Vnq+qFnTWyAw5S1a/hRtCOBv4aj0xDRp4bRkIxsTAymbfxgp95I1GnezWOj0TkJO+c63EB3uaKm8uiTETeEJH3vfSNMOdNktZ5QQaIyNNe3P93RGR3b/813hwCr4nIAhGJKi6qugk4GzhORAZ4+fxMRN7z8r82eK6I/ErcHAhvipt74zJv/2sicouIzAYuEpHBIvKkl8d7IrKfd15Pz753vWCOSY/SbHQf7K3EyEhEJBcXriMYGO143IjxPXAxed4Tkf/gghdepqpHedf1AA5R1a0iUokb9Tqhg/MmhdzuWuADVT1ORA4GHvDuBbALLsxDb+BzEfmzurhTYVHVDSKyEBe+pi8uBMTeuFG1M0XkAGALcIL3efJxo4JDA0UWqOoEz9aHgZtV9VNSCnsAAAJDSURBVE0R2Qk3z8OuuJHDr6jqGV5z3bsi8rKqbo5awIbRDhMLI9MoFpG5uBrFp8BL3v79gUdUtRkXkO914OvAhnbX5wN/EpFxQDMutHQ09sc9uFHVV0RkoIgE5wh4Vt0cBttEZAUurPgSH3kGI5Ie6qUPvO1eOPHoDTyjqluBrSLyr3bXPxay/m1grLQGOe0jLiryobgAk5d5+4uAnXDlZhgxYWJhZBpbVHWcV0P4N67PIpaO6IuB5bg39hxgaxft2Ray3oyP/5SI9Mb1kXyBE43fq+pf250zNUo2obWDHGBfT1hC8xDgBFX9PJpNhhEN67MwMhJ1EVAvBC71OnnfAE4SNw/5YNwsfe8CG3Fv6UH6AnWq2gKcipuulw7OC+UN4AewvXlqlbabw8Qv3hv/HcDTqroWJ3hnePsRkeEiMgT4L3C0uDnCe+FChIfjReCCkHsEm8j+DVzgiQYismdnbDYMsJqFkcGo6gciMg84GRfxcyIuuqYCl6tqvYisBppF5EPgftyD+kkR+RHwAq1v6PPanfdByK2uAe717vUlrWHEY+FV76GdA/wTmOZ9hhdFZFfgbe+Zvgn4oaq+JyIzPbuW4yKDrg+T94XA7Z59ecB/cJ3o03AzKM4TkRzc/OORRMcwwmJRZw0jTRGRXqq6yWty+w9wlnrzvxtGsrGahWGkL3eKyFhcx/TfTCiMVGI1C8MwDCMq1sFtGIZhRMXEwjAMw4iKiYVhGIYRFRMLwzAMIyomFoZhGEZUTCwMwzCMqPw/BAkI5/jKG4QAAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX8AAAA1CAYAAACp8OvZAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAADTxJREFUeJzt3V1MHGUXB/Czy7J8G1pWQitWUk3TIIloGkIoNpC0EmKkkqgtCQWJRiBsmxKCXJDQxA80VWsN0WqRi4oGmoJKiL1zbdwUTWmxSmi1dRvshu62tWLBrmVndv7vBZl5l9qPmWcQoXN+SS+62Tl75tln/vOx06kNADHGGLMW+3/dAGOMsYXH4c8YYxbE4c8YYxbE4c8YYxbE4c8YYxbE4c8YYxbE4c8YYxbE4c8YYxbE4c8YYxbE4c8YYxbkWODPW5BnSSiKQna7uf2aLMvkcJgbnkgkQjExMaZqMMbuHpIkUWxs7L/9MTY9b7orj/ztdjtNTk6aquFwOCgUCpmqMR/BH4lETNdgjJknSZLpGgsQ/LotuvB/7bXX6ODBg6ZqnDx5kjZt2kTHjx8XDs9QKERJSUm0a9cu4T52795NNpuNjh8/LlwjJiaG9u7dS+Pj40QktjMIh8NzlpVlWbgfFT8QkC0ERVH+8zpXrlyhF198kS5evCj82cFgkGpra2nTpk00MzMj3Munn35q+qqGatGF/wMPPEAHDhwwVSMrK4tOnDhBfX19QiEFgBITEyk7O5va29uFelAUhZ599lmqq6ujhoYGCgaDQr1cvHiRDh8+TBUVFfThhx8KnU04nU4iInrllVdoeHiYHA6H0MYgSZK2A7HZbLwDYP8KRVG0uWW3202Fpd/vp8cff1woMNW53traSl1dXfT7778L9aAoCvX09NCXX35JpaWl2vZoFAA6cOAArVy5Umj5mxZcwD93VF5ejt7eXj1vvS0igsfjMVWjrq4Os0MkLhQKgYjw1VdfIRwOC9eprq6Gw+EAAMzMzAjVaGtrg8vlMjUuRIS4uDiUlpYCgOF1UnsfGRnBt99+K9wHm1+SJCESiUBRFMPLRiIRRCIR7e+nTp3C0aNH57ymtw4AjI+Po7i4GE6nE42NjYbrALPrk5WVBSJCRUWF4eUVRUEoFMILL7wAIsJDDz2kvW5UOBwGESE5OdnwstG2bdsGIkIwGLzTW3Xl8aIK/5GRETidToRCoTuPxC0oioLLly+DiDA6OgpZloXqyLKMiooKU+GvThQiQk1NjdDEUZfp7OwU7kUdg8uXL+O9997DM888A0VRhHZGg4ODyM3NndOLJEmGaoTDYSQmJuKee+5BX1+f9pqRfhRFgSRJCIVC8Pv92vpIkiQ0zkuFum6yLCMSiRge++g6N4b29PS0qd6++OILNDU1weVyGZ6r6nr19/ejqKgINHtzCM6cOSPUy6FDh0BEcLlcOHv2rOHlJUmC1+tFTk4OiouL4fP5hOfVxMQEiAh5eXnC3xcwG/42m03PW5de+Dc3N+tdudsaGxsDEaGlpUX4KBkAurq6kJiYiNHRUVNf2tNPP236DEKWZfT09ICI8NZbbwGA8JGVekQ0PDwsVAcAent7QUSIj4/H1atXARg7Krp69SrOnDkDm82m7Ry//vpr3ctLkoSamhosX74caWlpICK43W54vV5cunRJe5+edTtx4gR6e3uxefPmeTnrvJmpqSn8+eef/3j9dv2pO+3oOXzu3DmcP38eo6OjePvtt/HRRx+hs7MTIyMjuvpQa4VCIRw5cgT79u3DihUrtLCNj4/HH3/8ccc66vbg9/vhdru1A4LMzExs375dd+Cq6//DDz+gvb0ddrsdRARJkhAOh4XmZn5+PogI9fX1AIwfrUciEUxNTYGIkJCQgOvXrxvuQf3ciYkJlJWVoba2FoFAQGh9VDab7e4N/8rKynkJ/7Nnz4KIUFJSInypJRKJoLu7G6tWrTK9E6mrq0NSUhI6OzuFawDA8PCwtpGqPYrYu3cv0tPTkZqaKtxLOBzWjtBWr16NgYEB3ctGn411dHRg3bp12np1d3djampK187W5/NhZGQEHo8HVVVVWo2YmBh0dHRgYmJCVz8JCQnahmWz2ZCenq792bBhA44dO6b7DHJ4eBgtLS3weDwYHR1FS0sLSkpKUFhYiNraWnR2dqKhoQH9/f0IBAK6agLQahUUFCA/Px+5ubnIyMgAESElJUW7HHCzOaGGX/RZZGFhoTZeashlZGTA4/EYOlvesGGDVuPJJ5+cs+PVQ/2eCwsLkZCQgPLycuzevRuA8UuKADAwMAAiQmxsrPCZTDgcRnd395xLRiJH/bIso6OjAytXrjS0fdzKXR3+69evNx3+0ZdaCgsLTdUCgKqqKhQUFMypLdJPQ0MDHnvsMRw7dkyoD3WDbGpqAhFhcHBQqI5qcnISDocD7e3tAMR3JKdPn0ZycjKICL/++qtwrcHBQezcuVMLEq/XCwC3DSL1GrU6xn///TfefPNN1NbWanWKioqwZcsWQ718//33aG5uRmVlpTYn1TMUm82G7u5u7Wwn2tTUFHJyclBfX4+tW7fi4Ycf1sL6qaee0kI7Pj4eSUlJc3bkN65nMBiEw+HQ3pOYmIhVq1ahqqoKra2t+Oyzz7B//354vV50dXXhwoULt5yfgUAAhw8fRk1NDYgISUlJWLduneE5FN2j2+0GEWHt2rXw+Xza60YCe2ZmBoFAAGvXrtUOsgDjlxHVZVpaWkBEePnllwGIzUO1fyLCzp07DS+vUr+LmJgYOJ3OOa+JstlsqKys1PPWpRf+BvZst6ROUCJCbW2tqVoA0Nraivz8fNN1BgcHQUQ4cuSIcNBKkgSfz4fU1FS43W4AYhNcHaOysjKkpaXh6NGjQrXUybxnzx7ExsaivLwchw4dMtxP9EZRUVGBhIQE5OTk6Plh65YGBgbm7AT27Nlj6EhbNTMzg7GxMWzduhWpqanaHL3xwCL68sXk5CSA2d9YxsbG5lwC+fnnnzE9PY33338fdXV1aGxs/MdnXrt2DXl5ecjOzobb7UZFRQW6urrQ3d1tuH8AyMvLQ0JCAogITqcTdXV1+OabbwD8f+zvFLjq+3788UdtLjudTnzwwQe6lr+VJ554QrtEMzk5KVRHDez4+HgQEU6fPi3US3QtswdY6noQEZYtWwbg9gcyenD430H0oO/atcvUNTYAGBoawurVqwGI77mjz0b27dtnuqf+/n7k5uZiaGjI1I5kZmYGdrsdJSUlwr2o4+31ehEbG4s1a9YAMD5W0Rt9W1sbiAjV1dXw+/2611FRFMiyrH12IBBAXl4ekpOTkZ6ejpdeeslQTzcKBoMoKyvTPU+jx0DtLdqNP7hG27JlC4aGhm56/V0dK7Xe7ULll19+ARFh/fr1ePXVV+HxeLRLmCJB++CDD8LhcCAzMxM9PT1CdSRJwrVr1+DxeEBEyMrKwvj4uOFegLljrJ7p3fi6UeodeleuXBGuEb0TycnJATA/4d/c3KznrUsz/Ldt26Zn5W7rp59+QkpKClpbW00P+P79+5GRkQFA/NKIioiwYsUKUzWia6WlpZmu8/zzz4OI8PHHH5uu9fnnn8PlcqG8vBznz58XqqFutD6fTztqF92Qo++Geeedd7Bx40YAYteS/wvRl7REybKMc+fOafVE5/D169dRWVmJxsZG7TKPmd7uvfdeuFwu+P1+4Rrq55eWloKI0Nvba+q7VRQFfX19yMrKQkdHh3AdVTAYBBHh4MGDpmuFQiE4nU69P+wvrfBXB2o+wn96ehp2ux0NDQ2mawUCARCRqeBXA0g99Y5+zSh1cmdmZpq+g0i9NTI3NxfFxcVaSIjWmp6e1n4Efv31102FNgDU19ebHi+VoijCd20sVWbH7EbvvvuudknFzPYgyzIcDgfa2tpM1wKAuLg45Obmmqqh9pCdnY24uDhTtYDZ+dbU1ISqqirTB6DA7N115eXlet++tML/vvvuw8aNG+flHu3nnnsO999/P0ZHR01PrK6uLqSkpJjuCYB2d8b09LTpvrZv3w4iwqlTp0z35ff7QUTYsWOHqTpq2HR0dODkyZOm7pBii0t0gM3HNvrdd9/NSy1ZluflBghVYmLivPxW6Pf7UVBQoPsW3DspLS3FJ598ovftuvJ40Tze4cKFC1RdXU02m64H0t2Soii0fPlyevTRR8nlcpl+DkZ2djb99ddfpmoQzf5T8c2bNxMRUXJysqm+IpEI7dixg9asWWP6AXaKolBmZiY1NjbSG2+8YeoZKOpTUN1uNz3yyCPC/4ydLT7RjxUxu42Gw2HKz8+fl1oxMTHk9XqpqKhoXp4DJMsypaSkmK6TlpY2r/O/urqafvvtt3mrR0RkAxb0+SwL8mE+n48yMzMpLi7OdC0AdOnSJUpPTzc9USORCE1NTdGyZctM96Waj8dXM8Zmt3Wz2/gioWsl7srwX+z4Of+M3d0AkCRJ/9XZL4c/Y4xZkK7wX+j/yeuuOKdijLGlji8WM8aYBXH4M8aYBXH4M8aYBXH4M8aYBXH4M8aYBXH4M8aYBXH4M8aYBXH4M8aYBXH4M8aYBXH4M8aYBXH4M8aYBXH4M8aYBXH4M8aYBXH4M8aYBXH4M8aYBXH4M8aYBXH4M8aYBXH4M8aYBXH4M8aYBXH4M8aYBXH4M8aYBXH4M8aYBXH4M8aYBf0PwIACOHRGfV4AAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "rotating_image_classification(digit_one, sess4, prob4, X4, keep_prob4, u4)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Some Other Data Uncertainty Experiments" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Consider the case that we mix two digits from the MNIST dataset and query a classifier trained on MNIST dataset to classify it. For example, the following image is created by overlaying digit 0 with digit 6. The resulting image have similarities to both digits but neither 0 nor 6." ] }, { "cell_type": "code", "execution_count": 317, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAACFCAYAAABL2gNbAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAErtJREFUeJzt3Xt0VdWdB/DvLzcJgQQCIUAxBAgQAiioGB8oWhStSjviWo5dPnCQUukssFKrXWLtOKvLqWOnykyt7SgWRUdHsVoU0dElNL6qIIoSBYQgDwkSIEQghEdee/7gevb5Xb1JvLn33Ht3vp+1WPx2fif3bPJLNif77rOPGGNARETpLyPZHSAiovjggE5E5AgO6EREjuCATkTkCA7oRESO4IBOROQIDuhERI7o1IAuIpeIyEYR2Swi8+LVKUou1tVdrK3bJNYbi0QkBGATgIsAVANYDeBqY8z6+HWPgsa6uou1dV9mJz73DACbjTFbAEBEngYwFUDUb45s6WZykNuJU1I8HEUDGs0xiZJmXdNUO3UFvmVtWdfUUY8va40x/do7rjMDehGAHb52NYAz2/qEHOTiTJnciVNSPKwyK9pKs65pqp26At+ytqxr6lhunt3ekeM6M6B3iIjMAjALAHLQI9Gno4Cwrm5iXdNbZ94U3Qmg2NceFP6YYoxZYIwpN8aUZ6FbJ05HAWFd3dVubVnX9NaZAX01gFIRKRGRbABXAVgan25RErGu7mJtHRfzlIsxpllEbgTwKoAQgEeMMevi1jNKCtbVXayt+zo1h26MeRnAy3HqC6UI1tVdrK3beKcoEZEjOKATETmCAzoRkSM4oBMROSLhNxYRpYqWSeNVO/PO3V78Yplevbe75YhqXz/9Ji8OVaxJQO8oVpklQ1T7i+8XefHB4a0ql9Gkd0YY+eAuL27esi3+nQsYr9CJiBzBAZ2IyBGccgk7+g9neHH3/9O/UpvyMV689TK9+9y5F3ys2m/9bWzUcwx8t8WLc158L6Z+Uuzyfq13MFg8YpkXt0Ycu71Z72Oyd66dgvlORdy7Rt9SaHSpF1d/T29CWH/aUS8uH6b3tLqg4FPV/m3BpV48ZIl+nR5b9ntxy4aq2DsbIF6hExE5ggM6EZEjOKATETmiS82hhwr7enHL4u4q93TpfC/e3ZKlcvkZr3vx4Mx29oie/mbU1J5ph734i/uzVe4nd8/14r4Pv9v2OajDGq6wz2+4a/ADUY8btXSOapf9uUG1C/pxb/CgZfSwX/OaGaeo3JHv1nvxM2fMV7khmfaxmnfsmqRy/7H6YtXO7tnoxTsu1DXOaCr0YmkuVLmSpb7vj5WV39T9pOAVOhGRIzigExE5oktNuWz6/WAv3jhqYUTW/rrVP6Qzf9o/0ovX1A9WueqG3lHPFxK9GO6lshejnmPxr37nxf+84UaVy3j7o6jnIC2z6ATVfuDe+714dLa+frlk/T96cdlN+mtsmhpVW0+QURDqrjjZi/ePbVa51RMe9OLCkF5KPHvnWV781mJ9d3C+vgFYi3i8trrLVM/CYsvl9pwja/SY0Lzt8zZOkli8QicicgQHdCIiR3BAJyJyhNNz6GbCyaq9+OyHfC39T3/liJ1Dv+cX01Wu57pa29hbp3IZX+6Ifv4MPVE+8r7ZXrz+h39QueFZeV585FcHVS7/+gFe3FyzGxRd7QV6573IeXPlvv5eaJq2Rz+OApFZPEi1vxxt4z4nHFC5A612aeJ5q6apXPG9tuaDPtLbeLQePYqoRE+i511utwP54lyda82x5991SZHKDfifffa4Br38NdF4hU5E5AgO6EREjnB6yqUpXy82OyXb/nNbYVTuF4/+yIuLl7yjci2IUav+zBE3r/Ti0dl6aWLl1N978Rtjn1W5cy60UzX5T3DKpS17JurlbRm+a5bzP75S5XJfWR1In6hjTPduqt3S3f6MTjxhi8rN3WprmbO8p36hlfZO68hdNNvugB4TeixZ5cUDQ2eqXM0EOwVzoEx/Xv+RQ23jw3Xfpgedxit0IiJHcEAnInIEB3QiIkc4PYfekiNRc+PeuV61B//mnW8+MEFK56xS7WUXDvTiK/P2qdz+y+zSp/wnEtuvdJQ5pNiL5058TeVafbOo5pH+0LaAUkimXubb2sO+B1VztJfKbVhjl6eOfOdL/XkJ6Frus/rnNevEs724sbc+Y93Jtq99PkxAZ9rAK3QiIke0O6CLyCMiskdEPvF9rEBEXhORqvDffRLbTYo31tVdrG3X1ZEpl0UAHgDwuO9j8wCsMMbcIyLzwu3b4t+9zim7PfqSodAHPaPmkuGO1Zd78ZXn650g55xoH5qxDHH7OVyENK1rpM032DsMn+/9vMptbba7Jnbfq3dQdNgipGFtqy/VD5E4/cSNXnywMUflMhrtdGrGIX33ZyKmXCLlf2bPsvc0nTtUZPsW9P+a7V6hG2PeBFAX8eGpAB4Lx48BuByUVlhXd7G2XVesb4oOMMbsCsc1AAZEO1BEZgGYBQA54GO8Uhzr6q4O1ZZ1TW+dflPUGGOAiNsudX6BMabcGFOehW7RDqMUw7q6q63asq7pLdYr9N0iMtAYs0tEBgLYE89OdUbGuFFePKm3XsK2qcnOtRVWNgXWp47o84ZvjvD8pHUjZevaFhl5KGruqf2ne3GoYk3U47qAlKxt5nfsLwpNeTpX3N0uR3zpixNVLvuAnadu3rItIX1rS/5m+8D3vaelzm8ysV6hLwXw1R6z0wG8EJ/uUJKxru5ibbuAjixbfArAuwDKRKRaRGYCuAfARSJSBeDCcJvSCOvqLta262p3ysUYc3WU1OQ49yUuqqbbhzZflbdX5SZWXufFvV7u2jvtpVtd2/LgaU9Gzf3lfyd5cRGCvRs4WdKptvVn2Ts+I++4zMyw7aad+kHQRauOJbZjaYp3ihIROYIDOhGRIzigExE5wrndFm++9CUv9i9TBIDsP/b1tT4LqEeUaBli51qzRO/YN+hi+/DnjSVnqFzZyJ1e/GLZUpWLfJ0mY3f+W9JQoHL/9uC1XnzC/e+pnGnWT1AirfYkOwSFjuml8X+pHO/FEc9vRt0Yu0a+R+FZKne4v71OPTg8YiOAyA1YfafMPqCvb4tet+NH6I2At02MEa/QiYgcwQGdiMgRzk25+D207zzVzln2XpQjKZ21Gntd0mT0HcAvlPl2Xyxr4zUi2vPrSlV7Th+789/U3FqVm3qLfcD3uL43qdzQO94FdUy3fXo+RJrttEpzrp6OOTis1RdHvpKvmhEbHOTs1dewRwvtsY299HfB1svsQ+aLc8sj+pqayyZ5hU5E5AgO6EREjuCATkTkiLSfQw/1zlftnhnVSeoJpaLtvicW3brtCpXbsdhOvvao1fOnvZZVqvZzP/ieF7fO0FtKvD7uaS+uvP5+lTt7u51TL1zA+XTpprfkNTFeUvqfWJS7U8+9919jd0IM1euly2b9Zt2fMSO8eM8E/Xyh/WV2An7HZL2Mtfen/q0Iou4yHTheoRMROYIDOhGRIzigExE5Iu3n0Ktn6ieZXNuzwovXNAwNuDexOzblQNTc4dbsqDkCZrz6Yy/eNPW/Ve77f5/jxcOu+Ujl+qMm6mtGrkvPe2alF4dW6Fv/H31rqO1L/jaVOzTYxvqZ9l1T8wT989rYx36lM4/o68vCtTZXc7aeJy9Yb+Pej0ffFjmyjpFM5ade3H9TjsodudVuPXCsQL9SXbnd0iGjQc+vS3snTSBeoRMROYIDOhGRI9J+yiVdNV9wmmo/feoDvpZe2rXkt/ZBM/lYCdKy60JRc9NOtNs9vIP4TF217KtT7fmVtj4zzl0Yl3MQkNXgn7vQNT5UZKdgeiM+Wo/qJY55O+xyxKYxOjdigN3+YXPlIJUrfsVOnwa9oJFX6EREjuCATkTkCA7oRESO4Bx6gPzz5nVzG1RuVJadN5+98xyV6714jRenzk3GqSPzsJ1PjXzSUM+QnfvM6KFnW1sPH0YsWiaNV+3Hz3jYnoPXSHGT0eibQ4+oq//JQ5Kph7FYnxKVWTJEtfeeZ7di/um4N1Vu5f4SL86t1jU3az9FsvC7j4jIERzQiYgckfZTLr22taj2tubYfo1OhMhfBfffXO/F749/WuVeO9Ldizf9i76bLrvp/QT0zh2D7rZ3Co4ZP03lPpzwqBc/uPBclRs+a4sXt9bXoy2h0fYJRqW/W6dyp3azUwORNwnm1EY+lbhry6rVP58Zx7pFORLIWv6BFxcUT1C5urH2K/3lNaerXJ+n7OeZpka0JdSvnxd/fmWRyv30zFe8+OcFW1RubJV9MHXxUn3HcUurHpOCxCt0IiJHtDugi0ixiFSIyHoRWScic8MfLxCR10SkKvx3n/Zei1IH6+om1rVr68gVejOAW4wxYwCcBWCOiIwBMA/ACmNMKYAV4TalD9bVTaxrF9buHLoxZheAXeG4XkQ2ACgCMBXApPBhjwF4HcBtCellG3KfW6Xar9w12ouH5+gny1QNOsmLm6t3xuX8rRNPUe2ts218xWi9u9/d/fW8ucrdOt2Lu7/6XtTj4iXV6xqrkht1zSvezvPij8/7s8qNXWB3aSx8vrvKNeXque/f/NJ+7ne763ngiiP2HLNfmqFypf8VfSfAREj1urZ+opf0ZR0624tbInZmCPXq5cWFz+gnSNWXjPPi2lMiF/Pa5cEFa/erzKFhvVR7xG1228Zb+j2kcqVZ9hb+G3ZcrHLFd9ilkS1Ven49mb7Vm6IiMhTAqQBWARgQ/uYBgBoAA6J8ziwAswAgBz1i7SclEOvqJta16+nwm6IikgfgOQA/M8Yc9OeMMQZR7nkxxiwwxpQbY8qzEP0dbUoO1tVNrGvX1KErdBHJwvFvjieNMX8Nf3i3iAw0xuwSkYEA9iSqk7Ga3Xurau9eZn/der9ucOThMbmnZIFqn5Id/Uv6QaNdznTdezNVbvjf7K+iQS16Ste6tqW5Zrdq33vDtbbx8JMqp6ZgztOvE3nHZ6tvQeLVn01RuYP/WuzFpRXJ3w0zXet6rFAv+vx8tp0izTqkjzUZ9v+jaZPfUrmSKXrazW9Krh4T+ofsw56fOaQfOP/j1+30Wb83s1Su4LM1SEUdWeUiABYC2GCMme9LLQXw1cTvdAAvxL97lCisq5tY166tI1fo5wC4DsDHIvLVu3y/BHAPgGdEZCaA7QB+mJguUoKwrm5iXbuwjqxyeRtqKxxlcpSPU4pjXd3EunZtaX/rf6RF9/7Ai/fM1Tuk/brfWtvwx52iv4TNvhnwtRF3HU9bfJMXl8x7V+WSd7Ow20IVdq7zvpnXqNx/3mnnWpeOWqJyM7brsW91hV0OO+yuD/U5jqbmfGo6GPyq3XKhenJPlTsyoNUXR3+NJ9fpW//vHL/Mi/+pV63KPd+gX+iJGnsL/9q/l6rc4LftT2XOMv3zmqq7nvLWfyIiR3BAJyJyhBxfkhqMXlJgzpTgpvFCI0pU+/zn7d1mP+9TFZdzjHrjR6qd/bG9GWPQvwd7l2BHrTIrcNDUxW0bwKDrSt8s3esa6lug2tt/MsqLj/aL3McyNoUf6i9P3k47L5q54oPIw1PGcvPsB8aY8vaO4xU6EZEjOKATETmCAzoRkSOcW7bo17JZ3+a7/CS7LGo5xkceHpNh+Kj9g4ioXS376lTb/yQq6hheoRMROYIDOhGRIzigExE5ggM6EZEjOKATETmCAzoRkSM4oBMROYIDOhGRIzigExE5ggM6EZEjOKATETmCAzoRkSM4oBMROSLQJxaJyF4A2wEUAqht5/CgdMW+DDHG9IvXi7Gu7WJd46er9qVDtQ10QPdOKvJ+Rx6nFAT2JX5Sqf/sS/ykUv/Zl7ZxyoWIyBEc0ImIHJGsAX1Bks77TdiX+Eml/rMv8ZNK/Wdf2pCUOXQiIoo/TrkQETki0AFdRC4RkY0isllE5gV57vD5HxGRPSLyie9jBSLymohUhf/uE0A/ikWkQkTWi8g6EZmbrL7EA+uq+uJMbVlX1Ze0qGtgA7qIhAD8EcClAMYAuFpExgR1/rBFAC6J+Ng8ACuMMaUAVoTbidYM4BZjzBgAZwGYE/5aJKMvncK6fo0TtWVdvyY96mqMCeQPgAkAXvW1bwdwe1Dn9513KIBPfO2NAAaG44EANiahTy8AuCgV+sK6srasa/rWNcgplyIAO3zt6vDHkm2AMWZXOK4BMCDIk4vIUACnAliV7L7EiHWNIs1ry7pGkcp15ZuiPub4f7OBLfsRkTwAzwH4mTHmYDL74rJkfC1Z28RjXb8uyAF9J4BiX3tQ+GPJtltEBgJA+O89QZxURLJw/BvjSWPMX5PZl05iXSM4UlvWNUI61DXIAX01gFIRKRGRbABXAVga4PmjWQpgejiejuNzYwklIgJgIYANxpj5yexLHLCuPg7VlnX1SZu6BvxGwhQAmwB8BuCOJLyR8RSAXQCacHxOcCaAvjj+7nQVgOUACgLox0Qc/9WsEsBH4T9TktEX1pW1ZV3dqSvvFCUicgTfFCUicgQHdCIiR3BAJyJyBAd0IiJHcEAnInIEB3QiIkdwQCcicgQHdCIiR/w/JhQsBvSk4w8AAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "im0 = mnist.test.images[10]\n", "im6 = mnist.test.images[21]\n", "img = im0 + im6\n", "img /= img.max()\n", "plt.subplot(1,3,1)\n", "plt.imshow(im0.reshape(28,28))\n", "plt.subplot(1,3,2)\n", "plt.imshow(im6.reshape(28,28))\n", "plt.subplot(1,3,3)\n", "plt.imshow(img.reshape(28,28))\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The neural network trained with softmax cross entropy loss has the following prediction for the classification of this image, where the image is classifed as 0 with probability 0.9." ] }, { "cell_type": "code", "execution_count": 318, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "softmax prob: [0.901 0. 0. 0. 0. 0.017 0.078 0. 0.003 0. ]\n" ] } ], "source": [ "p1 = sess1.run(prob1, feed_dict={X1:img[None,:], keep_prob1:1.0})\n", "print('softmax prob: ', np.round(p1[0], decimals=3))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "When we do the same experiments on the neural net trained using the loss function in Eq. 7, we have a much different results. The neural network could not generate any evidence to classify the image into one of 10 digits. Hence, it provides uniform distribution as its prediction. It implies I do not know by providing maximum uncertainty." ] }, { "cell_type": "code", "execution_count": 324, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "uncertainty: 1.0\n", "Dirichlet mean: [0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1]\n" ] } ], "source": [ "uncertainty2, p2 = sess2.run([u, prob2], feed_dict={X2:img[None,:], keep_prob2:1.0})\n", "print('uncertainty:', np.round(uncertainty2[0,0], decimals=2))\n", "print('Dirichlet mean: ', np.round(p2[0], decimals=3))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "When we use the loss function in Eq. 5, the exepcted probability is highest for digit 0. It is around 0.32, however, the associated uncertainty is quite high around 0.73 as shown below." ] }, { "cell_type": "code", "execution_count": 325, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "uncertainty: 0.73\n", "Dirichlet mean: [0.325 0.073 0.075 0.074 0.073 0.081 0.078 0.073 0.074 0.074]\n" ] } ], "source": [ "uncertainty3, p3 = sess3.run([u3, prob3], feed_dict={X3:img[None,:], keep_prob3:1.0})\n", "print('uncertainty:', np.round(uncertainty3[0,0], decimals=2))\n", "print('Dirichlet mean: ', np.round(p3[0], decimals=3))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The uncertainty increase to 0.85 while the expected probability for the digit 0 decreases to 0.184 when the loss function in Eq. 6 is used." ] }, { "cell_type": "code", "execution_count": 326, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "uncertainty: 0.85\n", "Dirichlet mean: [0.184 0.085 0.085 0.085 0.085 0.097 0.123 0.085 0.087 0.085]\n" ] } ], "source": [ "uncertainty4, p4 = sess4.run([u4, prob4], feed_dict={X4:img[None,:], keep_prob4:1.0})\n", "print('uncertainty:', np.round(uncertainty4[0,0], decimals=2))\n", "print('Dirichlet mean: ', np.round(p4[0], decimals=3))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Lets try another settings where each of these two digits can be recognizable easily. You can see below an image which is created by combining images for digit 0 and digit 6 without any overlap. " ] }, { "cell_type": "code", "execution_count": 330, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAD8CAYAAAC4nHJkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAETJJREFUeJzt3X2UVPV9x/HPd5ddQJ6RSCigoCI+F8wWayUGfIpSK+ZYPVL1YLRijtgQa3NqTWvsSTW0jebYxGhQMViNktagSKwJJSixGmQlBkSj+LAcobCgqwURYR++/WOHuOre3x3n6c7ye7/O2cPsfOd379eRD3dmfnPvz9xdAOJTk3UDALJB+IFIEX4gUoQfiBThByJF+IFIEX4gUoQfiBThByLVq5I7q7fe3kf9KrlLICofaKf2+G7L57FFhd/MzpB0q6RaSXe5+9zQ4/uon463U4rZJYCAlb4s78cW/LLfzGol3SbpTElHSpphZkcWuj0AlVXMe/5Jkl5199fdfY+kByVNL01bAMqtmPCPlPRml9835u77CDObZWaNZtbYqt1F7A5AKZX90353n+fuDe7eUKfe5d4dgDwVE/5NkkZ3+X1U7j4APUAx4V8laZyZjTWzekkXSFpcmrYAlFvBU33u3mZmV0n6uTqn+ua7+7qSdQagrIqa53f3xyQ9VqJeAFQQX+8FIkX4gUgRfiBShB+IFOEHIkX4gUgRfiBShB+IFOEHIkX4gUgRfiBShB+IFOEHIlXRS3cD1aJ9ynHBeq/rm4P1R8eHL13R3L4rWL9k5lcTa7XLVwfHlgpHfiBShB+IFOEHIkX4gUgRfiBShB+IFOEHIsU8f84HfzYpWO/7X8lzr94QXp/0jbPDy5J//uS1wfqvfnlMsB4y4pn2YL3Po88WvO2erP8/hteXWXjokmC9I2X7G9r2C9a3zUn+HsBnl6dsvEQ48gORIvxApAg/ECnCD0SK8AORIvxApAg/EKmi5vnNrEnSDkntktrcvaEUTRWidtj+wXr7wr7B+oPjbgnWm9vrEmuDap4Ijj2wV3jON9XMFQUP3XrR+8H6//5bfbB+xU1zgvX973zmU/dUKTvPPT6x9q0Dv1/Utg9fPDtYH3/XzmB96GeK/DtRAqX4ks9Ud3+rBNsBUEG87AciVWz4XdIvzOw5M5tVioYAVEaxL/snu/smMztA0lIz+527f+QNau4fhVmS1EfZv88B0KmoI7+7b8r9uVXSIkmfODvG3ee5e4O7N9SpdzG7A1BCBYffzPqZ2YC9tyWdLumFUjUGoLyKedk/XNIiM9u7nR+7++Ml6QpA2Zm7V2xnA22oH2+nlGXbr90/MVh/ecrdZdmvJP3g3bHB+uodBwbrG3cOLmr/tZZ8dvnPxj9a1Lab2sLfE/jKhVcF6zVPPV/U/kN6jfyDYP3GpxYl1o6oD7/oPePFPw/We0/bHKx7655gvVxW+jJt9xbL57FM9QGRIvxApAg/ECnCD0SK8AORIvxApHrUpbv9hD9MrC38kx+mjA7/pz6+K/zV47lfn5lYG7Au5aTGbS3Bcs07b4bHp/Ca2sTaYTdfGRz74vnfC9YPqesfrO/6++3B+qBLhifW2raEl8FO89bJBwXradN5QTcfECx764bCt10lOPIDkSL8QKQIPxApwg9EivADkSL8QKQIPxCpHjXP3zoo+TLTE+rD/ykdCp+6/PV7Lg3WRy96OrEWXgS7AjqSOzj06l8Hhx5RHz4ld830W4P1J4/5z2D9xFOTv2cw6L7i5vm3Tm4L1msCx7apa88Lju33+KqCeupJOPIDkSL8QKQIPxApwg9EivADkSL8QKQIPxCpHjXP394nrysSd+vYpy8J1g+8MXkef182bvbKYH3JqSOC9fP6vx2sv3t28lLVg+4LDlWvg0YH63MmLw3WO5R8SXOfHz5fX3o9pd7zceQHIkX4gUgRfiBShB+IFOEHIkX4gUgRfiBSqfP8ZjZf0lmStrr70bn7hkpaKGmMpCZJ57v7O+Vrs9P4v1tX8Nja5waUsJN4fGPVOcH6eVPDS5/PPmpFYm2JhgTHvnr5qGD94cEPB+tvtCUvk913WzZLaFeTfI78P5J0xsfuu1bSMncfJ2lZ7ncAPUhq+N19haSPLzkzXdKC3O0FksKHBwBVp9D3/MPdfXPu9hZJyWsyAahKRX/g5+4uJV8gz8xmmVmjmTW2anexuwNQIoWGv9nMRkhS7s+tSQ9093nu3uDuDXXqXeDuAJRaoeFfLGnvsrUzJT1SmnYAVEpq+M3sAUnPSBpvZhvN7DJJcyWdZmbrJZ2a+x1AD5I6z+/uMxJKp5S4F9Uce3iwPmVw8vnbr7R+EBw7bE1rQT3FbsiTfcIPmFq+fdth7xU1/oF3/yixVrt8dVHb3hfwDT8gUoQfiBThByJF+IFIEX4gUoQfiFRVXbp7/czBwfoF/bcl1iavuTg4duBj+/6Sy/uaOz53f1Hj/+PHUxJrIxXnpdq74sgPRIrwA5Ei/ECkCD8QKcIPRIrwA5Ei/ECkqmqe/+ozfxash07brb9t/5Stv1ZAR8hSjSUvsS1JdVYbrI/64obE2stjJwXHjj9sU7D+6PjFwXpab63enlhbtHNocOw/3XFh8nbv+3VwbFcc+YFIEX4gUoQfiBThByJF+IFIEX4gUoQfiFRVzfOn+eHbJyXW+ix5toKdoBI6PHxsavXw5dgfGR9Ywnt8IR19KPwNBOmWlnHB+uwhLyfWpvd7Kzh2+jW3JtZOXJq4eNYncOQHIkX4gUgRfiBShB+IFOEHIkX4gUgRfiBSqfP8ZjZf0lmStrr70bn7bpB0uaS9F9K/zt0fS91Wba1qBw5KrA+o2ZhHy0B+NrTtSaz9TdO5wbFvLjw4WN/vrfBM/8Ala4L1h846PbHW8eXk9Skk6YljHwzW85XPkf9Hks7o5v7vuvuE3E9q8AFUl9Twu/sKSS0V6AVABRXznv8qM1tjZvPNbEjJOgJQEYWG/3ZJh0iaIGmzpJuTHmhms8ys0cwa9/iuAncHoNQKCr+7N7t7u7t3SLpTUuLVEN19nrs3uHtDvfUttE8AJVZQ+M1sRJdfvyTphdK0A6BS8pnqe0DSFEnDzGyjpG9KmmJmEyS5pCZJV5SxRwBlkBp+d5/Rzd13F7KzPcP6aOPFRyXWLxywPDh+9c4xhewWRdg97f+KGv9+R33BY7/8878M1l+Zfnuw/qf/MzuxdvBfPB8ce4C2BOtp0s737/+T5Ovr1y4LX7f/nl+NSay93f52yp4/xDf8gEgRfiBShB+IFOEHIkX4gUgRfiBSPerS3Si9tpM/F6w/OPH7KVvoHawu+udTEmuDFF5Our4lvMx1mouOSr6c+9MqfAqy3NrfDp9Hd8ua5Oe0edf6vPfDkR+IFOEHIkX4gUgRfiBShB+IFOEHIkX4gUgxz7+PS5vHb5mzM1g/vC48j3/lphOD9cELVyfWPDhS6vW+Bet1Fv4ewIDaDxJrNfsNDo7teP/9YL2c2qccF6zfO+nOxNqlKct7d8WRH4gU4QciRfiBSBF+IFKEH4gU4QciRfiBSFV0nr92jzSwqT2x3tSW3dxqT2a9kv83vnv1juDYxuPCyz0v3RVeZemVf0i+FLsk1bc2Busho256Olg/8riLgvXfnHBPYu2Ouz8fHHvIrNeD9Y4d4ec1Te0R4xJr4/51XXDsxN7JFwbfL/zViI/gyA9EivADkSL8QKQIPxApwg9EivADkSL8QKTMPXxWtZmNlnSvpOHqPAV7nrvfamZDJS2UNEZSk6Tz3f2d0LYG2lA/3pKvOX72i+HlhTs8+d+qx794dHBs28ZNwXqWOiZPCNbfuDI8/twjkpebvumA5PPp8zF19leC9b4PJ18bv9x6fXZ4sP5XTz2RWJva973g2GNWhJcHH/Zw+PsPrf3CE+43XndXYu0LfcPfd1m+q39ibc7017R+7a68ZvvzOfK3SbrG3Y+U9MeSZpvZkZKulbTM3cdJWpb7HUAPkRp+d9/s7qtzt3dIeknSSEnTJS3IPWyBpHPK1SSA0vtU7/nNbIykiZJWShru7ptzpS3qfFsAoIfIO/xm1l/SQ5K+5u7bu9a884ODbj88MLNZZtZoZo2t2l1UswBKJ6/wm1mdOoN/v7v/NHd3s5mNyNVHSNra3Vh3n+fuDe7eUJeyqCOAykkNv5mZpLslveTut3QpLZY0M3d7pqRHSt8egHLJ55TeEyVdLGmtme2dU7pO0lxJPzGzyyRtkHR+eVr80JWD30isNS8ZGBzb2HJgqdspmblj5wXrE+oLP/P6uT3Jp1BL0sXPXhasH/LL3wXr4a2XV9uW5mD9O5dfmFy88/7g2LUnJU/FSZJOCpdrUo6rHUo+LXfGa9OCY7d/c3RirXnDbeHGukj9W+XuT0lKmjdMnrQHUNX4hh8QKcIPRIrwA5Ei/ECkCD8QKcIPRCr1lN5SSjult+XSE4Ljp81ZkVi7ftjagvuqdm0ps+m/3ZNcu2jhV4Njx177TCEt9XgdX5gYrNdcvy1YX3z4omD9sg2nBeurlh+RWDv4W78Jju34IHnp8ZW+TNu9pWSn9ALYBxF+IFKEH4gU4QciRfiBSBF+IFKEH4hUVc3zp6k9dGxiberDa4Jj/3rI+oL3W26HP3lpsF6/dr9gfdS3w0tZIx7M8wNIRfiBSBF+IFKEH4gU4QciRfiBSBF+IFI9ap4fQBjz/ABSEX4gUoQfiBThByJF+IFIEX4gUoQfiFRq+M1stJktN7MXzWydmc3J3X+DmW0ys+dzP+FFxQFUlV55PKZN0jXuvtrMBkh6zsyW5mrfdffvlK89AOWSGn533yxpc+72DjN7SdLIcjcGoLw+1Xt+MxsjaaKklbm7rjKzNWY238yGJIyZZWaNZtbYqt1FNQugdPIOv5n1l/SQpK+5+3ZJt0s6RNIEdb4yuLm7ce4+z90b3L2hTr1L0DKAUsgr/GZWp87g3+/uP5Ukd29293Z375B0p6RJ5WsTQKnl82m/Sbpb0kvufkuX+0d0ediXJL1Q+vYAlEs+n/afKOliSWvN7PncfddJmmFmEyS5pCZJV5SlQwBlkc+n/U9J6u784MdK3w6ASuEbfkCkCD8QKcIPRIrwA5Ei/ECkCD8QKcIPRIrwA5Ei/ECkCD8QKcIPRIrwA5Ei/ECkCD8QqYou0W1m2yRt6HLXMElvVayBT6dae6vWviR6K1QpezvI3T+TzwMrGv5P7Nys0d0bMmsgoFp7q9a+JHorVFa98bIfiBThByKVdfjnZbz/kGrtrVr7kuitUJn0lul7fgDZyfrIDyAjmYTfzM4ws5fN7FUzuzaLHpKYWZOZrc2tPNyYcS/zzWyrmb3Q5b6hZrbUzNbn/ux2mbSMequKlZsDK0tn+txV24rXFX/Zb2a1kl6RdJqkjZJWSZrh7i9WtJEEZtYkqcHdM58TNrOTJL0n6V53Pzp3379IanH3ubl/OIe4+99WSW83SHov65WbcwvKjOi6srSkcyRdogyfu0Bf5yuD5y2LI/8kSa+6++vuvkfSg5KmZ9BH1XP3FZJaPnb3dEkLcrcXqPMvT8Ul9FYV3H2zu6/O3d4hae/K0pk+d4G+MpFF+EdKerPL7xtVXUt+u6RfmNlzZjYr62a6MTy3bLokbZE0PMtmupG6cnMlfWxl6ap57gpZ8brU+MDvkya7+3GSzpQ0O/fytip553u2apquyWvl5krpZmXp38vyuSt0xetSyyL8mySN7vL7qNx9VcHdN+X+3Cppkapv9eHmvYuk5v7cmnE/v1dNKzd3t7K0quC5q6YVr7MI/ypJ48xsrJnVS7pA0uIM+vgEM+uX+yBGZtZP0umqvtWHF0uambs9U9IjGfbyEdWycnPSytLK+LmruhWv3b3iP5KmqfMT/9ckfSOLHhL6OljSb3M/67LuTdID6nwZ2KrOz0Yuk7S/pGWS1kv6b0lDq6i3f5e0VtIadQZtREa9TVbnS/o1kp7P/UzL+rkL9JXJ88Y3/IBI8YEfECnCD0SK8AORIvxApAg/ECnCD0SK8AORIvxApP4fYskg6RI7YOkAAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "img = np.zeros((28,28))\n", "img[:,:-6] += mnist.test.images[10].reshape(28,28)[:,6:]\n", "img[:,14:] += mnist.test.images[21].reshape(28,28)[:,5:19]\n", "img /= img.max()\n", "plt.imshow(img)\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Below, you can see the prediction of the neural network trained with softmax cross entropy for this example. The prediction of the network is digit 2 with probability 0.775. Hence, the network associates quite high probability with the wrong label. " ] }, { "cell_type": "code", "execution_count": 331, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "softmax prob: [0. 0.199 0.775 0.007 0.003 0. 0. 0.015 0. 0. ]\n" ] } ], "source": [ "p1 = sess1.run(prob1, feed_dict={X1:img.reshape(1,-1), keep_prob1:1.0})\n", "print('softmax prob: ', np.round(p1[0], decimals=3))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "On the otherhand, when we do the same using the network trained based on the loss in Eq. 7, the output of the neural network is uniform distribution with uncertainty 1.0, as shown below." ] }, { "cell_type": "code", "execution_count": 332, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "uncertainty: 1.0\n", "Dirichlet mean: [0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1]\n" ] } ], "source": [ "uncertainty2, p2 = sess2.run([u, prob2], feed_dict={X2:img.reshape(1,-1), keep_prob2:1.0})\n", "print('uncertainty:', np.round(uncertainty2[0,0], decimals=2))\n", "print('Dirichlet mean: ', np.round(p2[0], decimals=3))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The neural networks, trained using the loss functions defined in Eq. 5 and Eq. 6 in the paper, also have very high uncertainty for their predictions. These networks assing small amount of evidence for the classification of the image as digit 2. However, they associate very high uncertainty with their misclassifications of the image." ] }, { "cell_type": "code", "execution_count": 333, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "uncertainty: 0.92\n", "Dirichlet mean: [0.092 0.094 0.143 0.12 0.092 0.092 0.092 0.093 0.092 0.092]\n" ] } ], "source": [ "uncertainty3, p3 = sess3.run([u3, prob3], feed_dict={X3:img.reshape(1,-1), keep_prob3:1.0})\n", "print('uncertainty:', np.round(uncertainty3[0,0], decimals=2))\n", "print('Dirichlet mean: ', np.round(p3[0], decimals=3))" ] }, { "cell_type": "code", "execution_count": 334, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "uncertainty: 0.93\n", "Dirichlet mean: [0.093 0.093 0.16 0.098 0.093 0.093 0.093 0.094 0.093 0.093]\n" ] } ], "source": [ "uncertainty4, p4 = sess4.run([u4, prob4], feed_dict={X4:img.reshape(1,-1), keep_prob4:1.0})\n", "print('uncertainty:', np.round(uncertainty4[0,0], decimals=2))\n", "print('Dirichlet mean: ', np.round(p4[0], decimals=3))" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [] } ], "metadata": { "accelerator": "GPU", "colab": { "collapsed_sections": [], "default_view": {}, "name": "EDL MNIST Demo.ipynb", "provenance": [ { "file_id": "1AAN37ioPFkhPfTxBaLeFywBwIliguMu1", "timestamp": 1527923401234 } ], "version": "0.3.2", "views": {} }, "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.4" } }, "nbformat": 4, "nbformat_minor": 1 }