{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Artificial Intelligence\n", "## L2 International, Univ. Bordeaux" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Lab #5, Supervised Learning (4)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In this lab, we will see how to build a neural network with the Keras framework in Python. As usual, we give an incomplete python program. Some instructions will be used as they are and you do not need to change them. We will comment some of them but we do not need to understand them all. We will use the dataset MNIST, one of the most used datasets in machine learning research. This dataset consists of 70000 images of handwritten digits (of size 28x28)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Artificial neural networks\n", "\n", "Recall that an artificial neural network (ANN for short) consists of a set of layers disposed linearly, and each layer is a set of (artificial) neurons. The model is simplified so that signals can only circulate from the bottom layer to the top layer. Each neuron in the $k$-th layer of the neural network is connected to all the neurons in the $(k − 1)$-th layer, and the neurons in a given layer are all independent from each other.\n", "\n", "A neural network consists of the following components\n", "\n", " - An input layer, $x$\n", " \n", " - An arbitrary amount of hidden layers\n", " \n", " - An output layer, $y$\n", " \n", " - A set of weights and biases between each layer, $w$ and $b$\n", "\n", " - A choice of activation function for each hidden layer, $\\sigma$." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As usual, we need to import necessary python modules:" ] }, { "cell_type": "code", "execution_count": 21, "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "import pandas as pa" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "And the, load the dataset:" ] }, { "cell_type": "code", "execution_count": 22, "metadata": {}, "outputs": [], "source": [ "from keras.datasets import mnist\n", "(X_train, y_train), (X_test, y_test) = mnist.load_data()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can start by some manipulations to see what tha data looks like:" ] }, { "cell_type": "code", "execution_count": 23, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "X_train shape: (60000, 28, 28)\n", "Y_train shape: (60000,)\n" ] } ], "source": [ "print('X_train shape: ',X_train.shape)\n", "print('Y_train shape: ',y_train.shape)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can see that we have 60k images. Each image correspond to 28x28 pixels.\n", "\n", "We can also visualise an image:" ] }, { "cell_type": "code", "execution_count": 25, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAD4CAYAAAAq5pAIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAOKklEQVR4nO3df6zV9X3H8ddLuIBFbEEHIiXKDNkkzcR5g3ZuxkbrqP2hbXWRuMYlrPizK5n7YXRL/WNNWNfqzNpqoDJxUZuuLZOkppZSG9f9IF4oAkpb/EEVYWBHW7EVuMh7f9zDctX7/ZzL+e19Px/JzTnn+z7f833ncF98zz2f7/f7cUQIwNh3XLcbANAZhB1IgrADSRB2IAnCDiQxvpMbm+CJMUmTO7lJIJUD+pUOxUGPVGsq7LYXSrpL0jhJX4mIZaXnT9JkneuLmtkkgIL1sa6y1vDHeNvjJH1J0gckzZO0yPa8Rl8PQHs18zf7AknPRMRzEXFI0lclXdaatgC0WjNhnyXpxWGPd9aWvYHtJbYHbA8M6mATmwPQjGbCPtKXAG859jYilkdEf0T092liE5sD0Ixmwr5T0uxhj98taVdz7QBol2bC/oSkubbn2J4g6SpJa1rTFoBWa3joLSIO275J0qMaGnpbGRFPtawzAC3V1Dh7RDwi6ZEW9QKgjThcFkiCsANJEHYgCcIOJEHYgSQIO5AEYQeSIOxAEoQdSIKwA0kQdiAJwg4kQdiBJAg7kARhB5Ig7EAShB1IgrADSRB2IAnCDiRB2IEkCDuQBGEHkiDsQBKEHUiCsANJEHYgCcIOJEHYgSSamsUVY99x8+cV6z9eenyxvv39Kypr41ze1/z6yKFi/b2fX1qsn3rPxsrakQMHiuuORU2F3fYOSfslvS7pcET0t6IpAK3Xij37+yLiZy14HQBtxN/sQBLNhj0kfcf2BttLRnqC7SW2B2wPDOpgk5sD0KhmP8afHxG7bE+XtNb2jyLi8eFPiIjlkpZL0omeFk1uD0CDmtqzR8Su2u1eSaslLWhFUwBar+Gw255se8rR+5IukbS1VY0BaK1mPsbPkLTa9tHXeTAivt2SrtAyHl/+J971Z+UPY1/51F3F+jkTxh1zT0f9x4Ejxfp5E8u9b/zLLxbrH3rs6urik9uK645FDYc9Ip6TdFYLewHQRgy9AUkQdiAJwg4kQdiBJAg7kASnuI4Be2/4vcraL+YPFtd95oPl4SupPLT2vq0fL9aPrJheWZvyo18W15236ifF+udOGSjWT7p7d2Xt5eq3bMxizw4kQdiBJAg7kARhB5Ig7EAShB1IgrADSTDO/jbw4t+UB4WfvP6fKmvHycV1Nx06XKz/1eLri/XjH6u+XLMkKZ6vLJVPcJW2XTy1/IQ6V0/459PWVdYuWXhdcd0J336i/OJvQ+zZgSQIO5AEYQeSIOxAEoQdSIKwA0kQdiAJxtl7wLip5fHkpVf/W7FeGkvf/fqvi+v+xXXlaY8nfK98zng7xWuvFetf/sWcYv2Gd1WP8Uf58IMxiT07kARhB5Ig7EAShB1IgrADSRB2IAnCDiTBOHsP8NR3FuuLT9zZ8Gtf8PDNxfrcR9c3/NrtduTAgWL9/ufPLdZvOLt6nD2junt22ytt77W9ddiyabbX2t5eu61zlQEA3Taaj/H3SVr4pmW3SFoXEXMlras9BtDD6oY9Ih6XtO9Niy+TtKp2f5Wky1vcF4AWa/QLuhkRsVuSareVE3rZXmJ7wPbAoA42uDkAzWr7t/ERsTwi+iOiv08T2705ABUaDfse2zMlqXa7t3UtAWiHRsO+RtI1tfvXSHq4Ne0AaJe64+y2H5J0oaSTbe+U9BlJyyR9zfZiSS9IurKdTY51gzPf1dT6LxXOWf+tFeU50Otdux1jR92wR8SiitJFLe4FQBtxuCyQBGEHkiDsQBKEHUiCsANJcIprD3j2iklNrX/Jf1dPq3za5i1NvTbGDvbsQBKEHUiCsANJEHYgCcIOJEHYgSQIO5AE4+wdMH7WqcX63R++t6nXH/fDKU2t36uOe8c7ivXP/vbqDnUyNrBnB5Ig7EAShB1IgrADSRB2IAnCDiRB2IEkGGfvgF+dNatYv+j45qbFmvjzaGr9XuXx5V/Peu/b/x55rbLW9+rhhnp6O2PPDiRB2IEkCDuQBGEHkiDsQBKEHUiCsANJMM4+Bsx4YGtlLfOUzKt++TuVteP+/Ycd7KQ31N2z215pe6/trcOW3W77Jdubaj+XtrdNAM0azcf4+yQtHGH5nRExv/bzSGvbAtBqdcMeEY9L2teBXgC0UTNf0N1ke3PtY/7UqifZXmJ7wPbAoJo7BhxA4xoN+92SzpA0X9JuSV+oemJELI+I/ojo79PEBjcHoFkNhT0i9kTE6xFxRNIKSQta2xaAVmso7LZnDnv4UUnVYz8AekLdcXbbD0m6UNLJtndK+oykC23PlxSSdki6to09Iqmf3vieOs/4frH64D1/WFmbrv889obe5uqGPSIWjbC4uVkNAHQch8sCSRB2IAnCDiRB2IEkCDuQBKe4dsCkdZuL9Qf2Ty/Wr56yt5Xt9Izxc04r1r/0p/c09fqnfuulylq+C0mzZwfSIOxAEoQdSIKwA0kQdiAJwg4kQdiBJBhn74A4WL4c14GY0KFOesuei08t1v9gUnk0/GDUGS2PsTmVdaPYswNJEHYgCcIOJEHYgSQIO5AEYQeSIOxAEoyzjwVnzK6ubXq6c32MYPxp1b197FPfK65bbxz9vf+wtFg/ZUe+y0WXsGcHkiDsQBKEHUiCsANJEHYgCcIOJEHYgSQYZ+8Bf//oR4r1xVd+uVh/9qp3VtbmbGqopVHz+PKv0NO3nVJZW3PSw8V1v3/g+GL9lLsYRz8Wdffstmfbfsz2NttP2f50bfk022ttb6/dTm1/uwAaNZqP8Ycl3RwRZ0o6T9KNtudJukXSuoiYK2ld7TGAHlU37BGxOyI21u7vl7RN0ixJl0laVXvaKkmXt6tJAM07pi/obJ8u6WxJ6yXNiIjd0tB/CJJGnLDM9hLbA7YHBlW+FhuA9hl12G2fIOkbkpZGxCujXS8ilkdEf0T092liIz0CaIFRhd12n4aC/kBEfLO2eI/tmbX6TEljc6pRYIyoO/Rm25LulbQtIu4YVloj6RpJy2q35XEUVJq61eUnXFku/93HHqysrfrH84rrHv6fPeUXr2PPdQuK9Wc++MXK2pZDg8V1P3vtJ4v1Pm0o1vFGoxlnP1/SJyRtsX101PZWDYX8a7YXS3pBdX8lAXRT3bBHxA8kVe16LmptOwDahcNlgSQIO5AEYQeSIOxAEoQdSIJTXHvAjG89X6xvuq18SeWPT/55Ze2Wvz29uO6Zy/qK9e03FC5TLenri+4o1qXq6aiv+Hr5UtBnfPe/6rw2jgV7diAJwg4kQdiBJAg7kARhB5Ig7EAShB1IwhHRsY2d6GlxrjlR7lgNXnxOsb76vupzxk9w+epAGw69XqyfVT1MLkkar3HF+gVbrqisTfnQC8V143D5+AK81fpYp1di34hnqbJnB5Ig7EAShB1IgrADSRB2IAnCDiRB2IEkOJ/9baDvu+Xroy+4788ra//6x3cW1z1nQp2B9Drmrr6+WD9z2c7K2mHG0TuKPTuQBGEHkiDsQBKEHUiCsANJEHYgCcIOJFH3fHbbsyXdL+kUSUckLY+Iu2zfLumTkl6uPfXWiHik9Fqczw60V+l89tEcVHNY0s0RsdH2FEkbbK+t1e6MiM+3qlEA7TOa+dl3S9pdu7/f9jZJs9rdGIDWOqa/2W2fLulsSetri26yvdn2SttTK9ZZYnvA9sCgDjbVLIDGjTrstk+Q9A1JSyPiFUl3SzpD0nwN7fm/MNJ6EbE8Ivojor9P5euhAWifUYXddp+Ggv5ARHxTkiJiT0S8HhFHJK2QtKB9bQJoVt2w27akeyVti4g7hi2fOexpH5W0tfXtAWiV0Xwbf76kT0jaYntTbdmtkhbZni8pJO2QdG1bOgTQEqP5Nv4HkkYatyuOqQPoLRxBByRB2IEkCDuQBGEHkiDsQBKEHUiCsANJEHYgCcIOJEHYgSQIO5AEYQeSIOxAEoQdSKLupaRbujH7ZUk/HbboZEk/61gDx6ZXe+vVviR6a1QrezstIn5jpEJHw/6WjdsDEdHftQYKerW3Xu1LordGdao3PsYDSRB2IIluh315l7df0qu99WpfEr01qiO9dfVvdgCd0+09O4AOIexAEl0Ju+2Ftn9s+xnbt3Sjhyq2d9jeYnuT7YEu97LS9l7bW4ctm2Z7re3ttdsR59jrUm+3236p9t5tsn1pl3qbbfsx29tsP2X707XlXX3vCn115H3r+N/stsdJ+omk90vaKekJSYsi4umONlLB9g5J/RHR9QMwbF8g6VVJ90fEe2rLPidpX0Qsq/1HOTUi/rpHertd0qvdnsa7NlvRzOHTjEu6XNKfqIvvXaGvP1IH3rdu7NkXSHomIp6LiEOSvirpsi700fMi4nFJ+960+DJJq2r3V2nol6XjKnrrCRGxOyI21u7vl3R0mvGuvneFvjqiG2GfJenFYY93qrfmew9J37G9wfaSbjczghkRsVsa+uWRNL3L/bxZ3Wm8O+lN04z3zHvXyPTnzepG2EeaSqqXxv/Oj4jflfQBSTfWPq5idEY1jXenjDDNeE9odPrzZnUj7DslzR72+N2SdnWhjxFFxK7a7V5Jq9V7U1HvOTqDbu12b5f7+X+9NI33SNOMqwfeu25Of96NsD8haa7tObYnSLpK0pou9PEWtifXvjiR7cmSLlHvTUW9RtI1tfvXSHq4i728Qa9M4101zbi6/N51ffrziOj4j6RLNfSN/LOSbutGDxV9/aakJ2s/T3W7N0kPaehj3aCGPhEtlnSSpHWSttdup/VQb/8iaYukzRoK1swu9fb7GvrTcLOkTbWfS7v93hX66sj7xuGyQBIcQQckQdiBJAg7kARhB5Ig7EAShB1IgrADSfwfMuQp1XNnKkQAAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "0\n" ] } ], "source": [ "from matplotlib import pyplot as plt\n", "%matplotlib inline \n", "plt.imshow(X_train[1000])\n", "plt.show()\n", "\n", "print(y_train[1000])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now, we need to preprocess our data. In this lab, we will use \"simple\" architecture for our ANN. Thus, we need to flatten the images to transform them into a vector of size 28x28 = 784 :" ] }, { "cell_type": "code", "execution_count": 26, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "(60000, 784)\n" ] } ], "source": [ "num_pixels = X_train.shape[1] * X_train.shape[2]\n", "X_train = X_train.reshape(X_train.shape[0], num_pixels)\n", "X_test = X_test.reshape(X_test.shape[0], num_pixels)\n", "\n", "y_train = y_train.reshape(y_train.shape[0], 1)\n", "y_test = y_test.reshape(y_test.shape[0], 1)\n", "print(X_train.shape)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We also transform the values of pixels into floats and then normalise the values:" ] }, { "cell_type": "code", "execution_count": 27, "metadata": {}, "outputs": [], "source": [ "X_train = X_train.astype('float32')\n", "X_test = X_test.astype('float32')\n", "y_train = y_train.astype('float32')\n", "y_test = y_test.astype('float32')\n", "X_train = X_train / 255\n", "X_test = X_test / 255" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now, we can define our network. We need to load some necessary libraries:" ] }, { "cell_type": "code", "execution_count": 28, "metadata": {}, "outputs": [], "source": [ "from keras.models import Sequential \n", "from keras.layers import Dense \n", "from keras.optimizers import adam" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We also use \"one-hot\" encoding to encode the classes:" ] }, { "cell_type": "code", "execution_count": 29, "metadata": {}, "outputs": [], "source": [ "y_train = keras.utils.to_categorical(y_train)\n", "y_test = keras.utils.to_categorical(y_test)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "then encode the ANN:" ] }, { "cell_type": "code", "execution_count": 30, "metadata": {}, "outputs": [], "source": [ "num_classes = y_train.shape[1]\n", "\n", "def neural_network():\n", " model = Sequential()\n", " model.add(Dense(num_pixels, input_dim=num_pixels, kernel_initializer='normal', activation='relu'))\n", " model.add(Dense(num_classes, kernel_initializer='normal', activation='softmax')) \n", " model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])\n", " return model\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We instantiate the model and print it:" ] }, { "cell_type": "code", "execution_count": 31, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Model: \"sequential_5\"\n", "_________________________________________________________________\n", "Layer (type) Output Shape Param # \n", "=================================================================\n", "dense_4 (Dense) (None, 784) 615440 \n", "_________________________________________________________________\n", "dense_5 (Dense) (None, 10) 7850 \n", "=================================================================\n", "Total params: 623,290\n", "Trainable params: 623,290\n", "Non-trainable params: 0\n", "_________________________________________________________________\n" ] } ], "source": [ "model = neural_network()\n", "model.summary()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "and train our supervised learning model:" ] }, { "cell_type": "code", "execution_count": 32, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Train on 60000 samples, validate on 10000 samples\n", "Epoch 1/1\n", "60000/60000 [==============================] - 3s 54us/step - loss: 0.2296 - accuracy: 0.9341 - val_loss: 0.1203 - val_accuracy: 0.9661\n", "10000/10000 [==============================] - 0s 20us/step\n", "Neural network accuracy: 96.61%\n" ] } ], "source": [ "import warnings\n", "warnings.filterwarnings(\"ignore\")\n", "\n", "model.fit(X_train, y_train, validation_data=(X_test, y_test), epochs=1, batch_size=100)#98.36%\n", "\n", "scores = model.evaluate(X_test, y_test)\n", "print(\"Neural network accuracy: %.2f%%\" % (scores[1]*100))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### To Do:\n", "\n", "Try different hyper parameters and find the best ones to get the best accuracy" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.4" } }, "nbformat": 4, "nbformat_minor": 2 }