{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "view-in-github"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/SotaYoshida/Lecture_DataScience/blob/main/notebooks/Python_chapter_ArtificialNeuralNetwork.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sKor3X3ievgT"
      },
      "source": [
        "# 機械学習: ニューラルネットワークによる回帰"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4qZBy99AnBTn"
      },
      "source": [
        "\n",
        "\n",
        "この章では、最も単純な、入力層・隠れ層・出力層からなるニューラルネットワークを使って、データから尤もらしい予測を与える関数を構築してみましょう。\n",
        "\n",
        "対象とする(疑似)データは、多項式回帰の際に用いたデータと同じsin関数＋ノイズで生成することにします。\n",
        "\n",
        "\n",
        "* すすんだ注: このノートブックでは「ニューラルネットワークをPythonで表現してみる」ことに重きをおくため、使用するデータを訓練データ,検証データ,テストデータに分けることはせず、データは全てニューラルネットワークの訓練データとして使うこととします。\n",
        "授業で説明するとおり、一般に[教師あり学習]の文脈でニューラルネットワークを考える際は、本来データを上の様に複数用途に分けながら、モデル選択を行ったり、汎化性能の評価に使ったりします。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 135,
      "metadata": {
        "id": "nme6EtxcerdN",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "import numpy as np\n",
        "def create_toy_data(sample_size, std):\n",
        "    np.random.seed(1234) #毎回同じデータになるように乱数の種を固定しておく \n",
        "    x = np.linspace(0, 0.5, sample_size)\n",
        "    t = np.sin(2*np.pi*x) + np.random.normal(scale=std, size=x.shape)                                                                                                                 \n",
        "    return x, t\n",
        "\n",
        "xt,yt = create_toy_data(40,5.e-2) "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 136,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 279
        },
        "id": "kZDRP8GFCSl1",
        "outputId": "b2d6c33e-26ad-45d1-ed2a-30b914602522",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmEAAAEGCAYAAAAt7EI0AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deXiV1dX38e8iTJZBEIIiszLJLIShoqIWEVBBBRUUW62KqDjUPj7Vt9a59qljrdJWFCdUEKwiAooWZ0AgKMokiMgQpAVRqcgM+/1jBTOdQICcc+ec8/tcFxece0hWuCFZZ++117YQAiIiIiKSWOWiDkBEREQkHSkJExEREYmAkjARERGRCCgJExEREYmAkjARERGRCJSPOoD9Vbt27dC4ceOowxARERHZp7lz534TQsiMdS7pkrDGjRuTnZ0ddRgiIiIi+2RmK4s7p+lIERERkQgoCRMRERGJgJIwERERkQgkXU2YiIiIJIcdO3aQk5PD1q1bow4l7ipXrkz9+vWpUKFCie9REiYiSWX2bPjb32DJEmjQAC6/HE49NeqoRCSWnJwcqlWrRuPGjTGzqMOJmxACGzZsICcnhyZNmpT4Pk1HikjSeOopOOssaNsW7r8ffvELGDYM7rgj6shEJJatW7dSq1atlE7AAMyMWrVq7feIn0bCRCQpbNgAv/mNj4Q1b+7HuneHc87xpGzgQGjdOtoYRaSoVE/A9jiQr1MjYSKSFF55BU47LS8B2yMzEy6+GF54IZKwREQOmJIwEUkK338PRx4Z+9yRR/p5EUluu3bB9OnwxhvwzTel8zEzMjLo0KEDrVu3pn379jzwwAPs3r17r/esWLGCFxLwzk5JmIgkha5dYcoUiPW9c/Jk6NYt8TGJSOl56y1o2hSuvhoeeACaNYPrr4edOw/u4x5yyCHMmzePhQsX8tZbb/H6669zxz4KSZWEiYjkc/zxUKeO14Vt2eLHduyA++6DL7+Ec8+NNj4ROXALFsCFF8ITT8C8eZ6QLVvmx3//+9L7PHXq1GHkyJE8+uijhBBYsWIFJ5xwAh07dqRjx47MmDEDgJtuuokPPviADh068NBDDxV73cGyEEKpfKBEycrKCto7UiQ9ffstXHopfPihF+MvWeLvnJ95Bho3jjo6ESls8eLFHHPMMfu87rLL4Oij4eabCx5fuxZatYKVK6F69QOLoWrVqmzatKnAsRo1arBkyRKqVatGuXLlqFy5Ml988QWDBw8mOzubd999l/vvv59JkyYBsHnz5pjXleTrNbO5IYSsWLFpdaSIJI3DDvMC/ZUr/V1y/frQokXUUYnIwZo926chC6tb199oLVoUn5KDHTt2MHz4cObNm0dGRgZLly49qOv2l5IwEUk6jRr5LxFJDTVrwpo1cOyxBY/v2uWjYTVrlt7nWr58ORkZGdSpU4c77riDww8/nE8//ZTdu3dTuXLlmPc89NBDJbpuf6kmTERERCJ10UVe37ljR8Hjzz0H9eqV3oj3+vXrGTZsGMOHD8fM2LhxI3Xr1qVcuXKMHj2aXbt2AVCtWjV++OGHn+4r7rqDpSRMRCQNfPYZXHEFnHACnH8+vPlm1BGJ5PnVr+DQQ+HEE2HMGPjXv+Daa+F3v4PHHju4j71ly5afWlT07NmTXr16cdtttwFw1VVX8cwzz9C+fXs+//xzqlSpAkC7du3IyMigffv2PPTQQ8Ved7DiVphvZk8CZwDrQghtYpw34GGgL7AZuDiE8PG+Pq4K80VSx7ZtMGECLF3q+0AOHAhVq0YdVeoZPx6GD/cfascf7wsa7rvPV5Tec0/U0UkqK2lhPngrinHjPAn74Qf/t3rllT4Sliz2tzA/nknYicAm4NlikrC+wDV4EtYVeDiE0HVfH1dJmEhq+PRTOPNM74DfrRvMnw8zZvg34ZNPjjo6TxBfeQU+/9wXAJx33oGvztqbjRt9ymXxYv9h88tfluyHzoIFXqxcrx4cdxwUt2PKpk1ePzdtGnTokHf822+hfXtPgjt1Kp2vRaSw/UnCUsH+JmFxm44MIbwPfLuXS/rjCVoIIXwE1DCzuvGKR0TKju3boV8/uPden3a4+2549VVPwM47zxOEKM2f740iR42CELx791FHwdSppft5Zs3yJPSDD/z3Vau89caYMcXfs26db1zeu7f/fV1xhS/h//TT2Ne/9hr8/OcFEzDwlaaXX+4JoIhEI8rVkfWA1fle5+QeW1v4QjMbCgwFaNiwYUKCE5GSC8H3bnzkEfjiC2jY0JODoUOhXIy3epMmQZMmMGhQweMnnwx9+sDo0XDddYmJvbCdOz1B/L//gwsuyDs+fTr07++jT3XqHPzn2b4dBgzw5pRnnpl3/OqroUcPH90qvAI0BDj7bK/rmjoVypfP+7vv08djq1Gj4D3ffVf8yFq9erB8+cF/LSJ7E0JIi028D2RmMSkK80MII0MIWSGErMzMzKjDEZFC/vAH+POf4ZZbfPrugQc8kbrsstjXf/EFdO4c+1znzl4jFpUpUzw5yZ+AAXTvDmedBU8/XXqfp2nTggkYQJs2MGRI7M8zc6bvp3fPPZ6AgU9DXnihFzSPHl30nq5dPWGLtZjr9dehS5eD/lJEilW5cmU2bNhwQAlKMgkhsGHDhv1uXRHlSNgaoEG+1/Vzj4lIElmxAv7+dy/2rl3bj51yitd5tW4Nc+YUTbgaNYK334798T79NNoGrMuWFZ8gZmXBx/tcPlQyq1Z5whVLmzbw0UdFj3/yif/dxhpd7NnTa+oK69TJp1avucaT40MO8f03R4706dAnnzy4r0Nkb+rXr09OTg7r16+POpS4q1y5MvXr19+ve6JMwiYCw81sLF6YvzGEUGQqUkTKtldf9Wm1PQnYHj/7mReZv/RS0aTmrLN8D8gpU6Bv37zjc+Z4MfzixfGPuzhNmngdVSzz5vn50tC8OTz7bOxzH33k5wurXduT3lhWrCj6DPYYP963e2rYEDp29JHGzEyvx4vHYgORPSpUqECT0vpPk4LiNh1pZmOAmUALM8sxs0vNbJiZDcu9ZAqwHFgGPA5cFa9YRCR+tm/3hCuWKlX8fGGVK8PLL8Mll3hbivvv9z5Bp53m03ClUXN1oE4/3UfDXnml4PHsbE9mLr449n0heJF7ly5QrRq0bOkjT4WbT+5x6qm+DH/kyILH33vPE9tLLil6z5lneqI6a1bB42vXem3ZL38Z+3PVqAH//Kff+5vf+MefPVtbPolETRt4i8hBmTfPC9mXLYOKFfOO797tI2B33+1F47H897++EnDJEh+lufBCH6GJ2ty5cMYZnlAddxwsXOiLCZ56yovzY7ntNk8s773X71m0CG691RtQjh8fu4XE0qWe9GVmes3ZokWeHI0d6ysgY5k0yRO0X//a+ygtXuwLIoYP98aWIlK2RNInLF6UhImUPQMHeuH3ww97MvWf/8D/+3+eXL3/fuwaprLuxx89GVqyxAv1L7yw+Om+NWu8tcTnnxccxdu2zffC+9vf4KSTYt+7c6cXyO/pE3b22cWPLO7x5Zfwj3/4PUce6VONXffZZVFEoqAkTETiats2Xxn55JNe+L1pk2+Nc9996VFz9Nhj3sIiVo3XvffC6tU+WiUi6WdvSViUhfkiUoatXes9pBo08NGtvalUyROuu+7yZqK1a+97NCeV7NjhdW6xVK5cfF2YiKS3JJwkEJF4+v57H8Vq3Rr+53+8xcHpp3tSti+VK3vClk4JGHiR/auvwubNBY+H4DVvvXtHE5eIlG1KwkTkJyF44Xnt2t7HauZMn0rr2BF69dKITnFatPBE9ayz8hrNrl3r2wKBF/mLiBSmJExEfvLBBz6d+MgjULWqH6tcGe68E2rW9NGeZPTFF3Djjd7i4aqrSq/han4jR/oejccf78X5xxzjXe3feCOvu72ISH5KwkTkJ9One6JSeDWjmbehmD49mrgOxoQJ3jIiI8O3Uapf30emHn64dD9P+fJwxx2+UnL+fPj3v30F46GHlu7nEZHUofdnIvKTatW8JUMs69b5+WTyww/eT+utt7y2DXy69aKLfIq1b1/f0qc0VagAhx9euh9TRFKTRsJE5CcDB/qU46pVBY9/8423Xxg8OJq4DtQrr/jG1nsSsD0aNPAO/bE2vBYRSRSNhInIT444wuu/TjgBbr7ZG4B+9hnccw8MHep1Tslk/fri93ps0gQWLEhsPOlg+3aYONGnZOvW9ZW2NWtGHZVI2aSRMBEp4Jpr4JlnYNo03yfxn/+Ehx7y5CzZHHusb1Idqyf1v/4FHTokPqZUtmSJ75k5YoS/fvddOPro5F3QIRJvGgkTSRM7d5Z8ld5JJxW/zU4yOflk38/yjjvgD3/w4vwQfGp1zhxNR5am3bu9RcfNN+e15gDfh/O00zwh3lfTX5F0o5EwkRS2axc88IBPvVWs6HsT3nWXTxmlAzN47TV4+2046ig47zxo08a3Enrjjbw2HHLw3nnH25lcdlnB4506+b6bo0ZFE5dIWaYkTCSFDRvm9Tnjx3tCNnWqN2AdNCj2FF0qOvJI30T81VdhwADv57VggSdjsndLl8JvfuONei++2P8ei7NsGXTu7IlvYVlZfl5EClISJpKiFi2CSZNgyhT/IWjmiceECZ6EzJgRdYSJ1aGDF4l37x47UZCCXnvN/65+9jO44QafTvzlL31qN5bGjWHevNjn5s0rfoGESDpTEiYSgRDg8cf9B1u1atC2rRcz795dep9j0iSffqtSpeDxihXhggv8h6xILJs3e3+1SZPgj3/0vS+vu87r6EaOhE8+KXpPz57w3Xfw3HMFj8+f7ws9Lr00MbGLJBMV5otE4PrrfSTq/vt9CmfePC9o/uQTeOKJ+H9+s/SZjpT9N3my13J17VrweGamtyoZPdrfQOSXkQEvv+wNcF94AXr08NWSEybA3/+ukTCRWDQSJpJgS5bA2LHeIuEXv4Dq1b2h6Jtv+q9YowwHom9frwXbvLng8R07/IekNpWW4mzYUPxKxkaNvHlvLG3beh3Z+ef7x2jbFj7/3F+LSFFKwkQSbMIE/6FUeE/BKlVgyBAfTSgNbdr4NNIZZ+TV6ixe7MXpLVv6RtMisWRl+ZuEXbuKnnvzTT9fnEMO8d0I7r3Xi/rr1IlfnCLJTkmYSILt2OFL+WOpXNnPl5bHH/dErF8/rwU75RTfM3H8eBWnS/GysrzQ/re/zWtnsqe/2nvveYG+iBw8C0lWGJKVlRWys7OjDkPkgM2Z4wXzS5Z4YrTHrl3Qrp0X6Jd2o9QQYNs2qFRJyZeUzIYNnmzNneu1YUuWeLPfMWN8mlFESsbM5oYQYo4fqzBfJME6d/Z2CYMGwYMP+ohDTg7ceKM3U+3Ro/Q/p1nxo28isdSq5QX6S5b4NHbdutCli5J4kdKk6UiRCIwZA02b+rTP4Yf7CNgRR3i9mH7ISVnSooVvR9S1q/5tipQ2TUeKlJJt23wK57DDSj7qtH173j2VKsU3PhERSby9TUdqJEzkIG3d6gXMdet6b6W6db2xZeHWELFUrOjXKwETEUk/SsJEDkIIXmS/cqW3gVi71juEr1sHZ5+thqgiIlI8JWEiB2HWLN+jccyYvOaW9ev71i2rV+99w2MREUlvSsJEDsKbb8LAgVChQsHjGRk+QjZ1ajRxiYhI2ackTOQgVKjgNWGxbNlSsA+YiIhIfnFNwsyst5ktMbNlZnZTjPMNzewdM/vEzD4zs77xjEektJ19tu8DuXFjweObNsHzz8M550QTl4iIlH1xS8LMLAMYAfQBWgGDzaxVoctuAcaFEI4FBgF/i1c8IvHQsqXvA9mzJ7z1Fnz3Hbz9Npx6Kpx+uvf/EhERiSWeHfO7AMtCCMsBzGws0B9YlO+aAFTP/fOhwNdxjEckLv7yF3jySfjd7+DLL6FJExg2DIYOjToyEREpy+KZhNUDVud7nQN0LXTN7cCbZnYNUAXoGcd4REpk506YMgWWL/eEqm/fooX3+ZnBpZf6LxERkZKKujB/MPB0CKE+0BcYbWZFYjKzoWaWbWbZ69evT3iQkj4++wyaNYN774WvvoL77/fX8+ZFHZmIHKwff4QnnoBrr4V77vE2MiJRimcStgZokO91/dxj+V0KjAMIIcwEKgO1C3+gEMLIEEJWCCErMzMzTuFKutu61eu47rkHPvwQHn4YPvjAE7LTT/fVjiKSnObP930wJ02Co46CNWugQwd46qmoI5N0Fs8kbA7QzMyamFlFvPB+YqFrVgG/ADCzY/AkTENdEol//hNat4bBgwseP+88/2Y9blw0ceU3bZqvuGzXDvr182lTkXQ0ebIviKlTB9q39zdNO3bEvnb3bhgwAP78Z5gwAa6/HkaMgJkzvZZzyZLExi6yR9ySsBDCTmA4MBVYjK+CXGhmd5pZv9zLfgtcbmafAmOAi0Oy7SguKWPxYjjuuNjnuneHzz9PbDyFPfig15316QPPPus/VK6/Hm6/Pdq4RBLtkUfgmmvg8su9hODRR+G11/wN0+7dRa9/5x2oVg0uuKDg8ebN4bLLfIpSJArxLMwnhDAFmFLo2K35/rwI6B7PGERKqn59ePfd2OcWLYJu3RIaTgFr1sDdd/sPnPr1/ViHDr5ooHVr/+HSvHl08Ykkyvffw623wscf+8IZgCOO8FHhzp19l4o+fQres3IltG3ri2gKa9cOXn45/nGLxBJ1Yb5ImTFokPf6mjWr4PE5c/wbfOF30Yn00ks+DbknAdsjMxOGDPGGsSLp4PXX4cQT8xKwPSpW9JGxl14qek/z5jB7NsSaZ5k1S29gJDpxHQkTSSY1asDo0V6EP2AAdOrk77bHj4dnnoHDDosuth9+8IQrljp1YMOGxMYjEpVt26Bq1djnqlSJvY1Y9+5QqZLXjV1/fd7xOXP8//zcufGJVWRfNBImkk/fvrBggb/Lnj0bGjb0VVVnnhltXN27e81L4XfyIcCrr/p5kXRw8snwxhv+xqSwceO8WL8wM59yHDkSunaFm27yN1q9e/vqyEaN4h+3SCyWbHXwWVlZITs7O+owRBIqBDjhBDj2WPjTn3wkYMsWuPNOn57JzobyGteWJLVpk0+pz58Pdev6FHvhqff8rrjCmyk/9pi3m9i40VvLTJrk/xcOOST2fbt2eQL32WdeRzZgAFSvHvtakdJiZnNDCFmxzmkkTCQJmPlI2H/+46Nz3bpBgwa+YGDqVCVgkrzmzfP+XZMnQ+PGXkTfrp2vAC7OiBHw85/7qFbjxv5/YtUq37e1uAQMICPDyw1uvhkuuUQJmERPI2EiSWbtWu/m37Dh3kcLRMq6Xbu8KP6Pf/SFMXt8/rmP/M6cCU2bFn//tm3w9dder3noofGPV+RAaCRMJIXUrev9zJSASbKbNg1q1SqYgAG0bOkjVaNG7f3+SpW8flMJmCQrJWEiIhKJPf27YmnXzs+LpDIlYSIiEolmzdS/S9KbkjAREYlEjx7++4gRBY/PmgVjxvg2XSKpTGuqREQkEma+oXbfvvD8856ULV0K773nqyMbNIg6QpH4UhImIiKROfpoWLjQtwZbsMAbIz/zjG+4LZLqlISJiEikypeHfv38l0g6UU2YiIiISASUhImIiIhEQEmYiIiISASUhImIiIhEQEmYiIiISASUhImIiIhEQEmYiIiISATUJ0xERKQY33wDTz8Nn30Ghx8OF18MrVtHHZWkCo2EiYiIxDB7tidc8+fDKadApUr++0MPRR2ZpAqNhImIiBSyaxecdx6MHAn9++cdHzYMOnf2ZKx9++jik9SgkTAREZFCpk3z6cf8CRhA/fpw5ZXw5JPRxCWpRUmYiIhIIV9/DS1axD7XsqWfFzlYSsJEREQKadMGpk+H3buLnnv//b0X5y9ZAtdeCz16+JTmlCkQQvxileSlJExERKSQTp2gbl24/faCidi778KLL8Jll8W+b8oUOP54qFHD7+3VC377W/jNb5SISVEWkuxfRVZWVsjOzo46DBERSXFr18LZZ8N33/mo1rJlsGgRvPCCF+YXtm0bNGoE//wndO+ed3zjRk/qnnwSTjwxcfFL2WBmc0MIWbHOaXWkJI1Fi3yl0ldfwVFHwdChcMwxUUclIqmqbl2YORNmzPA2FX36QN++3qoilrfe8jqy/AkYwKGHwlVXwejRSsKkoLhOR5pZbzNbYmbLzOymYq45z8wWmdlCM3shnvFI8nr2WTjpJKheHS65BKpW9Xemzz0XdWQiksrMPKkaNsxHxYpLwMBHzOrVi32ufn349tv4xCjJK24jYWaWAYwATgVygDlmNjGEsCjfNc2Am4HuIYTvzKxOvOKR5PXvf8N11/k70pYt/dhZZ8Hgwf7NsVcvqKN/OSISsc6d4X//F7Zvh4oVC557/XXo2jWauKTsiudIWBdgWQhheQhhOzAWKNRxhcuBESGE7wBCCOviGI8kqTFj4Jxz8hKwPVq1gn79YOzYaOISEcmvZUtPtK64An780Y+F4NOQb7wBv/518ff+618+1VmvHhx7LDz8MOzYkZi4JTrxTMLqAavzvc7JPZZfc6C5mU03s4/MrHcc45EktW6d14DFctRRsH59YuMRESnO6NGwZQs0bAi9e0Pz5nDffZ6E1a4d+55Ro7zM4vzz4aOPfFuk116DAQO8c7+krqgL88sDzYCTgPrA+2bWNoTwff6LzGwoMBSgYcOGiY5RItahAzz2GPz+90XPTZvm/XhERMqCatV8dH7VKli40EslOnb02rJYNm2CG2/04v89o/0NGnipRbduMHmyj/hLaornSNgaoEG+1/Vzj+WXA0wMIewIIXwFLMWTsgJCCCNDCFkhhKzMzMy4BSxl09lnw8qVMGJEXp+dEOCvf/Ul5IW3FRERiVrDhr6aslOn4hMw8BWVnTsXLbeoUMFXgI8bF984JVrxTMLmAM3MrImZVQQGARMLXTMBHwXDzGrj05PL4xiTJKGKFWHqVHjiCW9JMXiwf8N6+mk/XqFC1BGKiByYrVt99CyW6tX9vKSuuE1HhhB2mtlwYCqQATwZQlhoZncC2SGEibnnepnZImAXcGMIYUO8YpLk1bQpfPyx10ssX+6rJbt23fs7TBGRsu6EE+Dqq72h66GHFjw3fjz07BlNXJIY6pgvKW/XLvjvf/1dZUZG1NGIiBR01VWwdCk8/jg0aeJ1Yg884EX+n3xS/EiZJIe9dczX3pGSsrZt82L+unX9G1vduv5627aoIxMRyfPXv8LPf+61YU2bemPXefN8n0olYKlNI2GSkkLIW979wAP+jW3ZMt9It1w5ePllTWWKSNmydauvqqxVy39JatDekZJ2Zs+GTz+FxYvzOlc3beo1Fscc4+fVvVpEypLKlb2vmKQPTUdKSnr9dW98WHjrkIoV/fiUKdHEJSIisoeSMElJGRnFb/mxc6cK9EVEJHr7TMLM7Bozq5mIYERKS79+vufknv3b9vjxR3jhBTV4FRGR6JVkJOxwYI6ZjTOz3mYqZ5ayr317OO0071g9c6aviJw501+fdpqfFxFJZj/8AP/4h28Y/oc/wBdfRB2R7K99JmEhhFvwrYRGARcDX5jZPWZ2dJxjEzkoI0fCwIG+MW7Vqv77gAF+XEQkmc2f7zuH/Otfvr/u1q1w3HHwyCNRRyb7o8QtKsysPXAJ0Bt4B+gGvBVC+N/4hVeUWlSUPVOm+LuxFSvg6KO9+7O6PIuIxMfu3dCqlfc9vOiivOOrVvmq7ylT4Nhjo4tPCjqoZq1mdp2ZzQXuBaYDbUMIVwKdgAGlGqkknTvv9C2EzjnHuzuffjpcfjncf3/x94QAr7wCvXpBs2bQuzdMLLyrqIiIxPThh75n7pAhBY83bOhvgkeNiiYu2X8l6RN2GHBOCGFl/oMhhN1mdkZ8wpJksGwZPPooLFgAder4sfbtve6qbVsYNMg7Pxd2yy2ehN12m79by86GG2/0DtG33prYr0FEJNnk5EDr1rEbTrdu7d9TJTnsMwkLIdy2l3OLSzccSSZjx8KFF+YlYHvUq+e1WOPGwQ03FDy3dCk88QQsWpTXEbp5c5++bNXKh9abNElM/CIiyahFC5g1y3cEKdxuZ+ZMPy/JQX3C5ID98ANkZsY+V6eOny9s3Di44IKiW3LUqeNNVF96qfTjFBFJJZ06wZFHwh//6OUde8yZA089BUOHRheb7B8lYXLAjj/ea7kKr+0IwY8ff3zRe378EQ47LPbHq1kTNm0q/ThFRFLNuHEwYQK0awfXX++9EXv39nqwo9W7IGkoCZMD1rcvbN8ON90Emzf7sU2bvFC/WjU45ZSi9/To4fVghRO33bv9G8pJJ8U9bBGRpFevHsydCw8/7AX5554LK1d6MibJQ0mYHLCMDHjjDViyBBo08KXRDRvC2rXw2muxi0Z79fL9G6+5BjZu9GPffw9XXeUjZErCRERKxszf7N5wg9fTVq0adUSyv0qyOlKkWHXq+AjWmjXeo6ZxY6hbt/jry5XzzbWvvtqvbdDA7zvzzOITNxERkVSkJExKRb16/qskatb0/Ru/+caXWjdoULRQX0REJNUpCZPI1K7tv0RERNKRasJEREREIqAkTERERCQCSsKkiJwcmDHDVzmKiEjq2b076ggElIRJPv/5j/eY6dDBlzy3bg3nnQfffht1ZCIiUhqef9737C1f3nc8ufHG2LubSGIoCRMAdu6E007zxGv1avjoI28dccQR3j6icHNVERFJLvfdB3ff7b/v2OHf59etg1NPhW3boo4uPSkJEwAmT4af/QzuuQcOOcSPVa3q3Zg3boR33ok2PhEROXDff+/f3998E3r29GbbRx8NTz8NVarAiy9GHWF6UhImAHz4oU9FFm6WaubHp0+PJi4RETl4b78NP/+592XMzwwuucT3+5XEUxImgO/1uH597HPr1vl5ERFJTiH4jiWxlCunkpOoKAkTAAYNguee84Qrv5wcePll3xxWRESS08kn+4xG4VXvIcCzz8Lpp0cTV7pTEiYANG/um2h37+41AvPmweOPwwknwC23lHxLIhERKXsOO8xXvffu7clYCL7n75VX+pvvwYOjjjA9adsi+cltt0GXLvCPf8CDD0KzZjBqFJxyStSRiUX5fHwAABHUSURBVIjIwbrlFqhbF379a1i5EipVgiFDYNq0vAVZklgW4jgRbGa9gYeBDOCJEML/FXPdAOAloHMIIXtvHzMrKytkZ+/1EhERESlGCLBliydhGRlRR5P6zGxuCCEr1rm4TUeaWQYwAugDtAIGm1mrGNdVA64DZsUrFhEREXFm3pJICVj04lkT1gVYFkJYHkLYDowF+se47i7gz8DWOMYiIiIiUqbEMwmrB6zO9zon99hPzKwj0CCEMHlvH8jMhppZtpllry+uj4KIiIhIEolsdaSZlQMeBH67r2tDCCNDCFkhhKzMzMz4ByciIiISZ/FMwtYA+Xvz1s89tkc1oA3wrpmtALoBE80sZvGaiIiISCqJZxI2B2hmZk3MrCIwCPhpY4QQwsYQQu0QQuMQQmPgI6DfvlZHioiIiKSCuCVhIYSdwHBgKrAYGBdCWGhmd5pZv3h9XhEREZFkENdmrSGEKcCUQsduLebak+IZi4iIiEhZom2LRERERCKgJExEREQkAkrCRERERCKgJExEREQkAkrCRERERCKgJExEREQkAnFtUSEiIiLJb+1aWLAAateGDh3ALOqIUoOSMBEREYlpyxa46ip49VU49lhYuRKqVIFnnvFkTA6OkjARERGJ6bLLYNcuWLECqleHEOCFF6B3b/j0Uzj88KgjTG6qCRMREZEivvoK3nwTnnrKEzDwacgLL4T+/eHxx6ONLxUoCRMREZEi5syBE0+EQw4peq5vX/joo8THlGqUhImIiEgRNWvC11/HPvf1135eDo6SsBS2bh3cfz9ccw389a/w7bdRRyQiIsnipJNg1Sp4772Cx3/8ER55BIYMiSSslKIkLEVNmgTHHAOLF0PTpjB7NrRoAe++G3VkIiKSDCpU8Hqwc8+F226DDz6AZ5+F446D7t2hV6+oI0x+FkKIOob9kpWVFbKzs6MOo0z75htPuF5/Hbp0yTv+9tswaJAXW1apEl18IiKSPJYsgUcfhY8/hsxMuPhiL8xXr7CSMbO5IYSsWOfUoiIFPfccnHFGwQQM4JRToFs3eOkl+NWvoolNRESSS4sWPv0opU/TkSlo1Spo2zb2ubZt/byIiIhES0lYCmreHGbNin1u1iw/LyIiItFSEpaCLrgA3n/fa8Lye/FFn9s/66xo4hIREZE8qglLQdWrwyuvwIAB0L69/5o9G7780ldNVqoUdYQiIpLK3nsPHngAPvkEatXyOuSrr4aKFaOOrGzRSFiKOu44WL7c/+HXqAHDh8OyZZ6QiYiIxMtzz/mMTP/+8OGH8PDD8MYb/nrnzqijK1vUoiICIfhS35UrvT6rTZuoIxIRETl4W7ZAw4YwbRq0a5d3fOdOOOEEuOEG7zuWTvbWokIjYQm2bBl07gznn+9N7/r0gZNPhrVro45MRETk4LzzDrRqVTABAyhfHoYNg3HjoomrrFISlkBbtsCpp8Ill8DSpTBhgjdO7dEDTj8ddu8u/t45c+Dyy71D8fDhMH9+4uIWEREpia1bvS45lurV/eeg5FESlkDjxkHLll6cWC73b758ed8OIgQfvo3lkUd8RWOzZj6UW6cO9OwJzz+fuNhFRET2pXt3394o1l7F48d703DJo9WRCfTxx7H32jLz49nZPlKW31dfwR13+L0NG/qx3r1h4ED/x96nDxx2WPxjFxER2ZfDD/fZnnPOgSefhKOOgs2bvTh/xgz429+ijrBs0UhYAtWq5cX4saxcCbVrFz3+/PO+ymRPArZHq1aegI0fX/pxioiIHKj77/da565dfQanXj1PwN5911frSx4lYQk0ZIgv3V29uuDxhQth6lQf3SpswwZo1Cj2x2vY0M+LiIiUFRkZXmazerX3ply6FF57DRo3jjqyskdJWAIddRT84Q++ifa993rflDvu8DnyESOgZs2i93TsCG+9VfR4CPDmm9CpU/zjFhER2V+VK/vm35mZUUdSdsU1CTOz3ma2xMyWmdlNMc7fYGaLzOwzM5tmZsWM+aSO667zVZFffAEPPgjr13tB/gUXxL7+3HP9XcSjj+atnty5E+680xOxwjVkIiIikhzi1qzVzDKApcCpQA4wBxgcQliU75qTgVkhhM1mdiVwUgjh/L193FRo1rq/li2DwYPhu++8sevcuXD00fDCC3DkkVFHJyIiIsXZW7PWeK6O7AIsCyEszw1iLNAf+CkJCyG8k+/6j4AhcYwnaTVt6ns/fvIJrFgBd9+tLvsiIiLJLp5JWD0gfwl6DtB1L9dfCrwe64SZDQWGAjQsvEwwTZh5fVjHjlFHIiIiIqWhTBTmm9kQIAu4L9b5EMLIEEJWCCErUxV+IiIikgLiORK2BmiQ73X93GMFmFlP4PdAjxDCtjjGIyIiIlJmxHMkbA7QzMyamFlFYBAwMf8FZnYs8BjQL4SwLo6xiIiIiJQpcUvCQgg7geHAVGAxMC6EsNDM7jSzfrmX3QdUBcab2Twzm1jMhxMRERFJKXHdOzKEMAWYUujYrfn+3DOen19ERESkrNIG3iIiIhK5TZu8/+Xcub6X8kUXQcuWUUcVX2VidaSIiIikr0WL4JhjfDu/Dh1g1y7o0cM3A09lGgkTERGRyIQAgwbB7bfDpZfmHb/2WujaFU44wX9PRRoJExERkcjMng3bt8Ovf13w+JFHeiL2+OPRxJUIGgkTERGRyOTk+FSkWdFzrVvD228Xf28I8M47kJ0NtWrBgAFQo0b8Yi1tGgkTERGRyLRs6aNhO3YUPTdjBrRoEfu+f/8bunSB66+Hdetg6lQ46ih48cW9f74PPoCzzoIGDaB9e3jgAdi69eC/jgOhJExEREQi07q1J2K33eYjW3t88gmMHAnDhsW+74ILoHdv+PRTL+AfNw7eew+uuQYWLox9z4svwvnnQ9++8OGH8OijPtJ25pmxk8B4s5D/K04CWVlZITs7O+owREREpJT85z/Qvz/8979w6qmwYoWPWD3xBJxzTtHrFy6E007z68oXKqy6/XbYsAEeeaTg8W3boFEjmDIFOnbMO75rF5x0kid7F15Yul8XgJnNDSFkxTqnkTARERGJ1OGHw8yZ8I9/QOPGMHAgrFwZOwEDWLoUsrKKJmAA3br5+cLef9+nK/MnYAAZGXDllTB+/EF/GftNhfkiIiISOTM48UT/tS8NGvho2O7dUK7QcNKCBX6+sK1boXr12B+venXYsmX/Yz5YGgkTERGRpNKpE1SrBo89VvB4Tg785S9w+eVF7znuOJg1C9avL3pu/Hg45ZT4xLo3GgkTERGRpGIGY8dCr14webL/vnIlPPss3Hxz7OautWrB0KFw9tnw1FPQrJmPfj36qLe5ePDBxH8dSsJEREQk6TRv7tsdvfii7zdZq5bXlTVtWvw9f/oT3HMPdO/uU5AbNvgI2bvv+v2JptWRIiIikla2bfORs5o1ITMzvp9rb6sjNRImIiIiaaVSJR9Ji5oK80VEREQioCRMREREJAJKwkREREQioCRMREREJAJKwkREREQioCRMREREJAJKwkREREQioCRMREREJAJq1prPypXw9NOwahW0bAm/+hXUqbPv+7ZsgW+/hdq1vQGciIiIyL5oJCzXmDG+K/u330K3bvD559C6Nbz9dvH3/PADXHkl1K0LWVlQrx7cdBNs3564uEVERCQ5KQkDcnJg+HB47z14+GG4/HIYNQrGj4dBg2Dz5qL37NoFffvC1q2esK1dC3Pm+GaiF12U+K9BREREkouSMODZZz3Zat264PGTTvIRrgkTit4zdaonZ6NGwRFH+LEmTTxxmz4dPvss7mGLiIhIElMSho+EHXNM7HOtWsGaNUWPv/kmnH8+lCv0N1ipEpxzjp8XERERKY6SMDzRmjEj9rnp02MnaBUq+FRkLFu2+HkRERGR4sQ1CTOz3ma2xMyWmdlNMc5XMrMXc8/PMrPG8YynOEOGwLRpMGlSweOPPQbffAN9+hS9Z8AAeOYZT7jy27ABXn4ZzjorfvGKiIhI8otbiwozywBGAKcCOcAcM5sYQliU77JLge9CCE3NbBDwZ+D8eMVUnBo14NVXPbFq3hzatIGPPvLVj5MnQ0ZG0Xu6doXu3eG00+Cuu6BtW7/n5pvhiiugUaNEfxUiIiKSTOLZJ6wLsCyEsBzAzMYC/YH8SVh/4PbcP78EPGpmFkIIcYwrpm7d4KuvPOlavRrOOANOPbVozdceZvDUU/D3v8M113hvsRYt4H/+x0fWRERERPYmnklYPWB1vtc5QNfirgkh7DSzjUAt4Jv8F5nZUGAoQMOGDeMVLxUrwtlnl/z6jAxvbTF8eNxCEhERkRSVFIX5IYSRIYSsEEJWZmZm1OGIiIiIHLR4JmFrgAb5XtfPPRbzGjMrDxwKbIhjTCIiIiJlQjyTsDlAMzNrYmYVgUHAxELXTAR+lfvngcDbUdSDiYiIiCRa3GrCcmu8hgNTgQzgyRDCQjO7E8gOIUwERgGjzWwZ8C2eqImIiIikvHgW5hNCmAJMKXTs1nx/3gqcG88YRERERMoiS7bZPzNbD6yM86epTaEVmlIm6LmUPXomZZOeS9mjZ1I2JeK5NAohxFxVmHRJWCKYWXYIISvqOKQgPZeyR8+kbNJzKXv0TMqmqJ9LUrSoEBEREUk1SsJEREREIqAkLLaRUQcgMem5lD16JmWTnkvZo2dSNkX6XFQTJiIiIhIBjYSJiIiIREBJmIiIiEgE0joJM7PeZrbEzJaZ2U0xzlcysxdzz88ys8aJjzL9lOC5nGhmH5vZTjMbGEWM6aYEz+QGM1tkZp+Z2TQzaxRFnOmmBM9lmJnNN7N5ZvahmbWKIs50sq9nku+6AWYWzExtK+KsBP9PLjaz9bn/T+aZ2WWJii1tkzAzywBGAH2AVsDgGN+gLgW+CyE0BR4C/pzYKNNPCZ/LKuBi4IXERpeeSvhMPgGyQgjtgJeAexMbZfop4XN5IYTQNoTQAX8mDyY4zLRSwmeCmVUDrgNmJTbC9FPSZwK8GELokPvriUTFl7ZJGNAFWBZCWB5C2A6MBfoXuqY/8Ezun18CfmFmlsAY09E+n0sIYUUI4TNgdxQBpqGSPJN3Qgibc19+BNRPcIzpqCTP5b/5XlYBtBIrvkrycwXgLvxN/dZEBpemSvpMIpHOSVg9YHW+1zm5x2JeE0LYCWwEaiUkuvRVkuciibW/z+RS4PW4RiRQwudiZleb2Zf4SNi1CYotXe3zmZhZR6BBCGFyIgNLYyX9/jUgt5ziJTNrkJjQ0jsJE5FSZmZDgCzgvqhjERdCGBFCOBr4HXBL1PGkMzMrh08J/zbqWKSA14DGueUUb5E3AxZ36ZyErQHyZ7v1c4/FvMbMygOHAhsSEl36KslzkcQq0TMxs57A74F+IYRtCYotne3v/5WxwFlxjUj29UyqAW2Ad81sBdANmKji/Lja5/+TEMKGfN+zngA6JSi2tE7C5gDNzKyJmVUEBgETC10zEfhV7p8HAm8HdbeNt5I8F0msfT4TMzsWeAxPwNZFEGM6KslzaZbv5enAFwmMLx3t9ZmEEDaGEGqHEBqHEBrj9ZP9QgjZ0YSbFkry/6Ruvpf9gMWJCq58oj5RWRNC2Glmw4GpQAbwZAhhoZndCWSHECYCo4DRZrYM+BZ/eBJHJXkuZtYZeAWoCZxpZneEEFpHGHZKK+H/lfuAqsD43LUrq0II/SILOg2U8LkMzx2h3AF8R96bSomDEj4TSaASPpNrzawfsBP/WX9xouLTtkUiIiIiEUjn6UgRERGRyCgJExEREYmAkjARERGRCCgJExEREYmAkjARERGRCCgJExEREYmAkjARERGRCCgJE5G0ZWadczftrWxmVcxsoZm1iTouEUkPatYqImnNzO4GKgOHADkhhD9FHJKIpAklYSKS1nL3k5sDbAWOCyHsijgkEUkTmo4UkXRXC9/3sho+IiYikhAaCRORtGZmE4GxQBOgbghheMQhiUiaKB91ACIiUTGzXwI7QggvmFkGMMPMTgkhvB11bCKS+jQSJiIiIhIB1YSJiIiIREBJmIiIiEgElISJiIiIREBJmIiIiEgElISJiIiIREBJmIiIiEgElISJiIiIROD/A6MtUXdQon1aAAAAAElFTkSuQmCC",
            "text/plain": [
              "<Figure size 720x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          },
          "output_type": "display_data"
        }
      ],
      "source": [
        "###グラフにしてみる\n",
        "import matplotlib.pyplot as plt\n",
        "fig = plt.figure(figsize=(10,4))\n",
        "ax = fig.add_subplot(111)\n",
        "ax.set_xlabel(\"x\"); ax.set_ylabel(\"y\")\n",
        "ax.scatter(xt, yt, facecolor=\"none\", edgecolor=\"b\", s=50, label=\"Data\")\n",
        "ax.legend()\n",
        "plt.show()\n",
        "plt.close()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qG3GbElypvo7"
      },
      "source": [
        "ではいくつか必要な関数を適宜定義しながら進めていきましょう。\n",
        "\n",
        "*注: 以下のコードは入力・出力ともに1次元かつ、決まったニューラルネットワーク構造の場合に対して書かれているため、naiveに2層以上の隠れ層を持つニューラルネットワークに拡張するのはstraightfowardではなく、また効率的ではありません。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 137,
      "metadata": {
        "id": "uRYnoX_rhm6U",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "nhl = 8 ## 隠れ層のノードの数を指定 これを増やすほどニューラルネットワークの表現能力が上がる一方、データに過適合しやすくなる(例外あり)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 138,
      "metadata": {
        "id": "9NjbaZrFpw0K",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "#重み行列W,V(今はベクトル)と、隠れ層でのバイアスbs,出力層でのバイアスを正規乱数で初期化\n",
        "np.random.seed(1234)  #結果が実行ごとに同じになるよう乱数を固定(バグを見つけやすくする)\n",
        "W = np.random.normal(0.0,1.0,nhl)\n",
        "V = np.random.normal(0.0,1.0,nhl)\n",
        "bs = np.random.normal(0.0,1.0,nhl)\n",
        "b0 = np.random.normal()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OqSaHMtth5to"
      },
      "source": [
        "隠れ層で作用させる活性化関数を定義しておきましょう。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 139,
      "metadata": {
        "id": "z8XFRbf6h6q9",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "#シグモイド関数: 活性化関数の一つ\n",
        "def sigmoid(z):\n",
        "    return 1.0/(1.0+np.exp(-z))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iHf3g_gsiENm"
      },
      "source": [
        "最適化したい量(データとモデルの齟齬を表す量)を目的関数(target function)やloss functionなどと呼びます。(以下でもそれに倣う)\n",
        "\n",
        "以下では、データとANNのアウトプットの二乗誤差を目的関数として定めることにします。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 140,
      "metadata": {
        "id": "AsIEltHxiEY4",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "### データとANNの出力間の二乗誤差を計算する関数を作っておく。\n",
        "def calc_tloss(x,y,tW,tV,tbs,tb0,acf):\n",
        "    nhl = len(tW)\n",
        "    s=0.0\n",
        "    for i in range(len(x)):\n",
        "        s += (np.dot(tV, acf(tW*x[i]+tbs)) + tb0 - y[i])**2\n",
        "    return s"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Q_QhyjGnQVyJ"
      },
      "source": [
        "上ではcalc_tlossの引数にacfという変数を指定し、acfにsigmoidを指定しました。  \n",
        "この様にしておくと、sigmoid関数以外の活性化関数を使う際にも、上のcalc_tloss関数が使いまわせますね。\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6P_hprmIWE0w"
      },
      "source": [
        "## データの下処理\n",
        "\n",
        "機械学習などの分析では、データの値を中心0,分散1に変換して扱うのが基本です。\n",
        "\n",
        "このことを、データの標準化と呼びます。\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 141,
      "metadata": {
        "id": "ABVMF4Y9WCBN",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "ymean = np.mean(yt)\n",
        "ystd = np.std(yt)\n",
        "ny = (np.array(yt)-ymean)/ ystd #それぞれのデータを平均をひいて標準偏差で割る"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "p_n9AjxUYWUr"
      },
      "source": [
        "なぜ標準化が必要なのかは、今のような1次元入力データの場合よりもむしろ多変数を扱う際を考えてみるとわかります。\n",
        "\n",
        "変数ごとに標準的なスケールが違う値を扱う場合、スケールの大きな量に学習が引っ張られる、ということが起こりえます。\n",
        "\n",
        "たとえば目的関数を[体重と身長、それぞれについての二乗誤差の和]とする場合、  \n",
        "データが50kg、ニューラルネットワークの予測が55kgで10%違っていても、二乗誤差の値は25ですが、  \n",
        "身長が180cm vs 198cmと10%違っていたら、二乗誤差の値は324となります。  \n",
        "したがって、目的関数は身長の予測精度により強く依存することになり、  \n",
        "身長をより重視する(きちんと再現する)方向へ、ニューラルネットワークの学習が引っ張られてしまいます。\n",
        "\n",
        "もちろん、身長をより高い精度で推測したいニューラルネットワークを構築したいなら話は別ですが、  \n",
        "特定の値を特別視しない(全ての量を平等に扱う)のなら、通常は標準化を行います。\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 142,
      "metadata": {
        "id": "e01pxFk0Qiuj",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "acf = sigmoid #sigmoid関数をacfという名前で使う"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eUXQaPuVQ_FR"
      },
      "source": [
        "さて、初期値W,V,bs,b0と活性化関数にsigmoidを選んだニューラルネットワークとデータの値の二乗誤差は..."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 143,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Xsdv35IfQ_bs",
        "outputId": "958a2fc0-1afa-4736-bde7-4c3051d1aedb",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "初期値での二乗誤差 61.29358546868393\n"
          ]
        }
      ],
      "source": [
        "print(\"初期値での二乗誤差\",calc_tloss(xt,ny,W,V,bs,b0,acf))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "q5o7mlnFMPLF"
      },
      "source": [
        "データ1個あたり、ニューラルネットワークとデータ値との間にどれくらい誤差があるかというと..."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 144,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "84AJ8uM3MYxi",
        "outputId": "a374719e-bd7f-4d97-ad61-2d02ef322096",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "データ1個あたりの誤差: 1.2378770684995737\n"
          ]
        }
      ],
      "source": [
        "print(\"データ1個あたりの誤差:\", np.sqrt(calc_tloss(xt,ny,W,V,bs,b0,acf)/len(xt))) #データ1個あたりどれほど誤差*があるか *標準化された誤差"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iV0s8cpJrN2e"
      },
      "source": [
        "ランダムに生成した重み(W,V)やバイアス項(bs,b0)では、まだニューラルネットワークは訓練がなされていないデタラメな関数なので、図にプロットしてみると...\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 145,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 279
        },
        "id": "pSmbEHP9MhfL",
        "outputId": "3b7c5b66-55f3-4e47-dfca-3e5e2184ae62",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmEAAAEGCAYAAAAt7EI0AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deXxU5dn/8c+VsFkBFxaLLIKyCSIoQRDcV0BFEBSwuCui4tqnv2pt3drqU0WtFdqK4oYVBKuIgKLirggERREQpAgS5ZHFpaKsyf3740rIzGQCATJzMsn3/Xrllcx9nzNzTQ7kXHOvFkJARERERNIrK+oARERERKoiJWEiIiIiEVASJiIiIhIBJWEiIiIiEVASJiIiIhKBalEHsLPq168fmjdvHnUYIiIiIjs0d+7ctSGEBsnqMi4Ja968Obm5uVGHISIiIrJDZraitDp1R4qIiIhEQEmYiIiISASUhImIiIhEIOPGhImIiEhm2LJlC3l5eWzcuDHqUFKuVq1aNGnShOrVq5f5HCVhIpJRZs+Gv/8dFi+Gpk3hssvg5JOjjkpEksnLy6NOnTo0b94cM4s6nJQJIbBu3Try8vJo0aJFmc9Td6SIZIzHHoO+faFDBxgxAk48EYYNg9tvjzoyEUlm48aN1KtXr1InYABmRr169Xa6xU8tYSKSEdatg+uv95aw1q29rEcPOOssT8oGDID27aONUURKquwJWJFdeZ9qCRORjPD883DqqcUJWJEGDeDCC+HppyMJS0RklykJE5GM8P33sP/+yev239/rRUSSmTRpEmbGZ599BsDy5csxMx588MFtxwwfPpzHH38cgAsvvJDGjRuzadMmANauXUsqdutREiYiGaFrV5g2DQoKStZNnQrduqU/JhHJDOPGjeOoo45i3Lhx28oaNmzIAw88wObNm5Oek52dzaOPPprSuJSEiUhGOOooaNjQx4Vt2OBlW7bAPffAf/4DZ58dbXwiUjGtX7+ed999lzFjxjB+/Pht5Q0aNODEE0/kiSeeSHreddddx/3338/WrVtTFpsG5otIRjCDF16ASy6BZs18MP7ixdCyJbz2GtSqFXWEIrI9t7+4gIVf/7dcn7Pd/nW59Yztz8h54YUX6NmzJ61bt6ZevXrMnTuXevXqAfDb3/6WXr16cfHFF5c4r1mzZhx11FGMHTuWM844o1zjLqKWMBHJGPvu6wP0c3Ph5pvh9dfhrbcgBUM1RKSSGDduHIMGDQJg0KBBcV2SBx54IF27duXpUmb23HTTTdxzzz0UJBsHUQ7UEiYiGeeAA/xLRDLHjlqsUuHbb7/l9ddfZ/78+ZgZ+fn5mBlXXXXVtmN+97vfMWDAAI499tgS57dq1YpOnToxYcKElMSnljARERGplJ599lnOO+88VqxYwfLly1m5ciUtWrRg5cqV245p27Yt7dq148UXX0z6HDfffDMjRoxISXxKwkREqoBPPoHLL4ejj4aBA+GVV6KOSCT1xo0bR79+/eLK+vfvz1133RVXdvPNN5OXl5f0Odq3b8/hhx+ekvgshJCaJzZ7FDgdWB1COCRJvQEPAL2Bn4ELQwgf7uh5c3JyQm5ubnmHKyIR2LQJJk2CJUt8H8gBA6B27aijqnwmToThw+Gaa3yW6eLFPqv07LPhzjujjk4qs0WLFnHwwQdHHUbaJHu/ZjY3hJCT7PhUjgl7HBgJPFlKfS+gVeFXV+Afhd9FpAr4+GM44wxfAb9bNx9w/5vfwIQJcPzxUUfnCeLzz8Nnn0GTJnDOOVC3bvm/zg8/wFNPwaJF0LgxnH++f9+RTz+FhQv92O7dffZoMuvX+/6aM2ZAp05eduyxnvB27Aj9+0PnzuX3fkSk7FLWHRlCeBv4djuHnAk8GdwHwN5m1ihV8YhIxbF5M/TpA3ff7ctL/OlPvvzEhAme7Hy7vb8caTB/PrRqBWPGQAjw8stw4IEwfXr5vs6sWZ6EvvOOf//yS196I2byVgmrV/vG5T17+u/r8suhXTtPapN58UU48sjiBKzIvvvCZZd5Aigi0YhydmRjYGXM47zCslWJB5rZUGAo+LodIlKxhOB7Nz74IHz+ua/jdfnlMHQoZCX5qDdlCrRoAYWzxrc5/njo1QvGjoVrr01P7Im2bvUE8X//F849t7j8vffgzDO99alhw91/nc2bvRXqkUe8RbDIVVd5S1X37iVngIYA/fr5uK7p06FateLffa9eHtvee8ef8913pbesNW4My5bt/nsRkV2TEQPzQwijQwg5IYScBg0aRB2OiCT4wx/gL3+B3//eu+/uvdcTqUsvTX78559Dly7J67p08TFiUZk2zZOT2AQMoEcP6NsXCreWK5fXadkyPgEDOOQQGDIk+evMnAlr1/o4rmqFH6HN4Fe/gmOO8d95oq5dPWHLzy9Z99JLcMQRu/1WRGQXRZmEfQU0jXncpLBMRDLI8uXwj3/4wqmnnw4NGsAJJ8Crr8Ibb8CcOSXPOeAAn62XzMcfR7v46tKlpSeIOTleXx6+/NITrmQOOcTrE330kf9uk7UunnSS1yfq3Nm7Vq++uni7p4IC+Oc/vTt0yJBdfw8isnuiTMImA+eb6wb8EEIo0RUpIhXbCy94t1r9+vHlv/iFDzJ/9tmS5/Tt60nYtGnx5XPm+GD4Cy5IXbw70qIFzJuXvG7ePK8vD61bwwcfJK/74AOvT1S/vie9ySxfXvIaFJk4Eb75xruJTz0VDjoIHn3Ux+OlYrKBiJRNypIwMxsHzATamFmemV1iZsPMbFjhIdOAZcBS4GHgylTFIiKps3mzJ1zJ7Lmn1yeqVQueew4uushn6Y0Y4YnXqad6N1x5jLnaVaed5q1dzz8fX56b68nMhRcmPy8EH+R+xBFQpw60bevdslu2JD/+5JPhxx9h9Oj48rfe8sT2ootKnnPGGZ6ozpoVX75qlY8tO//85K+1997w73/7uddf788/eza0aZP8eJGo5Of7+MuXX/au9/KQnZ1Np06daN++PR07duTee+/d4TZEy5cvL3Uro/KUsoH5IYTBO6gPwFXbO0ZEKr6TTy6e6VijRnF5QQE884zPfEzmyCN9bNi4cb5u1WGHeTIW9bDPGjV87bLTT/eEsHt3WLDAJxM89hg0KmUO9223eWJ5991+zsKFcMstPo5r4sSSS0hkZ/vMxdNO89fp0cPPmT3bf2/JEtFf/MKPPf10uPhiX/Nr0SKfEHH99aV3bxZp3lz7bErF9eqrPplnr73870Burn84GzGieAzkrthjjz2YV9i8vXr1as4991z++9//cvvtt5d6TlESdm7i4NBylrLFWlNFi7WKVDwDBvgn2Ace8C6vb76B3/3Ok6u3304+hqmi++knGD/e30Pjxj74vbTuvq++8qUlPvssPnnatMmTy7//HY47Lvm5W7f6APmidcL69Su9ZbHIf/7jY7oWLYL994dLLvEB+CIVTVkXa/30Ux/vOG6cL8ECsG6d7+7QubNP/NlVtWvXZv369dseL1u2jC5durB27VpWrFjBeeedx08//QTAyJEj6d69O926dWPRokW0aNGCCy64gH79+iU9rizvd3uLtSoJE5HdtmmTz4x89FHYYw9fIHTgQF+VvSqMOXroIe9CeTLJ0tR33w0rV3prlUhVU9Yk7NJLfaziTTfFl69a5evgrVix639LEpMwgL333pvFixdTp04dsrKyqFWrFp9//jmDBw8mNzeXN998kxEjRjBlyhQAfv7556THleX9RrVivohksFWrfA2ppk29dWt7atb0hOuPf/TFROvX33FrTmWyZYuPc0umVq3Sx4WJiJs929fIS9SokS/lsnCh76xR3rZs2cLw4cOZN28e2dnZLCllfZyyHrezMrCTQERS6fvvvRWrfXv4n//xroDTTvOkbEdq1fKErSolYODj4l54AX7+Ob48BO9e6dkzmrhEMsU++3i3fqL8fP/bs88+5fday5YtIzs7m4YNG3L//fez33778fHHH5Obm8vmZDOJoMzH7SwlYSKyTQi+Knz9+r5O1cyZ3pV2+OFwyilq0SlNmzaeqPbtW7zQ7KpVvi0Q+EB6ESndeed5a3ri35innvKxkuU1k3fNmjUMGzaM4cOHY2b88MMPNGrUiKysLMaOHUt+4arGderU4ccff9x2XmnH7S4lYSKyzTvveHfigw9C7dpeVqsW3HGHfxJ94YVo49tVn3/um4OfcQZceSV8+GH5v8bo0T7j86ijfHD+wQf7jK6XX969mV0iVcEFF/isyGOO8dbj116Da66B3/7Wx1zujg0bNmxbouKkk07ilFNO4dZbbwXgyiuv5IknnqBjx4589tln7LnnngAceuihZGdn07FjR+6///5Sj9tdGpgvItvcdZfvNXj33SXrRozw7oL7709/XLtj0iRvkbrkEk+SFiyAkSP9j3sq9qfcssU3IN9rr9LHiYlUFWUdmA8+U3jCBE/CfvzRP9BccUXpe59WRBqYLyK7rE4dX5IhmdWrvT6T/Pijr6f16qs+tg28u/W887yLtXdv39KnPFWvDvvtV77PKVIVVKvme7ameGmuCkXdkSKyzYAB3uWYuG/h2rW+/MLg7S7BXPE8/7x3bxQlYEWaNvXuj2QbXouIpItawkRkm1/+0sd/HX20r9fTtavv8Xjnnb6SdRl7FSqMNWtK3+uxRQtfIFLK1+bNMHkyzJ/vywsMHFi+M9sk84QQsMQtIyqhXRnepZYwEYlz9dXwxBMwY4bvk/jvf/s4sDvuiDqynXfYYT7AN9nfxtdeg06d0h9TZbZ4se+ZOWqUP37zTV+AM1MndMjuq1WrFuvWrdulBCWThBBYt24dtXZyIKgG5otUEVu3Vr1ZeiFATo7PivzDH3y/xhC8a/Xmm32boaJZoLJ7Cgp8bbkbbihemgNg7lzfmP3DD3e86K9UPlu2bCEvL4+NGzdGHUrK1apViyZNmlC9evW4cg3MF6mi8vPhr3/12YArVnj30LBhPjMwdrPtysrMN8keNMg33+7a1WdHgi8doQSs/Lzxhs8GvfTS+PLOnX3fzTFjYDv7JUslVb16dVqUNiZA1B0pUpkNG+bjcyZO9IRs+nRfgHXQoORddJXR/vv7JuIvvAD9+/t6Xp9+CoccEnVkFd+SJXD99b5Q74UX+u+xNEuXQpcunvgmysnxehGJpyRMpJJauBCmTIFp0/wmaOaJx6RJnoS8/37UEaZXp04+SLxHj+SJgsR78UX/Xf3iF97FeNhhcP75pbdmNW8O8+Ylr5s3r/QJEiJVmZIwkQiEAA8/7De2OnWgQwcfzFxQUH6vMWUKnHMOJC7sXKOGr8Pz4ovl91pSufz8s6+vNmUK/PnPvvfltdfCnDnekvjRRyXPOekkX+j3qafiy+fP94kel1ySnthFMonGhIlE4LrrvCVqxAjvwpk3z5eE+OgjeOSR1L++WdXpjpSdN3Wqj+Xq2jW+vEEDX6pk7Fj/ABErOxuee84XwH36aTj2WJ8tOWkS/OMfagkTSUYtYSJptngxjB/vSySceCLUresLir7yin8la2XYFb17+1iwn3+OL9+yxW+S2lRaSrNuXekzGQ84wBfvTaZDBx9HNnCgP0eHDj4DdeDA1MUqksmUhImk2aRJflPaa6/48j33hCFDvDWhPBxyiHcjnX568VidRYt8cHrbtr4vm0gyOTn+ISE/v2TdK694fWn22MN3I7j7bh/U37Bh6uIUyXRKwkTSbMuW0jd2rlXL68vLww97Itanj48FO+EE3zNx4kQNTpfS5eT4QPtf/9pXwIfi9dXeessH6IvI7tNirSJpNmeOD5hfvDh+ra78fDj0UB+gf9xx5fuaIcCmTVCzppIvKZt16zzZmjvXx4YtXuyL/Y4b592MIlI2WqxVpALp0sWXSxg0CO67z1sc8vLgN7+Bxo19QHN5Myu99U0kmXr1fID+4sXejd2oERxxhJJ4kfKk7kiRCIwbBy1berfPfvt5C9gvf+njxXSTk4qkTRvo29dbw/RvU6R8qTtSpJxs2uRdOPvuW/ZWp82bi8+pWTO18YmISPptrztSLWEiu2njRh/A3KiRr63UqJEvbJm4NEQyNWr48UrARESqHiVhIrshBB9kv2KFLwOxapWvEL56NfTrpwVRRUSkdErCRHbDrFm+R+O4ccWLWzZp4lu3rFy5/Q2PRUSkalMSJrIbXnkFBgyA6tXjy7OzvYVs+vRo4hIRkYpPSZjIbqhe3ceEJbNhQ/w6YCIiIrFSmoSZWU8zW2xmS83sxiT1zczsDTP7yMw+MbPeqYxHpLz16+f7QP7wQ3z5+vXwr3/BWWdFE5eIiFR8KUvCzCwbGAX0AtoBg82sXcJhvwcmhBAOAwYBf09VPCKp0Lat7wN50knw6qvw3Xfw+utw8slw2mm+/peIiEgyqVwx/whgaQhhGYCZjQfOBBbGHBOAuoU/7wV8ncJ4RFLir3+FRx+F3/4W/vMfaNEChg2DoUOjjkxERCqyVCZhjYGVMY/zgK4Jx9wGvGJmVwN7AielMB6RMtm6FaZNg2XLPKHq3bvkwPtYZnDJJf4lIiJSVlEPzB8MPB5CaAL0BsaaWYmYzGyomeWaWe6aNWvSHqRUHZ98Aq1awd13wxdfwIgR/njevKgjE5Hd9dNP8MgjcM01cOedvoyMSJRSmYR9BTSNedyksCzWJcAEgBDCTKAWUD/xiUIIo0MIOSGEnAYNGqQoXKnqNm70cVx33gnvvgsPPADvvOMJ2Wmn+WxHEclM8+f7PphTpsCBB8JXX0GnTvDYY1FHJlVZKpOwOUArM2thZjXwgfeTE475EjgRwMwOxpMwNXVJJP79b2jfHgYPji8/5xz/Yz1hQjRxxZoxw2dcHnoo9Onj3aYiVdHUqT4hpmFD6NjRPzRt2ZL82IIC6N8f/vIXmDQJrrsORo2CmTN9LOfixemNXaRIypKwEMJWYDgwHViEz4JcYGZ3mFmfwsN+DVxmZh8D44ALQ6btKC6VxqJF0L178roePeCzz9IbT6L77vNxZ716wZNP+k3luuvgttuijUsk3R58EK6+Gi67zIcQjBwJL77oH5gKCkoe/8YbUKcOnHtufHnr1nDppd5FKRKFVA7MJ4QwDZiWUHZLzM8LgR6pjEGkrJo0gTffTF63cCF065bWcOJ89RX86U9+w2nSxMs6dfJJA+3b+82ldevo4hNJl++/h1tugQ8/9IkzAL/8pbcKd+niu1T06hV/zooV0KGDT6JJdOih8NxzqY9bJJmoB+aLVBiDBvlaX7NmxZfPmeN/4BM/RafTs896N2RRAlakQQMYMsQXjBWpCl56CY45pjgBK1KjhreMPftsyXNat4bZsyFZP8usWfoAI9FJaUuYSCbZe28YO9YH4ffvD507+6ftiRPhiSdg332ji+3HHz3hSqZhQ1i3Lr3xiERl0yaoXTt53Z57Jt9GrEcPqFnTx41dd11x+Zw5/n9+7tzUxCqyI2oJE4nRuzd8+ql/yp49G5o181lVZ5wRbVw9eviYl8RP8iHACy94vUhVcPzx8PLL/sEk0YQJPlg/kZl3OY4eDV27wo03+getnj19duQBB6Q+bpFkLNPGwefk5ITc3NyowxBJqxDg6KPhsMPgrru8JWDDBrjjDu+eyc2FamrXlgy1fr13qc+fD40aeRd7Ytd7rMsv98WUH3rIl5v44QdfWmbKFP+/sMceyc/Lz/cE7pNPfBxZ//5Qt27yY0XKi5nNDSHkJKtTS5hIBjDzlrBvvvHWuW7doGlTnzAwfboSMMlc8+b5+l1Tp0Lz5j6I/tBDfQZwaUaNgiOP9Fat5s39/8SXX/q+raUlYADZ2T7c4Kab4KKLlIBJ9NQSJpJhVq3y1fybNdt+a4FIRZef74Pi//xnnxhT5LPPvOV35kxo2bL08zdtgq+/9vGae+2V+nhFdoVawkQqkUaNfD0zJWCS6WbMgHr14hMwgLZtvaVqzJjtn1+zpo/fVAImmUpJmIiIRKJo/a5kDj3U60UqMyVhIiISiVattH6XVG1KwkREJBLHHuvfR42KL581C8aN8226RCozzakSEZFImPmG2r17w7/+5UnZkiXw1ls+O7Jp06gjFEktJWEiIhKZgw6CBQt8a7BPP/WFkZ94wjfcFqnslISJiEikqlWDPn38S6Qq0ZgwERERkQgoCRMRERGJgJIwERERkQgoCRMRERGJgJIwERERkQgoCRMRERGJgJIwERERkQhonTAREZFSrF0Ljz8On3wC++0HF14I7dtHHZVUFmoJExERSWL2bE+45s+HE06AmjX9+/33Rx2ZVBZqCRMREUmQnw/nnAOjR8OZZxaXDxsGXbp4MtaxY3TxSeWgljAREZEEM2Z492NsAgbQpAlccQU8+mg0cUnloiRMREQkwddfQ5s2yevatvV6kd2lJExERCTBIYfAe+9BQUHJurff3v7g/MWL4Zpr4NhjvUtz2jQIIXWxSuZSEiYiIpKgc2do1Ahuuy0+EXvzTXjmGbj00uTnTZsGRx0Fe+/t555yCvz613D99UrEpCQLGfavIicnJ+Tm5kYdhoiIVHKrVkG/fvDdd96qtXQpLFwITz/tA/MTbdoEBxwA//439OhRXP7DD57UPfooHHNM+uKXisHM5oYQcpLVaXakZIyFC32m0hdfwIEHwtChcPDBUUclIpVVo0Ywcya8/74vU9GrF/Tu7UtVJPPqqz6OLDYBA9hrL7jyShg7VkmYxEtpd6SZ9TSzxWa21MxuLOWYc8xsoZktMLOnUxmPZK4nn4TjjoO6deGii6B2bf9k+tRTUUcmIpWZmSdVw4Z5q1hpCRh4i1njxsnrmjSBb79NTYySuVLWEmZm2cAo4GQgD5hjZpNDCAtjjmkF3AT0CCF8Z2YNUxWPZK7/+z+49lr/RNq2rZf17QuDB/sfx1NOgYb6lyMiEevSBf7f/4PNm6FGjfi6l16Crl2jiUsqrlS2hB0BLA0hLAshbAbGAwkrrnAZMCqE8B1ACGF1CuORDDVuHJx1VnECVqRdO+jTB8aPjyYuEZFYbdt6onX55fDTT14WgndDvvwyXHxx6ee+9pp3dTZuDIcdBg88AFu2pCduiU4qk7DGwMqYx3mFZbFaA63N7D0z+8DMeqYwHslQq1f7GLBkDjwQ1qxJbzwiIqUZOxY2bIBmzaBnT2jdGu65x5Ow+vWTnzNmjA+zGDgQPvjAt0V68UXo399X7pfKK+qB+dWAVsBxQBPgbTPrEEL4PvYgMxsKDAVo1qxZumOUiHXqBA89BDffXLJuxgxfj0dEpCKoU8db57/8EhYs8KEShx/uY8uSWb8efvMbH/xf1NrftKkPtejWDaZO9RZ/qZxS2RL2FdA05nGTwrJYecDkEMKWEMIXwBI8KYsTQhgdQsgJIeQ0aNAgZQFLxdSvH6xYAaNGFa+zEwL87W8+hTxxWxERkag1a+azKTt3Lj0BA59R2aVLyeEW1av7DPAJE1Ibp0QrlUnYHKCVmbUwsxrAIGBywjGT8FYwzKw+3j25LIUxSQaqUQOmT4dHHvElKQYP9j9Yjz/u5dWrRx2hiMiu2bjRW8+SqVvX66XySll3ZAhhq5kNB6YD2cCjIYQFZnYHkBtCmFxYd4qZLQTygd+EENalKibJXC1bwocf+niJZct8tmTXrtv/hCkiUtEdfTRcdZUv6LrXXvF1EyfCSSdFE5ekh1bMl0ovPx/++1//VJmdHXU0IiLxrrwSliyBhx+GFi18nNi99/og/48+Kr2lTDLD9lbM196RUmlt2uSD+Rs18j9sjRr5402boo5MRKTY3/4GRx7pY8NatvSFXefN830qlYBVbmoJk0ophOLp3ffe63/Yli71jXSzsuC559SVKSIVy8aNPquyXj3/kspBe0dKlTN7Nnz8MSxaVLxydcuWPsbi4IO9XqtXi0hFUquWrysmVYe6I6VSeuklX/gwceuQGjW8fNq0aOISEREpoiRMKqXs7NK3/Ni6VQP0RUQkejtMwszsajPbJx3BiJSXPn18z8mi/duK/PQTPP20FngVEZHolaUlbD9gjplNMLOeZhrOLBVfx45w6qm+YvXMmT4jcuZMf3zqqV4vIpLJfvwR/vlP3zD8D3+Azz+POiLZWTtMwkIIv8e3EhoDXAh8bmZ3mtlBKY5NZLeMHg0DBvjGuLVr+/f+/b1cRCSTzZ/vO4e89prvr7txI3TvDg8+GHVksjPKvESFmXUELgJ6Am8A3YBXQwj/L3XhlaQlKiqeadP809jy5XDQQb76s1Z5FhFJjYICaNfO1z0877zi8i+/9Fnf06bBYYdFF5/E263FWs3sWjObC9wNvAd0CCFcAXQG+pdrpJJx7rjDtxA66yxf3fm00+Cyy2DEiNLPCQGefx5OOQVatYKePWFy4q6iIiKS1Lvv+p65Q4bElzdr5h+Cx4yJJi7ZeWVZJ2xf4KwQworYwhBCgZmdnpqwJBMsXQojR8Knn0LDhl7WsaOPu+rQAQYN8pWfE/3+956E3Xqrf1rLzYXf/MZXiL7llvS+BxGRTJOXB+3bJ19wun17/5sqmWGHSVgI4dbt1C0q33Akk4wfD7/6VXECVqRxYx+LNWEC3HBDfN2SJfDII7BwYfGK0K1be/dlu3betN6iRXriFxHJRG3awKxZviNI4nI7M2d6vWQGrRMmu+zHH6FBg+R1DRt6faIJE+Dcc0tuydGwoS+i+uyz5R+niEhl0rkz7L8//PnPPryjyJw58NhjMHRodLHJzlESJrvsqKN8LFfi3I4QvPyoo0qe89NPsO++yZ9vn31g/fryj1NEpLKZMAEmTYJDD4XrrvO1EXv29PFgB2ntgoyhJEx2We/esHkz3Hgj/Pyzl61f7wP169SBE04oec6xx/p4sMTEraDA/6Acd1zKwxYRyXiNG8PcufDAAz4g/+yzYcUKT8YkcygJk12WnQ0vvwyLF0PTpj41ulkzWLUKXnwx+aDRU07x/Ruvvhp++MHLvv8errzSW8iUhImIlI2Zf9i94QYfT1u7dtQRyc4qy+xIkVI1bOgtWF995WvUNG8OjRqVfnxWlm+ufdVVfmzTpn7eGWeUnriJiIhURkrCpFw0buxfZbHPPr5/49q1PtW6adOSA/VFREQqOyaCQ0YAABamSURBVCVhEpn69f1LRESkKtKYMBEREZEIKAkTERERiYC6I6WEvDwfLN+ixfYH2YuIiAshEAIUhEBB4ffix14WYuoKEo8v2MnjQyiuL9jx8fkFIa4+v8DXCUr6fLGvX5Ds/Wzn+MT3X7CTxyc+f0FZjk/2/mOOL4BA8eMQc/5lxxzI4COaRfbvRkmYbPPNN7759vvvQ8uWvsXQSSfBP/9Z+gKrIpJaRTf3/JgbSkEI5IdAKIgtL7zZbvs5/uabH3dD85vwthtWQeHzFR5fEPNa+YU3w/yC2BtezOOY1w2xz1v4PCWeNzbO2MexccaeX0Ap5fEJR35Bkvh38P4Tf69lTxJK3vRD2PG1rOyyDLLMyDLDtv1M8eMs21ZmMXWlHp9Yl1VUF3tuzHNlZZX6XEXHGP48VvjcDWrXjPR3piRMANi6FU491TfffuYZ2GMPX3j1d7/z5SPefVfLR0ixoptt0Y0tv+imuK0sxJTF1BedF3cj9RttYllp5fHnFycdcfXbyoiLp+i4+BhJHneJJKTkTT8+2WCnk5DSjo99XFAJb+7ZWfE34Owsv1Fmb7tJe/22x4U332xLclwWheUW97zVsrLK8LyFjxNv7FllSRKSJQJFycBOHl/mRCN5khKfiJR+/JNPGJOeN26+GY7sanz9tTFqJCxbZkx4BmrV3E6SlFV6klR0vOw8JWECwNSp8ItfwJ13FidbtWv7aswdOsAbbyRfAT9TFH1i3VpQQEFB8ff8EJKW5RcUkF+QcHySpKBkAkCpN/viMuJu9iHxvF1IFvILilsmSkt2tr1usvhjW1hKnE+JYzPtU3/sjTc7y2/mWVkWU0ZcWdHPRTecZDftohtmtWpFN3sje9uNtPhmb4XJQ/ENsuj5kyQh287dwflZ8TfAbcftdHITn4RsOz8mTov73RETT0xyFBt/wvtPFqek3/ffw+i74ZNPfFkggFa/hGPGeI/Hu9Ph/POjjbEqUhKWwcozsXju3QIO7wVvLYlNNvz4Q88o4Im3AuvqhoS6wNaCxLLC76WWFScMXlb8XMnKiuIsLoupS3jvRUnI1vyCkr+XDEkaihOA5ElBdtGNMrbeLC7B8LLiZKBaVhY1qxWXlzw2IRmJrS9KWpIlK3GJDEmOjT2GkmUJMRS9p+RxxSceO3OePqWLwOuvw5FHFidgRczgoot80W0lYemnJCzB8rU/8dqib1KaWGz7qkiJRTbwE0x5rJT6TfDWs2V/umpZRQlA4U0xu/jmWHSDrFZKmXclFJ6fbdSsXi2+LOG5E8uKbr6JZUWPs2PLEl4vu5Sy2PgTk4LkyUpMghDXshJfnpiYiIikQgi+Y0kyWVkl9/OV9FASlmDxNz/yp6mLktYV3WSzkyQEsclEsoQg9quoZSKuPEmikrQstm4Xy4oSjNiyVV8ZVw83xo41GtQrLl+z2jirrzF1irF/o5LPvS0ZSigTEZGK4/jj4dJLfW/f2FnvIcCTT8I550QXW1WmJCzB8W0a8vGtp8S3lmRVgcSiJSwYAFecDTffDJ06wcw5Pkbsd1dD1/ZRBygiIrtq3319o++ePeHvf4fu3eHrr+GPf4TVq2Hw4KgjrJqUhCWoUS2LGtWq5hq2t94KRxzhS1Lcdx+0agVjxmT2gHwREXG//723gl18MaxYATVrwpAhMGOGz4iX9LOQwo5gM+sJPICPOHokhPC/pRzXH3gW6BJCyN3ec+bk5ITc3O0eIiIiIqUIATZs8CQsOzvqaCo/M5sbQshJVpeyJh8zywZGAb2AdsBgM2uX5Lg6wLXArFTFIiIiIs7MlyRSAha9VPa7HQEsDSEsCyFsBsYDZyY57o/AX4CNKYxFREREpEJJZRLWGFgZ8zivsGwbMzscaBpCmLq9JzKzoWaWa2a5a9asKf9IRURERNIsshHoZpYF3Af8ekfHhhBGhxByQgg5DRo0SH1wIiIiIimWyiTsKyB2bd4mhWVF6gCHAG+a2XKgGzDZzJIOXhMRERGpTFKZhM0BWplZCzOrAQwCJhdVhhB+CCHUDyE0DyE0Bz4A+uxodqSIiIhIZZCyJCyEsBUYDkwHFgETQggLzOwOM+uTqtcVERERyQQpXaw1hDANmJZQdkspxx6XylhEREREKpKquTS8iIiISMSUhImIiIhEQEmYiIiISASUhImIiIhEQEmYiIiISASUhImIiIhEIKVLVIiIiEjmW7UKPv0U6teHTp3ALOqIKgclYSIiIpLUhg1w5ZXwwgtw2GGwYgXsuSc88YQnY7J7lISJiIhIUpdeCvn5sHw51K0LIcDTT0PPnvDxx7DfflFHmNk0JkxERERK+OILeOUVeOwxT8DAuyF/9Ss480x4+OFo46sMlISJiIhICXPmwDHHwB57lKzr3Rs++CD9MVU2SsJERESkhH32ga+/Tl739ddeL7tHSVgltno1jBgBV18Nf/sbfPtt1BGJiEimOO44+PJLeOut+PKffoIHH4QhQyIJq1JRElZJTZkCBx8MixZBy5Yweza0aQNvvhl1ZCIikgmqV/fxYGefDbfeCu+8A08+Cd27Q48ecMopUUeY+SyEEHUMOyUnJyfk5uZGHUaFtnatJ1wvvQRHHFFc/vrrMGiQD7bcc8/o4hMRkcyxeDGMHAkffggNGsCFF/rAfK0VVjZmNjeEkJOsTktUVEJPPQWnnx6fgAGccAJ06wbPPgsXXBBNbCIiklnatPHuRyl/6o6shL78Ejp0SF7XoYPXi4iISLSUhFVCrVvDrFnJ62bN8noRERGJlpKwSujcc+Htt31MWKxnnvG+/b59o4lLREREimlMWCVUty48/zz07w8dO/rX7Nnwn//4rMmaNaOOUEREKrO33oJ774WPPoJ69Xwc8lVXQY0aUUdWsaglrJLq3h2WLfN/+HvvDcOHw9KlnpCJiIikylNPeY/MmWfCu+/CAw/Ayy/7461bo46uYtESFREIwaf6rljh47MOOSTqiERERHbfhg3QrBnMmAGHHlpcvnUrHH003HCDrztWlWxviQq1hKXZ0qXQpQsMHOiL3vXqBccfD6tWRR2ZiIjI7nnjDWjXLj4BA6hWDYYNgwkToomrolISlkYbNsDJJ8NFF8GSJTBpki+ceuyxcNppUFBQ+rlz5sBll/kKxcOHw/z56YtbRESkLDZu9HHJydSt6/dBKaYkLI0mTIC2bX1wYlbhb75aNd8OIgRvvk3mwQd9RmOrVt6U27AhnHQS/Otf6YtdRERkR3r08O2Nku1VPHGiLxouxTQ7Mo0+/DD5XltmXp6b6y1lsb74Am6/3c9t1szLevaEAQP8H3uvXrDvvqmPXUREZEf22897e846Cx59FA48EH7+2Qfnv/8+/P3vUUdYsaglLI3q1fPB+MmsWAH165cs/9e/fJZJUQJWpF07T8AmTiz/OEVERHbViBE+1rlrV+/BadzYE7A33/TZ+lJMSVgaDRniU3dXrowvX7AApk/31q1E69bBAQckf75mzbxeRESkosjO9mE2K1f62pRLlsCLL0Lz5lFHVvEoCUujAw+EP/zBN9G++25fN+X2272PfNQo2Gefkuccfji8+mrJ8hDglVegc+fUxy0iIrKzatXyzb8bNIg6koorpUmYmfU0s8VmttTMbkxSf4OZLTSzT8xshpmV0uZTeVx7rc+K/PxzuO8+WLPGB+Sfe27y488+2z9FjBxZPHty61a44w5PxBLHkImIiEhmSNlirWaWDSwBTgbygDnA4BDCwphjjgdmhRB+NrMrgONCCAO397yVYbHWnbV0KQweDN995wu7zp0LBx0ETz8N++8fdXQiIiJSmu0t1prK2ZFHAEtDCMsKgxgPnAlsS8JCCG/EHP8BMCSF8WSsli1978ePPoLly+FPf9Iq+yIiIpkulUlYYyB2CHoe0HU7x18CvJSswsyGAkMBmiVOE6wizHx82OGHRx2JiIiIlIcKMTDfzIYAOcA9yepDCKNDCDkhhJwGGuEnIiIilUAqW8K+AprGPG5SWBbHzE4CbgaODSFsSmE8IiIiIhVGKlvC5gCtzKyFmdUABgGTYw8ws8OAh4A+IYTVKYxFREREpEJJWRIWQtgKDAemA4uACSGEBWZ2h5n1KTzsHqA2MNHM5pnZ5FKeTkRERKRSSenekSGEacC0hLJbYn4+KZWvLyIiIlJRaQNvERERidz69b7+5dy5vpfyeedB27ZRR5VaFWJ2pIiIiFRdCxfCwQf7dn6dOkF+Phx7rG8GXpmpJUxEREQiEwIMGgS33QaXXFJcfs010LUrHH20f6+M1BImIiIikZk9GzZvhosvji/ff39PxB5+OJq40kEtYSIiIhKZvDzvijQrWde+Pbz+eunnhgBvvAG5uVCvHvTvD3vvnbpYy5tawkRERCQybdt6a9iWLSXr3n8f2rRJft7//R8ccQRcdx2sXg3Tp8OBB8Izz2z/9d55B/r2haZNoWNHuPde2Lhx99/HrlASJiIiIpFp394TsVtv9ZatIh99BKNHw7Bhyc8791zo2RM+/tgH8E+YAG+9BVdfDQsWJD/nmWdg4EDo3RvefRdGjvSWtjPOSJ4EppqF2HecAXJyckJubm7UYYiIiEg5+eYbOPNM+O9/4eSTYflyb7F65BE466ySxy9YAKee6sdVSxhYddttsG4dPPhgfPmmTXDAATBtGhx+eHF5fj4cd5wne7/6Vfm+LwAzmxtCyElWp5YwERERidR++8HMmfDPf0Lz5jBgAKxYkTwBA1iyBHJySiZgAN26eX2it9/27srYBAwgOxuuuAImTtztt7HTNDBfREREImcGxxzjXzvStKm3hhUUQFZCc9Knn3p9oo0boW7d5M9Xty5s2LDzMe8utYSJiIhIRuncGerUgYceii/Py4O//hUuu6zkOd27w6xZsGZNybqJE+GEE1IT6/aoJUxEREQyihmMHw+nnAJTp/r3FSvgySfhppuSL+5arx4MHQr9+sFjj0GrVt76NXKkL3Nx333pfx9KwkRERCTjtG7t2x0984zvN1mvno8ra9my9HPuugvuvBN69PAuyHXrvIXszTf9/HTT7EgRERGpUjZt8pazffaBBg1S+1rbmx2pljARERGpUmrW9Ja0qGlgvoiIiEgElISJiIiIREBJmIiIiEgElISJiIiIREBJmIiIiEgElISJiIiIREBJmIiIiEgElISJiIiIRECLtcZYsQIefxy+/BLatoULLoCGDXd83oYN8O23UL++LwAnIiIisiNqCSs0bpzvyv7tt9CtG3z2GbRvD6+/Xvo5P/4IV1wBjRpBTg40bgw33gibN6cvbhEREclMSsKAvDwYPhzeegseeAAuuwzGjIGJE2HQIPj555Ln5OdD796wcaMnbKtWwZw5vpnoeeel/z2IiIhIZlESBjz5pCdb7dvHlx93nLdwTZpU8pzp0z05GzMGfvlLL2vRwhO3996DTz5JedgiIiKSwZSE4S1hBx+cvK5dO/jqq5Llr7wCAwdCVsJvsGZNOOssrxcREREpjZIwPNF6//3kde+9lzxBq17duyKT2bDB60VERERKk9IkzMx6mtliM1tqZjcmqa9pZs8U1s8ys+apjKc0Q4bAjBkwZUp8+UMPwdq10KtXyXP694cnnvCEK9a6dfDcc9C3b+riFRERkcyXsiUqzCwbGAWcDOQBc8xscghhYcxhlwDfhRBamtkg4C/AwFTFVJq994YXXvDEqnVrOOQQ+OADn/04dSpkZ5c8p2tX6NEDTj0V/vhH6NDBz7npJrj8cjjggHS/CxEREckkqVwn7AhgaQhhGYCZjQfOBGKTsDOB2wp/fhYYaWYWQggpjCupbt3giy886Vq5Ek4/HU4+ueSYryJm8Nhj8I9/wNVX+9pibdrA//yPt6yJiIiIbE8qk7DGwMqYx3lA19KOCSFsNbMfgHrA2tiDzGwoMBSgWbNmqYqXGjWgX7+yH5+d7UtbDB+espBERESkksqIgfkhhNEhhJwQQk6DBg2iDkdERERkt6UyCfsKaBrzuElhWdJjzKwasBewLoUxiYiIiFQIqUzC5gCtzKyFmdUABgGTE46ZDFxQ+PMA4PUoxoOJiIiIpFvKxoQVjvEaDkwHsoFHQwgLzOwOIDeEMBkYA4w1s6XAt3iiJiIiIlLppXJgPiGEacC0hLJbYn7eCJydyhhEREREKiLLtN4/M1sDrEjxy9QnYYamVAi6LhWPrknFpOtS8eiaVEzpuC4HhBCSzirMuCQsHcwsN4SQE3UcEk/XpeLRNamYdF0qHl2Tiinq65IRS1SIiIiIVDZKwkREREQioCQsudFRByBJ6bpUPLomFZOuS8Wja1IxRXpdNCZMREREJAJqCRMRERGJgJIwERERkQhU6STMzHqa2WIzW2pmNyapr2lmzxTWzzKz5umPsuopw3U5xsw+NLOtZjYgihirmjJckxvMbKGZfWJmM8zsgCjirGrKcF2Gmdl8M5tnZu+aWbso4qxKdnRNYo7rb2bBzLRsRYqV4f/JhWa2pvD/yTwzuzRdsVXZJMzMsoFRQC+gHTA4yR+oS4DvQggtgfuBv6Q3yqqnjNflS+BC4On0Rlc1lfGafATkhBAOBZ4F7k5vlFVPGa/L0yGEDiGETvg1uS/NYVYpZbwmmFkd4FpgVnojrHrKek2AZ0IInQq/HklXfFU2CQOOAJaGEJaFEDYD44EzE445E3ii8OdngRPNzNIYY1W0w+sSQlgeQvgEKIgiwCqoLNfkjRDCz4UPPwCapDnGqqgs1+W/MQ/3BDQTK7XKcl8B+CP+oX5jOoOrosp6TSJRlZOwxsDKmMd5hWVJjwkhbAV+AOqlJbqqqyzXRdJrZ6/JJcBLKY1IoIzXxcyuMrP/4C1h16Qptqpqh9fEzA4HmoYQpqYzsCqsrH+/+hcOp3jWzJqmJ7SqnYSJSDkzsyFADnBP1LGICyGMCiEcBPwW+H3U8VRlZpaFdwn/OupYJM6LQPPC4RSvUtwDlnJVOQn7CojNdpsUliU9xsyqAXsB69ISXdVVlusi6VWma2JmJwE3A31CCJvSFFtVtrP/V8YDfVMakezomtQBDgHeNLPlQDdgsgbnp9QO/5+EENbF/M16BOicptiqdBI2B2hlZi3MrAYwCJiccMxk4ILCnwcArwetbptqZbkukl47vCZmdhjwEJ6ArY4gxqqoLNelVczD04DP0xhfVbTdaxJC+CGEUD+E0DyE0BwfP9knhJAbTbhVQln+nzSKedgHWJSu4Kql64UqmhDCVjMbDkwHsoFHQwgLzOwOIDeEMBkYA4w1s6XAt/jFkxQqy3Uxsy7A88A+wBlmdnsIoX2EYVdqZfy/cg9QG5hYOHflyxBCn8iCrgLKeF2GF7ZQbgG+o/hDpaRAGa+JpFEZr8k1ZtYH2Irf6y9MV3zatkhEREQkAlW5O1JEREQkMkrCRERERCKgJExEREQkAkrCRERERCKgJExEREQkAkrCRERERCKgJExEREQkAkrCRKTKMrMuhZv21jKzPc1sgZkdEnVcIlI1aLFWEanSzOxPQC1gDyAvhHBXxCGJSBWhJExEqrTC/eTmABuB7iGE/IhDEpEqQt2RIlLV1cP3vayDt4iJiKSFWsJEpEozs8nAeKAF0CiEMDzikESkiqgWdQAiIlExs/OBLSGEp80sG3jfzE4IIbwedWwiUvmpJUxEREQkAhoTJiIiIhIBJWEiIiIiEVASJiIiIhIBJWEiIiIiEVASJiIiIhIBJWEiIiIiEVASJiIiIhKB/w8LZkCDWEqD/wAAAABJRU5ErkJggg==",
            "text/plain": [
              "<Figure size 720x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          },
          "output_type": "display_data"
        }
      ],
      "source": [
        "xp = np.linspace(0.0,0.5,300)\n",
        "yp = np.array([np.dot(V, sigmoid(W*xp[i]+bs)) for i in range(len(xp))])\n",
        "\n",
        "fig = plt.figure(figsize=(10,4))\n",
        "ax = fig.add_subplot(111)\n",
        "ax.set_xlabel(\"x\"); ax.set_ylabel(\"y\")\n",
        "ax.scatter(xt, yt, facecolor=\"none\", edgecolor=\"b\", s=50, label=\"Data\") \n",
        "ax.plot(xp,yp*ystd+ymean,label=\"ANN\") #ニューラルネットワークの予測ypは、\"標準化された\"yの値に従って学習されているので、元のスケールに戻さないといけない。\n",
        "ax.legend()\n",
        "plt.show(); plt.close()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zfYX7Od8NEDt"
      },
      "source": [
        "当然ですが、全然だめですね。\n",
        "\n",
        "状況を改善するためにニューラルネットワークのパラメータを徐々に更新(学習)していきましょう。  \n",
        "そのためには、まず勾配を計算する関数を用意しておきます。\n",
        "\n",
        "loss functionを$f$と書くことにすると、必要な勾配は4種類で\n",
        "$\\frac{\\partial f}{\\partial W}, \\frac{\\partial f}{\\partial V}, \\frac{\\partial f}{\\partial b}, \\frac{\\partial f}{\\partial b_0}$です。  \n",
        "プログラムではそれぞれ```dw,dv,dbs,db0```とでも名前をつけることにして、勾配を返り値として与える関数を定義します。\n",
        "\n",
        "以下では、勾配降下法, Adamの2通りの最適化手法を用いてパラメータを更新することとします。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wOxttwbLdc1E"
      },
      "source": [
        "## 勾配降下法"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nyeC4YIfjnua"
      },
      "source": [
        "勾配降下法とは、目的関数を微分した勾配の値のみを使ってパラメータを更新する方法です。  \n",
        "たとえば,重み$W$の$i$番目を更新する際には  \n",
        "$W_i := W_i - \\eta \\frac{\\partial f}{\\partial W_i}$  \n",
        "とします。($f$は目的関数で、$\\eta$は学習率(パラメータ更新のスケールを決めるパラメータ)です。)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 146,
      "metadata": {
        "id": "xyL8Tvp1r6F-",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "def calc_der(x,y,tW,tV,tbs,tb0,acf,acfder):\n",
        "    tdw = np.zeros(nhl)\n",
        "    tdv =  np.zeros(nhl)\n",
        "    tdbs = np.zeros(nhl)\n",
        "    tdb0 = 0.0\n",
        "    #以下の勾配の計算は、目的関数が二乗誤差かつ全データでの勾配の和を使用する場合にのみ正しい\n",
        "    for i in range(len(x)):\n",
        "        g = np.dot(tV, acf(tW*x[i]+tbs) ) + tb0 - y[i]\n",
        "        tdb0 += 2.0 * g\n",
        "        for jth in range(nhl):                    \n",
        "            tdv[jth]  += 2.0 * g * acf(tW[jth]*x[i]+tbs[jth])\n",
        "            tdw[jth]  += 2.0 * g * tV[jth] * acfder(tW[jth]*x[i]+tbs[jth]) *x[i]\n",
        "            tdbs[jth] += 2.0 * g * tV[jth] * acfder(tW[jth]*x[i]+tbs[jth])\n",
        "    return tdw, tdv, tdbs, tdb0\n",
        "\n",
        "#シグモイド関数の微分: 勾配の計算を具体的に求めるのに使う\n",
        "def sigmoid_der(z):\n",
        "    return np.exp(-z)/ ((1.0+np.exp(-z))**2)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UOWJiYG6Nrp5"
      },
      "source": [
        "さてW,V,bs,b0の初期値での勾配の値は"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 147,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "6Px2oXdcNzTQ",
        "outputId": "413b3795-ca17-4929-b956-278ea19b15f9",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "(array([ 5.21872673e-02, -8.03960749e+00,  1.83009108e+00,  1.91103332e+00,\n",
              "         3.24927634e+00, -7.23670010e+00, -1.12232153e+00,  7.58282404e-03]),\n",
              " array([36.82875279, 29.24447765, 49.19261588,  9.67161219, 23.84412956,\n",
              "        23.11524231, 35.19158249, 35.04412075]),\n",
              " array([ 2.14818071e-01, -3.23527933e+01,  9.04076394e+00,  8.00770445e+00,\n",
              "         1.34159140e+01, -2.80674027e+01, -4.66782630e+00,  2.97730658e-02]),\n",
              " 58.69466810466825)"
            ]
          },
          "execution_count": 147,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "acf = sigmoid\n",
        "acfder = sigmoid_der #sigmoid関数の微分sigmoid_derをacfderという名前で使う\n",
        "calc_der(xt,ny,W,V,bs,b0,acf,acfder)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "k7vvCZExdmnK"
      },
      "source": [
        "と計算できるようになりました。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 148,
      "metadata": {
        "id": "skJRBuEBrn4b",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "def fitGD(x,y,tW,tV,tbs,tb0,acf,acfder,nepoch,eta,verbose):\n",
        "    for i in range(nepoch):\n",
        "        tdw,tdv,tdbs, tdb0 = calc_der(x,y,tW,tV,tbs,tb0,acf,acfder)\n",
        "        tW = tW - eta * tdw\n",
        "        tV = tV -eta * tdv\n",
        "        tbs =  tbs -eta * tdbs\n",
        "        tb0 = tb0 -eta * tdb0        \n",
        "        if verbose == 1:\n",
        "            print(i, \"tloss =\", calc_tloss(x,y,tW,tV,tbs,tb0,acf))\n",
        "    return tW,tV,tbs,tb0,tdw,tdv,tdbs, tdb0"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "W86sRWDOSZfU"
      },
      "source": [
        "では実際に上の関数を使って、パラメータの値を更新してみましょう。  \n",
        "(nhlの値に依りますが、ちょっぴり計算に時間がかかります)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 149,
      "metadata": {
        "id": "Zy6GcfmpSZwc",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "nepoch = 2000\n",
        "acf = sigmoid; acfder=sigmoid_der\n",
        "verbose=0\n",
        "eta = 0.01 #学習率(パラメータ更新のスケールを決めるパラメータ)\n",
        "W,V,bs,b0,dw,dv,dbs,db0=fitGD(xt,ny,W,V,bs,b0,acf,acfder,nepoch,eta,verbose)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "N-vuF2Z9SRXU"
      },
      "source": [
        "更新された重み・バイアス(W,V,bs,b0)を使って、データとの二乗誤差を計算してみると..."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 150,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "uZUXTpfhS1Y2",
        "outputId": "97daaae6-ed44-4498-fc52-6a61ed1cbb00",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "学習後の二乗誤差 31.426829182338306\n"
          ]
        }
      ],
      "source": [
        "print(\"学習後の二乗誤差\",calc_tloss(xt,ny,W,V,bs,b0,acf))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3MfVJVWmTJlU"
      },
      "source": [
        "すると、さっきより小さくはなっていますが、そこまで二乗誤差が減っていません。  \n",
        "\n",
        "実際にplotしてみても"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 151,
      "metadata": {
        "id": "fSpr9Db1SQZ1",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "xp = np.linspace(0, 0.5, 500) \n",
        "yp = 0.0*xp \n",
        "for i in range(len(yp)):\n",
        "    yp[i] = np.dot(V, sigmoid(W*xp[i]+bs)) + b0 "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 152,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 279
        },
        "id": "M4SvRcWLSSyg",
        "outputId": "8dab58c0-6aed-44f0-972f-4b1df0857d19",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmEAAAEGCAYAAAAt7EI0AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3dd3xUVf7/8dfJpFFCCwklofeOEIqgYkEFFRVBQcXeFduu/lZXdy3rul/7uqi7FnStKOiKiigi9gYEpEgvUoLU0EmfnN8fJyFtAhEyuZnM+/nY7GTOvXfymYxMPnPO55xjrLWIiIiISNWK8DoAERERkXCkJExERETEA0rCRERERDygJExERETEA0rCRERERDwQ6XUAv1fjxo1t69atvQ5DRERE5LDmzZu3w1qbEOhYyCVhrVu3JjU11eswRERERA7LGLO+vGMajhQRERHxgJIwEREREQ8oCRMRERHxQMjVhImIiEhoyM3NJS0tjaysLK9DCbrY2FiSk5OJioqq8DVKwkQkpMyZA889BytWQIsWcM01cOqpXkclIoGkpaURFxdH69atMcZ4HU7QWGtJT08nLS2NNm3aVPg6DUeKSMh45RU491zo0QMefxxOOQWuvx4eeMDryEQkkKysLOLj42t0AgZgjCE+Pv539/ipJ0xEQkJ6Otx+u+sJ69jRtQ0eDOed55Ky0aOhWzdvYxSRsmp6AlboSJ6nesJEJCS8/z6cfnpRAlYoIQEuvxzeesuTsEREjpiSMBEJCbt3Q/PmgY81b+6Oi4gEMnXqVIwxLF++HIB169ZhjGHChAkHzxk/fjz//e9/Abj88stJSkoiOzsbgB07dhCM3XqUhIlISBgwAKZPh/z8ssc+/hgGDqz6mEQkNEyaNInjjjuOSZMmHWxLTEzk6aefJicnJ+A1Pp+Pl19+OahxKQkTkZBw3HGQmOjqwjIzXVtuLjz2GKxZA+ef7218IlI97d+/n++++46JEyfy9ttvH2xPSEjglFNO4dVXXw143W233cZTTz1FXl5e0GJTYb6IhARj4IMP4KqroGVLV4y/YgW0bw+ffw6xsV5HKCKH8sBHS1j6295Kfcyuzetx34hDz8j54IMPGDZsGB07diQ+Pp558+YRHx8PwJ/+9CeGDx/OlVdeWea6li1bctxxx/H6668zYsSISo27kHrCRCRkNGrkCvRTU+Gee+CLL+DrryEIpRoiUkNMmjSJsWPHAjB27NgSQ5Jt27ZlwIABvFXOzJ67776bxx57jPxAdRCVQD1hIhJyWrVyXyISOg7XYxUMO3fu5IsvvmDx4sUYY/D7/RhjuOmmmw6e8+c//5nRo0czZMiQMtd36NCB3r17M3ny5KDEp54wERERqZHeffddLrnkEtavX8+6devYuHEjbdq0YePGjQfP6dy5M127duWjjz4K+Bj33HMPjz/+eFDiUxImIhIGFi2C666D44+HMWPgs8+8jkgk+CZNmsTIkSNLtI0aNYp//OMfJdruuece0tLSAj5Gt27d6NOnT1DiM9ba4DywMS8DZwHbrLXdAxw3wNPAGUAGcLm1dv7hHjclJcWmpqZWdrgi4oHsbJg6FVaudPtAjh4Ndet6HVXNM2UKjB8Pt9ziZpmuWOFmlZ5/Pjz8sNfRSU22bNkyunTp4nUYVSbQ8zXGzLPWpgQ6P5g1Yf8FngFeK+f4cKBDwdcA4N8FtyISBhYuhBEj3Ar4Awe6gvs774TJk+Gkk7yOziWI778Py5dDcjJccAHUq1f5P2fPHnjjDVi2DJKS4NJL3e3h/PILLF3qzh00yM0eDWT/fre/5qxZ0Lu3axsyxCW8vXrBqFHQt2/lPR8RqbigDUdaa78Bdh7ilHOA16zzE9DAGNMsWPGISPWRkwNnnw2PPuqWl3joIbf8xOTJLtnZeah3jiqweDF06AATJ4K18Omn0LYtzJhRuT9n9myXhH77rbvdsMEtvVFs8lYZ27a5jcuHDXO/r+uug65dXVIbyEcfwbHHFiVghRo1gmuucQmgiHjDy9mRScDGYvfTCto2lz7RGHMtcC24dTtEpHqx1u3dOGECrFrl1vG67jq49lqICPBRb9o0aNMGCmaNH3TSSTB8OLz+Otx6a9XEXlpenksQ/+//4KKLitq//x7OOcf1PiUmHv3PyclxvVAvveR6BAvddJPrqRo0qOwMUGth5EhX1zVjBkRGFv3uhw93sTVoUPKaXbvK71lLSoK1a4/+uYjIkQmJwnxr7QvW2hRrbUpCQoLX4YhIKX/5CzzyCNx7rxu+e+IJl0hdfXXg81etgn79Ah/r18/ViHll+nSXnBRPwAAGD4Zzz4WCreUq5ee0b18yAQPo3h3GjQv8c378EXbscHVckQUfoY2Biy+GE05wv/PSBgxwCZvfX/bYJ59A//5H/VRE5Ah5mYRtAloUu59c0CYiIWTdOvj3v93CqWedBQkJcPLJMHMmfPklzJ1b9ppWrdxsvUAWLvR28dXVq8tPEFNS3PHKsGGDS7gC6d7dHS/t55/d7zZQ7+LQoe54aX37uqHVm28u2u4pPx/+8x83HDpu3JE/BxE5Ol4mYR8ClxpnILDHWltmKFJEqrcPPnDDao0bl2yvXdsVmb/7btlrzj3XJWHTp5dsnzvXFcNfdlnw4j2cNm1gwYLAxxYscMcrQ8eO8NNPgY/99JM7Xlrjxi7pDWTdurKvQaEpU2DrVjdMfPrp0K4dvPyyq8cLxmQDEamYoCVhxphJwI9AJ2NMmjHmKmPM9caY6wtOmQ6sBVYDLwI3BisWEQmenByXcAVSp447XlpsLPzvf3DFFW6W3uOPu8Tr9NPdMFxl1FwdqTPPdL1d779fsj011SUzl18e+DprXZF7//4QFwedO7th2dzcwOefeirs2wcvvFCy/euvXWJ7xRVlrxkxwiWqs2eXbN+82dWWXXpp4J/VoAG895679vbb3ePPmQOdOgU+X8Qrfr+rv/z0Uzf0Xhl8Ph+9e/emW7du9OrViyeeeOKw2xCtW7eu3K2MKlPQCvOttRce5rgFbjrUOSJS/Z16atFMx+joovb8fHjnHTfzMZBjj3W1YZMmuXWrjjnGJWNel31GR7u1y846yyWEgwbBkiVuMsErr0CzcuZw33+/SywffdRds3Qp/PWvro5rypSyS0j4fG7m4plnup8zeLC7Zs4c93sLlIjWru3OPessuPJKt+bXsmVuQsTtt5c/vFmodWvtsynV18yZbjJP/frufSA11X04e/zxohrII1GrVi0WFHRvb9u2jYsuuoi9e/fywAMPlHtNYRJ2Ueni0EoWtMVag0WLtYpUP6NHu0+wTz/thry2boU//9klV998E7iGqbo7cADefts9h6QkV/xe3nDfpk1uaYnly0smT9nZLrl87jk48cTA1+bluQL5wnXCRo4sv2ex0Jo1rqZr2TJo3hyuusoV4ItUNxVdrPWXX1y946RJbgkWgPR0t7tD375u4s+Rqlu3Lvv37z94f+3atfTr148dO3awfv16LrnkEg4cOADAM888w6BBgxg4cCDLli2jTZs2XHbZZYwcOTLgeRV5vodarFVJmIgctexsNzPy5ZehVi23QOiYMW5V9nCoOXr+eTeE8lqApakffRQ2bnS9VSLhpqJJ2NVXu1rFu+8u2b55s1sHb/36I38vKZ2EATRo0IAVK1YQFxdHREQEsbGxrFq1igsvvJDU1FS++uorHn/8caZNmwZARkZGwPMq8ny9WjFfRELY5s1uDakWLVzv1qHExLiE629/c4uJNm58+N6cmiQ319W5BRIbW35dmIg4c+a4NfJKa9bMLeWydKnbWaOy5ebmMn78eBYsWIDP52NlOevjVPS83ysEBwlEJJh273a9WN26wR13uKGAM890SdnhxMa6hC2cEjBwdXEffAAZGSXbrXXDK8OGeROXSKho2NAN65fm97v3noYNK+9nrV27Fp/PR2JiIk899RRNmjRh4cKFpKamkhNoJhFU+LzfS0mYiBxkrVsVvnFjt07Vjz+6obQ+feC009SjU55OnVyieu65RQvNbt7stgUCV0gvIuW75BLXm176PeaNN1ytZGXN5N2+fTvXX38948ePxxjDnj17aNasGREREbz++uv4C1Y1jouLY9++fQevK++8o6UkTEQO+vZbN5w4YQLUrevaYmPhwQfdJ9EPPvA2viO1apXbHHzECLjxRpg/v/J/xgsvuBmfxx3nivO7dHEzuj799OhmdomEg8suc7MiTzjB9R5//jnccgv86U+u5vJoZGZmHlyiYujQoZx22mncd999ANx44428+uqr9OrVi+XLl1OnTh0Aevbsic/no1evXjz11FPlnne0VJgvIgf94x9ur8FHHy177PHH3XDBU09VfVxHY+pU1yN11VUuSVqyBJ55xr25B2N/ytxctwF5/frl14mJhIuKFuaDmyk8ebJLwvbtcx9obrih/L1PqyMV5ovIEYuLc0syBLJtmzseSvbtc+tpzZzpatvADbdecokbYj3jDLelT2WKioImTSr3MUXCQWSk27M1yEtzVSsajhSRg0aPdkOOpfct3LHDLb9w4SGXYK5+3n/fDW8UJmCFWrRwwx+BNrwWEakq6gkTkYOaNnX1X8cf79brGTDA7fH48MNuJesKjipUG9u3l7/XY5s2boFIqVw5OfDhh7B4sVteYMyYyp3ZJqHHWospvWVEDXQk5V3qCROREm6+GV59FWbNcvskvveeqwN78EGvI/v9jjnGFfgGem/8/HPo3bvqY6rJVqxwe2Y++6y7/9VXbgHOUJ3QIUcvNjaW9PT0I0pQQom1lvT0dGJ/ZyGoCvNFwkReXvjN0rMWUlLcrMi//MXt12itG1q95x63zVDhLFA5Ovn5bm25P/yhaGkOgHnz3Mbs8+cfftFfqXlyc3NJS0sjKyvL61CCLjY2luTkZKKiokq0qzBfJEz5/fDPf7rZgOvXu+Gh6693MwOLb7ZdUxnjNskeO9Ztvj1ggJsdCW7pCCVglefLL91s0KuvLtnet6/bd3PiRDjEfslSQ0VFRdGmvJoA0XCkSE12/fWuPmfKFJeQzZjhFmAdOzbwEF1N1Ly520T8gw9g1Ci3ntcvv0D37l5HVv2tXAm33+4W6r38cvd7LM/q1dCvn0t8S0tJccdFpCQlYSI11NKlMG0aTJ/u/gga4xKPqVNdEvLDD15HWLV693ZF4oMHB04UpKSPPnK/q9q13RDjMcfApZeW35vVujUsWBD42IIF5U+QEAlnSsJEPGAtvPii+8MWFwc9erhi5vz8yvsZ06bBBRdA6YWdo6PdOjwffVR5P0tqlowMt77atGnw97+7vS9vvRXmznU9iT//XPaaoUPdQr9vvFGyffFiN9HjqquqJnaRUKKaMBEP3Hab64l6/HE3hLNggVsS4uef4aWXgv/zjQmf4Uj5/T7+2NVyDRhQsj0hwS1V8vrr7gNEcT4f/O9/bgHct96CIUPcbMmpU+Hf/1ZPmEgg6gkTqWIrVsDbb7slEk45BerVcwuKfvaZ+wrUy3AkzjjD1YJlZJRsz811fyS1qbSUJz29/JmMrVq5xXsD6dHD1ZGNGeMeo0cPNwN1zJjgxSoSypSEiVSxqVPdH6X69Uu216kD48a53oTK0L27G0Y666yiWp1ly1xxeufObl82kUBSUtyHBL+/7LHPPnPHy1OrltuN4NFHXVF/YmLw4hQJdUrCRKpYbm75GzvHxrrjleXFF10idvbZrhbs5JPdnolTpqg4XcqXkuIK7f/4R7cCPhStr/b1165AX0SOnhZrFalic+e6gvkVK0qu1eX3Q8+erkD/xBMr92daC9nZEBOj5EsqJj3dJVvz5rnasBUr3GK/kya5YUYRqRgt1ipSjfTr55ZLGDsWnnzS9TikpcGdd0JSkitormzGlN/7JhJIfLwr0F+xwg1jN2sG/fsriRepTBqOFPHApEnQvr0b9mnSxPWANW3q6sX0R06qk06d4NxzXW+Y/tsUqVwajhSpJNnZbginUaOK9zrl5BRdExMT3PhERKTqHWo4Uj1hIkcpK8sVMDdr5tZWatbMLWxZemmIQKKj3flKwEREwo+SMJGjYK0rsl+/3i0DsXmzWyF82zYYOVILooqISPmUhIkchdmz3R6NkyYVLW6ZnOy2btm48dAbHouISHhTEiZyFD77DEaPhqioku0+n+shmzHDm7hERKT6UxImchSiolxNWCCZmSXXARMRESkuqEmYMWaYMWaFMWa1MeauAMdbGmO+NMb8bIxZZIw5I5jxiFS2kSPdPpB79pRs378f3nwTzjvPm7hERKT6C1oSZozxAc8Cw4GuwIXGmK6lTrsXmGytPQYYCzwXrHhEgqFzZ7cP5NChMHMm7NoFX3wBp54KZ57p1v8SEREJJJgr5vcHVltr1wIYY94GzgGWFjvHAvUKvq8P/BbEeESC4p//hJdfhj/9CdasgTZt4Prr4dprvY5MRESqs2AmYUnAxmL304ABpc65H/jMGHMzUAcYGsR4RCokLw+mT4e1a11CdcYZZQvvizMGrrrKfYmIiFSU14X5FwL/tdYmA2cArxtjysRkjLnWGJNqjEndvn17lQcp4WPRIujQAR59FH79FR5/3N1fsMDryETkaB04AC+9BLfcAg8/7JaREfFSMJOwTUCLYveTC9qKuwqYDGCt/RGIBRqXfiBr7QvW2hRrbUpCQkKQwpVwl5Xl6rgefhi++w6efhq+/dYlZGee6WY7ikhoWrzY7YM5bRq0bQubNkHv3vDKK15HJuEsmEnYXKCDMaaNMSYaV3j/YalzNgCnABhjuuCSMHV1iSfeew+6dYMLLyzZfsEF7s168mRv4ipu1iw347JnTzj7bDdsKhKOPv7YTYhJTIRevdyHptzcwOfm58OoUfDIIzB1Ktx2Gzz7LPz4o6vlXLGiamMXKRS0JMxamweMB2YAy3CzIJcYYx40xpxdcNofgWuMMQuBScDlNtR2FJcaY9kyGDQo8LHBg2H58qqNp7Qnn3R1Z8OHw2uvuT8qt90G99/vbVwiVW3CBLj5ZrjmGldC8Mwz8NFH7gNTfn7Z87/8EuLi4KKLSrZ37AhXX+2GKEW8EMzCfKy104Hppdr+Wuz7pcDgYMYgUlHJyfDVV4GPLV0KAwdWaTglbNoEDz3k/uAkJ7u23r3dpIFu3dwfl44dvYtPpKrs3g1//SvMn+8mzgA0bep6hfv1c7tUDB9e8pr166FHDzeJprSePeF//wt+3CKBeF2YL1JtjB3r1vqaPbtk+9y57g2+9KfoqvTuu24YsjABK5SQAOPGuQVjRcLBJ5/ACScUJWCFoqNdz9i775a9pmNHmDMHAo2zzJ6tDzDinaD2hImEkgYN4PXXXRH+qFHQt6/7tD1lCrz6KjRq5F1s+/a5hCuQxERIT6/aeES8kp0NdesGPlanTuBtxAYPhpgYVzd2221F7XPnun/z8+YFJ1aRw1FPmEgxZ5wBv/ziPmXPmQMtW7pZVSNGeBvX4MGu5qX0J3lr4YMP3HGRcHDSSfDpp+6DSWmTJ7ti/dKMcUOOL7wAAwbAXXe5D1rDhrnZka1aBT9ukUBMqNXBp6Sk2NTUVK/DEKlS1sLxx8Mxx8A//uF6AjIz4cEH3fBMaipEql9bQtT+/W5IffFiaNbMDbGXHnov7rrr3GLKzz/vlpvYs8ctLTNtmvu3UKtW4Ov8fpfALVrk6shGjYJ69QKfK1JZjDHzrLUpgY6pJ0wkBBjjesK2bnW9cwMHQosWbsLAjBlKwCR0LVjg1u/6+GNo3doV0ffs6WYAl+fZZ+HYY12vVuvW7t/Ehg1u39byEjAAn8+VG9x9N1xxhRIw8Z56wkRCzObNbjX/li0P3VsgUt35/a4o/u9/dxNjCi1f7np+f/wR2rcv//rsbPjtN1evWb9+8OMVORLqCROpQZo1c+uZKQGTUDdrFsTHl0zAADp3dj1VEyce+vqYGFe/qQRMQpUGMURExBOF63cF0rNneO4IYa0l30KuPx9/viUv35Kfb/Fbd5tvOfi9P9+Sb92XP59S9wu/L2gvfAzLwWv91mILrz34+EXX+vMp9XguPnDt1oIt9j24x7a4OtZ8676n4Lb0NRR+X+qa4r8Hiy1oBw5+H/hxCq9xZxbFenC87+CxonjPPSaJ07s1Dc6LWQFKwkRExBMdOsC//uX++JZeSPVI1+/Kz7fk5ueTk5dPrt+S63ff5/jzD36f688nu/B4sWPZBceK2uzBtjx/Pnn5ljy/LbgtSpLy8vOL2guO5RUkOiWvK/ren1/yfvHHqIkiDBhjMECEMeD+R4QxmGLfH2yPcOcaY4gwAO68CAPm4PfuPxpjOHi/8Bp3ReH/HbwpccwY2J2RUyXPvzxKwkREpFJZa8nx55OZ4ycz1092bj5ZeQW3uX6y8vLJzvWT1SCf3GQ/1zySz8DBfrLz3PF1G/J5b52fkSn53Pq2n6zcomPu1l2fXZAsFSZWuQWJU2WL8hkiIyKI9BkiIwyRvoiCW9fuizAl7kdGGHwRhlpRPnwxkUT53P3C63wRhqiICHw+Q1SEwRcRUXROwXm+CEOUzxBhXLsvouj7iIKEo7DdGIPPGHwRrv3geRGuPcK4pKbwMSIMpR6v6NqD5xReW9he8LiYsglVYQJdOqEypijpkcCUhImIhCl/vuVATh77s/I4kJ3HgRw/GTl5ZOb4ycjxF9zmkZHrP9jm2vPcbW6Athw/Gbl+/BXt0ekFn++Gzz8uuG/B5vlo2CuCeZt8xERFEBtZdFs3JpL4Oj5ioyKIjowgJjKCaF8EUT53v/DWtRmiI30Ft4HOM0T7fERFmhLHon0RRBV7DCUSEixKwkREQkxOXj57MnPZm5XLgWyXRO3LdonU/uw89mUVfb8/q+C2+FdBW0aOv8I/0xioHeWjVnQktaN91I72UavgtmHt6DJttaMjqRXl7seWSqRioiKIiXTtMZE+Ik0E33zpY+UyH0nNDOefb4iLC+IvUKSaUBImIlLFrLUcyPGzJzOXPRkumdqTmcvezKLbvVl5JdoKk649mblk5eYf9mdERhjiYiOpExNJ3ZhI4mIjaVQnmpaNarv26EjqxrpjdWPceXVifNSKKp1kufsxkRFB7REac17QHlqk2lISJiJyFKy17M3KY9eBHHZm5LjbAznszsgtcX9XRlH77szcQw7XGQNxMZHUrx1Fvdgo6teKol1CXerXiipoi6R+rSjq1YoqkUTFxRZ9H+ykSUSOnpIwEZFiCnuptu/LZsf+bHYU3G7fl832/Tklkq1dGTnsyig/oYryGRrWjqZRnWga1o6mU9M4GtaOpkFtl1jVr1WUZNWrVXQbFxNJRIQSKJGaTkmYiISFzBw/W/dmsb14YrU/pyjZKki0duzPDjjcZwzE1ylKqNon1qVhnWgaFSRVjepEH7zfqI5rqxsTqd4oESmXkjARCWn5+ZadGTls2ZPF1r1ZbNmbxdY97nbL3my27sli855M9mbllbm2MLFqXDeGxnVjaN26Do3ruvsJcTEH2xPiYmhUJxqfeqdEpBIpCRORastay+6MXDbtziRtVyabdmfy2+7MEonWtr3Z5PhL9lxFGEiIi6FpvVhaxddmQNtGNKkXS5N6sSQWJldxrtcq0qfd20TEG0rCRMQz+fmWbfuy2bQ742CStanUbellFGpF+WhaP5am9WLp19olV03rxdC0vkuymtaPJaFujJIrEan2lISJSFDtz85jffoBNqRnsH5nBuvTM9iw8wAbd2ayeU9mmRXOG9aOIqlhLdom1OH4DgkkNaxFUoNaJBfcNqgdpTorEakRlISJyFGx1pJ+IIf16QdYn16YZGW4xGtnBjv2l9ybrWHtKFrF16FXiwac0aMZSQ1rkdyg1sFkq06M3pak+tixA/77X1i0CJo0gcsvh27dvI5Kagq924lIhWTl+lmXfoA12w6wdvt+1u44wJrt+1m7/QD7s4uK3o2B5vVr0bJRbYZ2aULL+Nq0alSHVvG1aRlfm3qxUR4+C5GKmzMHRoyAYcPg5JNh9Wp3e9ddcPvtXkcnNYGSMBE5yFrL9n3ZrNlelGCt2b6ftTv2k7YrE1ts5LB5/VjaJdZlVJ8kWjcuSLIa1aFFo1rERPq8exIilcDvhwsugBdegHPOKWq//nro188lY716eRef1AxKwkTCVPr+bFZs3cfKLftYsXU/K7fuY+XWfewrtpRDrSgfbRPq0LtFQ0b1SaZtQl3aJdShTeM61I7W24fUXLNmueHH4gkYQHIy3HADvPwyPP20N7FJzaF3UZEabn92nkuwtuxj+ZZ9B5Ot4rVaDWpH0alJHOf2TqJ9Yl3aJtShXUJdmtaL1crtEpZ++w06dQp8rHNnmDKlauORmklJmEgNYa3ltz1ZLNm0hyW/7WXJb3tZtnkvm3ZnHjyndrSPDk3iOLlzIh2bxNGpqftKqBujGYcixXTvDn/7G+TnQ0Sp1U6++ebQxfkrVsCzz8LChUXF/MOHu3pJkeKUhImEIH++Ze32/SzdvLcg4XKJ1+6MXMAtVto2oS59WjXkogEt6VSQcCU1qKWeLZEK6NsXmjWD++93X4WJ2FdfwTvvwM8/B75u+nS47DI3ZHn//bBmDfzxj/DZZ/DUU0rEpCRjbeCNZ6urlJQUm5qa6nUYIlUmz5/Pqm37WZS2m0VpLtlavmXvwf0NoyMj6Nw0jm7N69G1eX26Na9H56ZxqtkSOUqbN8PIkbBrFwwZ4mZHLl0Kb73lCvNLy86GVq3gvfdg8OCi9j17XFL38stwwglVF79UD8aYedbalEDH9C4tIWPpUjdT6ddfoW1buPZa6NLF66gql7WWtF2ZLNi4m0Vpu1m4cQ+LN+0hM9etGh8XG0m35vW4qH8rujWvR7ekerRLqEuUVocXqXTNmsGPP8IPP8DixW5I8YwzICYm8PkzZ7o6suIJGED9+nDjjfD660rCpKSgJmHGmGHA04APeMla+38BzrkAuB+wwEJr7UXBjElC02uvwR13uOnhV1wB8+a5T6ZPPgnjxnkd3ZHbdSCHBRt3szBtNws37mZh2h52HnAF89GREXRvXo+x/VvQK7kBvVo0oHV8bdVuiVQhY1xSVTqxCmTXLkhKCiWinrYAACAASURBVHwsORm+/75yY5PQF7QkzBjjA54FTgXSgLnGmA+ttUuLndMBuBsYbK3dZYxJDFY8Erq2bIFbb3WfSDt3dm3nngsXXujeGE87DRJD4L8cay1rdxxg3rpdpK7fSer6XazdfgBwb/QdE+MY2iWRnskN6N2iAZ2axqmHSySE9OsH/+//QU4OREeXPPbJJzBggDdxSfUVzJ6w/sBqa+1aAGPM28A5wNJi51wDPGut3QVgrd0WxHgkRE2aBOedV5SAFeraFc4+G95+G265xZvYDiUr18/iTXtIXbeLeet3MX/DroO9XA1qR9G3ZUNG903mmBYN6ZFcn7rarkckpHXu7BKt666DZ56BOnXAWnjjDfj0U3jssfKv/fxz17O/cKH7UHn55W4IM0obTNRowXzXTwI2FrufBpT+HNARwBjzPW7I8n5r7adBjElC0LZtrgYskLZtYfv2qo2nPPuyckldt4uf1qYzd91Oftm0lxy/K55v07gOJ3dOJKVVQ1JaN6Rt47qapShSA73+OlxzDbRs6XrG1qyBWrVcEta4ceBrJk50Mykfegief95d89BDbsHY998HnzagqLG8/ugdCXQATgSSgW+MMT2stbuLn2SMuRa4FqBly5ZVHaN4rHdv98Z0zz1lj82a5V0v2P7sPOau28lPa9P5ae1Oftm0B3++Jcpn6JFUn8sHt6Zvq4b0bdWQxnXLqeQVkRolLs71zm/YAEuWuF6tPn3KX5pi/364805X/F/Y29+ihSu1GDgQPv7Y9fhLzRTMJGwT0KLY/eSCtuLSgNnW2lzgV2PMSlxSNrf4SdbaF4AXwC1REbSIpVoaORL+/Ge3+OGNN7o3M2thwgQ3hbz0tiLBcuBg0uUSr8XFkq5eyQ248cR2DGwbT5+WDakVrY+uIuGsZUv3dTgzZ7oes9LlFlFRbgb45MlKwmqyYCZhc4EOxpg2uORrLFB65uNU4ELgFWNMY9zw5NogxiQhKDoaZsyA8893idcxx8D8+a7eYsaM4NVM5PrzWbBxN9+u2sF3q7azMM0lXZERhl4tGnDDkIKkq1UDrcklIkckK8v1ngVSr547LjVX0P5yWGvzjDHjgRm4eq+XrbVLjDEPAqnW2g8Ljp1mjFkK+IE7rbXpwYpJQlf79i7x+uknWLvWzZYcMKByV5+21rIuPYNvV23n21U7+HFNOvuz84gw0DO5Aded0JZj28XTt1VDJV0iUimOPx5uuskt6Fq/fsljU6bA0KHexCVVQyvmS43n98Peve5TZekC1z0ZufywZgffrNrBt6u2k7bL7bOY3LAWx3dI4IQOjRnUrjH1a2uKkogEx403wsqV8OKL0KaNqxN74glX5P/zz+X3lElo0Ir5Epays+HBB90bW+G6PVdfbRlz3T6+W7uNL5dvY/6GXeRbqBsTybHt4rnuhLYc3yGBVloUVUSqyL/+BQ884GrDGjSAHTvgpJPcPpVKwGo29YRJjWQtjBrlesH+/oifLfk7mDpnGzMWbyM3yhVZdE+qx0mdEhnSMYFeLRpoYVQR8VRWlptVGR/vvqRmUE+YhJ1pX2bw8/5tHD9mG+e9kU5OXj61o30M6dmYWa914Lm/JDL8xFivwxQROSg2Fjp29DoKqUpKwqRGsNaydPNeZizZymdLtrB8yz7oAxt21ubiAS05uXMi/ds0IibSx59XwE9fwvATvY5aRETCmZIwCVn+fMu89buYsWQLM5ZsIW1XJsZAv1aN6B/dhUaZifznzrplrsvLc584RUREvHTYJMwYczPwRuH+jiJeys7z88PqdGYs2cLny7ayY38O0b4IBrePZ/xJ7RnatQmN68awcCGceSYcuN+tJ1bowAF46y23CrWIiIiXKtIT1gSYa4yZD7wMzLChVs0vIS0r1883K7czbdFmvli+jf3ZedSNieTETgmc3q0pJ3ZKIC625BISvXrB6afD8OHwyCNu25D58+FPf3LtvXp59GRERCrJvn3w5ptuGYvERLj0UujQweuo5Pc4bBJmrb3XGPMX4DTgCuAZY8xkYKK1dk2wA5TwlJOXz3ertzNt4WZmLt3Kvuw8GtaO4qyezTi9W1MGtY8nJvLQWwO98ILb6uiKK9yGuO3awQ03wPjxVfQkRESCZPFiGDYMjj0WTjnFLWI9aBD89a9w881eRycVVeElKowxvXBJ2DDgS2AgMNNa+/+CF15ZWqKi+pk+Hf7zH1i3ziU6N910ZKs85/rz+WFNOtMW/saMJVvYm5VHvdhITu/WlLN6NWdQu3gtIyEiYS8/H7p2hXvugUsuKWrfsMHtJDJ9utveTaqHo1qiwhhzK3ApsAN4Cbe1UK4xJgJYBVRpEibVy4MPulWd77nH/aOfOxeuucYlYnfcEfgaa2HqVPj3v2Htr5ZmvdJpMXgzy/ZtZldGLnVjIjmtaxPO7NmM4zskEB2pxEtEpNB337k9c8eNK9nesqV77504EZ55xpvY5PepSE1YI+A8a+364o3W2nxjzFnBCUtCwerV7h/6L7+4egRwtVbDh0OPHjB2LCQnl73u3nvhvc/3MXDMJnYe2MTGA1ls3OSjbUwT/u+SZgzpmEBs1KGHGkVEwlVaGnTrFnjv3G7dQINFoaMiNWH3HeLYssoNR0LJ22/DxRcXJWCFkpJg9GiYPBn+8Iei9m37spg48zde37aJiJP28u0OwwkdGjOyTxd6N25Cn14+Ol4NsdqmUUSkXJ06wezZbkeQ0vvh/vijOy6hQeuEyRHbtw8SEgIfS0x0xzNz/Hy2dAv/m7+Jb1dtJ99Co4b1ueWsrozo1ZyEuJiD14wZA+++C3feWUVPQEQkBPXtC82bw9//Dn/5S1GP2Ny58Mor8NNP3sYnFackTI7Ycce5N4G77y7ZLZ6fb/nft7s4ZtRGUh7azIEcP83rx3L9kHas+yqJZnFxXHlc2cdr2BD276+6+EVEQtXkyTBiBEyZUjQ78vvvXRLWrp3X0UlFaQNvOWJ+P/TrB6eeCvfdB/v9WUz6cRMvfL6RAxEHqBsTyRk9mjLymGQGtGlERITh00/hz3+GefNKJ27QsydMmAAnneTdcxIRCRXWwpdfwoIFblRi5EioW3aTEPHYoWZHKgmTo/LblnwuvmM7K/M2EtVyG0RY6mY25I7zWnDBsc2oHV2yszU/361lk5LietHq14fdu+Guu2DpUvj668DFpiIiIqHoqJaoEAnk1x0HmJy6kffmpbEtOZvEWtEMTmrDxce24Nhu5X8Ui4iATz5x06hbt4YWLdzaNiNGwEcfKQETEZHwoSRMKiwnL59Pl2zhzZ/WM/vXnUQYOKlTIhf0a8HJnRMrvJBqw4Zu/8YdO9xU6xYtID4+yMGLiIhUM0rC5LA27c5k0uwNvD13Azv259CiUS3uPL0To/ok07R+7BE/buPG7ktERCQcKQmTgPLzLd+s2s4bP23gi+VbscApnRO5eGArhnRIICJC44YiIiJHQ0mYlLDzQA5TUjfy1pwNrE/PoHHdaG44sR0X9m9JcsPaXocnIiJSYygJEwAWp+3hlR9+ZdqizeTk5dM5vhEPDuvE2OOaau9GEZEaJj/fTZQSb+klCGN5/nw+WbyZ8//zAyOe+Y5PF28hblsLMt47gd3vHctt5zVn3EUR7NzpdaQiIlIZ3nwTjjkGIiPd2mJ33ul2NxFvKAkLQ3syc3nxm7UMeewrbnhzPlv2ZnHPGV2Jmn4Kp8V3Z/2iOH76yS0d0bSpWz4ixJaTExGRUh57DB56yN3m5rrtjbZtcwtuZ2d7HV140mKtYeTXHQf47/e/MmVeGhk5fga0acSVx7VhaJcmTPvI8MgjbtuL4mt1WQs9esC//gUnn+xd7CIicuR274Y2bWDRIrcsUCFrYehQuOwyuPRS7+KrybRYaxiz1vLjmnQmfvcrX6zYRmSE4exeSVwxuDXdk+ofPO+77+Dss8sulmqMa//+eyVhIiKh6osv4NhjSyZg4N7jr7gCpk5VEuYFJWE1VJ4/n48Xb+b5r9eydPNe4utEc/PJHRg3sCWJcWXX9oqLg+3bAz/Wtm3QvXuQAxYRkaCxtvxC/IgIlZx4RcORNUxmjp/JqRt58du1pO3KpF1CHa49oS3n9E4iNspX7nUrV8Lxx8PixZCYWNSeluY21l68GJKSquAJiIhIpdu5E9q1c3v0NmtW1G4tDB8OF1wAV17pXXw1mYYjw8CuAzm8+uM6Xv1hHbsycunTsgF/PasrQ7s0qdDCqh07wo03wuDBcM890Ls3zJ0LDz8M996rBExEJJQ1agR/+AMMGwbPPQeDBsFvv8Hf/uZGOy680OsIw5OSsBC3cWcGE7/7lXfmbiQz18/QLolcN6Qd/Vo3+t2Pdd990L8//Oc/8OST0KEDTJyoWjARkZrg3ntdL9iVV8L69RATA+PGwaxZUKuW19GFp6AORxpjhgFPAz7gJWvt/5Vz3ijgXaCftfaQY40ajnSWb9nLv79aw7RFmzHAOb2TuG5IWzo2ifM6NBERqcashcxMl4T5yq9SkUriyXCkMcYHPAucCqQBc40xH1prl5Y6Lw64FZgdrFhqkl827eFfs1bx2dKt1In2ccWg1lx5XBuaN9DHGBEROTxjoLZ2oasWgjkc2R9Yba1dC2CMeRs4B1ha6ry/AY8AdwYxlpA3f8MuJsxaxZcrthMXG8ktp3TgysGtaVA72uvQRERE5AgEMwlLAjYWu58GDCh+gjGmD9DCWvuxMabcJMwYcy1wLUDLli2DEGr1NXttOhO+WM13q3fQsHYUd57eiUuObUW92CivQxMREZGj4FlhvjEmAngSuPxw51prXwBeAFcTFtzIvGet5fvV6fzri1XM+XUnjevG8OczOnPxgFbUidFcChERkZogmH/RNwHF1+ZNLmgrFAd0B74ybpn2psCHxpizD1ecX1NZa/l65XaenrWKnzfspkm9GO4b0ZUL+7c85BpfIiIiEnqCmYTNBToYY9rgkq+xwEWFB621e4DGhfeNMV8Bd4RrAvbD6h08MXMl89bvIqlBLf52bnfO75us5EtERKSGCloSZq3NM8aMB2bglqh42Vq7xBjzIJBqrf0wWD87lMxdt5MnPlvBT2t30rReLH8f2Z3z+7YgOrKc/SVERESkRghqgZG1djowvVTbX8s598RgxlLdLNy4mydmruSbldtpXFfDjiIiIuFGVd5VbOlve3ly5ko+X7aVhrWjuHt4Zy49tjW1opV8iYiIhBMlYVVk9bZ9PDVzFR8v3ky92EjuOK0jlw9uQ13NdhQREQlLygCC7LfdmTw1cyXvzU+jVpSPW05uz1XHt6V+La3zJSIiEs6UhAXJ7owcnvtqDf/9YR1YuHJwG248qT2N6miFexEREVESVukyc/y88sOv/PurNezPzuO8Y5K5/dQOJDfURl0iIiJSRElYJcnz5zNlXhr//HwlW/dmc0rnRO4c1onOTet5HZqIiIhUQ0rCjpK1lhlLtvLojOWs3X6APi0bMOHCPvRv08jr0ERERCrF5s3wyy/QuDH07g1uoxs5WkrCjsK89Tt56ONl/LxhN+0T6/LCJX05tWsTjP7rFBGRGiAzE268ET74AI45Btavhzp14NVXXTImR0dJ2BHYkJ7BI58u5+PFm2lSL4ZHR/XkvD5JRPq0yr2IiNQcV18Nfj+sWwf16oG18NZbMGwYLFwITZp4HWFoUxL2O+zJzOW5L1fzyvfr8EUYbh/akWtOaEPtaP0aRUSkZvn1V/jsM9iwAWrVcm3GwMUXwzffwIsvwr33ehtjqFP2UAG5/nwmzdnAPz9fxa6MHEb3SeaO0zvRpF6s16GJiIgExdy5cMIJRQlYcWec4ZIwOTpKwg7BWsuXK7bx94+XsWb7AY5tG8+9Z3WhW/P6XocmIiISVA0bwm+/BT7222/uuBwdJWHlWPrbXv4+fSnfr06nbeM6vHRpCqd0SQypovtt2+C111whZYcOMG4cNNKkTRERqYATT3RDkV9/DUOGFLUfOAATJsBTT3kWWo2hSvJStu3N4k/vLuLMCd+y5Le93D+iKzNuP4GhITbrcdo06NIFli2D9u1hzhzo1Am++srryEREJBRERcErr8D558N998G337oP9oMGweDBcNppXkcY+tQTVsrCtD387+c0rhrchptP7kD92qG3x+OOHXDZZfDJJ9C/f1H7F1/ABRe4Yss6dbyLT0REQsNpp7nk65ln4K67ICEBHngAzjlHa4VVBmOt9TqG3yUlJcWmpqYG7fGttWzek0XzBgEqEUPEP/8JP//s1nEp7eyzYdQol6SJiIhIcBlj5llrUwId03BkKcaYkE7AwI3h9+gR+FiPHu64iIiIeEtJWA3UsSPMnh342OzZ7riIiIh4S0lYDXTRRW4hvU8+Kdn+zjuwYgWce643cYmIiEgRFebXQPXqwfvvu9qvXr3c15w5sGaNmzUZE+N1hCIiUpN9/TU88YSrT46Pd3XIN90E0dFeR1a9qCeshho0CNaudf/hN2gA48fD6tUuIRMREQmWN95wIzLnnAPffQdPPw2ffuru5+V5HV31otmRHrAW5s93i6h27Ajdu3sdkYiIyNHLzISWLWHWLOjZs6g9Lw+OPx7+8Ae37lg40ezIamT1aujXD8aMcYveDR8OJ50Emzd7HZmIiMjR+fJL6Nq1ZAIGEBkJ118Pkyd7E1d1pSSsCmVmwqmnwhVXwMqVMHWqWzh1yBA480zIzy//2rlz4Zpr3MJ548fD4sVVF7eIiEhFZGW5uuRA6tVzfweliJKwKjR5MnTu7IoTIwp+85GRbjsIa133bSATJrgZjR06uK7cxEQYOhTefLPqYhcRETmcwYPdCvs7d5Y9NmUKnHxy1cdUnWl2ZBWaPz/wXlvGuPbUVNdTVtyvv7otIubPd+PsAMOGwejR7j/24cO1KbeIiFQPTZq40Z7zzoOXX4a2bSEjwxXn//ADPPec1xFWL+oJq0Lx8a4YP5D166Fx47Ltb77pZpkUJmCFunZ1CdiUKZUfp4iIyJF6/HFX6zxggBvBSUpyCdhXX7nZ+lJESVgVGjfOTd3duLFk+5IlMGOG690qLT0dWrUK/HgtW7rjIiIi1YXP58psNm50a1OuXAkffQStW3sdWfWjJKwKtW0Lf/kLDBwIjz7q1k154AE3Rv7ss9CwYdlr+vSBmTPLtlsLn30GffsGP24REZHfKzYWOnWChASvI6m+gpqEGWOGGWNWGGNWG2PuCnD8D8aYpcaYRcaYWcaYcvp8ao5bb3WzIletgiefhO3bXUH+RRcFPv/8892niGeeKZo9mZcHDz7oErHSNWQiIiISGoK2WKsxxgesBE4F0oC5wIXW2qXFzjkJmG2tzTDG3ACcaK0dc6jHrQmLtf5eq1fDhRfCrl1uYdd586BdO3jrLWje3OvoREREpDyHWqw1mLMj+wOrrbVrC4J4GzgHOJiEWWu/LHb+T8C4IMYTstq3d3s//vwzrFsHDz2kVfZFRERCXTCTsCSgeAl6GjDgEOdfBXwS6IAx5lrgWoCWpacJhgljXH1Ynz5eRyIiIiKVoVoU5htjxgEpwGOBjltrX7DWplhrUxJU4SciIiI1QDB7wjYBLYrdTy5oK8EYMxS4Bxhirc0OYjwiIiIi1UYwe8LmAh2MMW2MMdHAWODD4icYY44BngfOttZuC2IsIiIiItVK0JIwa20eMB6YASwDJltrlxhjHjTGnF1w2mNAXWCKMWaBMebDch5OREREpEYJ6t6R1trpwPRSbX8t9v3QYP58ERERkepKG3iLiIiI5/bvd+tfzpvn9lK+5BLo3NnrqIKrWsyOFBERkfC1dCl06eK28+vdG/x+GDLEbQZek6knTERERDxjLYwdC/ffD1ddVdR+yy0wYAAcf7y7rYnUEyYiIiKemTMHcnLgyitLtjdv7hKxF1/0Jq6qoJ4wERER8UxamhuKNKbssW7d4Isvyr/WWvjyS0hNhfh4GDUKGjQIXqyVTT1hIiIi4pnOnV1vWG5u2WM//ACdOgW+bssW6N8fbrsNtm2DGTOgbVt4551D/7xvv4Vzz4UWLaBXL3jiCcjKOvrncSSUhImIiIhnunVzidh997merUI//wwvvADXXx/4uosugmHDYOFCV8A/eTJ8/TXcfDMsWRL4mnfegTFj4Iwz4Lvv4JlnXE/biBGBk8BgM7b4Mw4BKSkpNjU11eswREREpJJs3QrnnAN798Kpp8K6da7H6qWX4Lzzyp6/ZAmcfro7L7JUYdX990N6OkyYULI9OxtatYLp06FPn6J2vx9OPNElexdfXLnPC8AYM89amxLomHrCRERExFNNmsCPP8J//gOtW8Po0bB+feAEDGDlSkhJKZuAAQwc6I6X9s03briyeAIG4PPBDTfAlClH/TR+NxXmi4iIiOeMgRNOcF+H06KF6w3Lz4eIUt1Jv/zijpeWlQX16gV+vHr1IDPz98d8tNQTJiIiIiGlb1+Ii4Pnny/ZnpYG//wnXHNN2WsGDYLZs2H79rLHpkyBk08OTqyHop4wERERCSnGwNtvw2mnwccfu9v16+G11+DuuwMv7hofD9deCyNHwiuvQIcOrvfrmWfcMhdPPln1z0NJmIiIiIScjh3ddkfvvOP2m4yPd3Vl7duXf80//gEPPwyDB7shyPR010P21Vfu+qqm2ZEiIiISVrKzXc9Zw4aQkBDcn3Wo2ZHqCRMREZGwEhPjetK8psJ8EREREQ8oCRMRERHxgJIwEREREQ8oCRMRERHxgJIwEREREQ8oCRMRERHxgJIwEREREQ8oCRMRERHxgBZrLWb9evjvf2HDBujcGS67DBITD39dZibs3AmNG7sF4EREREQORz1hBSZNcruy79wJAwfC8uXQrRt88UX51+zbBzfcAM2aQUoKJCXBXXdBTk7VxS0iIiKhSUkYkJYG48fD11/D00/DNdfAxIkwZQqMHQsZGWWv8fvhjDMgK8slbJs3w9y5bjPRSy6p+ucgIiIioUVJGPDaay7Z6tatZPuJJ7oerqlTy14zY4ZLziZOhKZNXVubNi5x+/57WLQo6GGLiIhICFMShusJ69Il8LGuXWHTprLtn30GY8ZARKnfYEwMnHeeOy4iIiJSHiVhuETrhx8CH/v++8AJWlSUG4oMJDPTHRcREREpT1CTMGPMMGPMCmPMamPMXQGOxxhj3ik4PtsY0zqY8ZRn3DiYNQumTSvZ/vzzsGMHDB9e9ppRo+DVV13CVVx6Ovzvf3DuucGLV0REREJf0JaoMMb4gGeBU4E0YK4x5kNr7dJip10F7LLWtjfGjAUeAcYEK6byNGgAH3zgEquOHaF7d/jpJzf78eOPwecre82AATB4MJx+Ovztb9Cjh7vm7rvhuuugVauqfhYiIiISSoK5Tlh/YLW1di2AMeZt4BygeBJ2DnB/wffvAs8YY4y11gYxroAGDoRff3VJ18aNcNZZcOqpZWu+ChkDr7wC//433HyzW1usUye44w7XsyYiIiJyKMFMwpKAjcXupwEDyjvHWptnjNkDxAM7ip9kjLkWuBagZcuWwYqX6GgYObLi5/t8bmmL8eODFpKIiIjUUCFRmG+tfcFam2KtTUlISPA6HBEREZGjFswkbBPQotj95IK2gOcYYyKB+kB6EGMSERERqRaCmYTNBToYY9oYY6KBscCHpc75ELis4PvRwBde1IOJiIiIVLWg1YQV1HiNB2YAPuBla+0SY8yDQKq19kNgIvC6MWY1sBOXqImIiIjUeMEszMdaOx2YXqrtr8W+zwLOD2YMIiIiItWRCbXRP2PMdmB9kH9MY0rN0JRqQa9L9aPXpHrS61L96DWpnqridWllrQ04qzDkkrCqYIxJtdameB2HlKTXpfrRa1I96XWpfvSaVE9evy4hsUSFiIiISE2jJExERETEA0rCAnvB6wAkIL0u1Y9ek+pJr0v1o9ekevL0dVFNmIiIiIgH1BMmIiIi4gElYSIiIiIeCOskzBgzzBizwhiz2hhzV4DjMcaYdwqOzzbGtK76KMNPBV6XE4wx840xecaY0V7EGG4q8Jr8wRiz1BizyBgzyxjTyos4w00FXpfrjTGLjTELjDHfGWO6ehFnODnca1LsvFHGGGuM0bIVQVaBfyeXG2O2F/w7WWCMubqqYgvbJMwY4wOeBYYDXYELA7xBXQXssta2B54CHqnaKMNPBV+XDcDlwFtVG114quBr8jOQYq3tCbwLPFq1UYafCr4ub1lre1hre+NekyerOMywUsHXBGNMHHArMLtqIww/FX1NgHestb0Lvl6qqvjCNgkD+gOrrbVrrbU5wNvAOaXOOQd4teD7d4FTjDGmCmMMR4d9Xay166y1i4B8LwIMQxV5Tb601mYU3P0JSK7iGMNRRV6XvcXu1gE0Eyu4KvJ3BeBvuA/1WVUZXJiq6GviiXBOwpKAjcXupxW0BTzHWpsH7AHiqyS68FWR10Wq1u99Ta4CPglqRAIVfF2MMTcZY9bgesJuqaLYwtVhXxNjTB+ghbX246oMLIxV9P1rVEE5xbvGmBZVE1p4J2EiUsmMMeOAFOAxr2MRx1r7rLW2HfAn4F6v4wlnxpgI3JDwH72ORUr4CGhdUE4xk6IRsKAL5yRsE1A8200uaAt4jjEmEqgPpFdJdOGrIq+LVK0KvSbGmKHAPcDZ1trsKootnP3efytvA+cGNSI53GsSB3QHvjLGrAMGAh+qOD+oDvvvxFqbXuw96yWgbxXFFtZJ2FyggzGmjTEmGhgLfFjqnA+Bywq+Hw18YbW6bbBV5HWRqnXY18QYcwzwPC4B2+ZBjOGoIq9Lh2J3zwRWVWF84eiQr4m1do+1trG1trW1tjWufvJsa22qN+GGhYr8O2lW7O7ZwLKqCi6yqn5QdWOtzTPGjAdmAD7gZWvtEmPMg0CqtfZDYCLwujFmNbAT9+JJEFXkdTHG9APeBxoCI4wxD1hru3kYdo1WykHB5AAAAUlJREFUwX8rjwF1gSkFc1c2WGvP9izoMFDB12V8QQ9lLrCLog+VEgQVfE2kClXwNbnFGHM2kIf7W395VcWnbYtEREREPBDOw5EiIiIinlESJiIiIuIBJWEiIiIiHlASJiIiIuIBJWEiIiIiHlASJiIiIuIBJWEiIiIiHlASJiJhyxjTr2DT3lhjTB1jzBJjTHev4xKR8KDFWkUkrBljHgJigVpAmrX2Hx6HJCJhQkmYiIS1gv3k5gJZwCBrrd/jkEQkTGg4UkTCXTxu38s4XI+YiEiVUE+YiIQ1Y8yHwNtAG6CZtXa8xyGJSJiI9DoAERGvGGMuBXKttW8ZY3zAD8aYk621X3gdm4jUfOoJExEREfGAasJEREREPKAkTERERMQDSsJEREREPKAkTERERMQDSsJEREREPKAkTERERMQDSsJEREREPPD/AWecLMB6YrngAAAAAElFTkSuQmCC",
            "text/plain": [
              "<Figure size 720x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          },
          "output_type": "display_data"
        }
      ],
      "source": [
        "fig = plt.figure(figsize=(10,4))\n",
        "ax = fig.add_subplot(111)\n",
        "ax.set_xlabel(\"x\"); ax.set_ylabel(\"y\")\n",
        "ax.scatter(xt, yt, facecolor=\"none\", edgecolor=\"b\", s=50, label=\"Data\")\n",
        "ax.plot(xp,yp*ystd+ymean,label=\"ANN\") ## ニューラルネットワークの出力は標準化した値に対して学習されていることに注意\n",
        "ax.legend()\n",
        "plt.show()\n",
        "plt.close()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cINfIqP4TTAV"
      },
      "source": [
        "ほとんど学習が進んでいません...(絶望)\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tDt2m0JKf1Ja"
      },
      "source": [
        "学習の様子を都度printしてみる(```verbose=1```に設定する)ことにして  \n",
        "最初からやりなおしてみると..."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 153,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "U_Je7tlWfX9v",
        "outputId": "72e92421-1808-4794-b074-74511b10a2e5",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "学習前のloss 61.29358546868393\n",
            "0 tloss = 155.20176990047585\n",
            "1 tloss = 636.6940976770327\n",
            "2 tloss = 1982.0790483266553\n",
            "3 tloss = 2056.051066857846\n",
            "4 tloss = 75.1510373285995\n",
            "5 tloss = 41.306942787588895\n",
            "6 tloss = 40.230205073524374\n",
            "7 tloss = 40.1956281449949\n",
            "8 tloss = 40.19379266944327\n",
            "9 tloss = 40.19298621925973\n",
            "10 tloss = 40.19221490236367\n",
            "11 tloss = 40.19144755457859\n",
            "12 tloss = 40.19068317848521\n",
            "13 tloss = 40.18992172293257\n",
            "14 tloss = 40.189163166751285\n",
            "15 tloss = 40.188407489902666\n",
            "16 tloss = 40.18765467254862\n",
            "17 tloss = 40.186904695016516\n",
            "18 tloss = 40.186157537795744\n",
            "19 tloss = 40.18541318153565\n"
          ]
        }
      ],
      "source": [
        "np.random.seed(1234)\n",
        "W = np.random.normal(0.0,1.0,nhl)\n",
        "V = np.random.normal(0.0,1.0,nhl)\n",
        "bs = np.random.normal(0.0,1.0,nhl)\n",
        "b0 = np.random.normal()\n",
        "\n",
        "nepoch=20 #20回だけ学習の様子を表示\n",
        "verbose=1 \n",
        "print(\"学習前のloss\", calc_tloss(xt,ny,W,V,bs,b0,acf))\n",
        "#学習\n",
        "W,V,bs,b0,dw,dv,dbs,db0=fitGD(xt,ny,W,V,bs,b0,acf,acfder,nepoch,eta,verbose)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TJaqgLw7Uaau"
      },
      "source": [
        "あるところからは、ほとんど学習が進んでいない事がわかります。\n",
        "\n",
        "原因として考えられるのは\n",
        "* loss functionをパラメータ(超)空間上にプロットした際にプラトーが存在する\n",
        "* 最適化手法や学習率の設定が適切でない\n",
        "* 初期値が悪い\n",
        "\n",
        "などがあります。\n",
        "\n",
        "勾配降下法は、最もシンプルな勾配を使った最適化手法ですが、学習の途中で勾配がほとんど0になってしまって(勾配消失ともいう)、学習が進まなくなってしまう、といったことがよく起こります。  \n",
        "\n",
        "「勾配が小さいなら勾配にかける学習率を大きくすればええんとちゃいまんの...？」  \n",
        "と思うかもしれませんが、学習率を単純に大きくしてしまうと、明後日の方向にパラメータを更新するせいで目的関数が発散してしまいます。(eta=0.1などとして試してみてください)\n",
        "\n",
        "注) 勾配降下法を拡張した、データを部分的に使うことで学習が停滞することを防ぐ、確率的勾配降下法(Stochastic Gradient Descent; SGD)は現在もよく使われています。\n",
        "\n",
        "以下では、Adamと呼ばれる別の最適化手法を試してみましょう。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "A3NSz6JulqFB"
      },
      "source": [
        "## Adam"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "n0QGVWTmltgl"
      },
      "source": [
        "Adamは、勾配降下法の様にその都度の勾配の情報だけを使うのではなく、\n",
        "以前の勾配の情報も有効活用する手法です。\n",
        "\n",
        "Adamは2014年に提唱された比較的新しい手法で、以降の機械学習の論文では、Adamが最もよく使われています。(*最も\"良い\"という意味では必ずしもありません)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 154,
      "metadata": {
        "id": "AdFBxZceoktL",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "def updateAdam(A,mt,vt,i,beta1,beta2,eps):\n",
        "    mhat = mt / (1.0-beta1**(i+1))\n",
        "    vhat = vt / (1.0-beta2**(i+1))\n",
        "    return mhat / (np.sqrt( vhat )+eps)\n",
        "\n",
        "def fitAdam(x,y,tW,tV,tbs,tb0,acf,acfder,nepoch,eta,verbose):\n",
        "    mts = [ np.zeros(nhl), np.zeros(nhl), np.zeros(nhl), np.zeros(1) ]\n",
        "    vts  = [ np.zeros(nhl), np.zeros(nhl), np.zeros(nhl), np.zeros(1) ]\n",
        "    ## Adamで使用するパラメータ\n",
        "    beta1 = 0.9; beta2 = 0.999; eps = 1.e-6\n",
        "    omb1 = 1.0-beta1; omb2 = 1.0-beta2\n",
        "    ## 最適化\n",
        "    for i in range(nepoch):\n",
        "        tmp = calc_der(x,y,tW,tV,tbs,tb0,acf,acfder)   ### 勾配を計算するところまでは同じ。\n",
        "        for n,mt in enumerate(mts):\n",
        "            mts[n] = beta1 * mt + omb1 * tmp[n]\n",
        "            vts[n]  = beta2 * vts[n] + omb2 * (tmp[n]**2)\n",
        "        ### 重み・バイアスの更新\n",
        "        tW += -eta * updateAdam(tW, mts[0],vts[0],i,beta1,beta2,eps)\n",
        "        tV  += -eta * updateAdam(tV, mts[1],vts[1],i,beta1,beta2,eps)\n",
        "        tbs += -eta * updateAdam(tbs,mts[2],vts[2],i,beta1,beta2,eps)\n",
        "        tb0 += -eta *  (mts[3]/(1.0-beta1**(i+1))) / ( np.sqrt( vts[3]/ (1.0-beta2**(i+1))) + eps)\n",
        "        if verbose and i % 500 == 0:\n",
        "            print(i, \"tloss =\", calc_tloss(x,y,tW,tV,tbs,tb0,acf))        \n",
        "    return tW,tV,tbs,tb0"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZzDaa2zCoeIm"
      },
      "source": [
        "それでは重みを初期化して、再び学習をしてみましょう"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 161,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "4mkH2DuglqLS",
        "outputId": "de51a9b4-c624-4aba-b11d-24dfd9ee7183",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "学習前のloss 61.29358546868393\n",
            "学習後のloss [0.89838996]\n"
          ]
        }
      ],
      "source": [
        "np.random.seed(1234) ## Gradient descentと同条件でスタートするためseedを固定\n",
        "W = np.random.normal(0.0,1.0,nhl)\n",
        "V = np.random.normal(0.0,1.0,nhl)\n",
        "bs = np.random.normal(0.0,1.0,nhl)\n",
        "b0 = np.random.normal()\n",
        "\n",
        "nepoch=2000\n",
        "verbose=False\n",
        "eta = 0.05\n",
        "\n",
        "acf = sigmoid ; acfder =sigmoid_der\n",
        "print(\"学習前のloss\", calc_tloss(xt,ny,W,V,bs,b0,acf))\n",
        "W,V,bs,b0=fitAdam(xt,ny,W,V,bs,b0,acf,acfder,nepoch,eta,verbose)\n",
        "print(\"学習後のloss\", calc_tloss(xt,ny,W,V,bs,b0,acf))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0PJr6ceIFVS-"
      },
      "source": [
        "さっきよりlossの値が小さくなっています。学習がうまく行ってそうですね。\n",
        "\n",
        "グラフにしてみると..."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 156,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 279
        },
        "id": "QlVmPtFxVr3u",
        "outputId": "b04eee9d-d29e-4add-9f68-7aeac862198d",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmEAAAEGCAYAAAAt7EI0AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzddXhW5RvA8e+zZE2M3OjuZoB0ju4WpAUFFTFQFBH9iSgqSgnI6G4kRUBJ6QbpAaPXvb3be35/PNRglGx7F/fnuna5nfOc895nw+1+n7gfZRgGQgghhBAiZVlZOgAhhBBCiIxIkjAhhBBCCAuQJEwIIYQQwgIkCRNCCCGEsABJwoQQQgghLMDG0gG8LHd3d6NAgQKWDkMIIYQQ4rkOHTrkbxhG9sTOpbkkrECBAhw8eNDSYQghhBBCPJdS6srTzslwpBBCCCGEBUgSJoQQQghhAZKECSGEEEJYQJqbEyaEEEIIMJlM+Pn5ER0dbelQBJApUyY8PT2xtbV94WskCRNCpCn798OUKXD2LOTNCwMGQOPGlo5KiJTn5+eHi4sLBQoUQCll6XAyNMMwCAgIwM/Pj4IFC77wdTIcKYRIM2bNgrZtoWxZGD8eGjaEQYPgyy8tHZkQKS86Opps2bJJApYKKKXIli3bS/dKSk+YECJNCAiAYcN0T1ixYvrYa69B+/Y6KevYEUqXtmyMQqQ0ScBSj//ys5CeMCFEmrBqFTRt+jABuy97dujdGxYutEhYQgjxn0kSJoRIE4KDIU+exM/lyaPPCyFS3urVq1FK8e+//wLg6+uLUoqJEyc+aDNkyBBmz54NQO/evfHw8CAmJgYAf39/MupOOJKECSHSBC8v2LABzOYnz61fD9Wrp3xMQghYtGgRtWrVYtGiRQ+O5ciRg59//pnY2NhEr7G2tsbHxyelQky1JAkTQqQJtWpBjhx6XlhUlD5mMsH338PFi9Cpk2XjEyIjCg8PZ9euXcycOZPFixc/OJ49e3YaNmzInDlzEr3uvffe46effiIuLi6lQk2VZGK+ECJNUArWrIF+/SBfPj0Z/+xZKFIE/vwTMmWydIRCWNDGEXDrRNLeM1dZaPbtM5usWbMGb29vihUrRrZs2Th06BDZsmUD4OOPP6ZZs2b07dv3ievy5ctHrVq1mDdvHq1atUrauNMQ6QkTQqQZWbPqCfoHD8LIkbBtG/z9N2TQ6SRCWNyiRYvo2rUrAF27dk0wJFmoUCG8vLxY+JRVM5988gnff/895sTmGGQQ0hMmhEhz8ufXH0KIe57TY5UcAgMD2bZtGydOnEApRXx8PEop3n777QdtPv30Uzp27EjdunWfuL5o0aJUqFCBpUuXpmTYqYr0hAkhhBDipS1fvpyePXty5coVfH19uXbtGgULFuTatWsP2pQoUYJSpUrx+++/J3qPkSNHMn78+JQKOdWRJEwIIdI5w4DZs6FCBbC11cO3X30F9yoECPGfLFq0iHbt2iU41qFDB8aOHZvg2MiRI/Hz80v0HqVLl6ZSpUrJFmNqpwzDSJ4bK+UDtATuGIZRJpHzCvgZaA5EAr0Nwzj8vPtWqVLFOHjwYFKHK4SwgJMnYepUPcE+Xz4YOFBKTSSHUaP0ooYff9SrTM+ehc8+06tL168HK3k7niadOXOGkiVLWjoM8YjEfiZKqUOGYVRJrH1y/q83G/B+xvlmQNF7HwOBqckYixAilVm0SO/9mDMnfPQRlCmjy0z88IOlI9OOHNGbg9eqBd266UUAyeHvv6FDByhZEho1giVLdM/VswQGwoQJOmn98kvw9X1625s3YdIkvYK0YUOwt4dy5WDlSrh1CzZvTtLHEUK8hGSbmG8Yxg6lVIFnNGkDzDV0V9w/SqnMSqnchmHcTK6YhBDJ6/ZtuHQJ8uYFT8+ntwsKgrfegp07dfIF0KQJdOmih8xat4aiRV8tlui4aAKiAwiKDiIyJozISH8io/yJjA4iIjacqLgoDHMchjkew4jDMOIxUKCsuXHVngN7MlGxrAPdmjrgf9eNT37KSu1/cjFsqAdZHbLhYOPwagEC06bB11/rnqoxY+DMGf31jh0weXLi1+zZozcxb9pU7535779QubLu5XrjjSfbb9gAzZrp7Z0eZWOjt3tavVqfF0KkPEuujvQArj3ytd+9Y08kYUqpgejeMvLly5ciwQkhXlxICAweDJs26eTp4kWoUQNmzIBcuZ5sv2IFNG78MAG7z8MDevWC+fN1D8/TmOJN3Iy4ybXgS/jdPYFf0EVuhF/HPzqQAFMYAeYYwnn5Ze/qXheUYaOgDmwCiAAcgY6wGdi8Urd1MCAL1mS1siennSseDtnJ4+yBZ+ZC5MlRFo8cFXC0d37qawUGwogRutxG4cL6WOnSOrkqW1Z/H7y8El4TG6t7C2fPhubNHx4fPFgnZLVrQ6FCCa+Jj9fzwBJjY6PPCyEsI02UqDAMYzowHfScMAuHI4R4hGFAu3Z6Y+0rV8DFRVe0/+YbnWgdPvxkEhAYqOeAJSZ/fjh3Tn8eGx/L5ZDLnLtzlPM39nMu6By+UXe4GR+ZIMWyN5vJHRdP9vh4Siobstk4ks3ODXc7V7LYZ8bJPjOODllwdHDH0SErjnYuZLJ1xtrGHmVth7KxBysbMOJZszqW3bti+OyLCKLjIog2RRAVHUxUdBBbd/kTYx1EzoIhBMaGEWQKI8AUyeXYUHZH3SA6+IR+O3lPDrOiiI0zhR1zUTRzUQrnrkzhAg1wcnRn7Vo9/Hg/AbvPxQX694fFi59MwjZs0EnuowkYQPHiOmmbNUtPuH9U06bw6acQGgqurgl/bgsWwAcfJP5zEEIkP0smYdeBvI987XnvmBAiDdmzB/z89Jyj+xO8HRz08Nrff+vhrse3FKpcWW8/ZBi6Ej5AnDmOC8EXWHdxLx4Vd9Ju4Xl8TcHc39TE1jAoFGuifJyZlnau5HXMjWeWwuR1L417tuJYuXmCax6wfbVhwkuREJ0FXIuA62Pn9pyAc8fg048eO2EYGNEhBPj/y3X/U9wIOMv1EF8uR9zgvCmUZaFniQ4/D34b4MBX5DUr8kdnJX/FYhw7WoMSRZpj75zzwe3c3XVC+zg/PyhVKvG4S5eG3bufPJ4/v57T1qKFHuIsV07fZ9QofT4DFysXwuIsmYStBYYopRYDXkCIzAcTIu3ZtUvP4Xp8hZ1S0KaNPv94Ela/Ptg6RTJw7EFK1t7Oydv7ORnhRxRmqAQx8fGUCYmlnnKgmGsBiuasQH7P17DNURLc8ibrcj4vL71i02x+8mU2btS9fk9QCuWQGfe81bkTVp3IECjrCf2q6e9DvCma6zcOcMFvNxfuHudMqC/H7f254xHA5mN7sTn6A8XN1lRwzEPV3NXZuaU9TVuXAVSClylRAn77LWHyet+ePfp8Yn7+WS94aN4cwsL0c/XsqSflP22oUgiR/JItCVNKLQLqAe5KKT/gC8AWwDCMX4EN6PIUF9AlKvokVyxCiOTj4gKnTyd+7u5dfR7AZDZx4u4J/rn8B/uu/UX8oBv8g8HBiwbFY2NpHqFwDihE00o1KFO+NipPBXDIknIPck+NGpA7N7z/Pnz7rd6TMj5erzA8eVKvXkzMnTvQo4f+XlSrpifZ29vrVaClSmUiX/7a5MtfmwaPXNOlxxnc826iQOV9nIm8xIqoayzw9UO1WobJZHB3QW6qerxG5dLdcMpeggYNIC4OJk6EoUMfJmLbtsHatXDqVOKxWVnBhx/C8OF6WNLJSZIvkTRu377NsGHD+Oeff8iSJQt2dnZ89NFHT9QPS06+vr60bNmSkydPPjh24sQJevbsCcDVq1dxc3PDzc0Nd3d3/vzzzxe65549e+jevTsAs2fP5uDBg0yaNClJY0/O1ZHdnnPeAN5+VhshROrXoYPex/Hq1YTzvAICYO7Ku3wybStDfl/D/qAzRBnxKMOgdGwsveNt8MpSklwODbkUWY8slUtQsZLVEz08KU0pvT9lnz76eSpU0CsQPT31kKuj45PXGIZesVinjp63ZWurj82apedknT79MBl9lM/0krz3Xkm+Gagn1F+7HkWTNuup3HQTxyNOs8B0i9lXVmLju4Kq8VbUdSvOvAltGfB+a+bMcaFWLZ3sHTsGy5ZBjhzPfjYrK8icOWm+T0IYhkHbtm154403HuwPeeXKFdauXftE27i4OGxsUm7wrWzZshw9ehSA3r1707JlSzp27PjCMfn6+rJw4cIHSVhySbZirclFirUKkfr88ose7vp0pIFHuQtsPruJI/7rCXHX0zw9THHUio6hhktBqhT0xq1ES8he4skxtVTm8mW9SMDD48mVnI/auVPXFDtz5slH6tBBL1AYNOjp1wcG6tIeuXPr17ovyhTJsfPr2H1hLX8HneEysQAUjjVR1ZyD7BFNyJGzH94dcpMp0ys8qEiTLF2sdevWrYwZM4a///470fOzZ89m5cqVhIeHEx8fz6pVq+jbty+XLl3C0dGR6dOnU65cOUaPHo2zszMf3FslUqZMGdatWwdAs2bNqFWrFnv27MHDw4M1a9bg4ODAoUOH6Nu3LwBNmjRh48aNCXrCHvVoElavXj0qVKjArl276NatGydOnEiQoDk7OxMeHk716tU5c+YMBQsW5I033iBLliysXbuWyMhILl68SLt27fjuu++eeK2XLdaaJlZHCiFSt+Y9L3DNYyVz7/5O6NlgAMo6x1A3wob6eV6jaPG2qEJ14RklG1KjggX1x/McPqwTrcRyyiZNdBmKZ8maVX88zsHWkeqlOlO9VGeGA1eDLvD3yfn87fc3y2P8icu0hBzRCzi30BXv/I0oW74Pyr3ICz2bSF/G7R/Hv4H/Juk9S2QtwcfVPn7q+VOnTj13y6HDhw9z/PhxsmbNytChQ6lYsSKrV69m27Zt9OrV60Fv1dOcP3+eRYsWMWPGDDp37syKFSt4/fXX6dOnD5MmTaJOnTp8+OGHL/VcsbGx3O/M6d27d6Jtvv32W8aPH/8gGZw9ezZHjx7lyJEj2NvbU7x4cYYOHUrevHkTvf5FSRImhHjCuXMwfbr+b/78upenXLmEbfzC/Nh08Xc2nF3O+eg7WBkGXkTTONyeegWakr1sF/CsmiH2xMmWDbZuTfzclSt6tWNSyJelCD1rj6YnEBYTyo7Ti9l0fhWLI/2Y57cWj8sraaKcaVaoJSUqDUC55UmaFxbiBbz99tvs2rULOzs7Dhw4AEDjxo3Jeu8dxq5du1ixYgUADRo0ICAggNDQ0Gfes2DBglSoUAGAypUr4+vrS3BwMMHBwdSpUweAnj17snHjxheOs0uXLi/9bAANGzbEzc0NgFKlSnHlyhVJwoQQSWv5cl38c8AA6NsXjh/XvTljxkCPPhFsuryRVafmcSz0EgAVomP4JN6WJgWb4162K3hUSvXDjEmtTRt4913dI/Zox8DNm+DjkzxbHrnYu9Ki4kBaVBxIaGwo2/5dzqazy5kXcY1ZV5dT7PxC2tnnokXJ7mQp1xUyuSV9ECLVeFaPVXIpXbr0g6QKYPLkyfj7+1OlysORNycnp+fex8bGBrP5YeW/6OjoB5/b29s/+Nza2pqoqKhXDTtBTI++ttlsJjY29qnXPR5LXFzcU9u+qPT/FlUI8cKCg/V+hH/+qYuttm0Ln39uMHPTUSaeHE79hbUYvfdLwvzP8m5IBJucqzCv8Qy6DzqBe7PvwbNyqk3A7t6Ffft0jayk5uKiS0d4e+uNsTdu1HPkqlXT9dCeVtsrqbjaudK2XF9+7bSB7V138lnpAdi55GIcgTQ4PZH3Z1Vhx5KOxF366/kbUwrxgho0aEB0dDRTpz7c+jkyMvKp7WvXrs2CBQsA+Ouvv3B3d8fV1ZUCBQpw+PBhQA9fXr58+ZmvmzlzZjJnzsyuXbsAHtzzvyhQoACHDh0CYO3atZhMJgBcXFwICwv7z/d9UdITJoR4YNkyPbepfHkIiQlh9flVrDw9n0tRt3GoaMY7LJL2LkUpX7UPqmSrNDHHKywM3n4bfv8dihTRk+29vPRw66OT4F9Vu3Z6E+5ff9V1uTw8YOlSXfIiJWXOlJkuVd6hS5V3OBd4ltVHZ7DObxtbos+SY/tgOm6xo2PxzmSvMgCcn7OcUohnUEqxevVqhg0bxnfffUf27NlxcnJi3LhxibYfPXo0ffv2pVy5cjg6OjJnzhwAOnTowNy5cyldujReXl4UK1bsua89a9Ys+vbti1KKJk2a/OdnGDBgAG3atKF8+fJ4e3s/6CUrV64c1tbWlC9fnt69e5MlS/KUy5HVkUKIB/73P7gRc5GcjX1Yd3kDUUYc5aJjaB8L7uEd2XF3AJ//+AIz1VMRb2/Ikwd++gnc3PSWSuPG6XpfR4/qWl7pnSnexA7fLSw7No3dYZewMQwaR0bRPUt5ynu9iypYJ9X2YIqns/TqSPEkWR0phHhp8eZ4dl7fyZH8v3E2/hh2Fw1ahEfQ3aEgJWoMglJtaNHG/onK96ndgQN6ccH69WBtrY85OMDo0bqsxIoVkMxlgFIFW2tbGhZuTsPCzbkSeoXFR35lzZVNbIw9T8k/B9INV5pXGox9+W6vvO2TEOLFSRImRAYWHRfN6vOrmHN8On7R/uSMi+PtkCjKRDWlZt+hWOUug2HAzJm6Wvzy5ZaO+OXs3AktWz5MwB51f0uljJCEPSq/a34+rjuWoabPWXd+FYuO/8aoGH9+Pjae1//5ls7FOuHq9Ra4/fexWpNJz8HLnDnx4rZCCE2SMCEyoNDYUJaeWcy8kz4ExkVQLjqG96LMNCjXmzueA+nQKwd3faBqVb060tpa7zPokMY6SVxcwN8/8XP+/olXsc8oHG0d6VyqB51KdmffzX+YfXACPwed5rfrq+k0ZyGv56lDzjojIMeLD3fFx+sFHZMn66+joqBjR/j++8TroAmR0cmcMCEyEP8of+admsPSM4sIN8fwWmQU/c3OVK76NqpCD7DT3RaGAYcO6aG8AgX05PK0OGXI319Pxj9yJGHR1eBgXQH/99+hYkXLxZfa/Bv4Lz6HJ7L5+g6sDIOW4RH0y1qRArU/gbxVn3v9W2/pLZ6mToXixXVv2JgxsHs37N2bMebfpSSZE5b6vOycMEnChEjnfH1hos9dDlnNIDj/UgwVT9OICPpa56BE7RFQshVYJTJel05MmaI34h45UieTp07pBQhNmsCPP1o6utTJL8yPucens+riWmLNcbQIj+RN56Lkr/URFG6QaEbu6wtVqujVp4/2MBoGNGyo6851e+aOwuJlSRKW+kgSJoR4YMGqAL74fQYutRdjqHjahIdT82Ie7Ip9Sv03m2WIavYAf/0FkybpXpq8eXVC0K5d2uzdS0n+kf58s3k6f4Usw4yJFuERDLLPR976n0Phhgm+gTNn6u/zvHlP3mfGDD3/7l5FApFEJAlLfWR1pBCCoOggph2axaKA+TjVMdEiPJw37fORt8l4zpqbUKOm4kTrpK2TlZrVq6c/xIu7fh3atXMnNPRTvBoM5IzTDNaVWMp66wBabRrAmw6F8GzwBRTUW8fY2sIjhc4TiIrS54VlhYfrYWFra3jttaQZHra2tqZs2bKYTCZsbGzo1asXw4YNw+oZb/B8fX3Zs2cP3TPaqphEZIy3wUKkQpcv6+rqXbvCiBFw/vyr3zPSFMnUo1PxXtaYhedn0SwmmDXRznzdZBp5+/8FxZpSvISic2eYP//VX0+kT4ahd0to3RrOnIE5U9zZ//0njMy5hYhd3dng6kZr69uM+70nQbObwZW9NG+ud1q4fj3hvUwmvXVThw6WeRahf57jx+t9YL/+Wv/eyZcPZs169Xs7ODhw9OhRTp06xZYtW9i4cSNffvnlM6/x9fVl4cKFr/7i6YAkYUJYwMqVeuVhZKQulWAYULMm/NffS3HmOJadW0aL5Y2ZcmwKr4UFMfu6FRVufkeBgbuhWJMEQ0dFi8Lt20n0MCLd2b1b95qMHJlwyLZzS3dqmz+l1Z3NtC7SloVubjQ3rjFjZWecNrbnh49P0bAhrFmjFz/s26cTuXz5oGlTyz1PRufjoxOugwfh779hzx694fzo0bqGXlLJkSMH06dPZ9KkSRiGga+vL7Vr16ZSpUpUqlSJPXv2ADBixAh27txJhQoV+Omnn57aLkMwDCNNfVSuXNkQIi0LDDSMLFkM4/DhhMdPndLHb9588XuZzWZj+9XtRuvl3kaZ2WWMnr8WMY78VNww9v9mrFkVa9Ssmfh1LVoYho/Pf38Gkb5NmWIYAwcmfm72bMPo2VN/fiHogjFky2CjzOwyRoOZpYzl4/MY538abLSuf91wdTWM4sUN47vvDCMmJuViz0hOnz793DZms2EUKWIYe/Y8eW7ZMsOoXfvVYnBycnrimJubm3Hr1i0jIiLCiIqKMgzDMM6dO2fc//u9fft2o0WLFg/aP61dWpTYzwQ4aDwlp5E5YUKksCVLdK/A46URSpWC9u1hwQIYPvz59zkVcIrx+77l4N2jFDDFMSE0igaV30bVeBvsHGleET78GH75BYYO1T0ahqGHIY8e1fsaCpGY3Lmf/u/j7Fl9HqBw5sJMbDSFw7cP8+OB7xhtfYoisdv4pNnvrBk5EF57FzK5plzg4glBQXDnDlSv/uS5Zs3g9deT77VNJhNDhgzh6NGjWFtbc+7cuVdqlx5JEiZECrt1Sw8HJqZYMX3+WQKjA/nl0M+svLCSLPFmPgsKpn2Rtth2GQUuOR+0s7GBjRt1Yjdt2sPCqxERsGmTVDIXT9e8ua75tWWL3tD9vitX4LffYMeOhO0r5azEvBaL2Hp1K+P3f8sAu9s0PDub4UfnkLfOJ1C5d7oug5KaOTrqIrpBQU8WzPXzg6Tel/rSpUtYW1uTI0cOvvzyS3LmzMmxY8cwm81kypQp0Wt++umnF2qXHsmcMCFSWNmyel5GYv76S59PjMlsYsGZBbRc7s2a8yvpGRLKOqv8dOm+Eds2kxMkYPcVKqQLlf76K9SpoyfnnjmjC5UK8TR2drB4sd7SacAAXXZi5EioVg2++AJKlHjyGqUUjfI3Yk379bxT8R32uGahTTYHft7zJZHT6sCVDDTPJxXJlEkvsvjhh4THDUPXz+vVK+le6+7duwwaNIghQ4aglCIkJITcuXNjZWXFvHnziI+PB8DFxYWwsLAH1z2tXUYgdcKESGEmE5QsqYccBw16OEw4dy58/rke7nl8e6B9N/fx7d6vuBB2hZqRUXwc50Shxt9A8eZS7Eokm1u39KTu06f1EGSfPnrY/EXcjrjNz4cn8PuldWQ3G7zvH0CLAs1QTb56pX0pxUMvWifsxg39JszLSw8/xsbqHs0bN/QE/cyZ/3sMj5eo6NmzJ++//z5WVlacP3+eDh06oJTC29ubyZMnEx4ejslkomnTpgQEBNC7d29atmyZaLu0SIq1CpEGnD//sFholSp6jlZkpF41Wbr0w3Z3I+8ybv9YNl/ZgkdcPB8FhVG/yhDUa++Cbcbpshdp19E7R/l23zecCjyDV3QMI4MjKVjjPagxRP4Nv6KXKdYaHKwTr40bdZ2wdu3gjTdkWkJSkyRMiDTCMPTcmgsX9P6M9es/LGBvNswsO7uMCQfHExsXzYDgYPq4e2Hf/HvIWvCZ9xUitYk3x7Pi/AomHPyR6LhI+gUF019lwb7lT3obJPGfSMX81OdlkzCZEybEKzKb4eef9YbFNjZ6w+gfftCTYZ9FKahbF/r103vr3U/AzgWdo+e6bny972tKhwezMsRgkPev2PdYJgmYSJOsrazpXLwza9uvo0mh5vyaxY12Lmb2LO8GKwZA+F1LhyiERUgSJsQrGjQIli+H2bP19iwLF8K6dXr+zMuIiovip4M/0WVtJ675n+Qb/2BmFO5O/sH7oEQLmfsl0jx3B3e+rf0tM5rMwCpzPt7MlYOPbm8ncEoVODRbv6MRLyWtjWalZ//lZyHDkUK8gpMnoUkTPcfLyenh8agovYJs5UqoXPn599lzYw9jdo/ieuRt2oaF836mQmRpPRlyJLIMTYh0ICY+Bp8TPkw/Ph0Xs5lP796hadayqFY/Qw4ZYnsRly9fxsXFhWzZsqHkTZpFGYZBQEAAYWFhFCyYcMRCNvAWIpmsXav3fnw0AQO9urFHD719y7OSsPDYcMYf+J4VF1ZSwBSPT3A4VWt9CtUGSF0lka7ZW9szuMJgGuVvxKjdo/hQmdkUfY2RM+qSvdYHUGsYWMuu38/i6emJn58fd+/KcG5qkClTJjw9PV/qGknChHgF8fF6HlhibGyePS9sz409fLFzJHei/OkTEsrbWSpg3+8XyJI/eYIVIhUqmqUo85rPY97peUw6MpG2nnn4+MAEWp1Zi2o7FXI9pXCewNbW9oleF5G2JOucMKWUt1LqrFLqglJqRCLn8ymltiuljiiljiulmidnPEIktebN9fYuMTEJj5tMuthlixZPXhNhiuDL3aN5c8ubZAq7xdyACN6vOxb7nqslARMZko2VDX3K9GF56xUUzlGOkdmz8ZZVILdmNoDtYyEu1tIhCpEski0JU0pZA5OBZkApoJtS6vEyf58BSw3DqAh0BaYkVzxCJIfKlXUV8fbtdZFV0PPDOnXSBVlr1EjYfu+NvbRb1YoV51fQOziUZc6VKD9wL1ToLhPvRYZX0K0gs71nM6LaCA45OtHO04PfD/2CMaMe3Dxm6fCESHLJ2RNWDbhgGMYlwzBigcVAm8faGMD93V3dgBvJGI8QyWL+fF1wtU4dcHGBmjV1VfGlSx/mVVFxUXy99ysGbhmIfehN5gaEM7zBD2TqMj/R7YaEyKislBU9SvZgRZuVFMtRnk+zu/OBVTAhvzWAbV9Lr5hIV5JtdaRSqiPgbRhG/3tf9wS8DMMY8kib3MAfQBbACWhkGMahZ91XVkeK5BYVpVc1Xryo917s0OHJbYQSYzZDWBg4O+uK1PedDjjNx38N50q4Hz1DQhmauSKZ2k4B1zzJ9xBCpAPx5nhmnZrF5COTyYoVX93wo2aWEtB+BmQvZunwhHghqblYazdgtmEYnkBzYJ5S6omYlFIDlVIHlVIHZRWISE7790PhwrBggZ7XtWgRFCwIe/c+/1orK3Bze4m6JPcAACAASURBVJiAmQ0zPidm0mN9NyJDrjLjbggf1hhFpp6rJAET4gVYW1nTv2x/FrZYiLOrJ2/mzsHY+FtET68DB37T2068hLt34bvv9P6JH3ygS8wIYUnJmYRdB/I+8rXnvWOP6gcsBTAMYy+QCXB//EaGYUw3DKOKYRhVsmfPnkzhiowuMhLatIFp02DDBvjqK1i/Xm9g3LYtvMx+srcibjFgU29+OjyB+uHhrDTnwqvPdqjaT+Z+CfGSSmYryZKWS+hRsgcLnezo4pmH01tGwMIuxAXfeaFcbO9evS/rv//q2n4ODtCoEfz0U/LHL8TTJGcSdgAoqpQqqJSyQ0+8X/tYm6tAQwClVEl0EiZdXcIili3Tc7tatUp4vHlzPc9ryZIXu89m3810WN2GE7cPM8Y/iB/KDMat7x+QrfArxWcYMG8eeHlBtmx6UcCMGVJkXGQMmWwyMaLaCKY1nka4Q2a65/Fg5q39BPyvBq9X3chHHz39jVJcnK7n5+OjP3r10m+yDh7UPWPHj6fsswhxX7IlYYZhxAFDgM3AGfQqyFNKqTFKqdb3mg0HBiiljgGLgN5GWivhL9KN8+ehatXEz1Wtqs8/S6Qpks93fcYHf39A/ohglodZ067rGlSdD5Kk8OrHH+s9Kb/4Qr+bHzcOZs6EwYNf+dZCpBk189SkyqlVmC/WY4K7C2NKujKpdXeamt6jbYtITKYnr/nzT8iTB1q2THjc01NvO+bjkzKxC/G4ZC3WahjGBmDDY8dGPfL5aeC15IxBiBdVoABs3Jj4uWPHoH79p197IegCw7e9w+WwawwMDmFQ3qbYtvwR7F2SJLYLF2DWLDh3DrJk0ccaNYLq1XUpjCNHoGLFJHkpIVK127dh+i9u/PvvBLYGLmL8wfF0LFiEb+0WMb3qP2xbMpumryfc7uvGDShePPH7FS8Oq1alQOBCJMLSE/OFSDW6dIEdO2D79oTHd+yArVuhW7cnrzEMg1XnV9FtXWdCQq4wPSCUofW/x7bDjCRLwABWr4bOnR8mYPc5O+tJxitXJtlLCZGq/fGHfgOSM6eie8nuLGi+AAfnnPTPnZN1eYOpfa4+HFmQYNJ+2bKwa1fiQ/c7dui5YkJYgiRhQtzj4qJre3XurBOu8eOhe3ddiHXxYr3y8VGRpkg+2zmCUXtGUT4ijOXx2aneZzuU75LkscXGgqNj4uecnPR5ITKKR9e23J+037xQC2ZktqVvLg/urBsKqwZBjJ4kVqUK5MwJX36ZMBHbulXPBe3fP4UfQIh7kq1OWHKROmEiuQUH6xIV9+uE9ejxZA/U+aDzfLDtHS6H+TE4OJiBpfpg3fDzZNtweP9+PbH47FmwfeQlzGaoUAF+/FH3DgiR3t26pYfgz52DRxfLx8cb1Oi/hrh6/8PFyuDbG37UcPSETrMhVxlu3tRvqAICoF49Pcfz3391GZp69Sz0MCJDeFadMEnChHhJq86v4pt/vsLJFMO4kGi8WkyFYk2S9TUNQ5fPsLfXS+o9PfUfoxEjwNcXtm3TdcqESIsOH4aJE+HECcidG/r10//en1bNZeRI2LQJfvlFr1z29dULVi5eBJ81lxixeziXgi8yJMJEv4AArJp9C5X7YKD45x+9GjJXLmjWDOzsUvRRRQaUmou1CpFmxMTHMHrX5wmGH736/JXsCRjoP0ZLlujkq1w5yJtX9wY4OsK6dZKAibRryRJdBqZ0aZgyRc/N/PxzeOedp9di/fprGDAA+vTRSVTlyrpXbPNmKO5eiAXNF9C0oDe/ONnwbr5ChG54H1a9iTJFUqMGvPmmTvIkAROWJj1hQryAm+E3Gbb1bU4Fn2dAcAhvF+2KdZOvwSblf4tHRcGdO/qPztPmiQmRFkREQL58uie3fPmHx0ND9TD73LlQq9az7xEbq4foH+81MwyDRf8u4vsD35PLOhMTrlygeJZi0HnuK9fsE+JlSE+YEK/gn5v/0GVNO3wDzzEhIJx3Gv2MdfPvLJKAga70nT+/JGAi7duwAapVS5iAAbi66t6qhQuffw87u8SHLZXSqydnec8i1taRHnnzsib2NkyvD2efUotGiBQmSZgQT2EYBrNOzOTNPwaSNTKYRdHONOy1BUq3s3RoQqQLoaF61WJicuWCkJBXf40KOSqwtNVSyueoyGeZHRiTIwexi7rCtq/BHP/qLyDEK5AkTIhERJgiGL7tHX48PIGGEREscK9LwX7bwL2IpUMTIt2oXl3X/UqsxMq6dXrSfVLI5pCNaY2n0bdMX5bZxNC3SBn8d/8ACzpCZGDSvIgQ/4EkYUI85kroFXqs7cjWa38xPCiUH6qOxKndNLCT8T8hklLp0no4sn9/3SsGep/HiRNh3z7o2TPpXsvGyoZhlYfxQ90fOKdMdClUgpM39sG0OnDjaNK9kBAvQZIwIR6x98Zeuq/tREDIVaaHxNK7/VJU1b6WDkuIdGv+fD2nq0ABqFNHz3dcskQXUnV1TfrXa1KgCfOazcPG3pU38uTid1sDfLzh5IqkfzEhnkNWRwqBnv+1+N+FjNs/joKxMUxUefDsvBBcc1s6NCEyhFu3dDHiXLmevs9jUgqMDuSDvz/gwK0D9DY7896V01jXHg71P5OaLyJJSbFWIZ7BZDYxdvdoll1aS72ISMZ6NMG5xQSwsbd0aEKIZGQym/hu/3csPruY12yyMu7iMdyKeEP76ZApGbrhRIYkJSqEeIqg6CAGrnudZZfW0i8knAlVP8G5zVRJwITIAGytbBlZfSRf1PiCfeZQehQtyyXfbTCzMQRctHR4IgOQJExkWBeCLtBtdVuOB57im1AT77VdhHVV2clXiIymY7GO+DT1IczKitfzF+Cf2ACY0QAubrd0aCKdkyRMZEh/X/uL13/vTEzEXWaZMtPqjW2Qr7qlwxJCWEjFHBVZ2GIhOZ09GJzNmRVZs8P8DrBvuqVDE+mYJGEiQzEMg7knfBi6bSj5o8NZ5FyRcr23gGseS4cmhLAwD2cP5jabS7XcXoy2j+HHQuUxb/wQNnwkhV1FspAkTGQYceY4vtk9iu8P/0SjiEhmF+5Jrs7zwdbB0qEJIVIJFzsXJjecTJfiXZhl9md4CS+iDkyHRd0gJszS4Yl0RpIwkSFEmiJ5b1M/Fl9cTZ/QCMbX+haHhp8nvumcECJDs7GyYaTXSD6q+hFbY27Rp0QV7l7eBj7NIMTP0uGJdESSMJHu3Ym8Q+817dl55xCfhZl4v90yrMp1snRYQohUTClFz1I9+aXBL1wyhdIpf0nOhF2DGQ3hxhFLhyfSCUnCRLp2LvAsPVa1wTfsGhNjnejS80/wrGzpsIQQaUBMDKwaXw/fb+cQEGJH58w5WBtlh9mnOZxZZ+nwRDogSZhIt/b67eKNdV2Jjw5mjn1x6vTeBm6elg5LCJFGDBwIV67Aia0l+bPPQornKshnea34xVQQY8nrsGcipLGC5yJ1kSRMpEurzizmra2DyR0TycI8zSnZdTnYOVk6LCFEGnHpEmzYAAsXQs6ckNMpJ3OazaKGR3VmFgrjm2xVMf74DNa/D/Fxlg5XpFGShIl0xTAMJu/7jlH7/0e1qGjmln2HXM1+kL3ghBAvZdcuaNwYHB0fHnOydWJSw0mUUa1Z7HqLL0vXIe6gDyztCbGRlgtWpFnyl0mkG3HmOMZsH86v/86jbUQ0k+pOwLn625YOSwiRBjk7Q2Dgk8dtrWypEfI1ua8MYEWkL++VrUPkuU0wtzVEBAB6hHL3bpgyBVasgOjoFA5epBmShIl0IToumuEberP82hYGRMQxpvVibEu2sHRYQog0qmlTOHAATp5MeDwqCqb9qvig+jt85vUZOyOuMqB0DQLvnISZjblz1pcaNaBvXzh+HKZOhfz5YcsWyzyHSN1sLB2AEK8qJCaEd37vxpHwq4yIsaNHt3WQJb+lwxJCpGFOTvDzz9CkCYwcCfXrw4ULMHYsVKyohyqV6oK7ozsf7/iYnkVK8+uVizjObswbTZYx6MsKD8oQ7tgBHTrA4cOQN69ln0ukLspIYys7qlSpYhw8eNDSYQgLiI/X7yYvX4ZChaBRI/CPvsWgNZ3wjQ1irOGOd9dV4JDF0qEKIdKJPXtgwgTdq5UzJ/TpA716JZxmevTOUYZsGwImGHMuhPoOgajOc6BIowdt3n0X3NxgzBgLPISwKKXUIcMwqiR6TpIwkRacPAlt20LWrFCpEhw6BKE25/Ec1I0IcxQ/O5bCq8N8sLG3dKhCiAzoUsgleq4eTERcENPirfC6dR5aT4QK3QFYvhwWLdJzxETG8qwkLFnnhCmlvJVSZ5VSF5RSI57SprNS6rRS6pRSamFyxiPSpuhoaNYMvvgC9u+HX3+FGWv2k21gJ+LjIvnNvRFenZZIAiaEsJhCboV4P9c8zCEeDHaIZWv+irB6MOwYD4bBv/9CnjyWjlKkNsmWhCmlrIHJQDOgFNBNKVXqsTZFgU+A1wzDKA28l1zxiLRr+XIoXRp69tRf77i0kQF/9CdzfDRN9/XiRPjPUoJCCGFxbRvlIPy32eSkJO+ru6wqWR+2fUXYipFMnWKmb9+nX3vrFnz9NXTsCG+/rRcFiPQvOf9yVQMuGIZxyTCMWGAx0OaxNgOAyYZhBAEYhnEnGeMRadSpU1C7tv5805nFvLvjIwrGxjK30ghMniM4dcqy8QkhBIC1NSyZ48aRkTNw8vdiVPRFRrs0xeXkZLa8PZiK5UyJXrdnD5QrB35+0LkzeHpC+/bw1Vcp/AAixSVnEuYBXHvka797xx5VDCimlNqtlPpHKeWd2I2UUgOVUgeVUgfv3r2bTOGK1CpPHjh7FpYfnspH+/5HuVgTM+uMx71Sb+niF0KkKlWrwumjjrRjElnuNGGF+xm+KdqckqbFsKQnmKIStI+Lg27dwMdHT7Xo3Bk++UTPe50+Hfbts9CDiBRh6TEcG6AoUA/oBsxQSmV+vJFhGNMNw6hiGEaV7Nmzp3CIwtK6dYNz/I8vT0zhtdg4fvWehUuJlhw6BJs2Qdeulo5QCCEeypoVPnzfju3Dv6NjsY4sijvJmIrNiT+3Cea1h+iQB223b9erLlu2THiPHDn0sOTs2Skbu0hZyZmEXQcerYjiee/Yo/yAtYZhmAzDuAycQydlQgB6G6LFxz8gouFiagVBzjMrWbKtBkOH6vo9Pj76F54QQqQ21lbWjKo+iv5l+7M8+CQfVWqKye8AzGoBYbcBuH0bihRJ/PoiRfR5kX4lZxJ2ACiqlCqolLIDugJrH2uzGt0LhlLKHT08eSkZYxJpiNkwM25jf6Zd20z7OFtGNNkEbiXZtg3c3eHoUV22QgghUiulFO9WepfhlYfzR9BphpSvR1TQRfBpCkG+lCsHO3fqYcnHbd+u54qJ9CtZ64QppZoDEwBrwMcwjP8ppcYABw3DWKuUUsAPgDcQD/zPMIzFz7qn1AnLGOLMcYz+vQdrgk/T03Dmwy4bUQ5PjFQLIUSasfL8SkbvGU3lzMWY9O8BnKwzQc9VNH69FOXKwbhxYHNvH5s//oAePfSbTY/HZ1OLNEWKtYo0JTY+lhGrOrAlwpe3rNwZ1GU9ys7xpe9jNsOkSXoT3UuXdJX9t96CIUOkooUQwjI2XNrAp7s+pbRrIaZePIWrKYqQlsvo8J4X585B3bpw8SJcvQoLF0KdOpaOWLwqScJEmhEZG8GwlW3YE3Obj2zz0rPzarCx+0/36t9fr6ocNw4qV9arjT7+GIoVg5kzkzhwIYR4QVuvbOWDHR9Q1CUf065dIUvobei6kKNh9Tl2TE/Kb9QIbG0tHalICpKEiTQhIiaMt1a04GhsIKOdStCuwxKwsv5P9zp2DFq00EmYk9MjrxEBxYvD+vVQvnwSBS6EEC9pp99Ohv01jLxOuZlx6w7u/peh8xwo3szSoYkkZrFti4R4UWFRQQxc1pRjsYGMy1yFdh2X/ecEDGDtWl3a4tEEDPTX3brBmjWvGLAQQryC2p61mdxwMtcjb9M7R1Zu5SwOS16HkystHZpIQZKECYsLibjLgGVNOR0Xyg856uHdZhYo9Ur3jI9/ele+ra0+L4QQluSV24tpjacREBNM76wO+HlWhBX94Mh8S4cmUogkYcKigsNu0H+5N+fMkUzwbEHD5pNeOQED8PaGJUsgNjbh8dhYfbyZ9PgLIVKBijkq8luT3wgzRfCGs5nLBWrAmrdh/wxLhyZSgCRhwmICgi7Td0ULLhkx/FKoM3UbjUuye3t56fo6nTrBhQv62IULekuQsmX1eSGESA1Ku5fGp6kPcUY8vTNFcK5oA9jwAeya8MzrAgPhxx+hVy8YPhxOnEihgEWSeW4SppQaqpTKkhLBiIzDP+Ac/Va35ZphYlLxPtSqMypJ768ULF4MZcpAzZqQOTPUqAGlSumesCTobBNCiCRTPGtxZnnPwkbZ0M/an7Mlm8GfX8C2/0EiC+gOHdK/z44cgQYN9HzXxo3hu+8sELz4z567OlIp9TW62v1hwAfYbFhwSaWsjkxdDAMWLICpU8HXFwoX1vudde789ETn9t3T9F/XldvEM7nsUKpWGZSsMcbFQWgouLo+LIQohBCp0bXQa/T9oy9Rpih+sytMiROrofrb0PR/D36pms261M7Ysbq3/74bN/QG4qtX6/+K1OGVVkcahvEZej/HmUBv4LxS6hulVOEkjVKkSe+/Dz/8ACNGwL59MGwYfPUVfPZZ4u1v3j5On3VduUs8hTYPp0H9QVSoANOnJ/pmL0nY2Oj9JSUBE0Kkdnld8+LT1AcHWwf6x17kTKVu8M9kWPeezr6Av/8GFxfo2DHhtXny6GLUPj4WCFz8Jy80J+xez9etex9xQBZguVJKOj4zsFOn9NDeX39Bq1bg6Qnt2ulfENOn6yr1j7p+4xB9NvQgyIinyLaPGTy4L9ev6zkNM2fCO+9Y5DGEECJVyeuiEzFHW0f6R5zgTLU+cGg2rB4E8XHcvKnrHSY22lCiBNy8meIhi//oReaEvauUOgR8B+wGyhqGMRioDHRI5vhEKrZsGbz+Ori5JTyeLZsejlyx4uGx6zcO0Gdzb8IMM27LPufXqb2oW1cPETZoAFu2wPLlcOZMyj6DEEKkRvcTMSdbJ/qHHOD0a2/B8SWwsj+lS5rYsyfxTb937NBzxUTa8CI9YVmB9oZhNDUMY5lhGCYAwzDMQMtkjU6katHRuks8Ma6uEBWlP79xfT99N/UlAjPNb42hfLmuODs/2b5rV1i1KnljFkKItMLTxROfpj442zozwH8Hp2u/C6dWUf5cH4oXieXzzx+MUAKwaxfMnw8DB1ouZvFyXmRO2BeGYVx5yjnpt8jAGjTQvV2P/hIAXQh15Up9/qbfP/Td3JcwZTCjxtdERnQgU6bE75cpE5hMyR+3EEKkFZ4unvh460Ss/50/OVX3fTjzO+ve6M3enTEUK6aTrkaNoH17vVCqQAFLRy1elNQJE/9Zo0a6B+vNNyEgQB+7cwf69NG/BArn20vfP/oTqmBGzf9RqkQ7mjaFpUuf7EY3mfRxb+8UfwwhhEjVPJw98PH2wcXWhQG3NnOq/kfYX17P9sE9WTAnmkqVYPBguHoVmjSxdLTiZUgSJv4zKytYt073hBUuDEWL6smijo4wbeJu+m/pT7CCaa+NpXTxNgDUrg358+u5ZNeu6ftcuaKHIsuUgWrVLPhAQgiRSt1PxFztXBlwYz2nGo5And+M16XuDOoXRYcOPHWUQaRekoSJV+Lqqlc2Xr2qE7KrV+GrT3YzdOdA/K0UU18bS9lirR60V0rP+8qTB8qXh5w5oVIlKFRIiqgKIcSzeDh74NP0XiLm9zunG38GF7fBoq4QG2np8MR/8NxiramNFGtN3fyv6SHIW9aKaa99S8WiT1+7EROjt93Ilg3s7FIwSCGESMNuhN+gz6Y+RMRFMDN/R4pvHg0FakH3JWDnZOnwxGNeqVirEC8qwG8//e8lYFNrfvPMBAzA3h5y55YETAghXkYe5zz81vQ37K3tGXh1FRebfQNXdsP8jhATZunwxEuQJEwkiUC/A/Tf3Icb1orJ1b+icrHWlg5JCCHSrbwueZnZZCZWyor+V5bh22IcXNsH8ztAdKilwxMvSJIw8cqCbhyi/+be+FkrJlX/kqol2lk6JCGESPcKuBXgtya/YTbM9Lu8hGutfoDrh2BeW4gKtnR44gVIEiYSuHtXbyP0zjswcSIEBT27fcjNowzY2Iur1opfqo2iWgnZREEIIVJK4cyFmd54OjHxMfS/tJibbX6Bm8dhbhuIDEz0mjNnYNEi+PNPXddRWI4kYeKBDRv0vmMnTujVivv2QbFiem/IxITfOsGg9a9zyVrxc5VPqVGqc4rGK4QQAopnLc60xtMIiw2j38UF3G43Ge6chrmtEyRiwcHQsiU0bAirV8Onn+rf9bt2WTD4DE5WRwpAF1stVgzWr4fq1R8e37YNunSBy5dJsNVQ5J3TDF7bmeM28GPlD6lf9o2UD1oIIcQDx+4eY+AfA8nhmINZxfrivvJNyF4Meq0Fx6y0agV588LPP4Otrb5m40Z44w04fBg8PS0bf3olqyPFcy1YAC1aJEzAQG89VLOm3lz7vti753hvTWeO2sDY8u9IAiaEEKlA+ezlmdJoCrcjbzPgwlyCOk6Hu+dgbhvOHQvk4EGYMOFhAgbQrJl+oz19uuXizsgkCROArlpftmzi58qV0+cBTIGXGL66A3vtFF+WeRPvirJTrBBCpBaVc1ZmYoOJXAu7xsALCwjpMAPunsV9Qxu86wYmWhKoYUM4ejTlYxWShIl7ihWD/fsTP3d/blh8iB+frmjLX3bwaYletK0yNGWDFEII8Vxeub2YUH8CF4Mv8ualRYR1nIlb7Fk+yt020cn6ly9D9uwWCFRIEia0bt1gxw49P+BRy5fD6dPQpskdRi9rySY7g2GFO9HN60PLBCqEEOK5annU4sd6P3I28CxDfJcT1Wk2hZzPEDIpYSIWGAi//AK9e1su1oxMJuaLB/bsgfbtoUIF/XHgAJw9C+uXB7LuVGMW28QyKH9z3q43ztKhCiGEeAGbLm/iox0fUdOjJoPD21DiUC9uGyU5Xn4N565lYeJE6NoVxo61dKTpl8Um5iulvJVSZ5VSF5RSI57RroNSylBKJRqkSBk1a+pu6V699Mbcb74JF06FsvV0MxbbxPJGnnq8VfdbS4cphBDiBXkX9OaLGl+w+/pu5mTeQnCzueSxPkOJA23xPRPE3LmSgFmSTXLdWCllDUwGGgN+wAGl1FrDME4/1s4FeBfYl1yxiBfn4ADdu9/7IjaSqQuaMcs6ki45qjO80S8opSwanxBCiJfToVgHIkwRfH/wexwLOzKm+1yKLO3FL0XaQZXVQGZLh5hhJWdPWDXggmEYlwzDiAUWA20SafcVMA6ITsZYxMuKi2HO4uZMsQqlddbyfOo9TRIwIYRIo3qV7sXg8oNZc3EN34Ucw+g0F26dlC2OLCw5kzAP4NojX/vdO/aAUqoSkNcwjPXPupFSaqBS6qBS6uDdu3eTPlKRULyJ5YtbM94IoIlbcb5sMRsrJWs4hBAiLRtcfjA9S/VkwZkFTI66CF3m30vE2kkiZiEW+8uqlLICfgSGP6+tYRjTDcOoYhhGleyyjjZ5mePZtLQjY+KuU8spP9+2WoSNVbKNWgshhEghSik+rPIh7Yu2Z9rxacw23YIu8+DWCZjfHqJDLB1ihpOcSdh1IO8jX3veO3afC1AG+Esp5QtUB9bK5HwLMpvZveJ1Pom5SAWHXPzYdhm21rbPv04IIUSaoJRiVPVRNC3QlB8O/cAyFQ6d5+pNv+e1k0QshSVnEnYAKKqUKqiUsgO6AmvvnzQMI8QwDHfDMAoYhlEA+AdobRiG1J+wBMPg6NoBDAs/TmG7rExqtxIHGwdLRyWEECKJWVtZM7bWWGp71OarvV+xwQ7oPOdeIiY9Yikp2ZIwwzDigCHAZuAMsNQwjFNKqTFKqdbJ9brivzm76X3eCvyH7LYu/Np2Ja52rpYOSQghRDKxtbblx3o/UjlnZUbuGslfTk73ErGjML8DRIdaOsQMQYq1Cq5uH0OvS4uxtnNkbtvVeLh4WjokIYQQKSA8NpwBfwzgXNA5pjaaSrXgO7DsDchTEV5fCZnkDfmrslixVpH63f5nIgMvLiLexp7pLRZKAiaEEBmIs50zUxtNJZ9rPt7Z/g6ncxSCTrPhxhHpEUsBkoRlYMHHFvDmickE2doy1duHwlmLWTokIYQQKSxzpsz82uhX3OzcGPznYHzzlIWOs+DGYZ2IxYRZOsR0S5KwDCry7Abe2jeGa7a2TGwwiTI5K1o6JCGEEBaS0ykn05tMB+DNLW9yO7+XTsSuH5JELBlJEpYBxV7ZzTt/DeO0nR3f1/qGannrWDokIYQQFpbfNT9TG00lJDaEQX8OIqRwXeg0C/wOwvyOkoglA0nCMpi4m8f5eGM/9mWy48uqH9GgSCtLhySEECKVKJWtFL/U/4UroVd4a+tbRBZtDB19wO8ALOgkiVgSkyQsAzECLjFmTVf+dLDl4zIDaFO6p6VDEkIIkcpUy12N7+t8z0n/k7z/9/uYSrSAjjPh2v57iVi4pUNMNyQJyyhCb/LT8jascrBmUJGOvF75HUtHJIQQIpVqmL8ho6qPYvf13YzcPRJT8TaENP4NQxKxJCVJWEYQGcicxS2ZlQm6eDTgrZqjLB2REEKIVK5DsQ68W/FdNl7eSOkh4yjcuh19fp9B/JV9xM7uLIlYEpAkLL2LCWfdolaMt4+lsXtFPmnwI0opS0clhBAiDbi8tB/WR3phX2MhX2+dxo9/dGBx7Aysr+8lfl5niI2wdIhpmiRh6VlcDLsXt+dz6xCquRTiW+/fsLaytnRUQggh0gB/f5gyWbHpk+G0LtyayUcns+XOEnqM7cBPV2eg/PbCwi6ShzD3wAAAGmBJREFUiL0CG0sHIJKJOZ4Ty3vw//buO7qqKnH7+HenV0KRIh0BQRQBJ6CIwlBUekdBAbFRIyo2FERpo0gLAgF9AQtKRxSkWFCKtKEOUgQCgoCFpiCEEJLs3x8nAwkGCPNy70lyn89as1buOfsmz529Qh5P2ee51MOUDStCbNOPCfIPcjuViIjkEMuXwz33QJHCfgwsOJBT504xdN1QokKiKNm4LaMWwQvmKaeIPTwLgsLcjpzj6EhYbmQtP30RQ6+EHeQPimJCsxlEBkW6nUpERHKQwEBITHS+DvALYHid4VQrVI1XVr7CrsTVrD3dFlq9BwdWwbQHISnB3cA5kEpYLnTku8F0P7IMAkK5P2k6498uyMcfw9mzbicTEZGcol492LgR4uOd1yEBIYytP5abom5iVtJz1Gy1HW5vBy0nOkVs+kMqYtdIJSyXObX+Pbrv/YQ/AoLYO/pDVnxeCmth2jQoWxY2bHA7oYiI5AQRETB0KNx/P3z6KZw6BT/tzEPqrIn4JeZlvn9PDp46CFUecorYTythensVsWugEpaLJO6cz9ObRvBTYBC/TxrLRyNu47PPYOBAWLQIxo+H5s0vHl4WERG5kh49IDYWRo+GokWhRQuoUq4gM9pNJJVUun3TjeNnjztFrNVE+GkFzOgA53XqJStUwnKJlANreHnZC2wKCaZ+4iBqFq9N/foZx7RqBZUrw7x57mQUEZGcp3lzWLkSTp+G/fvhjTfglsJlGFd/HEcTjtJraS8SzidAlfbQcgLsW+4cEVMRuyqVsFzA/r6DIQu78G1YMH2rxJC6uxXVq2c+Njr64vl9ERGR/1WVglUYUWcEP574kT7L+nA+9TxU7QAt49KKmI6IXY1KWE538hBxn7ZjTngQT5ZvxyNVu1G6NGzZkvnwLVugTBmvJhQRkVyqTok6DKg5gFW/rOKN1W9grYWqD0OL8bBvGcx4GM7rGpjLUQnLyRJOMHNGcyaG+dGqWB1613wNgI4dnWvAVq3KOHzJEudOl9atXcgqIiK5UuvyrelVtRfz985nzKYxzsZqj0CLcbD3OxWxK9BirTlVUgJfz2jB0OAk6uSvzIB6sRceR5Q/v3M3ZPPm8MADUKUKrF8PK1Y414OFaT09ERG5jrrd3o2jCUeZvG0yBcMKkif+EUaM6MitSZa4B2LYM/QRivX5hLA8IW5HzVZ0JCwnSklm46yH6Gv+oEpEKYY3mkKAX8Y+/cADsGcP1KrlPHri/vth717ntYiIyPVkjOHVO1+lXol6vLVuGH3eXcLTT8ObSzpxsNpYyvMNW1/tSFKCjoilZ6y1bme4JtHR0XZDDl/s6vx5WLgQfv4Zbr4Z7rsP/LP6SEdr2TfvcTr9uY78oTcwtdV88obk9WheERGRrDh+KpGaI7sSXnYb793/LtWLOHeJpW74EL8venM47H6K9fkYAoJdTuo9xpiN1trozPbpSJiXrV8PN93krLmyeze89hpUqgS7dmXt/Ue+7kf3P9YSGBjOhKbTVMBERCTbWLMihGL/HkupPCXp/W1vdp1w/rj5RT/K2kJjKJbwFczsCMnnXE6aPaiEedGpU9CsGYwd6zwYddw4p5S9+CI0aQLJyZm/z1qYMQOGdx5Hz/1zOW6CaR/+AcUji3v3A4iIiFzB+fMQ5hfFxPsmEhYYRo9vevDL6V8AOFy4C+MOxMKer2BmJxUxVMK8ato0uPdeaNky4/Ynn4QiRZxTlJnp0wfWfDSP3XePIT4oiNaB7zDy5Vv51788n1lERCSratd27sw3p4swscFEElMS6f5Nd/5M/JNp08Cv+mPQdDTs+RJmdfb5IqYS5kU7dlz+wvh77nH2X2rrVohfto6wpi+yNjSEN+56jX6darNyJYwYAYcOeTaziIhIVhUoADExzlmfMwfK807ddzj812GavR/Dth/P0qkTEP04NBkFu5fArEd9uoiphHlRsWKwc2fm+3bscPZfaumseOo82pkvwkOJufVxWlZ8EHCOnLVuDXPnejCwiIjINRo8GDp0cIpYo8rRHHp3GH+GbqX28JcIDU+77qb6E9BkJOxeDLO7QHKSq5ndohLmRZ07w+zZ8MMPGbevXAlr1kCbNpe84fRRovK25sN8QbQpeR9d//Fsht1580KCHlYvIiLZiDHw3HPOcya3bYOfvmpAv7teZe3RZbz177e4sCpD9Seh8QjYtchni5hHS5gxpqExZpcxJt4Y0zeT/X2MMTuMMVuNMUuNMaU8mcdtN94IEybAP/8JvXvDpEnw1FPOg7WnT4fw8HSDk87w7YwWjLkhlahDVehX++0Li7ECpKTAZ58530tERCS78feHwoWdBcLbV2zPY7c9xsxdM5mybcrFQTWeSitiC2HOYz5XxDxWwowx/sB4oBFQCehgjKl0ybDNQLS19nZgDvC2p/JkFw89BJs3O+fNV6+GcuVg+3Zo0CDdoJRk/jOrPS/7n6RSREnOLniPF/oEcPKks/vYMXj8cShbFu66y5WPISIick2eveNZGpVuROymWBbuS3cnWo2noNFw+PELnytinnxsUQ0g3lq7D8AYMwNoAVy4/Nxa+1268WuBjh7Mk22ULAmvv36Zndayf0FPYpL2UjAkP+OaTIW6YcTEQOnSULQoHD4Mbds614OlOzgmIiKSbfkZP4bcM4SjZ4/Sf1V/CoUVurCYK3d2BSwsfsm5a/LBD31iQVdPno4sBhxM9/pQ2rbLeQJY7ME8OcKxZUPpfmwlfoFhTGw6jQKhBShQwDldGR/vrBe2f79zKjMiwu20IiIiWRfkH0Rs3VhKRZbimW+fIf6P+Is77+zmnJrcvTjtod9n3QvqJdniwnxjTEcgGhh+mf1djTEbjDEbjh496t1wXpSwaSq99nzE8cAgxj0wmZJ5SmbYX6AAVK7sXJAvIiKSE0UFRxHXII7ggGB6LO3BkYQjF3fWeAqavQPxS2F6e0jK3XefebKEHQZKpHtdPG1bBsaYBkA/oLm1NtPFQqy171lro6210QULFvRIWLcl713KC+sG8WNQMCPqjKRyoSpuRxIREfGIohFFiasfx6lzp+j5TU9OJ52+uPMfj0LLONi3HKY9COdOX/4b5XCeLGHrgfLGmDLGmCCgPTA//QBjTDXgXZwCdiST7+ET7K8/MPjLHqwMC6F/9IvUKd3g6m8SERHJwW4pcAuj/jmK+D/j6bOsD+dTz1/cWfVhaP3/4MAq+KQtnPvLvaAe5LESZq1NBmKAL4GdwCxr7XZjzCBjTPO0YcOBCGC2MWaLMWb+Zb5d7nXyMBPnPcSn4cF0vbkD7W7r7HYiERERr6hVrBav13ydNb+uYeDqgRfXEAO4vR20nQIH/w1TW0HiSfeCeogn747EWrsIWHTJtgHpvvbtQz6JJ5k3swVx4f60KFqHmLtecTuRiIiIV7Uq34pfz/zKhP9MoGhEUXpW7Xlx562twC8AZj8GH7WATvMgNJ97Ya+zbHFhvk9KTmL1jNYMDErk7rwVeb3+6AyLsYqIiPiKHlV60LJcSyb8ZwIzts8jJSXdzluawUNT4fft8GEzOHPctZzXm0qYG6xl16dd6GN/o1xoYUY1/oBAv0C3U4mIiLjEUHLXAFL33c2QdQO5seYquneH4//tWxUaQfvpcHS3U8RO546VElTCXPD7V6/Q86/NhAdGMK7pJ4QHhl/9TSIiIrnUsGEwZlQgo+uNpOIN5Sjduw9n8+ykbl04cyZtUPkG8PBMOLEPPmgCf/3maubrQSXMy06vHU+vA/M4ExBMXOOPKBJexO1IIiIirjl50ilhixfD/XUiiGsQR1RIHg5G96JYxV/5+ON0g8vWhUdmw8lDThE79Ytrua8HlTAvOr9jAS9siSU+OIiR9d6hQv4KbkcSERFx1YoVUL06lEhbWbRQWCEm1J9AYnIi55r2YP5Xl9wVWeZe6DjXORL2fmP482fvh75OVMK8xB7cwNDvnmNVaAiv1XiFWiVqux1JRETEdcZAamrGbeXylSO2bizH7QGO1HyWpJRLHupdqiZ0+oykkyf47c1GVCy4l8KF4aWX4NSpK/88a2HNGhg3DqZNg9MurgWrEuYNx/cy+fNHmBsRylMVHqHNLQ+7nUhERCRbqFMHNm2Cffsybq9epAZRq4eQWHAD/b/vT6rN2NTGz69Om/kLyBd5lp0vNWT9wh0cOQL168PZyzx28tgx5+d17gw7djglrFQpmDfPQx/uKlTCPO3MMRbNas2YyCAaF63N03e+7HYiERGRbCMyEgYMgIYNYckSSElxCtkTT0DCxib0qvwMi/cvZuzmsRfec+aM855R06sQ3HURxvhRcmlj3h+6ifz5nXKVmc6dnVOfu3ZBXBx88QV8/TV06wY//uilD5yOSpgnJSWwcUZr+oem8I+8NzO4ntYCExERudSzz8KgQdCvHwQGQo0akC8fLF0K3ao9Qbub2zHph0nM2T0HgOXLoUoVKF8eKFQRHl8CwZGYD5vzcvvVfP7533/Grl2weTO8+Sb4pWs/d9wB3bs7pczbVMI8JTWFn+Z0orc5RrHQgoxpOIUg/yC3U4mIiGRL7dvDxo2QnOycNhw5EqKiwBjDq3e+Sq1itRiydgirDq8iNRUC0j/zJ38ZeGwJRBah9s+tuT186d++/86dzlGwoEz+FN9zj7Pf21TCPMFaji/qQ8+E7QQERRDX+COigqPcTiUiIpLt+WXSTAL8AhhZZyTl8pbj+eXPU+T2XaxfDwcPphsUVQweW8zBhLIMrNAedn6R4XsUK+acckz/eMr/2rkTiha9vp8jK1TCPODsqlh6H17MscAQxj0wmRKRJdyOJCIikqOFB4Yzvv54wgPDeXldT2L6/k6TJrB6tVOsjhyBPgMK8vCXC6BIZZjVGbbOvvD+6GgIC4MpUzJ+399/h9Gj4cknvfyBUAm77lK2zeWVreP4ISSYt+qMoHLBym5HEhERyRUKhxcmrn4cZ86fYectvej29Bm6dHEu7i9b1ln49fOv8uP/2OdQ6m749CnY+AHgLIUxbRq88Qa0aweTJjkX91er5hSwe+/1/udRCbueDqxh5HcvsjQ8jJfu6EP90g3cTiQiIpKrVMhfgZF1RhL/Zzw/lHme7TuT+eUXOHECJk+GQoWA4EhnZf1yDWDBM7DGueq+UiXYvh3q1nWOoJ09C19+Cf37u/NZjM3s5Gg2Fh0dbTds2OB2jL87todPpjfmrTwhPFKuDX1rveF2IhERkVxrzu45DFwzkLY3t2XAXQMyX30gOQnmPgE750PdflD7ReeQmBcZYzZaa6Mz26cjYdfD6SN8N7M1b0cGU7fwnbxY8zW3E4mIiORqbW9uy5OVn2TO7jlM2TYl80EBQdD2fajSAb4bCl/1z/zKfJcEXH2IXFHSGbZNb81LYZZKUTcxrMFY/P383U4lIiKS6z1d7WkO/3WY2E2xFIsoRsMyDf8+yD8AWsRBUASsGQeJJ6HZGMgGf6tVwv5/pCRzeHZHYvxPUCD0BsY2nEJoQKjbqURERHyCn/Fj8D2D+T3hd/p9349CYYW4o/AdmQz0g8bDITQfrHjbKWJtJkFAsPdDp4/l6k/Pyazl5MLn6Jmwk6TAMOIafsANoTe4nUpERMSnBPsHM6buGG6MuJHe3/XmwKkDmQ80Bur1gwf+5VwjNr09JJ3xbthLqIT9j5K+H8lzvyzh56Agxtw3gZvy3uR2JBEREZ+UNyQvE+pPwA8/enzTgxOJJy4/uGYvaDEe9i2DlSO9ljEzKmH/A7t1Nq9vjWN9aAiDaw2lepHqbkcSERHxaSXylOCdeu9wJOEIvb/tTWJy4uUHV+sInT+H2i95L2AmVMKu1f7viVv2El9EhPN0lR40LdfM7UQiIiICVC1UlTfvfZOtR7fS7/t+pNrUyw8uUxsCQ7wXLhMqYdfiyI/M+/xRJkZF0LpME56q0sPtRCIiIpLOfaXu4/no5/nqwFfEbop1O84V6e7IdE6dglmz4OefoWJFaN0aQv5bkv/6jTUz2zIoKpSaBavR/57BGRaGS0kBf/fvdhUREfF5nSt15uBfB3l/2/sUjyjOgxUedDtSpnQkLM3XX8NNN8GSJc6drB99BOXKwZYtwLm/2D2tNX0iLGUiSzKqQRyBfoGcOweDBjlPZg8MdJ5bFRsLqVc4+ikiIiKeZYyhb42+1C5em6HrhrLi0Aq3I2VKJQw4dgw6dIB582DOHOfhnkuWOE9Vb9Uimd9mdKRXwB+EheQlruEUIoIisBYefBDWr3eeO5WSAtOnO++PiXH7E4mIiPi2AL8AhtceToV8FXhh+QvsPL7T7Uh/oxKGc9SrWbO/P0G9XVvLiEZP0/vcj5wKCmX8A5MpEl4EgOXLYc8e+PRTuO02Z/mRGjVg8WKniO3e7cIHERERkQvCAsMYV38cUcFRxCyN4bczv7kdKQOVMCA+HqIzebRm8oq3WVrla3YFBTOi7hgq5q94Yd+CBdCxo3MaMr3ISGjTBhYu9HBoERERuapCYYUYX388CckJ9Fzak9NJp92OdIFKGFCmDGzenHGb3TKdN7dOYGVYKI2CXuPe4hkPk1l7+Qex+/llq+eDioiI+LSb893MqH+O4qc/f+L55c9zPvW825EAD5cwY0xDY8wuY0y8MaZvJvuDjTEz0/avM8aU9mSey+nc2bkebN26tA37lvHBspeZlSeSc8u7MLBNu7+9p2lT+OQTSE7OuP3MGZg7F5o08XxuERERyZqaRWsyoOYAVv+ymqFrh2KzwdESj5UwY4w/MB5oBFQCOhhjKl0y7AngD2ttOWA0MMxTea6kcGGYOhUaN4a+j29nwdzHGZUvipSt9Zna/TmCM3m+Z926UKoUtGt38fqvzZud8tW8OVSo4N3PICIiIlfWqnwrut7elbl75jJ522S343j0SFgNIN5au89amwTMAFpcMqYF8GHa13OA+sZc7iSfZzVu7FxoX7Daf3j9hnBKpN7K968P484amf9fZIxzxOvWW50L+oODoWVLp4RNmODl8CIiIpIlMVVjaFymMWM2jWHRvkWuZvHkYq3FgIPpXh8C7rzcGGttsjHmJFAAOJZ+kDGmK9AVoGTJkp7KS/780LLTnez+d02G1X6bvCGZHAJLJyQEhgyBwYMhMdF57U6FFBERkawwxjC41mASkxMpGFbQ1Sw5YsV8a+17wHsA0dHRHj2JWzZvWd69/71reo8xEBrqoUAiIiJyXQX5BzGm3hi3Y3j0dORhoES618XTtmU6xhgTAEQBxz2YSURERCRb8GQJWw+UN8aUMcYEAe2B+ZeMmQ88mvZ1W+Bbmx1uVxARERHxMI+djky7xisG+BLwB6ZYa7cbYwYBG6y184HJwFRjTDxwAqeoiYiIiOR6Hr0mzFq7CFh0ybYB6b5OBP6+CJeIiIhILqcV80VERERcoBImIiIi4gKVMBEREREXqISJiIiIuMDktBUhjDFHgQMe/jE3cMmq/ZItaF6yH81J9qR5yX40J9mTN+allLU206X5c1wJ8wZjzAZrbbTbOSQjzUv2oznJnjQv2Y/mJHtye150OlJERETEBSphIiIiIi5QCcvctT3BW7xF85L9aE6yJ81L9qM5yZ5cnRddEyYiIiLiAh0JExEREXGBSpiIiIiIC3y6hBljGhpjdhlj4o0xfTPZH2yMmZm2f50xprT3U/qeLMxLbWPMJmNMsjGmrRsZfU0W5qSPMWaHMWarMWapMaaUGzl9TRbmpbsx5gdjzBZjzPfGmEpu5PQlV5uTdOPaGGOsMUbLVnhYFn5Puhhjjqb9nmwxxjzprWw+W8KMMf7AeKARUAnokMk/UE8Af1hrywGjgWHeTel7sjgvPwNdgGneTeebsjgnm4Foa+3twBzgbe+m9D1ZnJdp1trK1tqqOHMyyssxfUoW5wRjTCTwDLDOuwl9T1bnBJhpra2a9r9J3srnsyUMqAHEW2v3WWuTgBlAi0vGtAA+TPt6DlDfGGO8mNEXXXVerLX7rbVbgVQ3AvqgrMzJd9bahLSXa4HiXs7oi7IyL6fSvQwHdCeWZ2Xl7wrAYJz/qE/0ZjgfldU5cYUvl7BiwMF0rw+lbct0jLU2GTgJFPBKOt+VlXkR77rWOXkCWOzRRAJZnBdjTC9jzF6cI2G9vZTNV111TowxdwAlrLULvRnMh2X13682aZdTzDHGlPBONN8uYSJynRljOgLRwHC3s4jDWjveWlsWeBno73YeX2aM8cM5Jfy821kkgwVA6bTLKb7m4hkwj/PlEnYYSN92i6dty3SMMSYAiAKOeyWd78rKvIh3ZWlOjDENgH5Ac2vtOS9l82XX+rsyA2jp0URytTmJBG4Dlhlj9gN3AfN1cb5HXfX3xFp7PN2/WZOAf3gpm0+XsPVAeWNMGWNMENAemH/JmPnAo2lftwW+tVrd1tOyMi/iXVedE2NMNeBdnAJ2xIWMvigr81I+3csmwB4v5vNFV5wTa+1Ja+0N1trS1trSONdPNrfWbnAnrk/Iyu/JjeleNgd2eitcgLd+UHZjrU02xsQAXwL+wBRr7XZjzCBgg7V2PjAZmGqMiQdO4EyeeFBW5sUYUx2YB+QDmhljBlprb3Uxdq6Wxd+V4UAEMDvt3pWfrbXNXQvtA7I4LzFpRyjPA39w8T8qxQOyOCfiRVmck97GmOZAMs7f+i7eyqfHFomIiIi4wJdPR4qIiIi4RiVMRERExAUqYSIiIiIuUAkTERERcYFKmIiIiIgLVMJEREREXKASJiIiIuIClTAR8VnGmOppD+0NMcaEG2O2G2NuczuXiPgGLdYqIj7NGDMECAFCgUPW2jddjiQiPkIlTER8Wtrz5NYDicDd1toUlyOJiI/Q6UgR8XUFcJ57GYlzRExExCt0JExEfJoxZj4wAygD3GitjXE5koj4iAC3A4iIuMUY0xk4b62dZozxB1YbY+pZa791O5uI5H46EiYiIiLiAl0TJiIiIuIClTARERERF6iEiYiIiLhAJUxERETEBSphIiIiIi5QCRMRERFxgUqYiIiIiAv+Dw6/lkRZjhKxAAAAAElFTkSuQmCC",
            "text/plain": [
              "<Figure size 720x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          },
          "output_type": "display_data"
        }
      ],
      "source": [
        "xp = np.linspace(0, 0.5, 500) \n",
        "yp = 0.0*xp \n",
        "for i in range(len(yp)):\n",
        "    yp[i] = np.dot(V, sigmoid(W*xp[i]+bs)) + b0 \n",
        "ytruth = np.sin(2*np.pi*xp)\n",
        "fig = plt.figure(figsize=(10,4))\n",
        "ax = fig.add_subplot(111)\n",
        "ax.set_xlabel(\"x\"); ax.set_ylabel(\"y\")\n",
        "ax.scatter(xt, yt, facecolor=\"none\", edgecolor=\"b\", s=50, label=\"Data\")\n",
        "ax.plot(xp,yp*ystd+ymean,color=\"C01\",label=\"ANN\")\n",
        "ax.plot(xp,ytruth,color=\"C02\",label=\"Ground Truth\")\n",
        "ax.legend()\n",
        "plt.show()\n",
        "plt.close()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jOaGjLWiGE0-"
      },
      "source": [
        "のように、データの特徴をそこそこうまく捉えたニューラルネットワークへと学習が進みました。\n",
        "\n",
        "実際には、ニューラルネットワークの精度(良さ)は、前述のような検証データに対する汎化性能で評価します。  \n",
        "\n",
        "上で示した例では、3層のニューラルネットワークにデータからそれらしい関数を学習させてみました。\n",
        "\n",
        "ニューラルネットワークの構造をより複雑化したりしながら、より複雑で高次元な回帰問題に応用したり、回帰問題だけではなく分類問題・画像生成・物体検知などなど、各種の楽しい実社会の問題に応用していきます。 (例: 第2回で説明した敵対的生成ネットワーク)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qMB-1YMwdqij"
      },
      "source": [
        "## 最適化手法に関するまとめ\n",
        "\n",
        "ニューラルネットワークの学習がうまく進むかどうかは一般に\n",
        "1. ネットワークの構造(アーキテクチャとも言ったりします)や活性化関数(とその微分)の持つ性質\n",
        "2. 最適化手法や手法内のパラメータ\n",
        "3. 重みやバイアスの初期値\n",
        "\n",
        "などに強く依存します。\n",
        "\n",
        "1.に関して   \n",
        "回帰問題における代表的な活性化関数としては\n",
        "最近の傾向として、sigmoidよりも以下のReLU関数が使われることが多いです。  \n",
        "\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 157,
      "metadata": {
        "id": "LKDihzzDT0-H",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "def relu(z):\n",
        "    return (z > 0)* z   \n",
        "\n",
        "def relu_der(z):\n",
        "     return (z > 0)*1.0 \n",
        "     \n",
        "#いずれも、zが実数値でもnp.array型のベクトルでも対応可能な表式"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 158,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 265
        },
        "id": "M962DFirJgu9",
        "outputId": "ce0db346-fd65-47d4-8fcd-911179489442",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlMAAAD4CAYAAADIBWPsAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deXyU5bn/8c+VyU7YQxAJS5QoiyhLQK0bCiJuIKdV6GlPtfXUc1r12GM3u6Noq91+bdVzLC6FWpe6tBYR9SiouFAlYZF9UQIkLElICATIMjP3748kGBGGQGbmmeX7fr3yyswzTybXZJI71zzPPffXnHOIiIiIyIlJ8boAERERkXimZkpERESkA9RMiYiIiHSAmikRERGRDlAzJSIiItIBqV5949zcXDdw4ECvvr2IeKCkpKTKOdfL6zo6SuOXSPIJNX551kwNHDiQ4uJir769iHjAzLZ4XUM4aPwSST6hxi+d5hMRERHpADVTIiIiIh2gZkpERESkAzybM3UkTU1NlJWVUV9f73UpMS8zM5P8/HzS0tK8LkVESL7xS2OQyCdiqpkqKyujc+fODBw4EDPzupyY5Zxj9+7dlJWVUVBQ4HU5IkJyjV8ag0Q+7Zin+czsMTOrMLNVR7ndzOwPZrbJzD40s1EnWkx9fT09e/ZM+IGoo8yMnj17Js0rYJFwOdJ4ZmY9zOw1M9vY8rn7idx3Mo1fGoNEPq09c6ZmA5NC3H45UNjycRPwvx0pKBkGonDQz0nkhMzms+PZHcAC51whsKDl+glJpr/LZHqsIsdyzNN8zrlFZjYwxC5TgD875xzwTzPrZmZ9nHM7wlSjSEILBh31/gAHGwMcbArQ4A9S3xSg0R+kKeBoCgRpCgTxBxz+oMMfDBIIOgLB5uvBoCPoIOAczn1yPegcrvUz4Bw4mrdB86kaDm3/5PbWbYc7tKnNjRlpPm6+eFDEfjbhdpTxbAowruXyHOBN4PtRK0pEoq6+KcC9L6/jpgtP4eRuWR2+v3DMmeoLbGtzvaxl22eaKTO7ieajV/Tv3z8M3zoy7rnnHp588kl8Ph8pKSn88Y9/5OGHH+b2229n6NChEfu+V1xxBU8++STdunX71PYZM2aQk5PDd77znYh9bwkP5xy1B5so33OQHXvq2bWvnqp9jeze30D1/kZqDzZRe7CJffV+9tX7OdDo50BjwOuyj1vrQYkumWlx1UwdRe82L/52Ar2PtFM8jF8+n4/hw4fj9/spKCjg8ccf/8x40taRxpYbbriBq666ii984QuHtuXk5FBXVxfR2kWi6dF3NjP7vVIuHdo7ZpqpdnPOzQJmARQVFR3hta/3Fi9ezLx581i6dCkZGRlUVVXR2NjII488EvHvPX/+/Ih/DwmfXXvrWVVey9ode9lYUcemijq27j7Avgb/Z/btmpVGj07pdM1Ko3t2OgN6diInI5WcDB/Z6alkp/vISveRmeYjIzWFjNQU0lNTSPf5SPMZqb4U0n0p+FIMX4qR6jN8Zoeu+1IMM0ix5u1mkJJiGM3bzMCwQ03QZ67TfNrGoM0+yXcaxznnzOyIY1M8jF9ZWVksX74cgOuvv54HH3yQH/3oRx5XJRJbdtbW8+Abm7hsWG/OG5QblvsMRzNVDvRrcz2/ZVtc2rFjB7m5uWRkZACQm9v8gx43bhy//vWvKSoq4tFHH+W+++6jW7dunHXWWWRkZPDAAw9www03kJWVxbJly6ioqOCxxx7jz3/+M4sXL+bss89m9uzZADz11FP8/Oc/xznHlVdeyX333Qd8ElGRm5vLPffcw5w5c8jLy6Nfv36MHj3ak5+HfGJb9QHe2lDJ+5urWbK5mp17P5l827dbFoPyciga0J1+PbLp2y2LPt2y6N0lg56dMkhP1ZJuMWxX69QEM+sDVHhdUDice+65fPjhhwB89NFH3HzzzVRWVpKdnc3DDz/M4MGDPa5QxBv3vrwWf9Dx4yvDd6YpHM3UXOAWM3saOBuoDcd8qTtfXM2a7Xs7XFxbQ0/uws+uHhZyn4kTJ3LXXXdx2mmnMWHCBKZNm8ZFF1106Pbt27czc+ZMli5dSufOnbnkkks466yzDt1eU1PD4sWLmTt3LpMnT+bdd9/lkUceYcyYMSxfvpy8vDy+//3vU1JSQvfu3Zk4cSIvvPAC11xzzaH7KCkp4emnn2b58uX4/X5GjRqlZsojH1XW8Y/l23l55Q42VjSf5ujdJYOxBT0Z2a8bw/O7MqRPF3IyYmqVETk+c4HrgXtbPv+jo3fo1fjVKhAIsGDBAm688UYAbrrpJh566CEKCwt5//33+eY3v8nChQvDWp9IPCgureaF5du59ZJB9OuRHbb7PeZ/ADN7iubJmblmVgb8DEgDcM49BMwHrgA2AQeAr4atOg/k5ORQUlLC22+/zRtvvMG0adO49957D93+wQcfcNFFF9GjRw8Arr32WjZs2HDo9quvvhozY/jw4fTu3Zvhw4cDMGzYMEpLS9myZQvjxo2jV6/m4OkvfelLLFq06FPN1Ntvv83UqVPJzm5+oidPnhzxxy2faPQHeXnVDma/V8qyrXswg3MKejJtTD8uHpzHKbmdkvIUWCI4ynh2L/CMmd0IbAGu867Cjjl48CAjRoygvLycIUOGcOmll1JXV8d7773Htddee2i/hoaGo97HkX639fsuiSAQdMx4cTUndcnkG+NODet9t+fdfF88xu0OuDlsFbVo7yuwSPD5fIwbN45x48YxfPhw5syZ0+6vbT09mJKScuhy63W/36/VgmNYoz/IX5ds5YE3NrFrbwOn5HbiR1cMYfKIk+ndJdPr8iQMQoxn48P5fbwav1rnTB04cIDLLruMBx98kBtuuIFu3bodmkt1LD179qSmpubQ9erq6kPTHUTi2bPF21hVvpffTx9Bdnp4zyZoIsdh1q9fz8aNGw9dX758OQMGDDh0fcyYMbz11lvU1NTg9/t5/vnnj+v+x44dy1tvvUVVVRWBQICnnnrqU6cRAS688EJeeOEFDh48yL59+3jxxRc79qAkJOcc81fu4JLfvMlP/rGaAT06MfurY3j99ov4+oWnqJGSuJOdnc0f/vAHfvOb35CdnU1BQQHPPvss0Pz7vmLFiqN+7bhx4/jrX/9KY2MjALNnz+biiy+OSt0ikVJ7sIlfvrqeMQO7M/msk8N+/5rocZi6ujpuvfVW9uzZQ2pqKoMGDWLWrFmH3ibct29ffvjDHzJ27Fh69OjB4MGD6dq1a7vvv0+fPtx7771cfPHFhyagT5ky5VP7jBo1imnTpnHWWWeRl5fHmDFjwvoY5RPlew7ykxdWsXBdBUP7dGHO14ZzYWGuTmtI3Bs5ciRnnnkmTz31FE888QTf+MY3uPvuu2lqamL69OmH5nrefffd/O53vzv0dWVlZZSUlDB69Gh8Ph+nnnoqDz30kFcPQyQsfv/6RmoONPKzq8dGZHw3d6TV+aKgqKjIFRcXf2rb2rVrGTJkiCf1HI+6ujpycnLw+/1MnTqVr33ta0ydOjXqdcTLzytWvbJqJ999bgWBoOP2S0/jhs8NJNWng7WRZGYlzrkir+voqHgev8IpGR+zxJ9NFfuY9Lu3ubYon1/8y5knfD+hxi8dmToBM2bM4PXXX6e+vp6JEyd+avK4xD5/IMgvXl7Ho+9s5qz8rjzwr6PC+q4OERGJDc457nxxDVnpPr4z8fSIfR81Uyfg17/+tdclyAmqbwpw61PLeG3NLq4/dwA/vHIIGak+r8sSEZEIeG3NLt7eWMVPrxpKz5yMY3/BCYq5Zso5p/kq7eDV6dl4VnuwiX+fs4TiLTXcOXkY139uoNclSYJJpvFLY5DEuvqmAHe/tJbCvBz+7dwBx/6CDoipCSKZmZns3r1bf6TH4Jxj9+7dZGbqXWbtdaDRz1f/9AHLt+3h/i+OVCMlYZdM45fGIIkHj76zma3VB/jZ1cNIi/B82Jg6MpWfn09ZWRmVlZVelxLzMjMzyc/P97qMuNDoD/Kff1nK8m17+J8vjWLSGX28LkkSULKNXxqDJJa15u9NHNqb8wsjv05aTDVTaWlpFBQUeF2GJBDnHN9//kMWbajkvs8PVyMlEaPxSyR2RCJ/L5SYOs0nEm6z3yvl78vKuf3S05g2pr/X5YiISISVbGnO37vpglPo3zM679RWMyUJq7i0mnteWsuEIXnccvEgr8sREZEICwQdM+au4aQumXzz4vDm74WiZkoSUs3+Rr75xFL6ds/iN9eNICUlOd5hJSKSzJ4t3sbK8lp+cMXgsOfvhRJTc6ZEwuXOF1dTvb+RF244j65ZCpcWEUl0tQeb+FUE8/dCUTMlCef1Nbt4Yfl2bhtfyBl925+bKCIi8esPCzZSfaCRORHK3wtFp/kkodQeaOKHf1/J4JM6c7PmSYmIJIVNFfuY814p08f08+RFtI5MSUL57Wvrqapr4NHrx5CeqtcKIiKJLlr5e6Hov40kjE0Vdfzl/a3869n9GZ6v03siIsng9bUVvL2xiv+ecFpE8/dCUTMlCeMX89eSnebjWxNO87oUERGJgvqmADPnrYlK/l4oaqYkIbyzsYoF6yq4+ZJB5Hr0ykRERKIrmvl7oaiZkrjnnOPeV9aS3z2LGxRgLCKSFKKdvxeKmimJe2+ur2RV+V7+a3whmWk+r8sREZEoiHb+XihqpiSuOef4w8KN9O2WxdSRfb0uR0REoqC4NPr5e6GomZK4tvij3Szbuof/HHeqp+fLRUQkOgJBx4wXV0c9fy8U/feRuHb/wk3kdc7g2tH5XpciIiJR8GzxNlaV7416/l4oaqYkbq0sq2Xxx7v5+gWnaK6UiEgSaM3fKxoQ/fy9UGKjpRM5AX9eXEp2uo/rxvTzuhQREYmCQ/l7k6OfvxeKjkxJXKrZ38g/Vmxn6si+dM1K87ocERGJMK/z90JRMyVx6a/F22j0B/nKuQO9LkVERCIsFvL3QlEzJXEnEHQ8vngL55zSg9NP6ux1OSIiEmGxkL8XipopiTtvrKugfM9BrXYuIpIEGvwB7n7J+/y9UNRMSdx5tmQbuTkZTBjS2+tSREQkwh59ZzNbdh/gp1cPjdn1BNtVlZlNMrP1ZrbJzO44wu39zewNM1tmZh+a2RXhL1WkeeL5wnUVXDPiZFJj9I9K4pOZ/beZrTazVWb2lJllel2TSLLbWVvPAwub8/cuKOzldTlHdcz/RmbmAx4ELgeGAl80s8ODcH4MPOOcGwlMB/4n3IWKAMxdsZ2mgOPzWqRTwsjM+gL/BRQ5584AfDSPZSLiofteWRcz+XuhtOel/Vhgk3PuY+dcI/A0MOWwfRzQpeVyV2B7+EoU+cTzS8sY2qcLQ/p0OfbOIscnFcgys1QgG41jIp4q2VLD35eVx0z+Xijtaab6AtvaXC9r2dbWDODLZlYGzAduPdIdmdlNZlZsZsWVlZUnUK4ks4279vFhWa2OSknYOefKgV8DW4EdQK1z7v/a7qPxSyR6gkHHjLmxlb8XSrgmnXwRmO2cyweuAB43s8/ct3NulnOuyDlX1KtX7J77lNj03NIyUlOMKSNiJ0JAEoOZdaf5iHsBcDLQycy+3HYfjV8i0fNsyTZWltfGVP5eKO1ppsqBtnkd+S3b2roReAbAObcYyARyw1GgCDQv2DZvxQ4uKMwlNwbXGJG4NwHY7JyrdM41AX8DPudxTSJJaW99bObvhdKeZmoJUGhmBWaWTvOkzLmH7bMVGA9gZkNobqZ0HFzCZkVZLeV7DnLlmfHxhyVxZytwjpllW3Pg13hgrcc1iSSlP7y+kd37G5kxeVhM5e+FcsxmyjnnB24BXqV5cHnGObfazO4ys8ktu30b+LqZrQCeAm5wzrlIFS3J5+WVO0jzGZdqbSmJAOfc+8BzwFJgJc1j4yxPixJJQpsq9jH7vVKmFcVe/l4o7ToR6ZybT/PE8rbbftrm8hrgvPCWJtLMOcf8VTs4b1AuXbMVaiyR4Zz7GfAzr+sQSVbOOe6at5asdB/fvSz28vdC0aqHEvNWle9lW/VBrhjex+tSREQkQhasrWDRhsqYzd8LRc2UxLz5q3aQmmJMHKpTfCIiiajBH2BmjOfvhaJmSmKac475K3dw7qk96Zad7nU5IiISAfGQvxdK/FUsSWXDrjq27D7A5WfoFJ+ISCLatTc+8vdCUTMlMW3Bul0AjB+S53ElIiISCfe+HB/5e6GomZKYtmBtBcP7dqV3l0yvSxERkTBrzd/7+gUFMZ+/F4qaKYlZ1fsbWbq1hksG66iUiEii+VT+3rhBXpfTIWqmJGa9sa4C52CCFuoUEUk4bfP3OmXEfv5eKGqmJGYtXFdBXucMhp3cxetSREQkjOIxfy8UNVMSkxr9QRZtqOSSwXmkpMRHNpOIiLRPPObvhaJmSmJScWk1+xr8jNcpPhGRhNKavzd9THzl74WiZkpi0lsbKknzGecN6ul1KSIiEibOOe58cQ1Z6T6+MzG+8vdCUTMlMWnRxiqKBvQgOz2+JyWKiMgnXl9bwdsbq/hWHObvhaJmSmJOxb561u7YywWn5XpdioiIhEmDP8DdL61hUF4OX4nD/L1Q1ExJzHl3UxUAF8ZprICIiHzWofy9q+Izfy+UxHo0khDe3lBFj07pDO2jJRFERBLBztrm/L1Lh/bmwtMS74WymimJKcGgY9HGKs4flKslEUREEsR9r6zDH3D8+MohXpcSEWqmJKas27mPqroGLijUfCkRkURwKH/vwgIG9OzkdTkRoWZKYsrbGysBuEDzpURE4l4w6LjzxdX07pIR9/l7oaiZkpjyzqYqCvNyOKlrpteliIhIBz1XUsaHZbX84PIhcZ+/F4qaKYkZjf4gS0qrOW+QTvGJiMS7vfVN/PLVdYwe0J0pI+I/fy+UxG0TJe4s37aH+qYg556qVc9FROJda/7en24YmxD5e6HoyJTEjMUf7cYMzilQMyUiEs82VdQx+71SphX1Y3h+YuTvhaJmSmLGex9VMezkLnTNTvO6FBEROUHOOe6a15K/d1ni5O+FomZKYkJ9U4BlW/dw7ik6KiUiEs8WrK1g0YZKvjXhNHITKH8vFDVTEhNKttTQGNB8KRGReNbgDzAzQfP3QlEzJTFh8Ue78aUYYwb28LoUERE5QY+9U5qw+XuhJM8jlZi2+OPdDO/blc6Zmi8lIhKPdu2t5/6FGxM2fy8UNVPiuf0NflZs26NTfCIicezelxM7fy8UNVPiuaVba/AHHedo8rmISFxqzd/79wsSN38vlHY1U2Y2yczWm9kmM7vjKPtcZ2ZrzGy1mT0Z3jIlkX2wuZoUg9EDuntdiiQxM+tmZs+Z2TozW2tm53pdk0g8CAYdM+Y25+/dfHHi5u+FcswV0M3MBzwIXAqUAUvMbK5zbk2bfQqBHwDnOedqzCwvUgVL4nl/czVn9O1KTgLnNklc+D3winPuC2aWDmR7XZBIPHi2ZBsry2v53bQRCZ2/F0p7jkyNBTY55z52zjUCTwNTDtvn68CDzrkaAOdcRXjLlERV3xRg+bY9nF2gd/GJd8ysK3Ah8CiAc67RObfH26pEYt/e+iZ+9er6pMjfC6U9zVRfYFub62Ut29o6DTjNzN41s3+a2aQj3ZGZ3WRmxWZWXFlZeWIVS0JZsW0Pjf4gYxUhI94qACqBP5nZMjN7xMw+NfFD45fIZ7Xm7824eljC5++FEq4J6KlAITAO+CLwsJl1O3wn59ws51yRc66oV6/ketukHNkHm6sBGDNQ86XEU6nAKOB/nXMjgf3Ap+aHavwS+bRky98LpT3NVDnQr831/JZtbZUBc51zTc65zcAGmpsrkZA+KK1m8Emd6Zad7nUpktzKgDLn3Pst15+jubkSkSNIxvy9UNrTTC0BCs2soGVS5nRg7mH7vEDzUSnMLJfm034fh7FOSUBNgSAlW2o0X0o855zbCWwzs9b/CuOBNSG+RCSptebv3Ta+MGny90I55rR755zfzG4BXgV8wGPOudVmdhdQ7Jyb23LbRDNbAwSA7zrndkeycIl/q8prOdAY0HwpiRW3Ak+0vGj8GPiqx/WIxKTW/L1Te3Xi+s8N9LqcmNCu9zA65+YD8w/b9tM2lx1we8uHSLssKW2ZL1Wg+VLiPefccqDI6zpEYl1r/t6cr41Nqvy9UPRTEM98sLmGgT2zyeuc6XUpIiLSDq35exOG9OaiJMvfC0XNlHgiGHSUbKlmzEDNlxIRiRf3teTv/eSq5MvfC0XNlHji46o6ag40qZkSEYkTJVtq+FsS5++FomZKPPHB5hoAirS+lIhIzAsGHXe+mNz5e6GomRJPFJdWk5uTTkGuXt2IiMS650rK+LCslh9cPiRp8/dCUTMlnliypZqiAT2SOn5ARCQe7K1v4pevrkv6/L1Q1ExJ1O2srWdb9UHGaLFOEZGYp/y9Y1MzJVF3aH0pzZcSEYlpyt9rHzVTEnXFpdVkp/sY2qeL16WIiMhRKH+v/dRMSdQtKa1hZP9upGrlXBGRmKX8vfbTfzOJqr31TazbuVfrS4mIxLAGf4C7X1rDoLwc5e+1g5opiaplW/cQdFA0QM2UiEiseuydUkp3H+CnVw1V/l476CckUVVSWo0vxRjRv5vXpYiIyBG0zd+7UPl77aJmSqJqSWkNQ/t0IUeLvomIxCTl7x0/NVMSNU2BIMu21ShCRkQkRi3d2py/d6Py946LmimJmtXb91LfFNTkcxGRGBQMOu6cu5q8zsrfO15qpiRqilsW6ywaoCNTIiKx5rmlZawoq+UHVwzWVIzjpGZKoqa4tIb+PbLJ65LpdSkiItLG3vomfvnKOkb178Y1I/p6XU7cUTMlUeGco3hLteZLiYjEoPsXNOfv3Tn5DOXvnQA1UxIVpbsPUFXXqPWlRERizKaKOv70binXjVb+3olSMyVR0RpuPLZAR6ZERGKFc46ZLfl7352k/L0TpWZKomLJ5mq6Z6dxaq8cr0sREZEWC9dV8Jby9zpMzZRERfGWGooG9tC5eBGRGNHgDzBznvL3wkHNlERcxb56NlftZ4wmn4uIxAzl74WPfnoScSWlNQBarFNEJEbs2lvPA8rfCxs1UxJxS0pryExLYdjJepeIiEgsuO/ldTQpfy9s1ExJxC0prWZEv26kp+rXTUTEa8rfCz/9d5OIqmvws3p7LWN1ik9ExHPK34sMNVMSUcu21hB0UKRmSkTEc8rfiww1UxJRSzZXk2IwSuHGIiKeUv5e5KgtlYh6f3M1Z/TtqldAIiIea83fe+yGMVrzL8zadWTKzCaZ2Xoz22Rmd4TY7/Nm5sysKHwlSryqbwqwbNsezi7QKT6JD2bmM7NlZjbP61pEwumjyub8vWtH53Nmfjevy0k4x2ymzMwHPAhcDgwFvmhmQ4+wX2fgNuD9cBcp8enDsloa/UHGFvT0uhSR9roNWOt1ESLhNnPeGrLSfHxv0mCvS0lI7TkyNRbY5Jz72DnXCDwNTDnCfjOB+4D6MNYnceyDzbsxQyufS1wws3zgSuARr2sRCaeF63bx5vpKbpug/L1IaU8z1RfY1uZ6Wcu2Q8xsFNDPOfdSqDsys5vMrNjMiisrK4+7WIkv72+u5vTenemWne51KSLt8Tvge0DwSDdq/JJ41OAPcNeLazi1Vyfl70VQh9/NZ2YpwG+Bbx9rX+fcLOdckXOuqFcvLV+fyJoCQUq21Gi+lMQFM7sKqHDOlRxtH41fEo/+9G5L/t7Vw5S/F0Ht+cmWA/3aXM9v2daqM3AG8KaZlQLnAHM1CT25rd6+lwONAc2XknhxHjC5ZQx7GrjEzP7ibUkiHVOxt577FzTn712k/L2Iak8ztQQoNLMCM0sHpgNzW290ztU653KdcwOdcwOBfwKTnXPFEalY4sIHm3cDMKZA86Uk9jnnfuCcy28Zw6YDC51zX/a4LJEOufcV5e9FyzGbKeecH7gFeJXmd7k845xbbWZ3mdnkSBco8en9j6s5JbcTeZ0zvS5FRCTpLN1aw9+WKn8vWtq1kqJzbj4w/7BtPz3KvuM6XpbEM38gyAebq7l6xMlelyJy3JxzbwJvelyGyAkLBh0zlL8XVZqNJmG3avte9jX4+dypmi8lIhJtz5WU8aHy96JKzZSE3eKPmudLnXOKmikRkWjaW9/EL19V/l60qWWVsHvvoypO791Zi8OJiESZ8ve8oSNTElaN/iDFpTWcq1N8IiJRpfw976iZkrBaUbaHg00BneITEYki5xx3vaj8Pa+omZKwWvxRcx7fOado5XMRkWhZuK6CtzYof88raqYkrN77qIqhfbooj09EJEoa/AFmzlvDKb068ZVzB3pdTlJSMyVhU98UYOnWPZyrU3wiIlFzKH/vqqGkp+rfuhf0U5ew+WBzNY3+IOcX5npdiohIUvgkfy+PcafneV1O0lIzJWHz9sZK0n0pnK1wYxGRqGjN3/vxlUO9LiWpqZmSsHl7YxVjCrqTle7zuhQRkYS3rE3+3sBc5e95Sc2UhEXF3nrW7dzHBYW9vC5FRCThKX8vtqiZkrB4e2MVABdovpSISMQ9v7SMFWW13HG58vdigZopCYu3N1aSm5POkJO6eF2KiEhC21ffxH2vrGek8vdihtpZ6bBg0PHOpirOH5RLSoqyoEREIun+hZvYvb+BR68v0pgbI3RkSjps7c69VNU1ar6UiEiENefvbeba0fmc1U/5e7FCzZR02JvrKwHNlxIRibSZ89aQkerju5cpfy+WqJmSDluwdhdn5nclr0um16WIiCSshet28eb6Sr41oZBenZW/F0vUTEmHVNU1sGzbHsYP7u11KSIiCavBH+CuF5W/F6vUTEmHvLm+Eudg/BDFGIiIRIry92KbnhHpkIXrdtG7SwbDTtaSCCIikaD8vdinZkpOWKM/yKINVVwyOA8zvT1XRCQS7ntlvfL3YpyaKTlhH2yupq7Br/lSIiIRsmxrDc8vLVP+XoxTMyUn7PW1u8hITeG8QVoSQUQk3JS/Fz/UTMkJCQYdr67eyfmDcslK93ldjohIwlH+XvxQMyUnZEXZHnbU1nPF8D5elyIiknBa8/dGKX8vLqjVlRMyf+UO0nzGhKGaLyUiEm6t+XuP3aD8vXigI1Ny3JxzzF/ZfI61lhYAAA3sSURBVIqva1aa1+WIiCSUtvl7Z+Yrfy8eqJmS4/ZhWS3lew7qFJ+ISATMnLeGTOXvxRU1U3Lc5q/aQWqKMXHoSV6XIiKSUFrz925T/l5caVczZWaTzGy9mW0yszuOcPvtZrbGzD40swVmNiD8pUoscM7x8sqdnDcol67ZOsUnicHM+pnZGy3j2Gozu83rmiT5NPgDzJy3Vvl7ceiYzZSZ+YAHgcuBocAXzezwZViXAUXOuTOB54BfhrtQiQ1Lt9awtfoAV52pU3ySUPzAt51zQ4FzgJuPMM6JRNSf3i1lc9V+5e/FofY8W2OBTc65j51zjcDTwJS2Ozjn3nDOHWi5+k8gP7xlSqx4rqScrDQfl2u+lCQQ59wO59zSlsv7gLWA3o8uUdOavzd+sPL34lF7mqm+wLY218sIPcjcCLx8pBvM7CYzKzaz4srKyvZXKTGhvinAvA+3c/kZJ2kBOUlYZjYQGAm8f9h2jV8SMa35ez+5SgdE41FYjyOa2ZeBIuBXR7rdOTfLOVfknCvq1atXOL+1RMFra3axr97P50frwKMkJjPLAZ4HvuWc29v2No1fEimt+XtfO1/5e/GqPYcXyoF+ba7nt2z7FDObAPwIuMg51xCe8iSWPL+0jD5dMznnlJ5elyISdmaWRnMj9YRz7m9e1yPJoW3+3i2XKH8vXrXnyNQSoNDMCswsHZgOzG27g5mNBP4ITHbOVYS/TPFaxd56Fm2oZOrIvvi0Gq8kGDMz4FFgrXPut17XI8mjNX/v+5OUvxfPjtlMOef8wC3AqzRPynzGObfazO4ys8ktu/0KyAGeNbPlZjb3KHcncerZkjKCDp3ik0R1HvBvwCUtY9hyM7vC66IksbXm743s342pI/V+h3jWrjbYOTcfmH/Ytp+2uTwhzHVJDPEHgjzxzy2cN6gnp/bK8bockbBzzr0D6JCrRNX9CzdRVdfAo9crfy/eaSELOabX11awvbZei8iJiITJx23y987qp/y9eKdmSo7pz4tL6dsti/GDtfaJiEg4tObvfW+S8vcSgZopCWnjrn2899FuvnROf1J9+nUREemohet28Yby9xKK/jtKSHMWl5LuS2FaUb9j7isiIqE1+oPK30tAaqbkqCr21vNMcRlTR/alZ45ePYmIdNSf3t2s/L0EpGdSjmrWoo/xB4J88+JTvS5FRCTuVeyt5w8LNjJhiPL3Eo2aKTmi3XUNPPH+VqaM6MuAnoo3EBHpqNb8vR9fqfy9RKNmSo7o0Xc2U+8PcLOOSomIdJjy9xKbmin5jKq6Bv68eAuXn3ESg/I6e12OiEhcU/5e4lMzJZ/xu9c3cLApwO2XnuZ1KSIica81f++Oy5W/l6jUTMmnbNi1jyff38qXz+6vo1IiIh3UNn/vmhHK30tUapHlU+55aS2dMlK5bYKOSomIdJTy95KDjkzJIQvW7uKtDZXcNr6QHp3SvS5HRCSuKX8veaiZEgBqDzbxw7+v5PTenbUqr4hIGCh/L3momRIA7nlpDVV1jfzq2jO1Kq+ISAe15u/913jl7yUD/dcU3tpQyTPFZdx04Smcma9D0SIiHdE2f+/6zw30uhyJAjVTSW5nbT3ffmY5g/JyuG18odfliIjEvdb8vZ8ofy9p6N18SazRH+TmJ5dyoDHAU18fRWaaz+uSRETiWmv+3vjBeVys/L2koWYqid3z0hpKttTwwL+OpLC31pQSEemo+15ZT2MgyI+vUv5eMtHxxyQ1a9FHzFm8hRvPL+CqM0/2uhwRkbjXmr934/mnUKD8vaSiZioJPbNkGz+fv46rzuzDD68Y4nU5IiJxT/l7yU3NVJJ5vqSMO/72IRee1ovfXjcCn1bkFRHpsNb8ve9PUv5eMtIzniScc8xa9DG/eHkd5w3qyUNfHqV3mYiIhEHb/L2pI5W/l4zUTCWBBn+AmfPW8Jd/buWqM/vwm+vOIiNV79wTEQkH5e+JmqkEt3X3AW5+cikry2v5jwtP4fuTBuuPXUQkTJS/J6BmKmEFgo7HF5fy6//bQIrBrH8bzcRhJ3ldlohIQpk5bw0ZqT6+O+l0r0sRD6mZSkBLSqu5+6W1rNi2hwsKc/n51OH065HtdVkiIgmlNX/vR1cMIa9zptfliIfUTCUI5xxLt+7h/oUbeXN9Jb06Z/D76SOYfNbJmOm0nohIOCl/T9pSMxXn6hr8vLJqJ48vLmVFWS3dstP4weWD+cq5A8lK1yRzEZFIaM3f+9NXx+id0aJmKh7tOdDIWxsqeW3NLl5fu4v6piCn9urEzCnD+JdR+XTSGiciIhGj/D05nP7rxjjnHNtr61lZVktxaTVLSqtZWV5L0EGPTul8YXQ+14zoy+gB3XU6T0QkCpS/J4drVzNlZpOA3wM+4BHn3L2H3Z4B/BkYDewGpjnnSsNbauJyzlFzoIkdtQcpqznItuoDbK7az8aKOjbu2kfNgSYA0lNTGNGvG7dcUsjFp/fizPxuWsFcJEyONc6JALy2ZhfPLy3jPy5S/p584pjNlJn5gAeBS4EyYImZzXXOrWmz241AjXNukJlNB+4DpkWiYC855wgEHf5gy+eAozEQpCkQpNEfpDEQpKEpSIM/wMGmAAcbmz/XNfjZ3+BnX33zx54Djew52ET1/kaq9jVQtb+RRn/wU9+ra1YahXk5TDrjJIae3JUzTu7C0JO7aLFNkQho5zgnSW7Oe6Xc+eJqhvftyq2XFHpdjsSQ9hyZGgtscs59DGBmTwNTgLaDzBRgRsvl54AHzMycc66jBW6rPsBXZy8BmpuZw7nPXGi+6Jxr+Qyu5Ubnmj847PbgocuOoGteoynoHMGgI+AcwSAEWhqpjjCDnIxUumal0T07ne6d0hmUl0NuTgYndcmkT9dM8rtn069HFl2z0nTaTiR62jPOnZBbn1rGuh17O3o34rFA0PFx1X4uHdqb308fQXa6ZsnIJ9rz29AX2Nbmehlw9tH2cc75zawW6AlUtd3JzG4CbgLo379/uwrMSE3h9N6d29zJZ/dp3dS2+TCamxdr2W5tbjCszW2QYta8j4HPjBSDlBQjxQzfoc/gS0nBZ0aqz0hNMdJ8KaT5mj+npzZ/ZKT6yExLISvNR2aaj6x0HzkZqWSn++iUnqrVx0Vi0zHHuRMZvwD6dc8iEAwee0eJeZ8fnc9/XnSqplfIZ0S1tXbOzQJmARQVFbXrME9el0we/NKoiNYlInIsJzJ+AXxv0uCI1SQisaE9i2OUA/3aXM9v2XbEfcwsFehK80R0EZF40J5xTkTkiNrTTC0BCs2swMzSgenA3MP2mQtc33L5C8DCcMyXEhGJkvaMcyIiR3TM03wtc6BuAV6l+S3DjznnVpvZXUCxc24u8CjwuJltAqppHohEROLC0cY5j8sSkTjRrjlTzrn5wPzDtv20zeV64NrwliYiEj1HGudERNpDgUIiIiIiHaBmSkRERKQD1EyJiIiIdICaKREREZEOMK9WMDCzSmDLcXxJLoetqJ6AkuExgh5nIjnexzjAOdcrUsVEi8avo0qGx5kMjxH0OI/kqOOXZ83U8TKzYudckdd1RFIyPEbQ40wkyfAYwyFZfk7J8DiT4TGCHufx0mk+ERERkQ5QMyUiIiLSAfHUTM3yuoAoSIbHCHqciSQZHmM4JMvPKRkeZzI8RtDjPC5xM2dKREREJBbF05EpERERkZijZkpERESkA2K6mTKza81stZkFzazosNt+YGabzGy9mV3mVY3hZmYzzKzczJa3fFzhdU3hYmaTWp6vTWZ2h9f1RIqZlZrZypbnr9jresLFzB4zswozW9VmWw8ze83MNrZ87u5ljbEm2cawRB6/QGNYPIv0+BXTzRSwCvgXYFHbjWY2FJgODAMmAf9jZr7olxcx/885N6LlIyFS7FuenweBy4GhwBdbnsdEdXHL85dI67TMpvnvra07gAXOuUJgQct1+UQyjmEJN36BxrAEMJsIjl8x3Uw559Y659Yf4aYpwNPOuQbn3GZgEzA2utXJcRoLbHLOfeycawSepvl5lDjhnFsEVB+2eQowp+XyHOCaqBYV4zSGJRSNYXEs0uNXTDdTIfQFtrW5XtayLVHcYmYfthyWTJTTJon+nLXlgP8zsxIzu8nrYiKst3NuR8vlnUBvL4uJI4n895CI4xck9nN2uGQZw8I2fqWGp54TZ2avAycd4aYfOef+Ee16oiHUYwb+F5hJ8y/zTOA3wNeiV52EwfnOuXIzywNeM7N1La+KEppzzplZ0q21kmxjmMavpJB0Y1hHxy/Pmynn3IQT+LJyoF+b6/kt2+JCex+zmT0MzItwOdES18/Z8XDOlbd8rjCzv9N8eiBRB6JdZtbHObfDzPoAFV4XFG3JNoYl6fgFcfycHa8kGsPCNn7F62m+ucB0M8swswKgEPjA45rCouUJbTWV5gmsiWAJUGhmBWaWTvPk27ke1xR2ZtbJzDq3XgYmkjjP4ZHMBa5vuXw9kHBHYiIkIcewBB6/QGNYIgrb+OX5kalQzGwqcD/QC3jJzJY75y5zzq02s2eANYAfuNk5F/Cy1jD6pZmNoPkweSnwH96WEx7OOb+Z3QK8CviAx5xzqz0uKxJ6A383M2j++3rSOfeKtyWFh5k9BYwDcs2sDPgZcC/wjJndCGwBrvOuwtiThGNYQo5foDHM25I6LtLjl+JkRERERDogXk/ziYiIiMQENVMiIiIiHaBmSkRERKQD1EyJiIiIdICaKREREZEOUDMlIiIi0gFqpkREREQ64P8DUhtX1HfheXwAAAAASUVORK5CYII=",
            "text/plain": [
              "<Figure size 720x288 with 2 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          },
          "output_type": "display_data"
        }
      ],
      "source": [
        "##適当な区間のxの値を用意する\n",
        "xp = np.linspace(-10.0,10.0,100)\n",
        "yp_sigmoid = sigmoid(xp)\n",
        "yp_relu = relu(xp)\n",
        "\n",
        "fig = plt.figure(figsize=(10,4))\n",
        "ax1 = fig.add_subplot(121)\n",
        "ax1.plot(xp,yp_sigmoid,label=\"Sigmoid\")\n",
        "ax1.legend()\n",
        "ax2 = fig.add_subplot(122)\n",
        "ax2.plot(xp,yp_relu,label=\"ReLU\")\n",
        "ax2.legend()\n",
        "plt.show()\n",
        "plt.close()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f9ELoAn8hYNr"
      },
      "source": [
        "sigmoid関数はx->+∞で1.0, x=-∞で-1.0に漸近します。\n",
        "\n",
        "一方でReLU関数はx=0までは0.0で、x>0.0で、xとなるような関数です。\n",
        "\n",
        "\n",
        "なぜReLUがよく使われる様になったかと言うと、  \n",
        "(特に隠れ層の数が多い深層学習において)学習するにつれて勾配の値が小さくなって学習が進まない、\n",
        "という問題を解決するためです。\n",
        "\n",
        "それぞれの関数の微分を表示してみると"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 159,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 265
        },
        "id": "dj-P7bDfvYpr",
        "outputId": "c8bf0288-f777-4bb1-efb2-d1546c867027",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlkAAAD4CAYAAADfJ/MlAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deXyc9XXv8c/RaF9s2dbiRbZlyza2wWDjJSEQMIkhhiQQUiikSQuFV2jTkt6Wm9ukoU24JG1p0tyb5JYGSEIgaYDsxElJSNgJqxdswDYGyXiRvEmWbGuxlpn53T9mRh7LI3kkzcwzy/f9egnNPPPMzBnP8NOZ8zvP7zHnHCIiIiKSWHleByAiIiKSjZRkiYiIiCSBkiwRERGRJFCSJSIiIpIESrJEREREkiDf6wCGqqqqcvX19V6HISIptHHjxjbnXLXXcSSCxjCR3DLS+JV2SVZ9fT0bNmzwOgwRSSEz2+11DImiMUwkt4w0fmm6UERERCQJlGSJiIiIJIGSLBEREZEkSLueLJFMMzAwQHNzM729vV6HkvaKi4upq6ujoKDA61BSKtc+I7n6PosMpSRLZJyam5upqKigvr4eM/M6nLTlnOPw4cM0NzczZ84cr8NJqVz6jOTy+ywyVFzThWa21sx2mFmjmX0uxu23mtk2M3vNzJ4ws9lRtwXMbHP4Z10igxdJB729vUyZMiXr/3iOl5kxZcqUtKnmmNl9ZnbIzN4Y5nYzs2+Gx73XzOzcsT5XLn1G0u19FvHSaZMsM/MBdwGXAYuBj5nZ4iG7vQqscM6dDfwU+ErUbcedc0vDP1ckKG6RtJILfzwTIc3+ne4H1o5w+2XA/PDPzcC3xvNkafbakyqXXqvISOKZLlwFNDrndgKY2cPAlcC2yA7Ouaei9n8J+EQig5Ts9tbBTn69ZR8AVRVFfGzVLAp8OiZDkss596yZ1Y+wy5XA951zDnjJzCrNbJpzbn9KApSM89+v7WfHgWNehyEJMK+2givOmT7ux4knyZoB7I263gy8a4T9bwJ+E3W92Mw2AH7gTufcI0PvYGY3E/qmyKxZs+IISbLF8f4AN9z3CvuO9mIGzkHfQJBPXjjX69Ayyj//8z/z4IMP4vP5yMvL45577uHb3/42t956K4sXDy08J87ll1/Ogw8+SGVl5Unbb7/9dsrLy/nMZz6TtOdOgVhj3wzglCQrE8Ywn8/HkiVL8Pv9zJkzhx/84AenvG/RYr2HN9xwAx/60Ie4+uqrB7eVl5fT1dWV1Ngzxd//dAvd/QFUyMt8l501NWVJVtzM7BPACuCiqM2znXMtZjYXeNLMXnfONUXfzzl3L3AvwIoVK1wiY5L09q1nmth3tJcf/8V5rJozmRvvX883nnibK5dNp6ai2OvwMsKLL77Ir3/9azZt2kRRURFtbW309/fzne98J+nP/eijjyb9OTJBJoxhJSUlbN68GYDrr7+eu+66i9tuu83jqLLLQMDxlxc18LnLFnodiqSJeOZkWoCZUdfrwttOYmZrgNuAK5xzfZHtzrmW8O+dwNPAsnHEK1lkb3sPdz/TxBXnTGfVnMkA/NOHFtPnD/CV3+7wOLrMsX//fqqqqigqKgKgqqqK6dOns3r16sHTu3z3u99lwYIFrFq1ik9+8pPccsstQKgy8alPfYp3v/vdzJ07l6effpobb7yRRYsWccMNNww+x0MPPcSSJUs466yz+OxnPzu4vb6+nra2NiBUTVuwYAEXXHABO3ZkxfsX19iXic477zxaWkIvpampibVr17J8+XLe+9738uabb3ocXeYKOIc6HSRaPJWs9cB8M5tDaIC5DviT6B3MbBlwD7DWOXcoavskoMc512dmVcD5nNwULznsXx7djs+Mf7j8xLe+OVVl3HTBXO5+pomPv2sWy2ZN8jDC0fvfv9rKtn2J7clYPH0CX/zwmcPefumll3LHHXewYMEC1qxZw7XXXstFF50oJu/bt48vfelLbNq0iYqKCt73vvdxzjnnDN7e0dHBiy++yLp167jiiit4/vnn+c53vsPKlSvZvHkzNTU1fPazn2Xjxo1MmjSJSy+9lEceeYSPfOQjg4+xceNGHn74YTZv3ozf7+fcc89l+fLlCf138MA64JZwH+q7gKOJ6Mfy4jMSLRAI8MQTT3DTTTcBcPPNN3P33Xczf/58Xn75Zf7qr/6KJ598MqHx5YpA0OHTXKFEOW2S5Zzzm9ktwGOAD7jPObfVzO4ANjjn1gFfBcqBn4SPKtkTPpJwEXCPmQUJVc3udM5ti/lEklMOHuvlN28c4NPvm8e0iSUn3XbL++bx0Ct7+MGLuzMuyfJCeXk5Gzdu5LnnnuOpp57i2muv5c477xy8/ZVXXuGiiy5i8uRQtfCaa67hrbfeGrz9wx/+MGbGkiVLqK2tZcmSJQCceeaZ7Nq1i927d7N69Wqqq0Mnmf/4xz/Os88+e1KS9dxzz3HVVVdRWloKwBVXpP+BxGb2ELAaqDKzZuCLQAGAc+5u4FHgcqAR6AH+3JtIE+P48eMsXbqUlpYWFi1axCWXXEJXVxcvvPAC11xzzeB+fX19wz5GrKMGdSRhSDAYmiXOy9O/h5wQV0+Wc+5RQgNO9LYvRF1eM8z9XgCWjCdAyU6Pbz8IELOxsLwon/cvrOGpHYfwB4LkZ1D9Pd5qQqL5fD5Wr17N6tWrWbJkCQ888EDc941MM+bl5Q1ejlz3+/1Zu2q3c+5jp7ndAX+d6Of16jMS6cnq6enhAx/4AHfddRc33HADlZWVg71apzNlyhQ6OjoGr7e3t1NVVZWskDNKwIWSLFWyJFrm/PWSrPL4toPMnlLKvJrymLevWVxLR88Am/YcSXFkmWfHjh28/fbbg9c3b97M7NmD6wGzcuVKnnnmGTo6OvD7/fzsZz8b1eOvWrWKZ555hra2NgKBAA899NBJ05EAF154IY888gjHjx+ns7OTX/3qV+N7UZI0paWlfPOb3+RrX/sapaWlzJkzh5/85CdAaLX2LVu2DHvf1atX86Mf/Yj+/n4A7r//fi6++OKUxJ3uAqpkSQw6rY6kXHefn+ebDvOn75497FTDhQuqKfTl8fj2g4NN8RJbV1cXn/70pzly5Aj5+fnMmzePe++9d/Aw+xkzZvD5z3+eVatWMXnyZBYuXMjEiRPjfvxp06Zx5513cvHFF+Oc44Mf/CBXXnnlSfuce+65XHvttZxzzjnU1NSwcuXKhL5GSaxly5Zx9tln89BDD/HDH/6QT33qU3z5y19mYGCA6667brBn78tf/jJf//rXB+/X3NzMxo0bWb58OT6fj4aGBu6++26vXkZaCUYqWUqyJIo5l15HG69YscJFjoiS7PTbNw7wl/+1kYc++W7Oa5gy7H5/dt8rNLf38ORnVqcuuDHYvn07ixYt8jqMEXV1dVFeXo7f7+eqq67ixhtv5KqrrvIkllj/Xma20Tm3wpOAEizWGJYJn5FEy7XX3Nk7wJLbf8dtly/SOn85ZqTxS9OFknKPbz/IxJICVtSP3NR+yaIadrZ109SqhQ7H6/bbb2fp0qWcddZZzJkz56SmdREZv2Aw9FvThRJN04WSUoGg48k3D3HxGdWnPXXO+xfV8k+/3Mrj2w7ScFHs3i2Jz7//+797HYJIVjvR+O5xIJJWVMmSlNq8t4P27n7WLK497b7TK0s4c/qEwSMR01m6Tbunq1z+d8ql155LrzUi0viuniyJpiRLUuqVd0KHf7+nIb7Dvs+fV8WWvUfp8weSGda4FBcXc/jw4Zz8wzIazjkOHz5McXHunS4plz4jufo+RxrfNV0o0TRdKCm1eW8H9VNKmVxWGNf+y2ZWcm8gyPb9nSydOfzJbL1UV1dHc3Mzra2tXoeS9oqLi6mrq/M6jJTLtc9ILr7Pg5UsrZMlUZRkSco459i05wgXzIt/8cLIiu+v7ulI2ySroKCAOXPmeB2GpDF9RrKf1smSWDRdKCmz72gvrZ19o0qWpk4sZuqEYl7VoqQiksaCWvFdYlCSJSmzOZwoLZs1uorUslmVbN6rJEtE0pca3yUWJVmSMq/u6aAwP4+FUyeM6n5LZ1ayp72Hw13Dn7hWRMRLanyXWJRkScps3nuEJTMmUpg/uo9dpC9L1SwRSVeB8GKkmi6UaEqyJCX6/UFebznKsjE0ry+ZMRFfnqkvS0TS1onpQo8DkbSij4OkxJsHjtHnD7J0lP1YACWFPhZOreDVvR1JiExEZPwGpwtVyZIoSrIkJSJTfZGpv9FaNquSLXuPDn5bFBFJJ2p8l1iUZElKbN57hOqKIqZPHNsq0EtnTqKrz89OnSxaRNJQQI3vEoOSLEmJ7fs7WTxtAjbGUvriaaEjErcf6ExkWCIiCRHUiu8Sg5IsSbqBQJDGQ50snFYx5sdoqCkjP894c/+xBEYmIpIYmi6UWJRkSdK909bNQMCxaJTrY0UryvfRUF3ODlWyRCQNBdT4LjEoyZKk2x6uPp0xdeyVrMj931SSJSJpKBhZJ0uVLImiJEuS7s0DneTnGQ3V5eN6nIXTKmg5cpyjxwcSFJmISGJEKllaJ0ui6eMgSbfjQCfzaspHvdL7UJHpxrcOqpolIukl0viu6UKJpiRLku7N/cfGPVUIJ6Yb1fwuIulGje8Si5IsSaqjxwfYd7R31CeFjmXaxGImFOerL0tE0o4a3yUWJVmSVJGjARcmoJJlZiycOkFJloiknaAqWRKDkixJqjcPhKb2xrNGVrSF0yrYcaAT53R6HRFJHyca35VkyQlKsiSp3jzQycSSAqZOGNvpdIZaOHUCXX1+mjuOJ+TxREQSIaDGd4lBSZYkVaTpfayn0xlqsPldU4YikkaCqmRJDEqyJGmcc7x1sCsh/VgROsJQRNJRILIYqSpZEiWuJMvM1prZDjNrNLPPxbj9VjPbZmavmdkTZjY76rbrzezt8M/1iQxe0tv+o7109fmZX5u4JKu8KJ8ZlSU0tnYl7DFFRMZrcJ0slS4kymk/DmbmA+4CLgMWAx8zs8VDdnsVWOGcOxv4KfCV8H0nA18E3gWsAr5oZpMSF76ks8ZDoURo3jhXeh+qoaZ88LFFxiOOL5CzzOwpM3s1/CXyci/ilPSnxneJJZ6cexXQ6Jzb6ZzrBx4GrozewTn3lHOuJ3z1JaAufPkDwO+dc+3OuQ7g98DaxIQu6W4wyapJbJI1r7qcptauwW+OImMR5xfIfwR+7JxbBlwH/Gdqo5RMMbgYqaYLJUo8SdYMYG/U9ebwtuHcBPxmNPc1s5vNbIOZbWhtbY0jJMkETa1dTCwpoKq8MKGPO6+mnN6BIC1HdIShjMtpv0ACDoispDsR2JfC+CSDRBrf81TJkigJnT02s08AK4CvjuZ+zrl7nXMrnHMrqqurExmSeKjxUBcN1WUJO7IwoqG6LPT46suS8YnnS+DtwCfMrBl4FPh0rAfSF0VRJUtiiSfJagFmRl2vC287iZmtAW4DrnDO9Y3mvpKdmlq7Ej5VCCemH5vUlyXJ9zHgfudcHXA58AMzO2Xc1BdFGVwnS5UsiRJPkrUemG9mc8yskFBfwrroHcxsGXAPoQTrUNRNjwGXmtmkcMP7peFtkuWO9PTT1tWflCRrSnkRk0oL1Pwu4xXPl8CbgB8DOOdeBIqBqpREJxlF62RJLKdNspxzfuAWQsnRdkJNoFvN7A4zuyK821eBcuAnZrbZzNaF79sOfIlQorYeuCO8TbJcspreI+bpCEMZv9N+gQT2AO8HMLNFhJIszQfKKbROlsSSH89OzrlHCfUjRG/7QtTlNSPc9z7gvrEGKJnpxPINiVsjK9q8mnJ++8aBpDy25AbnnN/MIl8gfcB9kS+QwAbn3DrgfwLfNrO/I9QEf4PTiTMlhkAwlGVpnSyJFleSJTJajYe6KMrPY8akkqQ8fkN1OR09Axzu6mNKeVFSnkOyXxxfILcB56c6Lsk8qmRJLMq5JSkaW7uYW12etP6EyDSkpgxFJB1oMVKJRUmWJEWyjiyMGEyytIyDiKSBYNBhRsKXrJHMpiRLEq53IEBzx/GEn04n2vSJJZQU+FTJEpG0EHBOU4VyCiVZknBNrV04l7wjCyG0Fk1DTZmSLBFJC8Gg0xpZcgolWZJwTa3dADTUlCX1eRqqy9kZfi4RES8FgqpkyamUZEnC7WztwgzqpyQ3yZpbVU7LkeMc7w8k9XlERE4n4Jya3uUUSrIk4Xa2djOjsoTiAl9Sn2du+ByG77SpmiUi3goGHcqxZCglWZJwO9tCyzckWyTJ2tmmviwR8ZYqWRKLkixJKOcc77R2M7cquVOFAHPCz6G+LBHxWiCoNbLkVEqyJKEOHuujuz9AQ3Xyk6zSwnymTyxmp9bKEhGPhaYLlWTJyZRkSUJFEp5UTBdGnmenerJExGOaLpRYlGRJQjUNJlnJr2RFnmdnazc6Z6+IeEmVLIlFSZYkVFNrN6WFPqZOKE7J882tKqOrz09rZ19Knk9EJBZVsiQWJVmSUDvbuplTVZay83dFpiWb1PwuIh4KBJVkyamUZElC7WxNzfINEVrGQUTSQdBpnSw5lZIsSZjegQAtR46nZPmGiOkTSyguyNMyDiLiqUDQkZ+nP6lyMn0iJGF2He7GudQ1vUPoRNH1U8q0jIOIeCoQRCeIllMoyZKEiVSTGlI4XRh5Pi3jICJeCjqHT39RZQh9JCRhItWkOSmcLoRQ5Wxvew99fp0oWkS8EQg6fFrCQYZQkiUJs7O1m6kTiikryk/p886tLiPoYM/hnpQ+r4hIRNA5TRfKKZRkScI0tXWntB8rYm6VlnEQEW+pkiWxKMmShAidGLrLmyQr/JzvqC9LRDwSCKqSJadSkiUJcbi7n2O9/sGqUipVFBdQXVGkIwxFxDNBp0qWnEpJliRE5MjCOR5UsiB0eh0dYSgiXtGK7xKLkixJiEgVqcGDShaETq+jSpaIeCXgtE6WnEpJliTEzrZuCvPzmDGpxJPnb6guo6NngI7ufk+eX0RyWzDo8CnHkiGUZElC7Gzton5KqWflcp3DUES8pOlCiUVJliTEztZuT5reI7SMg4h4KXSCaCVZcjIlWTJuA4Ege9p7PFm+IaJuUgkFPtOJokXEE6pkSSxKsmTc9rT34A865qb4nIXR8n15zNaJokXEIwGt+C4xxJVkmdlaM9thZo1m9rkYt19oZpvMzG9mVw+5LWBmm8M/6xIVuKSPSPXIy0oWhM6ZqGUcZDRON7aF9/ljM9tmZlvN7MFUxyiZIagV3yWG055kzsx8wF3AJUAzsN7M1jnntkXttge4AfhMjIc47pxbmoBYJU15vXxDxNzqMp7ecQh/IEi+T0VaGVk8Y5uZzQf+ATjfOddhZjXeRCvpLuA0XSiniucv0Sqg0Tm30znXDzwMXBm9g3Nul3PuNSCYhBglzb3T1s2UskImlhZ4GkdDVTkDAUfLkeOexiEZ47RjG/BJ4C7nXAeAc+5QimOUDBEMosZ3OUU8SdYMYG/U9ebwtngVm9kGM3vJzD4Sawczuzm8z4bW1tZRPLSkg52t3pwYeqjBZRzU/C7xiWdsWwAsMLPnw2PY2lgPpDFMQo3vXkch6SYVH4nZzrkVwJ8AXzezhqE7OOfudc6tcM6tqK6uTkFIkkg727qYU5UOSVZkGQc1v0vC5APzgdXAx4Bvm1nl0J00hommCyWWeJKsFmBm1PW68La4OOdawr93Ak8Dy0YRn6S5Iz39tHX10+DhkYURk8sKmVRaoLWyJF7xjG3NwDrn3IBz7h3gLUJJl8hJgkGtkyWniifJWg/MN7M5ZlYIXAfEdZSgmU0ys6Lw5SrgfGDbyPeSTBJJaObVeJ9kQSgOVbIkTvGMbY8QqmJFxrAFwM5UBimZQZUsieW0SZZzzg/cAjwGbAd+7JzbamZ3mNkVAGa20syagWuAe8xsa/jui4ANZrYFeAq4c8hRiZLhmg6FjyxMg0oWhOKIxCQyknjGtvBth81sG6Ex7H855w57E7Gks4AqWRLDaZdwAHDOPQo8OmTbF6IurydUah96vxeAJeOMUdJYU2sXhb486jw6MfRQDdXlPNy9l47ufiaVFXodjqS5OMY2B9wa/hEZVlArvksMOhZCxqWpNdT0ni7rUkWmLXWiaBFJJU0XSizp8ZdRMlbjoS4aarw/sjAiMm3ZqClDEUkhrZMlsSjJkjHr8wfY097DvDTpxwKYMamEwvw8HWEoIikVqmR5HYWkG30kZMx2H+4h6KAhTY4sBPDlGXOrytT8LiIpFdC5CyUGJVkyZul2ZGFEg5ZxEJEUCgYdAHnqyZIhlGTJmEX6ntLhlDrRGqrL2dPeQ+9AwOtQRCQHBFwoyVIlS4ZSkiVj1tTaxYzKEkoL41oJJGUaqssIutB0pohIsgVUyZJhKMmSMWtKkxNDDxVZxkFThiKSCsFIJUtJlgyhJEvGJBh0NLV2pc3pdKLNrdIyDiKSOpFKlqYLZSglWTImB4710tMfSLumd4CSQh8zKktUyRKRlAgGQ781XShDKcmSMWlM0yMLIxpqylXJEpGUONH47nEgknaUZMmYvHWwE4D5temZZM0PJ1mRMr6ISLIMTheqkiVDKMmSMXn7YBdTygqpKi/yOpSYzqitoM8fZG+7jjAUkeSKNL5rulCGUpIlY/LWoc60rWLBiQpbpOImIpIsanyX4SjJklFzztF4sIsFtRVehzKs+eHY3lZflogkmdbJkuEoyZJR23+0l84+/2Aik47Ki/KZUVmiSpaIJF1QK77LMJRkyahFEpcFabhGVrT5teW8dVCVLBFJLjW+y3CUZMmovR1OXNJ5uhBC8TW16ghDEUkuNb7LcJRkyai9dbCTqvIiJpUVeh3KiObXlNPvD7L7cLfXoYhIFguEFyPVdKEMpSRLRu2tQ10sSOMjCyMilTZNGYpIMp2YLvQ4EEk7+kjIqISOLOxM+6lCOHGi6LfV/C4iSTQ4XahKlgyhJEtGpeXIcbr7A2m9RlZEWVE+dZNKeEvLOIhIEvnV+C7DUJIlo5IpTe8RC2orVMkSkaTSOlkyHCVZMionlm/IjCRrfm05O1u78Uc6U0VEEkzrZMlwlGTJqOw42ElNRRETSwu8DiUuZ9RW0B8IsktHGIpIkmidLBmOkiwZle37O1k4bYLXYcRt4dRQrNv2a8pQRJIjGFTju8SmJEvi1u8P0niok0XTMmOqEEJHGBb4jO37j3kdiohkqYBTJUtiU5IlcWtq7WIg4FicQZWswvw8GqrLlWSJSNJonSwZjj4SErdIorIog5IsgMXTJijJEpGk0TpZMhwlWRK37fuPUZifx9yqMq9DGZVF0yZw8Fgf7d39XociIllo8LQ6mi6UIZRkSdy27+9kQW05+RlWE49U3lTNEpFkCKjxXYYR119LM1trZjvMrNHMPhfj9gvNbJOZ+c3s6iG3XW9mb4d/rk9U4JJazjm27z/GoqmZNVUIDDbqK8mSoU43tkXt90dm5sxsRSrjk8wQVOO7DOO0SZaZ+YC7gMuAxcDHzGzxkN32ADcADw6572Tgi8C7gFXAF81s0vjDllRr7ezjcHd/xvVjAUwpL6KmoohtSrIkSpxjG2ZWAfwP4OXURiiZQutkyXDiqWStAhqdczudc/3Aw8CV0Ts453Y5514Dhi6r/QHg9865dudcB/B7YG0C4pYU25ahTe8Ri6ZNYLvWypKTnXZsC/sS8G9AbyqDk8yhxncZTjxJ1gxgb9T15vC2eMR1XzO72cw2mNmG1tbWOB9aUimSoGTS8g3RFk2bQOOhTvr9Or2ODDrt+GRm5wIznXP/PdIDaQzLbapkyXDSooPZOXevc26Fc25FdXW11+FIDNv3H2P6xOKMOZ3OUIumVTAQcDS1dnkdimQIM8sD/g/wP0+3r8aw3DaYZKmSJUPEk2S1ADOjrteFt8VjPPeVNLJ9/7GMnSqEExW4bfvUlyWDTjc+VQBnAU+b2S7g3cA6Nb/LUIPThWlRtpB0Es9HYj0w38zmmFkhcB2wLs7Hfwy41MwmhRveLw1vkwxyvD/AzrZuFk/P3CRrTlUZxQV5bFWSJSeMOLY5544656qcc/XOuXrgJeAK59wGb8KVdBVZJytfWZYMcdpPhHPOD9xCKDnaDvzYObfVzO4wsysAzGylmTUD1wD3mNnW8H3bCTWNrg//3BHeJhlk676jBIKOs+sqvQ5lzPJ9eZw5fSKvtxzxOhRJE/GMbSLxCKiSJcPIj2cn59yjwKNDtn0h6vJ6QqX2WPe9D7hvHDGKx15rPgrA2XUTPY5kfJbMmMiP1u/FHwhm3IKqkhynG9uGbF+dipgk8wTVkyXD0F8aOa3Xmo9QO6GI2gnFXocyLufMnMjxgQBNrd1ehyIiWURHF8pwlGTJab3WcjSjpwojlswIvYYtzZoyFJHEOdH4riRLTqYkS0Z0rHeAna3dnD0js6cKAeZWlVFelM/r4elPEZFE0BIOMhwlWTKiN1rC/VgzM7+SlZdnnDVjAq+pkiUiCRTQuQtlGEqyZESRpvclWVDJAjinrpLt+7Xyu4gkTqTxXafVkaGUZMmIXm8+yszJJUwuK/Q6lIRYUjeR/kCQtw7qPIYikhiRdbJUyZKhlGTJiLY0H+HsGZk/VRhxTp2a30UksQbXyVKOJUMoyZJhtXf309xxPOPXx4pWN6mEytICNb+LSMIEg448A9N0oQyhJEuGFan2LMmiJMvMOLuuks17VckSkcQIOKepQolJSZYMa+OuDnx5xtIsOLIw2rmzKtlxsJNjvQNehyIiWSBUyVKSJadSkiXDWr+rnTOnT6C0MK6zL2WMlfWTcQ427e7wOhQRyQKBoCpZEpuSLImp3x9kS/MRls+e5HUoCbd0ZiW+PGOjkiwRSYCAc1qIVGJSkiUxbd13lN6BICvrJ3sdSsKVFeWzeNoE1u9q9zoUEckCwaDTKXUkJiVZElOkyrMiCytZAMtnT2Lz3iMMBLQoqYiMjxrfZThKsiSm9bvamTW5lJoJxV6HkhQr6yfTOxBk675jXociIhkuENRq7xKbkiw5hXOOjbs7sraKBbCiPvTaNmjKUETGKRh0+PTXVGLQx0JOsetwD21d/Syvz94kq3ZCMXWTStiwS83vIjI+anyX4SjJkmFONSwAABPWSURBVFNEqjvZ2PQebWX9ZDbs7sCFT4khIjIWanyX4SjJklO88k47E0sKmFdd7nUoSbWifhJtXX2809btdSgiksHU+C7DUZIlJ3HO8XxjG+fNnZL138ze01AFwPNNhz2OREQyWSCo6UKJTUmWnGTX4R72He3l/PlVXoeSdPVTSplRWcLzb7d5HYqIZLCg03ShxKYkS07yh7dbAbhgXvYnWWbG+fOm8EJTG4Gg+rJEZGxUyZLhKMmSk/yhsY0ZlSXUTyn1OpSUuGB+Ncd6/bzectTrUEQkQwWCqJIlMSnJkkGBoOOFpsNcMK8Ky5FvZe9pmALA842aMhSRsQk6rZMlseljIYNeaz5CZ6+fC3KgHyuiqryIxdMm8Fx4mlREZLQ0XSjDUZIlgyLVnEh1J1dcML+KTbuP0NPv9zoUEclAanyX4SjJkkF/aGxj8bQJTCkv8jqUlDp/XhX9gSDrtfq7iIyBKlkyHCVZAsCx3gE27u7gvQtyZ6owYlX9ZIry83jqzUNehyIiGSigFd9lGEqyBICnd7QyEHBcsqjW61BSrqTQxwXzqvj9toM6xY6IjFpQ5y6UYSjJEgB+t/UAVeWFLJuVvSeFHsmlZ9bScuQ42/Yf8zoUEckwgaBOqyOxxZVkmdlaM9thZo1m9rkYtxeZ2Y/Ct79sZvXh7fVmdtzMNod/7k5s+JII/f4gz+xo5f0La3N2oHjfwlrM4PfbDnodiqRQHGPbrWa2zcxeM7MnzGy2F3FKegs4rZMlsZ02yTIzH3AXcBmwGPiYmS0esttNQIdzbh7wf4F/i7qtyTm3NPzzlwmKWxLopZ2H6ezzc+mZuTdVGFFdUcTyWZOUZOWQOMe2V4EVzrmzgZ8CX0ltlJIJgkGHTzmWxBBPJWsV0Oic2+mc6wceBq4css+VwAPhyz8F3m+5spplFvjdtgOUFPg4PwdOpTOSSxbXsnXfMZo7erwORVLjtGObc+4p51zkA/ESUJfiGCUDaLpQhhNPkjUD2Bt1vTm8LeY+zjk/cBSILLY0x8xeNbNnzOy9sZ7AzG42sw1mtqG1VYtCplIw6Hh82yEuWlBNcYHP63A8demZUwF4XNWsXBHP2BbtJuA3sW7QGJbbgs6Rp7qCxJDsxvf9wCzn3DLgVuBBM5swdCfn3L3OuRXOuRXV1dVJDkmivdZylAPHerlkce5OFUbMqSpjXk05v916wOtQJM2Y2SeAFcBXY92uMSy3qZIlw4knyWoBZkZdrwtvi7mPmeUDE4HDzrk+59xhAOfcRqAJWDDeoCVxHnm1hcL8PNYoyQLgg0um8fI77ew/etzrUCT54hnbMLM1wG3AFc65vhTFJhkkoBXfZRjxJFnrgflmNsfMCoHrgHVD9lkHXB++fDXwpHPOmVl1uLkUM5sLzAd2JiZ0Ga+BQJBfbdnHJYtqmVhS4HU4aeGj587AOXjk1X1ehyLJd9qxzcyWAfcQSrC0Wq3EFNSK7zKM0yZZ4R6rW4DHgO3Aj51zW83sDjO7Irzbd4EpZtZIaFowcij0hcBrZraZUEP8Xzrn2hP9ImRsnn2rlcPd/Vy1bKQ2lNwye0oZy2dP4uebmrUwaZaLc2z7KlAO/CS8DM3QL5giBJymCyW2/Hh2cs49Cjw6ZNsXoi73AtfEuN/PgJ+NM0ZJkp9vamFyWSEXnaEekmhXLZvBPz7yBlv3HeOsGRO9DkeSKI6xbU3Kg5KMEwyixneJSSu+56ijxwf4/faDfPjsaRT49DGI9qGzp1Hoy+Pnm05pzxEROUWo8d3rKCQd6WORo37z+n76/UE+eq6W/RmqsrSQ9y2sYd2WFvyBoNfhiEia03ShDEdJVg5yzvHDl/cwr6acs+s0HRbL1cvraOvq1wrwInJawaDWyZLYlGTloE17jvB6y1Guf089Wpg/tosX1lA3qYTvvbDL61BEJM35tU6WDENJVg66/4VdVBTn81EdVTgsX55x/Xn1vPJOO1v3HfU6HBFJY6pkyXCUZOWYA0d7+c3r+7lu5UzKiuI6uDRn/fHKmZQU+Lj/+V1ehyIiaUw9WTIcJVk55r9e2k3AOf7svHqvQ0l7E0sK+KPlM/jlln0c7tJC3yISm06rI8NRkpVDuvr8/PDl3bx/YS0zJ5d6HU5GuOE99fT7g3z/xd1ehyIiaUoniJbhKMnKIQ+8sIuOngFued88r0PJGPNqKvjAmbXc94d3ONLT73U4IpKGtE6WDEcfixxxrHeAe5/dyfsX1rB0ZqXX4WSUv7tkAV39fr79nE67KSInc84RdOjchRKTkqwc8d3n3uHo8QH+7pIFXoeScRZOncAHl0zje8/vUm+WiJwkGD7FaZ56siQGJVk5oKO7n/v+8A5rz5yqc/GN0d+uWUDvQIC7n2nyOhQRSSOBcJalSpbEoiQrB3zlsR30DAS49VJVscZqXk05f3RuHfe/sIvGQ11ehyMiaSLoQkmWKlkSi5KsLLdpTwcPr9/Dn7+nngW1FV6Hk9E+e9lCSgp8/NMjb+DCA6uI5LbBSpaSLIlBSVYW8weC/OMv3qC2opi/VS/WuFWVF/H3axfy4s7D/HLzPq/DEZE0EHCaLpThKcnKYve/sItt+4/xhQ8vplyruyfEn6yaxTkzK/nyf2+jvVtLOojkumBQ04UyPCVZWeqNlqN85bc7WLOolsvOmup1OFkjL8/416uWcOy4n//1ky2aNhTJcSca3z0ORNKSkqws1NXn55YHNzG5rJCvXn02pjJ2Qi2ePoHPX76QJ948xPd0XkORnDY4XahKlsSgJCvLOOe47Revs6e9h29ct5RJZYVeh5SVrn9PPWsW1fKvv9nOlr1HvA5HRDwSDIZ+a7pQYlGSlWW+/vjb/HLzPm69ZAHvmjvF63Cylpnx1avPpqaimJse2MDe9h6vQxIRD6jxXUaiJCuLPPzKHr7xxNtcs7yOv75Y5ydMtkllhTxw40oGAkGu/94rdKgRXiTnBLWEg4xASVaWWLdlH7c98gYXLajmXz66RH1YKTKvpoLvXL+C5o7j3KBESyTnaJ0sGYmSrCzwXy/t5n88/CrLZ0/iro+fS4FOB59SK+sn862Pn8v2A51ce++LHDzW63VIIpIianyXkeivcQYLBB3/53c7+MdH3uDiM2r4/o2rtB6WR96/qJb7/3wlLR3HufruF9i+/5jXIYlICgyuk6XZA4lBSVaGau/u54bvvcI3n2zkmuV13POnyyku8HkdVk57T0MVD37y3fT7g1z1n8/z043NXockIkmmSpaMRElWBnps6wEu+8azvPxOO//60SV85eqzNUWYJs6ZWcmvP/1els2cxGd+soVbHtxEa2ef12GJSJIEVMmSEWhuKYPsPtzNvzy6nce2HmTh1Aq+e/1Kzpox0euwZIjqiiJ+cNMq/vPpJv7jyUaee7uNv197Bn+8YqaSYZEsE1knS5UsiUVJVgZo7ujhW0838aP1e8n3GX+/9gw++d65+oOdxvJ9efzN++dz+ZKpfP7nb3DbL97g28/u5G/XLOCDZ0/TeyeSJU5MF3ociKQlJVlpKhh0vLTzMN9/cTe/23YAX57xJ++axS0Xz6NmQrHX4Umc5tVU8KO/eDdPbD/Ev/9uB3/7o83822/f5BPvns01y+v0XopkOE0XykiUZKWRgUCQjbs7eGL7QX61ZT8HjvVSWVrAzRc28KfnzWZGZYnXIcoYmBlrFtfyvoU1PPHmIR54YRdffWwHX/vdDs5rmMIHl0xn9RnVTNf7K5Jxgmp8lxEoyfJQT7+fbfuOsXF3B+t3tfPyO+109vop8BkXLajhtg8uYs2iWkoKddRgNsjLMy5ZXMsli2tpau3il5v3sW5zC5//xesALKgt511zprCifhJLZ1Yya3KpFpUVSXODi5Hq/1WJIa4ky8zWAt8AfMB3nHN3Drm9CPg+sBw4DFzrnNsVvu0fgJuAAPA3zrnHEhZ9Buj3BznU2Utzx3H2tvew+3APjYe6ePtQJzvbugl/CWJOVRkfXDKN1WdUc/68KiqKC7wNXJKqobqcWy9ZwN+tmU/joS6e2nGI595u4+ebmvnBS7sBqCjOZ+HUCubVlNNQXU7dpFJmTi5h+sQSKksLlIAlwHjGNhGIWidLlSyJ4bRJlpn5gLuAS4BmYL2ZrXPObYva7Sagwzk3z8yuA/4NuNbMFgPXAWcC04HHzWyBcy6Q6BeSCM45AkGHP+joDwTxBxz9/iADgSB9/gC9A6Hfx/uD9PT76e73090XoLPXT1ffAEd6BjhyfICO7n7au/tp6+qjrevk06z48ozZk0tpqCnnw+dM56zpEzlnZiXVFUUevWrxkpkxv7aC+bUV3HxhA/5AkDcPdPJGy1FebznKWwc7+e0bB+joGTjpfkX5edRMKGJKWRFTygqpLC2ksrSAiuJ8KooLqCjKp7TIR1lhPsUFPkoKfRTl51Fc4KMwP48Cn1Hoy6PAl0e+zyjIy8u5PxLjGdtSH62kK62TJSOJp5K1Cmh0zu0EMLOHgSuB6IHoSuD28OWfAv9hoa/ZVwIPO+f6gHfMrDH8eC8mIvi/eehVtu47CoAL/yfoHA5wkcsulDwFHTgcgWBoeyDoCAYdgfDlSHI1Vr48Y2JJAZWlBVSWFDBrcinLZk2idkIRUycUM2NSCTMnlTK9soTCfB2GIrHl+/I4a8ZEzpoxkeuith/p6Wdv+3H2dvSw/2gvB4/10trZR1tXH/uP9vLmgU46evrp6R/79xczwskW5OflkWehz7UvzzAz8izU3JtnhllofyO03cwwAAMjdP2OK87kPfOqxvkvklRjHtucc2MfLMJ+tWUf33zi7fE+jHgs8v+cGt8llniSrBnA3qjrzcC7htvHOec3s6PAlPD2l4bcd8bQJzCzm4GbAWbNmhVv7MyaXBqaDw9/tvPCA72FB/rQH4PQHwRf+HdeXuiPgs+MvDzDZ4bPF/qd78sjP8/ID3/Lz88zCvN9FPhssAJQEq4KlBT4KCvKp7TQR0VxPiUFPk3fSNKEKlWFLKkbeV00fyBIV5+frj4/Pf0Buvv8HB8I0DsQoG8gSK8/wIDf0RcI0u8P4g+EKrX+oMMfOPGlwx9wJ76MODf4hSW0LfSF5eQvMKHLkS87AOXFad/yOZ6xrS16p7GMYRNLCphfWz6mwCW9nD9vCoumVXgdhqShtBgFnXP3AvcCrFixIu5viJ/5wBlJi0kkE+X78gYTMkmdsYxhFy6o5sIF1UmNS0S8Fc+8VQswM+p6XXhbzH3MLB+YSKhJNJ77ioh4YTxjm4jIacWTZK0H5pvZHDMrJNTIvm7IPuuA68OXrwaeDPcsrAOuM7MiM5sDzAdeSUzoIiLjMp6xTUTktE47XRjuQ7gFeIzQYc73Oee2mtkdwAbn3Drgu8APwo3t7YQGK8L7/ZhQI6kf+Ot0PbJQRHLLeMY2EZF4WLp9KVuxYoXbsGGD12GISAqZ2Ubn3Aqv40gEjWEiuWWk8UtrCYiIiIgkgZIsERERkSRQkiUiIiKSBEqyRERERJIg7RrfzawV2D2Ku1QxZPXlLJULrzMXXiPodcYy2zmXFStzjnIM02chu+TC68yF1wgJGr/SLskaLTPbkC1HJY0kF15nLrxG0OuUE3Ll30ivM3vkwmuExL1OTReKiIiIJIGSLBEREZEkyIYk616vA0iRXHidufAaQa9TTsiVfyO9zuyRC68REvQ6M74nS0RERCQdZUMlS0RERCTtKMkSERERSYKMTbLM7Boz22pmQTNbMeS2fzCzRjPbYWYf8CrGRDKz282sxcw2h38u9zqmRDKzteH3q9HMPud1PMliZrvM7PXwe5gVZxE2s/vM7JCZvRG1bbKZ/d7M3g7/nuRljOkm18YvyO4xTONXZkvmGJaxSRbwBvBR4NnojWa2GLgOOBNYC/ynmflSH15S/F/n3NLwz6NeB5Mo4ffnLuAyYDHwsfD7mK0uDr+H2bLWzP2E/l+L9jngCefcfOCJ8HU5IRfHL8jCMUzjV1a4nySNYRmbZDnntjvndsS46UrgYedcn3PuHaARWJXa6GSUVgGNzrmdzrl+4GFC76NkAOfcs0D7kM1XAg+ELz8AfCSlQaU5jV9ZReNXhkvmGJaxSdYIZgB7o643h7dlg1vM7LVwaTObpl+y+T0bygG/M7ONZnaz18EkUa1zbn/48gGg1stgMki2/7+QjWNYtr9n0XJl/IIEjWH5iYsn8czscWBqjJtuc879MtXxJNtIrxf4FvAlQh/yLwFfA25MXXSSIBc451rMrAb4vZm9Gf4WlbWcc87Mcm6tmFwbv0BjWA7IufELxjeGpXWS5ZxbM4a7tQAzo67XhbelvXhfr5l9G/h1ksNJpYx9z0bLOdcS/n3IzH5BaKohGwepg2Y2zTm338ymAYe8DijVcm38gpwdwzL6PRuNHBq/IEFjWDZOF64DrjOzIjObA8wHXvE4pnELv8kRVxFqnM0W64H5ZjbHzAoJNf6u8zimhDOzMjOriFwGLiW73sdo64Drw5evB7KycpMEWTl+QVaPYRq/slNCxrC0rmSNxMyuAv4fUA38t5ltds59wDm31cx+DGwD/MBfO+cCXsaaIF8xs6WESu27gL/wNpzEcc75zewW4DHAB9znnNvqcVjJUAv8wswg9P/eg86533ob0viZ2UPAaqDKzJqBLwJ3Aj82s5uA3cAfexdh+snB8QuydAzT+JX5kjmG6bQ6IiIiIkmQjdOFIiIiIp5TkiUiIiKSBEqyRERERJJASZaIiIhIEijJEhEREUkCJVkiIiIiSaAkS0RERCQJ/j87BWludsVk2wAAAABJRU5ErkJggg==",
            "text/plain": [
              "<Figure size 720x288 with 2 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          },
          "output_type": "display_data"
        }
      ],
      "source": [
        "xp = np.linspace(-10.0,10.0,100)\n",
        "yp_sigmoid = sigmoid_der(xp)\n",
        "yp_relu = relu_der(xp)\n",
        "\n",
        "fig = plt.figure(figsize=(10,4))\n",
        "ax1 = fig.add_subplot(121)\n",
        "ax1.plot(xp,yp_sigmoid,label=\"Sigmoid\")\n",
        "ax1.legend()\n",
        "ax2 = fig.add_subplot(122)\n",
        "ax2.plot(xp,yp_relu,label=\"ReLU\")\n",
        "ax2.legend()\n",
        "plt.show()\n",
        "plt.close()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "B0FHTFBwvtLP"
      },
      "source": [
        "sigmoid関数は微分の値が最大で0.25なのに対して、ReLU関数では最大1.0となるため勾配の消失が起こりにくいのです。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PiFx3X4xJd7V"
      },
      "source": [
        "問題ごとに何が最適なネットワーク構造だったり活性化関数なのかは、  \n",
        "予め分かることはなく、試行錯誤が必要です。  \n",
        "ここまでこの授業で勉強してきた皆さんは既に、  \n",
        "「この試行錯誤自体を人力ではなくコンピュータにやらせる方法はないか」という点に思い至るのではないでしょうか?\n",
        "\n",
        "これに関連したお話はベイズ最適化の回で説明します。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hpVMAFUiwUtT"
      },
      "source": [
        "3.に関して\n",
        "\n",
        "また、ネットワークの重みやバイアスをどのような値から始めるかに学習が依存する場合もあります。\n",
        "\n",
        "というのも、今考えた３層のニューラルネットワークでは、  \n",
        "重み$W$の学習に使う勾配の表式は、$V$に比例しています。\n",
        "\n",
        "したがって単純に勾配の情報のみを使う最適化手法では、  \n",
        "$V$の初期値を0に取ったり、学習の過程で偶然$V$の値が0に近くなってしまうと、  \n",
        "$V$が更新されノンゼロの値を持つまで$W$の学習は始まりません。\n",
        "\n",
        "どのような初期値を採用するべきかに関しても、予め知ることは一般にはできませんが、  \n",
        "いくつかの特定の場合に関して、推奨される方法というのは存在しています。\n",
        "\n",
        "例: ReLU関数を活性化関数に使うときはHeの初期値というものが推奨されている"
      ]
    }
  ],
  "metadata": {
    "accelerator": "TPU",
    "colab": {
      "authorship_tag": "ABX9TyPpQYMbqGEZMRisgxppnqSm",
      "collapsed_sections": [],
      "include_colab_link": true,
      "name": "Python_chapter_ArtificialNeuralNetwork.ipynb",
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}