{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "gpuType": "T4"
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "# Pytorch Tutorial\n",
        "\n",
        "In this tutorial, we will learn basic concepts in Pytorch as follows.\n",
        "\n",
        "- Overview of Pytorch components\n",
        "- Tensors\n",
        "- Calculating gradients (autograd)\n",
        "- Layers in `torch.nn`\n",
        "- Training loop\n"
      ],
      "metadata": {
        "id": "GbQBOjkd4MLC"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Pytorch components\n",
        "\n",
        "There are some modules in Pytorch you should learn and remembers:\n",
        "- `torch`\n",
        "- `torch.autograd`\n",
        "- `torch.nn`\n",
        "- `torch.optim`\n",
        "- `torch.utils.data`\n",
        "- `torch.onnx`"
      ],
      "metadata": {
        "id": "vvHZ00zzPT1X"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Tensors\n",
        "\n",
        "Tensor is a data structure in Pytorch to store multi-dimensional arrays."
      ],
      "metadata": {
        "id": "oYeMzJoLwL14"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Creating Tensors\n",
        "\n",
        "We have various ways to create tensors"
      ],
      "metadata": {
        "id": "FeQz-O1XaYPD"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import numpy as np\n",
        "import torch\n",
        "\n",
        "# Create a tensor from a list\n",
        "t = torch.tensor([[1, 2], [3, 4.]])\n",
        "\n",
        "# Create a tensor on cuda\n",
        "t = torch.tensor([[1, 2], [3, 4.]], device=\"cuda:0\")\n",
        "\n",
        "# Create a tensor with double-precision\n",
        "t = torch.tensor([[1, 2], [3, 4.]], dtype=torch.float64)\n",
        "\n",
        "t = torch.arange(0, 10)\n",
        "\n",
        "t = torch.zeros(100, 10).to(\"cuda:0\")\n",
        "\n",
        "# Create a Tensor with random values\n",
        "t = torch.randn(100, 10)\n",
        "\n",
        "t.size()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "5iiUT7jlbL8f",
        "outputId": "614deabb-93d6-4ffe-9fb7-34ff8064d07f"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "torch.Size([100, 10])"
            ]
          },
          "metadata": {},
          "execution_count": 1
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Transform data to Tensors\n",
        "\n",
        "As you see in the notebook that implement [Feed-forward Neural Network](https://colab.research.google.com/drive/1wQUxcyfuAUam9zZDV56tMUzRFMOoT3Zh?usp=sharing#scrollTo=-5Et7l80zf0M), we can us the following statement to convert data to Tensors with `torch.tensor`."
      ],
      "metadata": {
        "id": "d_15eYMfab8E"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# X_train_vectors is an Numpy array\n",
        "# X_train_tensor = torch.tensor(X_train_vectors, dtype=torch.float32)"
      ],
      "metadata": {
        "id": "cor15PsJa5c3"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Indexing and Operations on Tensors\n",
        "\n",
        "Very similar to Numpy"
      ],
      "metadata": {
        "id": "KVktparzcxIv"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Gradients\n",
        "\n",
        "Tensors have attribute `requires_grad`. If we assign `requires_grad=True`, Pytorch enable a computation graph to compute gradients for Tensors. We can call the function `backward` to calculate the gradients."
      ],
      "metadata": {
        "id": "BmdmRGf1wZnT"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "x = torch.tensor(2.0, requires_grad=True)\n",
        "y = x ** 2\n",
        "y.backward()\n",
        "print(x.grad)  # Output: 4.0 (dy/dx = 2 * x)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "5XxHgmSvdgLP",
        "outputId": "41e05652-849b-4c2d-a670-c44a0736da47"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor(4.)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## torch.nn\n",
        "\n",
        "In order to make implementation of neural network easier, Pytorch provide pre-built layers, models in the module `torch.nn`."
      ],
      "metadata": {
        "id": "e0nWSRUPweAW"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from torch import nn\n",
        "\n",
        "net = nn.Linear(in_features=3, out_features=1, bias=False)"
      ],
      "metadata": {
        "id": "gMTJ6RVvhpPt"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Loss function\n",
        "\n",
        "There are many pre-built loss function in Pytorch."
      ],
      "metadata": {
        "id": "mxN9h2GIih1s"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "criterion = nn.CrossEntropyLoss()"
      ],
      "metadata": {
        "id": "n4GxPtLbikD0"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Optimization"
      ],
      "metadata": {
        "id": "BaKrYdlWww5_"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# optimizer = torch.optim.Adam(model.parameters(), lr=0.001)"
      ],
      "metadata": {
        "id": "DdrfkQq0i3np"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Data Utility\n",
        "\n",
        "In order to simplify the process of training with mini-batch or doing parallel computing, Pytorch provides class `TensorDataset` and `DataLoader`"
      ],
      "metadata": {
        "id": "2chFah6ew0ed"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from torch.utils.data import TensorDataset, DataLoader, RandomSampler, SequentialSampler"
      ],
      "metadata": {
        "id": "_xfbCkLTjEXn"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## torch.onnx\n",
        "\n",
        "ONNX (Open Neural Network eXchange) is a modern format proposed primarily by Facebook and Microsoft to standardize neural network models. It provides a mechanism to run inference in one framework using a model created in another framework."
      ],
      "metadata": {
        "id": "YHIkDH9Pw3Kw"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## References\n",
        "\n",
        "1. [Text Classification Using Feed-forward Neural Networks](https://colab.research.google.com/drive/1wQUxcyfuAUam9zZDV56tMUzRFMOoT3Zh#scrollTo=nBXYLRO4owGI)\n",
        "2. Pytorch Tutorial: [https://pytorch.org/tutorials/index.html](https://pytorch.org/tutorials/index.html)"
      ],
      "metadata": {
        "id": "ZdPvkKmEOErT"
      }
    }
  ]
}