PyTorch Examples

Linear Regression is used to predict a numerical value, \(\hat{y}\), from observations, \(X = \{x_1, x_2, \cdots, x_{n}\}\) with the equation \[\hat{y}(X) = \sum_{i=0}^{n} w_i f_i (X)\] where the linear name (in linear regression) comes from the fact that the equation is linear w.r.t. the weights, \(w_i\).

The combination of function and data point \(f_i (X)\) is called a feature. Features make up the basis for the regression, thus should be unique. The features are up to us; we most only keep the prediction equation linear w.r.t. the weights.

The goal is to have the predictions, \(\hat{y}(X)\), be accurate. This is done by having, hopefully, lots of data, \[\{ y^0, X^0\},\{y^1, X^1\}, \cdots, \{y^{N-1}, X^{N-1}\}\] which now includes the observed values of the parameter we hope to predict, \(y\), to compare to. We do this by minimizing the loss, \[loss = \frac{1}{N}\sum_{i=0}^{N-1}\left( y^i - \hat{y}(X^i) \right)^2\], where the only free parameters are the weights in \(\hat{y}\).

footnote: In a linear algebra context, we would consider the fully system as matrix multiplication, notionally written \[\bf{\hat{y}} = \textbf{X}\textbf{w}\], and be concerned whether it was over/under determined and how to regularize. In these notes we expect that the system is over determined (there are more data examples than features).

In the machine learning context, we calculate the gradient of the loss w.r.t. the weights, \(\nabla loss\), and take a step in that direction by updating the weights with \[w^{new} = w^{old} + n \nabla loss\] where \(n\) is a scalar called the learning rate.

This is done iteratively, either a number of iteration or until the loss reaches a desired threshold. It must be said that this scheme is a local minimum finder; it will find a local minimum, but this is necessarily the lowest possible cost, also that the local minimum depends greatly on the starting set of weights.

It should be obvious that the choice of loss function greatly dictates the optimized weights. This is often stated as the crux of machine learning. At this point, we will only point out how it plays out the example of Linear Regression via a small generalization of our previous loss function,

\[loss(m) = \frac{1}{N} \sum_{i=0}^{N-1} \left( | y^i - \hat{y}(X^i) | \right)^m\]

where the larger the value of \(m\), the more large outliers in prediction error \[\left(|y - \hat{y}|\right)\] contribute to the loss and, thus, the higher their priority for minimization during the optimization.

• The following section shows PyTorch's use of Gradient Descent to fit a line to noisy data set.
• The standard linear regression naming conventions are used: the input data \(x\) and \(y\), the fit parameter \(w\) and bias \(b\), and the predicted dependent value, \[\hat{y} = w x +b.\]
• Each regression is found using the mean squared error (MSE) cost function, \[loss = \frac{1}{N} \sum ( y_i - \hat{y}_i)^2.\]
• Each epoch moves the parameters such that the MSE(\(\hat{y},y)\) is minimized.
  • Note: epoch 0 line in each figure displays the initial values of the parameters.

Example script Now we use linear regression and with the sigmoid function to find the line/plane/hyperplane between two classes, here [0,1].

#create noisy data
class NoisyBinaryData(Dataset):
    def __init__(self, N=100, x0=-3, x1=5, stdDev=2):
        xlist = []; ylist = []
        for i in range(N):
            #class 0
            if np.random.rand()<0.5:
                xlist.append(np.random.normal(x0,stdDev))
                ylist.append(0.0)
            #class 1
            else:
                xlist.append(np.random.normal(x1,stdDev))
                ylist.append(1.0)

        self.x = torch.tensor(xlist).view(-1,1)
        self.y = torch.tensor(ylist).view(-1,1)

    def __getitem__(self, index):
        return self.x[index], self.y[index]

    def __len__(self):
        return len(self.x)

np.random.seed(0)
data = NoisyBinaryData()
trainloader = DataLoader(dataset = data, batch_size = 20)

# create my "own" linear regression model
class logistic_regression(nn.Module):
    def __init__(self, input_size, output_size):
        #call the super's constructor and use it without having to store it directly.
        super(logistic_regression, self).__init__()
        self.linear = nn.Linear(input_size, output_size)

    def forward(self, x):
        """Prediction"""
        return torch.sigmoid(self.linear(x))

• Used to linearly classify between two or more classes.
• Softmax Equation:

\[S(y_i) = \frac{exp(y_i)}{\sum exp(y_j)}\]

• where, notably, \(S(y_i) \in [0,1]\) and \(\sum S(y_i) = 1\)

• Softmax relies on the classic argmax programming function, \[\hat{y} = argmax_i(S(y_i))\]
• Softmax uses parameter vectors where the dot product is used to classify.
• The complicated part here is the loss. How to incentivize this behavior with a decent gradient for learning.

Example ./MLP_makemore.md, which can be run in VSCode with the Jupytext extension or converted to a Jupyter Notebook by executing:

jupytext softMax_linLayer_makemore.md -o softMax_linLayer_makemore.ipynb

• Find three functions, on for each class, where the function that corresponds to each class has the largest value in the region where the class resides.
  • Then argmax is used to retrieve the class designation.

• \(z0 = - x\), \(z1 = 1\), and \(z2 = x -1\) and \(f(x) = [z0(x), z1(x), z2(x)]\),

  • class 0 for \(x \in (-\infty, -1)\)
  • class 1 for \(x \in (-1, 2)\)

  • class 2 for \(x \in (2, \infty)\)

    	z0	z1	z2	\(\hat{y}\)
    arg	0	1	2	argmax
    f(-5)	10	1	-6	0
    f(1)	-1	1	0	1
    f(4)	-4	1	3	2

1. Load Data
2. Create Model
3. Train Model
4. View Results