---
title: "pytorch - GPU"
subtitle: "Lecture 24"
author: "Dr. Colin Rundel"
footer: "Sta 663 - Spring 2023"
format:
  revealjs:
    theme: slides.scss
    transition: fade
    slide-number: true
    self-contained: true
execute: 
  echo: true
---

```{python setup}
#| include: false
import numpy as np
import matplotlib as mpl
import matplotlib.pyplot as plt
import pandas as pd
import seaborn as sns
import scipy

import torch

import os

plt.rcParams['figure.dpi'] = 200

np.set_printoptions(
  edgeitems=30, linewidth=60,
  precision = 5, suppress=True
)

pd.set_option("display.width", 130)
pd.set_option("display.max_columns", 10)
pd.set_option("display.precision", 6)
```

```{r r_setup}
#| include: false
knitr::opts_chunk$set(
  fig.align="center",
  cache=FALSE
)

local({
  hook_err_old <- knitr::knit_hooks$get("error")  # save the old hook
  knitr::knit_hooks$set(error = function(x, options) {
    # now do whatever you want to do with x, and pass
    # the new x to the old hook
    x = sub("## \n## Detailed traceback:\n.*$", "", x)
    x = sub("Error in py_call_impl\\(.*?\\)\\: ", "", x)
    #x = stringr::str_wrap(x, width = 100)
    hook_err_old(x, options)
  })
  
  hook_warn_old <- knitr::knit_hooks$get("warning")  # save the old hook
  knitr::knit_hooks$set(warning = function(x, options) {
    x = sub("<string>:1: ", "", x)
    #x = stringr::str_wrap(x, width = 100)
    hook_warn_old(x, options)
  })
  
  hook_msg_old <- knitr::knit_hooks$get("output")  # save the old hook
  knitr::knit_hooks$set(output = function(x, options) {
    if (is.null(options$wrap))
      options$wrap = TRUE
    
    x = stringr::str_replace(x, "(## ).* ([A-Za-z]+Warning:)", "\\1\\2")
    x = stringr::str_split(x, "\n")[[1]]
    #x = stringr::str_wrap(x, width = 120, exdent = 3)
    x = stringr::str_remove_all(x, "\r")
    if (options$wrap)
        x = stringi::stri_wrap(x, width=120, exdent = 3, normalize=FALSE)
    x = paste(x, collapse="\n")
    
    #x = stringr::str_wrap(x, width = 100)
    hook_msg_old(x, options)
  })
})
```

## CUDA

> CUDA (or Compute Unified Device Architecture) is a parallel computing platform and application programming interface (API) that allows software to use certain types of graphics processing unit (GPU) for general purpose processing, an approach called general-purpose computing on GPUs (GPGPU). CUDA is a software layer that gives direct access to the GPU's virtual instruction set and parallel computational elements, for the execution of compute kernels.

<br/>

Core libraries:

:::: {.columns}
::: {.column width='33%'}
* cuBLAS

* cuSOLVER

* cuSPARSE
:::

::: {.column width='33%'}
* cuFFT

* cuTENSOR

* cuRAND
:::

::: {.column width='33%'}
* Thrust

* cuDNN
:::
::::



## CUDA Kernels

::: {.small}
```c
// Kernel - Adding two matrices MatA and MatB
__global__ void MatAdd(float MatA[N][N], float MatB[N][N], float MatC[N][N])
{
    int i = blockIdx.x * blockDim.x + threadIdx.x;
    int j = blockIdx.y * blockDim.y + threadIdx.y;
    if (i < N && j < N)
        MatC[i][j] = MatA[i][j] + MatB[i][j];
}
 
int main()
{
    ...
    // Matrix addition kernel launch from host code
    dim3 threadsPerBlock(16, 16);
    dim3 numBlocks(
        (N + threadsPerBlock.x -1) / threadsPerBlock.x, 
        (N+threadsPerBlock.y -1) / threadsPerBlock.y
    );
    
    MatAdd<<<numBlocks, threadsPerBlock>>>(MatA, MatB, MatC);
    ...
}
```
:::


##

![](imgs/gpu_bench1.png){fig-align="center" width="100%"}


##

![](imgs/gpu_bench2.png){fig-align="center" width="100%"}

## GPU Status

::: {.small}
```{bash}
nvidia-smi
```
:::

## Torch GPU Information

::: {.small}
```{python}
torch.cuda.is_available()

torch.cuda.device_count()

torch.cuda.get_device_name("cuda:0")
torch.cuda.get_device_name("cuda:1")

torch.cuda.get_device_properties(0)
torch.cuda.get_device_properties(1)
```
:::


## GPU Tensors

Usage of the GPU is governed by the location of the Tensors - to use the GPU we allocate them on the GPU device.

:::: {.columns .small}
::: {.column width='50%'}
```{python}
cpu = torch.device('cpu')
cuda0 = torch.device('cuda:0')
cuda1 = torch.device('cuda:1')

x = torch.linspace(0,1,5, device=cuda0); x
y = torch.randn(5,2, device=cuda0); y
z = torch.rand(2,3, device=cpu); z
```
:::

::: {.column width='50%' .fragment}
```{python error=TRUE}
x @ y
y @ z
y @ z.to(cuda0)
```
:::
::::


## NN Layers + GPU

NN layers (parameters) also need to be assigned to the GPU to be used with GPU tensors,

::: {.small}
```{python error=TRUE}
nn = torch.nn.Linear(5,5)
X = torch.randn(10,5).cuda()
```
:::

. . .

::: {.small}
```{python error=TRUE}
nn(X)
```
:::

. . .

:::: {.columns .small}
::: {.column width='50%'}
```{python}
nn.cuda()(X)
```
:::

::: {.column width='50%'}
```{python}
nn.to(device="cuda")(X)
```
:::
::::


## Back to MNIST

Same MNIST data from last time (1x8x8 images),

::: {.small}
```{python}
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split

digits = load_digits()
X, y = digits.data, digits.target

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.20, shuffle=True, random_state=1234
)

X_train = torch.from_numpy(X_train).float()
y_train = torch.from_numpy(y_train)
X_test = torch.from_numpy(X_test).float()
y_test = torch.from_numpy(y_test)
```
:::

. . .

To use the GPU for computation we need to copy these tensors to the GPU,

::: {.small}
```{python}
X_train_cuda = X_train.to(device=cuda0)
y_train_cuda = y_train.to(device=cuda0)
X_test_cuda = X_test.to(device=cuda0)
y_test_cuda = y_test.to(device=cuda0)
```
:::


## Convolutional NN

::: {.small}
```{python}
class mnist_conv_model(torch.nn.Module):
    def __init__(self, device):
        super().__init__()
        self.device = torch.device(device)
        
        self.model = torch.nn.Sequential(
          torch.nn.Unflatten(1, (1,8,8)),
          torch.nn.Conv2d(
            in_channels=1, out_channels=8,
            kernel_size=3, stride=1, padding=1
          ),
          torch.nn.ReLU(),
          torch.nn.MaxPool2d(kernel_size=2),
          torch.nn.Flatten(),
          torch.nn.Linear(8 * 4 * 4, 10)
        ).to(device=self.device)
        
    def forward(self, X):
        return self.model(X)
    
    def fit(self, X, y, lr=0.001, n=1000, acc_step=10):
      opt = torch.optim.SGD(self.parameters(), lr=lr, momentum=0.9) 
      losses = []
      for i in range(n):
          opt.zero_grad()
          loss = torch.nn.CrossEntropyLoss()(self(X), y)
          loss.backward()
          opt.step()
          losses.append(loss.item())
      
      return losses
    
    def accuracy(self, X, y):
      val, pred = torch.max(self(X), dim=1)
      return( (pred == y).sum() / len(y) )
```
:::


## CPU vs Cuda

:::: {.columns .small}
::: {.column width='50%'}
```{python}
m = mnist_conv_model(device="cpu")
loss = m.fit(X_train, y_train, n=1000)
loss[-1]
m.accuracy(X_test, y_test)
```
:::

::: {.column width='50%'}
```{python}
m_cuda = mnist_conv_model(device="cuda")
loss = m_cuda.fit(X_train_cuda, y_train_cuda, n=1000)
loss[-1]
m_cuda.accuracy(X_test_cuda, y_test_cuda)
```
:::
::::

## Performance

:::: {.columns .small}
::: {.column width='50%'}
CPU performance:
```{python}
m = mnist_conv_model(device="cpu")

start = torch.cuda.Event(enable_timing=True)
end = torch.cuda.Event(enable_timing=True)

start.record()
loss = m.fit(X_train, y_train, n=1000)
end.record()

torch.cuda.synchronize()
print(start.elapsed_time(end) / 1000) 
```
:::

::: {.column width='50%'}
GPU performance:
```{python}
m_cuda = mnist_conv_model(device="cuda")

start = torch.cuda.Event(enable_timing=True)
end = torch.cuda.Event(enable_timing=True)

start.record()
loss = m_cuda.fit(X_train_cuda, y_train_cuda, n=1000)
end.record()

torch.cuda.synchronize()
print(start.elapsed_time(end) / 1000) 
```
:::
::::


## Profiling CPU - 1 forward step

::: {.small}
```{python}
m = mnist_conv_model(device="cpu")
with torch.autograd.profiler.profile(with_stack=True, profile_memory=True) as prof_cpu:
    tmp = m(X_train)
```

```{python, wrap=FALSE}
print(prof_cpu.key_averages().table(sort_by='self_cpu_time_total', row_limit=5))
```
:::

## Profiling GPU - 1 forward step

::: {.small}
```{python}
m_cuda = mnist_conv_model(device="cuda")
with torch.autograd.profiler.profile(with_stack=True) as prof_cuda:
    tmp = m_cuda(X_train_cuda)
```

```{python, wrap=FALSE}
print(prof_cuda.key_averages().table(sort_by='self_cpu_time_total', row_limit=5))
```
:::


## Profiling CPU - fit

::: {.small}
```{python}
m = mnist_conv_model(device="cpu")
with torch.autograd.profiler.profile(with_stack=True, profile_memory=True) as prof_cpu:
    losses = m.fit(X_train, y_train, n=1000)
```

```{python, wrap=FALSE}
print(prof_cpu.key_averages().table(sort_by='self_cpu_time_total', row_limit=5))
```
:::

## Profiling GPU - fit

::: {.small}
```{python}
m_cuda = mnist_conv_model(device="cuda")
with torch.autograd.profiler.profile(with_stack=True) as prof_cuda:
    losses = m_cuda.fit(X_train_cuda, y_train_cuda, n=1000)
```

```{python, wrap=FALSE}
print(prof_cuda.key_averages().table(sort_by='self_cpu_time_total', row_limit=5))
```
:::



# CIFAR10

<br/>

::: {.small}
[homepage](https://www.cs.toronto.edu/~kriz/cifar.html)
:::

## Loading the data

::: {.small}
```{python}
#| results: hide
import torchvision

training_data = torchvision.datasets.CIFAR10(
    root="/data",
    train=True,
    download=True,
    transform=torchvision.transforms.ToTensor()
)

test_data = torchvision.datasets.CIFAR10(
    root="/data",
    train=False,
    download=True,
    transform=torchvision.transforms.ToTensor()
)
```
:::

## CIFAR10 data

::: {.small}
```{python}
training_data.classes
training_data.data.shape
test_data.data.shape
```
:::

. . .

::: {.small}
```{python}
training_data[0]
```
:::

## Example data

```{python echo=FALSE, out.width="85%"}
fig, axes = plt.subplots(nrows=4, ncols=6, figsize=(10, 6), layout="constrained")

for i, ax in enumerate([ax for row in axes for ax in row]):
    ax.set_axis_off()
    img, cls = training_data[i]
    
    p = ax.imshow(img.numpy().transpose((1,2,0)))
    t = ax.set_title(f"{training_data.classes[cls]}")
    
plt.show()
```

## Data Loaders

```{python}
batch_size = 100

training_loader = torch.utils.data.DataLoader(
    training_data, 
    batch_size=batch_size,
    shuffle=True,
    num_workers=4,
    pin_memory=True
)

test_loader = torch.utils.data.DataLoader(
    test_data, 
    batch_size=batch_size,
    shuffle=True,
    num_workers=4,
    pin_memory=True
)
```

## Loader generator

```{python}
training_loader
```

. . .

```{python}
X, y = next(iter(training_loader))
X.shape
y.shape
```


## CIFAR CNN

::: {.small}
```{python}
class cifar_conv_model(torch.nn.Module):
    def __init__(self, device):
        super().__init__()
        self.device = torch.device(device)
        self.epoch = 0
        self.model = torch.nn.Sequential(
            torch.nn.Conv2d(3, 6, kernel_size=5),
            torch.nn.ReLU(),
            torch.nn.MaxPool2d(2, 2),
            torch.nn.Conv2d(6, 16, kernel_size=5),
            torch.nn.ReLU(),
            torch.nn.MaxPool2d(2, 2),
            torch.nn.Flatten(),
            torch.nn.Linear(16 * 5 * 5, 120),
            torch.nn.ReLU(),
            torch.nn.Linear(120, 84),
            torch.nn.ReLU(),
            torch.nn.Linear(84, 10)
        ).to(device=self.device)
        
    def forward(self, X):
        return self.model(X)
    
    def fit(self, loader, epochs=10, n_report=250, lr=0.001):
        opt = torch.optim.SGD(self.parameters(), lr=lr, momentum=0.9) 
      
        for j in range(epochs):
            running_loss = 0.0
            for i, (X, y) in enumerate(loader):
                X, y = X.to(self.device), y.to(self.device)
                opt.zero_grad()
                loss = torch.nn.CrossEntropyLoss()(self(X), y)
                loss.backward()
                opt.step()
    
                # print statistics
                running_loss += loss.item()
                if i % n_report == (n_report-1):    # print every 100 mini-batches
                    print(f'[Epoch {self.epoch + 1}, Minibatch {i + 1:4d}] loss: {running_loss / n_report:.3f}')
                    running_loss = 0.0
            
            self.epoch += 1
```
:::

::: {.aside}
Based on [source](https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html)
:::


## CNN Performance - CPU (1 step)

::: {.small}
```{python}
X, y = next(iter(training_loader))

m_cpu = cifar_conv_model(device="cpu")
tmp = m_cpu(X)

with torch.autograd.profiler.profile(with_stack=True) as prof_cpu:
    tmp = m_cpu(X)
```

```{python, wrap=FALSE}
print(prof_cpu.key_averages().table(sort_by='self_cpu_time_total', row_limit=5))
```
:::


## CNN Performance - GPU (1 step)

::: {.small}
```{python}
m_cuda = cifar_conv_model(device="cuda")
Xc, yc = X.to(device="cuda"), y.to(device="cuda")
tmp = m_cuda(Xc)
    
with torch.autograd.profiler.profile(with_stack=True) as prof_cuda:
    tmp = m_cuda(Xc)
```

```{python, wrap=FALSE}
print(prof_cuda.key_averages().table(sort_by='self_cpu_time_total', row_limit=5))
```
:::




## CNN Performance - CPU (1 epoch)

::: {.small}
```{python}
m_cpu = cifar_conv_model(device="cpu")

with torch.autograd.profiler.profile(with_stack=True) as prof_cpu:
    m_cpu.fit(loader=training_loader, epochs=1, n_report=501)
```

```{python, wrap=FALSE}
print(prof_cpu.key_averages().table(sort_by='self_cpu_time_total', row_limit=5))
```
:::

## CNN Performance - GPU (1 epoch)

::: {.small}
```{python}
m_cuda = cifar_conv_model(device="cuda")

with torch.autograd.profiler.profile(with_stack=True) as prof_cuda:
    m_cuda.fit(loader=training_loader, epochs=1, n_report=501)
```

```{python, wrap=FALSE}
print(prof_cuda.key_averages().table(sort_by='self_cpu_time_total', row_limit=5))
```
:::






## Loaders & Accuracy

```{python}
def accuracy(model, loader, device):
    total, correct = 0, 0
    with torch.no_grad():
        for X, y in loader:
            X, y = X.to(device=device), y.to(device=device)
            pred = model(X)
            # the class with the highest energy is what we choose as prediction
            val, idx = torch.max(pred, 1)
            total += pred.size(0)
            correct += (idx == y).sum().item()
            
    return correct / total
```


## Model fitting

::: {.small}
```{python eval=FALSE}
m = cifar_conv_model("cuda")
m.fit(training_loader, epochs=10, n_report=500, lr=0.01)
## [Epoch 1, Minibatch  500] loss: 2.098
## [Epoch 2, Minibatch  500] loss: 1.692
## [Epoch 3, Minibatch  500] loss: 1.482
## [Epoch 4, Minibatch  500] loss: 1.374
## [Epoch 5, Minibatch  500] loss: 1.292
## [Epoch 6, Minibatch  500] loss: 1.226
## [Epoch 7, Minibatch  500] loss: 1.173
## [Epoch 8, Minibatch  500] loss: 1.117
## [Epoch 9, Minibatch  500] loss: 1.071
## [Epoch 10, Minibatch  500] loss: 1.035
```
:::

::: {.small}
```{python eval=FALSE}
accuracy(m, training_loader, "cuda")
## 0.63444
accuracy(m, test_loader, "cuda")
## 0.572
```
:::


## More epochs

If continue fitting with the existing model,

::: {.small}
```{python eval=FALSE}
m.fit(training_loader, epochs=10, n_report=500)
## [Epoch 11, Minibatch  500] loss: 0.885
## [Epoch 12, Minibatch  500] loss: 0.853
## [Epoch 13, Minibatch  500] loss: 0.839
## [Epoch 14, Minibatch  500] loss: 0.828
## [Epoch 15, Minibatch  500] loss: 0.817
## [Epoch 16, Minibatch  500] loss: 0.806
## [Epoch 17, Minibatch  500] loss: 0.798
## [Epoch 18, Minibatch  500] loss: 0.787
## [Epoch 19, Minibatch  500] loss: 0.780
## [Epoch 20, Minibatch  500] loss: 0.773
```

```{python eval=FALSE}
accuracy(m, training_loader, "cuda")
## 0.73914
accuracy(m, test_loader, "cuda")
## 0.624
```
:::

## More epochs (again)

::: {.small}
```{python eval=FALSE}
m.fit(training_loader, epochs=10, n_report=500)
## [Epoch 21, Minibatch  500] loss: 0.764
## [Epoch 22, Minibatch  500] loss: 0.756
## [Epoch 23, Minibatch  500] loss: 0.748
## [Epoch 24, Minibatch  500] loss: 0.739
## [Epoch 25, Minibatch  500] loss: 0.733
## [Epoch 26, Minibatch  500] loss: 0.726
## [Epoch 27, Minibatch  500] loss: 0.718
## [Epoch 28, Minibatch  500] loss: 0.710
## [Epoch 29, Minibatch  500] loss: 0.702
## [Epoch 30, Minibatch  500] loss: 0.698
```

```{python eval=FALSE}
accuracy(m, training_loader, "cuda")
## 0.76438
accuracy(m, test_loader, "cuda")
## 0.6217
```
:::

## The VGG16 model

::: {.small}
```{python}
class VGG16(torch.nn.Module):
    def make_layers(self):
        cfg = [64, 64, 'M', 128, 128, 'M', 256, 256, 256, 'M', 512, 512, 512, 'M', 512, 512, 512, 'M']
        layers = []
        in_channels = 3
        for x in cfg:
            if x == 'M':
                layers += [torch.nn.MaxPool2d(kernel_size=2, stride=2)]
            else:
                layers += [torch.nn.Conv2d(in_channels, x, kernel_size=3, padding=1),
                           torch.nn.BatchNorm2d(x),
                           torch.nn.ReLU(inplace=True)]
                in_channels = x
        layers += [
            torch.nn.AvgPool2d(kernel_size=1, stride=1),
            torch.nn.Flatten(),
            torch.nn.Linear(512,10)
        ]
        
        return torch.nn.Sequential(*layers).to(self.device)
    
    def __init__(self, device):
        super().__init__()
        self.device = torch.device(device)
        self.model = self.make_layers()
    
    def forward(self, X):
        return self.model(X)
```
:::

::: {.aside}
Based on code from [pytorch-cifar](https://github.com/kuangliu/pytorch-cifar), original [paper](https://arxiv.org/abs/1409.1556)
:::

## Model

::: {.small}
```{python}
VGG16("cpu").model
```
:::


## VGG16 performance - CPU

::: {.small}
```{python}
X, y = next(iter(training_loader))
m_cpu = VGG16(device="cpu")
tmp = m_cpu(X)

with torch.autograd.profiler.profile(with_stack=True) as prof_cpu:
    tmp = m_cpu(X)
```

```{python, wrap=FALSE}
print(prof_cpu.key_averages().table(sort_by='self_cpu_time_total', row_limit=5))
```
:::


## VGG16 performance - GPU

::: {.small}
```{python}
m_cuda = VGG16(device="cuda")
Xc, yc = X.to(device="cuda"), y.to(device="cuda")
tmp = m_cuda(Xc)

with torch.autograd.profiler.profile(with_stack=True) as prof_cuda:
    tmp = m_cuda(Xc)
```

```{python, wrap=FALSE}
print(prof_cuda.key_averages().table(sort_by='self_cpu_time_total', row_limit=5))
```
:::


## VGG16 performance - Apple M1 GPU (mps)

::: {.small}
```{python}
#| eval: false
m_mps = VGG16(device="mps")
Xm, ym = X.to(device="mps"), y.to(device="mps")

with torch.autograd.profiler.profile(with_stack=True) as prof_mps:
    tmp = m_mps(Xm)
```

```{python}
#| eval: false
print(prof_mps.key_averages().table(sort_by='self_cpu_time_total', row_limit=5))
```

```
--------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  
                            Name    Self CPU %      Self CPU   CPU total %     CPU total  CPU time avg    # of Calls  
--------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  
         aten::native_batch_norm        35.71%       3.045ms        35.71%       3.045ms     234.231us            13  
          aten::_mps_convolution        19.67%       1.677ms        19.88%       1.695ms     130.385us            13  
    aten::_batch_norm_impl_index        11.92%       1.016ms        36.02%       3.071ms     236.231us            13  
                     aten::relu_        11.29%     963.000us        11.29%     963.000us      74.077us            13  
                      aten::add_        10.40%     887.000us        10.44%     890.000us      68.462us            13  
--------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  
Self CPU time total: 8.526ms
```
:::


## Fitting w/ `lr = 0.01`

::: {.small}
```{python eval=FALSE}
m = VGG16(device="cuda")
fit(m, training_loader, epochs=10, n_report=500, lr=0.01)

## [Epoch 1, Minibatch  500] loss: 1.345
## [Epoch 2, Minibatch  500] loss: 0.790
## [Epoch 3, Minibatch  500] loss: 0.577
## [Epoch 4, Minibatch  500] loss: 0.445
## [Epoch 5, Minibatch  500] loss: 0.350
## [Epoch 6, Minibatch  500] loss: 0.274
## [Epoch 7, Minibatch  500] loss: 0.215
## [Epoch 8, Minibatch  500] loss: 0.167
## [Epoch 9, Minibatch  500] loss: 0.127
## [Epoch 10, Minibatch  500] loss: 0.103
```
:::

. . .

::: {.small}
```{python eval=FALSE}
accuracy(model=m, loader=training_loader, device="cuda")
## 0.97008
accuracy(model=m, loader=test_loader, device="cuda")
## 0.8318
```
:::


## Fitting w/ `lr = 0.001`

::: {.small}
```{python eval=FALSE}
m = VGG16(device="cuda")
fit(m, training_loader, epochs=10, n_report=500, lr=0.001)

## [Epoch 1, Minibatch  500] loss: 1.279
## [Epoch 2, Minibatch  500] loss: 0.827
## [Epoch 3, Minibatch  500] loss: 0.599
## [Epoch 4, Minibatch  500] loss: 0.428
## [Epoch 5, Minibatch  500] loss: 0.303
## [Epoch 6, Minibatch  500] loss: 0.210
## [Epoch 7, Minibatch  500] loss: 0.144
## [Epoch 8, Minibatch  500] loss: 0.108
## [Epoch 9, Minibatch  500] loss: 0.088
## [Epoch 10, Minibatch  500] loss: 0.063
```
:::

. . .

::: {.small}
```{python eval=FALSE}
accuracy(model=m, loader=training_loader, device="cuda")
## 0.9815
accuracy(model=m, loader=test_loader, device="cuda")
## 0.7816
```
:::

## Report

::: {.small}
```{python}
from sklearn.metrics import classification_report

def report(model, loader, device):
    y_true, y_pred = [], []
    with torch.no_grad():
        for X, y in loader:
            X = X.to(device=device)
            y_true.append( y.cpu().numpy() )
            y_pred.append( model(X).max(1)[1].cpu().numpy() )
    
    y_true = np.concatenate(y_true)
    y_pred = np.concatenate(y_pred)

    return classification_report(y_true, y_pred, target_names=loader.dataset.classes)
```
:::

##

::: {.small}
```{python}
#| eval: false
print(report(model=m, loader=test_loader, device="cuda"))

##               precision    recall  f1-score   support
## 
##     airplane       0.82      0.88      0.85      1000
##   automobile       0.95      0.89      0.92      1000
##         bird       0.85      0.70      0.77      1000
##          cat       0.68      0.74      0.71      1000
##         deer       0.84      0.83      0.83      1000
##          dog       0.81      0.73      0.77      1000
##         frog       0.83      0.92      0.87      1000
##        horse       0.87      0.87      0.87      1000
##         ship       0.89      0.92      0.90      1000
##        truck       0.86      0.93      0.89      1000
## 
##     accuracy                           0.84     10000
##    macro avg       0.84      0.84      0.84     10000
## weighted avg       0.84      0.84      0.84     10000
```
:::


# Some state-of-the-art examples

## Hugging Face

This is an online community and platform for sharing machine learning models (architectures and weights), data, and related artifacts. They also maintain a number of packages and related training materials that help with building, training, and deploying ML models.

Some notable resources,

* [`transformers`](https://huggingface.co/docs/transformers/index) - APIs and tools to easily download and train state-of-the-art (pretrained) transformer based models 

* [`diffusers`](https://huggingface.co/docs/diffusers/index) - provides pretrained vision and audio diffusion models, and serves as a modular toolbox for inference and training

* [`timm`](https://huggingface.co/docs/timm/index) - a library containing SOTA computer vision models, layers, utilities, optimizers, schedulers, data-loaders, augmentations, and training/evaluation scripts


## Stable Diffusion

::: {.small}
```{python}
#| warning: false
#| message: false
#| results: hide
from diffusers import StableDiffusionPipeline

pipe = StableDiffusionPipeline.from_pretrained(
  "stabilityai/stable-diffusion-2-1-base", torch_dtype=torch.float16
).to("cuda")
```
:::

. . .

::: {.small}
```{python sd1-1}
#| results: hide
#| cache: true
prompt = "a picture of thomas bayes with a cat on his lap"
generator = [torch.Generator(device="cuda").manual_seed(i) for i in range(6)]
fit = pipe(prompt, generator=generator, num_inference_steps=20, num_images_per_prompt=6)
```
:::

. . .

::: {.small}
```{python sd1-2}
#| cache: true
fit.images
```
:::

##

```{python sd1-plot}
#| echo: false
#| cache: true
fig, axes = plt.subplots(nrows=2, ncols=3, figsize=(10, 6), layout="constrained")

for i, ax in enumerate([ax for row in axes for ax in row]):
    ax.set_axis_off()
    p = ax.imshow(fit.images[i])
    
plt.show()
```

::: {.aside}
[Thomas Bayes GIS](https://www.google.com/search?q=thomas+bayes&source=lnms&tbm=isch&sa=X&biw=1280&bih=590&dpr=2)
:::


## Customizing prompts

::: {.small}
```{python sd2}
#| results: hide
#| cache: true
prompt = "a picture of thomas bayes with a cat on his lap"
prompts = [
  prompt + t for t in 
  ["in the style of a japanese wood block print",
   "as a hipster with facial hair and glasses",
   "as a simpsons character, cartoon, yellow",
   "in the style of a vincent van gogh painting",
   "in the style of a picasso painting",
   "with flowery wall paper"
  ]
]

generator = [torch.Generator(device="cuda").manual_seed(i) for i in range(6)]
fit = pipe(prompts, generator=generator, num_inference_steps=20, num_images_per_prompt=1)
```
:::

## 

```{python sd2-plot}
#| echo: false
#| cache: true
fig, axes = plt.subplots(nrows=2, ncols=3, figsize=(10, 6), layout="constrained")

for i, ax in enumerate([ax for row in axes for ax in row]):
    ax.set_axis_off()
    p = ax.imshow(fit.images[i])
    
plt.show()
```



## Increasing inference steps

::: {.small}
```{python sd3}
#| results: hide
#| cache: true
generator = [torch.Generator(device="cuda").manual_seed(i) for i in range(6)]
fit = pipe(prompts, generator=generator, num_inference_steps=50, num_images_per_prompt=1)
```
:::

```{python sd3-plot}
#| echo: false
#| cache: true
fig, axes = plt.subplots(nrows=2, ncols=3, figsize=(10, 6), layout="constrained")

for i, ax in enumerate([ax for row in axes for ax in row]):
    ax.set_axis_off()
    p = ax.imshow(fit.images[i])
    
plt.show()
```




## Alpaca LoRA

::: {.small}
```{python}
#| eval: false
from transformers import GenerationConfig, LlamaTokenizer, LlamaForCausalLM

tokenizer = LlamaTokenizer.from_pretrained("chainyo/alpaca-lora-7b")

model = LlamaForCausalLM.from_pretrained(
    "chainyo/alpaca-lora-7b",
    load_in_8bit=True,
    torch_dtype=torch.float16,
    device_map="auto",
)

generation_config = GenerationConfig(
    temperature=0.2,
    top_p=0.75,
    top_k=40,
    num_beams=4,
    max_new_tokens=128,
)
```
:::

::: {.aside}
Based on [chainyo/alpaca-lora-7b](https://huggingface.co/chainyo/alpaca-lora-7b), see also [alpaca lora](https://github.com/tloen/alpaca-lora), [stanform alpaca](https://github.com/tatsu-lab/stanford_alpaca) & [llama](https://ai.facebook.com/blog/large-language-model-llama-meta-ai/)
:::

## Generate a prompt

::: {.small}
```{python}
#| include: false
def generate_prompt(instruction, input_ctxt = None):
    if input_ctxt:
        return f"""Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.

### Instruction:
{instruction}

### Input:
{input_ctxt}

### Response:"""
    else:
        return f"""Below is an instruction that describes a task. Write a response that appropriately completes the request.

### Instruction:
{instruction}

### Response:"""
```


```{python}
#| eval: false
instruction = "Write a short childrens story about Thomas Bayes and his pet cat"
input_ctxt = None 
prompt = generate_prompt(instruction, input_ctxt)
print(prompt)
```

```
Below is an instruction that describes a task. Write a response that appropriately completes the request.

### Instruction:
Write a short childrens story about Thomas Bayes and his pet cat

### Response:
```
:::


## Running the model

::: {.small}
```{python}
#| eval: false
input_ids = tokenizer(prompt, return_tensors="pt").input_ids.to(model.device)

with torch.no_grad():
    outputs = model.generate(
        input_ids=input_ids,
        generation_config=generation_config,
        return_dict_in_generate=True,
        output_scores=True,
    )

response = tokenizer.decode(outputs.sequences[0], skip_special_tokens=True)
print(response)
```

```
Below is an instruction that describes a task. Write a response that appropriately completes the request.

### Instruction:
Write a short childrens story about Thomas Bayes and his pet cat

### Response:
Once upon a time, there was a little boy named Thomas Bayes. He had a pet cat named Fluffy, and 
they were the best of friends. One day, Thomas and Fluffy decided to go on an adventure. They 
traveled far and wide, exploring new places and meeting new people. Along the way, Thomas and 
Fluffy learned many valuable lessons, such as the importance of friendship and the joy of discovery.
Eventually, Thomas and Fluffy made their way back home, where they were welcomed with open arms. 
Thomas and Fluffy had a wonderful time.
```
:::