- Tue 30 January 2018
- python
Linear Algebra using Torch¶
Here is a list of linear algrebra functions provided by pytorch. The order of the concepts in this notebook is based on deep learning book (by Goodfellow et. al) chapter 2.
Tensors¶
A tensor is an array of numbers, that may have,
- zero dimensions, and be a scalar ,
- one dimension, and be a vector
- two dimensions, and be a matrix
- or more dimensions.
importing torch¶
import torch
Scalars¶
A scalar is a number (Integers, real numbers, rational numbers) usually denoted by italic font
x = 12
x
Vectors¶
A vector is a 1-D array of numbers $$ \textbf{x} = \left[ \begin{array}{c} x_1 \\ x_2 \\ x_3 \\ ... \\ x_n \\ \end{array} \right] $$
x = torch.Tensor(5) # construct a vector lenght 5, uninitialized
x
y = torch.randn(5) # construct a vector lenght 5 with random values
y
# make a vector from a range
torch.range(1, 5)
x.size() # get the size of a vector
z = x + y # sum of two vectors (they should have same size)
z
Distance between two vectors¶
Returns the p-norm of (input - other)
dist1 = torch.dist(x,y)
dist2 = torch.dist(x,y,3)
print(dist1, dist2)
Matrix¶
A matrix is a 2-D array of numbers(Integers, real numbers, rational numbers) usually denoted by captial letters $$ \textbf{A} = \left[ \begin{array}{c} x_{11} & x_{12} & x_{13} & ... & x_{1n} \\ x_{21} & x_{22} & x_{23} & ... & x_{2n} \\ x_{31} & x_{32} & x_{33} & ... & x_{3n} \\ ... & ... & ... & ... & ...\\ x_{m1} & x_{m2} & x_{m3} & ... & x_{mn} \\ \end{array} \right] $$
# construct a matrix with 5 rows and 3 columns
A = torch.Tensor(2, 3)
print('A = ',A)
# construct a matrix with 2 rows and 3 columns with random values
B = torch.randn(2, 3)
print('B = ',B)
# converting a list of lists into a squared torch matrix
C = torch.Tensor([[6.80, -2.11, 5.66],
[-6.05, -3.30, 5.36],
[-0.45, 2.58, -2.70]]).t()
print('C = ', C)
# change vector to matrix
D = torch.range(1, 6).view(3, 2)
print('D = ', D)
Identity Matrix¶
A 2-D tensor with ones on the diagonal and zeros elsewhere
# create an identity matrix with lenght of 3
I3 = torch.eye(3)
I3
Matrix Addition¶
$$ E = A + B $$E = A + B
E
Matrix (Dot) Product¶
If matrix A is a n x m Tensor, matrix B is a m x p Tensor, out will be a n x p Tensor.
$$ C = AB $$$$ C_{i,j} = \sum_{k} A_{i,k}B_{k,j} $$# Matrix (Dot) Product
A = torch.randn(2, 3)
B = torch.randn(3, 3)
C = torch.mm(mat1, mat2)
C
Matrix-vector Product¶
Performs a matrix-vector product of the matrix mat and the vector vec.
$$ C = vA $$$$ C_{i,j} = \sum_{k} A_{i,k}v_{k} $$mat = torch.randn(2, 3)
vec = torch.randn(3)
torch.mv(mat, vec)
Slicing¶
B[:,1] # get second row
B[2,:] # get third column
Trace¶
$$ Tr(A) = \sum_{i} A_{i,i} $$# Trace Sum_{i} A_{i,i}
torch.trace(B)
Matrix Transpose¶
$$ (A^T)_{i,j} = A_{j,i}$$torch.transpose(B, 0, 1) # for matrix transpose you should always use 0 and 1
Matrix Inversion¶
X = torch.rand(4, 4)
print ('X = ',X)
X_inv = torch.inverse(X)
print ('X_inv = ',X_inv)
print ('X.X_inv = ', torch.mm(X, X_inv))
Special Matrices¶
Symmetric Matrix $$ A = A^T $$
A = torch.Tensor([[1, 2, 3],
[2, 4, 5],
[3, 5, 6]]).t()
print ('A= ', A)
AT = torch.transpose(A, 0, 1)
print ('AT= ',AT)
Orthogonal matrix $$ A^T A = A A^T = I $$ $$ A^{-1} = A^{T} $$
A = torch.Tensor([[0, -0.8, -0.6],
[0.8, -0.36, 0.48],
[0.6, 0.48, -0.64]]).t()
print ('A= ', A)
AT = torch.transpose(A, 0, 1)
print ('AT.A = ',torch.mm(AT,A))
Systems of Equations¶
$$ Ax = b $$X, LU = torch.gesv(B, A) returns the solution to the system of linear equations represented by AX=BAX=B LU contains L and U factors for LU factorization of A.
A = torch.Tensor([[6.80, -2.11, 5.66, 5.97, 8.23],
[-6.05, -3.30, 5.36, -4.44, 1.08],
[-0.45, 2.58, -2.70, 0.27, 9.04],
[8.32, 2.71, 4.35, -7.17, 2.14],
[-9.67, -5.14, -7.26, 6.08, -6.87]]).t()
B = torch.Tensor([[4.02, 6.19, -8.22, -7.57, -3.03],
[-1.56, 4.00, -8.67, 1.75, 2.86],
[9.81, -4.09, -4.57, -8.61, 8.99]]).t()
X, LU = torch.gesv(B, A)
print ('X = ',X)
print ('LU = ', LU)
torch.dist(B, torch.mm(A, X)) # Checking accuracy
Norm¶
Functions that measure how “large” a vector is, similar to a distance between zero and the point represented by the vector $$ L^{p} - Norm $$
# L2 norm
l2_norm = torch.norm(A)
# L3 norm
l3_norm = torch.norm(A,3)
# L1 norm
l1_norm = torch.norm(A, 1)
# max norm
max_norm = torch.max(torch.abs(A))
print ('L1 norm = ', l1_norm)
print ('L2 norm = ', l2_norm)
print ('L3 norm = ', l3_norm)
print ('Max norm = ', max_norm)
# Max norm
Eigendecomposition¶
Computing the eigenvalues and eigenvectors of a real square matrix.
## Eigenvector and eigenvalue
eigenvector, eigenvalue = torch.eig(A, True)
print('eigenvector = ', eigenvector)
print('eigenvalue = ', eigenvalue)
Singular Value Decomposition¶
$$ A = UDV^{T} $$a = torch.Tensor([[8.79, 6.11, -9.15, 9.57, -3.49, 9.84],
[9.93, 6.91, -7.93, 1.64, 4.02, 0.15],
[9.83, 5.04, 4.86, 8.83, 9.80, -8.99],
[5.45, -0.27, 4.85, 0.74, 10.00, -6.02],
[3.16, 7.98, 3.01, 5.80, 4.27, -5.31]]).t()
u, s, v = torch.svd(a)
print(u,s,v)
# accuracy
print(torch.dist(a, torch.mm(torch.mm(u, torch.diag(s)), v.t())))