Support Vectors Machines (w. kernel)

Class 5

February 26th, 2020

Data Science: Techniques and Applications (DSTA)

Review of Kernelization

The Kernel matrix

Often…

K_{i j} = x_{i}^{T} \cdot x_{j}

$K_{ij} = x_i^T \cdot x_j$

$\Theta(n^2)$ dot products, costing $\Theta(d^2)$

3 model kernels

$\phi(\cdot): \mathcal{D_1}, \dots \mathcal{D_d} \rightarrow \mathcal{F}^e$

identity: $\phi(\vec{x}) = \vec{x}$

poly/non-linear: $\phi([x, y]^T) = [x^2, y^2, xy]^T$

Mercer: $\phi(\vec{x_i}) = \sqrt(\Lambda) \vec{U_i}$

Kernel trick

Show that

the analysis task (min/max/var/mean etc.) can be reduced to dot prod. in the feature space: $\phi(x_i)^T \cdot \phi(x_j)$

show that the dot prod. above can be replaced with look-ups in the Kernel matrix, i.e., $K(\vec{x_i}, \vec{x_j}) = \phi(\vec{x_i})^T \cdot \phi(\vec{x_j})$

it can be safely deployed to the three model kernels

SVMs (finally)

Goals

Solve classification over not-so-scattered datapoints

Ideas

map datapoints over a feature space where they would appear to fall farther apart from each other
find areas of maximum separations between groups
determine the cutting (hyper)plane that cuts across the clear area
use it as a classifier

SVD (and K-means): underlying assumptions

members of a class have individual/family variations that make their values oscillate around a typical class value
such classes are somewhat non-overlapping: there must be an emergent difference in features between individuals of class A and B.

in the vicinity of class A datapoints, values could only be proper for a class A object
datapoints should not be too close to the border to the next class

points lying at the mimimum distance from the cutting hyperplane are called support vectors
each class will have its support vectors and its minimum distance $\delta^{c}$ from the cutting h..
on to the Zaki-Meria presentation…

Assortativity

Assortativity […] is a preference for […] attaching to others that are similar in some way.

[…] At one point [he] moved from the Mathematics Department at MIT to that at Boston University.

Which singlehandedly raised the average IQ in both departments.

Support Vectors Machines (w. kernel) Class 5 February 26th, 2020