Tag Archives: regression

支持向量机

Posted by on April 12, 2012 No comments | 338 views

支持向量机(Support Vector Machines, SVM)最初由Vladimir Vapnik于1997年提出，SVM最简单的形式就是找出一下区分界限（descision boundary），也称之为超平面(hyperplane)，使得离它最近的点（称之为support vectors）与之间隔最大。

这和logistic regression有些相似，区别在于logistic regression要求所有的点尽可能地远离中间那条线，而SVM是要求最接近中间线的点尽可能地远离中间线。也就是说SVM的主要目标是区分那些最难区分的点。

SVM对于hyperplane的定义，在形式上和logistic regression一样,logistic regression的decision boundary由\( \theta^TX=0\)确定,SVM则用\( w^TX+b=0\)表示,其中b相当于logistic regression中的\( \theta_0\)，从形式上看，两者并无区别，当然如前面所说，两者的目标不一样，logistic regression着眼于全局，SVM着眼于support vectors。有监督算法都有label变量y，logistic regression取值是{0,1}，而SVM为了计算距离方便，取值为{-1,1}。

那么SVM的问题就变成为，确定参数w和b，以满足：
\( w^T x_i + b \ge +1\) when \( y_i = +1\)
\( w^T x_i + b \le +1\) when \( y_i = -1\)
Read more »

Machine Learning Ex 5.2 - Regularized Logistic Regression

Posted by on October 26, 2011 3 comments | 2,464 views

Now we move on to the second part of the Exercise 5.2, which requires to implement regularized logistic regression using Newton's Method.

Plot the data:

x < - read.csv("ex5Logx.dat", header=F)
y <- read.csv("ex5Logy.dat", header=F)
y <- y[,1]

d <- data.frame(x1=x[,1],x2=x[,2],y=factor(y))
require(ggplot2)
p <- ggplot(d, aes(x=x1, y=x2))+
    geom_point(aes(colour=y, shape=y))

Read more »

Machine Learning Ex 5.1 - Regularized Linear Regression

Posted by on October 25, 2011 10 comments | 1,887 views

The first part of the Exercise 5.1 requires to implement a regularized version of linear regression.

Adding regularization parameter can prevent the problem of over-fitting when fitting a high-order polynomial.

Plot the data:

?View Code RSPLUS
1
2
3
4
5
6
7
8
9

x < - read.table("ex5Linx.dat")
y <- read.table("ex5Liny.dat")
 
x <- x[,1]
y <- y[,1]
 
require(ggplot2)
d <- data.frame">data.frame(x=x,y=y)
p <- ggplot(d, aes(x,y)) + geom_point(colour="red", size=3)

Read more »

Machine Learning Ex4 - Logistic Regression

Posted by on October 24, 2011 5 comments | 1,739 views

Exercise 4 required implementing Logistic Regression using Newton's Method.

The dataset in use is 80 students and their grades of 2 exams, 40 students were admitted to college and the other 40 students were not. We need to implement a binary classification model to estimates college admission based on the student's scores on these two exams.

plot the data

?View Code RSPLUS
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

x < - read.table("ex4x.dat",header=F, stringsAsFactors=F)
x <- cbind(rep(1, nrow(x)), x)
colnames(x) <- c("X0", "Exam1", "Exam2")
x <- as.matrix(x)
 
y <- read.table("ex4y.dat",header=F, stringsAsFactors=F)
y <- y[,1]
 
## plotting data
d <- data.frame">data.frame(x,
                y = factor(y,
                levels=c(0,1),
                labels=c("Not admitted","Admitted" )
                )
                )
 
require(ggplot2)
p <- ggplot(d, aes(x=Exam1, y=Exam2)) +
    geom_point(aes(shape=y, colour=y)) +
    xlab("Exam 1 score") +
    ylab("Exam 2 score")

Read more »

Machine Learning Ex3 - Multivariate Linear Regression

Posted by on March 29, 2011 No comments | 1,722 views

Part 1. Finding alpha.
The first question to resolve in Exercise 3 is to pick a good learning rate alpha.

This require making an initial selection, running gradient descent and observing the cost function.

I test alpha range from 0.01 to 1.

?View Code RSPLUS
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51

##preparing data input.
x < - read.table("ex3x.dat", header=F)
y <- read.table("ex3y.dat", header=F)
 
#normalize features using Z-score.
x[,1] <- (x[,1] - mean(x[,1]))/sd(x[,1])
x[,2] <- (x[,2] - mean(x[,2]))/sd(x[,2])
 
x <- cbind(x0=rep(1, nrow(x)), x)
x <- as.matrix(x)
 
##gradient descent algorithm.
gradDescent_internal <- function(theta, x, y, m, alpha) {
  h <- sapply(1:nrow(x), function(i) t(theta) %*% x[i,])
	j <- t(h-y) %*% x
	grad <- 1/m * j
  theta <- t(theta) - alpha * grad 
	theta <- t(theta)
  return(theta)
}
 
## cost function.
J <- function(theta, x, y, m) {
  h <- sapply(1:nrow(x), function(i) t(theta) %*% x[i,])
  j <- 2*sum((h-y)^2)/m
  return(j)
}
 
## calculate cost function J for every iteration at specific alpha value.
testLearningRate <- function(x,y, alpha, niter=50) {
  j <- rep(0, niter)
  m <- nrow(x)
  theta <- matrix(rep(0, ncol(x)), ncol=1)
  for (i in 1:niter) {
    theta <- gradDescent_internal(theta,x,y,m, alpha)
    j[i] <- J(theta, x, y, m)
  }  
  return(j)
}
 
 
## test learning rate.
alpha=c(0.01, 0.03, 0.1, 0.3, 1)
xxx=sapply(alpha, testLearningRate, x=x, y=y)
colnames(xxx) <- as.character(alpha)
 
require(ggplot2)
xxx <- melt(xxx)
names(xxx) <- c("niter", "alpha", "J")
p <- ggplot(xxx, aes(x=niter, y=J))
p+geom_line(aes(colour=factor(alpha))) +xlab("Number of iteractions") +ylab("Cost J")

Read more »

  • Page 1 of 2
  • 1
  • 2
  • >