Thanh-Nam blog

Contour visualization with Plots.jl

Tue, 22 Feb 2022 00:00:00 +0000

I found that the official tutorial of contours plot on Plots.jl website is not enough for us to generate good visualization. So below is my attempt to have a nice plot.

Prerequisite

Julia 1.7.1
Plots.jl 1.25.7
Backend: GR

We use Himmelblau’s function to illustrate our visualization. The function is straightforward to define in Julia:

using Plots
f(x, y) = (x^2 + y - 11)^2 + (x + y^2 -7)^2

range_x = -6:0.1:6
range_y = -6:0.1:6

The first attempt

contour(range_x, range_y, f, xlabel="x", ylabel="y")

The code snippet above means that we need to draw the contour of function f with range of its parameter x and y defined by range_x and range_y respectively. Here is the result

The result is quite terrible since it does not allow us to recognize local minima. Can we do better?

The second attempt

The answer is yes since contour provides several parameters and we try to explore in order to make better plot.

The most important parameter we will explore is levels

contour(range_x, range_y, f, xlabel="x", ylabel="y", levels=200)

Basically, it will plot 200 contour lines in this visualization and here is the output

The figure above is a little bit better than the one of the first attempt. From the figure, we can somehow recognize that there are 4 local minima. However, since the number of contour lines is too large, the gap between 2 consecutive contour lines near the edge of figure is so small. Sequentially, it makes some illusion that there are more than 4 local minima. Hmm, it is not clear enough for us.

The third attempt

In this attempt, we try to make the number of contour lines small enough for clear visualization but at the same time, the lines must be representative enough. How can we achieve these two goals at the same time? We observe that Himmelblau’s function is always positive since it is the sum of two squares. Hence, we want to have contour lines when function has values 0. We also choose some other values to put contour lines. Now, we have a list of values of function that we want to draw contour lines and the list can be set by levels parameter.

contour(range_x, range_y, f, xlabel="x", ylabel="y",levels=[0,1,2,5,7,15,30,60, 120, 250, 500])

Now, the figure is awesome, we now can see clearly the 4 local minima.

The final attempt

Black and white figure is boring. We want to have some color to make it more exciting.

contour(range_x, range_y, f, xlabel="x", ylabel="y",levels=[0,1,2,5,7,15,30,60, 120, 250, 500], fill=true)

Finally, the contour plot is 90% on par with the one on Wikipedia page.

Closing thoughts

For me, to have good contour plot, there are two points we need to consider

First, we need to find the good value range of variable. Specifically, within this range, the function should have local maxima and minima.
Second, we should somehow predict the min/max value of function so that we can put these values for parameter levels

Some thoughts after Converting ISLR book to Julialang

Fri, 25 Dec 2020 00:00:00 +0000

An Introduction to Statistical Learning with Applications in R (ISLR) is an excellent book for introducing statistical learning to beginners. Its authors are well-known researchers in this field but this book is very intuitive and easy to follow. As a Julialang advocate, I really want to see the Julia-version of this book. For this reason, I have translated all its materials to Julia with some inspiration from the Python project - ISLR-python.

My project is named ISLR.jl. It is a long project since I got the idea 3 years ago but I only seriously devote my time to this project last month. After finishing (nearly) all, I proudly shared it in /r/julia and surprisingly, my post is well received by other advocates. Right now, it is my most starred repo in Github.

ISLR is a must-read book for beginners of data science so re-implementing the book in Julia is kind-of a proxy for me to measure the readiness of Julia eco-system for data science. In this post, I will share my thoughts on this topic.

What do data scientist need?

In my opinion, a programming language and its eco-system are ready for data scientists when they can quickly apply their standard techniques e.g. PCA, linear regression to torture and plot their given data. For this reason, data scientists require

Good library of statistic
Good visualization library

The good of Julia

Julia is getting more and more attention from researchers and developers around the world. Hence, its eco-system is becoming more and more mature. It means all basic need for data science is ready in Julia. For example, if we want to use support vector machine, we have libsvm.jl. For visualization, we have Plots.jl.

In case of not having ready library, we are able to borrow from other programming languages. For example, PyCall.jl allows us to call Python within Julia or PyPlot triggers matplotlib.

The bad of Julia

There are several disadvantage of Julia. Here are my thoughts.

The poor document: Since Julia is quite new, the maintainers of its libraries somehow do not care much about document. It is very hard when we encounter some issues during our coding process. For example, I have to dig into the source code of Plots.jl to find out how to draw vertical/horizontal lines. An other example is that Generalized linear model is extremely popular in stat community but the document of GLM.jl is not that useful, I have to code and guess multiple times. Hopefully, the document becomes more reliable for users in future.
The fragment of library: In Python, matplotlib is a de-facto library for visualization. However, I have found that besides Plots.jl, I still need to use StatsPlots.jl for some plots related to statistic. Moreover, we do not have a standard library which is similar to sklearn in Python. There are some efforts such as MLJ.jl but it is not enough since these libraries are still in early state.
The integrity of eco-system: I believe Julia libraries should be implemented in Julia to use all of its benefits. However, there are still many libraries implemented in Julia but behind the scenes, they call other programming language. For example, ScikitLearn.jl totally calls scikit-learn and some parts of MLJ.jl are used other Python libraries. I understand that this approach can reduce the cost of development but in long run, I think it is better if the entire is done via Julia.

Conclusion

Julia is a fantastic language and I believe it is the future of scientific computing. It cannot replace Python or R in near future but it gradually gets traction of the community. Hopefully, I will see more and more libraries and tools written in Julia.

Learning neural network from real courses

Sat, 09 Nov 2019 00:00:00 +0000

Today, neural network and deep learning are more and more popular but what materials to learn will create a little headache due to the huge numbers of resources online. Different people have different tastes. Someones like online tutorials, some take Coursera courses. For me, I prefer learning neural networks and its application from courses of some big universities (e.g. Stanford, Berkeley, CMU). I like courses with slides, lecture notes, lecture record and homeworks. For this reason, I summarize these courses, categorize them according to their difficulty; I try to keep the post updated.

Beginner

In this section, I will list some courses that provide foundation for neural network learning.

Designing, Visualizing and Understanding Deep Neural Networks from UC Berkeley.
Stats 202: Data Mining and Analysis from Stanford. They also include some solutions for homework.

Intermediate

After learning foundation about neural network, we move to some of its applications

CS224n: Natural Language Processing with Deep Learning from Stanford (Winter 2019). The note and materials are fantastic. Chris Manning is also a big name in the field. Must try course.
CS231n: Convolutional Neural Networks for Visual Recognition from Stanford (2016). An other masterpiece. It will help you understand the whole neural network application for computer vision.

Advanced

We move forward to more advanced technique of deep learning.

CS 285 Deep Reinforcement Learning from Berkeley. It helps us to understand deep RL. There is another course from Stanford but I feel that the materials of Stanford are not that details to follow.
Statistical Learning from Stanford. It is so advanced, be careful.
Convex Optimization. The instructor is Stephen Boyd. He is a god in this field. Follow the sequence if you enjoy the course.

Bonus

Some useful materials for neural network

The Annotated Transformer from Harvard. It is for us to understand Transformer - an attention neural network.
How to Escape Saddle Points Efficiently from Berkeley. I think the title says all, I dont need to explain :)

Hopefully, these courses are useful for your study.

Install Google Chrome inside Docker on Ubuntu 16.04 LTS

Mon, 16 Oct 2017 00:00:00 +0000

Google Chrome is a great browser with a huge number of plug-in. I think I dont need to advertise more for Chrome since it already has a huge popularity nowadays.

Last week, one question raised from my head “How can I install and use Google Chrome browser via Docker?” and this question got stuck inside me for a quite time. Let me explain my scenario a little bit about the reason of this question.

I want to have the second Chrome installed in my computer. Searching around does not give me any positive results.
I want to have the safe browsing. Some examples are testing new interesting plug-ins of Chrome or checking some websites.

All of the requirements are satisfied by Docker but running GUI apps via Docker is quite tough task. After searching around, I have found the instruction of Jessie Frazelle. She created some Dockerfile for the easy of using Chrome via Docker. I follow her guide with some modification to be able to run in my computer.

TL;DR: Install image using her Dockerfile and run

docker run -it --net host --cpuset-cpus 0  -v /tmp/.X11-unix:/tmp/.X11-unix -e DISPLAY -v $HOME/Downloads:/root/Downloads --name chrome --cap-add=SYS_ADMIN jess/chrome

Below are step by step guide and some explanation

Pull her Dockerfile and create and image in our machine docker pull jess/chrome.
Run the docker run command above to open Chrome. Notice that I have changed some arguments from the original recommendation of Jessie.
- To overcome the error Failed to move to new namespace: PID namespaces supported, Network namespace supported, but failed: errno = Operation not permitted, we need to add --cap-add=SYS_ADMIN to assign some admin capacities for docker containers to control the host machine. There are a little documents about this docker option. I am not sure if it makes any harms for the host machine but from the discussion here, we can trust this option over --privileged one.
- I use -e DISPLAY instead of -e DISPLAY=unix$DISPLAY since docker is intelligent enough to automatically select our current display and make use. In her setting, the unix$DISPLAY is a little bit unique for her computer. To check the X display, you can use xauth list to see if it matches the default setting. If not match, we usually get the error

  No protocol specified

  (google-chrome:1): Gtk-WARNING **: cannot open display: unix:0

That is all. Hura, now we can use the second Chrome in our computer within docker.

Shapefile drawing using Python

Tue, 08 Mar 2016 00:00:00 +0000

Last week, I needed to draw some maps and display data on these maps also. I have spent some hours to discover how to complete this task. Thus, in this post, I will summarize step by step to accomplish the initial goal.

Basic drawing

First time first, we need to download the shapefile before doing anything. Luckily, Open street map is an open source data for us to download the shapefile of boundary of many countries. However, there are some services extracted the shapefile of each countries for us. The data is quite complete and ready to use. I also put the demo data of Singapore link1 link2 link3, and they are used to illustrate latter steps (please download the whole three).
Secondly, we need to choose the library to draw. BaseMap is a common lib for drawing shapefile but it is overkilled in my situation i.e. I do not need to use its fancy features. Thus, pyshp is my selection. It is lightweight but still powerful enough to satisfy my requirement. Moreover, installation and usage are as easy as ABC :-) . Last but not least, I have used Python 2.7 and matplotlib 1.4.2 to draw but the conversion to Python 3.x is not hard.
Let us draw :-D

# import libraries
import shapefile as shp
import matplotlib.pyplot as plt

#load shapefile
sf = shp.Reader('SGP_adm0.shp')

plt.figure()
for shape in sf.shapeRecords():
    
    # end index of each components of map
    l = shape.shape.parts
    
    len_l = len(l)  # how many parts of countries i.e. land and islands
    x = [i[0] for i in shape.shape.points[:]] # list of latitude
    y = [i[1] for i in shape.shape.points[:]] # list of longitude
    l.append(len(x)) # ensure the closure of the last component
    for k in xrange(len_l):
        # draw each component of map.
        # l[k] to l[k + 1] is the range of points that make this component
        plt.plot(x[l[k]:l[k + 1]],y[l[k]:l[k + 1]], 'k-')

# display
plt.show()

The final result is not bad.

Display histogram

Using the same setup as above, we will display the histogram in 2D map to illustrate the density. There are some tricks to complete the task so let us view the code first.

import shapefile as shp
import matplotlib.pyplot as plt
import matplotlib as mpl
from mpl_toolkits.axes_grid1 import make_axes_locatable
import numpy as np

def drange(start, stop, step):
    result = list()
    r = start
    i = 0.0
    while r < stop:
        result.append(r)
        i += 1.0
        r = round(start + i * step, 3)
    result.append(r)
    return result

def readLoc(fname):
    f = open(fname, 'r')
    lines = f.readlines()
    f.close()
    x = list()
    y = list()
    for line in lines:
        comp = line.strip('\n').split(',')
        x.append(float(comp[1]))
        y.append(float(comp[0]))
    return x, y

#load shapefile
sf = shp.Reader('SGP_adm0.shp')

plt.figure()
for shape in sf.shapeRecords():
    l = shape.shape.parts
    len_l = len(l)  # how many parts of countries i.e. land and
    x = [i[0] for i in shape.shape.points[:]]
    y = [i[1] for i in shape.shape.points[:]]
    l.append(len(x))
    for k in xrange(len_l):
        plt.plot(x[l[k]:l[k + 1]],y[l[k]:l[k + 1]], 'k-')

lower_x = 103.6
lower_y = 1.2
upper_x = 104.1
upper_y = 1.5
# plot the frame of city
x = [lower_x, lower_x, upper_x, upper_x, lower_x]
y = [lower_y, upper_y, upper_y, lower_y, lower_y]
plt.plot(x, y)

# load data
fname = '1.35_103.9_1.4_103.95'
comp = fname.split('_')
loc = [float(i) for i in comp]
x, y = readLoc('user_density.txt')

# plot the line
area_x = [loc[1], loc[1], loc[3], loc[3], loc[1]]
area_y = [loc[0], loc[2], loc[2], loc[0], loc[0]]
# plot a box in map
plt.plot(area_x, area_y, 'b--')

# histogram
xedges = drange(lower_x, upper_x, 0.01) # 0.01 is the size of bin
yedges = drange(lower_y, upper_y, 0.01)
H, xedges, yedges = np.histogram2d(x, y, bins=(xedges, yedges))

im = plt.imshow(H.T, cmap='OrRd', interpolation='bilinear', origin='low', extent=[xedges[0], xedges[-1], yedges[0], yedges[-1]])

# add label to axis and do not scale unit of x, y axises
plt.ylabel('Latitude', fontsize=20)
plt.xlabel('Longitude', fontsize=20)
ax = plt.gca()
ax.ticklabel_format(useOffset=False)
divider = make_axes_locatable(ax)                          # set size of color bar
cax = divider.append_axes("right", size="5%", pad=0.05)    # set size of color bar
plt.colorbar(im, cax=cax)                                  # set size of color bar 

# finally, show the plot
plt.show()

The file user_density.txt could be downloaded via this link. The data inside this file is generated randomly. Each line of file contains 2 information latitude and longitude of each point and both are separated via comma. Function readLoc is used to load its data.

There are 2 important notes for the above code

I do not use the normal order in numpy manual which is (y, x)

H, xedges, yedges = np.histogram2d(x, y, bins=(xedges, yedges))

We use the transpose matrix of H.T when using imshow. Option bilinear is used to make it look like heatmap.

im = plt.imshow(H.T, cmap='OrRd', interpolation='bilinear', origin='low', extent=[xedges[0], xedges[-1], yedges[0], yedges[-1]])

The final result is quite beautiful

Periodic Mobility Model

Thu, 06 Aug 2015 22:30:00 +0000

Disclamer

This post is giant wall of text and formula. I am not sure it is 100% correct so please read with your caution.

“Friendship and Mobility: User movement in location-based social networks” is an interesting paper which discusses about user movement. I think its intuition is quite clear and trivial but it is (maybe) the first paper which separates the location of users into two clusters named Home and Work. The data analysis of this paper is very nice. After all, the authors proposed a model name Periodic Mobility Model aka PMM to predict the Home and Work location of users. The whole process is follow

Firstly, given the time of day, it will define which is the current cluster (i.e. Home or Work) of user.
Secondly, based on cluster, the location of user is drawn.

Before going to the formula, let me introduce the symbol system

$c_i = k$ means the check-in $i$ is in cluster $k$. In the original paper, $k = 1 ; 2$ because there are only two clusters.
$l_i$ is the location (latitude and logitude) of check-in $i$.
$t_i$ is the time of check-in $i$.
$x_i$ represents the check-in $i$. Formally, it is a pair of location and time of check-in $x_i = (l_i, t_i) $.

The first attempt

The joint distribution of each check-in will be calculated by

\[\begin{align} p(c\_i = k, l\_i, t\_i \| \theta) &= p(l\_i \| c\_i = k, t\_i) \; p(t\_i, c\_i = k) \\\\ &= p(l\_i \| c\_i = k) \; p(c\_i = k \| t\_i ) \; p(t\_i) \end{align}\]

Taking the log becomes

\[\begin{align} \log p(c\_i = k, l\_i, t\_i \| \theta) =& \log p(l\_i \| c\_i = k) + \log p(c\_i = k \| t\_i ) + \log p(t\_i) \\\\ =& \log \mathcal{N}(\mu\_k, \Sigma\_k) + \log N\_k(t\_i) - \log \sum\_{k' = 1}^K N\_{k'}(t\_i) + \log p(t\_i) \\\\ =& - \frac{\log \| \Sigma\_k \|}{2} - \frac{1}{2} (l\_i - \mu\_k)^T \Sigma\_k^{-1} (l\_i - \mu\_k) \\\\ & + \log P\_{c\_k} - \log \sigma\_k - (\frac{\pi}{12})^2 \frac{(t\_i - \tau\_k)^2 }{2 \sigma\_k^2} \\\\ & + \log \sum\_{k'=1}^K N\_{k'}(t\_i) + const \end{align}\]

Follow the EM-algorithm, we must calculate

\[\begin{align} p(c\_i = k \| l\_i\; t\_i) &= \frac{p(l\_i \| c\_i = k) \; p(c\_i = k \| t\_i ) \; p(t\_i)}{\sum\_{k' = 1}^K p(l\_i \; t\_i; \; c\_i= k')} \\\\ &= \frac{ p(l\_i \| c\_i = k) \; p( c\_i = k \| t\_i) }{\sum\_{k'=1}^K p(l\_i \| c\_i = k') \; p(c\_i = k' \| t\_i) } \\\\ &= \gamma\_{ik} \end{align}\]

where

\[\begin{align} p(l\_i \| c\_i = k) &\sim \mathcal{N}(\mu\_k, \Sigma\_k) \\\\ p( c\_i = k \| t\_i) &= \frac{N\_k}{\sum\_{k'=1}^K N\_{k'}} \\\\ N\_k &= \frac{P\_{c\_k}}{\sqrt{2\pi \sigma^2\_k}} \exp ( - $\frac{\pi}{12}$^2 \frac{(t\_i - \tau\_k)^2}{2\sigma^2\_k} ) \end{align}\] \[\begin{align} Q(\theta, \; \theta^{old}) &= \mathbb{E}\_{C \| X, \theta^{old}} [ L (\theta; X, C)] \\\\ &= \sum\_{i=1}^N \sum\_{k=1}^K p(c\_i = k \| x\_i, \theta^{old}) \; \log p(x\_i, \; c\_i = k \| \theta) \\\\ \end{align}\]

Updating $ \mu_k $ and $\Sigma_k$ are similar to Gaussian mixture model so we have

\[\begin{align} \frac{\partial Q}{\partial \mu\_k} &= \sum\_{i=1}^N \gamma\_{ik}[ \Sigma\_k^{-1} (l\_i - \mu\_k)] = 0 \\\\ \mu\_k &= (\sum\_{i=1}^N \gamma\_{ik})^{-1} \; \sum\_{i=1}^N \gamma\_{ik} \; l\_i \end{align}\] \[\begin{align} \frac{\partial Q}{\partial \Sigma\_k} &= \sum\_{i=1}^N \frac{\gamma\_{ik}}{2} [ -\Sigma\_k^{-1} + \Sigma\_k^{-1} (l\_i - \mu\_k)(l\_i - \mu\_k)^T \Sigma\_k^{-1}] = 0 \\\\ \Sigma\_k &=\frac{\sum\_{i=1}^N \gamma\_{ik} (l\_i - \mu\_k) \; (l\_i - \mu\_k)^T }{ \sum\_{i=1}^N \gamma\_{ik}} \\\\ \end{align}\]

We take the differentiation of Q over other parameters

\[\begin{align} \frac{\partial Q}{\partial P\_{c\_k}} &= \sum\_{i=1}^N \frac{\gamma\_{ik}}{P\_{c\_k}} [1 + \frac{N\_k(t\_i)}{\sum\_{k'=1}^K N\_k' (t\_i)}] \\\\ \end{align}\] \[\begin{align} \frac{\partial Q}{\partial \tau\_k} &=\frac{1}{\sigma\_k^2} (\frac{\pi}{12})^2 \sum\_{i=1}^N \gamma\_{ik}(t\_i - \tau\_k) [1 + \frac{N\_k(t\_i)}{\sum\_{k'=1}^K N\_k' (t\_i)}] \\\\ \end{align}\] \[\begin{align} \frac{\partial Q}{\partial \sigma\_k} &= \sum\_{i=1}^N \gamma\_{ik} [1 + \frac{N\_k(t\_i)}{\sum\_{k'=1}^K N\_k' (t\_i)}] [-\sigma\_k^{-1} + (\frac{\pi}{12})^2 \frac{(t\_i - \tau\_k)^2}{\sigma\_k^3} ] \\\\ \end{align}\]

Setting them equal to 0 cannot give us a closed form solution because $N_k(t_i)$ contains all other parameters inside. Moreover, there are some conditions that we need to follow. First of all, $P_{c_k}$ is positive and the sum over k must be equal to 1. In other word, it is belong to the simplex. Secondly, $\tau_k$ is time so it must in range of 0 and 24. Finally, $\sigma_k$ should be positive also.

Optimizing Q with these constraints is the pain in the neck. Moreover, I could not find the closed form solution. It is quite weird because the authors claimed that they could derive the closed form solution.

Another look of model

\[\begin{align} p(c\_i = k, l\_i, t\_i \| \theta) &= p(l\_i \| c\_i = k, t\_i) \; p(t\_i, c\_i = k) \\\\ &= p(l\_i \| c\_i = k) \; p(t\_i \| c\_i = k) \; p(c\_i = k) \end{align}\]

where

\[\begin{align} p(l\_i \| c\_i = k) &\sim \mathbb{N}(\mu\_k, \Sigma\_k) \\\\ p(t\_i \| c\_i = k) &=\frac{1}{\sqrt{2\pi \sigma^2\_k}} \exp ( - $\frac{\pi}{12}$^2 \frac{(t\_i - \tau\_k)^2}{2\sigma^2\_k} ) \\\\ p(c\_i = k) &= P\_{c\_k} \end{align}\]

This viewpoint does not follow the original intuition of model. In this modification, user will choose his/her cluster first (Home or Work) and from this cluster, he/she will choose time and location of checkin.

To calculate for the first step of EM $\begin{align} p(c\_i = k \| l\_i\; t\_i) &= \frac{ p(l\_i \; t\_i \| c\_i = k) \; p(c\_i = k)}{\sum\_{k' = 1}^K p(l\_i \; t\_i; \; c\_i= k')} \\\\ &= \frac{ p(l\_i \| c\_i = k) \; p(t\_i \| c\_i = k) \; p(c\_i = k)}{\sum\_{k'=1}^K p(l\_i \| c\_i = k') \; p(t\_i \| c\_i = k') \; p(c\_i = k')} \\\\ &= \gamma\_{ik} \end{align}$

\[\begin{align} \log p(l\_i, \; t\_i, \; c\_i = k) &= \log p(l\_i \| c\_i = k) + \log p(t\_i \| c\_i = k) + \log P\_{c\_k} \\\\ &= -\frac{1}{2} \log \| \Sigma\_k\| -\frac{1}{2} (l\_i - \mu\_k)^T \Sigma\_k^{-1} (l\_i - \mu\_k) - \log \sigma\_k - (\frac{\pi}{12})^2 \frac{(t\_i - \tau\_k)^2}{2 \sigma^2\_k} + \log P\_{c\_k} + const \end{align}\] \[\begin{align} Q(\theta, \; \theta^{old}) &= \mathbb{E}\_{C \| X, \theta^{old}} [ L (\theta; X, C)] \\\\ &= \sum\_{i=1}^N \sum\_{k=1}^K p(c\_i = k \| x\_i, \theta^{old}) \; \log p(x\_i, \; c\_i = k \| \theta) \\\\ &= \sum\_{i=1}^N \sum\_{k=1}^K \gamma\_{ik} [ -\frac{1}{2} \log \| \Sigma\_k\| -\frac{1}{2} (l\_i - \mu\_k)^T \Sigma\_k^{-1} (l\_i - \mu\_k) - \log \sigma\_k - (\frac{\pi}{12})^2 \frac{(t\_i - \tau\_k)^2}{2 \sigma^2\_k} + \log P\_{c\_k}] \end{align}\]

We have one constraint that $\sum\_{k=1}^K P\_{c\_k} = 1$ so apply Lagrange multiplier we have $Q(\theta, \; \theta^{old})= \sum\_{i=1}^N \sum\_{k=1}^K \gamma\_{ik} [ -\frac{1}{2} \log \| \Sigma\_k\| -\frac{1}{2} (l\_i - \mu\_k)^T \Sigma\_k^{-1} (l\_i - \mu\_k) - \log \sigma\_k - (\frac{\pi}{12})^2 \frac{(t\_i - \tau\_k)^2}{2 \sigma^2\_k} + \log P\_{c\_k}] + \lambda ( \sum\_{k=1}^K P\_{c\_k} - 1)$

To optimize $P_{c_k}$ we take the derivative of Q and set it to 0. Moreover, taking advantage of the contraint to infer the value of $\lambda$ and plug it back to find the final value of $P_{c_k}$

\[\begin{align} \frac{\partial Q}{\partial P\_{c\_k}} &= \sum\_{i=1}^N \gamma\_{ik} \frac{1}{P\_{c\_k}} + \lambda = 0 \\\\ \lambda \; P\_{c\_k} &=- \sum\_{i=1}^N\gamma\_{ik} \\\\ \lambda &= - \sum\_{i=1}^N \sum\_{k=1}^K \gamma\_{ik} \\\\ P\_{c\_k} &= \frac{\sum\_{i=1}^N\gamma\_{ik}}{\sum\_{i=1}^N \sum\_{k=1}^K \gamma\_{ik} } \end{align}\]

We do the same to optimize $\tau_k$

\[\begin{align} \frac{\partial Q}{\partial \tau\_k} &= \sum\_{i=1}^N \gamma\_{ik}[( \frac{\pi}{12})^2 2 \frac{t\_i - \tau\_k}{2 \sigma^2\_k} ] = 0 \\\\ \sum\_{i=1}^N \gamma\_{ik} \; t\_i &= \sum\_{i=1}^N \gamma\_{ik} \tau\_k \\\\ \tau\_k &= \frac{\sum\_{i=1}^N \gamma\_{ik} \; t\_i}{ \sum\_{i=1}^N \gamma\_{ik}} \\\\ \end{align}\]

Note that using this formula $\tau_k$ is still in range 0 and 24.

Below is the update rule for $\sigma_k^2$

\[\begin{align} \frac{\partial Q}{\partial \sigma\_k} &= \sum\_{i=1}^N \gamma\_{ik}[-\frac{1}{\sigma\_k} + \frac{\pi^2 \; (t\_i - \tau\_k)^2}{12^2\; \sigma^3\_k} ] = 0 \\\\ \sigma\_k^2 &= (\frac{\pi}{12})^2 \frac{ \sum\_{i=1}^N \gamma\_{ik} (t\_i - \tau\_k)^2 }{ \sum\_{i=1}^N \gamma\_{ik} } \end{align}\]

Updating $ \mu_k $ follows the rule

Last but not least, $\Sigma_k$

\[\begin{align} \frac{\partial Q}{\partial \Sigma\_k} &= \sum\_{i=1}^N \frac{\gamma\_{ik}}{2} [ -\Sigma\_k^{-1} + \Sigma\_k^{-1} (l\_i - \mu\_k)(l\_i - \mu\_k)^T \Sigma\_k^{-1}] = 0 \\\\ \Sigma\_k &=\frac{\sum\_{i=1}^N \gamma\_{ik} (l\_i - \mu\_k) \; (l\_i - \mu\_k)^T }{ \sum\_{i=1}^N \gamma\_{ik}} \\\\ \end{align}\]

Stochastic Variance Reduced Gradient

Thu, 16 Jul 2015 21:30:00 +0000

Optimization has a long history but it is still developed by many researchers around the world. With the raise of machine learning and large dataset nowaday, optimization is now evoluating as the heart of parameter learning process. Researchers are trying to make it faster and more efficient.

Among all algorithms, Stochastic Gradient Descent(SGD) is considered as a good choice. It does not require a huge memory and each of its step is cheap. However, SGD is fast to converge to its local neighborhood rather than the optimal solution. Stochastic Variance Reduced Gradient(a.k.a SVRG) is a recent research work and could be viewed as a modified version of SGD. This algorithm will converge faster to optimal value by reducing the variance.

The proposed algorithm SVRG is simple and elegent so I try to implement it and compare with SGD. Actually, we can download an implementation of SVRG from author website but it is written in C++ and not easy to match with the paper. For this reason, I try to translate the algorithm from paper to program using Julia ver 0.3. All of my implementation could be found in this link.

First of all, I create a training data of n points:

\[w = [1; 2; 3; 4]\] \[x = r + ra\] \[y = w' * x\]

where $r \sim U(-50, 50)$ and $ra \sim \mathcal{N}(0, 1)$. It means r is drawn from uniform distribution from -50 to 50. This task is implemented in file genData.jl.

Secondly, I will minimize the squared loss of training data to get back vector w. Formally, we have

\[P = \frac{1}{n} \sum\_{i=1}^n \phi\_i(w)\]

where

\[\phi\_i = (w^T * x\_i - y\_i)^2\]

Finally, $w\_{opt} = argmin\_w P$. The algorithm is in svrg.jl.

In my running, n is selected to be 100 000, update frequency $m = 2n$, learning rate is 0.0001. To run the code, we could type to terminal

julia main.jl

The output will appear

341317718146
1973361489718
7982712672765
1467179096912
26535395599187
986501037489642
284287345559172
8046052335362346
8103649634668636
15111556209992194
10221744358936354
03395696393082646
0030038768910633142
000374384037301745
9358523058164334e-5
338155215045428e-6
[0.9999890714244761,2.0000090291730728,3.00007147492654,3.99995174501473]

All the lines are value of objective function per each iteration and the last line is the final $w$. It is nearly similar to the groundtruth so my implementation is 95% correct :-).

Some thought of SVRG

The procedure is easy to understand and implement.
The proof is not so hard to follow but we need to be familiar with SGD first.
The learning rate of algorithm is still a parameter to tune. In the paper, authors advises us to chose learning rate $\eta < \frac{1}{L}$ with L is Lipschitz constant. An inappropriate $\eta$ could create divergence to our program. For example, applying learning rate $\eta = 0.01$ will make the objective function increase over time. This condition will help SVRG to converge but in practice, finding L is hard.
In backtracking SGD, the learning rate must be decreased over time to ensure the convergence. However, SVRG will keep the suitable value of learning rate until the convergence so it could achieve the better result by reducing the variance.

Logistic Regression: does label matter?

Sun, 04 Jan 2015 00:00:00 +0000

Logistic Regression

Changing to blue Dropbox icons

Fri, 12 Dec 2014 00:00:00 +0000

Dropbox in Linux has been upgraded to version 3 with a lot of changes. However, it also uses a black/while color for the indicator. These icons, for me, are very ugly. I love the blue icons of Dropbox indicator so much so I decide to switch back to the blue versions.

First of all, we need to download an older version of Dropbox such as 2.10.52.
Then extract it. After extract, go to

downloadPath/.dropbox-dist/dropbox-lnx.x86-2.10.52/images/hicolor/16x16/status/

and move all of these images to override your current Dropbox icons in

~/.dropbox-dist/dropbox-lnx.x86_64-3.0.3/images/hicolor/16x16/status

Note: After extracting, it is the hidden folder so press Ctrl-H to show or just use terminal to navigate.

Finally, open terminal and restart Dropbox

dropbox stop

dropbox start

Done. Dropbox icons are blue again :-)