Machine Learning Series

Putting pieces of Machine Learning (ML) together is not easy. It involves knowledge from multiple disciplines. Let’s try to approach the topics more systematically. Some people may avoid math in ML. However, you will not go too far. Fortunately, many are high school math. The real issue is whether it is introduced properly. So I will not shy from it but I will give you enough background for it.

Fundamental

ML algorithms cover areas including information theory, probability, data distribution, Bayesian inference, matrix decomposition, and statistics. We will cover them in this fundamental article first.

Machine Learning Summary (Fundamental)

The second article covers the basic ML algorithms including linear regression, classification, kernels, and Bayesian linear regression. You may know some of the topics already. So we will be brief for those well-understood areas. But we will spend more time in Gaussian Process and SVM since the concept is relatively complex. We will also detail many different approaches in the clustering and decision tree. I will also cover basic ML concepts including the cost function, regularization, and model optimization (MLE, MAP). We will also cover some deep learning topics including normalization, LSTM, and energy-based models like RBM.

Machine Learning Summary -Algorithm

These two articles involve many proves. But in order not to overwhelm you with details. It will separate it into a different article.

Machine Learning Summary (Proof & Terms)

Linear algebra

You may overwhelm by math in many research papers. Probability distribution, calculus, linear algebra, statistics, and optimization are the major math that we need to learn in ML. The last two are huge topics so I will cover them whenever it is needed only. If you have problems in understanding the linear algebra so far in the previous articles, the article below can refresh your memory. Pay special attention to areas related to matrix factorization, it will be used heavily in the dimensional reduction in ML.

Machine Learning & Linear Algebra

Eigenvector may look simple but it is a powerful concept and play a major role in ML. Google ranks page based on a similar concept. Here we will dedicate an article on it.

Machine Learning & Linear Algebra — Eigenvalue and eigenvector

If you are irritated in hearing “the matrix is positive semidefinite and therefore …”, you may take a look here on some of the properties for some special matrics constantly used in ML.

Machine Learning & Linear Algebra — Special Matrices

SVD and PCA are a core topic in ML. Imagine that we can use linear algebra to untangle data to discover the structure of the data. Instead of processing raw data, we start learning knowledge.

Machine Learning — Singular Value Decomposition (SVD) & Principal Component Analysis (PCA)

Optimization

Many AI problems are rephrased as an optimization problem. People may already familiar with conventional methods like gradient descent. We will cover more advanced optimization methods here. The first one is the Lagrange multiplier in optimizing problems under constraints.

Machine Learning — Lagrange multiplier & Dual decomposition

Many machine learning (ML) problems deal with a dilemma: If we know latent factor A, we can determine latent factor B. Or, if we know B, we can determine A. Either way solves the ML problem. Unfortunately, we don’t know A or B. But in ML, it can be solved by one powerful algorithm called Expectation-Maximization Algorithm (EM).

Machine Learning — Expectation-Maximization Algorithm (EM)

Graphical model

The Graphical model (GM) is a branch of ML which uses a graph to represent a domain problem. Many ML algorithms can be generalized and discussed as GM. Studying it allows us a bird’s eye view on many ML algorithms.

Machine Learning — Graphical Model

Inference

Inference is about making predictions from data that we know. Here, we find an exact inference when GM is known.

Machine Learning — Graphical Model Exact inference (Variable elimination, belief propagation…

But finding an exact solution is usually hard. Here, we will approximate it with linear programming and dual decomposition.

Machine Learning — MAP inference with Linear Programming & Dual decomposition

Next, we switch our gear to find the solution through re-establish p(x). This can be done through sampling with methods like Monte Carlo Sampling, importance sampling, Metropolis algorithm and Gibbs sampling that are commonly used in ML.

Machine Learning — Sampling-based Inference

In ML, there are two major approximation approaches. They are sampling and variational inference. In the next article, we model the posterior with variational inference with a method like Mean Field Variational Inference.

Machine Learning — Variational Inference

Finally, we will conclude our GM with a discussion on how to learn the model.

Machine Learning — Learning Graphical Model

ML topics

Finally, we will look into some advanced ML algorithms.

Machine Learning — Hidden Markov Model (HMM)

Machine Learning — Latent Dirichlet Allocation LDA

Machine Learning — Visualization, Dimension Reduction & Anomaly detection

Machine Learning - Recommender System

Github (These are old and less polished articles)

"Machine learning - Regression, Logistic regression, SVM, MAP and Kernels"

"Machine learning - PCA, SVD, Matrix factorization and Latent factor model"

"Machine learning - Clustering, Density based clustering and SOM"

"Machine learning - Recommendation, Collaborative filtering and ranking"

"Machine learning - Naive bayes classifier, Bayesian inference"

"Machine learning - Anomaly detection"

"Machine learning - Decision tree, Random forest, Ensemble methods and Beam searach"

"Machine learning - Gaussian Process"

"Machine learning - Visualization, multi-dimensional scaling, Sammon mapping, IsoMap and t-sne"

"Machine learning - Nonsupervised and semi-supervised learning"

"Machine learning - Restricted Boltzmann Machines."