Models properties

Bias & variance

Course:

• Principles and Methodology of Machine Learning - David Gianazza (slides 233-236)

Some sources:

• Bias–variance tradeoff - Wikipedia
• Bias vs Variance - Gunjan Agicha (Medium)

High bias / low variance models generalize better on new tasks.

Low bias / high variance models perform better on the task they have been trained on.

Parametric & non-parametric

• What exactly is the difference between a parametric and non-parametric model? - stats.stackexchange
• Parametric and nonparametric ML algorithms - Machine learning mastery

In very brief:

• Parametric models have fixed set of parameters. Their complexity is determined by the number of parameters. The parameters are fitted to the data. Popular parametric models:
  • Linear regression, Logistic regression
  • Neural networks
• Non-parametric models don't have a fixed parameters, but rather are built from the data. Their complexity depends on the data on which they are built on.
  • Tree-based models: CART, random forests, gradient boosting regressor or classifier
  • K nearest neighbors
  • Support Vector Machines

Computational complexity

• Computational Complexity of ML Models - Paritosh Kumar (Medium)