On Certain Limitations of Recursive Representation Model

Stanisław Jastrzębski,

Igor Sieradzki

There is a strong research eort towards developing models that can achieve state-of-the-art results without sacricing interpretability and simplicity. One of such is recently proposed Recursive Random Support Vector Machine (R2SVM) model, which is composed of stacked linear models. R2SVM was reported to learn deep representations outperforming many strong classi-ers like Deep Convolutional Neural Network. In this paper we try to analyze it both from theoretical and empirical perspective and show its important limitations. Analysis of similar model Deep Representation Extreme Learning Machine (DrELM) is also included. It is concluded that models in its current form achieves lower accuracy scores than Support Vector Machine with Radial Basis Function kernel.
Słowa kluczowe: support vector machines, random recursive support vector machine, extreme learning machine, representation learning, stacked generalization

[1] Bengio Y., Learning deep architectures for AI. Foundations and Trends in Machine Learning, 2009, 2.
[2] Podolak I.T., Roman A., Theoretical foundations and experimental results for a hierarchical classier with overlapping clusters. Computational Intelligence, 2013, 29 (2), pp. 357388.
[3] Czarnecki W.M., Tabor J., Extreme entropy machines: Robust information theoretic classication. arXiv preprint arXiv:1501.05279, 2015.
[4] Huang G.B., Zhu Q.Y., Siew C.K., Extreme learning machine: Theory and applications. Neurocomputing, 2006, 70 (13), pp. 489501.
[5] Tissera M., McDonnell M., Deep extreme learning machines for classication. In: Proceedings of ELM-2014 Volume 1. vol. 3 of Proceedings in Adaptation, Learning and Optimization. Springer International Publishing 2015, pp. 345354.
[6] Vinyals O., Jia Y., Deng L., Darrell T., Learning with recursive perceptual representations. In: Advances in Neural Information Processing Systems 25. Curran Associates, Inc. 2012, pp. 28252833.47
[7] Yu W., Zhuang F., He Q., Shi Z., Learning deep representations via extreme
learning machines. Neurocomputing, 2015, 149, pp. 308315.

[8] Wolpert D.H., Stacked generalization. Neural Networks, 1992, 5, pp. 241259.
[9] Cortes C., Vapnik V., Support-vector networks. Machine Learning, 1995, 20 (3), pp. 273297.
[10] Tang Y., Deep learning using linear support vector machines. In: In ICML, 2013.
[11] Lichman M., UCI machine learning repository, 2013.
[12] Chang C.C., Lin C.J., LIBSVM: A library for support vector machines. ACM Transactions on Intelligent Systems and Technology, 2011, 2, pp. 27:127:27.
[13] Pedregosa F., Varoquaux G., Gramfort A., Michel V., Thirion B., Grisel O., Blondel M., Prettenhofer P., Weiss R., Dubourg V., Vanderplas J., Passos A., Cournapeau D., Brucher M., Perrot M., Duchesnay E., Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 2011, 12, pp. 2825

Czasopismo ukazuje się w sposób ciągły on-line.
Pierwotną formą czasopisma jest wersja elektroniczna.

Wersja papierowa czasopisma dostępna na www.wuj.pl