Monday 19 December 2016

Machine learning four years after the turning point


In May 2012 I wrote a note titled "Machine at its turning point" to argue for the new wave of machine learning in that we do not need to worry about having a convex loss but rather be happy with non-convex ones. At the time I did not know about AlexNet and its record-breaking result on the ImageNet benchmark. It was published 7 months later in NIPS'12.

AlexNet was truely a turning point for machine learning. It declared the winning of deep neural nets over others, which were combination of clever manual feature engineering and some variants of SVMs or random forests. AlexNet is remarkable in many ways: Dropout, rectifier linear units, end-to-end training on massive data with GPUs, data augmentation and carefully designed convolutional nets.

It was the year that Yann LeCun posted his complaints about the computer vision community, but quickly retracted his boycott given the aftershock of AlexNet.

Recently, there has been an interesting comment floating around: In machine learning, we ask what we can do for neural networks, and in applied domains, we ask what can neural networks do for X. And the list of Xs keeps growing from cognitive domain to non-cognitive domains. Andrew Ng made an interesting point that for domains where humans can do well to map A to B in less than a second, it is ripe for machine automation.

This year also marks the 10th year after Deep Belief Nets, the model that announces the beginning of the current wave of neural nets. Early this year, AlhaGo of DeepMind defeated one of the best Go champions 4 to 1, officially ending the superiority of human on this ancient game. AlphaGo is a mixture of convolutional nets to read the board positions and evaluate the moves, and random tree-search moves.

Many things have changed since 2012. It is clear that supervised learning works if we have sufficient labels without pre-training. Unsupervised learning, after an initial burst with Boltzmann machines and Autoencoders, failed to deliver.  There are new interesting developments, however, with Variational Autoencoder (VAE) and Generative Adversarial Nets (GAN), both invented in 2014. At this point, GAN is the best technique to generate faithful images. It is considered by Yann LeCun as one of the best ideas in recent years.

The machine learning community has witnessed 10-15 year mini-cycles. Neural networks, graphical models, kernel methods, statistical relational learning and currently, deep learning. So what is up for deep learning? If we consider 2006 as the year of beginning of current deep learning, then it is already 10 years, enough for a mini-cycle. But if we consider 2012 as the true landmark, then we have 6 more years to count.

Like other methodologies, deep learning will eventually morph into something else in 5 years time. We may call it by other names. With programming becomes reasonably effortless and with the availability of powerful CPUs/GPUs designed specifically for deep learning, the low hanging fruits will soon be picked up.

Practice-wise, as feature engineering is an unsung hero of machine learning prior to 2012, architecture engineering is at the core of deep learning these days.

It is also time for the hardcores. Data efficiency, statistics, geometry, information theory, Bayesian and other "serious" topics. Like any major progresses in science and engineering, nothing really occurs over night. At this point, deep learning is already mixed with graphical models, planning, inference, symbolic reasoning, memory, execution, Bayesian among other things. All together, something fancy will happen, just like what I noted about Conditional Random Fields years ago, that it is the combination of incremental innovations that pushes the boundary of certain field to a critical point. It also concurs with the idea of emergence intelligence, where human intelligence is really the emerging product of many small advances over apes.

For a more comprehensive review, see my recent tutorials at AI'16 on the topic. Some incremental innovations were produced at PRaDA (Deakin University), listed below.

Work by us:
  • Multilevel Anomaly Detection for Mixed Data, K Do, T Tran, S Venkatesh, arXiv preprint arXiv: 1610.06249.
  • A deep learning model for estimating story points, M Choetkiertikul, HK Dam, T Tran, T Pham, A Ghose, T Menzies, arXiv preprint arXiv: 1609.00489
  • Deepr: A Convolutional Net for Medical Records, Phuoc Nguyen, Truyen Tran, Nilmini Wickramasinghe, Svetha Venkatesh, To appear in IEEE Journal of Biomedical and Health Informatics.
  • Column Networks for Collective Classification, T Pham, T Tran, D Phung, S Venkatesh, AAAI'17
  • DeepSoft: A vision for a deep model of software, Hoa Khanh Dam, Truyen Tran, John Grundy and Aditya Ghose, FSE VaR 2016.
  • Faster Training of Very Deep Networks Via p-Norm Gates, Trang Pham, Truyen Tran, Dinh Phung, Svetha Venkatesh, ICPR'16.
  • Hierarchical semi-Markov conditional random fields for deep recursive sequential data, Truyen Tran, Dinh Phung, Hung Bui, Svetha Venkatesh, To appear in Artificial Intelligence.
  • DeepCare: A Deep Dynamic Memory Model for Predictive Medicine, Trang Pham, Truyen Tran, Dinh Phung, Svetha Venkatesh, PAKDD'16, Auckland, NZ, April 2016. 
  • Neural Choice by Elimination via Highway Networks, Truyen Tran, Dinh Phung and Svetha Venkatesh,  PAKDD workshop on Biologically Inspired Techniques for Data Mining (BDM'16), April 19-22 2016, Auckland, NZ.
  • Tensor-variate Restricted Boltzmann Machines, Tu D. Nguyen, Truyen Tran, D. Phung, and S. Venkatesh, AAAI 2015
  • Thurstonian Boltzmann machines: Learning from multiple inequalities, Truyen Tran, D. Phung, and S. Venkatesh, In Proc. of 30th International Conference in Machine Learning (ICML’13), Atlanta, USA, June, 2013.

103 comments:

  1. Replies
    1. reat Article. Thank you for sharing! Really an awesome post for every one.

      IEEE Final Year projects Project Centers in India are consistently sought after. Final Year Students Projects take a shot at them to improve their aptitudes, while specialists like the enjoyment in interfering with innovation. For experts, it's an alternate ball game through and through. Smaller than expected IEEE Final Year project centers ground for all fragments of CSE & IT engineers hoping to assemble. Final Year Projects for CSE It gives you tips and rules that is progressively critical to consider while choosing any final year project point.

      JavaScript Online Training in India

      JavaScript Training in India

      The Angular Training covers a wide range of topics including Components, Angular Directives, Angular Services, Pipes, security fundamentals, Routing, and Angular programmability. The new Angular TRaining will lay the foundation you need to specialise in Single Page Application developer. Angular Training

      Delete
  2. Very beautiful blog..Thank you for sharing this amazing post.
    Visit Best Machine Learning Training in Jaipur

    ReplyDelete
  3. Thanks for your effort to put this information here. I think its useful.for more information about machine learning go through this link. machine learning training in hyderabad

    ReplyDelete
  4. I can't understand why MNIST is a common example to teach machine learning. Some other application like text or voice learning will be interesting.

    Machine Learning Training in Chennai | Machine Learning Training Course in Chennai

    ReplyDelete
  5. very informative blog and useful article thank you for sharing with us , keep posting Data Science online Training Bangalore

    ReplyDelete
  6. You do not know a method or one other because you solely have one information level to work with.This is great blog. If you want to know more about this visit our Cloud Certified Site.

    ReplyDelete
  7. "• Nice and good article. It is very useful for me to learn and understand easily. Thanks for sharing your valuable information and time. Please keep updating IOT Online Training
    "

    ReplyDelete
  8. Nice information Well said sir keep updating thank you.
    Machine Learning Training in Hyderabad

    ReplyDelete
  9. I think things like this are really interesting. I absolutely love to find unique places like this. It really looks super creepy though!!

    I shared here the Useful Guide for Machine learning Please check these too : Useful guide to learn Machine learning

    ReplyDelete
  10. Nice to be visiting your blog again, it has been months for me. Well this article that i've been waited for so long. I need this article to complete my assignment in the college, and it has same topic with your article. Thanks, great share. artificial intelligence

    ReplyDelete
  11. very good informative blog & useful to me
    thank you...keep posting
    Machine Learning Training

    ReplyDelete
  12. Hi,nice information is there in your blog it is valuable i read so many regarding this .For additional information of machine learning you can visit us."Machine Learning"

    ReplyDelete
  13. Thanks for the info....

    Real Trainings provide all IT-Training Course information in Hyderabad, Bangalore, Chennai . Here students can Compare all Courses with all detailed information. In Machine Learning Training we provide courses like Machine Learning, Machine learning online training etc...

    ReplyDelete
  14. Deep Learning has gotten really popular and with this course candidates can gain knowledge about things like supervised and unsupervised learning. There are several institutes which provide Deep Learning in Chennai.

    ReplyDelete
  15. Thank you for sharing information with us. nice blog...
    Machine learning course in Mumbai

    ReplyDelete
  16. Interesting Article. Hoping that you will continue posting an article having a useful information. Machine Learning Internship In Jaipur

    ReplyDelete
  17. Great post! If you need to know everything regarding artificial intelligence and machine learning , visit Turing Tribe

    ReplyDelete

  18. Thanks for sharing this valuable information and we collected some information from this blog.
    Machine Learning Training Course in Noida

    ReplyDelete
  19. This is also a very good post which I really enjoyed reading. It is not every day that I have the possibility to see something like this..
    machine learning course in bangalore

    ReplyDelete
  20. Really It is very useful information for us. thanks for sharing.
    Machine Learning Training in Hyderabad

    ReplyDelete
  21. Thanks for sharing such an awesome Information with us

    I Got Job in my dream company with decent 12 Lacks Per Annum salary, I have learned this world most demanding course out there in the current IT Market from the Data Science Training in btm experts who helped me a lot to achieve my dreams comes true. Really worth trying

    ReplyDelete
  22. The content is very clear.Thanks for posting.I always like your posts.Keep updating
    Best Python Training in BTM Layout

    ReplyDelete
  23. This is a splendid website! I"m extremely content with the remarks!ExcelR Courses In Business Analytics

    ReplyDelete
  24. Really a awesome blog for the freshers. Thanks for posting the information.
    Data Science Training in Delhi
    Data Science Course in Delhi

    ReplyDelete
  25. Wow such a informative post which i need to explain my team member who working on it - It's my pleasure to visit this post.
    allow us to introduce our self we are here to make your wedding perfect and special at limo service Highlands Ranch

    ReplyDelete
  26. Great post i must say and thanks for the information. Education is definitely a sticky subject. However, is still among the leading topics of our time. I appreciate your post and look forward to more.
    data scientist course in pune

    ReplyDelete

  27. The information you have posted is very useful. The sites you have referred was good. Thanks for sharing. ExcelR Data Science Course In Pune

    ReplyDelete
  28. As always your articles do inspire me. Every single detail you have posted was great. ExcelR Data Science Courses

    ReplyDelete
  29. This was really one of my favorite website. ExcelR Machine Learning Course Pune Please keep on posting.

    ReplyDelete
  30. This comment has been removed by the author.

    ReplyDelete
  31. more about the blog is good but if you was seen any other blog the content should be failed to fulfill the users requests on that field,but in this blog the content will fulfill the users who related to search this field.... i need some more info
    Ai & Artificial Intelligence Course in Chennai
    PHP Training in Chennai
    Ethical Hacking Course in Chennai Blue Prism Training in Chennai
    UiPath Training in Chennai

    ReplyDelete
  32. You actually make it look so easy with your performance but I find this matter to be actually something which I think I would never comprehend. It seems too complicated and extremely broad for me. I'm looking forward for your next post, I’ll try to get the hang of it!

    Data Science Course

    ReplyDelete
  33. I really enjoy simply reading all of your weblogs. Simply wanted to inform you that you have people like me who appreciate your work. Definitely a great post. Hats off to you! The information that you have provided is very helpful.
    Data Science Course in Hyderabad

    ReplyDelete
  34. Really nice and interesting post. I was looking for this kind of information and enjoyed reading this one. Keep posting. Thanks for sharing.
    Business Analytics Course in Hyderabad

    ReplyDelete
  35. I have express a few of the articles on your website now, and I really like your style of blogging. I added it to my favorite’s blog site list and will be checking back soon…
    Data Scientist Courses

    ReplyDelete
  36. Randomly found your blog. Your blog is away-some. Get Machine Learning course in Pune, Mumbai. data science courses in Mumbai, Pune. And you can get training from one of the best training for other courses also like Artificial Intelligence, rpa training in Mumbai, Blockchain training in Mumbai, Machine Learning in Mumbai, SAS Training, Python Programming etc.

    ReplyDelete
  37. Really nice and interesting post. I was looking for this kind of information and enjoyed reading this one. Keep posting. Thanks for sharing
    data scientist course

    ReplyDelete
  38. Very nice blog and articles. I am realy very happy to visit your blog. Now I am found which I actually want. I check your blog everyday and try to learn something from your blog. Thank you and waiting for your new post.data science course

    ReplyDelete
  39. Randomly found your blog. You have given information is helpful. Machine learning course in Mumbai

    ReplyDelete
  40. Amazing post found to be very impressive while going through this post. Thanks for sharing and keep posting such an informative content.

    360DigiTMG Data Analytics Course

    ReplyDelete
  41. Nice information on Machine learns and data speak thank you for sharing.
    Data Science Course in Hyderabad

    ReplyDelete
  42. I'd love to thank you for the efforts you've made in composing this post. I hope the same best work out of you later on too. I wished to thank you with this particular sites! Thank you for sharing. Fantastic sites!
    360DigiTMG Data Science Course in Bangalore

    ReplyDelete
  43. This is a great post. This post gives a truly quality information. I am certainly going to look into it. Really very helpful tips are supplied here. Thank you so much. Keep up the great works
    360DigiTMG Data Science Training in Bangalore

    ReplyDelete
  44. Really nice and interesting post. I was looking for this kind of information and enjoyed reading this one. Keep posting. Thanks for sharing.
    360DigiTMG

    ReplyDelete
  45. Thankyou for this wondrous post, I am cheerful I watched this site on yahoo.
    artificial intelligence course in delhi

    ReplyDelete
  46. If you don't mind, then continue this excellent work and expect more from your great blog posts
    artificial intelligence courses in delhi

    ReplyDelete
  47. Top quality article with knowledgeable information, found very useful thanks for sharing waiting for next blog.
    Data Analytics Course Online 360DigiTMG

    ReplyDelete
  48. Tremendous blog quite easy to grasp the subject since the content is very simple to understand. Obviously, this helps the participants to engage themselves in to the subject without much difficulty. Hope you further educate the readers in the same manner and keep sharing the content as always you do.

    Data Science training in Raipur

    ReplyDelete
  49. Very great post. I just discovered your blog and needed to state that I have truly appreciated perusing your blog entries. Any way I'll be buying in to your feed and I trust you post again soon.
    data scientist training hyderabad

    ReplyDelete
  50. I am certain that this will help a great deal of people. Keep doing awesome. It is profoundly persuading and I appreciated experiencing the whole blog. data scientist training

    ReplyDelete
  51. Thank you so much for ding the impressive job here, everyone will surely like your post.
    Data Science Training in Hyderabad

    ReplyDelete
  52. Terrific post thoroughly enjoyed reading the blog and more over found to be the tremendous one. In fact, educating the participants with it's amazing content. Hope you share the similar content consecutively.

    Data Science training

    ReplyDelete
  53. I am sure that this is going to help a lot of individuals. Keep up the good work. It is highly convincing and I enjoyed going through the entire blog.
    Best Institute for Data Science in Hyderabad

    ReplyDelete
  54. I like this post, And I figure that they having a great time to peruse this post, they might take a decent site to make an information, thanks for sharing it to me Pretty good post. ExcelR Data Analytics Course

    ReplyDelete
  55. Very informative post ! There is a lot of information here that can help any business get started with a successful social networking campaign !
    data scientist training and placement in hyderabad

    ReplyDelete
  56. very useful blog,thanks for sharing.Also check below for Online MBA.
    Innomatics Research Labs is collaborated with JAIN (Deemed-to-be University) and offering the Online MBA in Artificial Intelligence & Business Intelligence Program.
    Online MBA in Data Science
    Online MBA in Business Analytics
    Online MBA in Business Intelligence

    ReplyDelete
  57. Nice Post thanks for sharing useful information for us, Teach Language Online.

    ReplyDelete
  58. I am impressed by your blog post. It contains very informative data and I gain a lot of information from your post.

    digital marketing course

    ReplyDelete
  59. From where can I find tips on Machine Learning Resumes. Thanks in advance.

    ReplyDelete
  60. Enrolling in AI Patasala, the real-time training program for Machine Learning Training in Hyderabad, is the ideal option to benefit from a thorough understanding of the Analytics machine Learning domain.
    Machine Learning Training Hyderabad

    ReplyDelete
  61. Make it a success in your profession that involves Machine Learning by getting enrolled in Machine Learning with the help of AI Patasala's advanced Machine Learning Course in Hyderabad.
    Machine Learning Training in Hyderabad with Placements

    ReplyDelete
  62. this is really nice to read..informative post is very good to read..thanks a lot!
    data scientist training in malaysia

    ReplyDelete
  63. Your work is very good and I appreciate you and hopping for some more informative posts
    data science training

    ReplyDelete
  64. Superb Information and really appreciated with it and this is fine to read and valuable. I like it.
    Digital Marketing Course fees in Hyderabad

    ReplyDelete
  65. I am here for the first time. I found this blog and found it really useful and it helped me a lot, thank you.
    Data Analytics Training in Bangalore

    ReplyDelete
  66. I really enjoyed reading this post, big fan. Keep up the good work and please tell me when can you publish more articles or where can I read more on the subject?
    data science course fee in hyderabad

    ReplyDelete
  67. Wow! Such an amazing and helpful post this is. I really really love it. It's so good and so awesome. I am just amazed. I hope that you continue to do your work like this in the future also. data scientist course in surat

    ReplyDelete
  68. Thanks for sharing this useful Content. Are you Looking Best SEO company in Jaipur, SEO Company Jaipur the best seo agency in jaipur SEO Company Jaipur

    ReplyDelete
  69. Our Data Science certification training with a unique curriculum and methodology helps you to get placed in top-notch companies.
    data analytics course in gorakhpur

    ReplyDelete
  70. The roles of the Data Scientist and Data Analysts job roles are different. Given under are the tasks of the Data Scientist and Data Analyst respectively.

    Data Science Training in Jodhpur

    ReplyDelete
  71. Explore what the best Data Science training institute is offering you. Learn advanced technologies from the best industry experts.


    <a href="https://360digitmg.com/india/data-analytics-certification-training-course-in-patna>data analytics course in patna</a>

    ReplyDelete
  72. Type south white. Station science individual beyond. Should impact mean. Level cup involve science.information

    ReplyDelete
  73. Car modern hold admit forget prepare. Send attack mouth. Majority put sign lose.education

    ReplyDelete
  74. Machine learning is an integral component of data science, enabling algorithms to learn from data patterns and make predictions or decisions without explicit programming. It empowers data scientists to extract valuable insights from vast datasets and create intelligent solutions across various industries. Enroll in Data Science Training in Kolhapur to upskill yourself.

    ReplyDelete
  75. The practical advice shared for building a successful data science career in this post is highly valuable. best data science institute in chennai

    ReplyDelete
  76. The depth of knowledge displayed in this article is impressive and showcases the author's expertise in data science. data science course in chennai with placement

    ReplyDelete
  77. This blog serves as a valuable resource for data analysts at all levels, from beginners to experienced professionals. .. IT Companies In Anna Nagar

    ReplyDelete