Semester Winter 2016/17 Location & Time University of Haifa Education Building, Class 463, Tuesdays 10:15-11:45. Lecturer Dr. Amir Adler E-mail adleram@cs.technion.ac.il Syllabus (1) Review of Deep Learning applications and architectures: fully connected networks, convolutional networks (CNN), recurrent networks (RNN), Stochastic Gradient Descent, Back-Propagation, and Regularization Techniques. (2) Students Lectures (40 minutes each), based on state-of-the- art papers. Pre-requisites: Machine Learning or Algorithms Grading: 85% Lecture + 15% Class Presence & Participation
1 Course Plan
Topic Week
Introduction to Deep Learning: (1) Review of computer vison, natural language 1
processing, and speech processing applications. (2) Evolution of neural networks technology, and milestones during past 10 years, Machine Learning concepts review: supervised, unsupervised, semi-supervised 2 learning, representation learning, reinforcement learning. Fully connected and convolutional networks in detail. Multi-layer perceptron, 3 non-linear activation units, SoftMax layer, convolutional layer, max-pooling, average-pooling. Stochastic Gradient Descent (SGD) and loss functions (mean-squared error, 4 negative log likelihood). SGD with momentum. The Back-Propagation algorithm 5 Recurrent Neural Networks (RNN): Basic RNN structure, characters prediction 6 and image captioning with RNNs, exploding and vanishing gradients problem, Long Short Term Memory (LSTM), Gated Recurrent Unit (GRU). Students Lectures 7 Students Lectures 8 Students Lectures 9 Students Lectures 10 Students Lectures 11 Students Lectures 12 Students Lectures 13 Students Lectures 14
Text Book
1. Ian Goodfellow and Yoshua Bengio and Aaron Courville, "Deep Learning", MIT Press 2016. Available online: http://www.deeplearningbook.org/
2 Lectures
No. Paper Name Conference Link
/Journal 1 Beyond short snippets deep CVPR 2015 https://arxiv.org/abs/1503.08909 networks for video classification large scale video classification with CVPR 2014 http://vision.stanford.edu/pdf/karpat convolutional neural networks hy14.pdf
2 Deep Residual Learning for Image CVPR 2016 https://arxiv.org/abs/1512.03385
Recognition Identity Mappings in Deep ECCV 2016 https://arxiv.org/abs/1603.05027 Residual Networks 3 SSD: Single Shot MultiBox ECCV 2016 https://arxiv.org/abs/1512.02325 Detector Faster R-CNN: Towards Real-Time NIPS 2015 https://arxiv.org/abs/1506.01497 Object Detection with Region Proposal Networks 4 Rich feature hierarchies for CVPR 2014 https://arxiv.org/abs/1311.2524 accurate object detection and semantic segmentation OverFeat: Integrated Recognition, ICLR 2014 https://arxiv.org/pdf/1312.6229.pdf Localization and Detection using Convolutional Networks 5 Show, Attend and Tell: Neural ICML 2014 https://arxiv.org/pdf/1502.03044v3. Image Caption Generation with pdf Visual Attention Unifying Visual-Semantic NIPS 2014 https://arxiv.org/abs/1411.2539 Embeddings with Multimodal Neural Language Models 6 Mastering the game of Go with Nature 2016 http://www.nature.com/nature/journ deep neural networks and tree al/v529/n7587/full/nature16961.ht search (by Google DeepMind) ml
Human level control through deep Nature 2015 http://www.nature.com/nature/journ
reinforcement learning (by Google al/v518/n7540/full/nature14236.ht DeepMind) ml
7 Learning Hierarchical Features for PAMI 2013 http://yann.lecun.com/exdb/publis/p
Scene Labeling df/farabet-pami-13.pdf Indoor Semantic Segmentation ICLR 2013 https://arxiv.org/abs/1301.3572 using depth information 8 Understanding the difficulty of AISTATS http://jmlr.org/proceedings/papers/v training deep feedforward neural 2010 9/glorot10a/glorot10a.pdf 3 networks Batch Normalization: Accelerating ICML 2015 http://jmlr.org/proceedings/papers/v Deep Network Training by 37/ioffe15.pdf Reducing Internal Covariate Shift 9 Learning spatiotemporal features ICCV 2015 https://arxiv.org/pdf/1412.0767.pdf with 3d convolutional networks sequence to sequence video to text CVPR 2015 https://arxiv.org/abs/1505.00487
10 Very Deep Convolutional Networks ICLR 2015 https://arxiv.org/pdf/1409.1556.pdf
for Large-Scale Image Rcognition Going Deeper With Convolutions CVPR 2015 https://arxiv.org/abs/1409.4842
11 Object Contour Detection with a CVPR 2016 http://www.cv-
Learning a deep convolutional download?doi=10.1.1.642.1999&re network for image super-resolution p=rep1&type=pdf
12 Explain Images with Multimodal NIPS 2014 https://arxiv.org/pdf/1410.1090.pdf
Recurrent Neural Networks Workshop Long-term Recurrent Convolutional CVPR 2015 http://www.icsi.berkeley.edu/pubs/ Networks for Visual Recognition vision/longtermrecurrent15.pdf and Description 13 Zero-Shot Learning Through Cross- NIPS 2013 http://nlp.stanford.edu/pubs/Socher Modal Transfer GanjooManningNg_NIPS2013.pdf
DeViSE: A Deep Visual-Semantic NIPS 2013 http://static.googleusercontent.com/
Embedding Model media/research.google.com/en//pub s/archive/41473.pdf
14 Video Super-Resolution With IEEE Trans. http://ivpl.eecs.northwestern.edu/sit
Convolutional Neural Networks on Comp. es/default/files/07444187.pdf Imaging 2016 ICLR 2016 https://arxiv.org/pdf/1511.05440v6. Deep multi-scale video prediction pdf beyond mean square error
NIPS 2015 https://arxiv.org/pdf/1506.05439v3.
15 Learning with a Wasserstein Loss pdf
ICCV 2009 http://www.ariel.ac.il/sites/ofirpele/
Fast and Robust Earth Mover’s publications/ICCV2009.pdf Distances (background paper)