Geoff Hinton

The brain has about 1014 synapses and we only live for about 109 seconds. So we have a lot more parameters than data. This motivates the idea that we must do a lot of unsupervised learning since the perceptual input (including proprioception) is the only place we can get 105 dimensions of constraint per second.

4 thoughts on “Geoff Hinton

  1. shinichi Post author

    AMA Geoffrey Hinton (self.MachineLearning)

    reddit

    https://www.reddit.com/r/MachineLearning/comments/2lmo0l/ama_geoffrey_hinton/clyjogf/

    Here are some of my beliefs about the brain that have made a big difference to the kinds of machine learning I have done:

    The cortex is pretty much the same all over and if parts are lost early, other parts can take on the functions they would have implemented. This suggests its really worth taking a bet on there being a general purpose learning procedure.

    The brain is clearly using distributed representations.

    The brain does complex tasks like object recognition and sentence understanding with surprisingly little serial depth to the computation. So artificial neural nets should do the same.

    The brain has about 1014 synapses and we only live for about 109 seconds. So we have a lot more parameters than data. This motivates the idea that we must do a lot of unsupervised learning since the perceptual input (including proprioception) is the only place we can get 105 dimensions of constraint per second.

    Roughly speaking, spikes are noisy samples from an underlying Poisson rate. Over the short time periods involved in perception, this is an incredibly noisy code. One of the motivations for the idea of dropout was that very noisy spikes are a good way to get a very strong regularizer that can help the brain deal with the fact that it has thousands of times more parameters than experiences.

    Over a short time period, a neuron really is a binary all-or-none device (so far as other neurons are concerned). This was one of the motivations behind Boltzmann machines. Another was the paper by Crick and Mitchison suggesting that we do unlearning during sleep. There now seems to be quite a lot of evidence for this.

    Reply
  2. shinichi Post author

    Geoffrey E. Hinton

    http://www.cs.toronto.edu/~hinton/

    I am an Engineering Fellow at Google where I manage Brain Team Toronto, which is a new part of the Google Brain Team and is located at Google’s Toronto office at 111 Richmond Street. Brain Team Toronto does basic research on ways to improve neural network learning techniques. I also do pro bono work as the Chief Scientific Adviser of the new Vector Institute. I am also an Emeritus Professor at the University of Toronto.

    Reply
  3. shinichi Post author

    シナプス(synapse)は、神経細胞間あるいは筋繊維(筋線維)、神経細胞と他種細胞間に形成される、シグナル伝達などの神経活動に関わる接合部位とその構造である。化学シナプス(小胞シナプス)と電気シナプス(無小胞シナプス)、および両者が混在する混合シナプスに分類される。シグナルを伝える方の細胞をシナプス前細胞、伝えられる方の細胞をシナプス後細胞という。

    Reply
  4. shinichi Post author

    (sk)

    80 年生きるとして、 60 x 60 x 24 x 365 x 80 = 2,522,880,000 = 2.5 x 109 秒。

    脳には 1014 のシナプスがあるから、1秒あたり 105 ものシナプスが、知覚の入力(固有受容「目を閉じていても、自分の手や足の位置と、それを動かしていることが分かる感覚」を含む)に使われている。

    脳は、知覚の処理という、とても大変なことをしている。

    その処理がわからないといって、論理的でないなどと言ってはいけない。

    なんとなくとか、直観とか、たぶんとか、そういうことは、とても信頼できるものなのかもしれない。

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *