Re: Neural netss (was Re: death of the mind.)

From: Eray Ozkural exa (erayo_at_bilkent.edu.tr)
Date: 09/08/04 

Date: 8 Sep 2004 06:02:34 -0700

"Rick Craik" <rick@@icebergideas..com> wrote in message news:<vAk%c.12221$lP4.904168@news20.bellglobal.com>...
> > In my mind, there ought to be programming language-like thingies that
> > drive these virtual machines.
>
> In my mind, there is always the problem of what is data and
> what is a program. Can we say a NN is data driven and a VM
> is program driven? [Hmm, seems like nurture vs. nature.] Perhaps
> there is a point when altering Life automata, we find a data
> driven nature.

What kind of a programming language would emerge in a NN architecture?
It's a massively parallel, fine-grained, distributed memory
architecture. A programming language that doesn't match the hardware
will not work, hence I think one of the reasons for all the focus on
K-lines and so forth on Minsky's SOM.

But there is a very interesting angle here. Think of the kind of
program and data loading that a parallel architecture would demand.
There is a lot of redundancy, ie. parallel overhead in parallel
algorithms, but it brings increased efficiency due to concurrency.
However, a world model demands space optimization, and the most
obvious way is to use compact programmatic descriptions like LISP.
This is another problem, how are these computational tradeoffs decided
by the system, ie. what kind of *computational* optimizations are
present? I believe, computational constraints play a larger role than
it seems, one definition of intelligence is to get the most out of
your hardware.

A typical NN is data driven, but it's not that simple. First, a feed
forward NN itself implements a nonlinear / combinatorial function of
some sort. A recurrent neural net implements a Turing-complete
program. In both cases training turns the data into program (we have
little idea what the training algorithm is, our models are mostly
mathematical ideas that are not really biologically plausible). In the
second case, the program becomes data. I know that the distinction
between data and program is less visible in neural networks, but that
is because you are not familiar with the programming language, not
because there is no programming language. The recurrent neural net is
a low level machine code, like the Turing machine, the only big
difference is that one is a low level parallel machine, and the other
is a low level serial machine. Absolutely every theory using such
general computational notions as Kolmogorov complexity, time
complexity, etc. apply perfectly to neural networks. C-T thesis works
in the real-world.

In other words, neural networks do not have a fundamental difference
from other computational models.

There are only some common misunderstandings that confuse people who
are not familiar with theory of computation (or too familiar with it!)
 1. It doesn't look like one of the programming systems we use
 2. It doesn't look like a Turing Machine or other models in automata
theory
 3. It doesn't seem to harbor any language
 4. It can be trained, so our intuition thinks "Oh, so this is the
difference, the other computers can't be trained!!!!!"

Obviously, the worst of these is 4. An article showing how drastic
these misunderstandings in the minds of the researchers are would be
much more useful than the philosophical handwaving that is popular in
the literature.

Regards, 

--
Eray Ozkural

Relevant Pages

  • Re: How do the brain neurons compute?
    ... compute (or what kind of algorithm it use)? ... solved how neural networks correlate data. ... Never underestimate the power of the human mind, ... If I replaced a single neuron in my brain with an electronic ...
    (sci.physics)
  • Re: C# without .net?
    ... C# is merely a programming language. ... language designed with .NET in mind, but there's no reason it can't be ... Just keep in mind that the framework would be ... Maybe a compiler other than that included in .NET SDK... ...
    (microsoft.public.dotnet.languages.csharp)
  • Re: why so many PHP newsgroups??
    ... boggle your mind what some people consider a legitimate computer ... programming language, not even counting decades obsolete languages ...
    (comp.lang.php)
  • Re: What does Tcl lack?
    ... > as probably the only programming language in the world. ... Tcl, mind... ... When nobody needs the command $myobject, ... Perhaps some specialized allocation mechanism that you could ...
    (comp.lang.tcl)
  • Re: why so many PHP newsgroups??
    ... Seriously, search the newsgroups directory for "comp.lang", it will ... boggle your mind what some people consider a legitimate computer ... programming language, not even counting decades obsolete languages ...
    (comp.lang.php)