anhinga_anhinga (anhinga_anhinga) wrote,
anhinga_anhinga
anhinga_anhinga

Self-referential neural nets in 2018

Two series of experiments with self-referential neural nets with vector flows ("dataflow matrix machines") were done by us in 2018.

The ability of a neural net to modify itself on the fly was used to edit it interactively while it is running ("livecoding"). This also opens the way to have populations of neural nets editing each other.

Emerging "sleep-wake" behavior and other emerging bistability patterns were observed in randomly initialized neural nets. There is no theoretical understanding of this emerging dynamics yet.

These experiments are described in Section 1 of https://github.com/jsa-aerial/DMM/tree/master/technical-report-2018 (DMM technical report 11-2018, "Dataflow matrix machines: recent experiments and notes for next steps").

Also https://arxiv.org/abs/1610.00831 got an extra page (page 7) with two new appendices, and the main paper, https://arxiv.org/abs/1712.07447, has a new more compact abstract:

1) Dataflow matrix machines (DMMs) generalize neural nets by replacing streams of numbers with linear streams (streams supporting linear combinations), allowing arbitrary input and output arities for activation functions, countable-sized networks with finite dynamically changeable active part capable of unbounded growth, and a very expressive self-referential mechanism.

2) DMMs are suitable for general-purpose programming, while retaining the key property of recurrent neural networks: programs are expressed via matrices of real numbers, and continuous changes to those matrices produce arbitrarily small variations in the associated programs.

3) Spaces of V-values (vector-like elements based on nested maps) are particularly useful, enabling DMMs with variadic activation functions and conveniently representing conventional data structures.
We also have a better slide deck: https://github.com/jsa-aerial/DMM/blob/master/doc/DMM-IBM-Talk-Oct2018.pdf
 
Crosspost: https://anhinga-anhinga.dreamwidth.org/83296.html

Other blogs by this author:

https://anhinga-drafts.livejournal.com/  (mirror: https://anhinga-drafts.dreamwidth.org/ )
https://anhinga-travel.livejournal.com/ (mirror: https://dmm.dreamwidth.org/ )

Tags: dataflow matrix machines
Subscribe
  • Post a new comment

    Error

    default userpic

    Your reply will be screened

    When you submit the form an invisible reCAPTCHA check will be performed.
    You must follow the Privacy Policy and Google Terms of use.
  • 0 comments