PhysicalNeuralNetworks
JonathanLamont
November 16,2015
Contents
•
•
•
•
ArtificialNeuralNetwork
HistoryofDevelopment
AHaH Computation
FutureGoalsofAHaH System
NeuralComputation
• Understandingbiologicalneuralsystemsfalls
inaspectrumbetweenmimicryand
algorithmicsolutions
• Tomimicthebrain,shiftquestionfrom“how
dobrainscompute?”to“howdobrainsbuild
andrepairthemselvesasdissipativeattractorbasedstructures?”
WhatisanArtificialNeuralNetwork?
• ANNsaregenerallydenselyconnectedgroups
ofindividualcells,calledneurons,that
producedesiredoutputsfortrainedinputs
(trainedratherthanprogrammed)
• Inspiredbybiologicalneuralnetworks
• Usedinmachinelearningandcognitive
sciencetoestimatefunctions
NeuronDiagram
ANNDiagram
InspirationforPhysical
Implementation
• IBM’sSimulationwith1BillionNeuronsand10
TrillionSynapsesrequired147,456CPUs,144TBof
memoryrunning,andrequires2.9MWsofpowerat
1/83real-time.
– Wouldrequire7GWofpowertosimulatehumancortexbasedon
theseestimates, usingvonNeumannArchitecture
– Averagehumanneocortex contains150,000billionconnections
• CoreAdaptivePowerProblem:Energywastedduring
memory-processorcommunication.
AdaptivePowerProblem
• Constantdissipationoffreeenergyallowsliving
systemstoadaptatallscales
• Eachadaptationmustreducetomemory-processor
communicationasstatevariablesaremodified
– Energyconsumedinmovingthisinformationgrows
linearlywithnumberofstatevariablesthatmustbe
continuouslymodified
HistoryofDevelopment
• AlanTuring’s‘IntelligentMachinery’(1948)
– Describes machineconsistingofartificialneuronsconnectedinany
patternwithmodifierdevices
– Modifierdevices couldpassordestroyasignal
• Hebb proposedsynapticplasticity(1949)
– Describestheabilityofsynapsestostrengthorweaken over
timeinresponsetoincreasesordecreasesinactivity
• ADALINEor“AdaptiveLinearNeuron”(1960)
– Physicaldeviceusingelectrochemicalplatingofcarbonrodsto
emulatesynapticelementscalledmemistors.
– Memistor – Resistorwithmemory abletoperformlogic
operationsandstoreinformation.
HistoryofDevelopment(Continued)
• Memristor Theory(1971)– regulatesflowof
electricalcurrentincircuitandrememberscharge
thatpreviouslyflowed;retainmemorywithout
power(non-volatile)
• Very-large-scaleintegration(VLSI)(1980)–integrated
circuitcreatedbycombiningthousandsoftransistors
intosinglechip
– HelpedNeuralNetworks(Hopfield),Neuromorphic
Engineering(Mead),andPhysicsofComputation
(Feynman)
AHaH Computation
• Exploitcontroversial“FourthLawof
Thermodynamics”toretrainweightsfrom
Swenson(1989):
– “Asystemwillselectthepathorassemblyofpathsout
ofavailablepathsthatminimizesthepotentialor
maximizestheentropyatthefastestrategiventhe
constraints.”
• Thisthermodynamicprocesscanbeunderstood:
1)Particlesspreadoutalongavailablepathwaysanderode
differentials thatfavoronepathortheother
2)Pathwaysthatleadtodissipationarestabilized
Figure1.AHaHprocess.
NugentMA,MolterTW(2014)AHaHComputing–FromMetastableSwitchestoAttractorstoMachineLearning.PLoSONE9(2):e85175.
doi:10.1371/journal.pone.0085175
http://journals.plos.org/plosone/article?id=info:doi/10.1371/journal.pone.0085175
AHaH Plasticity
• Definesynapticweight:
w=Ga – Gb,whereGa andGb areconductancevariables
• Anti-Hebbian (erasepath)
– Modificationtosynapticweightthatreducesprobability
thatthesynapticstatewillremainthesameupon
subsequent measurement
• Hebbian (selectpath)
– Modificationtosynapticweightthatincreasesprobability
thatsynapticstatewillremainthesameuponsubsequent
measurement
Figure4.Adifferentialpairofmemristorsformsasynapse.
NugentMA,MolterTW(2014)AHaHComputing–FromMetastableSwitchestoAttractorstoMachineLearning.PLoSONE9(2):e85175.
doi:10.1371/journal.pone.0085175
http://journals.plos.org/plosone/article?id=info:doi/10.1371/journal.pone.0085175
LinearNeuronModel
• y=b+ΣNi=0xiwi
– yisthe output ofthe selected neuron,bisthe bias to the
function,xi isthe output ofneuron i,and wi isthe weight
between the selected neuron and neuron xi
• Weights and biases change based onthe AHaH rule
– αand βare constants that control the relative contribution
ofAnti-Hebbian and Hebbian plasticity
AHaH forMachineLearning
• AHaH node:usesAHaH ruletobisectinput
spaceusingahyperplane tomakeabinary
decision
• SincetherearemanywaystheAHaH nodecan
bisectinput,thehyperplane chosentobisect
theinputspaceis“theonethatmaximizesthe
separationofthetwoclasses”ofinputs.
Figure2.Attractorstatesofatwo-inputAHaHnode.
NugentMA,MolterTW(2014)AHaHComputing–FromMetastableSwitchestoAttractorstoMachineLearning.PLoSONE9(2):e85175.
doi:10.1371/journal.pone.0085175
http://journals.plos.org/plosone/article?id=info:doi/10.1371/journal.pone.0085175
AHaH PowerConsumption
• MajormotivationistoeliminatevonNeumann
bottleneckregardingmemory-processor
communication
• Energyoutputisdefined:
• Inpractice,theenergyconsumptionpersynapse
perupdateisaround12pJ,althoughscaling
trendsprojectenergyconsumptiontobelowered
to2pJ inthefuture.
FutureGoals
• Provideaphysicaladaptivelearninghardware
resource(AHaH circuit)thatprovidesmemory
similartohowRAMprovidesmemoryto
currentcomputingsystems.
• UseAHaH nodeasabuildingblockforhigherorderadaptivealgorithmsnotyetconceived.
Conclusion
• ArtificialNeuralNetworksaregreatfunction
approximations,butthereisamajorbottleneck
frommemory-processorcommunicationusing
vonNeumannarchitecture
• Reachingbacktothelate1940s,thistypeof
computationhasbeentheorizedandstudied
• AHaH computationexploitsphysical
phenomenontoimplementanartificialneural
network,lesseningthememory-processor
communicationbottleneckforpowerwhile
allowingmassiveparallelism.
Sources
• AHaH Computing-FromMetastableSwitchestoAttractorsto
MachineLearning
– http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0085175
• “NeuralNetworkLearning”
– http://web.eecs.utk.edu/~mclennan/Classes/420-527-S14/handouts/Part-2A1pp.pdf
• “Neuralnetworkmodelsforpatternrecognitionand
associativememory”
– http://www.sciencedirect.com/science/article/pii/089360808990035X
• “Perceptrons,Adalines,andBackpropagation”
– http://www-isl.stanford.edu/~widrow/papers/bc1995perceptronsadalines.pdf