Pony’s Stable

Second_look_into_Theano_core

The first feature was done and is being reviewed! This post is my reading notes of theano core this week and how I implements the share_memory feature.

Data sturctures

Function

Function is a callable object whose core is a function ```fn``` generate by linker. Every time a function is call, it will first examinate input data and then evaluate the ```fn``` to get output value.

PARAMETERS:

  • input/output_storages: two list of Container instance.
  • maker: the maker of this function
  • finder/inv_finder: provide mapping between Container and In( seems useless )
  • fn: Core, what the real ‘function’ is, a python function that evaluate graph.
  • default and indices: List of tuples, partially useful. default[i][2] is input_storage[i] ; indices[i][0] is inputs[i] in FunctionMaker.

METHODS:

  • __init__(): Initialize input containers value and set up “[]” operator of container and self
  • __call__(): verify input data types and then execute the self.fn. Pickup value in some of the the output storages and set up corresponding input_storage in there’s updates.

FunctionMaker

FunctionMaker is a Factory that create function. However, it's not like a factory very much cause every maker only corrsponds to one function. In FunctionMaker some important info of a theano.function are stored, such as Inputs/Outputs(represented by SymbolicKit), FunctionGraph.

PARAMS:

  • indices: ( almost deprecated, ignore )
  • inputs: List of In() instances. In()
  • output: List of Out() instances
  • expanded_inputs: equals to inputs plus sharedvariables
  • fgraph: FunctionGraph that represents the function it creates.
  • Linker: Linker that link storage and apply method to create a fn. By default, FAST_RUN mode use VM_Linker , FAST_COMPILE uses PerformLinker.
  • Optimizer, mode, profile …: some configuration that has less to do with my job

METHOD:

  • create(input_storages): Given input_storages(by default, list of In.value), start the linker and link the function using Linker.make_thunk() return a theano.function.

Linker/VM

Linker is a class that allocate storage for allpy_nodes and link them together. Understanding Linker and FunctionGraph definitely helps understands how theano works. The core method of Linker is make_thunk().

PARAMS:

  • fgraph: FunctionGraph accpeted by Linker.accept().
  • allow_gc,recycle…: some configuration bools

METHODS:

  • make_thunk/all… : make_thunk() is defined in class Linker. It calls method make_all(). Every subclass of linker will have slightly different implementation of make_all(). Bascially, at first, make_all() will toposort a fgraph, to acquire the order that apply_nodes should be executed. Next, it will call Op.make_thunk(), whick return a function. This function take input_storage of node, apply computation on them, and put result on output storage. Meanwhile, make_all() will allocate storage for all variables . Same variables in different node will have same storages. This is done by a dict sotarge_map that map variable to storage. At last, Linker return a function that executes list of thunks in certain order to acquire function outputs data.

Storage

Storage is quite a tricky thing in theano. Theano use a list with one element to be a storage. The element is true data. But all object only refer to the list. 
The list works like a pointer. When some objects refer to a storage, they refers to the list, not the true data. Therefore, modifying the data of storage will not change this reference. By doing this, theano can access and modify storage data from several places without change the reference relationship.

FunctionGraph

A representation of the computational graph of a function. It stores given the input and output variables of one function, by calling node.input and variable.owner recursively we can get the whole graph

PARAMS:

  • features: Still Don’t understand now, ignore it.
  • input/output: List of input/output variabls.
  • variables: Set of variables( all variables in the subgraph)
  • apply_nodes: Set of apply_nodes in the subgraph defined by inputs and outputs

METHODS

  • __import_r__ and __import__(): import variable and apply_node to this fgraph.
  • clone_get_equiv : Clone fgraph. Return new fgraph and a dict that map old variables to new variables.
  • replace() : Replace all certain variables in fgraph by given new variables.

How theano work?( Simplified version )

Without update:

1.Input Variables will be wraped and become In()s. Each In() contains variable, it’s value( defaults is none ) as well as some other configuration. 2.Generate fgraph by input and output variables, and then optimize it. 3.Linker toposorts fgraph to get an order of apply_nodes.Next, Linker allocates storages and links function based on this order. 4.Done

With update:

Update is given by a dict { ori_var:update_var ... }. Ori_var is limited to be an SharedVariable therefore it will transfer into In() and become the input of this function. update_var will be add into the output of fgraph. Everytime a function called, the function extract the storage of update_var and use it to set the value of corresponding input.

How to implement the share_memory feature?

This is simple after understand how theano works. I implements it following sevaral steps: 1. Copy In() and Out() instances in Maker 2. Copy fgraph and get the dict equiv that map old variables to new variables 3. Copy old storage_map in Function.fn.storage_map 4. Modify copied storage_map accord to equiv, so that in the copied storage_map, new variables are mapped to old storage if memory need to be shared. 5. Reinitialize the maker, linke the function using the copied storage_map 6. Done

Ok, basically this is the report of this week’s work. Now I need to figure out how to implement the following features.