Pony’s Stable

First Look Into Theano Core

It’s been a week and a half since google-melange anounced the accepted student for google summer of code. I was luckcy enough to be accepted by Theano – sub-foundation of Python organization, to help them add new features: Allow user to modify compiled function. As scheduled, from 27th April to 23rd May is the community bounding period. During these days I ought to get familiar with Theano core code and Theano dev community.

Before the application started, I’ve dived into Theano cored and got the basic idea of what I’am going to do. However, to make the idea more clear and to fit the requirement that student should post every week about their progress. I decide to write two post about Theano core – about how theano work. This is the first post. This post will talk about what is a function? And how a function is generate.

How a function is generated?

Just recall how we compiled a function func = theano.function( [ inputs ], output ), we can know that we should start out journey from method function(), which locates in theano/compile/founction.py.
In method function(), after some data verification, it will call orig_func() or pfunc() which return a function that user will get. Since pfunc() will also call orig_func(), we are going to look into pfunc() first.

pfunc.py

pfunc() have two major tasks:

  • Transfer input_variable into In() instances. So does shared_variable. ( In function graph, SharedVariabls are treated as input, updates are treated as output ).
  • Rebuild computational graph using updates and inputs, transform output into Out instances.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41

def pfunc( ... some arguments ... ):

    # Clones the replacements named in the givens argument, and points each Var1 to
    # the clone of Var2.
    # Transform params into theano.compile.In objects.
    inputs = [_pfunc_param_to_in(p, allow_downcast=allow_input_downcast)
                for p in params]

    #  clones the outputs and the update expressions.  This rebuilds a computation graph
    # from the inputs and the givens. ( Irrelative to me. Pass )

    output_vars = rebuild_collect_shared(outputs,
                                             in_variables,
                                             replace=givens,
                                             updates=updates,
                                             rebuild_strict=rebuild_strict,
                                             copy_inputs_over=True,
                                             no_default_updates=no_default_updates)
    # extracting the arguments
    input_variables, cloned_outputs, other_stuff = output_vars
    clone_d, update_d, update_expr, shared_inputs = other_stuff

    for i, iv in zip(inputs, input_variables):
        i.variable = iv

    for sv in shared_inputs:
        # pass value of None here
        # value will be stored in the resulting functions' defaults list
        # but since the value of shared variables never needs to be refed, it is not needed
        if sv in update_d:
            si = In(variable=sv, value=sv.container, mutable=True,
                    borrow=True, update=update_d[sv], shared=True)
        else:
            si = In(variable=sv, value=sv.container,
                    mutable=False, borrow=True, shared=True)
        inputs.append(si)

    return orig_function(inputs, cloned_outputs, mode,
            accept_inplace=accept_inplace, name=name, profile=profile,
            on_unused_input=on_unused_input, output_keys=output_keys

orig_func():

Now it time for a look into orig_func(). orig_func() will again makes sure that inputs and outputs are transformed into In an Out. And then it will use create method in FunctionMaker to make a function, which will be return.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

def orig_function(inputs, outputs, mode=None, accept_inplace=False,
                  name=None, profile=None, on_unused_input=None,
                  output_keys=None):

    # conver input variable into instances of In() instances
    inputs = map(convert_function_input, inputs)

    # so do outputs
    if outputs is not None:
        if isinstance(outputs, (list, tuple)):
            outputs = map(FunctionMaker.wrap_out, outputs)
        else:
            outputs = FunctionMaker.wrap_out(outputs)

    # In()s and Out()s will be passed into FunctionMaker and a function will be create from it by calling create() method.
    fn = Maker(inputs,
                   outputs,
                   mode,
                   accept_inplace=accept_inplace,
                   profile=profile,
                   on_unused_input=on_unused_input,
                   output_keys = output_keys).create(
                       defaults)         #     ^^^^

FunctionMaker:

FunctionMaker.__init()__ is where fgraph is extracted and optimized. FuncitonMaker.create() is where function will be compiled and linked. In fact, FunctionMaker.linker.make_thunk() is where function is linked.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83

class FunctionMaker:
    def __init__( ...args... ):
        # again, make sure that input/output are tranformed into In and Out
        inputs, outputs = map(self.wrap_in, inputs), map(self.wrap_out, outputs)

        # ???
        _inputs = gof.graph.inputs([o.variable for o in outputs] + [i.update for i in inputs if getattr(i, 'update', False)])

        # make indices ... which is useless
        indices = [[input] + self.expand_in(input, _inputs) for input in inputs]

        # get fgraph
        if fgraph is Node:
            fgraph, additional_outputs = std_fgraph(inputs, outputs, accept_inplace)
        else:
            _, additional_outputs = std_fgraph(inputs, outputs, accept_inplace)
        self.fgraph = fgraph

        # fetch optimizor and linker
        optimizer, linker = mode.optimizer, copy.copy(mode.linker)

        # optimize fgraph
        # if need_opt:
            # optimization code here

        # linker accept fgraph 
        if no_borrow:
            self.linker = linker.accept(fgraph, no_recycling=infer_reuse_pattern(fgraph, no_borrow))
        else:
            self.linker = linker.accept(fgraph)

        if hasattr(linker, 'accept_var_updates'):
            # hacky thing so VMLinker knows about updates
            self.linker.accept_var_updates(
                    fgraph_updated_vars(fgraph, inputs))

        # some other configeration here
        # ...

    def create(self, input_storage=None, trustme=False):
    """
    Create a function.

    Input storage is the 'value' attribute of each In instances
    """

        # construct input_storage_list and default list
        for i, ((input, indices, subinputs), input_storage_i) in enumerate(zip(self.indices, input_storage)):
            # a lot of codes here.

        """
        Q: What's indices?
        List of (SymbolicInput, incide, [SymbolicInput..]). The first one is the 
        input vaiable; the incide is the index of the input in the input list; 
        the [SymIn...] the relevant input(?); 
        According to the document, the last two is deprecated. So it can be regarded as list of SymbolicInput.
    
        Q: What's defaults?
        A: List of 3-tuples. Each tuple corresponds to one input_storage. 
        ( 
          Bool: Is this input required at each function call?,  
          Bool: Should this inputs value be reverted to default value after each call? 
          AnyType: The value(storage) associated with this input.
        )
        """

        # call make_thunk() from linker and get fn
        try:
            theano.config.traceback.limit = 0
            _fn, _i, _o = self.linker.make_thunk(input_storage=input_storage_lists)
            # ^   ^   ^ => (function, input_containers, output_containers)
            # where function is a thunk that operates on the returned variables.
            # Because the map_storag() in make_thunk()
            # from here on, the input/output_storage represent all I/O of all relative nodes
            # ALso, storage is container, Instead of SymbolicKit
        finally:
            theano.config.traceback.limit = limit_orig

        # get a function, here function_builder() is the constructor
        # of class Function.
        fn = self.function_builder(_fn, _i, _o, self.indices, self.outputs,
                    defaults, self.unpack_single, self.return_none, self.output_keys, self)
        return fn

What’s theano.function?

Each function is a callable object. theano.function is not a python function. Instead, it a class with method __call__(). Every funciton stores its own fgraph, maker, storages and many other configurations. However, the core of a function is a function fn() returned by linker.make_thunk(). Every time a function is called, it will first verify the input data, and then call self.fn() to get output values.

Now I know how is a function borned. Also, I know that to complete my missions, I need to focus more on FunctionMaker and Function rather that orig_func() and pfunc(). However, there still exist some question, such as: What does make_thunk() do? and What is In(), Out() and container? In the next post, I will have a look at this and other relative data structures.