FWEP 1 Json/Dictionary type of I/O definition

[rethore]

FWEP (FUSED-Wind Enhancement Proposal) 1:

Json/Dictionary type of I/O definition

Hey FUSED people,
We have had a short discussion with Justin on their upcoming big change of API. They are apparently planning to change the way they define the I/Os:

instead of

class MyComp(Component): 

   x = Float(x, iotype="in")    
   y = Float(x, iotype="out")

   def execute(self): 
       self.y = 2*self.x

you would do something like:

class MyComp(Component): 

   def __init__(self): 

       self.add_input('x', float)
       self.add_output('y', float)

   def execute(self, inputs, outputs): 
        outputs['y']= 2*inputs['x']

So this is a quite drastic change for us, as we rely on the components I/O definition to define our standard interfaces. What I propose is to add a layer on top of OpenMDAO (that might hopefully eventually be supported by OpenMDAO) so that things are defined this way:

@FUSED_IO(
     inputs=[
        {name:’x’, 'type':float},
        {name:’x2’, 'type':list},
        {name:’x3’, 'type':array},
    ],
    outputs=[
        {name:’y’, 'type':float},
        {name:’y2’, 'type':list, 'size':…, 'desc':‘blabla'},
        {name:’y3’, 'type':array, 'size':..},
    ])
class MyComp(Component):
   def execute(self, inputs, outputs): 
        outputs['y']= 2*inputs['x’]
    ...

Here are a few advantages of doing that:

• To do the repetitive add_input / add_output call under the hood through the decorator
• To have a set of thin decorators that can take a non openmdao function or class and wrap them into a FUSED-Wind openmdao component
• To have a list of standard FUSED-Wind variables:

wind_speed  =  {name:’wind_speed’, type:float, min:4.0, max:25.0, desc:’A wind speed float’, unit:’m/s’}
hub_wind_speed  =  expand(wind_speed, {name:’hub_height_wind_speed’, desc:’The wind speed at hub height'})
rotor_height = {’name’:’rotor_height’, ….}
wt_layout  =  {
    ‘wt_list’: {'name':’wt_list’, type:[wt_descriptor],…} # we would need to define some kind of elegant way to inform what are the contents of the lists
    ‘wt_positions’: {‘type’:array, ’size’:’[n,2]'}
    ‘wt_wind_roses’: {...}}

• expand could be function keeping track of inheritance, for multi-fidelity I/O definition, if necessary.

dico = {'name':'dico',
    'a':1,
    'b':2,
    'based_on':[]}
​
expand = lambda x, y: dict(x.items() + y.items() + [('based_on', [x['name']]+x['based_on'])])

expand(dico, {'name':'new_dico','c':3, 'a':3})

{'a': 3, 'b': 2, 'based_on': ['dico'], 'c': 3, 'name': 'new_dico'}

• These standard variables could then be used to define our FUSED-Wind interfaces:

from fusedwind.variables import wind_speed, wind_direction, wt_layout, net_aep, gross_aep

@FUSED_IO(
    inputs=[wind_speed, wind_direction, wt_layout],
    outputs=[net_aep, gross_aep])
class MyComp(Component):
...

• Our standard interfaces could then be defined as dictionaries:

GenericAEP = FUSED_IO(
     inputs=[wind_speed, wind_direction, wt_layout],
    outputs=[net_aep, gross_aep])

• Then any component satisfying this interface could be done like that:

@FUSED_IO(
    based_on=GenericAEP,
     inputs=[wind_speed, wind_direction, wt_layout],
     outputs=[net_aep, gross_aep, capacity_factor])
…

Or something looking like that

@GenericAEP
@FUSED_IO(outputs=[capacity_factor])
…

(a bit less verbose...)

• As everything is already defined in dictionary, it’s a simple step to go to Json files, defined standard inputs/outputs files, REST interfaces etc..
• To use this definition to generated automatically a standard RESTful web API. So being able to deploy quickly a FUSED-Wind compatible function of Component into a webmodel.
  So potentially somebody developing in another language could deploy their model online using the same standard API as we defined in FUSED-Wind and thereby make their model available to run in a FUSED-Wind assembly. We would “just” need to inform the web address and credentials, and the Component could automatically generate it’s input/outputs from the REST interface definition. This would be our way to open up FUSED-Wind collaboration beyond the python and openmdao community to anybody else able to deploy REST interfaces (hint: could be something we advocate through the IEA SE task).

What do you think about this idea?

[rethore]

For illustration purpose, here is a simple flask app that could deploy Hawc2 online:

from flask import Flask, request
from hawc2wrapper import Hawc2Wrapper
import json

app = Flask(__name__)

def deploy_FUSEDWIND(cls):
    c = cls()

    @app.route('/'+cls.__name__, methods=['GET', 'POST'])
    def myflask():
        if request.method == 'POST':
            inputs =  request.form.to_dict()
            # ... Optionally do some credential check here...
            c.execute(inputs, outputs)
            return json.dumps(output)

        # Publish the interface definition of the class
        return json.dumps(cls.fusedIO)

if __name__ == "__main__":
    deploy_FUSEDWIND(Hawc2Wrapper)
    app.run()    

and on client side:

from fusedwind.interfaces import RESTComponent

class A(Assembly):
    def configure(self):
        self.add('hawc2', RESTComponent(url='fusedwind.hawc2.dk/Hawc2Wrapper', credential='.hawc2.license'))
        ...

[dykesk]

If we don’t use there i/o structure and instead use this dictionary type set-up, don’t we break their dependency graph functionality and lose all the nice derivative chaining etc? It seems like at some point, we will have to perform the specific add_input, add_output to get variables into the graph and connect across components at the assembly level.

Seems like we could still do a variation of the below that Pierre proposed, but we would add the variables in the init and use them explicitly as openmdao variables as before.

[rethore]

Maybe I've been unclear. I'm not suggesting to not use their way of declaring the I/O. I'm suggesting to add another layer on top of it. So our decorator would actually call the add_input and add_output functions using the information contained in the dictionaries passed to the decorator.

[rethore]

@JustinSGray, I would also be interested to have your feed-backs about the feasibility of my proposal, when you have some spare time :-)

[dykesk]

Got it - I didn't see the add_input/add_output in your example code anywhere and it looked in your example like you are passing them in via the execute method...

[fzahle]

I like the idea of the standard variables Pierre mentions in fusedwind.variables. This would help prevent us from defining interfaces in different parts of the framework that refer to the same variable but are given different names since all interface variables have to defined in the variables.py file.

I'm not so much a fan of using the decorator to do the add_input and add_output under the hood since many components will most likely have a mix of FUSED-Wind interface variables mixed with specific model variables, which means that a class interface will declare I/O in two different ways: standard OpenMDAO and a FUSED-Wind decorator. I like the current way we do it where everything is defined in one place by the user, and the decorator really only doing a simple check of that.

Lastly, I think its way too ambitious to have added this to the 0.2 release (which is June), since this release should focus on getting FUSED-Wind and all our models working in OpenMDAO 0.12 with MPI. Also, we don't know if/when the proposed changes in OpenMDAO will take place.

[rethore]

@fzahle, I see your point that blending add_input and decorator input would be confusing. The way I see it the decorator input would replace most occurrence of add_input, even when the inputs/outputs are custom to the component. They would only be necessary when they are somehow automatically generated. You could also see it the other way around: if we do as you suggest to both have a decorator and explecitely do the add_inputs/add_outputs, it would be weird if the exposed interface in the decorator would not cover all the inputs/outputs defined in the init section. That would make things even more confusing I think.

import fusedwind.variables.api as fw

# Defined somewhere upstream
private_input1 = fw.float(1.0, desc='my private inputs')

@IO({'inputs': {
        'wind_speed': fw.wind_speed,
        'wind_direction': fw.wind_direction,
        'private_input1': private_input1
    },
    'outputs': {
        'net_aep': fw.net_aep,
        'private_output1':  fw.array(zeros([10,2]), desc='my private inputs') #defined in the dictionary
    }})
class MyComp(Component):
...

Another potential issue I see is if the add_inputs / outputs are somehow necessary to be done before the init function. Then it start to be tricky to do it "under the hood".

Edit:
Concerning your last point. We don't have to wait for their modification of API to take this decision. This can be done now with their current API. I have already a working decorator that does the job OK. Actually the sooner we move to a more abstract definition of the I/O, the smoother our transition to the new API will be for us and our community.