cavis/.old/ADRs/0004-Mapping_IR.md

Import IR

Status

Proposed

Proposed by: Adam Gibson (28-09-2020)

Discussed with: N/A

Context

Generally, every neural network file format defines a sequence of operations to execute mathematical operations that comprises a neural network.

Each element in the sequence is a node that contains information such as the desired operation, and a set of attributes that represent parameters in to the mathematical function to execute.

In order to write import/export for different frameworks, we need to adapt an attribute based format from various popular deep learning frameworks. Nd4j has a different list based format for operation execution arguments. In the previous ADR, we added an IR which makes it easier to interop with other frameworks.

In this ADR, this work is extended to add a file format for describing lists of operations as MappingRules which allow transformations from one framework to another.

These transformations manipulate protobuf as input and output Nd4j's new OpDescriptor format as output.

##Related work

See the import IR

Decision

We implement a mapping process framework that defines transforms on an input file format. A MappingProcess defines a list of MappingRules which represent a sequence of transformations on each attribute of an op definition.

To assist in mapping, a mapping context with needed information like rule arguments for transformation, current node, and whole graph are used as input.

The input is a protobuf file for a specific framework and the output is an op descriptor described here.

A MappingRule converts 1 or more attributes in to 1 more or arg definitions. A potential definition can be found in Appendix E.

Attributes are named values supporting a wide variety of types from floats/doubles to lists of the same primitive types. See Appendix C for a theoretical definition.

Arg Definitions are the arguments for an OpDescriptor described in the import IR ADR. See Appendix D for a potential definition of arg definitions.

All of this together describes how to implement a framework agnostic interface to convert between a target deep learning framework and the nd4j format.

Implementation details

In order to implement proper mapping functionality, a common interface is implemented. Below are the needed common types for mapping:

1. IRNodeDef: A node definition in a graph
2. IRTensor: A tensor type for mapping
3. IROpList: A list of operations
4. IRAttrDef: An attribute definition
5. IRAttrValue: An attribute value
6. IROpDef: An op definition for the IR
7. IRDataType: A data type
8. IRGraph: A graph abstraction

Each one of these types is a wrapper around a specific framework's input types of the equivalent concepts.

Each of these wrappers knows how to convert the specific concepts in to the nd4j equivalents for interpretation by a mapper which applies the mapping rules for a particular framework.

Doing this will allow us to share logic between mappers and making 1 implementation of mapping possible by calling associated getter methods for concepts like data types and nodes.

Serialization

In order to persist rules using protobuf, all rules will know how to serialize themselves. A simple serialize() and load() methods are implemented which covers conversion using interface methods up to the user to implement which describes how to persist the protobuf representation. This applies to any of the relevant functionality such as rules and processes.

Custom types

Some types will not map 1 to 1 or are directly applicable to nd4j. In order to combat this, when an unknown type is discovered during mapping, adapter functions for specific types must be specified.

Supported types include:

1. Long/Int
2. Double/Float
3. String
4. Boolean
5. Bytes
6. NDArrays

An example:

A Dim in tensorflow can be mapped to a long in nd4j.

Shape Information can be a list of longs or multiple lists depending on the context.

Consequences

Advantages

• Allows a language neutral way of describing a set of transforms necessary for mapping an set of operations found in a graph from one framework to the nd4j format.

• Allows a straightforward way of writing an interpreter as well as mappers for different frameworks in nd4j in a standardized way.

• Replaces the old import and makes maintenance of imports/mappers more straightforward.

Disadvantages

• More complexity in the code base instead of a more straightforward java implementation.

• Risks introducing new errors due to a rewrite

Appendix A: Contrasting MappingRules with another implementation

We map names and types to equivalent concepts in each framework. Onnx tensorflow does this with an attribute converter

This is done by a handler (one for each op). More can be found here

Appendix B: Challenges when mapping nd4j ops

The above formats are vastly different. Onnx and tensorflow are purely attribute based. Nd4j is index based. This challenge is addressed by the IR by adding names to each property.

In order to actually map these properties, we need to define rules for doing so. Examples of why these mapping rules are needed below:

1. Different conventions for the same concept. One example that stands out from conv is padding. Padding can be represented as a string or have a boolean that says what a string equals. In nd4j, we represent this as a boolean: isSameMode. We need to do a conversion inline in order to invoke nd4j correctly.

2. Another issue is implicit concepts. Commonly, convolution requires you to configure a layout of NWHC (Batch size, Height, Width, Channels) or NCHW (Batch size, Channels,Height, Width). Tensorflow allows you to specify it, nd4j also allows you to specify it. Onnx does not.

A more in depth conversation on this specific issue relating to the 2 frameworks can be found here In order to address these challenges, we introduce a MappingRule allowing us to define a series of steps to map the input format to the nd4j format in a language neutral way via a protobuf declaration.

Appendix C: A theoretical attribute definition

enum class AttributeValueType {
    FLOAT,
    LIST_FLOAT,
    BYTE,
    LIST_BYTE,
    INT,
    LIST_INT,
    BOOL,
    LIST_BOOL,
    STRING,
    LIST_STRING
}

interface IRAttribute<ATTRIBUTE_TYPE,ATTRIBUTE_VALUE_TYPE> {

    fun name(): String

    fun floatValue(): Double

    fun listFloatValue(): List<Float>

    fun byteValue(): Byte

    fun listByteValue(): List<Byte>

    fun intValue(): Long

    fun listIntValue(): List<Long>

    fun boolValue(): Boolean

    fun listBoolValue(): List<Boolean>

    fun attributeValueType(): AttributeValueType

    fun internalAttributeDef(): ATTRIBUTE_TYPE

    fun internalAttributeValue(): ATTRIBUTE_VALUE_TYPE
}

Appendix D: A theoretical kotlin definition of argument descriptors and op descriptors can be found below:

interface IRArgDef<T,DATA_TYPE> {
    fun name(): String

    fun description(): String

    fun dataType(): IRDataType<DATA_TYPE>

    fun internalValue(): T

    fun indexOf(): Integer
}

interface IROpDef<T,ARG_DEF_TYPE,DATA_TYPE,ATTRIBUTE_TYPE,ATTRIBUTE_VALUE_TYPE> {
    fun opName(): String

    fun internalValue(): T

    fun inputArgs(): List<IRArgDef<ARG_DEF_TYPE,DATA_TYPE>>

    fun outputArgs(): List<IRArgDef<ARG_DEF_TYPE,DATA_TYPE>>

    fun attributes(): List<IRAttribute<ATTRIBUTE_TYPE,ATTRIBUTE_VALUE_TYPE>>

}

##Appendix E: A theoretical kotlin definition of Mapping Rules, MappingProcess and ArgDef can be found below:

interface MappingProcess<T,TENSOR_TYPE,ATTRIBUTE_TYPE,ATTRIBUTE_VALUE_TYPE,DATA_TYPE> {
    fun opName(): String

    fun frameworkVersion(): String

    fun inputFramework(): String

    fun rules(): List<MappingRule<ATTRIBUTE_TYPE,ATTRIBUTE_VALUE_TYPE>>


    fun applyProcess(inputNode: IRNode<T,TENSOR_TYPE,ATTRIBUTE_TYPE,ATTRIBUTE_VALUE_TYPE,DATA_TYPE>): OpDeclarationDescriptor

    fun applyProcessReverse(input: OpDeclarationDescriptor): IRNode<T,TENSOR_TYPE,ATTRIBUTE_TYPE,ATTRIBUTE_VALUE_TYPE,DATA_TYPE>

    fun createDescriptor(argDescriptors: List<OpNamespace.ArgDescriptor>): OpDeclarationDescriptor
}

interface MappingRule<ATTRIBUTE_TYPE,ATTRIBUTE_VALUE_TYPE> {
    fun name(): String

    /**
     * Convert 1 or more attributes in to a list of {@link ArgDescriptor}
     */
    fun convert(inputs: List<IRAttribute<ATTRIBUTE_TYPE,ATTRIBUTE_VALUE_TYPE>> ): List<OpNamespace.ArgDescriptor>

    fun convertReverse(input: List<OpNamespace.ArgDescriptor>): List<IRAttribute<ATTRIBUTE_TYPE,ATTRIBUTE_VALUE_TYPE>>

}