Commit Graph

16 Commits (de625baea68e35b892c4045d237a0f5e44343d6b)

Author SHA1 Message Date
Alex Black 88d3c4867f
Refactor packages to fix split package issues (#411)
* Refactor nd4j-common: org.nd4j.* -> org.nd4j.common.*

Signed-off-by: Alex Black <blacka101@gmail.com>

* Fix CUDA (missed nd4j-common package refactoring changes)

Signed-off-by: Alex Black <blacka101@gmail.com>

* nd4j-kryo: org.nd4j -> org.nd4j.kryo

Signed-off-by: Alex Black <blacka101@gmail.com>

* Fix nd4j-common for deeplearning4j-cuda

Signed-off-by: Alex Black <blacka101@gmail.com>

* nd4j-grppc-client: org.nd4j.graph -> org.nd4j.remote.grpc

Signed-off-by: Alex Black <blacka101@gmail.com>

* deeplearning4j-common: org.deeplearning4.* -> org.deeplearning4j.common.*

Signed-off-by: Alex Black <blacka101@gmail.com>

* deeplearning4j-core: org.deeplearning4j.* -> org.deeplearning.core.*

Signed-off-by: Alex Black <blacka101@gmail.com>

* deeplearning4j-cuda: org.deeplearning4j.nn.layers.* -> org.deeplearning4j.cuda.*

Signed-off-by: Alex Black <blacka101@gmail.com>

* Import fixes

Signed-off-by: Alex Black <blacka101@gmail.com>

* deeplearning4j-nlp-*: org.deeplearning4.text.* -> org.deeplearning4j.nlp.(language).*

Signed-off-by: Alex Black <blacka101@gmail.com>

* deeplearning4j-ui-model: org.deeplearning4j.ui -> org.deeplearning4j.ui.model

Signed-off-by: Alex Black <blacka101@gmail.com>

* datavec-spark-inference-{server/model/client}: org.datavec.spark.transform -> org.datavec.spark.inference.{server/model/client}

Signed-off-by: Alex Black <blacka101@gmail.com>

* datavec-jdbc: org.datavec.api -> org.datavec.jdbc

Signed-off-by: Alex Black <blacka101@gmail.com>

* Delete org.deeplearning4j.datasets.iterator.impl.MultiDataSetIteratorAdapter in favor of (essentially identical) org.nd4j.linalg.dataset.adapter.MultiDataSetIteratorAdapter

Signed-off-by: Alex Black <blacka101@gmail.com>

* ND4S fixes

Signed-off-by: Alex Black <blacka101@gmail.com>

* Fixes

Signed-off-by: Alex Black <blacka101@gmail.com>

* nd4j-common-tests: org.nd4j.* -> org.nd4j.common.tests

Signed-off-by: Alex Black <blacka101@gmail.com>

* Trigger CI

Signed-off-by: Alex Black <blacka101@gmail.com>

* Fixes

Signed-off-by: Alex Black <blacka101@gmail.com>

* #8878 Ignore CUDA tests on modules with 'nd4j-native under cuda' issue

Signed-off-by: Alex Black <blacka101@gmail.com>

* Fix bad imports in tests

Signed-off-by: Alex Black <blacka101@gmail.com>

* Add ignore on test (already failing) due to #8882

Signed-off-by: Alex Black <blacka101@gmail.com>

* Import fixes

Signed-off-by: Alex Black <blacka101@gmail.com>

* Additional import fixes

Signed-off-by: Alex Black <blacka101@gmail.com>
2020-04-29 11:19:26 +10:00
Fariz Rahman 4cb87a94e8
tf.keras import test and fixes (#347)
* merge conf

* merge conf

* tfkeras tests

* parameterized tests

* rename

* cuda versions

* jccp versions

* 'updates'

* updates

* rnn+mlp passing

* repeat

* updates

* tests

* Update pom.xml

* Update pom.xml

* rem print

* cnn1d model conversion fixed

* cnn1d activate fixed

* cnn1d outptut shape fix

* cnn1d bprop fix

* cnn1d stack fix

* KerasModelEndToEndTest - Remove permutes for NWC and NHWC format tests

Signed-off-by: Alex Black <blacka101@gmail.com>

* Fixes and update test - input shapes (NCHW -> NHWC input)

Signed-off-by: Alex Black <blacka101@gmail.com>

* Ignore for known bad tests

Signed-off-by: Alex Black <blacka101@gmail.com>

* Multiple fixes - MergeVertex, CNN1D layers, etc

Signed-off-by: Alex Black <blacka101@gmail.com>

* Fix issue with RNN/FF preprocessors, time distributed etc with NWC format

Signed-off-by: Alex Black <blacka101@gmail.com>

* LSTM NWC dropout fix

Signed-off-by: Alex Black <blacka101@gmail.com>

* Add sequence embedding layer NWC support (configurable output format)

Signed-off-by: Alex Black <blacka101@gmail.com>

* Fix expected shape in a couple of tests - NWC expected

Signed-off-by: Alex Black <blacka101@gmail.com>

* Fix EmbeddingSequenceLayer backprop for NWC output case + add gradient checks

Signed-off-by: Alex Black <blacka101@gmail.com>

* CnnToFeedForwardPreprocessor: align with Keras/TF; fix Keras reshape/flatten

Signed-off-by: Alex Black <blacka101@gmail.com>

* Update ConvDataFormatTests to match new reshape behaviour

Signed-off-by: Alex Black <blacka101@gmail.com>

* Switch hard-coded path to ResourceUtils.listClassPathfiles for TestTFKerasModelImport

Signed-off-by: Alex Black <blacka101@gmail.com>

* TestUtils fix

Signed-off-by: Alex Black <blacka101@gmail.com>

* Fixes

Signed-off-by: Alex Black <blacka101@gmail.com>

* Fix JSON serde issue with data formats

Signed-off-by: Alex Black <blacka101@gmail.com>

* Fix for input dtype inference; fix 2 tests

Signed-off-by: Alex Black <blacka101@gmail.com>

* Test fixes

Signed-off-by: Alex Black <blacka101@gmail.com>

* #8891 Ignore for TestVertxUIMultiSession until fixed

Signed-off-by: Alex Black <blacka101@gmail.com>

* Restore but deprecate TensorFlowCnnToFeedForwardPreProcessor for older zoo models

Signed-off-by: Alex Black <blacka101@gmail.com>

* Ignore for deprecated preprocessor in DTypeTests

Signed-off-by: Alex Black <blacka101@gmail.com>

* Remove debug printlns

Signed-off-by: Alex Black <blacka101@gmail.com>

Co-authored-by: Alex Black <blacka101@gmail.com>
2020-04-28 20:31:09 +10:00
Alexander Stoyakin 455a5d112d
Fixes for codegen generated classes and build improvements (#367)
* Input format extended

* Deleted redundant code

* Added weights format to conv2d config

* Refactoring

* dl4j base test functionality

* Different tests base class per module

* Check base class for dl4j-graph subproject tests

* Check if test classes extend BaseDL4JTest

* Use nd4j-common-tests as transient dependency

* Enums and tests added

* Added codegenerated methods

* Use namespace methods

* Replace DifferentialFunctionFactory  with codegenerated classes

* Fixed linspace

* Namespaces regenerated

* Namespaces used instead of factory

* Regenerated base classes

* Input format extended

* Added weights format to conv2d config

* Refactoring

* dl4j base test functionality

* Different tests base class per module

* Check base class for dl4j-graph subproject tests

* Check if test classes extend BaseDL4JTest

* Use nd4j-common-tests as transient dependency

* Enums and tests added

* Added codegenerated methods

* Use namespace methods

* Replace DifferentialFunctionFactory  with codegenerated classes

* Fixed linspace

* Namespaces regenerated

* Regenerated base classes

* Regenerated namespaces

* Generate nd4j namespaces

* INDArrays accepting constructors

* Generated some ops

* Some fixes

* SameDiff ops regenerated

* Regenerated nd4j ops

* externalErrors moved

* Compilation fixes

* SquaredDifference - strict number of args

* Deprecated code cleanup. Proper base class for tests.

* Extend test classes with BaseND4JTest

* Extend test classes with BaseDL4JTest

* Legacy code

* DL4J cleanup

* Exclude test utils from base class check

* Tests fixed

* Arbiter tests fix

* Test dependency scope fix + pom.xml formatting

Signed-off-by: Alex Black <blacka101@gmail.com>

* Significant number of fixes

Signed-off-by: Alex Black <blacka101@gmail.com>

* Another round of fixes

Signed-off-by: Alex Black <blacka101@gmail.com>

* Another round of fixes

Signed-off-by: Alex Black <blacka101@gmail.com>

* Few additional fixes

Signed-off-by: Alex Black <blacka101@gmail.com>

* DataVec missing test scope dependencies

Signed-off-by: Alex Black <blacka101@gmail.com>

Co-authored-by: Alex Black <blacka101@gmail.com>
2020-04-20 10:27:13 +10:00
Shams Ul Azeem 4e8f3a025f
Fixing python object for obtaining scalars (#330)
* Fixing python object for obtaining scalars

Signed-off-by: shams <shamsazeem20@gmail.com>

* Fix variable name for stridePtr

Signed-off-by: shams <shamsazeem20@gmail.com>

* Fix variable name for stridePtr

Signed-off-by: shams <shamsazeem20@gmail.com>

Co-authored-by: Alex Black <blacka101@gmail.com>
2020-03-24 19:11:57 +11:00
Shams Ul Azeem 9c77bfa85f
Support for more numpy datatypes (#241)
* Adding more datatypes support in datavec-python

* Using numpy C API for creating numpy arrays

* Adding parameterized tests

* Adding support for BFLOAT16 (by converting it to FLOAT)

* Cleanup

* Using casting instead of creating an array

* Giving out a warning while casting array from BFLOAT16 to FLOAT

* Add syncToPrimary and syncToSpecial methods to BaseDataBuffer

Signed-off-by: Alex Black <blacka101@gmail.com>

* Python exec: sync to host before passing pointers

Signed-off-by: Alex Black <blacka101@gmail.com>

* Added copyright header

* use np api (#267)

* python exec / numpy - check object type before cast (#268)

* use np api

* verify object before cast

* fix cong

* cuda fix

* inplace test + tiny fix

* more test

* fix double alloc

* rem tags

* fix cuda check

* Fix implicit CUDA dependency in datavec-python tests; remove new method, add test

Signed-off-by: Alex Black <blacka101@gmail.com>

Co-authored-by: Alex Black <blacka101@gmail.com>
Co-authored-by: Fariz Rahman <farizrahman4u@gmail.com>
2020-03-19 00:48:37 +11:00
Shams Ul Azeem 5c9e0bc2bb
Ignore none type for pythonexception (#237)
* Making TypeName enum public

* Ignoring None type object for PythonExceptions

* better handling of None + test

Co-authored-by: Fariz Rahman <farizrahman4u@gmail.com>
2020-02-13 09:58:39 +11:00
Shams Ul Azeem b941186302
Making TypeName enum public (#235) 2020-02-12 21:17:30 +11:00
Fariz Rahman c9ffb6cbec Python: Use memoryview instead of bytearray (#225)
* memoryview

* cleanup
2020-02-10 21:54:44 +11:00
Fariz Rahman f6b3032def
Python executioner 2.0 (#134)
* wrapper

* builtins

* context mgr

* direct place

* ret all var

* call fix

* fix ndarray serde

* jobs

* try-with gil management

* cleanup

* exec tests passing

* list tests

* transforms test passing

* all pass

* headers

* dict fixes+test

* python path

* bool isinstance

* job tests

* nits

* transform fix+test

* transform tests

* leak fixes

* more mem leak fixes

* more fixes

* nits for adam

* PythonJob lombok builder

* checked exceptions

* more nits

* small leak fix

* more nits

* pythonexceptions

* fix jvm crash when bad python code

* Exception->PythonException

* Add support for boolean types in arrow records and ability to cast from float, double to int for TypeConversion (#178)

* nits for alex

* update tests

* fix test

* all pass

* refacc

* rem old code

* dtypes

* bytes working+exception pass through+cleanup (#209)

* more bytes tests

* header

* rem dummy test

* rem bad import

* alex nits + refacc

* Small error fixes (wrong type in msg) + minor formatting

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* use actual python type names (dictionary->dict, boolean->bool)

Co-authored-by: Shams Ul Azeem <shamsazeem20@gmail.com>
Co-authored-by: Alex Black <blacka101@gmail.com>
2020-02-04 20:23:59 +11:00
Fariz Rahman 709295281e suppress err (#123) 2019-12-12 09:40:29 +03:00
Fariz Rahman 0e8a4f77bc datavec python ensure host (#113)
* ensure host

* one more host ensure

* info->debug
2019-12-05 23:27:32 +11:00
Fariz Rahman 1adc25919c Python updates (#86)
* python updates

* fix cyclic deps

* konduit updates

* konduit updates

* fix list

* fixes

* sync pyvars test

* setuprun comments

* Version fix, other module test fixes

Signed-off-by: Alex Black <blacka101@gmail.com>

* bug fix using advanced hacking skillzz
2019-12-02 19:20:23 +11:00
Samuel Audet 5e07998e59 Add support for CUDA 10.2 (#89) 2019-11-29 16:31:03 +11:00
Alex Black 4d331f0993
Fix test profiles (#197)
* Fix DataVec profiles

Signed-off-by: Alex Black <blacka101@gmail.com>

* arbiter maven profile fixes

Signed-off-by: Alex Black <blacka101@gmail.com>
2019-08-30 11:33:31 +10:00
Serhii Shepel 0463ee4eba Fix backend dependencies for tests (#189) 2019-08-29 12:54:48 +09:00
skymindops b5f0ec072f Eclipse Migration Initial Commit 2019-06-06 15:21:15 +03:00