Commit Graph

19 Commits (2258799d2fb83e61d1928339681136cec3e3ae74)

Author SHA1 Message Date
agibsonccc 46dbd0b203 Update copyrights remove attic and relocate elsewhere 2021-02-09 13:16:31 +09:00
agibsonccc 4766032444 Update copyright to mention NOTICES file 2021-02-01 21:31:04 +09:00
agibsonccc c715aea405 Update LICENSE 2021-02-01 17:47:29 +09:00
agibsonccc 65c6a9a42e Dev commits 2021-02-01 14:31:20 +09:00
dariuszzbyrad 4394965cb5
Datavec code cleaup (#9071)
* removed unnecessary semicolons

Signed-off-by: Dariusz Zbyrad <dariusz.zbyrad@gmail.com>

* Use standard charset object

Signed-off-by: Dariusz Zbyrad <dariusz.zbyrad@gmail.com>

* Removed unused imports

Signed-off-by: Dariusz Zbyrad <dariusz.zbyrad@gmail.com>
2020-08-23 10:45:12 +09:00
Alex Black ee3e059b12
DL4J/DataVec: Fix Yolo2OutputLayer and ObjectDetectionRecordReader support for NHWC data format (#483)
* Fix Yolo2OutputLayer for NHWC data format

Signed-off-by: Alex Black <blacka101@gmail.com>

* ObjectDetectionRecordReader NHWC support

Signed-off-by: Alex Black <blacka101@gmail.com>
2020-06-05 11:49:02 +10:00
Alex Black 7651a486e1
#8201 ImageLoader/NativeImageLoader NHWC support (#432)
Signed-off-by: Alex Black <blacka101@gmail.com>
2020-05-04 17:17:57 +10:00
Alex Black 092e6b9891
ImageRecordReader - NHWC support (#431)
* #8201 ImageRecordReader - NHWC support

Signed-off-by: Alex Black <blacka101@gmail.com>

* Check values - NCHW vs. NHWC

Signed-off-by: Alex Black <blacka101@gmail.com>
2020-05-04 14:25:19 +10:00
Alex Black 88d3c4867f
Refactor packages to fix split package issues (#411)
* Refactor nd4j-common: org.nd4j.* -> org.nd4j.common.*

Signed-off-by: Alex Black <blacka101@gmail.com>

* Fix CUDA (missed nd4j-common package refactoring changes)

Signed-off-by: Alex Black <blacka101@gmail.com>

* nd4j-kryo: org.nd4j -> org.nd4j.kryo

Signed-off-by: Alex Black <blacka101@gmail.com>

* Fix nd4j-common for deeplearning4j-cuda

Signed-off-by: Alex Black <blacka101@gmail.com>

* nd4j-grppc-client: org.nd4j.graph -> org.nd4j.remote.grpc

Signed-off-by: Alex Black <blacka101@gmail.com>

* deeplearning4j-common: org.deeplearning4.* -> org.deeplearning4j.common.*

Signed-off-by: Alex Black <blacka101@gmail.com>

* deeplearning4j-core: org.deeplearning4j.* -> org.deeplearning.core.*

Signed-off-by: Alex Black <blacka101@gmail.com>

* deeplearning4j-cuda: org.deeplearning4j.nn.layers.* -> org.deeplearning4j.cuda.*

Signed-off-by: Alex Black <blacka101@gmail.com>

* Import fixes

Signed-off-by: Alex Black <blacka101@gmail.com>

* deeplearning4j-nlp-*: org.deeplearning4.text.* -> org.deeplearning4j.nlp.(language).*

Signed-off-by: Alex Black <blacka101@gmail.com>

* deeplearning4j-ui-model: org.deeplearning4j.ui -> org.deeplearning4j.ui.model

Signed-off-by: Alex Black <blacka101@gmail.com>

* datavec-spark-inference-{server/model/client}: org.datavec.spark.transform -> org.datavec.spark.inference.{server/model/client}

Signed-off-by: Alex Black <blacka101@gmail.com>

* datavec-jdbc: org.datavec.api -> org.datavec.jdbc

Signed-off-by: Alex Black <blacka101@gmail.com>

* Delete org.deeplearning4j.datasets.iterator.impl.MultiDataSetIteratorAdapter in favor of (essentially identical) org.nd4j.linalg.dataset.adapter.MultiDataSetIteratorAdapter

Signed-off-by: Alex Black <blacka101@gmail.com>

* ND4S fixes

Signed-off-by: Alex Black <blacka101@gmail.com>

* Fixes

Signed-off-by: Alex Black <blacka101@gmail.com>

* nd4j-common-tests: org.nd4j.* -> org.nd4j.common.tests

Signed-off-by: Alex Black <blacka101@gmail.com>

* Trigger CI

Signed-off-by: Alex Black <blacka101@gmail.com>

* Fixes

Signed-off-by: Alex Black <blacka101@gmail.com>

* #8878 Ignore CUDA tests on modules with 'nd4j-native under cuda' issue

Signed-off-by: Alex Black <blacka101@gmail.com>

* Fix bad imports in tests

Signed-off-by: Alex Black <blacka101@gmail.com>

* Add ignore on test (already failing) due to #8882

Signed-off-by: Alex Black <blacka101@gmail.com>

* Import fixes

Signed-off-by: Alex Black <blacka101@gmail.com>

* Additional import fixes

Signed-off-by: Alex Black <blacka101@gmail.com>
2020-04-29 11:19:26 +10:00
Alex Black 9e7395667f
MKLDNNConvHelper: don't permute; pass in weight format arg (#412)
* MKLDNNConvHelper: don't permute; pass in weight format arg

Signed-off-by: Alex Black <blacka101@gmail.com>

* Fix flaky tests unrelated to main branch change

Signed-off-by: Alex Black <blacka101@gmail.com>

* Don't log every extracted file for LFW and other auto-downloaded image loaders using BaseImageLoader.downloadAndUntar

Signed-off-by: Alex Black <blacka101@gmail.com>
2020-04-26 01:04:31 +10:00
Alexander Stoyakin ccb216a3ed
Fixing issues from Sonar report (#391)
* Fixing issues from Sonar report

* Proper logger of exceptions

* Coding style fixes

* Use dup parameter

* Cleanup, minor issues

* Cuda compilation fixed and some minor fixes
2020-04-23 08:36:49 +10:00
Alexander Stoyakin 455a5d112d
Fixes for codegen generated classes and build improvements (#367)
* Input format extended

* Deleted redundant code

* Added weights format to conv2d config

* Refactoring

* dl4j base test functionality

* Different tests base class per module

* Check base class for dl4j-graph subproject tests

* Check if test classes extend BaseDL4JTest

* Use nd4j-common-tests as transient dependency

* Enums and tests added

* Added codegenerated methods

* Use namespace methods

* Replace DifferentialFunctionFactory  with codegenerated classes

* Fixed linspace

* Namespaces regenerated

* Namespaces used instead of factory

* Regenerated base classes

* Input format extended

* Added weights format to conv2d config

* Refactoring

* dl4j base test functionality

* Different tests base class per module

* Check base class for dl4j-graph subproject tests

* Check if test classes extend BaseDL4JTest

* Use nd4j-common-tests as transient dependency

* Enums and tests added

* Added codegenerated methods

* Use namespace methods

* Replace DifferentialFunctionFactory  with codegenerated classes

* Fixed linspace

* Namespaces regenerated

* Regenerated base classes

* Regenerated namespaces

* Generate nd4j namespaces

* INDArrays accepting constructors

* Generated some ops

* Some fixes

* SameDiff ops regenerated

* Regenerated nd4j ops

* externalErrors moved

* Compilation fixes

* SquaredDifference - strict number of args

* Deprecated code cleanup. Proper base class for tests.

* Extend test classes with BaseND4JTest

* Extend test classes with BaseDL4JTest

* Legacy code

* DL4J cleanup

* Exclude test utils from base class check

* Tests fixed

* Arbiter tests fix

* Test dependency scope fix + pom.xml formatting

Signed-off-by: Alex Black <blacka101@gmail.com>

* Significant number of fixes

Signed-off-by: Alex Black <blacka101@gmail.com>

* Another round of fixes

Signed-off-by: Alex Black <blacka101@gmail.com>

* Another round of fixes

Signed-off-by: Alex Black <blacka101@gmail.com>

* Few additional fixes

Signed-off-by: Alex Black <blacka101@gmail.com>

* DataVec missing test scope dependencies

Signed-off-by: Alex Black <blacka101@gmail.com>

Co-authored-by: Alex Black <blacka101@gmail.com>
2020-04-20 10:27:13 +10:00
Alex Black 81ebfeead1
Small fixes (#355)
* #8787 DataVec test fix

Signed-off-by: Alex Black <blacka101@gmail.com>

* New nd4j test + fix bad datavec test

Signed-off-by: Alex Black <blacka101@gmail.com>

* #8745 Fix flaky arbiter test

Signed-off-by: Alex Black <blacka101@gmail.com>

* One more test

Signed-off-by: Alex Black <blacka101@gmail.com>
2020-04-01 15:11:39 +11:00
Samuel Audet 58aa5a3a9b
RL4J: Add TransformProcess, part 1 (#8711)
* Added TransformProcess, part 1

Signed-off-by: unknown <aboulang2002@yahoo.com>

* Renamed TemporalMergeTransform to HistoryMergeTransform

Signed-off-by: unknown <aboulang2002@yahoo.com>

* changed INDArrayHelper to use Nd4j.expandDims

Signed-off-by: Alexandre Boulanger <aboulang2002@yahoo.com>

* Adjusted copyrights

Signed-off-by: unknown <aboulang2002@yahoo.com>
2020-03-05 14:43:13 +09:00
Alex Black 29104083cc
Various fixes (#143)
* #8568 ArrayUtil optimization

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* #6171 Keras ReLU and ELU support

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Keras softmax layer import

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* #8549 Webjars dependency management

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Fix for TF import names ':0' suffix issue / NPE

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* BiasAdd: fix default data format for TF import

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Update zoo test ignores

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* #8509 SameDiff Listener API - provide frame + iteration

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* #8520 ND4J Environment

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Deconv3d

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Deconv3d fixes + gradient check

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Conv3d fixes + deconv3d DType test

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Fix issue with deconv3d gradinet check weight init

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* #8579 Fix BaseCudaDataBuffer constructor fix for UINT16

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* DataType.isNumerical() returns false for BOOL type

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* #8504 Reduce Spark log spam for tests

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Clean up DL4J gradient check test spam

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* More Gradient check spam reduction

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* SameDiff test spam reduction

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Fixes for FlatBuffers mapping

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* SameDiff log spam cleanup

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Tests should extend BaseNd4jTest

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Remove debug line in c++ op

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* ND4J test spam cleanup

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* DL4J test spam reduction

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* More Dl4J and datavec test spam cleanup

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Fix for bad conv3d test

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Additional test

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Embedding layers: don't inherit global default activation function

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Trigger CI

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Consolidate all BaseDL4JTest classes to single class used everywhere; make timeout configurable per class

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Test fixes and timeout increases

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Timeouts and PReLU fixes

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Restore libnd4j build threads arg for CUDA build

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Increase timeouts on a few tests to avoid spurious failures on some CI machines

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* More timeout fixes

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* More test timeout fixes

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Tweak timeout for one more test

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Final tweaks

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* One more ignore

Signed-off-by: AlexDBlack <blacka101@gmail.com>
2020-01-04 13:45:07 +11:00
Alex Black bfd9e3692a
Add op counting to TensorFlowImportValidator (#128)
* Add op counting to TensorFlowImportValidator

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Test tweak

Signed-off-by: AlexDBlack <blacka101@gmail.com>
2019-12-17 10:23:37 +11:00
Alex Black 4920f22fff
Check for empty streams for NativeImageLoader + test (#121)
Signed-off-by: AlexDBlack <blacka101@gmail.com>
2019-12-10 12:11:05 +11:00
Alex Black dcc2baa676
Version upgrades (#199)
* DataVec fixes for Jackson version upgrade

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* DL4J jackson updates + databind version 2.9.9.3

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Shade snakeyaml along with jackson

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Version fix

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Switch DataVec legacy JSON format handling to mixins

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Next set of fixes

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Cleanup for legacy JSON mapping

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Upgrade commons compress to 1.18; small test fix

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* New Jackson backward compatibility for DL4J - Round 1

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* New Jackson backward compatibility for DL4J - Round 2

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* More fixes, all but legacy custom passing

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Provide an upgrade path for custom layers for models in pre-1.0.0-beta JSON format

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Legacy deserialization cleanup

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Small amount of polish - legacy JSON

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Upgrade guava version

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* IEvaluation legacy format deserialization fix

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Upgrade play version to 2.7.3

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Update nd4j-parameter-server-status to new Play API

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Update DL4J UI for new play version

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* More play framework updates

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Small fixes

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Remove Spark 1/2 adapter code from DataVec

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* datavec-spark dependency cleanup

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* DL4J spark updates, pt 1

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* DL4J spark updates, pt 2

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* DL4J spark updates, pt 3

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* DL4J spark updates, pt 4

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Test fix

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Another fix

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Breeze upgrade, dependency cleanup

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Add Scala 2.12 version to pom.xml

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* change-scala-versions.sh - add scala 2.12, remove 2.10

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Move Spark version properties to parent pom (now that only one spark version is supported)

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* DataVec Play fixes

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* datavec play dependency fixes

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Clean up old spark/jackson stuff

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Cleanup jackson unused dependencies

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Dropping redundant dependency

Signed-off-by: Alexander Stoyakin <alexander.stoyakin@gmail.com>

* Removed scalaxy dependency

Signed-off-by: Alexander Stoyakin <alexander.stoyakin@gmail.com>

* DataVec fixes for Jackson version upgrade

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* DL4J jackson updates + databind version 2.9.9.3

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Shade snakeyaml along with jackson

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Version fix

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Switch DataVec legacy JSON format handling to mixins

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Next set of fixes

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Cleanup for legacy JSON mapping

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Upgrade commons compress to 1.18; small test fix

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* New Jackson backward compatibility for DL4J - Round 1

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* New Jackson backward compatibility for DL4J - Round 2

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* More fixes, all but legacy custom passing

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Provide an upgrade path for custom layers for models in pre-1.0.0-beta JSON format

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Legacy deserialization cleanup

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Small amount of polish - legacy JSON

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Upgrade guava version

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* IEvaluation legacy format deserialization fix

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Upgrade play version to 2.7.3

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Update nd4j-parameter-server-status to new Play API

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Update DL4J UI for new play version

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* More play framework updates

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Small fixes

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Remove Spark 1/2 adapter code from DataVec

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* datavec-spark dependency cleanup

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* DL4J spark updates, pt 1

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* DL4J spark updates, pt 2

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* DL4J spark updates, pt 3

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* DL4J spark updates, pt 4

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Test fix

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Another fix

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Breeze upgrade, dependency cleanup

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Add Scala 2.12 version to pom.xml

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* change-scala-versions.sh - add scala 2.12, remove 2.10

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Move Spark version properties to parent pom (now that only one spark version is supported)

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* DataVec Play fixes

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* datavec play dependency fixes

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Clean up old spark/jackson stuff

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Cleanup jackson unused dependencies

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Add shaded guava

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Dropping redundant dependency

Signed-off-by: Alexander Stoyakin <alexander.stoyakin@gmail.com>

* Removed scalaxy dependency

Signed-off-by: Alexander Stoyakin <alexander.stoyakin@gmail.com>

* Ensure not possible to import pre-shaded classes, and remove direct guava dependencies in favor of shaded

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* ND4J Shaded guava import fixes

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* DataVec and DL4J guava shading

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Arbiter, RL4J fixes

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Build fixed

Signed-off-by: Alexander Stoyakin <alexander.stoyakin@gmail.com>

* Fix dependency

Signed-off-by: Alexander Stoyakin <alexander.stoyakin@gmail.com>

* Fix bad merge

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Jackson shading fixes

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Set play secret, datavec-spark-inference-server

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Fix for datavec-spark-inference-server

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Arbiter fixes

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Arbiter fixes

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Small test fix

Signed-off-by: AlexDBlack <blacka101@gmail.com>
2019-08-30 14:35:27 +10:00
skymindops b5f0ec072f Eclipse Migration Initial Commit 2019-06-06 15:21:15 +03:00