Commit Graph

14 Commits (0a865f6ee3d9fdd8b956a9ad46a71b5fb939b2ed)

Author SHA1 Message Date
Tamás Fenyvesi f33b08e20f
shutdown hook for UI server (#9005)
* shutdown hook for UI server

Signed-off-by: Tamás Fenyvesi <tamas.fenyvesi@doknet.hu>

* logging exception

Signed-off-by: Tamás Fenyvesi <tamas.fenyvesi@doknet.hu>
2020-07-25 20:30:49 +09:00
Tamás Fenyvesi 9f47c8aca8 narrow down UIServer.stop() to declare throwing InterruptedException; fix typo in test
Signed-off-by: Tamás Fenyvesi <tamas.fenyvesi@doknet.hu>
2020-05-08 13:08:01 +02:00
Tamás Fenyvesi cf24728f35 tests for auto-attach and manual attach; improve JavaDoc
Signed-off-by: Tamás Fenyvesi <tamas.fenyvesi@doknet.hu>
2020-05-08 13:08:00 +02:00
Tamás Fenyvesi ef7c21c204 fix logging of UI server auto-stop, fix indentation in test
Signed-off-by: Tamás Fenyvesi <tamas.fenyvesi@doknet.hu>
2020-05-08 13:02:30 +02:00
Alex Black 88d3c4867f
Refactor packages to fix split package issues (#411)
* Refactor nd4j-common: org.nd4j.* -> org.nd4j.common.*

Signed-off-by: Alex Black <blacka101@gmail.com>

* Fix CUDA (missed nd4j-common package refactoring changes)

Signed-off-by: Alex Black <blacka101@gmail.com>

* nd4j-kryo: org.nd4j -> org.nd4j.kryo

Signed-off-by: Alex Black <blacka101@gmail.com>

* Fix nd4j-common for deeplearning4j-cuda

Signed-off-by: Alex Black <blacka101@gmail.com>

* nd4j-grppc-client: org.nd4j.graph -> org.nd4j.remote.grpc

Signed-off-by: Alex Black <blacka101@gmail.com>

* deeplearning4j-common: org.deeplearning4.* -> org.deeplearning4j.common.*

Signed-off-by: Alex Black <blacka101@gmail.com>

* deeplearning4j-core: org.deeplearning4j.* -> org.deeplearning.core.*

Signed-off-by: Alex Black <blacka101@gmail.com>

* deeplearning4j-cuda: org.deeplearning4j.nn.layers.* -> org.deeplearning4j.cuda.*

Signed-off-by: Alex Black <blacka101@gmail.com>

* Import fixes

Signed-off-by: Alex Black <blacka101@gmail.com>

* deeplearning4j-nlp-*: org.deeplearning4.text.* -> org.deeplearning4j.nlp.(language).*

Signed-off-by: Alex Black <blacka101@gmail.com>

* deeplearning4j-ui-model: org.deeplearning4j.ui -> org.deeplearning4j.ui.model

Signed-off-by: Alex Black <blacka101@gmail.com>

* datavec-spark-inference-{server/model/client}: org.datavec.spark.transform -> org.datavec.spark.inference.{server/model/client}

Signed-off-by: Alex Black <blacka101@gmail.com>

* datavec-jdbc: org.datavec.api -> org.datavec.jdbc

Signed-off-by: Alex Black <blacka101@gmail.com>

* Delete org.deeplearning4j.datasets.iterator.impl.MultiDataSetIteratorAdapter in favor of (essentially identical) org.nd4j.linalg.dataset.adapter.MultiDataSetIteratorAdapter

Signed-off-by: Alex Black <blacka101@gmail.com>

* ND4S fixes

Signed-off-by: Alex Black <blacka101@gmail.com>

* Fixes

Signed-off-by: Alex Black <blacka101@gmail.com>

* nd4j-common-tests: org.nd4j.* -> org.nd4j.common.tests

Signed-off-by: Alex Black <blacka101@gmail.com>

* Trigger CI

Signed-off-by: Alex Black <blacka101@gmail.com>

* Fixes

Signed-off-by: Alex Black <blacka101@gmail.com>

* #8878 Ignore CUDA tests on modules with 'nd4j-native under cuda' issue

Signed-off-by: Alex Black <blacka101@gmail.com>

* Fix bad imports in tests

Signed-off-by: Alex Black <blacka101@gmail.com>

* Add ignore on test (already failing) due to #8882

Signed-off-by: Alex Black <blacka101@gmail.com>

* Import fixes

Signed-off-by: Alex Black <blacka101@gmail.com>

* Additional import fixes

Signed-off-by: Alex Black <blacka101@gmail.com>
2020-04-29 11:19:26 +10:00
Fariz Rahman 4cb87a94e8
tf.keras import test and fixes (#347)
* merge conf

* merge conf

* tfkeras tests

* parameterized tests

* rename

* cuda versions

* jccp versions

* 'updates'

* updates

* rnn+mlp passing

* repeat

* updates

* tests

* Update pom.xml

* Update pom.xml

* rem print

* cnn1d model conversion fixed

* cnn1d activate fixed

* cnn1d outptut shape fix

* cnn1d bprop fix

* cnn1d stack fix

* KerasModelEndToEndTest - Remove permutes for NWC and NHWC format tests

Signed-off-by: Alex Black <blacka101@gmail.com>

* Fixes and update test - input shapes (NCHW -> NHWC input)

Signed-off-by: Alex Black <blacka101@gmail.com>

* Ignore for known bad tests

Signed-off-by: Alex Black <blacka101@gmail.com>

* Multiple fixes - MergeVertex, CNN1D layers, etc

Signed-off-by: Alex Black <blacka101@gmail.com>

* Fix issue with RNN/FF preprocessors, time distributed etc with NWC format

Signed-off-by: Alex Black <blacka101@gmail.com>

* LSTM NWC dropout fix

Signed-off-by: Alex Black <blacka101@gmail.com>

* Add sequence embedding layer NWC support (configurable output format)

Signed-off-by: Alex Black <blacka101@gmail.com>

* Fix expected shape in a couple of tests - NWC expected

Signed-off-by: Alex Black <blacka101@gmail.com>

* Fix EmbeddingSequenceLayer backprop for NWC output case + add gradient checks

Signed-off-by: Alex Black <blacka101@gmail.com>

* CnnToFeedForwardPreprocessor: align with Keras/TF; fix Keras reshape/flatten

Signed-off-by: Alex Black <blacka101@gmail.com>

* Update ConvDataFormatTests to match new reshape behaviour

Signed-off-by: Alex Black <blacka101@gmail.com>

* Switch hard-coded path to ResourceUtils.listClassPathfiles for TestTFKerasModelImport

Signed-off-by: Alex Black <blacka101@gmail.com>

* TestUtils fix

Signed-off-by: Alex Black <blacka101@gmail.com>

* Fixes

Signed-off-by: Alex Black <blacka101@gmail.com>

* Fix JSON serde issue with data formats

Signed-off-by: Alex Black <blacka101@gmail.com>

* Fix for input dtype inference; fix 2 tests

Signed-off-by: Alex Black <blacka101@gmail.com>

* Test fixes

Signed-off-by: Alex Black <blacka101@gmail.com>

* #8891 Ignore for TestVertxUIMultiSession until fixed

Signed-off-by: Alex Black <blacka101@gmail.com>

* Restore but deprecate TensorFlowCnnToFeedForwardPreProcessor for older zoo models

Signed-off-by: Alex Black <blacka101@gmail.com>

* Ignore for deprecated preprocessor in DTypeTests

Signed-off-by: Alex Black <blacka101@gmail.com>

* Remove debug printlns

Signed-off-by: Alex Black <blacka101@gmail.com>

Co-authored-by: Alex Black <blacka101@gmail.com>
2020-04-28 20:31:09 +10:00
Alex Black 0a1f75a5d9 Merge & fix small test conflict
Signed-off-by: Alex Black <blacka101@gmail.com>
2020-04-23 10:31:15 +10:00
Tamás Fenyvesi 722d5a052a
Fix UIServer features in multi-session mode, synchronous start and stop (#8856)
* fix Freemarker version mismatch: change version requested in TrainModule to 2.3.23 (freemarker.version in pom.xml)

Signed-off-by: Tamás Fenyvesi <tamas.fenyvesi@doknet.hu>

* fix checking multi-session mode in VertxUIServer.getInstance. Tested multiple calls in TestVertxUIMultiSession.

Signed-off-by: Tamás Fenyvesi <tamas.fenyvesi@doknet.hu>

* fix UIServer.getInstance() to return existing instance

Signed-off-by: Tamás Fenyvesi <tamas.fenyvesi@doknet.hu>

* extend timeout for manual UI tests from 30 to 600 seconds

Signed-off-by: Tamás Fenyvesi <tamas.fenyvesi@doknet.hu>

* start and stop UI server synchronously (wait until complete), tests

Signed-off-by: Tamás Fenyvesi <tamas.fenyvesi@doknet.hu>

* fix for auto-attaching StatsStorage given in VertxUIServer#getInstance(Integer, boolean, Function<String,StatsStorage>), test improvements

Signed-off-by: Tamás Fenyvesi <tamas.fenyvesi@doknet.hu>

* exception handling, test improvements

Signed-off-by: Tamás Fenyvesi <tamas.fenyvesi@doknet.hu>

* add asynchronous method to start UI server

Signed-off-by: Tamás Fenyvesi <tamas.fenyvesi@doknet.hu>

* fix UIServer.getInstance() to return existing instance

Signed-off-by: Tamás Fenyvesi <tamas.fenyvesi@doknet.hu>

* fix UI server language setting in multi-session mode

Signed-off-by: Tamás Fenyvesi <tamas.fenyvesi@doknet.hu>

* fix UI server system tab not loading data in multi-session mode

Signed-off-by: Tamás Fenyvesi <tamas.fenyvesi@doknet.hu>

* undo added InterruptedException in UIServer.getInstance()

Signed-off-by: Tamás Fenyvesi <tamas.fenyvesi@doknet.hu>

* fix async stopping of UIServer.stopAsync(Promise<Void>), added test

Signed-off-by: Tamás Fenyvesi <tamas.fenyvesi@doknet.hu>

* restore the daemon thread style behaviour of UIServer: don't keep the process alive just because the UI is running

Signed-off-by: Tamás Fenyvesi <tamas.fenyvesi@doknet.hu>

* speed up and don't @Ignore tests in TestVertxUI and TestVertxUIMultiSession, put longer tests to separate class TestVertxUIManual

Signed-off-by: Tamás Fenyvesi <tamas.fenyvesi@doknet.hu>
2020-04-23 10:26:51 +10:00
Alexander Stoyakin ccb216a3ed
Fixing issues from Sonar report (#391)
* Fixing issues from Sonar report

* Proper logger of exceptions

* Coding style fixes

* Use dup parameter

* Cleanup, minor issues

* Cuda compilation fixed and some minor fixes
2020-04-23 08:36:49 +10:00
Alexander Stoyakin 0a27e9f41d
Fix incompatibilities with generated code (#303)
* Cholesky fixed

* Constructors added

* MatMul wrapper

* Constructor added

* Missing wrappers added

* Generate Linalg namespace added

* Output data types

* Unit tests

* Added mmul

* Code generation

* Code generated

* Build fixed

* Fixing signatures

* Tests fixed

* Tests fixed

* Added enum

* Fix tests

* Some fixes

* Eye test fixed

* SameDiff: small fix for renameVariable - also replace variable name in lossVariable list if necessary

Signed-off-by: Alex Black <blacka101@gmail.com>

* Some fixes

* Tests fixed

* Revert wrong fix

* Some fixes

* Some fixes

* Extending base test class

* Added pad

* Fixed for generated signatures

* Fixes due to nd4j codegen

* Backwards compatibility fixes

* Fixed errors in tests, reverted wrong changes

* Test fixed

* Added missing operations used for nd4s operators

* Compilation fixed

* Added meshgrid

* Fixed constructors

* fixes

Signed-off-by: Alex Black <blacka101@gmail.com>

* Fix bad commit (incorrectly reverted change from master)

Signed-off-by: Alex Black <blacka101@gmail.com>

* Fixed test

Co-authored-by: Alex Black <blacka101@gmail.com>
2020-04-01 12:00:38 +11:00
Alex Black 0756e3fe70
Small fixes. (#206)
* Logging format tweaks for file logging

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Min abs error tweak for Util layer gradient checks

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* #8648 Fix SameDiff NPE instead of error for missing placeholders

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Test runtime reduction

Signed-off-by: AlexDBlack <blacka101@gmail.com>
2020-02-01 18:19:36 +11:00
Alex Black 29104083cc
Various fixes (#143)
* #8568 ArrayUtil optimization

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* #6171 Keras ReLU and ELU support

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Keras softmax layer import

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* #8549 Webjars dependency management

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Fix for TF import names ':0' suffix issue / NPE

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* BiasAdd: fix default data format for TF import

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Update zoo test ignores

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* #8509 SameDiff Listener API - provide frame + iteration

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* #8520 ND4J Environment

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Deconv3d

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Deconv3d fixes + gradient check

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Conv3d fixes + deconv3d DType test

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Fix issue with deconv3d gradinet check weight init

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* #8579 Fix BaseCudaDataBuffer constructor fix for UINT16

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* DataType.isNumerical() returns false for BOOL type

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* #8504 Reduce Spark log spam for tests

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Clean up DL4J gradient check test spam

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* More Gradient check spam reduction

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* SameDiff test spam reduction

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Fixes for FlatBuffers mapping

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* SameDiff log spam cleanup

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Tests should extend BaseNd4jTest

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Remove debug line in c++ op

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* ND4J test spam cleanup

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* DL4J test spam reduction

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* More Dl4J and datavec test spam cleanup

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Fix for bad conv3d test

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Additional test

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Embedding layers: don't inherit global default activation function

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Trigger CI

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Consolidate all BaseDL4JTest classes to single class used everywhere; make timeout configurable per class

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Test fixes and timeout increases

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Timeouts and PReLU fixes

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Restore libnd4j build threads arg for CUDA build

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Increase timeouts on a few tests to avoid spurious failures on some CI machines

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* More timeout fixes

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* More test timeout fixes

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Tweak timeout for one more test

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Final tweaks

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* One more ignore

Signed-off-by: AlexDBlack <blacka101@gmail.com>
2020-01-04 13:45:07 +11:00
Alex Black 4b50b920c7
Re-add UI auto-attach functionality with test; also fixes (#80)
Signed-off-by: AlexDBlack <blacka101@gmail.com>
2019-11-25 22:52:02 +11:00
Alex Black 6f514e9431
DL4J UI: Migrate from Play to Vertx (#68)
* Vertx transition initial steps

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Assets serving works

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Next steps

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Add other pages

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Add remaining modules

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Delete deeplearning4j-play module

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* SD UI module

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Main UI works

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Multi-session UI fixed

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* TSNE fixed

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Polish, fixes, re-add CLI launch option

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Fix remote UI functionality

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Final tweaks

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Remove Scala version suffix from deeplearning4j-ui

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Update Arbiter UI to vertx

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Missing version tag

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Remove (now unneeded) scala from arbiter UI

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Move vertx version to parent pom

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Switch template loading from classpath to a more reliable mechanism

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Small fix for I18N message loading

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Switch to shaded jackson for UI module JSON conversion

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* Small test tweaks and copyright header fix

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* pom file copyright headers

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* plugins -> pliginManagement for java 8 modules

Signed-off-by: AlexDBlack <blacka101@gmail.com>

* plugins -> pliginManagement for java 8 modules

Signed-off-by: AlexDBlack <blacka101@gmail.com>
2019-11-22 23:50:34 +11:00