cavis/deeplearning4j/dl4j-integration-tests/readme.md

#DL4J and SameDiff Integration Tests

These tests are designed to check a number of aspects of DL4J and SameDiff:

1. Predictions (i.e., network output)
2. Training (training curves, parameters, gradient calculation)
3. Evaluation (accuracy, etc)
4. Model serialization (saving + loading models)
5. Overfitting sanity checks (make sure we can overfit a single example)
6. Data pipelines
7. Parallel Wrapper
8. Validating conditions that should always hold (frozen layer params don't change, for example)

They are designed for the following purposes:

1. Detecting regressions: i.e., new commit changed or broke previously working functionality
2. Detecting integration issues - i.e., issues that show up only when components are used together (but not in isolation in unit test)
3. Detecting significant differences between CPU and CUDA backends
4. Validating implementation via sanity checks on training - i.e., can we overfit a single example?
5. Checking networks and data pipelines on real-world scale data and nets
6. Operating as fully automated pre-release checks (replacing manual sanity checks)

Main Classes

Explanation of the main classes:

• IntegrationTestBaselineGenerator: Run manually to generate and save "expected results" for comparing in the future. Output goes to dl4j-test-resources, for saving/uploading.
• IntegrationTestRunner: Actually runs the tests, and compares the output/result to those generated by the baseline generator
• TestCase: integration tests extend this
• testcases/*.java: the actual integration test definitions
• IntegrationTestsDL4J: entry point for running the DL4J integration tests
• IntegrationTestsSameDiff: entry point for running the SameDiff integration tests

Types of Test Components

The integration tests are set up to be able to run multiple types of tests on each network configuration.

Networks may be pretrained (from model zoo) or randomly initialized (from specified configuration).

Specifically, test cases can be run with any subset of the following components to be tested, by setting TestCase.XYZ boolean options to true or false:

1. testPredictions: Testing output (predictions) on some specified data vs. saved/known good arrays
2. testGradients: Testing gradients on some specified data vs. saved/known good arrays
3. testPretrain: Test layerwise pretraining parameters and training curves
4. testTrainingCurves: Train, and check score vs. iteration
5. testParamsPostTraining: validate params match post training
6. testEvaluation: test the evaluation performance (post training, if 4 or 5 are true)
7. testParallelInference: validate that single net and parallel inference results match
8. testOverfitting: sanity check - try to overfit a single example

See TestCase.java for more details.

Adding a New Integration Test

The process to add a new test is simple:

1. Add a method that creates and returns a TestCase object (example: testcases/MLPTestCases.getMLPMnist())
2. Add it as a unit test to IntegrationTests class (example: IntegrationTestsDL4J.testMLPMnist())
3. Run IntegrationTestBaselineGenerator with the new test case, to generate and save the "known good" results.
4. Run the new integration test to make sure it passes, on both CPU and CUDA backends
5. Commit the generated test resources from step 3 to dl4j-test-resources repo

Note that IntegrationTestBaselineGenerator assumes you have the dl4j-test-resources cloned parallel to the DL4J mono-repo.