* initial commit Signed-off-by: raver119 <raver119@gmail.com> * Implementation of hashcode cuda helper. Working edition. * Fixed parallel test input arangements. * Fixed tests for hashcode op. * Fixed shape calculation for image:crop_and_resize op and test. * NativeOps tests. Initial test suite. * Added tests for indexReduce methods. * Added test on execBroadcast with NDArray as dimensions. * Added test on execBroadcastBool with NDArray as dimensions. * Added tests on execPairwiseTransform and execPairwiseTransofrmBool. * Added tests for execReduce with scalar results. * Added reduce tests for non-empty dims array. * Added tests for reduce3. * Added tests for execScalar. * Added tests for execSummaryStats. * - provide cpu/cuda code for batch_to_space - testing it Signed-off-by: Yurii <yurii@skymind.io> * - remove old test for batch_to_space (had wrong format and numbers were not checked) Signed-off-by: Yurii <yurii@skymind.io> * Fixed complilation errors with test. * Added test for execTransformFloat. * Added test for execTransformSame. * Added test for execTransformBool. * Added test for execTransformStrict. * Added tests for execScalar/execScalarBool with TADs. * Added test for flatten. * - provide cpu/cuda code for space_to_Batch operaion Signed-off-by: Yurii <yurii@skymind.io> * Added test for concat. * comment unnecessary stuff in s_t_b Signed-off-by: Yurii <yurii@skymind.io> * Added test for specialConcat. * Added tests for memcpy/set routines. * Fixed pullRow cuda test. * Added pullRow test. * Added average test. * - correct typo in NDArray::applyPairwiseTransform(nd4j::pairwise::BoolOps op...) Signed-off-by: Yurii <yurii@skymind.io> * - debugging and fixing cuda tests in JavaInteropTests file Signed-off-by: Yurii <yurii@skymind.io> * - correct some tests Signed-off-by: Yurii <yurii@skymind.io> * Added test for shuffle. * Fixed ops declarations. * Restored omp and added shuffle test. * Added convertTypes test. * Added tests for execRandom. Eliminated usage of RandomBuffer with NativeOps. * Added sort tests. * Added tests for execCustomOp. * - further debuging and fixing tests terminated with crash Signed-off-by: Yurii <yurii@skymind.io> * Added tests for calculateOutputShapes. * Addded Benchmarks test. * Commented benchmark tests. * change assertion Signed-off-by: raver119 <raver119@gmail.com> * Added tests for apply_sgd op. Added cpu helper for that op. * Implement cuda helper for aplly_sgd op. Fixed tests for NativeOps. * Added test for assign broadcastable. * Added tests for assign_bp op. * Added tests for axpy op. * - assign/execScalar/execTransformAny signature change - minor test fix Signed-off-by: raver119 <raver119@gmail.com> * Fixed axpy op. * meh Signed-off-by: raver119 <raver119@gmail.com> * - fix tests for nativeOps::concat Signed-off-by: Yurii <yurii@skymind.io> * sequential transform/scalar Signed-off-by: raver119 <raver119@gmail.com> * allow nested parallelism Signed-off-by: raver119 <raver119@gmail.com> * assign_bp leak fix Signed-off-by: raver119 <raver119@gmail.com> * block setRNG fix Signed-off-by: raver119 <raver119@gmail.com> * enable parallelism by default Signed-off-by: raver119 <raver119@gmail.com> * enable nested parallelism by default Signed-off-by: raver119 <raver119@gmail.com> * Added cuda implementation for row_count helper. * Added implementation for tnse gains op helper. * - take into account possible situations when input arrays are empty in reduce_ cuda stuff Signed-off-by: Yurii <yurii@skymind.io> * Implemented tsne/edge_forces op cuda-based helper. Parallelized cpu-based helper for edge_forces. * Added kernel for tsne/symmetrized op heleper. * Implementation of tsne/symmetrized op cuda helper. Working edition. * Eliminated waste printfs. * Added test for broadcastgradientargs op. * host-only fallback for empty reduce float Signed-off-by: raver119 <raver119@gmail.com> * - some tests fixes Signed-off-by: Yurii <yurii@skymind.io> * - correct the rest of reduce_ stuff Signed-off-by: Yurii <yurii@skymind.io> * - further correction of reduce_ stuff Signed-off-by: Yurii <yurii@skymind.io> * Added test for Cbow op. Also added cuda implementation for cbow helpers. * - improve code of stack operation for scalar case Signed-off-by: Yurii <yurii@skymind.io> * - provide cuda kernel for gatherND operation Signed-off-by: Yurii <yurii@skymind.io> * Implementation of cbow helpers with cuda kernels. * minor tests tweaks Signed-off-by: raver119 <raver119@gmail.com> * minor tests tweaks Signed-off-by: raver119 <raver119@gmail.com> * - further correction of cuda stuff Signed-off-by: Yurii <yurii@skymind.io> * Implementatation of cbow op helper with cuda kernels. Working edition. * Skip random testing for cudablas case. * lstmBlockCell context fix Signed-off-by: raver119 <raver119@gmail.com> * Added tests for ELU and ELU_BP ops. * Added tests for eq_scalar, gt_scalar, gte_scalar and lte_scalar ops. * Added tests for neq_scalar. * Added test for noop. * - further work on clipbynorm_bp Signed-off-by: Yurii <yurii@skymind.io> * - get rid of concat op call, use instead direct concat helper call Signed-off-by: Yurii <yurii@skymind.io> * lstmBlockCell context fix Signed-off-by: raver119 <raver119@gmail.com> * Added tests for lrelu and lrelu_bp. * Added tests for selu and selu_bp. * Fixed lrelu derivative helpers. * - some corrections in lstm Signed-off-by: Yurii <yurii@skymind.io> * operator * result shape fix Signed-off-by: raver119 <raver119@gmail.com> * - correct typo in lstmCell Signed-off-by: Yurii <yurii@skymind.io> * few tests fixed Signed-off-by: raver119 <raver119@gmail.com> * CUDA inverse broadcast bool fix Signed-off-by: raver119 <raver119@gmail.com> * disable MMAP test for CUDA Signed-off-by: raver119 <raver119@gmail.com> * BooleanOp syncToDevice Signed-off-by: raver119 <raver119@gmail.com> * meh Signed-off-by: raver119 <raver119@gmail.com> * additional data types for im2col/col2im Signed-off-by: raver119 <raver119@gmail.com> * Added test for firas_sparse op. * one more RandomBuffer test excluded Signed-off-by: raver119 <raver119@gmail.com> * Added tests for flatten op. * Added test for Floor op. * bunch of tests fixed Signed-off-by: raver119 <raver119@gmail.com> * mmulDot tests fixed Signed-off-by: raver119 <raver119@gmail.com> * more tests fixed Signed-off-by: raver119 <raver119@gmail.com> * Implemented floordiv_bp op and tests. * Fixed scalar case with cuda implementation for bds. * - work on cuda kernel for clip_by_norm backprop op is completed Signed-off-by: Yurii <yurii@skymind.io> * Eliminate cbow crach. * more tests fixed Signed-off-by: raver119 <raver119@gmail.com> * more tests fixed Signed-off-by: raver119 <raver119@gmail.com> * Eliminated abortion with batched nlp test. * more tests fixed Signed-off-by: raver119 <raver119@gmail.com> * Fixed shared flag initializing. * disabled bunch of cpu workspaces tests Signed-off-by: raver119 <raver119@gmail.com> * scalar operators fix: missing registerSpecialUse call Signed-off-by: raver119 <raver119@gmail.com> * Fixed logdet for cuda and tests. * - correct clipBynorm_bp Signed-off-by: Yurii <yurii@skymind.io> * Fixed crop_and_resize shape datatype. * - correct some mmul tests Signed-off-by: Yurii <yurii@skymind.io>
272 lines
12 KiB
C++
272 lines
12 KiB
C++
/*******************************************************************************
|
|
* Copyright (c) 2015-2018 Skymind, Inc.
|
|
*
|
|
* This program and the accompanying materials are made available under the
|
|
* terms of the Apache License, Version 2.0 which is available at
|
|
* https://www.apache.org/licenses/LICENSE-2.0.
|
|
*
|
|
* Unless required by applicable law or agreed to in writing, software
|
|
* distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
|
* WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
|
* License for the specific language governing permissions and limitations
|
|
* under the License.
|
|
*
|
|
* SPDX-License-Identifier: Apache-2.0
|
|
******************************************************************************/
|
|
|
|
//
|
|
// @author raver119@gmail.com, created on 19.01.18.
|
|
// @author Yurii Shyrma (iuriish@yahoo.com)
|
|
//
|
|
|
|
#include <ops/declarable/helpers/s_t_b.h>
|
|
|
|
namespace nd4j {
|
|
namespace ops {
|
|
namespace helpers {
|
|
|
|
|
|
//////////////////////////////////////////////////////////////////////////
|
|
template <typename T>
|
|
static void batchToSpace_(const NDArray& input, NDArray& output, const uint cropBottom, const uint cropTop, const uint cropLeft, const uint cropRight) {
|
|
|
|
// input [bS, H * blockSize, W * blockSize, iC]
|
|
// output [bS, H * blockSize - cropBottom - cropTop, W * blockSize - cropLeft - cropRight, iC]
|
|
|
|
// if (cropTop = cropBottom = cropRight = cropLeft = 0) shapes are the same
|
|
// else:
|
|
// oH -> [cropBottom, iH - cropTop]
|
|
// oW -> [cropLeft, iH - cropRight]
|
|
// xLen > zLen
|
|
|
|
const T* x = input.bufferAsT<T>();
|
|
T* z = output.bufferAsT<T>();
|
|
|
|
const int rank = 4;
|
|
|
|
const Nd4jLong* xShapeInfo = input.getShapeInfo();
|
|
const Nd4jLong* zShapeInfo = output.getShapeInfo();
|
|
|
|
const uint bS = xShapeInfo[1];
|
|
const uint iH = xShapeInfo[2];
|
|
const uint iW = xShapeInfo[3];
|
|
const uint iC = xShapeInfo[4];
|
|
|
|
// loop through output array
|
|
PRAGMA_OMP_PARALLEL_FOR_SIMD_ARGS(collapse(4))
|
|
for (uint b = 0; b < bS; ++b) {
|
|
for (uint h = cropBottom; h < iH - cropTop; ++h) {
|
|
for (uint w = cropLeft; w < iW - cropRight; ++w) {
|
|
for (uint c = 0; c < iC; ++c) {
|
|
|
|
const Nd4jLong xOffset = b * xShapeInfo[5] + h * xShapeInfo[6] + w * xShapeInfo[7] + c * xShapeInfo[8];
|
|
|
|
const Nd4jLong zOffset = b * zShapeInfo[5] + (h - cropBottom) * zShapeInfo[6] + (w - cropLeft) * zShapeInfo[7] + c * zShapeInfo[8];
|
|
|
|
z[zOffset] = x[xOffset];
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
|
|
BUILD_SINGLE_TEMPLATE(template void batchToSpace_, (const NDArray& input, NDArray& output, const uint cropBottom, const uint cropTop, const uint cropLeft, const uint cropRight), LIBND4J_TYPES);
|
|
|
|
//////////////////////////////////////////////////////////////////////////
|
|
void batchToSpace(nd4j::LaunchContext* context, const NDArray& input, NDArray& output, const uint cropBottom, const uint cropTop, const uint cropLeft, const uint cropRight, const uint blockSize) {
|
|
|
|
// [bS*blockSize*blockSize, H/blockSize, W/blockSize, iC] is rearranged/permuted to [bS, oH, oW, iC]
|
|
// oH = H - cropTop - cropBottom
|
|
// oW = W - cropLeft - cropRight
|
|
|
|
NDArray inputRearranged0 = input.reshape(input.ordering(), {blockSize, blockSize, output.sizeAt(0), input.sizeAt(1), input.sizeAt(2), input.sizeAt(3)});
|
|
inputRearranged0.permutei({2, 3,0, 4,1, 5});
|
|
|
|
if(input.lengthOf() == output.lengthOf())
|
|
output.assign(inputRearranged0);
|
|
else {
|
|
NDArray inputRearranged1 = inputRearranged0.reshape(input.ordering(), {output.sizeAt(0), input.sizeAt(1) * blockSize, input.sizeAt(2) * blockSize, input.sizeAt(3)});
|
|
BUILD_SINGLE_SELECTOR(input.dataType(), batchToSpace_, (inputRearranged1, output, cropBottom, cropTop, cropLeft, cropRight), LIBND4J_TYPES);
|
|
}
|
|
}
|
|
|
|
|
|
//////////////////////////////////////////////////////////////////////////
|
|
template <typename T>
|
|
static void spaceToBatch_(const NDArray& input, NDArray& output, const uint padBottom, const uint padTop, const uint padLeft, const uint padRight) {
|
|
|
|
// input [bS, H * blockSize - padBottom - padTop, W * blockSize - padLeft - padRight, iC]
|
|
// output [bs, H * blockSize, W * blockSize, iC]
|
|
|
|
// if (padTop = padBottom = padRight = padLeft = 0) shapes are the same
|
|
// else:
|
|
// iH -> [padBottom, oH - padTop]
|
|
// iW -> [padLeft, oW - padRight]
|
|
// zLen > xLen
|
|
|
|
const T* x = input.bufferAsT<T>();
|
|
T* z = output.bufferAsT<T>();
|
|
|
|
const int rank = 4;
|
|
|
|
const Nd4jLong* xShapeInfo = input.getShapeInfo();
|
|
const Nd4jLong* zShapeInfo = output.getShapeInfo();
|
|
|
|
const uint bS = zShapeInfo[1];
|
|
const uint oH = zShapeInfo[2];
|
|
const uint oW = zShapeInfo[3];
|
|
const uint iC = zShapeInfo[4];
|
|
|
|
// loop through output array
|
|
PRAGMA_OMP_PARALLEL_FOR_SIMD_ARGS(collapse(4))
|
|
for (uint b = 0; b < bS; ++b) {
|
|
for (uint h = 0; h < oH; ++h) {
|
|
for (uint w = 0; w < oW; ++w) {
|
|
for (uint c = 0; c < iC; ++c) {
|
|
|
|
const Nd4jLong zOffset = b * zShapeInfo[5] + h * zShapeInfo[6] + w * zShapeInfo[7] + c * zShapeInfo[8];
|
|
|
|
if(h >= padBottom && h < oH - padTop && w >= padLeft && w < oW - padRight) {
|
|
const Nd4jLong xOffset = b * xShapeInfo[5] + (h - padBottom) * xShapeInfo[6] + (w - padLeft) * xShapeInfo[7] + c * xShapeInfo[8];
|
|
z[zOffset] = x[xOffset];
|
|
}
|
|
else
|
|
z[zOffset] = 0.f;
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
|
|
BUILD_SINGLE_TEMPLATE(template void spaceToBatch_, (const NDArray& input, NDArray& output, const uint padBottom, const uint padTop, const uint padLeft, const uint padRight), LIBND4J_TYPES);
|
|
|
|
//////////////////////////////////////////////////////////////////////////
|
|
void spaceToBatch(nd4j::LaunchContext* context, const NDArray& input, NDArray& output, const uint padBottom, const uint padTop, const uint padLeft, const uint padRight, const uint blockSize) {
|
|
|
|
// [bS, iH, iW, iC] is rearranged/permuted to [bS*blockSize*blockSize, (iH + padBottom + padTop)/blockSize, (iW + padLeft + padRight)/blockSize, iC]
|
|
|
|
NDArray outputRearranged0 = output.reshape(output.ordering(), {blockSize, blockSize, input.sizeAt(0), output.sizeAt(1), output.sizeAt(2), input.sizeAt(3)});
|
|
outputRearranged0.permutei({2, 3,0, 4,1, 5});
|
|
|
|
if(input.lengthOf() == output.lengthOf()) {
|
|
outputRearranged0.assign(input);
|
|
}
|
|
else {
|
|
NDArray outputRearranged1 = outputRearranged0.reshape(output.ordering(), {input.sizeAt(0), output.sizeAt(1) * blockSize, output.sizeAt(2) * blockSize, input.sizeAt(3)});
|
|
BUILD_SINGLE_SELECTOR(input.dataType(), spaceToBatch_, (input, outputRearranged1, padBottom, padTop, padLeft, padRight), LIBND4J_TYPES);
|
|
|
|
if(output.getBuffer() != outputRearranged1.getBuffer())
|
|
outputRearranged0.assign(outputRearranged1);
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
/*
|
|
template <int N, bool B2S>
|
|
struct SpaceToBatchHelper {
|
|
template <typename T>
|
|
static void run(T *ptrSpace, const Nd4jLong *space_shape, const Nd4jLong *space_strides, const Nd4jLong *block_shape, const Nd4jLong *pad_start, const Nd4jLong *block_offsets, T *ptrBatch, const Nd4jLong *batch_shape, const Nd4jLong *batch_strides) {
|
|
for (int batch_pos = 0; batch_pos < batch_shape[0]; ++batch_pos) {
|
|
const int space_pos = batch_pos * block_shape[0] + block_offsets[0] - pad_start[0];
|
|
if (space_pos >= 0 && space_pos < space_shape[0]) {
|
|
SpaceToBatchHelper<N - 1, B2S>::run(ptrSpace + space_pos * space_strides[0], space_shape + 1, space_strides + 1, block_shape + 1, pad_start + 1, block_offsets + 1, ptrBatch, batch_shape + 1, batch_strides + 1);
|
|
} else {
|
|
if (!B2S)
|
|
for (int i = 0; i < batch_strides[0]; i++)
|
|
ptrBatch[i] = (T) 0.f;
|
|
}
|
|
|
|
ptrBatch += batch_strides[0];
|
|
}
|
|
}
|
|
};
|
|
|
|
template <bool B2S>
|
|
struct SpaceToBatchHelper<0, B2S> {
|
|
template <typename T>
|
|
static void run(T *ptrSpace, const Nd4jLong *space_shape, const Nd4jLong *space_strides, const Nd4jLong *block_shape, const Nd4jLong *pad_start, const Nd4jLong *block_offsets, T *ptrBatch, const Nd4jLong *batch_shape, const Nd4jLong *batch_strides) {
|
|
int str = batch_strides[-1];
|
|
for (int i = 0; i < str; i++)
|
|
if (B2S)
|
|
ptrSpace[i] = ptrBatch[i];
|
|
else
|
|
ptrBatch[i] = ptrSpace[i];
|
|
}
|
|
};
|
|
|
|
template <typename T, int NUM_BLOCK_DIMS, bool B2S>
|
|
void _execute(nd4j::LaunchContext * context, void *vptrSpace, const Nd4jLong *space_shape, const Nd4jLong *space_strides, const Nd4jLong *block_shape, const Nd4jLong *pad_start, const Nd4jLong *block_offsets, void *vptrBatch, const Nd4jLong *batch_shape, const Nd4jLong *batch_strides) {
|
|
auto ptrSpace = reinterpret_cast<T *>(vptrSpace);
|
|
auto ptrBatch = reinterpret_cast<T *>(vptrBatch);
|
|
SpaceToBatchHelper<NUM_BLOCK_DIMS, B2S>::run(ptrSpace, space_shape, space_strides, block_shape, pad_start, block_offsets, ptrBatch, batch_shape, batch_strides);
|
|
};
|
|
|
|
Nd4jStatus _spaceToBatch(nd4j::LaunchContext * context, int internal_block_dims, NDArray *input, NDArray *output, std::vector<Nd4jLong> &internal_input_shape, std::vector<Nd4jLong> &internal_output_shape, Nd4jLong *block_shape, Nd4jLong *paddings) {
|
|
auto in = input->reshape('c', internal_input_shape);
|
|
auto out = output->reshape('c', internal_output_shape);
|
|
switch (internal_block_dims) {
|
|
case 1:
|
|
_prepare<1, false>(context, &in, &out, block_shape, paddings);
|
|
break;
|
|
case 2:
|
|
_prepare<2, false>(context, &in, &out, block_shape, paddings);
|
|
break;
|
|
case 3:
|
|
_prepare<3, false>(context, &in, &out, block_shape, paddings);
|
|
break;
|
|
case 4:
|
|
_prepare<4, false>(context, &in, &out, block_shape, paddings);
|
|
break;
|
|
default: {
|
|
return Status::THROW("SpaceToBatch: Wrong number of internal_block_dims");
|
|
}
|
|
}
|
|
|
|
return Status::OK();
|
|
}
|
|
|
|
Nd4jStatus _batchToSpace(nd4j::LaunchContext * context, int internal_block_dims, NDArray *input, NDArray *output, std::vector<Nd4jLong> &internal_input_shape, std::vector<Nd4jLong> &internal_output_shape, Nd4jLong *block_shape, Nd4jLong *crops) {
|
|
auto in = input->reshape('c', internal_input_shape);
|
|
auto out = output->reshape('c', internal_output_shape);
|
|
switch (internal_block_dims) {
|
|
case 1:
|
|
_prepare<1, true>(context, &in, &out, block_shape, crops);
|
|
break;
|
|
case 2:
|
|
_prepare<2, true>(context, &in, &out, block_shape, crops);
|
|
break;
|
|
case 3:
|
|
_prepare<3, true>(context, &in, &out, block_shape, crops);
|
|
break;
|
|
case 4:
|
|
_prepare<4, true>(context, &in, &out, block_shape, crops);
|
|
break;
|
|
default: {
|
|
return Status::THROW("BatchToSpace: Wrong number of internal_block_dims");
|
|
}
|
|
}
|
|
|
|
return Status::OK();
|
|
}
|
|
|
|
#define STB_DIM (0, 1),\
|
|
(1, 2),\
|
|
(2, 3),\
|
|
(3, 4)
|
|
|
|
#define STB_BOOL (0, false),\
|
|
(1, true)
|
|
|
|
BUILD_TRIPLE_TEMPLATE(template void _execute, (nd4j::LaunchContext * context, void *ptrSpace, const Nd4jLong *space_shape, const Nd4jLong *space_strides, const Nd4jLong *block_shape, const Nd4jLong *pad_start, const Nd4jLong *block_offsets, void *ptrBatch, const Nd4jLong *batch_shape, const Nd4jLong *batch_strides), LIBND4J_TYPES, STB_DIM, STB_BOOL);
|
|
|
|
#undef STB_BOOL
|
|
#undef STB_DIM
|
|
*/
|
|
|
|
}
|
|
}
|
|
} |