cavis/nd4j/nd4j-remote/nd4j-json-server
Samuel Audet 5e07998e59 Add support for CUDA 10.2 (#89) 2019-11-29 16:31:03 +11:00
..
src SameDiff cleanup and fixes (#12) 2019-10-26 12:38:08 +11:00
README.md [WIP] Remote inference (#96) 2019-08-14 12:11:09 +03:00
pom.xml Add support for CUDA 10.2 (#89) 2019-11-29 16:31:03 +11:00

README.md

SameDiff model serving

This modules provides JSON-based serving of SameDiff models

Example

First of all we'll create server instance. Most probably you'll do it in application that will be running in container

val server = SameDiffJsonModelServer.<String, Sentiment>builder()
                .adapter(new StringToSentimentAdapter())
                .model(mySameDiffModel)
                .port(8080)
                .serializer(new SentimentSerializer())
                .deserializer(new StringDeserializer())
                .build();

server.start();
server.join();

Now, presumably in some other container, we'll set up remote inference client:

val client = JsonRemoteInference.<String, Sentiment>builder()
                .endpointAddress("http://youraddress:8080/v1/serving")
                .serializer(new StringSerializer())
                .deserializer(new SentimentDeserializer())
                .build();

Sentiment result = client.predict(myText);

On top of that, there's async call available, for cases when you need to chain multiple requests to one or multiple remote model servers.

Future<Sentiment> result = client.predictAsync(myText);