148 lines
5.1 KiB
Markdown
148 lines
5.1 KiB
Markdown
# BRUTEX CAVIS Build Requirements #
|
|
|
|
You can build CAVIS on either x86_64 linux or x86_64 windows
|
|
(sometimes referred to as amd64). Support for native GPU (CUDA) or CPU (AVX2/ AVX512) support can be enabled.
|
|
|
|
Linux on x86_64 is the primary target platform.
|
|
CAVIS is written in Java, C, C++ and Kotlin, therefore you will need a Java JDK and
|
|
C/C++ compiler.
|
|
|
|
# Table of contents
|
|
1. [Introduction](#introduction)
|
|
2.
|
|
2. [Some paragraph](#paragraph1)
|
|
1. [Sub paragraph](#subparagraph1)
|
|
3. [Another paragraph](#paragraph2)
|
|
|
|
## Build Environment Prerequisites <a id="introduction"></a>
|
|
CAVIS uses the [Gradle Build Tool](https://gradle.org) and [CMake](https://cmake.org/) to execute the build.
|
|
A Gradle Wrapper is provided, that will pull the right version of gradle.
|
|
|
|
### Linux x86_64 Prerequisites
|
|
|
|
Linux Kernel 5.10 or higher
|
|
|
|
**Required build tools:**
|
|
|
|
| Tools | Version | Remarks |
|
|
|----------|------------|--------------------------------------------------|
|
|
| Gradle | \> 7.2 | |
|
|
| CMake | \> 2.24.10 | |
|
|
| GNU Make | 3.82 | |
|
|
|
|
| Compiler | Version | Remarks |
|
|
| -------- |----------------|---------------------------------------------------|
|
|
| GCC | 7.3.1 - 10.2.1 | C and C++ compiler |
|
|
| nvcc | \> 11.4 | only in case you want to compile for CUDA support |
|
|
| javac | JDK 11 | openjdk-11 |
|
|
|
|
#### Amazon Linux 2 ####
|
|
Preparation shell script
|
|
|
|
```shell
|
|
sudo amazon-linux-extras install java-openjdk11 && \
|
|
sudo yum install --assumeyes git make gcc gcc-c++ openssl-devel java-11-openjdk-devel && \
|
|
wget https://github.com/Kitware/CMake/releases/download/v3.24.2/cmake-3.24.2.tar.gz && \
|
|
tar -xf cmake-3.24.2.tar.gz && cd cmake-3.24.2/ && \
|
|
./bootstrap && make && sudo make install && cd ..
|
|
```
|
|
|
|
In case you have less than 8GB RAM, a swap file should be created
|
|
```shell
|
|
sudo dd if=/dev/zero of=/swapfile count=8k bs=1M && sudo chmod 600 /swapfile && \
|
|
sudo mkswap /swapfile && sudo swapon /swapfile
|
|
```
|
|
|
|
You can make the swap change permanent (surviving a reboot) by putting it into /etc/fstab file.
|
|
```shell
|
|
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab
|
|
```
|
|
|
|
In order to resolve Brutex Network IP6 endpoints, you need to add an additional nameserver
|
|
```shell
|
|
echo 'nameserver 8.8.8.8' | sudo tee -a /etc/resolv.conf
|
|
```
|
|
## Some paragraph <a id="paragraph1"></a>
|
|
The first paragraph text
|
|
|
|
### Sub paragraph <a id="subparagraph1"></a>
|
|
This is a sub paragraph, formatted in heading 3 style
|
|
|
|
## Troubleshooting <a id="paragraph2"></a>
|
|
### Gradle process just crashes ###
|
|
This is likely an out-of-memory problem, check
|
|
```shell
|
|
dmesg
|
|
```
|
|
for
|
|
```
|
|
oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=java,pid=16477,uid=1000
|
|
[ 6291.694315] Out of memory: Killed process 16477 (java) total-vm:11373580kB, anon-rss:1373168kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:11220kB oom_score_adj:0
|
|
```
|
|
and increase i.e. swapfile size (by adding another one).
|
|
|
|
|
|
|
|
sudo apt-get -q update
|
|
ret=$?
|
|
while [ $ret -ne 0 ];
|
|
do
|
|
echo Waiting for instance to be ready ...
|
|
sleep 20s
|
|
sudo apt-get -q update
|
|
ret=$?
|
|
done
|
|
|
|
echo Instance is ready.
|
|
|
|
#Create swap
|
|
echo Adding swap file
|
|
sudo fallocate -l 16G /swapfile
|
|
sudo chmod 600 /swapfile
|
|
sudo mkswap /swapfile
|
|
sudo swapon /swapfile
|
|
echo Swap created.
|
|
|
|
#echo "deb http://deb.debian.org/debian/ buster main" | sudo tee /etc/apt/sources.list
|
|
#echo "deb http://deb.debian.org/debian/ buster main contrib non-free" | sudo tee -a /etc/apt/sources.list
|
|
|
|
|
|
sudo apt-get -q --yes install build-essential default-jdk git ninja-build gnupg software-properties-common snapd make g++-multilib g++ g++-8 gcc gcc-8 libopenblas-base libopenblas-dev
|
|
ret=$?
|
|
while [ $ret -ne 0 ];
|
|
do
|
|
echo Waiting for instance to be ready ...
|
|
sleep 20s
|
|
sudo apt-get -q --yes install build-essential default-jdk git ninja-build gnupg software-properties-common snapd make g++-multilib g++ g++-8 gcc gcc-8 libopenblas-base libopenblas-dev
|
|
ret=$?
|
|
done
|
|
#sudo snap install core
|
|
#sudo snap install cmake --classic
|
|
|
|
wget https://github.com/Kitware/CMake/releases/download/v3.20.4/cmake-3.20.4-linux-x86_64.sh
|
|
chmod a+x cmake-3.20.4-linux-x86_64.sh
|
|
sudo sh cmake-3.20.4-linux-x86_64.sh --skip-license
|
|
|
|
echo "supersede domain-name-servers 172.31.0.2, 8.8.8.8" | sudo tee -a /etc/dhcp/dhclient.conf
|
|
echo "nameserver 8.8.8.8" | sudo tee -a /etc/resolv.conf
|
|
|
|
# Buildparameter: #
|
|
|
|
-P\<xxx>\
|
|
CAVIS_AVX_EXTENSION = {avx2 | avx512}, default is avx2
|
|
|
|
# Zeppelin Spark dependencies #
|
|
3
|
|
|
|
|
|
To add the dependency to the language models, use the following format in the Dependencies section of the of the Spark Interpreter configuration (Interpreters -> Spark -> Edit -> Dependencies):
|
|
|
|
groupId:artifactId:packaging:classifier:version
|
|
|
|
In your case it should work with
|
|
|
|
edu.stanford.nlp:stanford-corenlp:jar:models:3.8.0
|
|
|
|
|
|
Native cpu code under linux needs libc6-dev
|
|
/lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.29' not found |