build kubernetes csi development env by kind on Mac

Beautiful Autumn photo by HaoMa

Recently I’ve been investigating Model training acceleration, I found Alluxio, at the same time noted Fluid,

Our Service are all on kubernetes, Fluid is a good architecture strategy

I need a development environment where I can submit code and get involved

But no one used Kind to develop it, kind is a good kubernetes development tool

So follow me and start setting up the development environment by kind on Mac

Fluid currently contains four controllers

• alluxio
• csi-driver + node-driver-registrar(sidecar)
• dataset
• jindo

https://thenewstack.io/the-current-state-of-the-state-for-cloud-native-storage/

csi-driver need to communicate with kubelet, so we run them in node container

the rest of controllers run directly on the Mac

1. create kind kubernetes cluster

$ kind create cluster --config=cluster.yaml
# kind cluster yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: fluid-dev
nodes:
- role: control-plane
  image: kindest/node:v1.16.1
  extraMounts:
  - hostPath: /Users/mahao/go
    containerPath: /home/work/go
$ k get nodes
NAME                      STATUS   ROLES    AGE   VERSION
fluid-dev-control-plane   Ready    master   74s   v1.16.15

“/User/mahao/go” is my GOPATH on Mac

/User/mahao/go
    |__bin
    |__pkg
    |__src
    |   |__ github.com
    |   |__ sigs.k8s.io
    |   |__ k8s.io
    |__local
    |  |__go #(e.g.: go1.16.3.linux-amd64.tar.gz ) 
    |__tmp

local/go for building csi-driver code in kind node container, you can download it from https://golang.org/doc/install, I use go1.16.3.linux-amd64.tar.gz

2. create crds and csi driver to kind cluster

• alluxioruntimes.data.fluid.io
• databackups.data.fluid.io
• dataloads.data.fluid.io
• datasets.data.fluid.io
• jindoruntimes.data.fluid.io
• fuse.csi.fluid.io

$k apply -f crds.yaml
customresourcedefinition.apiextensions.k8s.io/alluxioruntimes.data.fluid.io created
customresourcedefinition.apiextensions.k8s.io/databackups.data.fluid.io created
customresourcedefinition.apiextensions.k8s.io/dataloads.data.fluid.io created
customresourcedefinition.apiextensions.k8s.io/datasets.data.fluid.io created
customresourcedefinition.apiextensions.k8s.io/jindoruntimes.data.fluid.io created
$k apply -f csi.yaml
$k get csidriver
NAME                CREATED AT
fuse.csi.fluid.io   2021-04-24T15:30:38Z

3. run csi-driver and node-driver-registrar code in kind node container

start fluid csi driver

$ docker exec -ti fluid-dev-control-plane /bin/bash
$ cd /home/work/go/src/github.com/allenhaozi/fluid/cmd/csi && sh start.sh
### 
### start.sh
### 
#! /bin/bash
set -x
export TMPDIR=/root/go/tmp
export GO111MODULE=on
export GOMODCACHE=/root/go/pkg/mod
export GOPROXY=https://goproxy.io
export GOPATH=/home/work/go
export GOROOT=/home/work/go/local/go
export GOBIN=/home/work/go/bin
export PATH=$PATH:$GOBIN:$GOROOT/bin
if [ ! -d $TMPDIR ]; then
  mkdir -p $TMPDIR
fi
# delete reg socket if exist
rm -rf /var/lib/kubelet/plugins_registry/fuse.csi.fluid.io-reg.sock
go run main.go \
 --kubelet-registration-path="/var/lib/kubelet/csi-plugins/fuse.csi.fluid.io/csi.sock" \
 --csi-address="/var/lib/kubelet/csi-plugins/fuse.csi.fluid.io/csi.sock" \
    --reg-path="/var/lib/kubelet/plugins_registry" \
    --v=5

start node-driver-registrar

$ docker exec -ti fluid-dev-control-plane /bin/bash
$ cd /home/work/go/src/github.com/allenhaozi/node-driver-registrar/cmd/csi-node-driver-registrar && sh start.sh
###
### start.sh
###
#! /bin/bash
set -x
export TMPDIR=/root/go/tmp
export GO111MODULE=on
export GOMODCACHE=/root/go/pkg/mod
export GOPROXY=https://goproxy.io
export GOPATH=/home/work/go
export GOROOT=/home/work/go/local/go
export GOBIN=/home/work/go/bin
export PATH=$PATH:$GOBIN:$GOROOT/bin
if [ ! -d $TMPDIR ]; then
  mkdir -p $TMPDIR
fi
rm -f /var/lib/kubelet/csi-plugins/fuse.csi.fluid.io/csi.sock
mkdir -p /var/lib/kubelet/csi-plugins/fuse.csi.fluid.io
cp /home/work/go/src/github.com/allenhaozi/fluid/csi/shell/check_mount.sh /usr/local/bin/check_mount.sh && chmod +x /usr/local/bin/check_mount.sh
go run main.go start \
 --endpoint="unix://var/lib/kubelet/csi-plugins/fuse.csi.fluid.io/csi.sock" \
    --nodeid="fluid-dev-control-plane" \
 --kubeconfig=/etc/kubernetes/kubelet.conf \
 --v=5

I made a small change to node-driver-registrar based on V1.3.0 , you can checkout it from https://github.com/allenhaozi/node-driver-registrar/tree/feat/fluid-dev-v1.3.0

The only change is to allow passing `reg-path` in parameters

4. start alluxio, dataset on the Mac

start alluxio

$ cd ~/go/src/github.com/allenhaozi/fluid/cmd/alluxio && sh start.sh
# start.sh
export ALLUXIO_INIT_IMAGE_ENV=registry.cn-hangzhou.aliyuncs.com/fluid/init-users:v0.4.0-a8ba7c9
export MOUNT_ROOT=/alluxio-mnt
go run main.go start

start dataset

$ cd ~/go/src/github.com/allenhaozi/fluid/cmd/dataset && sh start.sh
# start.sh
export ALLUXIO_INIT_IMAGE_ENV=registry.cn-hangzhou.aliyuncs.com/fluid/init-users:v0.4.0-a8ba7c9
go run main.go start \
 --metrics-addr=127.0.0.1:8082

load image to kind cluster

$ kind load docker-image registry.cn-hangzhou.aliyuncs.com/fluid/init-users:v0.4.0-a8ba7c9 --name=fluid-dev
$ kind load docker-image registry.cn-huhehaote.aliyuncs.com/alluxio/alluxio:2.3.0-SNAPSHOT-2c41226 --name=fluid-dev
$ kind load docker-image registry.cn-huhehaote.aliyuncs.com/alluxio/alluxio-fuse:2.3.0-SNAPSHOT-2c41226 --name=fluid-dev                                 
$ kind load docker-image nginx:latest --name=fluid-dev

4. run fluid demo

$ k apply -f dataset.yaml
$ k apply -f runtime.yaml
$ k apply -f pod.yaml
# check pod list
NAME                READY   STATUS    RESTARTS   AGE
demo-app            1/1     Running   0          31m
demo-fuse-ksdsr     1/1     Running   0          31m
demo-master-0       2/2     Running   0          36m
demo-worker-k2xhh   2/2     Running   0          31m
# check data
$ k exec -ti demo-app sh
$ ls /data/spark/
SparkR_3.0.2.tar.gz   spark-3.0.2-bin-hadoop2.7-hive1.2.tgz  spark-3.0.2-bin-hadoop3.2.tgz  spark-3.0.2.tgz
pyspark-3.0.2.tar.gz  spark-3.0.2-bin-hadoop2.7.tgz      spark-3.0.2-bin-without-hadoop.tgz

finish it,

If you have an easier method based on Kind, please leave a comment