Development environment

Linux Distributions supported list:

Coding guide lines

Building (Development Only)

  1. Install prerequisites GNU make, git and clone project
apt-get install git make
git clone git@github.com:Nexenta/edgefs.git
  1. Setup target destination, default target location (controlled by environment variable NEDGE_HOME) is /opt/nedge. Use NEDGE_HOME to set it to any other than /opt/edgefs location. Use default environment
rm -rf /opt/nedge; export NEDGE_HOME=/opt/nedge
cd edgefs
source env.sh
  1. Install development packages (OS is autodetected)
make deploy
  1. Build without ASAN (production)
make NEDGE_NDEBUG=1 world

Note, we believe that quality and clarity of the code and pull requests makes significant difference in contributions. It saves other’s people time, it helps project to achieve needed stability levels and community to progress faster! As such, our default build method selected to always compile with ASAN across all components where possible. So, to enable ASAN, just do not provide NEDGE_NDEBUG=1 in the command line above. Please make sure that all ASAN found issues needs to be addressed prior to sending pull request!

Working with dependencies and submodules.

EdgeFS project is using git submodules to track its dependencies. To speed up the build process, we updating modules with --depth=1 flag. To add a fix to dependency branch, you will need to follow similar to this procedure:

#
# make sure you cloned it into /opt/edgefs
# you can skip this step if you already run make world before
#
cd /opt/edgefs
export NEDGE_HOME=/opt/nedge
. env.sh
make world
#
# delete the reference to git submodule and deps itself
#
rm -rf .git/modules/deps/nfs-ganesha
rm -rf deps/nfs-ganesha/nfs-ganesha
#
# update submodule again
#
git submodule update --recursive --init deps/nfs-ganesha/nfs-ganesha
cd deps/nfs-ganesha/nfs-ganesha
git log

Now you should be able to do normal git operations on submodule and once you satisfied, make sure to update submodule in EdgeFS tree, so that it will point to a new reference:

git status deps/nfs-ganesha
git add deps/nfs-ganesha/nfs-ganesha
git commit

Running a test cluster in single node dev environment

  1. Ensure that environment set correctly and points to the correct target directory
cd edgefs
source env.sh
  1. Initialize simple single data directory configuration
mkdir /data
efscli config node -i eth0

This will enable use of IPv4 addressing for existing interface eth0 used as a primary backend interface. It will also create 4 subdirectories in /data/device{0,1,2,3} to emulate disks. Consider other profiles to enable more sophisticated configurations, e.g. All-Flash, All HDD, Hybrid HDD with Metadata Offload on SSD, or Mounted Directories.

  1. Start Target
edgefs-start.sh target
  1. Verify that HW (or better say emulated in this case) configuration look normal and accept it
efscli system fhtable print
efscli system fhtable set

At this point new dynamically discovered configuration checkpoint will be created at $NEDGE_HOME/var/run/flexhash-checkpoint.json

  1. Initialize cluster
efscli system init

This will create system “root” object, holding Site’s Namespace. Namespace may consist of more then single region.

  1. Create new local namespace (or we also call it “Region” or “Segment”)
efscli cluster create Hawaii
  1. Create logical tenants of cluster namespace “Hawaii”, also buckets if needed
efscli tenant create Hawaii/Cola
efscli bucket create Hawaii/Cola/bk1
efscli tenant create Hawaii/Pepsi
efscli bucket create Hawaii/Pepsi/bk1

Now cluster is setup, services can be now created and attached to CSI provisioner.

Running NFS server on a single node (Development Only)

  1. Ensure that environment set correctly and points to the correct target directory
cd edgefs
source env.sh
  1. Initialize simple single data directory configuration
mkdir /data
efscli config node -i eth0

This will enable use of IPv4 addressing for existing interface eth0 used as a primary backend interface. It will also create 4 subdirectories in /data/device{0,1,2,3} to emulate disks. Consider other profiles to enable more sophisticated configurations, e.g. All-Flash, All HDD, Hybrid HDD with Metadata Offload on SSD, or Mounted Directories.

  1. Start EdgeFS in solo mode
edgefs-start.sh solo
  1. Verify that no other NFS server is running but RPC is running
rpcinfo -p
   program vers proto   port  service
    100000    4   tcp    111  portmapper
    100000    3   tcp    111  portmapper
    100000    2   tcp    111  portmapper
    100000    4   udp    111  portmapper
    100000    3   udp    111  portmapper
    100000    2   udp    111  portmapper
ps -aef | grep rpc
root       429     2  0 05:50 ?        00:00:00 [rpciod]
root       653     1  0 05:50 ?        00:00:00 /sbin/rpcbind -f -w

Make sure RPC processes are running but no other NFS server is running.

  1. Create a bucket for NFS export
efscli cluster create cltest
efscli tenant create cltest/test
efscli bucket create cltest/test/bk1

This will create cluster (cltest), tenant(test) and bucket(bk1) objects.

  1. Create NFS service object
efscli service create nfs nfs01

This will create NFS service object nfs01.

  1. Export the bucket for NFS use
efscli service serve nfs01 cltest/test/bk1

This will export bucket bk1 via service object nfs01.

  1. Start NFS service to export the bucket
edgefs-start.sh nfs nfs01 --hostrpc
showmount -e
Export list for node-name:
/test/bk1 (everyone)

This will start EdgeFS nfs server exporting the bucket bk1. Verify the export using ‘showmount’ command.

Running Distributed SQL (DSQL) server on a single node (Development Only)

  1. Ensure that environment set correctly and points to the correct target directory
cd edgefs
source env.sh
  1. Initialize simple single data directory configuration
mkdir /data
efscli config node -i eth0

This will enable use of IPv4 addressing for existing interface eth0 used as a primary backend interface. It will also create 4 subdirectories in /data/device{0,1,2,3} to emulate disks. Consider other profiles to enable more sophisticated configurations, e.g. All-Flash, All HDD, Hybrid HDD with Metadata Offload on SSD, or Mounted Directories.

  1. Start EdgeFS in solo mode
edgefs-start.sh solo

Make sure RPC processes are running but no other NFS server is running.

  1. Create a bucket for DSQL use
efscli cluster create cltest
efscli tenant create cltest/test
efscli bucket create cltest/test/bk1

This will create cluster (cltest), tenant(test) and bucket(bk1) objects.

  1. Create DSQL service object
efscli service create dsql mydb

This will create DSQL service object mydb.

  1. Configure DSQL cluster
efscli service serve mydb cltest/test/bk1 1,192.168.1.102:9181
efscli service serve mydb cltest/test/bk1 2,192.168.1.104:9183

mydb will use the bucket cltest/test/bk1. It will serve a two node cluster. Node id 1 will be served from 192.168.1.102 at port 9181 and node id 2 will be served from 192.168.1.104 at port 9183. All the nodes in a cluster can serve only one bucket (cltest/test/bk1). There must be at least one node with id 1.

  1. Display DSQL service
efscli service show mydb
X-Service-Name: mydb
X-Service-Type: dsql
X-Description: DSQL Server
X-Status: disabled
[
  1,cltest/test/bk1@192.168.1.102:9181
  2,cltest/test/bk1@192.168.1.104:9183
]
  1. Start DSQL service
grpc-sql -s mydb

or

CCOW_SVCNAME=mydb /opt/edgefs/scripts/edgefs-start.sh sql

This will start DSQL server if it finds that the node has matching IPv4 address in the list not nodes (cluster) in the service mydb. This command will have to be run on respective nodes where DSQL server needs to be started. If more than one instance is configured for a node, then those many instances will be started on that node.

  1. Removing a node from DSQL cluster
efscli service unserve mydb  1,cltest/test/bk1@192.168.1.102:9181

This will remove node 192.168.1.102 from the cluster.

  1. Running a test client against DSQL Server
cd /opt/edgefs/src/dqlite/libdqclient
./test_client 1 "127.0.0.1:9181"

Running unit tests (Development Only)

  1. Compile unit tests
make NEDGE_NDEBUG=1 test
  1. Unit tests binaries isn’t getting installed into the target directory by default, so, make sure you install it first manually or execute in place. For example core libraries tests can be executed like this:
efscli cluster create cltest
efscli tenant create cltest/test
cd src/ccow/test
./get_test -n

Be aware of “-n” flag. It will skip start of Target daemon as a part of the test.