The NCI openstack cloud is a facility based at NCI where you can create 'virtual machines' and connect them to resources at NCI like the /g/data filesystem. A virtual machine is a private linux server can be used to run analysis or create services such as websites backed by NCI's data.
You can use the cloud to:
Run analyses on data stored under /g/data1
Create web services to visualise data stored at NCI
Run programs not suited for the HPC, e.g. cron jobs, long-running processes, GUIs
The CMS team maintains the following VMs on the NCI cloud:
climate-cms.nci.org.au: Web services maintained by the CMS team, including Ramadda and Jenkins
downloader.climate-cms.nci.org.au: Server for downloading external data sets to the /g/data1 area
Getting Access
To get access to the openstack cloud email the CMS helpdesk climate_help@nf.nci.org.au. The cloud credentials are provided as a shell file containing a number of environment variables, in order to use them it must be sourced into your environment.
Creating a new VM instance
There are a couple tools you can use to start up a virtual machine on the cloud. To use them you will need your openstack credentials loaded in your environment:
Vagrant
Vagrant is a high level tool - it can be used both for booting VM instances on the cloud as well as your local machine, as well as automatically configuring the instance for you using Puppet or similar tools.
To use Vagrant you create a configuration file, then run '`vagrant up`' to boot it. The configuration files are written in Ruby, and look like:
require 'vagrant-openstack-plugin'
Vagrant.configure("2") do | config |
config.vm.box = "openstack-dummy"
config.ssh.username = "ec2-user"
config.ssh.private_key_path = "~/.ssh/id_rsa"
config.vm.provision :puppet do |puppet|
puppet.module_path = "modules"
end
config.vm.provider :openstack do |os|
os.endpoint = "#{ENV['OS_AUTH_URL']}/tokens"
os.username = "#{ENV['OS_USERNAME']}"
os.api_key = "#{ENV['OS_PASSWORD']}"
os.tenant = "#{ENV['OS_TENANT_NAME']}"
os.region = "#{ENV['OS_REGION_NAME']}"
os.flavor = /m1.small/
os.image = /centos/
os.keypair_name = "walesnix"
os.floating_ip = "130.56.244.115"
os.security_groups = ["ssh","http"]
end
end
Nova
Nova is a low-level tool for booting VMs. It is configured entirely using the command line like so:
Both tools require the same information to boot an instance - Nova gives more control with its various other functions, while in Vagrant much of the complexity is taken care of for you.
Connecting to the VM
By default an instance only has a private IP address, so it's only visible to other cloud systems. To connect to it you have to first connect to the cloud login node 'cloudlogin.nci.org.au', then you can connect to the VM using the special 'ec2-user' user (using NCI accounts requires extra configuration). If you've forgotten your VM's IP address use 'nova list' to show it.
scott@localhost $ ssh saw562@cloudlogin.nci.org.au
saw562@cloudlogin $ nova list
+--------------------------------------+--------+--------+--------------------+
| ID | Name | Status | Networks |
+--------------------------------------+--------+--------+--------------------+
| 640c9191-b6b2-4e33-82ec-14515f53a003 | testvm | ACTIVE | private=10.0.0.52 |
+--------------------------------------+--------+--------+--------------------+
saw562@cloudlogin $ ssh ec2-user@10.0.0.52
ec2user@testvm.local $
Configuring the VM
At the moment the VM is very bare - it's not connected to the NCI accounts or filesystem, and only the bare essential programs have been installed. This is where configuration management tools like Puppet come in - you describe how you'd like the system to be set up & Puppet goes off and does the configuration.
Puppet uses a 'manifest' to define what to install, which looks like:
node default {
include ssh
class {'apache':
default_vhost => false,
}
package {'subversion':}
package {'python':}
}
You can see an example of a full Puppet configuration in the code for the CoE Data Portal
We're creating a Puppet module to help you out with common VM use cases, some functions are listed below:
General Issues
There are a number of basic items that need to be set up on any VM. These include:
Server security
Automatic updates
Admin emails
To set this up add to your Puppet manifest:
include climatecms
Shell Access
To enable shell access for users add to your manifest:
include climatecms::access::shell
If your VM is on the NCI cloud this will allow you to log on using your NCI account credentials
Web Sites
Setting up websites can be complicated, as there are many ways to do so (e.g. static pages, tomcat apps, CGI). We recommend using the nginx module to manage these, it allows you to easily connect different locations on a web site with different services.
By default all connections except SSH are firewalled off on the VMs, to enable http and https connections add:
include climatecms::access::web
This will by default set up the webserver to show files in /var/www/html.
Mounting /g/data1 projects
In order to mount a /g/data1 directory on a VM you will first need to request access through the helpdesk. With that done you can add (for each project to be mounted):
climatecms::nci::mount {'ua8': }
Backups
It is important to back up data on the cloud servers, as by default everything gets lost when you shut it down.
Misc. Programs
To install any program available as a Centos yum package use:
Table of Contents
You can use the cloud to:
The CMS team maintains the following VMs on the NCI cloud:
Getting Access
To get access to the openstack cloud email the CMS helpdesk climate_help@nf.nci.org.au. The cloud credentials are provided as a shell file containing a number of environment variables, in order to use them it must be sourced into your environment.
Creating a new VM instance
There are a couple tools you can use to start up a virtual machine on the cloud. To use them you will need your openstack credentials loaded in your environment:
Vagrant
Vagrant is a high level tool - it can be used both for booting VM instances on the cloud as well as your local machine, as well as automatically configuring the instance for you using Puppet or similar tools.
To use Vagrant you create a configuration file, then run '`vagrant up`' to boot it. The configuration files are written in Ruby, and look like:
require 'vagrant-openstack-plugin' Vagrant.configure("2") do | config | config.vm.box = "openstack-dummy" config.ssh.username = "ec2-user" config.ssh.private_key_path = "~/.ssh/id_rsa" config.vm.provision :puppet do |puppet| puppet.module_path = "modules" end config.vm.provider :openstack do |os| os.endpoint = "#{ENV['OS_AUTH_URL']}/tokens" os.username = "#{ENV['OS_USERNAME']}" os.api_key = "#{ENV['OS_PASSWORD']}" os.tenant = "#{ENV['OS_TENANT_NAME']}" os.region = "#{ENV['OS_REGION_NAME']}" os.flavor = /m1.small/ os.image = /centos/ os.keypair_name = "walesnix" os.floating_ip = "130.56.244.115" os.security_groups = ["ssh","http"] end endNova
Nova is a low-level tool for booting VMs. It is configured entirely using the command line like so:
Both tools require the same information to boot an instance - Nova gives more control with its various other functions, while in Vagrant much of the complexity is taken care of for you.
Connecting to the VM
By default an instance only has a private IP address, so it's only visible to other cloud systems. To connect to it you have to first connect to the cloud login node 'cloudlogin.nci.org.au', then you can connect to the VM using the special 'ec2-user' user (using NCI accounts requires extra configuration). If you've forgotten your VM's IP address use 'nova list' to show it.
Configuring the VM
At the moment the VM is very bare - it's not connected to the NCI accounts or filesystem, and only the bare essential programs have been installed. This is where configuration management tools like Puppet come in - you describe how you'd like the system to be set up & Puppet goes off and does the configuration.
Puppet uses a 'manifest' to define what to install, which looks like:
node default { include ssh class {'apache': default_vhost => false, } package {'subversion':} package {'python':} }You can see an example of a full Puppet configuration in the code for the CoE Data Portal
We're creating a Puppet module to help you out with common VM use cases, some functions are listed below:
General Issues
There are a number of basic items that need to be set up on any VM. These include:
To set this up add to your Puppet manifest:
Shell Access
To enable shell access for users add to your manifest:
If your VM is on the NCI cloud this will allow you to log on using your NCI account credentials
Web Sites
Setting up websites can be complicated, as there are many ways to do so (e.g. static pages, tomcat apps, CGI). We recommend using the nginx module to manage these, it allows you to easily connect different locations on a web site with different services.
By default all connections except SSH are firewalled off on the VMs, to enable http and https connections add:
This will by default set up the webserver to show files in /var/www/html.
Mounting /g/data1 projects
In order to mount a /g/data1 directory on a VM you will first need to request access through the helpdesk. With that done you can add (for each project to be mounted):
climatecms::nci::mount {'ua8': }Backups
It is important to back up data on the cloud servers, as by default everything gets lost when you shut it down.
Misc. Programs
To install any program available as a Centos yum package use:
package {'python':}