So where do I come in? Well thats an easy question to answer with a long explanation. Gerir wrote Theia as an internal tool for gathering statistical data from our production NetApps. Being an architecty kinda guy we keep him very busy, so unfortunately he doesn’t have as much time to work on Theia as we would like. This is where I come in.. After having some conversations with Gerir about Theia, we decided that she needed some attention and after a few revisions it would be time for Theia to be introduced to the Open Source community through http://github.com/ning

Getting ready for open source!
The main feature that was holding us back from open sourcing Theia was the lack of interface autodiscovery. This was a crucial part to open sourcing Theia because without this you would have to create a copy of Theia for every netapp you were planning to monitor. As part of the metrics collection we parse network interface statistics. Previously this required you to hard code the name of the NetApp interface into theia. From an Operations perspective this would be a nightmare to manage if you have more than one NetApp, or if the interfaces ever changed. In addition to the management nightmares python dictionary’s only support unique Key’s which Theia uses for text formatting which would prevent Theia from ever being able to display statistical data for multiple network interfaces.

I overcame these obstacles by utilizing modules that were already being imported as to limit overhead, and to keep things tidy. So first things first.. we need to “discover” the active interfaces on the NetApp. There are a few built in tools which are exposed to provide this functionality but they dont appear to be written with automation in mind.

By using the python subprocess library we can login to the netapp and collect some data as shown below:

cmd = "/usr/bin/ssh -o BatchMode=yes -2 -ax -i /root/.ssh/perfstat_dsa %s 'stats show ifnet'|sed 's/:/ /g'|sort|awk '{print $2}'|uniq|tr -d '\r'" % (hostname)

If we break down this command we can see that we are establishing an SSH connection to the NetApp and running ‘stats show ifnet’ which is a utility exposed by ONTAP 7.3. After we have the interface details we can then perform some text formatting so that we can parse the network interfaces and load them into a python list.

NetApp> stats show ifnet
ifnet:svif0:recv_packets:22510/s
ifnet:svif0:recv_errors:0/s
ifnet:svif0:send_packets:41798/s
ifnet:svif0:send_errors:0/s
ifnet:svif0:collisions:0/s
ifnet:svif0:recv_data:1969198b/s
ifnet:svif0:send_data:61504464b/s
ifnet:svif0:recv_mcasts:0/s
ifnet:svif0:send_mcasts:0/s
ifnet:svif0:recv_drop_packets:0/s
NetApp*>

Code:

    cmd = "/usr/bin/ssh -o BatchMode=yes -2 -ax -i /root/.ssh/perfstat_dsa %s 'stats show ifnet'|sed 's/:/ /g'|sort|awk '{print $2}'|uniq|tr -d '\r'" % (hostname)
    p = subprocess.Popen(cmd,shell=True,stdout=subprocess.PIPE,stderr=subprocess.PIPE)
    so = p.communicate()[0]

Note that the subprocess library returns both stdout and stderr so we must tell python to only return stdout by specifying [0] at the end our p.communicate() command.

Now that we have the names of the interfaces we can collect data for each individual interface by iterating over our python list. This is done by using a basic for loop as shown below.

    if_list = []
    if_list.append(so.rstrip('\n'))
 
    for interface in if_list:
            vifs = []
            vifs = interface.splitlines()

As you can see in the for loop listed above we are calling the buit in python library splitlines() as well as rstrip(). In order to properly format our list we must first remove the newline characters that are inserted by the stats show ifnet command, and then sort them into the “vifs” list. Here is the before and after using print statements to compare.

if_list.append(so.rstrip('\n'))

['e0b\ne0d\ne0e\ne0f\nvif0']


for interface in if_list:
vifs = []
vifs = interface.splitlines()


['e0b', 'e0d', 'e0e', 'e0f', 'vif0']

Up to this point we have been doing prep work leading in to the working portion of this code. In order for Theia to iterate over our active interfaces and collect the statistical data that we are looking for we need to present these values to theia in the form of a python dictionary. Dictionary’s are pretty simple and consist of a Key and a Value. The Key specifies the name of our Value. For example if you had a Key called “Food”, Pizza could be your value and would be represented as:

mealtime = { 
      'Food'    : "pizza"
}
print mealtime

Theia uses a dictionary called xnmetrics that will be used to collect the specific data from the NetApp. Since the python dictionary requires you to have unique Keys, we must also specify the interface name as part of the Key. We do this by writing another simple for loop, but we will append the data in our list into the xnmetrics dictionary.

    for vif in vifs:
            xnmetrics["NetRE %s" % (vif)] = "ifnet:%s:recv_errors" % (vif)
            xnmetrics["NetSE %s" % (vif)] = "ifnet:%s:send_errors" % (vif)

Here we are doing a few things. First and most obvious is our for loop but whats important is the action we are performing. While iterating through our list of interfaces we are appending the Key and Value for each interface as the vif into our xnmetrics dictionary. By appending the interface name into the key we can ensure that we always have our required unique Key name and as an afterthought we can also easily identify the name of the interface.

I hope you find this article useful, and as with all open source projects we are looking for contributors who would like to use and expand upon Theia. So if you are interested head over to the Ning github @ https://github.com/ning/Theia to get started!

autodiscover.py – code integrated into Theia for interface autodiscovery

 
#!/usr/bin/env python
import subprocess
import sys
 
# xnmetrics dictionary (Original Theia code)
xnmetrics = { 
      'APB'    : "system:system:avg_processor_busy",
      'CIFSO'  : None,
      'CPU' : "system:system:cpu_busy",
      'CPtime' : "system:system:cp_time",
      'CacheAge' : "system:system:cache_age",
      'CacheHit' : "system:system:cache_hit",
      'DiskRead' : "system:system:disk_data_read",
      'DiskUtil' : "system:system:disk_busy",
      'DiskWrite' : "system:system:disk_data_written",
      'HTTPO' : None,
      'NFSO' : "nfsv3:nfs:nfsv3_ops",
      'NFSRL' : "nfsv3:nfs:nfsv3_read_latency",
      'NFSRO' : "nfsv3:nfs:nfsv3_read_ops",
      'NFSWL' : "nfsv3:nfs:nfsv3_write_latency",
      'NFSWO' : "nfsv3:nfs:nfsv3_write_ops",
      'NetIn' : "system:system:net_data_recv",
      'NetOut' : "system:system:net_data_sent",
      'TPB' : "system:system:total_processor_busy",
      'TapeRead' : None,
      'TapeWrite' : None,
      'TotalO' : "system:system:total_ops",
      'cpu0' : "processor:processor0:processor_busy",
      'cpu1' : "processor:processor1:processor_busy",
      'cpu2' : "processor:processor2:processor_busy",
      'cpu3' : "processor:processor3:processor_busy"
}
 
# interface autodiscovery code (Open Source Initiative)
def vifautodiscover():
    hostname = sys.argv[1]
    cmd = "/usr/bin/ssh -o BatchMode=yes -2 -ax -i /root/.ssh/perfstat_dsa %s 'stats show ifnet'|sed 's/:/ /g'|sort|awk '{print $2}'|uniq|tr -d '\r'" % (hostname)
    p = subprocess.Popen(cmd,shell=True,stdout=subprocess.PIPE,stderr=subprocess.PIPE)
    so = p.communicate()[0]
 
    if_list = []
    if_list.append(so.rstrip('\n'))
 
    for interface in if_list:
            vifs = []
            vifs = interface.splitlines()
 
    for vif in vifs:
            xnmetrics["NetRE %s" % (vif)] = "ifnet:%s:recv_errors" % (vif)
            xnmetrics["NetSE %s" % (vif)] = "ifnet:%s:send_errors" % (vif)
 
    print xnmetrics
 
vifautodiscover()

With that in mind how are you going to host your website on IPv6 address space? There are many web servers available as free Open Source software such as nginx, apache (httpd), and lighthttpd. For the context of this article we will focus on apache, and assume that you already have IPv6 connectivity and an understanding of virtual hosts.

Getting Started

Here are a few things that you are going to need to get started:

• Linux based server
• Familiarity with Apache and the Linux command line (bash)
• IPv6 address
• Text editor

As the transition to IPv6 progresses you should see your hosting provider assigning IPv6 addresses. At this time many hosting companies have not yet caught up to this as implementing IPv6 networking and connectivity can be extremely time consuming and expensive. Currently Verio, Linode, and Liquid Web all provide web hosting with IPv6 connectivity.

Networking

Once you have confirmed that the network appliance your server is connected to has IPv6 capabilities, and your provider has assigned you an IPv6 address you can get started configuring your network interfaces. There are many ways this can be done but we will only cover binding an IPv6 address to an interface already serving traffic over IPv4.

First things first.. enable IPv6 networking by adding the following line to /etc/sysconfig/network on a redhat based Linux distribution.

NETWORKING_IPV6=yes

This is one of the files your init scripts consults with to identify what networking services to start. Now we can move on to configuring the interface. Mentioned previously we will be configuring a single interface to serve both IPv4 and IPv6 traffic. In this example we will be binding these addresses to eth0.

Here are the parameters we must edit:

• IPV6INIT – Accepts YES/NO and is used to determine if we should bring up IPv6
• IPV6ADDR – The IPv6 Address provided to you by your hosting company
• IPV6_DEFAULTGW – The default gateway which should also be provided by your hosting company

Example:

IPV6INIT=yes
IPV6ADDR=2600:3c01::f03c:91ff:fe93:eb17
IPV6_DEFAULTGW=fe80::1

At this point you should be ready to restart networking and test connectivity. This will differ slightly from some of the tools we use for testing IPv4. For example if you wanted to run a simple ping command you would use ping6 or a third party site such as SubnetOnline.

/etc/init.d/networking restart

PING 2600:3c01::f03c:91ff:fe93:eb17(2600:3c01::f03c:91ff:fe93:eb17) 32 data bytes
40 bytes from 2600:3c01::f03c:91ff:fe93:eb17: icmp_seq=0 ttl=245 time=176 ms
40 bytes from 2600:3c01::f03c:91ff:fe93:eb17: icmp_seq=1 ttl=245 time=144 ms
40 bytes from 2600:3c01::f03c:91ff:fe93:eb17: icmp_seq=2 ttl=245 time=144 ms
40 bytes from 2600:3c01::f03c:91ff:fe93:eb17: icmp_seq=3 ttl=245 time=144 ms
 
--- 2600:3c01::f03c:91ff:fe93:eb17 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3010ms
rtt min/avg/max/mdev = 144.641/152.757/176.961/13.982 ms, pipe 2

If a ping from your workstation does not work do not be alarmed.. your local network may not have IPv6 connectivity. If you should encounter any issues up to this point you should consult with your hosting provider to ensure your interfaces are configured properly and you have the correct configuration.

Virtual Hosts

Now that we have networking configured we can move on to configuring our virtual hosts. Virtual hosts are pretty easy to configure, however there are multiple steps. First and foremost you must configure apache to listen on your IPv6 interface. This is pretty easy to do however there is an important syntax change that you need to be aware of..

When configuring apache to listen on an IPv4 interface using port 80 you would typically use the following statement in httpd.conf:

Listen 127.0.0.1:80

This changes very slightly but the IPv6 address requires brackets around it like in this example:

Listen [2600:3c01::f03c:91ff:fe93:eb17]:80

Below is an example virtual host configuration. This is pretty standard and does not provide any cache control or any additional features. Here is a break down on the configuration:

• ServerName – this will be the name of your virtual host or domain name
• ServerAlias – aliases work similar to ServerName, however they are used to redirect a URL to the content specified in DocumentRoot
• DocumentRoot – this is where you store your website content
• ErrorLog – pretty self explanatory however any errors that may be generated will be logged here. (404, 500 etc..)
• CustomLog – CustomLog provides you with an ‘all in one’ solution for logging HTTP requests. It specifies the path to the log file as well as the log format. For more details on how to customize this please refer to the apache docs here.
• Options -Indexes – this disables directory listings on folder’s that do not have an index page.
• AllowOverride – this directive is what tells apache to ignore or utilize the contents of a .htaccess file
• Order – Order provides a very basic means of access control.
• Allow – apache docs describe this best.. “Controls which hosts can access an area of the server”
• Typically most people append their virtual host configuration to the httpd.conf file, however it can be stored elsewhere as httpd.conf provides an include directory. For example you could do something like..

  mkdir /etc/httpd/vhost.d
  echo "Include vhost.d/*.conf" >> /etc/httpd/conf/httpd.conf

  This would provide you with the ability to create a file such as “/etc/httpd/vhost.d/mydomain.conf” where you would store your virtual host configuration. Doing this keeps httpd.conf clean, and also makes things a bit easier to manage.

      ServerName  www.yoursite.com
      ServerAlias www.yoursite.com yoursite.com
      UseCanonicalName Off
      DocumentRoot    /home/yoursite/public_html
      ErrorLog    /var/log/httpd/yoursite-error_log
      CustomLog   /var/log/httpd/yoursite-access_log combined
   
          Options -Indexes FollowSymLinks
          AllowOverride All
          Order allow,deny
          Allow from all

  Now that your virtual hosts are configured its time to restart apache and start hosting your website. I hope whoever reads this finds it helpful. If you have any questions please feel free to comment below.
  
  
  

Over the next few day’s I will be updating my servers to have IPv6 compatibility and networking which will give me the ability to host this blog in IPv4 as well as IPv6 address space. Following these updates I will document some of the changes required to start using IPv6 as well as how to configure apache virtual hosts to serve traffic over an IPv6 interface.

 
[root@www ~]# dig AAAA h1tman.com
 
; <<>> DiG 9.7.0-P2-RedHat-9.7.0-5.P2.el6_0.1 <<>> AAAA h1tman.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 3229
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 2, ADDITIONAL: 0
 
;; QUESTION SECTION:
;h1tman.com.			IN	AAAA
 
;; ANSWER SECTION:
h1tman.com.		3600	IN	AAAA	2600:3c01::f03c:91ff:fe93:eb17
 
;; AUTHORITY SECTION:
h1tman.com.		3600	IN	NS	ns23.domaincontrol.com.
h1tman.com.		3600	IN	NS	ns24.domaincontrol.com.
 
;; Query time: 235 msec
;; SERVER: 74.207.241.5#53(74.207.241.5)
;; WHEN: Fri Sep  2 09:53:03 2011
;; MSG SIZE  rcvd: 108
 
[root@www ~]#

So why the change of pace? Well, thats a complicated question to ask but here are some highlights.. Ning employ’s some of the most technical, and well known people in the industry. With that in mind (and a new environment) this provides an excellent opportunity to learn new technologies, expand upon experience, learn new way’s of implementing, deploying, and maintaining software and infrastructure as well as meeting new people.

By now you may have asked yourself “what does ‘ning’ do?”. Well, basically we provide a social networking platform that empowers the non-technical bloggers, businesses, and people who enjoy social networking an easy platform to design and maintain their own social network without the overhead of an IT department and developers. Bored and got time to surf the net? Visit our site: http://www.ning.com/

1. Familiar with linux text editors
2. Have a basic understanding of load balancing
3. Familiarity with Apache, MySQL, and rsync
4. The ability to create at least 4 virtual machines

What does this set-up provide?

This set-up will provide you with some very basic failover at the web servers by using a load balancer. Keep in mind I will not be discussing HA at the load balancer level so you will still have a single point of failure with this guide, as it is only intended to provide some fundamental knowledge on how to set-up a replicated and load balanced pair of web servers.

Why is this useful?

If your website needs some redundancy as well as the need to be able to handle large volumes of traffic then this will give you a good starting point. This solution will not fit every website’s needs so you must design to fit your application, performance requirements, disaster recovery plan, redundancy and scalability. This guide will however provide you with the basics to configure a small web server farm that will handle most ‘run of the mill’ websites.

Getting Started..

First and foremost we need four virtual machines running CentOS 5.5. Just about any hypervisor will work. If you do not have one readily available you can download VMWare Server, or Virtual box for free. Once your hypervisor is installed you need to proceed with creating your virtual machines. These will not require much disk space for this “how to”. Creating these virtual machines with 256MB RAM and 10GB of disk space should be sufficient for a testing environment (not for a production website). When creating these virtual machines keep in mind to set their host names respective to their roles for easy identification. For example:

Web Server 1 – WWW1
Web Server 2 – WWW2
Load Balancer – LB1
Database Server – DB1

Feel free to set these to whatever you would like as long as you can remember their roles. For this example we will use the host names listed above. Once you have all virtual machines installed go ahead and ensure they have network connectivity and that they are up to date. To verify this run the following commands:

yum clean all
yum -y update
yum -y upgrade

Now that we have our environment ready to go we can get started with configuring the virtual machines.

LB1 – HAProxy Load Balancer

Before we get started with the installation of the load balancer we need to ensure that we have the correct YUM repositories installed. With that said the only additional repository that we will need is EPEL. Here is a link for the EPEL repository that will outline all the details for the repository: http://fedoraproject.org/wiki/EPEL

To install the EPEL repository simply run the following:

rpm -Uvh http://download.fedoraproject.org/pub/epel/5/i386/epel-release-5-4.noarch.rpm

Now that our repository is ready go ahead and install HAProxy:

yum -y install haproxy

Now comes the configuration. Fortunately HAProxy is fairly simple to configure however we need some basic information before we can proceed.

1. eth0 IP of the load balancer
2. eth0 IP of both web servers

Once you have this information you can go ahead and edit the configuration file located at /etc/haproxy/haproxy.cfg

Example:

global
 
    #logging is designed to work with syslog facility's due to chrooted environment
    #log loghost    local0 info - By default this is commented out
 
    #chroot directory
    chroot /usr/share/haproxy
 
    #user/group id
    uid 99
    gid 99
 
    #running mode
    daemon
defaults
 
    #HTTP Log format
    mode http
 
    #number of connection retries for the session
    retries 3
 
    #try another webhead if retry fails
    option redispatch
 
    #session settings - max connections, and session timeout values
    maxconn 10000
    contimeout 10000
    clitimeout 50000
    srvtimeout 50000
 
listen webfarm 127.0.0.5:80
 
    #HTTP Log format
    mode http
    #stats uri /haproxy
    stats uri /haproxy
 
    #balance roundrobin - Typical Round Robin
    #balance leastconn - Least Connections
    #balance static-rr - Static Round Robin - Same as round robin, but weights have no effect
    balance roundrobin
 
    #cookie <COOKIENAME> prefix - Used for cookie-based persistence
    cookie webpool insert
 
    #option httpclose - http connection closing
    option  httpclose
 
    #option forwardfor - best stated as "Enable insertion of the X-Forwarded-For header to requests sent to the web heads" aka send EU IP
    option forwardfor
 
    #Web Servers (Examples)
    server WWW1 127.0.0.1:80 check port 80 # Active in rotation   
    server WWW2 127.0.0.2:80 check port 80 # Active in rotation

HAProxy is now configured. As you can see this was a pretty simple process, however there are a few things left to do..

1. Ensure that HAProxy starts on reboot

chkconfig haproxy on

2. Configure iptables to allow HTTP access to port 80

iptables -I INPUT -m tcp -p tcp --dport 80 -j ACCEPT

3. Start the service

/etc/init.d/haproxy start

At this point you should be able to open a browser and view the HAProxy status page. If your eth0 IP address for your load balancer was 127.0.0.5 you could browse to http://127.0.0.5/haproxy to view the current status.

WWW1/2 – Apache Web Servers

This part isnt much more difficult than setting up your load balancer however the apache configuration file syntax is a little more cryptic. So first things first.. Install apache and configure iptables.

yum -y install httpd

chkconfig httpd on

This will provide you with a basic installation of apache that will work for the purpose of this example. At the bottom of this portion I will provide an example of a more advanced configuration file for use with virtual hosts. As with our load balancer we will need to configure iptables to allow HTTP traffic access to port 80.

iptables -I INPUT -m tcp -p tcp --dport 80 -j ACCEPT

By default without any configuration changes apache sets your document root to /var/www/html. This is where you will store your website content. This can be changed to fit your needs when creating virtual hosts, or for security concerns. To create an example web page simply run the following command. This will allow us to ensure the load balancer is working later on.

On WWW1:

echo "WWW1" > /var/www/html/index.html

On WWW2:

echo "WWW2" > /var/www/html/index.html

Now that you have installed a fake temporary website we should be able to start apache and test our load balancer. Starting apache is very similar to starting HAProxy. Simply run:

/etc/init.d/httpd start

If you type http://127.0.0.5/ into your browser you should be greeted by one of your web servers with a response of “WWW1″ or “WWW2″ on the screen. If you press F5 on your keyboard (command + R for mac) to refresh the page multiple times you should see the text change back and forth. This is because we configured the load balancer with ’round robin’ which sends one request to WWW1 the second to WWW2 the third to WWW1 and so on. At this point once you verify your load balancer worked as expected you can configure a basic cron job using rsync to replicate your data.

In this example we are only using basic one way replication provided by rsync. In order to do this we need to setup a key pair for our web servers. Simple way to do this is as follows:

[root@WWW1 ~]$ ssh-keygen -t rsa -b 2048

Once you have saved the key you need to copy the public key over to WWW2 in /root/.ssh/authorized_keys. This file may not exist so you will need to create it. After setting our public key on WWW2 verify you can login as the root user from WWW1 to WWW2. If this is successful add the following to /etc/crontab:

05 * * * * rsync -aPSv -e 'ssh' /var/www/html 127.0.0.2:/var/www/html

Make sure you edit the IP addresses listed in this article to be the same as those of your virtual machines. The ones used here will not work and are only for examples. At this point we have set up and configured our load balancer as well as our web servers and replication. To test replication simply login to WWW1 and run

touch /var/www/html/text.txt

Give it a few minutes and you should see this file show up on WWW2. If all is well then move on to the next section to configure your database server.

Example Virtual Host:

<VirtualHost 127.0.0.2:80>
	ServerName	example.com
	ServerAlias	example.com
	UseCanonicalName Off
	DocumentRoot	/home/user/public_html/example.com
	ErrorLog	/var/log/httpd/example.com-error_log
	CustomLog	/var/log/httpd/example.com-access_log combined
	<Directory /home/user/public_html/example.com>
		Options Indexes FollowSymLinks
		IndexOptions FancyIndexing
		AllowOverride All
		Order allow,deny
		Allow from all
	</Directory>
		php_admin_value open_basedir "/home/user/public_html/example.com:/tmp"
		php_admin_flag safe_mode On
</VirtualHost>

DB1 – MySQL Database Server

This section is going to be pretty brief as the exact configuration for MySQL is dependant on your application. With that in mind here we go..

First and foremost install the service:

yum -y install mysql mysql-server

chkconfig mysqld on

Once installed you can proceed with any configuration changes in /etc/my.cnf or simply start the service as is by running:

/etc/init.d/mysqld start

As with apache and HAProxy we must configure the firewall to be able to communicate with our web servers, however this time not on port 80.

iptables -I INPUT -m tcp -p tcp --dport 3306 -j ACCEPT

And thats it! If you are planning on using a database driven application such as wordpress you will want to set your wp-config.php to point to the eth0 IP of your DB1 server during the configuration portion of the wordpress setup.

I hope this helps anyone looking for a similar configuration, as well as provide some ideas on how to set up something like this. If you have questions please feel free to ask by posting here!

Simple enough, however I feel that if memcached cant handle the current load then there is probably a reason and it should be revised.

Initial Configuration:
Database Server
- MySQL 5.0.77
- Memcached 1.4.5

Memcached(defaults):
64MB Cache
1024 Max Connections
Listening on private IP

As you can see here this is not optimal for a fairly high traffic website. Defaults are in most cases not the best for use in production unless your application can perform the way its expected. Additionally there is no fault tolerance in the event that memcached should fail, and since the service was running on a production database server it was robbing a minimal amount of memory from MySQL.

While troubleshooting the issue I found that my resource usage was relatively low on my web servers which provided me with some room to disable memcached on my database server, and deploy local instances on the web servers themselves. This not only frees up network / memory usage on my database server, but it also allows me to re-configure the services to allow for a larger cache per server. Another performance impacting improvement by moving the instances locally to the web servers is reduction in latency. Theoretically since the cache is now stored on the web servers, they should be able to access the data much faster as well as not have to worry about the network.

New Configuration:
Web Servers:
- Apache 2.2.3
- Memcached 1.4.5

Memcached(defaults):
128MB Cache
1024 Max Connections
Listening on 127.0.0.1

Now each of the web servers has a local instance of memcached running and the overall response time seems much quicker. However even with this configuration there is no fault tolerance for memcached but the likeliness of it becoming overloaded within this configuration is much lower. Watchdog or some other type of monitoring software such as Nagios could also be a viable option for checking and automatically responding should memcached fail. Unfortunately I do not know of any open source projects at this time which allow for this automation, though I do not think it would be difficult to write a script for this.

Memcached Configuration:

PORT=11211
USER=memcached
MAXCONN=1024
CACHESIZE=128

If you are considering using memcached just keep in mind that it currently has no form of authentication. This can be particularly troublesome if you do not take the proper security measures to prevent unauthorized access. This can be easily done by blocking or restricting access to the port.

iptables example:

# define your chain
:MEMCACHE - [0:0]
 
# specify protocol and port for the chain
-A INPUT -p tcp -m tcp              --dport 11211        -j MEMCACHE
 
# apply rules to the chain
-A MEMCACHE -s 10.176.12.18/32 -j ACCEPT
-A MEMCACHE -s 10.176.12.17/32 -j ACCEPT
-A MEMCACHE -j DROP

As shown in the iptables example above, the final rule for the MEMCACHE chain is to drop all traffic. This allows you to block all access to the specified port except for those that you explicitly allow.

Red Hat has announced general availability of Red Hat Enterprise
Linux 6. Effective with the release of version 6, we will
no longer offer the Red Hat Certified Technician (RHCT) exam.
We are launching a new certification, Red Hat Certified System
Administrator (RHCSA) in its place. You can read about this
new program at

https://www.redhat.com/certification/rhcsa/

You will continue to retain your RHCT and it will still be
possible to verify your certification at

https://www.redhat.com/training/certification/verify

In addition, because you earned RHCT on Red Hat Enterprise
Linux 5, we will issue a Red Hat Enterprise Linux 5 RHCSA to
you retroactively by January 1, 2011. While there are some
differences between RHCT and RHCSA, we believe that the common
areas between them justify issuing this certification to you
and hope that it proves useful to you and your career.

Please direct questions or comments to the certification team at

https://www.redhat.com/training/certification/comments.html

Best wishes!
Red Hat Certification

All in all the summit is going well, and it seems we are going to be making good progress in the future releases. Below is some more information about the summit including the agenda, and registration links. Also keep in mind that OpenStack is an open source cloud computing platform which means that if you like to get your hands dirty and write some code, you can download it and hack away. The best part is that since its open source its free!

Agenda:
11/09/10
9:00 am – 11:30 am Day 1 Morning General Session
12:00 pm – 1:00 pm Day 1 Lunch
1:30 pm – 4:30 pm Day 1 Afternoon Business Session
1:30 pm – 4:30 pm Day 1 Afternoon Developer Session
6:00 pm – 10:30 pm Day 1 Evening Design Summit Reception
Reception and tour at Rackspace Headquarters.
Buses will depart from the Weston Centre lobby at 6:00pm.

11/10/10
8:00 am – 11:30 am Day 2 Morning Business Session
8:00 am – 11:30 am Day 2 Morning Developer Session
12:00 pm – 1:00 pm Day 2 Lunch
1:30 pm – 4:30 pm Day 2 Afternoon Business Session
1:30 pm – 4:30 pm Day 2 Afternoon Developer Session
4:30 pm – 5:30 pm Day 2- OpenStack Governance Discussion

11/11/10
9:00 am – 11:30 am Day 3 Morning Doc Sprint
9:00 am – 12:00 pm Day 3 Morning InstallFest
12:00 pm – 5:00 pm Day 3 Afternoon OpenStack Developer Coding Experience
12:00 pm – 1:00 pm Day 3 Lunch
6:00 pm – 10:00 pm Day 3 Evening OpenStack Developer Party
Buses will depart the Weston Centre lobby at 6:00pm.

11/12/10
9:00 am – 11:30 am Day 4 Morning Doc Sprint
9:00 am – 12:00 pm Day 4 Morning OpenStack Developer Coding Experience

http://openstack.org/

http://openstack.org/register/
http://www.openstack.org/blog/2010/11/openstack-design-summit-day-1-pictures/

Update:
Seems I was spotted in the following youtube video, while working on our OpenStack cluster for the install fest today.

Update #2:
Pics of the openstack cluster: http://picasaweb.google.com/tim.galyean/2010OpenStackSummitSanAntonio#

Ryan over at http://www.ryanuber.com/ brought up some interesting questions that I left out in my original post based off of the assumption that the reading audience was already familiar with CloudFiles.

Here are two very important and relevant questions that he asked:

Both of these are great questions to ask because without knowing the answer to these knowing how to use the API is a waste of time. So what is CloudFiles and what can I use it for? By definition CloudFiles is a ‘distributed object store’. It can be utilized to store any type of data that resides in some sort of a file format such as images, documents, videos, audio files, and so on..

Some basic terms you need to know, and understand are objects and containers. Objects are your files regardless of their type (aka what you are going to store), and containers would be the folders where you store objects in within CloudFiles.

Now that you have a base understanding of what CloudFiles is, we can take a look at some of its practical uses.

These are just a few examples of what you can use CloudFiles for, however there are many uses beyond what I can list as companies all over use it in different ways to best fit their needs. Here is some technical information provided by Rackspace on how CloudFiles works, along with some basic explanation of the CDN and how it works http://www.rackspacecloud.com/cloud_hosting_products/files/technology

Here are some additional resources for more information on ReST API’s and CDN Technology
http://en.wikipedia.org/wiki/Representational_State_Transfer
http://en.wikipedia.org/wiki/Content_delivery_network

So now that you know what CloudFiles is, and why its API is important lets get into some use cases such as uploading a file via cURL.

Creating a new container:

[h1tman@www1 ~]$ curl -X PUT -d -s -H "X-Auth-Token: XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXX" <container URL>/container
201 Created
[h1tman@www1 ~]$

Uploading an object:

[h1tman@www1 ~]$ curl -X PUT -T cf-list.sh -H "X-Auth-Token: XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXX" <container URL>/container
202 Accepted
The request is accepted for processing.
[h1tman@www1 ~]$

Deleting a container (destroy’s all its objects also):

[h1tman@www1 ~]$ curl -X DELETE -H "X-Auth-Token: XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXX" <container URL>/container
[h1tman@www1 ~]$

As you can see the API performs many functions which can easily be implemented via a shell script using curl, or by utilizing the API bindings for PHP, Ruby, Python etc.. What you decide to use it for is entirely up to you as is the means in which integrate it into your applications. Hope this helps answer some of the questions I previously left out. Enjoy!

CloudFiles uses a a “ReST” API which allow’s you to interact with it in a variety of ways by using curl, PHP, Python, and Ruby. This particular script was written in bash so that I could utilize it from a shell, however the same principals can be adapted to the other technologies as the API will interact with them in the same manner. So how does it work?

First the API expects a username and a password to provide initial authentication. In this case you will pass your username and API key provided by the rackspace control panel located HERE. Within the script you will see the following code which performs this action:

${CURL} "X-Auth-User: ${USER}" -H "X-Auth-Key: ${APIKEY}" https://auth.api.rackspacecloud.com/v1.0 2>&1 \ 
| grep Storage | awk -F'<' '{print $2}'

After passing your username and API key to the ReST API it will return you a Storage Token, and a Storage URL. These will be used to access your containers. Keep in mind that the Storage Token should not be saved within a database as the token does expire and is required for any operation which lists, uploads or deletes data. Once we have established authentication and we have our storage token we can then list and modify contents. In this example we are only listing contents, however modifying data is not much different. Here are a few examples on how to list container’s and their contents.

List all containers:

curl -s -H "X-Auth-Token: XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXX" <container URL>

List a containers contents:

curl -s -H "X-Auth-Token: XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXX" <container URL>/<container>

Additional information on how to manipulate data can be found in the CloudFiles developers guide located on the rackspace website at the following URL:
http://docs.rackspacecloud.com/files/api/cf-devguide-latest.pdf
You can also find various bindings for PHP, Python, Ruby, Java etc.. here:
http://www.rackspacecloud.com/cloud_hosting_products/files/api

Now that we have covered the basics on how the script works, here are some example usage scenarios:

Basic container listing:

$./cf-list.sh USERNAME APIKEY

Container listing with verbosity(outputs to a log file):

$./cf-list.sh USERNAME APIKEY 1

#!/bin/bash
# 
# File:         cf-list.sh
#
# Written by:   Tim Galyean
# Date:         8/4/2010
#
##################################################
 
USER="${1}"
APIKEY="${2}"
DEBUG="${3}"
LOG="cflist.log"
 
function f_exec() {
    if [ "${DEBUG}" = "1" ]; then
        # setting debug to 1 allows you to see full HTTP headers
        # using this option will redirect the output to cflist.log
        echo "redirecting verbose output to ${LOG}"
        CURL="curl -v -H"
        f_grabauthtoken 2>&1 > ${LOG}
        f_list 1>>${LOG} 2>&1
 
        # this url can be used to parse your containers and their contents
        # Example: curl -s -H "X-Auth-Token: XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXX" <container URL>/<container>
        echo "curl -s -H \"X-Auth-Token: ${STORAGETOKEN}\" $STORAGEURL"
    else
        # without using debug this script will just print your containers to stdout
        CURL="curl -s -H"
        f_grabauthtoken
        f_list
    fi
}
 
# Grab API Authentication Token, and Storage URL
function f_grabauthtoken() {
    # Establish initial authentication
    ${CURL} "X-Auth-User: ${USER}" -H "X-Auth-Key: ${APIKEY}" https://auth.api.rackspacecloud.com/v1.0 2>&1 | grep Storage | awk -F'<' '{print $2}'
 
    # Parse storage token to be used as an authentication key
    STORAGETOKEN=`curl -v -H "X-Auth-User: ${USER}" -H "X-Auth-Key: ${APIKEY}" https://auth.api.rackspacecloud.com/v1.0 2>&1 | \
    grep Storage | awk -F'<' '{print $2}' | grep Token | awk '{print $2}' | tr -d '\r'`
 
    # Parse the storage URL which will be the location of your containers
    STORAGEURL=`curl -v -H "X-Auth-User: ${USER}" -H "X-Auth-Key: ${APIKEY}" https://auth.api.rackspacecloud.com/v1.0 2>&1 | \
    grep Storage | awk -F'<' '{print $2}' | grep Url | awk '{print $2}'`
 
}   
 
# List Containers
function f_list() {
    if [ -n ${STORAGETOKEN} ]; then
        # This sends the API your token and storage url
        # the API will return your container listing
        ${CURL} "X-Auth-Token: ${STORAGETOKEN}" $STORAGEURL
    else
        echo "Please authenticate first"
    fi
}
 
# Usage instructions
function f_verify() {
    # basic usage instructions
    # to enable debugging use: ./cf-list.sh <username> <apikey> 1
    if [ -z "${USER}" ] || [ -z "${APIKEY}" ]; then
        echo "Usage: ./cf-list.sh <username> <apikey>"
    else
        f_exec
    fi
}
 
f_verify

Download it here: http://www.h1tman.com/cf-list.sh

I wrote it in ruby because I like its logger library which gives enough options and information to accomplish the task. Additionally ruby has network / http libraries built right into it without having to install any additional software, or hack together some curl command to accomplish this with bash. So here it is, feel free to modify or use it as you see fit. If you have any suggestions or feedback please leave a reply. Thanks

Configuration:
Add to a cron job at any given interval. I set mine to just run every minute:

[h1tman@h1tman ~]$ crontab -l
* * * * * ruby ~/urlmon.rb www.h1tman.com
[h1tman@h1tman ~]$

Basic Usage:

[h1tman@h1tman ~]$ ./urlmon.rb domain.com
[h1tman@h1tman ~]$ ./urlmon.rb www.domain.com

Keep in mind that http:// is not required as the domain is appended to it within the script itself.

#!/usr/bin/env ruby
# File:         urlmon.rb
#
# Written by:   Tim Galyean
# Date:         9/2/2010
 
require 'net/http'
require 'logger'
include Net
 
$LOG = Logger.new('monitor.log')
domain = ARGV[0]
 
if domain == nil
    usage = "Usage: ./urlmon.rb domain.com"
    $LOG.debug usage
elsif domain != nil
    begin
        url = URI::parse("http://"+domain.to_s)
        request = HTTP.start(url.host, url.port) do |http|
            begin
                response = http.request_get(url.request_uri)
                test = response.code + " " + response.message
                $LOG.info url.to_s + " " + test
            end
        end
    end
end

Example log output:

# Logfile created on Sat Sep 04 00:38:53 +0000 2010 by logger.rb/1.5.2.9
I, [2010-09-04T00:38:53.794756 #12547]  INFO -- : http://google.com 301 Moved Permanently
I, [2010-09-04T00:38:58.264756 #12549]  INFO -- : http://www.google.com 200 OK

Download it Here: http://www.h1tman.com/urlmon.rb