LVM to the rescue

Recently I was reminded of the magic of LVM (Logical Volume Manager) when a friend called to say that a virtual server I had built for him on Citrix XenServer was running out of disk space. The server was built with Debian GNU/Linux 6.0 and luckily I elected to use LVM partitioning during the install.

There is surprisingly little written about the history of LVM. The concept of Logical Volumes to manage storage has been around a long time and existed in Unix systems under various names. LVM version 1 became part of the Linux kernel in January 2001 with the release of version 2.4. The 2.6 kernel integrated LVM2 and this became the default for many distros. It replaced version 1 in the 2005 release of Red Hat Enterprise Linux RHEL 4.

While championed by Red Hat in the early days, most distributions now offer an option to select LVM during installation and you would be wise to do so. Old concerns over I/O latency have pretty much been overcome by faster hard drives and modern file systems.

Let's get back to the situation I spoke of above. I was asked to create a VM to replace an aging physical machine. The original server had a 100GB drive and was using about 30 percent of this so I figured a 100GB disk image in XenServer would be adequate. Of course the usage pattern changed and this quickly became too small for the server usage.

In the days before LVM there would have been a couple of ways to deal with this. If the disk usage was for a none root directory such as /var that could live on its own parition we could easily add another drive and mount it at /var. We would of course have to copy all the stuff already in /var over to the new partition and if these were active log files this would mean stopping services while we do this. If the root partition itself was taking up the space we could grow the disk image in XenServer but would then have to use something like PartitionMagic to grow the file system. This would result in significant down time.

With LVM things become much more simple. On a physical machine we could simply add another drive and then let perform the LVM magic illustrated below to make use of it within the existing disk configuration. On a VM this gets even better because we can add a disk image without physical access to the server.

I won't go into adding a disk image in XenServer as it is simple enough through the Windows app XenCenter. In my case when I added an additional 200GB image it showed up as /dev/xvdb. I could then log into the server and begin integrating this into LVM.

The first thing is to create a partition on the new drive:

$> sudo fdisk /dev/xvdb

Next we make a file system:

$> sudo mkfs -t ext3 /dev/xvdb1

We get the usual warning about this new filesystem being checked after 33 reboots and I don't want that so I use tune2fs to fix this:

$> sudo tune2fs -c 0 /dev/xvdb1

Next we create the physical volume:

$> sudo pvcreate /dev/xvdb1

Before we go any further we want to our current disk layout.

$> sudo lvdisplay

$> sudo lvdisplay 
  --- Logical volume ---
  LV Name                /dev/www6/root
  VG Name                www6
  LV UUID                2lhl2U-h1L7-jgeO-2weG-LVQJ-EZvF-fIFLUw
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                285.67 GiB
  Current LE             73132
  Segments               2
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           254:0
   
  --- Logical volume ---
  LV Name                /dev/www6/swap_1
  VG Name                www6
  LV UUID                gqWkKS-q9t8-MxEK-iFOu-zBp5-E9Qp-ulwBid
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                4.09 GiB
  Current LE             1046
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           254:1

So now we need to expand the size of the volume group to accomdate the new disk:

$> sudo vgextend www6 /dev/xvdb1

Now we expand the Logical Volume:

$> sudo lvextend -L+190G /dev/www6/root

In Debian we use resize2fs to make the file system fit the volume:

$> sudo resize2fs /dev/www6/root

To see how cool LVM is, open up another shell and run df to see the file system growing live without interrupting running services on the system. All this with zero downtime!

Open Source JobScheduler

Introduction

The company I work for uses ActiveBatch Job Scheduling from Advanced Systems Concepts, Inc. For the longest time I have been looking for a viable replacement for this. It is not that there is anything inadequate about the software. On the contrary I suspect we are only using a tiny portion of what it is capable of. The problem is that software runs only on Microsoft Windows. We are a Linux/Mac shop and I resent having to run a Windows system to schedule jobs on Linux servers. In addition, the software costs $12,000 annually to maintain support.

JobScheduler from sos-berlin.com

So I was delighted when I found a link to the JobScheduler at http://www.sos-berlin.com/.

JobScheduler has exactly what I was looking for: Job Chains with dependencies, email notification, logging and in particular, event driven scheduling. The one downside I have found so far is the requirement of a 32 bit JVM. The FAQ anticipates this question stating, "You could ask why has this software not been ported to 64bit? The simple answer is: there is no need for this." It is true that there is no need in terms of the extra memory requirement of the 64 bit JVM but in 2012 one may reasonably say there is no need to stay behind the times. The vast majority of systems today are installed as 64 bit systems, so why not update JobScheduler What this means practically for 64 bit Linux users like myself is an extra configuration step after installation to ensure the 32 bit JVM is used.

On Debian I simply installed the 32 bit JVM with apt-get:

#apt-get install ia32-sun-java6-bin

and added the following line to jobscheduler_environment_variables.sh located in the bin directory under the freshly installed jobscheduler:

JAVA_HOME="/usr/lib/jvm/ia32-java-6-sun/jre"

Documentation
The documentation is pretty thorough. In the course of this write-up I will be working through the documents bundled with the software and trying out the sample implementations. I this way I hope to highlight any weaknesses as well as pointing out the strengths of the software. While there is a supported commercial version available, I will be using only the GPLd version and therefore reliant on the published documentation.


What is a Job?
The documentation defines a job as a program to be executed, its rn time and the event of an error occurring. A job also contains any parameters to be used, pre and post processing, file locks and follow-on jobs. These are all part of the configuration defined in XML, using the tag.

Jobs can be configured as part of the central configuration file in ./config/scheduler.xml but this requires a restart of the JobScheduler. A better solution is to use the hot directory at ./config/live where changes will be picked up with no restart necessary.

Jobs can either be executable programs or implemented using the Job Scheduler API. The API supports a number of languages including Java, Perl, and JavaScript.

My Environment
Before we get into some examples I should set out my installation environment. JobScheduler is installed at the following location:

/opt/sos-berlin.com/jobscheduler/scheduler

The main directory of interest under here is the bin directory. In addition there is a directory structure under the home directory of the user who launched the daemon. In my case it is here:

~/sos-berlin.com/jobscheduler/scheduler

and below here we have: config, jobs, logs, mail, and a sym link to scheduler_home.

The jobs directory contains sample job configuraitons and the config/live directory is where we will put the XML configurations for jobs to be picked up without restart.

Example One
Run a shell script every minute which will log the current date.

The script called getDate.sh is stored under jobs. It includes only the simple date command.

The XML controlling this job looks like this:

<?xml version="1.0" encoding="iso-8859-1"?>
<!--This very verbose example will automatically start the job getDate.sh every minute on the days specified within weekdays within the period 12am-23:59pm. Furthermore you can start the job manually in the period 8am-8pm as defined by run_time. The job executes the./jobs/getDate.sh shell script. -->

<job>
<script language="shell">
<include file="jobs/getDate.sh"/>
<!-- for Windows use "jobs\my_shell_script.cmd" -->
</script>
<run_time begin = "00:00"
end = "23:59">
<weekdays>
<day day="1">
<period begin = "00:00" end = "23:59" repeat = "60"/>
</day>
<day day="2">
<period begin = "00:00" end = "23:59" repeat = "60"/>
</day>
<day day="3">
<period begin = "00:00" end = "23:59" repeat = "60"/>
</day>
<day day="4">
<period begin = "00:00" end = "23:59" repeat = "60"/>
</day>
<day day="5">
<period begin = "00:00" end = "23:59" repeat = "60"/>
</day>
<day day="6">
<period begin = "00:00" end = "23:59" repeat = "60"/>
</day>
<day day="7">
<period begin = "0:00" end = "23:59" repeat = "60"/>
</day>
</weekdays>
</run_time>
</job>

We can now see the job in the GUI of JobScheduler:

Here we see the job getDate on the left hand side of the screen, with the logs created for each 1 minute run on the right hand side. This is a very simplified example but shows the basics of how JobScheduler works.

Example Two
In this example we will schedule a job by watching a directory for changes.

Again in the live directory we add a file called listDir.job.xml:

<?xml version="1.0" encoding="iso-8859-1"?>
<job>
<script language="shell">
<include file="jobs/listDir.sh"/>
<!-- for Windows use "jobs\my_shell_script.cmd" -->
</script>
<start_when_directory_changed directory = "~/sos-berlin.com/jobscheduler/scheduler/jobs/notification_dir" regex = "\.txt$"/> <run_time/> </job>

Now if we add a file to the notification_dir folder, the script listDir.sh will be executed. This type of execution can be combined with time based execution detailed above.