A bit of background

Alfresco comes with the ability to be extended in a nice easy way, that is through the use of Alfresco Module Packages. In essence it is the delta of changes you would have made to the raw source code if you wanted to make some sort of customisation or apply one of the ones alfresco supplies like the S3-connector.

Over the last 3 years I’ve seen it done a number of ways, using the mmt tool to apply them manually, a shell script to do it and now I decided that wasn’t good enough.

Using the mmt tool manually is obviously not brilliant, some poor person has to sit there and run it to apply the amps. So you may have guessed this is not a good idea.

What about wrapping the mmt tool in a shell script that can be triggered by say a sysadmin to apply all the amps or just have it executed once per amp using some configuration managent tool like puppet. This is good. You put the amp into the configuration management tool push the right buttons and it magically get’s applied to the war files and all is well. Well sort of, what happens if someone just throws an amp on the server? who puts it in configuration management? who’s made a backup? Well I decided that I’d write a new script for applying amps so that it can be used both with a CM and as a ad-hock script.

What does it do?

I’ve written it so it will trawl through a directory and pull up every amp in the directory and it will apply the amps to alfresco or share as needed. What’s quite handy is that it will take several versions of an amp and work out what the latest version is, it will check the latest version against what is already installed in the war and then if the amp is a newer version it will apply it after making a backup.

For some odd reason I also made it cope with a variety of amp naming schemes, so you could upload alfresco-bob-123.amp or you could upload frank-share-super-1.2.3.4-5.amp it’s your amp, call it what you want. All the script cares about is the correlation of terms between the file name and the amp info when it’s installed. So as long as you use 2 words from the file name that also appear in the amp description it will work it out for you. The higher the correlation the more accurate it would be, it is configurable too but I set it to 2 occurrence of at least 2 words to match, so far… it’s working.

I also forgot to mention that the script will stop alfresco clear the caches and restart it for you in a pretty safe way.

A Script

Firstly I realise this is a bad format to get the script I’ve in the past put them in a git repo and shared it that way, I have put this one in a git repo and I hope to share that repo with some of the things we have done at alfresco that are useful for either running servers in general or for running alfresco either way I hope to shortly get that out on a public repo but for now here it is:

#!/usr/bin/ruby
# Require libs
$:.unshift File.expand_path("../", __FILE__)
require 'lib/logging'
require 'fileutils'
require 'timeout'

# Set up logging provider
Logging::new('alfresco_apply_amps')
Logging.log_level("INFO",false)
$log.info "Starting"

#CONSTANTS
BKUP_LOCATION="/var/lib/alfresco/alf_data/backups"
ALF_MMT="/var/lib/tomcat6/bin/alfresco-mmt.jar"
WEBAPPS_DIR="/var/lib/tomcat6/webapps"
AMP_LOCATIONS=["/var/lib/alfresco/alf_data/amps/"]

#Defaults
@restart=false

#Methods
def available_amps(amp_dir)
  #Get a list of Amps
  amps_list = `ls #{amp_dir}*`
  amps_array = amps_list.split("\n")
end

def backup(war)
  version=`/usr/bin/java -jar #{ALF_MMT} list #{WEBAPPS_DIR}/#{war}.war | grep Version | awk '{print $3}'`

  #Date stamp the war
  $log.info "Backing up #{WEBAPPS_DIR}/#{war}.war to #{BKUP_LOCATION}/#{war}-#{current_date}.war"
  `cp -a #{WEBAPPS_DIR}/#{war}.war #{BKUP_LOCATION}/#{war}-#{current_date}.war`
end

def clear_caches()
  $log.debug "Cleaning caches"  
  delete_dir("#{WEBAPPS_DIR}/alfresco/")
  delete_dir("#{WEBAPPS_DIR}/share/")
  delete_dir('/var/cache/tomcat6/work/',true)
  delete_dir('/var/cache/tomcat6/temp/',true)
  $log.info "Caches cleaned"  
end

def compare_strings(str1,str2,options={})
  matches = options[:matches] || 2
  frequency = options[:frequency] || 2

  #Make one array of words
  words=Array.new
  words << str1.split(' ') << str2.split(' ')
  words.flatten!
  #Hash to store each unique key in and number of occurances
  keys = Hash.new
  words.each do |key|
    if keys.has_key?(key)
      keys[key] +=1
    else
      keys.merge!({key =>1})
    end
  end

  #Now we have a Hash of keys with counts how many matches and what frequency
  #where a match is a unique key >1 and frequency si the count of each key i.e. 
  #matches=7 will mean 7 keys must be >1 frequency=3 means 7 matches must be > 3
  
  act_matches=0
  keys.each_pair do |key,value|
    if value >= frequency
      act_matches +=1
    end
  end
  if act_matches >= matches
    true
  else
    false
  end
end

def compare_versions(ver1,ver2)
  #return largest
  v2_maj=0
  v2_min=0
  v2_tiny=0
  v2_release=0
  v1_maj=0
  v1_min=0
  v1_tiny=0
  v1_release=0
  if ver1 =~ /\./ && ver2 =~ /\./
    #both are dotted notation
    #Compare maj -> release

    #Conver '-' to '.'
    ver1.gsub!(/-/,'.')
    ver2.gsub!(/-/,'.')

    v1_maj = ver1.split('.')[0]
    v1_min = ver1.split('.')[1] || 0
    v1_tiny = ver1.split('.')[2] || 0
    v1_release = ver1.split('.')[3] || 0

    v2_maj = ver2.split('.')[0]
    v2_min = ver2.split('.')[1] || 0
    v2_tiny = ver2.split('.')[2] || 0
    v2_release = ver2.split('.')[3] || 0

    if v1_maj > v2_maj
      return ver1
    elsif v1_min > v2_min
      return ver1
    elsif v1_tiny > v2_tiny
      return ver1
    #Don't compare release for now as some amps don't put the release in the amp when installed so you end up re-installing
    #elsif v1_release > v2_release
    #  return ver1
    else
      return ver2
    end
  else
    #Validate both are not-dotted
    if ver1 =~ /\./ || ver2 =~ /\./
      $log.debug "Eiher both types aren't the same or there's only one amp"
      return ver2
    else
      result = ver1<=>ver2
      if result.to_i > 0 && !result.nil?
        return ver1
      else
        return ver2
      end
    end
  end
end

def current_date()
  year=Time.now.year
  month=Time.now.month
  day=Time.now.day
  if month < 10
    month = "0"+month.to_s
  end
  if day < 10
    day = "0"+day.to_s
  end
  "#{year.to_s+month.to_s+day.to_s}"
end

def current_version(app, amp_name)

#
# THIS needs to cope with multiple amps being installed, produce a array hash [{:amp=>"ampname",:version => ver},etc]
#

  if app == "alfresco" || app == "share"
    amp_info = `/usr/bin/java -jar #{ALF_MMT} list #{WEBAPPS_DIR}/#{app}.war`
    amp_title=""
    amp_ver=0
    #$log.debug "Amp info: #{amp_info}"
    amp_info.each_line do |line|
      if line =~ /Title/
        amp_title=line.split("Title:").last.strip.gsub(%r/(-|_|\.)/,' ')
      elsif line =~ /Version/
        # strip/replace ampname, downcase etc
        if compare_strings(amp_name.gsub(%r/(-|_|\.)/,' ').downcase,amp_title.downcase)
          amp_ver=line.split("Version:").last.strip
          $log.info "Installed Amp found for #{amp_name}"
          $log.debug "Installed version: #{amp_ver}"
        else
          $log.debug "No installed amp for #{amp_name} for #{app}"
        end
      end
    end
  else
    $log.warn "The application #{app} can not be found in #{WEBAPPS_DIR}/"
  end
  return amp_ver
end

def delete_dir (path,contents_only=false)
  begin
    if (contents_only)
      $log.debug "Removing #{path}*"
      FileUtils.rm_rf Dir.glob(path+"*")
    else
      $log.debug "Removing #{path}"
      FileUtils.rm_rf path
    end
  rescue Errno::ENOENT
    $log.warn "#{path} Does not exist"
  rescue Erro::EACCES
    $log.warn "No permissions to delete #{path}"
  rescue
    $log.warn "Something went wrong"
  end
end

def firewall(block=false)
  if block
    `/sbin/iptables -I INPUT -m state --state NEW -m tcp -p tcp  --dport 8080 -j DROP`
  else
    `/sbin/iptables -D INPUT -m state --state NEW -m tcp -p tcp  --dport 8080 -j DROP`
  end
end

def get_amp_details(amps)
  amps_hash = Hash.new
  amps.each do |amp|
    amp_hash = Hash.new
    #Return hash with unique amps with just the latest version
    amp_filename = amp.split("/").last
    amp_path = amp
    amp_name=""
    amp_version=""
    first_name=true
    first_ver=true
    #Remove the ".amp" extension and loop through
    amp_filename[0..-5].split("-").each do |comp|
      pos = comp =~ /\d/
      if pos == 0
        if first_ver
          amp_version << comp
          first_ver=false
        else
          #By commenting this out the release will get ignored which because some amps to put it in their version is probably safest
          #amp_version << "-" << comp
        end
      else
        if first_name
          amp_name << comp.downcase
          first_name=false
        else
          amp_name << "_" << comp.downcase
        end
      end
    end

    #If a key of amp name exists, merge the version down hash else merge the lot
    if amps_hash.has_key?(amp_name)
      amp_hash={amp_version => {:path => amp_path, :filename => amp_filename}}
      amps_hash[amp_name].merge!(amp_hash)
    else 
      amp_hash={amp_name =>{amp_version => {:path => amp_path, :filename => amp_filename}}}
      amps_hash.merge!(amp_hash)
    end
  end
  return amps_hash
end

def install_amp(app, amp)
  $log.info "applying amp to #{app}"
  $log.warn "amp path must be passed!" unless !amp.nil?

  $log.debug "Command to install = /usr/bin/java -jar #{ALF_MMT} install #{amp} #{WEBAPPS_DIR}/#{app}.war -nobackup -force"
  `/usr/bin/java -jar #{ALF_MMT} install #{amp} #{WEBAPPS_DIR}/#{app}.war -nobackup -force`
  restart_tomcat?(true)
  $log.debug "Setting flag to restart tomcat"
end

def latest_amps(amp_hash)
  amp_hash.each_pair do |amp,amp_vers|
    latest_amp_ver=0
    $log.debug "Comparing versions for #{amp}"
    amp_vers.each_key do |version|
      $log.debug "Comparing #{latest_amp_ver} with #{version}"
      latest_amp_ver = compare_versions(latest_amp_ver,version)
      $log.info "Latest version for #{amp}: #{latest_amp_ver}"
      if latest_amp_ver != version
        amp_vers.delete(version)
      end
    end
  end
  return amp_hash
end

def next_version?(ver, current_ver, app)
  #Loop through amp versions to work out which is newer than the installed
  #Turn list into array
  next_amp=false
  $log.debug "if #{ver} > #{current_ver}"
  if ( ver.to_i > current_ver.to_i)
    $log.debug "Next #{app} amp version to be applied:  #{ver}"
    next_amp=true
  end
end

def restart_tomcat()
  #If an amp was applied restart
  if (restart_tomcat?)
    $log.info "Restarting Tomcat.... this may take some time"
    $log.debug"Getting pid"
    if (File.exists?('/var/run/tomcat6.pid') )
      pid=File.read('/var/run/tomcat6.pid').to_i
      $log.debug "Killing Tomcat PID= #{pid}"
      begin
        Process.kill("KILL",pid)
        Timeout::timeout(30) do
          begin
            sleep 5
            $log.debug "Sleeping for 5 seconds..."
          end while !!(`ps -p #{pid}`.match pid.to_s)
        end
      rescue Timeout::Error
        $log.debug "didn't kill process in 30 seconds"
      end
    end
    $log.debug "Killed tomcat"

    #Clear caches
    clear_caches
    $log.info "blocking firewall access"
    firewall(true)
    $log.debug "starting tomcat"
    `/sbin/service tomcat6 start`
    if ($?.exitstatus != 0)
      $log.debug "Tomcat6 service failed to start, exitstatus = #{$?.exitstatus}"
    else
      #Tomcat is starting sleep until it has started
      #For now sleep for 180 seconds
      $log.info "Sleeping for 180 seconds"
      sleep 180
      $log.info "un-blocking firewall access"
      firewall(false)
    end
  else
    $log.info "No new amps to be installed"
  end
end

def restart_tomcat?(bool=nil)
  @restart = bool unless bool.nil?
  #$log.debug "Restart tomcat = #{@restart}"
  return @restart
end

# - Methods End

#
# doGreatWork()
#

#Store an Hash of amps
amps=Hash.new

#For each AMP_LOCATIONS find the latest Amps
AMP_LOCATIONS.each do |amp_loc|
  $log.debug "Looking in #{amp_loc} for amps"
  amps.merge!(get_amp_details(available_amps(amp_loc)))
end

#Sort through the array and return only the latest versions of each amp
latest_amps(amps)

amps.each do |amp, details|
  #The Amps in here are the latest of their kind available so check with what is installed
  details.each_pair do |version,value|
    if amp =~ /share/
      if next_version?(version,current_version("share",amp),"share")
        $log.debug "Backing up share war"
        backup("share")
        $log.info "Installing #{amp} (#{version}): #{value[:path]}"
        install_amp("share",value[:path])
      else
        $log.info "No update needed"
      end
    else
      if next_version?(version,current_version("alfresco",amp),"alfresco")
        $log.debug "Backing up alfresco war"
        backup("alfresco")
        $log.info "Installing #{amp} (#{version}): #{value[:path]}"
        install_amp("alfresco",value[:path])
      else
        $log.info "No update needed"
      end
    end
  end
end

$log.debug "Restart tomcat?: #{restart_tomcat?}"
restart_tomcat

$log.info "All done for now"

Okay 2 things, it’s a long script all in one file to make it easy to transport, I’ve also used a logging class that enables logging to screen / file that is …below :) you could also just remove the require at the top and replace “$log.debug” with “puts” up to you.

#
#   Set up Logging
#

require 'rubygems'
require 'log4r'

class Logging

  def initialize(log_name,log_location="/var/log/")
    # Create a logger named 'log' that logs to stdout
    $log = Log4r::Logger.new log_name

    # Open a new file logger and ask him not to truncate the file before opening.
    # FileOutputter.new(nameofoutputter, Hash containing(filename, trunc))
    file = Log4r::FileOutputter.new('fileOutputter', :filename => "#{log_location}#{log_name}.log",:trunc => false)

    # You can add as many outputters you want. You can add them using reference
    # or by name specified while creating
    $log.add(file)
    # or mylog.add(fileOutputter) : name we have given.

    # As I have set my logging level to ERROR. only messages greater than or 
    # equal to this level will show. Order is
    # DEBUG < INFO < WARN < ERROR < FATAL

    # specify the format for the message.
    format = Log4r::PatternFormatter.new(:pattern => "[%l] %d: %m")

    # Add formatter to outputter not to logger. 
    # So its like this : you add outputter to logger, and add formattters to outputters.
    # As we haven't added this formatter to outputter we created to log messages at 
    # STDOUT. Log messages at stdout will be simple
    # but the log messages in file will be formatted
    file.formatter = format
    
  end

  def self.log_level(lvl,verbose=false)
    # You can use any Outputter here.
    $log.outputters = Log4r::Outputter.stdout if verbose

    # Log level order is DEBUG < INFO < WARN < ERROR < FATAL
    case lvl
        when    "DEBUG"
            $log.level = Log4r::DEBUG
        when    "INFO"
            $log.level = Log4r::INFO
        when    "WARN"
            $log.level = Log4r::WARN
        when    "ERROR"
            $log.level = Log4r::ERROR
        when    "FATAL"
            $log.level = Log4r::FATAL
        else
             print "You provided an invalid option: #{lvl}"
    end
  end

end

I hope this helps people out, if there’s any issues just leave comments and i’ll help :)