Tuesday, December 31, 2013

Groovy: Line-by-line process output - stdout and stderr

It is very easy to start up command line processes with Groovy.   Just use the execute() method on a string, which returns a Java Process object:

def proc = "./some-shell-script.sh".execute()

But what if we want to capture the stdout and stderr of the process?  Turns out this is also very easy:

def proc = "./some-shell-script.sh".execute()
def rc = proc.waitFor()
def output = proc.text

This is all well and good, but like anything it has a few limitations.   The output is buffered and not available until the process completes.   That makes it not as useful for say, logging to the console in real time as you might want to do in a Jenkins job.   Also, it might be handy to prefix each line of output as it comes from the process with a timestamp or something like that.   The good news is that Groovy comes with some things that can be used for this purpose.

Groovy extends the Process object with some interesting methods that make it easy to handle stdout and stderr very easily.   We can also use the ANT LineOrientedOutputStream to give us line-by-line behavior:


  class LineOutput extends LineOrientedOutputStream
  {
    String prefix
    List lines

    @Override
    protected void processLine(String line) throws IOException
    {
      lines.add(line)
      println "${new Date().format('yyyy-MM-dd HH:mm:ss.SSS')} ${prefix} : ${line}"
    }
  }

We can start threads for stdout and stderr on the process like this:
 
def outLines = []
def errorLines = []
def proc = "./some-shell-script.sh".execute()
def outThread = proc.consumeProcessOutputStream(new LineOutput(prefix: "out", lines: outLines))
def errThread = proc.consumeProcessErrorStream(new LineOutput(prefix: "err", lines: errLines))

The threads will automatically start, and begin recording and echoing every line as soon as it happens. The we can wait for the process to terminate and clean up:
 
try { outThread.join(); } catch (InterruptedException ignore) {}
try { errThread.join(); } catch (InterruptedException ignore) {}
try { proc.waitFor(); } catch (InterruptedException ignore) {}

All of this can be combined into a class for convenient access, adding some configuration options to make it more useful:
 
class ShellCommand
{
  private final String cmd
  private final boolean echo
  private final Process proc
  private final Thread outThread
  private final Thread errThread
  private final List outLines = []
  private final List errLines = []

  private class LineOutput extends LineOrientedOutputStream
  {
    boolean echo
    String prefix
    List lines

    @Override
    protected void processLine(String line) throws IOException
    {
      lines.add(line)
      if (echo)
        println "${new Date().format('yyyy-MM-dd HH:mm:ss.SSS')} ${prefix} : ${line}"
    }
  }


  ShellCommand(String cmd, boolean echo = false,String outPrefix = "stdout", String errPrefix = "stderr")
  {
    this.cmd = cmd
    this.echo = echo
    // Start the process.
    this.proc = cmd.execute()
    // Start the stdout, stderr spooler threads
    outThread = proc.consumeProcessOutputStream(new LineOutput(echo: echo, prefix: outPrefix, lines: outLines))
    errThread = proc.consumeProcessErrorStream(new LineOutput(echo: echo, prefix: errPrefix, lines: errLines))
  }

  def waitForOrKill(int millis)
  {
    proc.waitForOrKill(millis)
    _done()
  }

  private void _done()
  {
    try { outThread.join(); } catch (InterruptedException ignore) {}
    try { errThread.join(); } catch (InterruptedException ignore) {}
    try { proc.waitFor(); } catch (InterruptedException ignore) {}
    proc.closeStreams()
  }

  def getRc()
  {
    def rc = null
    try { rc = proc.exitValue() } catch (IllegalThreadStateException e) {}
    return rc
  }
}

This also adds the waitForOrKill(millis) method, which is useful when running shell commands that are expected to complete in a reasonable amount of time (or something is wrong).

Friday, December 27, 2013

Reading Java Properties files with shell

This is an oldie but a goodie, so I'll re-post it here on this new blog.   How do you read Java properties files with shell?  Maybe another good question is: Why would you want to read Java properties files with shell?   Let's start with that.

Why?

Quite a few Java Enterprise systems that I've worked on used Java (ANT) properties files for build and runtime configuration.   It's an easy to edit and easy to read for both humans and Java programs.   However, when running Java Enterprise on Linux, you invariably get some shell scripts for process control, deployment, etc.   These need to know about database host names, and other things that are included in the configuration file.

Method 1 - Scan through the file to get an individual property

This works well if you just want to read a specific property in a Java-style properties file. Here's what it does:
  1. Remove comments (lines that start with '#')
  2. Grep for the property
  3. Get the last value using tail -n 1
  4. Strip off the property name and the '=' (using cut -d "=" -f2-) to handle values with '=' in them, as a commenter suggested).
sed '/^\#/d' myprops.properties | grep 'someproperty'  | tail -n 1 | cut -d "=" -f2-

It can also be handy to strip off the leading and trailing blanks with this sed expression:
 
s/^[[:space:]]*//;s/[[:space:]]*$//

Which makes the whole thing look like this:

sed '/^\#/d' myprops.properties | grep 'someproperty'  | tail -n 1 | cut -d "=" -f2- | sed 's/^[[:space:]]*//;s/[[:space:]]*$//'

Shell scripts can use this technique to set environment variables, for example:

JAVA_HOME=`sed '/^\#/d' build.properties | grep 'jdk.home'  | tail -n 1 | cut -d "=" -f2- | sed 's/^[[:space:]]*//;s/[[:space:]]*$//'`

Method 2 - Convert the properties file to a shell script, then run it

This method is good for when you want to read the whole properties file. However, it only works if the property names are valid shell variable names. For example, property names with a '.' in them will cause this to break. Here is the original article from +Michael Slinn that inspired this method.

Security Note

These techniques generate shell code from the properties file, then execute the generated code. This can be a potential security issue if the properties file has hidden shell code in it. Make sure you have appropriate access control on the properties file in this case. 


The steps are:
  1. Create a temp file.
  2. Stream the properties file, convert it to unix LF format (unless you are sure the file was produced on Linux).
  3. (not sure what the first 'sed' does)
  4. Double quote all the values, to preserve spaces.
  5. Source in the file. Shell should ignore Java properties file comments automatically.
  6. Erase the temp file.
Here is the code:

TEMPFILE=$(mktemp)
cat somefile.properties | dos2unix |
sed -re 's/"/"/'g|sed -re 's/=(.*)/="\1"/g'>$TEMPFILE
source $TEMPFILE
rm $TEMPFILE

Method 2 reloaded - Use AWK

This method is the same as the previous one except we use an inline awk script to replace any non-alpha-digit-underscore characters in the property names with underscore. This solves the problem the previous method has when it processes properties that have '.' in the name, for example.
Some clever person could probably figure out how to do the same thing with sed, but awk works.
Steps:
  1. Create a temporary file.
  2. Stream the properties file to stdout and (optionally) filter out dos style linefeeds.
  3. Split each line using "=" as the field separator. If there are exactly two fields, then substitute the non-identifier characters with underscore in the first one, and print the second field with quotes around it.
  4. If the line doesn't have an =, or seems to have more than two values, just print it out.
  5. Source in the temporary file and delete it.
Teh codez:

TEMPFILE=$(mktemp)
cat somefile.properties | dos2unix |
awk 'BEGIN { FS="="; } \
/^\#/ { print; } \
!/^\#/ { if (NF >= 2) { n = $1; gsub(/[^A-Za-z0-9_]/,"_",n); print n "=\"" $2 "\""; } else { print; } }' \
 >$TEMPFILE
source $TEMPFILE
rm $TEMPFILE

Method 2 - Using Groovy

This is my current favorite!   Since Groovy has all of the ANT facilities available, it can do all the property interpolation things that ANT does, plus handle all the quirks of Java Properties files that the other methods may not handle easily.

The example here can read two properties files: one for default values, and one for overrides that are environment specific.   This is great for when you have a lot of values that are the same most of the time, and only a few which change based on the environment.


#!/usr/bin/groovy
def generateEnvVars(File propertiesFile,File defaultsFile) {
    println "#!/bin/bash"
    AntBuilder ant = new AntBuilder()
    print "# Reading ${propertiesFile} ..."
    ant.property (file: propertiesFile)
    println "OK"
    if (defaultsFile) {
            print "# Reading ${defaultsFile} ..."
            ant.property (file: defaultsFile)
            println "OK"
    }
    // The properties are now in ant.project.properties
    def props = ant.project.properties
    props.keySet().toList().sort().each { String name ->
      // Shell-ify the property name.
      def shellVar = name.replaceAll(/[^A-Za-z0-9_]/, '_')
      def value = props[name]
      println "${shellVar}=\'${value}\'"
    }
}
 
// Main
// args[0] - Name of the main properties file
// args[1] (optional) - Name of the defaults properties file
generateEnvVars(new File(args[0]), args.length > 1 ? new File(args[1]) : null)

This reads the properties file (the first argument) and optionally the default properties file (the second argument), and writes the shell statements to stdout.  Since we're using ANT to read in the properties files, this version has a lot more functionality:
  1. All the standard ANT properties will be processed. 
  2. ANT will interpolate the property values. 

So, if we have the following properties file:

# properties example
some.property=this is a property
multiline.property=this is a java \
multi line property
interpolated=the value of some.property is "${some.property}"

The generated shell script will contain:
 
some_property='this is a property'
multiline_property='this is a java multi line property'
interpolated='the value of some.property is "this is a property"'

Edge Case - Shell commands in property values

What happens when the properties file looks like this? zzzhack=' echo "all your base are belong to us" ' When we use the Groovy script with this file, we hit an edge case:
 
$ groovy readprops.groovy my.properties > env-vars.sh
$ source env-vars.sh
all your base are belong to us

Uh oh! This property value could have done something very bad. We can change the Groovy script to escape any single quotes like this:
 
def value = props[name].replaceAll(/[\']/,"\\\\'")

However, this may break some things. Of course, this isn't a problem if you don't embed shell commands into your properties files... with single quotes around them. (yeah... well... maybe you shouldn't do that!)

Here's the previous generation of this post: http://shrubbery.homeip.net/c/display/W/Reading+Java-style+Properties+Files+with+Shell

Thursday, December 12, 2013

Write Your Own Seam 2 Interceptors

When using Seam2, there are two main ways to implement a backing bean component:
  1. EJB3 - If EJB3 is available, this is a good choice.   Not only do you get all the Seam capabilities (@In, etc.) you also get all the EJB3 capabilties such as Interceptors.
  2. POJO - This is a good choice when EJB3 is not available, or if for some reason you are EJB-averse (you should see a doctor about that).
Today I was working with some existing POJO Seam components that could not use the nice EJB3 interceptors that I had already created.   I usually just convert these to stateful session beans, but I decided I'd try my hand at using a Seam interceptor.

What you need to create a Seam interceptor

  1. An interceptor class, annotated with  org.jboss.seam.annotations.intercept.Interceptor.   Usually the @Interceptor annotation will specify the interceptor order, for example:

    @Interceptor(around = {
            BijectionInterceptor.class,
            MethodContextInterceptor.class,
            ConversationInterceptor.class,
            SynchronizationInterceptor.class,
            ConversationalInterceptor.class,
            RemoveInterceptor.class,
            SeamInterceptor.class,
            SecurityInterceptor.class,
            TransactionInterceptor.class,
            EventInterceptor.class,
            HibernateSessionProxyInterceptor.class,
            ManagedEntityInterceptor.class
    })
    public class MyInterceptor {
    ...
        @AroundInvoke
        public Object aroundInvoke(InvocationContext invocation) throws Exception
        {
    ...    
    }
    
    Note: This uses org.jboss.seam.annotations.intercept.AroundInvoke, not the EJB3 annotation.
  2. Extend AbstractInterceptor to simplify getting access to the Component (the meta-data) for the intercepted object.
  3. An annotation that allows you to apply the interceptor to components.

    import org.jboss.seam.annotations.intercept.Interceptors;
    
    import java.lang.annotation.ElementType;
    import java.lang.annotation.Retention;
    import java.lang.annotation.RetentionPolicy;
    import java.lang.annotation.Target;
    
    @Target(ElementType.TYPE)
    @Retention(RetentionPolicy.RUNTIME)
    @Interceptors(MyInterceptor.class)
    public @interface MyInterceptorAnnotation {}
    
    
  4. The component class, with the custom annotation on it.

    @Name("myBackingBean")
    @Scope(ScopeType.CONVERSATION)
    @MyInterceptorAnnotation
    public class MyBackingBean implements Serializable
    {
       ...
    

Similarities and Differences between Seam Interceptors and EJB3 Interceptors

  • In both systems, the interceptor instance is scoped to the component.   That is, there is one instance of the interceptor for each intercepted component by default.
  • Seam interceptors can be forced to be stateless, so you can have stateless interceptor instances used with stateful components.
  • EJB3 interceptors have specific lifecycle methods.   With Seam Interceptors, you need to determine this in the @AroundInvoke method.
  • It's not as easy to apply Seam Interceptors to all components.   With an EJB3 interceptor, you can specify default interceptor bindings in META-INF/ejb-jar.xml.   (I'm not currently sure how you apply custom Seam interceptors implicitly)