bash command fails in tomcat/java Runtime.getRuntime.exec(), but works from command line

I have a tomcat webapp that runs a process in the shell because several of the utilities are not available in java. This code works perfectly on other machines, but there is a mysterious problem on my public server.

String[] textAnalysisPipeline = {
    "/bin/sh",
    "-c",
    "/bin/cat " + inputfileLoc +
    " | tee /tmp/debug1 | " + loadJar + " " + jarOptLookupLoc + " " + optHfstLoc +
    " 2>/dev/null | " + "tail -n+5" + // get rid of the header that hfst-ol.jar produces
    " | tee /tmp/debug2 | cut -f 1-2" + // get rid of the "0.0" weights
    " | tee /tmp/debug3 | " + cgConvLoc +
    " | tee /tmp/debug4 | " + vislcg3Loc + " -g " + vislcg3DisGrammarLoc +  // disambiguate with the constraint grammar
    " | tee /tmp/debug5 > " + outputfileLoc};
log.debug("Text analysis pipeline: "+textAnalysisPipeline[2]);
Process process = Runtime.getRuntime().exec(textAnalysisPipeline);
process.waitFor();

I print the string to the log, and it looks like this: (path/to/ is not the actual paths)

/bin/cat /path/to/inputFile | tee /tmp/debug1 | java -jar /path/to/hfst-ol.jar /path/to/analyser.ohfst 2>/dev/null | tail -n+5 | tee /tmp/debug2 | cut -f 1-2 | tee /tmp/debug3 | /usr/local/bin/cg-conv | tee /tmp/debug4 | /path/to/vislcg3 -g /path/to/grammar.rlx | tee /tmp/debug5 > /path/to/outputFile

If I copy this pipeline from the log and run it from the bash command line, I get the desired output all the way to the end of the pipeline. However, when the tomcat server runs the command, it produces an empty file. The debug files debug1 and debug2 are as expected, but debug3 and thereafter is empty, which suggests that the pipeline fails at cut -f 1-2 (see UPDATE 1 below).

OS – Fedora 22
java – openjdk 1.8.0_77
tomcat – 7.0.39
sh –> bash – 4.3.42(1)-release

================================================================
UPDATE 1:

This does not seem to be a problem with cut. I wrote a short python script, cut.py to achieve the same functionality as cut -f 1-2 (remove 't0.0' from the end of each line)

import re, sys
myRE = re.compile( r's+0.0$' )
for line in sys.stdin :
    sys.stdout.write( myRE.sub( '', line ) )

Using cut.py in place of cut, I get the same problem. With the server debug3 and beyond is empty, but if I copy-paste from the log to the interactive shell, everything works fine.

================================================================
UPDATE 2:

I also wrote a simple bash script to run the pipeline so that tomcat/java only runs the bash script with one argument for the input/output filename. If I run the script from an interactive shell, it works, but the results are no different in tomcat, using cut or cut.py in the shell script.

Answer

By default installation on most systems, Tomcat does not have the same environment as your user. It does not run, for instance, your .bashrc .profile or whatever you have in your login script, so all variables set in your user shell environment are different.

You can check that by comparing the env command with both users: yours and by having it called by your java program, something like:

String[] textAnalysisPipeline = {"/bin/sh","-c","/usr/bin/env > /tmp/env.txt"}; //or wherever the 'env' command is in your system
Process process = Runtime.getRuntime().exec(textAnalysisPipeline);
...

And compare the content of the /tmp/env.txt with the env execution with your user… they are probably very different.

Look for the following variables:

• PATH
• CLASSPATH
• JAVA_HOME

I have already the same problem in the past. I suggest the following approach:

1. Use absolute path for everything, including the calls for “java”, “tee”, “tail” and libs (jar files)… you have in your command;

2. Change the environment configuration under which your Tomcat runs to reach all the applications you call in your command (usually by calling a script that configures all the necessary PATH variable (do not forget the JAVA_HOME and CLASSPATH to your jar files!). Check the Startup.sh and Catalina.sh for a suitable place to include your stuff;

3. Change your command to redirect the error output not to /dev/null as it is in your message, but to some log file somewhere in your system that your application can write (usually /tmp/exec.log is just fine) so you can be sure of the shell execution problem. I bet it will be something like: sh: ****: command not found or Error: Unable to access jarfile or some message of your application not being able to locate an object, so you will be sure of what application you call in your script is not in the PATH environment variable or what libs you don’t have in it… usually both

For additional info on this, check https://www.mulesoft.com/tcat/tomcat-classpath

Hope this helps…