Different behavior on local test server (Ansible) vs. NLB test server

[bertfrees]

We noticed the other day that the server doesn't behave exactly as expected (Java logging messages not showing up), so then I tried to reproduce it on my local test server, which has exactly the same version of the Debian package installed, and there it worked. This could mean that a different environment causes the difference in behavior. I'm thinking it could be the Java version. I've noticed that using Java 8 can cause some bundles to fail.

It would help if we could replicate the NLB test server even more by using a more similar image, and also by moving more of the scripts we use to install, update, launch and stop to the Ansible project. Currently a lot of scripts are in test-server-hubot. @josteinaj What do you think?

Note: another problem is that the About page does not always display the correct version of the Pipeline server, so that I can never be 100% sure. In this case I'm pretty sure which version is installed, but still it's annoying. I have to fix this in the Debian package. @josteinaj Could you please send me the files that are currently in /etc/opt/daisy-pipeline2? Might help me with the debugging.

[josteinaj]

The engine version on the about page should be pulled directly from the engines web api, so if it's showing the wrong version then presumably the web api is stating the wrong version.

I wouldn't mind if the scripts were moved from test-server-hubot to the ansible project.

This is the java -version for the engine:

java version "1.7.0_79"
OpenJDK Runtime Environment (IcedTea 2.5.5) (7u79-2.5.5-0ubuntu0.12.04.1)
OpenJDK 64-Bit Server VM (build 24.79-b02, mixed mode)

And this is the java -version for the webui:

openjdk version "1.8.0_45-internal"
OpenJDK Runtime Environment (build 1.8.0_45-internal-b14)
OpenJDK 64-Bit Server VM (build 25.45-b02, mixed mode)

The server is running Ubuntu 12.04, which was the LTS at the time we initially set it up. I could spend some time upgrading it to a newer Ubuntu version if that would be useful, but I'm not sure it's worth it while we're doing sprints?

There's two files in /etc/opt/daisy-pipeline2 and there doesn't seem to be anything sensitive in them so I'm including them here:

config-logback.xml

<configuration debug="false">

  <!-- see http://logback.qos.ch/manual/configuration.html#LevelChangePropagator -->
  <contextListener class="ch.qos.logback.classic.jul.LevelChangePropagator"/>
    <appender name="JOB" class="org.daisy.pipeline.logging.IgnoreSiftAppender">
    <!-- in the absence of the class attribute, it is assumed that the
         desired discriminator type is
         ch.qos.logback.classic.sift.MDCBasedDiscriminator -->
    <discriminator>
      <key>jobid</key>
      <defaultValue>default</defaultValue>
    </discriminator>
    <sift>
      <appender name="FILE-${jobid}" class="ch.qos.logback.core.FileAppender">
        <file>${org.daisy.pipeline.iobase}/${jobid}/${jobid}.log</file>
        <append>false</append>
        <layout class="ch.qos.logback.classic.PatternLayout">
            <Pattern>%date [%-5level] %logger{36} - %msg%n</Pattern>
        </layout>
      </appender>
    </sift>
  </appender>

  <appender name="FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
    <filter class="org.daisy.pipeline.logging.OSGIFilter"/>   
    <file>${org.daisy.pipeline.logdir}/daisy-pipeline.log</file>
    <rollingPolicy class="ch.qos.logback.core.rolling.FixedWindowRollingPolicy">
      <fileNamePattern>${org.daisy.pipeline.logdir}/daisy-pipeline.%i.log.zip</fileNamePattern>
      <minIndex>1</minIndex>
      <maxIndex>4</maxIndex>
    </rollingPolicy>
    <triggeringPolicy class="ch.qos.logback.core.rolling.SizeBasedTriggeringPolicy">
      <maxFileSize>10MB</maxFileSize>
    </triggeringPolicy>

    <encoder>
      <Pattern>%date [%-5level] %logger{36} - %msg%n</Pattern>
    </encoder>
  </appender>

  <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
          <filter class="org.daisy.pipeline.logging.OSGIFilter"/>   
          <encoder>
                  <Pattern>%-5level %marker [%40.40logger{40}] %msg   @%class{36}:%L#%M%n</Pattern>
          </encoder>
  </appender>

  <root level="DEBUG">
    <appender-ref ref="FILE"/>
    <appender-ref ref="STDOUT"/>
    <appender-ref ref="JOB"/>
  </root>
  <logger name="com.xmlcalabash" level="INFO"/>
  <logger name="org.eclipse.jetty.util.log" level="INFO" />
  <logger name="cz.vutbr.web" level="WARN"/>
</configuration>

system.properties

#fwk version
org.daisy.pipeline.version=1.9.9-SNAPSHOT
#misc conf
derby.stream.error.file=/var/log/daisy-pipeline2/derby.log
logback.configurationFile=file:/etc/opt/daisy-pipeline2/config-logback.xml
org.daisy.pipeline.iobase=${org.daisy.pipeline.data}/jobs
org.daisy.pipeline.logdir=/var/log/daisy-pipeline2
#number of parallel workers
#org.daisy.pipeline.procs=2
org.daisy.pipeline.xproc.configuration=${org.daisy.pipeline.home}/etc/config-calabash.xml
# org.ops4j.pax.logging.DefaultServiceLog.level=WARN
# org.ops4j.pax.logging.service.frameworkEventsLogLevel=TRACE

# To load AWT in headless mode when running the Braille modules
java.awt.headless=true

#WS configuration
org.daisy.pipeline.ws.host=localhost
#allow local filesystem interaction
#org.daisy.pipeline.ws.localfs=true
#org.daisy.pipeline.ws.authentication=false
#org.daisy.pipeline.ws.authentication.key=clientid
#org.daisy.pipeline.ws.authentication.secret=supersecret

#SSL
#org.daisy.pipeline.ws.ssl=true
#org.daisy.pipeline.ws.ssl.keystore=${org.daisy.pipeline.home}/etc/keystore
#org.daisy.pipeline.ws.ssl.keystorepassword=password
#org.daisy.pipeline.ws.ssl.keypassword=password

#persistence properties
#org.daisy.pipeline.persistence.url=jdbc:mysql://localhost:3306/daisy_pipeline
#org.daisy.pipeline.persistence.user=root
#org.daisy.pipeline.persistence.password=pass

#calabash properties
com.xmlcalabash.config.user=""

[josteinaj]

Actually, I just checked, and the engine reports version 1.9.9-SNAPSHOT, while the about page says 1.9.7.

I created an issue for it here: daisy/pipeline-webui#55

[bertfrees]

Yeah, weird.

I have Trusty on my local test server, but I don't expect that to be an issue. Java version could be important, but I'm using exactly the same version. Must be something else but there's no real way to investigate for me :(

[josteinaj]

Is there any way we could make the environment completely reproducible? I suppose that would mean using Docker or something?

[bertfrees]

Yes that is one way I guess. It's not the ultimate solution (some things we can't eliminate such as state/data and coincidental circumstances, but it's probably the closest thing we'll ever get to it.

[bertfrees]

Maybe we should start with finding out what the issue is in this particular case, and when we've found out think about how (if) we can make sure we detect these kind of problems in a local test.

[bertfrees]

The Java logging messages seem to work again now.

[josteinaj]

Ok. Does that mean we can close this issue?

[bertfrees]

Didn't mean that. If we're not going to take further action, we can close the issue. On second thought, the Docker solution isn't really a solution. That's just "giving up". I think a good start would be to gradually move more and more stuff from test-server-hubot to the Ansible script. However I'm not completely sure which files we can move, which ones are specific to your server, and how the scripts are invoked. Also we should probably wait with this until the web UI DEB is included in the Ansible script.

[josteinaj]

Ok, sure.

Off the top of my head there's not much in test-server-hubot that's specific to the test server.

[bertfrees]

Regarding the version number issue: see also daisy/pipeline-assembly#97.