SSP-2436 Altering tests and code to be Windows friendly

[shawnconnolly]

• Leverage java.io.tmpdir system property instead of using /tmp
• Fix the windows delete file issue
• Fixed a couple of typos

[shawnconnolly]

This method was unused so it was removed.

[dmccallum]

Thanks, Shawn. This definitely improves the tests when run on a Windows env. I'm still having trouble getting them to actually complete, though. I ended up pushing a rebased copy of your commit plus an additional fix-up commit for the SQLServer test DDL files to add CREATEs for two missing tables: external_student_financial_aid_file and external_student_financial_aid_award_term. The commits are on a feature branch: https://github.com/Jasig/ssp-data-importer/tree/SSP-2436

At the moment, I'm getting this when testing on SqlServer for the db and Wins for the app... one failure from BatchInitializerHeaderFailTest and then ErrorReportingTest just seems to run forever. But only on this env (Postgres + OSX works fine). I don't really think these problems are caused by your PR, but there do seem to be ongoing problems with SqlServer/Windows tests, at least when using the DDL that ships with the tests to initialize the test db.

Also, it seems that when the job encounters a db-related error or basically any sort of problem in SqlServerStagingTableWriter, the job is throwing itself into what seems like an endless loop. E.g. my ErrorReportingTest has been running effectively forever with repeats of this nonsense:

16:16:09.077 [taskExecutor-1] INFO  o.j.s.u.i.j.s.SqlServerExternalTableUpsertWr
iter - ******UPSERT****** batch start:41753 batchstop:41753
16:16:09.077 [taskExecutor-1] ERROR o.j.s.u.i.j.l.StagingAndUpsertSkipListener -
 ERROR on Stage/Upsert Write
java.lang.IllegalArgumentException: null
        at java.sql.Date.valueOf(Date.java:138) ~[na:1.6.0_45]
        at net.sourceforge.jtds.jdbc.Support.convert(Support.java:502) ~[jtds-1.
2.4.jar:1.2.4]
        at net.sourceforge.jtds.jdbc.JtdsPreparedStatement.setObjectBase(JtdsPre
paredStatement.java:371) ~[jtds-1.2.4.jar:1.2.4]
        at net.sourceforge.jtds.jdbc.JtdsPreparedStatement.setObject(JtdsPrepare
dStatement.java:674) ~[jtds-1.2.4.jar:1.2.4]
        at org.apache.commons.dbcp.DelegatingPreparedStatement.setObject(Delegat
ingPreparedStatement.java:163) ~[commons-dbcp-1.2.2.jar:1.2.2]
        at org.springframework.jdbc.core.StatementCreatorUtils.setValue(Statemen
tCreatorUtils.java:315) ~[spring-jdbc-3.2.2.RELEASE.jar:3.2.2.RELEASE]
        at org.springframework.jdbc.core.StatementCreatorUtils.setParameterValue
Internal(StatementCreatorUtils.java:217) ~[spring-jdbc-3.2.2.RELEASE.jar:3.2.2.R
ELEASE]
        at org.springframework.jdbc.core.StatementCreatorUtils.setParameterValue
(StatementCreatorUtils.java:145) ~[spring-jdbc-3.2.2.RELEASE.jar:3.2.2.RELEASE]
        at org.jasig.ssp.util.importer.job.staging.SqlServerStagingTableWriter$1
.doInPreparedStatement(SqlServerStagingTableWriter.java:138) ~[classes/:na]
        at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:
589) ~[spring-jdbc-3.2.2.RELEASE.jar:3.2.2.RELEASE]
        at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:
617) ~[spring-jdbc-3.2.2.RELEASE.jar:3.2.2.RELEASE]
        at org.jasig.ssp.util.importer.job.staging.SqlServerStagingTableWriter.w
rite(SqlServerStagingTableWriter.java:119) ~[classes/:na]
        at sun.reflect.GeneratedMethodAccessor37.invoke(Unknown Source) ~[na:na]

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
sorImpl.java:25) ~[na:1.6.0_45]
        at java.lang.reflect.Method.invoke(Method.java:597) ~[na:1.6.0_45]
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflecti
on(AopUtils.java:317) ~[spring-aop-3.2.2.RELEASE.jar:3.2.2.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJo
inpoint(ReflectiveMethodInvocation.java:183) ~[spring-aop-3.2.2.RELEASE.jar:3.2.
2.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(
ReflectiveMethodInvocation.java:150) ~[spring-aop-3.2.2.RELEASE.jar:3.2.2.RELEAS
E]
        at org.springframework.aop.support.DelegatingIntroductionInterceptor.doP
roceed(DelegatingIntroductionInterceptor.java:132) ~[spring-aop-3.2.2.RELEASE.ja
r:3.2.2.RELEASE]
        at org.springframework.aop.support.DelegatingIntroductionInterceptor.inv
oke(DelegatingIntroductionInterceptor.java:120) ~[spring-aop-3.2.2.RELEASE.jar:3
.2.2.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(
ReflectiveMethodInvocation.java:172) ~[spring-aop-3.2.2.RELEASE.jar:3.2.2.RELEAS
E]
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynami
cAopProxy.java:204) ~[spring-aop-3.2.2.RELEASE.jar:3.2.2.RELEASE]
        at com.sun.proxy.$Proxy14.write(Unknown Source) ~[na:na]
        at org.springframework.batch.item.support.CompositeItemWriter.write(Comp
ositeItemWriter.java:51) ~[spring-batch-infrastructure-2.2.2.RELEASE.jar:na]
        at sun.reflect.GeneratedMethodAccessor37.invoke(Unknown Source) ~[na:na]

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
sorImpl.java:25) ~[na:1.6.0_45]
        at java.lang.reflect.Method.invoke(Method.java:597) ~[na:1.6.0_45]
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflecti
on(AopUtils.java:317) ~[spring-aop-3.2.2.RELEASE.jar:3.2.2.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJo
inpoint(ReflectiveMethodInvocation.java:183) ~[spring-aop-3.2.2.RELEASE.jar:3.2.
2.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(
ReflectiveMethodInvocation.java:150) ~[spring-aop-3.2.2.RELEASE.jar:3.2.2.RELEAS
E]
        at org.springframework.aop.support.DelegatingIntroductionInterceptor.doP
roceed(DelegatingIntroductionInterceptor.java:132) ~[spring-aop-3.2.2.RELEASE.ja
r:3.2.2.RELEASE]
        at org.springframework.aop.support.DelegatingIntroductionInterceptor.inv
oke(DelegatingIntroductionInterceptor.java:120) ~[spring-aop-3.2.2.RELEASE.jar:3
.2.2.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(
ReflectiveMethodInvocation.java:172) ~[spring-aop-3.2.2.RELEASE.jar:3.2.2.RELEAS
E]
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynami
cAopProxy.java:204) ~[spring-aop-3.2.2.RELEASE.jar:3.2.2.RELEASE]
        at com.sun.proxy.$Proxy27.write(Unknown Source) ~[na:na]
        at org.springframework.batch.core.step.item.SimpleChunkProcessor.writeIt
ems(SimpleChunkProcessor.java:175) [spring-batch-core-2.2.2.RELEASE.jar:na]
        at org.springframework.batch.core.step.item.FaultTolerantChunkProcessor.
scan(FaultTolerantChunkProcessor.java:580) ~[spring-batch-core-2.2.2.RELEASE.jar
:na]
        at org.springframework.batch.core.step.item.FaultTolerantChunkProcessor.
access$900(FaultTolerantChunkProcessor.java:50) ~[spring-batch-core-2.2.2.RELEAS
E.jar:na]
        at org.springframework.batch.core.step.item.FaultTolerantChunkProcessor$
5.recover(FaultTolerantChunkProcessor.java:413) ~[spring-batch-core-2.2.2.RELEAS
E.jar:na]
        at org.springframework.retry.support.RetryTemplate.handleRetryExhausted(
RetryTemplate.java:435) [spring-retry-1.0.2.RELEASE.jar:na]
        at org.springframework.retry.support.RetryTemplate.doExecute(RetryTempla
te.java:304) [spring-retry-1.0.2.RELEASE.jar:na]
        at org.springframework.retry.support.RetryTemplate.execute(RetryTemplate
.java:188) [spring-retry-1.0.2.RELEASE.jar:na]
        at org.springframework.batch.core.step.item.BatchRetryTemplate.execute(B
atchRetryTemplate.java:217) [spring-batch-core-2.2.2.RELEASE.jar:na]
        at org.springframework.batch.core.step.item.FaultTolerantChunkProcessor.
write(FaultTolerantChunkProcessor.java:423) ~[spring-batch-core-2.2.2.RELEASE.ja
r:na]
        at org.springframework.batch.core.step.item.SimpleChunkProcessor.process
(SimpleChunkProcessor.java:199) [spring-batch-core-2.2.2.RELEASE.jar:na]
        at org.springframework.batch.core.step.item.ChunkOrientedTasklet.execute
(ChunkOrientedTasklet.java:75) [spring-batch-core-2.2.2.RELEASE.jar:na]
        at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransact
ionCallback.doInTransaction(TaskletStep.java:395) [spring-batch-core-2.2.2.RELEA
SE.jar:na]
        at org.springframework.transaction.support.TransactionTemplate.execute(T
ransactionTemplate.java:130) [spring-tx-3.2.0.RELEASE.jar:3.2.0.RELEASE]
        at org.springframework.batch.core.step.tasklet.TaskletStep$2.doInChunkCo
ntext(TaskletStep.java:267) [spring-batch-core-2.2.2.RELEASE.jar:na]
        at org.springframework.batch.core.scope.context.StepContextRepeatCallbac
k.doInIteration(StepContextRepeatCallback.java:77) [spring-batch-core-2.2.2.RELE
ASE.jar:na]
        at org.springframework.batch.repeat.support.RepeatTemplate.getNextResult
(RepeatTemplate.java:368) [spring-batch-infrastructure-2.2.2.RELEASE.jar:na]
        at org.springframework.batch.repeat.support.RepeatTemplate.executeIntern
al(RepeatTemplate.java:215) [spring-batch-infrastructure-2.2.2.RELEASE.jar:na]
        at org.springframework.batch.repeat.support.RepeatTemplate.iterate(Repea
tTemplate.java:144) [spring-batch-infrastructure-2.2.2.RELEASE.jar:na]
        at org.springframework.batch.core.step.tasklet.TaskletStep.doExecute(Tas
kletStep.java:253) [spring-batch-core-2.2.2.RELEASE.jar:na]
        at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep
.java:195) [spring-batch-core-2.2.2.RELEASE.jar:na]
        at org.springframework.batch.core.partition.support.TaskExecutorPartitio
nHandler$1.call(TaskExecutorPartitionHandler.java:139) [spring-batch-core-2.2.2.
RELEASE.jar:na]
        at org.springframework.batch.core.partition.support.TaskExecutorPartitio
nHandler$1.call(TaskExecutorPartitionHandler.java:136) [spring-batch-core-2.2.2.
RELEASE.jar:na]
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) [n
a:1.6.0_45]
        at java.util.concurrent.FutureTask.run(FutureTask.java:138) [na:1.6.0_45
]
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExec
utor.java:895) [na:1.6.0_45]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor
.java:918) [na:1.6.0_45]
        at java.lang.Thread.run(Thread.java:662) [na:1.6.0_45]
16:16:09.077 [taskExecutor-1] INFO  Q.c.j.s.u.i.j.s.SqlServerExternalTableUpsert
Writer -  MERGE INTO external_person as target USING stg_external_person as sour
ce ON (target.school_id = source.school_id ) WHEN NOT MATCHED AND source.batch_i
d >= 41754 and source.batch_id <= 41754 THEN INSERT (school_id,username,first_na
me,middle_name,last_name,birth_date,primary_email_address,address_line_1,address
_line_2,city,state,zip_code,home_phone,work_phone,office_location,office_hours,d
epartment_name,actual_start_term,actual_start_year,marital_status,ethnicity,gend
er,is_local,balance_owed,coach_school_id,cell_phone,photo_url) VALUES ( source.s
chool_id,source.username,source.first_name,source.middle_name,source.last_name,s
ource.birth_date,source.primary_email_address,source.address_line_1,source.addre
ss_line_2,source.city,source.state,source.zip_code,source.home_phone,source.work
_phone,source.office_location,source.office_hours,source.department_name,source.
actual_start_term,source.actual_start_year,source.marital_status,source.ethnicit
y,source.gender,source.is_local,source.balance_owed,source.coach_school_id,sourc
e.cell_phone,source.photo_url) WHEN MATCHED AND source.batch_id >= 41754 and sou
rce.batch_id <= 41754 THEN UPDATE SET target.school_id=source.school_id,target.u
sername=source.username,target.first_name=source.first_name,target.middle_name=s
ource.middle_name,target.last_name=source.last_name,target.birth_date=source.bir
th_date,target.primary_email_address=source.primary_email_address,target.address
_line_1=source.address_line_1,target.address_line_2=source.address_line_2,target
.city=source.city,target.state=source.state,target.zip_code=source.zip_code,targ
et.home_phone=source.home_phone,target.work_phone=source.work_phone,target.offic
e_location=source.office_location,target.office_hours=source.office_hours,target
.department_name=source.department_name,target.actual_start_term=source.actual_s
tart_term,target.actual_start_year=source.actual_start_year,target.marital_statu
s=source.marital_status,target.ethnicity=source.ethnicity,target.gender=source.g
ender,target.is_local=source.is_local,target.balance_owed=source.balance_owed,ta
rget.coach_school_id=source.coach_school_id,target.cell_phone=source.cell_phone,
target.photo_url=source.photo_url;

The batch index keeps creeping up... here it's up to 41754, but if I'm reading the test right, there's only 3 total rows in the external_person.csv file being tested. I get the same endless cycle of errors when running the test in isolation.

So... mainly I'm just wondering if you're getting any of the same behaviors in your environment. If so, I think we've probably got at least two things to do. Not sure if they can actually be separated, but...

1. Try to fix up the tests as best we can for this PR/ticket w/r/t any SQLServer-specific problems in test data/config/impl, then
2. Fix whatever allows the endless loops: https://issues.jasig.org/browse/SSP-2476