Factor out complicated dockerfile steps into shell scripts

[zbeekman]

Some of the dockerfile steps are a little bit too clever. In particular the steps that leverage caching heavily and have any complex logic for what to do upon cache hit vs miss or during CI should have better abstraction and separation of concerns.

Biggest places for cleanup/improvement are:

• Git checkout of LLVM Project:

  salt-dev/Dockerfile

  Lines 30 to 74 in fdd1f77

  	RUN --mount=type=cache,target=/git <<EOC
  	echo "Checking out LLVM."
  	echo "\$CI = $CI"
  	# If the job is killed during a git operation the cache might be broken
  	if [ -f /git/llvm-project.git/index.lock ]; then
  	echo "index.lock file found--git repo might be in a broken state."
  	echo "Removing /git/llvm-project.git and forcing a new checkout!"
  	rm -rf /git/llvm-project.git
  	fi
  	if ${CI:-false}; then
  	# Github CI never seems to use the cached git directory :-[
  	echo "Running under CI. \$CI=$CI. Shallow cloning will be used if a clone is required."
  	# export SHALLOW='--depth=1'
  	fi
  	if mkdir llvm-project && git --git-dir=/git/llvm-project.git -C llvm-project pull origin release/14.x --ff-only
  	then
  	echo "WARNING: Using cached llvm git repository and pulling updates"
  	cp -r /git/llvm-project.git /llvm-project/.git
  	git -C /llvm-project reset --hard HEAD
  	else
  	echo "Cloning a fresh LLVM repository"
  	git clone --separate-git-dir=/git/llvm-project.git \
  	${SHALLOW:-} --single-branch \
  	--branch=release/14.x \
  	--filter=blob:none \
  	https://github.com/llvm/llvm-project.git
  	if [ -f /llvm-project/.git ]; then
  	rm -f /llvm-project/.git
  	fi
  	cp -r /git/llvm-project.git /llvm-project/.git
  	fi
  	cd llvm-project/llvm
  	mkdir build
  	ls -lad /git/llvm-project.git
  	ls -la /llvm-project/.git
  	if [ -f /llvm-project/.git ]; then
  	echo "Contents of .git: "
  	cat /llvm-project/.git
  	real_git_dir="$(cat /llvm-project/.git | cut -d ' ' -f2)"
  	echo "Contents of $real_git_dir :"
  	ls -lad "$real_git_dir"
  	echo "$real_git_dir"/*
  	fi
  	git status
  	EOC

• CMake configure and build of LLVM with ccache output/status checking:

  salt-dev/Dockerfile

  Lines 86 to 113 in fdd1f77

  	# Configure and build LLVM/Clang components needed by SALT
  	RUN --mount=type=cache,target=/ccache/ <<EOC
  	nproc --all || lscpu || true
  	pwd
  	if ! git -C /llvm-project status ; then
  	echo "llvm-project git repository missing or broken!!!"
  	exit 1
  	fi
  	ccache -s
  	cmake -GNinja \
  	-DCMAKE_INSTALL_PREFIX=/tmp/llvm \
  	-DCMAKE_MAKE_PROGRAM=/usr/local/bin/ninja \
  	-DCMAKE_BUILD_TYPE=Release \
  	-DLLVM_ENABLE_PROJECTS="clang;clang-tools-extra" \
  	-DLLVM_TARGETS_TO_BUILD=X86 \
  	-S /llvm-project/llvm -B /llvm-project/llvm/build
  	# Build libraries, headers, and binaries
  	# Do build
  	ccache -s
  	cd /llvm-project/llvm/build
  	# Actually do the build
  	ninja install-llvm-libraries install-llvm-headers \
  	install-clang-libraries install-clang-headers install-clang install-clang-cmake-exports \
  	install-clang-resource-headers install-llvm-config install-cmake-exports
  	ccache -s
  	git -C /llvm-project status
  	EOC

• Download, configure and install of TAU configurations:

  salt-dev/Dockerfile

  Lines 158 to 175 in fdd1f77

  	# Download and install TAU
  	# http://tau.uoregon.edu/tau.tgz
  	# http://fs.paratools.com/tau-mirror/tau.tgz
  	# http://fs.paratools.com/tau-nightly.tgz
  	RUN --mount=type=cache,target=/home/salt/ccache <<EOC
  	ccache -s
  	echo "verbose=off" > ~/.wgetrc
  	wget http://tau.uoregon.edu/tau.tgz || wget http://fs.paratools.com/tau-mirror/tau.tgz
  	tar xzvf tau.tgz
  	cd tau*
  	./installtau -prefix=/usr/local/ -cc=gcc -c++=g++\
  	-bfd=download -unwind=download -dwarf=download -pthread -iowrapper -j
  	./installtau -prefix=/usr/local/ -cc=clang -c++=clang++\
  	-bfd=download -unwind=download -dwarf=download -pthread -iowrapper -j
  	cd ..
  	rm -rf tau*
  	ccache -s
  	EOC