update documentation

installation, MacOS notes, and some FAQ items

Author: tjparnell

docs/AdvancedInstallation.md

@@ -20,9 +20,9 @@ accordingly for their system. If that doesn't describe you, skip ahead to the
     information. 
 
     Install [HTSlib](https://github.com/samtools/htslib) for Bam file support. You may 
-    already have it; later versions may (usually) work just as well, but version 1.9 
-    is officially recommended. It defaults to installing in `/usr/local` but doesn't 
-    have to be; see below. 
+    already have it; version 1.9 is officially recommended, but later versions appear
+    to work just fine and are preferred, as it includes Cram support. It defaults to 
+    installing in `/usr/local`. 
 
         curl -o htslib-1.19.tar.bz2 -L https://github.com/samtools/htslib/releases/download/1.19/htslib-1.19.tar.bz2 
         tar -xf htslib-1.19.tar.bz2
@@ -118,10 +118,7 @@ switch between releases with a simple command. It also manages multiple `local::
 installations, in case you want to isolate packages. 
 
 BioToolBox does not utilize threading (it uses forks for parallel execution), so if you 
-have a choice, compile a non-threaded Perl for a (very) slight performance gain. For 
-those adventurous to try, BioToolBox does work under [cperl](https://github.com/perl11/cperl), 
-although installing some prerequisite modules is a trying experience (many failed 
-tests and partial functionality).
+have a choice, compile a non-threaded Perl for a slight performance gain. 
 
 ### System installation
 
@@ -283,6 +280,10 @@ An example for downloading on Linux:
     do curl -o $HOME/bin/$name http://hgdownload.soe.ucsc.edu/admin/exe/linux.x86_64/$name \
     && chmod +x $HOME/bin/$name; done;
 
+**NOTE** Current versions of these utilities do not support directly piping data into
+the utility using the `stdin` file name. You will need to either find an older binary
+version, compile your own from older source code (see below), or update BioToolBox;
+version 2.02 now supports a work-around.
 
 ## Legacy Perl modules
 
@@ -314,14 +315,16 @@ multi-user installations).
 The Bio::DB::Sam library _only_ works with the legacy Samtools version, which
 included both the C libraries, headers, and executables; use version
 [0.1.19](https://github.com/samtools/samtools/archive/0.1.19.tar.gz) for best
-results. You will need to compile the Samtools code, but you do not have to
-install it (the library is not linked). Before compiling, edit the Makefile to
-include the cflags `-fPIC` and (most likely) `-m64` for 64 bit OS. Export the
-`SAMTOOLS` environment variable to the path of the Samtools build directory, and
-then you can proceed to build the Perl module; it should find the necessary
-files using the `SAMTOOLS` environment variable. You may obtain the latest
-source from
-[here](https://github.com/GMOD/GBrowse-Adaptors/tree/master/Bio-SamTools).
+results. You will need to compile the Samtools code, but you do not have to install
+it (the library is not linked). Before compiling, edit the Makefile to include the
+cflags `-fPIC` and (most likely) `-m64` for 64 bit OS. Export the `SAMTOOLS`
+environment variable to the path of the Samtools build directory, and then you can
+proceed to build the Perl module; it should find the necessary files using the
+`SAMTOOLS` environment variable. You may obtain the latest source from
+[Github](https://github.com/GMOD/GBrowse-Adaptors/tree/master/Bio-SamTools) or by
+downloading a [tarball](https://github.com/GMOD/GBrowse-Adaptors/tarball/master).
+**Note** that this project and file contains multiple Perl adapters and cannot be
+used directly with `cpanm`, for example.
 
 ### UCSC BigFile library
 
@@ -330,40 +333,91 @@ The Bio::DB::BigWig and Bio::DB::BigBed modules are part of the same distributio
 use the code from the GitHub repository, as it should be compatible with recent UCSC 
 libraries, whereas the distribution on CPAN is out of date. 
 
+**NOTE** The UCSC library, when it encounters an error, will immediately terminate
+the Perl process, with no chance of trapping the error. The newer `libBigWig` C
+library used with Bio::DB::Big (detailed above) does not exhibit this behavior, plus
+it's considerably easier to install. Encountering errors rarely happens, however,
+because all bioinformatic data is always perfectly formatted and well behaved, right?
+
 You will need the UCSC source code; the
 [userApps](http://hgdownload.soe.ucsc.edu/admin/exe/) source code is sufficient, rather
-than the entire browser code. Version 375, at the time of this writing, works. This
-requires at least `OpenSSL` and `libpng` libraries to compile the required library; on
-MacOS, these need to be installed independently (see [Homebrew](https://brew.sh) for
-example). There are other requirements, such as MySQL client libraries, that are needed if
-you want to compile the actual command line utilities, if so desired. 
-
-For purposes here, only the library needs to be compiled. It does not need to be 
-installed, as nothing is linked. Therefore, you can safely ignore the main `Makefile` 
-commands. Below are the steps for compiling just the requisite C library for installing 
-the Perl module.
-
-Edit the file `kent/src/inc/common.mk`, and insert `-fPIC` into the `CFLAGS` variable. If 
-you have installed any libraries in non-standard locations, e.g. `openssl` installed via 
-HomeBrew on MacOS, then add these paths to the `HG_INC` variable. Save the file. 
-
-To simplify compilation, you can skip the main Makefile and simply compile only the 
-libraries that you need. First, export the `MACHTYPE` environment variable to an 
-acceptable simple value, usually `x86_64`.
-
-Next, move to the included `kent/src/htslib` directory, and compile this library by
-issuing the `make` command.
-
-Move to the `kent/src/lib` directory, and compile the library by issuing the `make` 
-command. If it compiles successfully, you should get a `jkweb.a` file in the `lib/x86_64` 
-directory.
-
-Finally, you can return to the Perl module. First, set the `KENT_SRC` environment 
-variable to the full path of the `kent/src` build directory (otherwise you will need to 
-interactively provide the Perl module Build script this path). Then issue the standard 
-`Build.PL` commands to build, test, and install the Perl modules.
-
-
+than the entire browser code. Versions 375 and 398, at the time of this writing, works
+successfully, but more recent versions appear to have increasing problems with
+successful compilation – YMMV. 
+
+**NOTE** If you are compiling the command line utilities, such as `wigToBigWig`, be
+aware that in version 439 and later, these utilities no longer accept `stdin` as a
+file input. The Bio::ToolBox::big_helper module uses this feature for convenience in
+applications such [bam2wig](apps/bam2wig.md). You can compile your own following
+these steps, but you do not need to install Bio::DB::BigWig.
+
+This requires at least `OpenSSL` and `libpng` libraries to compile the required
+library. For the command line utilities, if desired, you will also need MySQL
+libraries; MariaDB, for now, seems adequate as far as I can tell.
+
+On Linux, this is mostly not a problem as these libraries and development files are
+readily available through the package manager. If you're building on macOS, see the
+notes in the [macOS notes page](MacOSNotes.md).
+
+For purposes of installing the Perl adapter, only the library needs to be compiled.
+It does not need to be installed, as nothing is linked. So, you do not need to run
+the full `make` command. In the `userApps` folder, run
+
+	cd path/to/userApps
+	make installEnvironment
+
+This will generate `kent/src/inc/localEnvironment.mk` for your local machine. Edit this
+file to add at the end
+
+	CFLAGS = -fPIC
+
+If you have odd or non-standard locations for some libraries, for example in a
+computing cluster where the development files are brought in using environment
+[modules](https://modules.readthedocs.io/), you may be able to set additional paths
+in this `localEnvironment.mk` file or by directly hacking `kent/src/inc/common.mk`.
+**NOTE** Be careful about setting a generic path to the libraries, particularly if
+you also have `htslib` installed, since the UCSC userApp provides its own (presumably
+modified) `htslib` library, which will conflict with a system available library.
+
+To proceed with library compilation, follow the following steps. Note exporting
+environment variables which aid in building the Perl adapter.
+
+	cd path/to/kent/src
+	export KENT_SRC=$(PWD)
+	cd htslib
+	make
+	cd ../lib
+	make
+	cd
+
+To install the Perl adapter, download the tarball from Github (same source as
+Bio::DB::Sam above) and follow the steps below.
+
+	curl -o GBrowse-Adaptors.tar.gz -L https://github.com/GMOD/GBrowse-Adaptors/tarball/master
+	tar -xf GBrowse-Adaptors.tar.gz
+	cd GBrowse-Adaptors-master-85c29de/Bio-BigFile
+	export MACHTYPE=local
+	perl Build.PL
+	./Build
+	./Build test
+	./Build install
+
+If the environment variables have been set correctly and the library compiled
+successfully, then the Perl build process should proceed smoothly. There may be
+various warnings emitted during the build process, which can usually be ignored.
+
+Once you have compiled the main `kent/src/lib` library, you can proceed with
+compiling the command line utilities, if you desire. You can build just the ones you
+want by going into each utility subdirectory within `kent/src/utils/` and issuing
+`make`. For example:
+
+	cd /path/to/userApps
+	mkdir bin
+	cd kent/src/utils/wigToBigWig
+	make
+
+The compiled executable should be copied into `userApps/bin`, and you can move it from
+there to wherever.
 
 
 

docs/FAQ.md

@@ -7,6 +7,35 @@
 These may or may not have actually been asked, but it's a collection of hints that the 
 programmer understands but a casual user might not, as well as rationale.
 
+- Certain UCSC utilities no longer support `stdin`.
+
+	Several of the UCSC command line utilities for big files (bigWig and bigBed in
+	particular) used to support a barely documented feature of using `stdin` or
+	`stdout` as filenames, but more recent versions do not. This feature was used
+	extensively by
+	[Bio::ToolBox::big_helper](https://metacpan.org/pod/Bio::ToolBox::big_helper)
+	when reading and/or writing big files as a way of offloading computational
+	processing, avoiding intermediate disk writes, and for the very practical reason
+	that the Perl adapters did not support direct writing of files. Version 2.02 of
+	Bio::ToolBox (finally) now checks the version of these utilities, and adjusts
+	behavior as necessary to either writing directly to the utility or intermediate
+	text files and then calling the utility.
+
+	If you find you would like to use the older utilities that support  `stdin`, at
+	least for `wigToBigWig` (`bedToBigBed` hasn't supported this for a very, very
+	long time), then you will need to compile your own. The change occurs in
+	`userApps.v439` release (October 2022), so you will need an earlier version. See
+	the section on UCSC library in the [Advanced Install](AdvancedInstallation.md)
+	document for hints on compiling the utilities (you don't need to install the
+	library).
+	
+- Do you support `CSV` files?
+
+	CSV files appear perfectly benign, but are in fact a can of worms: mandatory or
+	optional quoting, empty or undefined values, spaces, character escaping, text
+	encoding, and so on. This mostly affects reading files. Most (all?) bioinformatic
+	text formats are tab-delimited, so CSV support is intentionally absent.
+	
 - Programs don't recognize a UCSC gene table (refFlat, knownGene, genePred, etc)
 
 	UCSC doesn't have official file extensions, and their downloads page just 
@@ -128,11 +157,6 @@ programmer understands but a casual user might not, as well as rationale.
 	(BED and refFlat are fastest), or import them into an annotation SQLite database
 	file (see above).
 
-	There is an (unofficial) [effort](http://perl11.org) to make Perl faster by tweaking 
-	the internals. BioToolBox will install under [cperl](http://perl11.org/cperl/), 
-	although getting the prerequisites installed is not a trivial task. The speed gain 
-	is modest.
-
 - Why do you fork instead of using threads?
 
 	It's easier? Forking a child process is less complicated, as memory is automatically 

docs/MacOSNotes.md

@@ -17,6 +17,25 @@ need administrator permissions.
 
 	$ xcode-select --install
 
+## Install Homebrew
+
+If you're not already familiar with [Homebrew](https://brew.sh), you should be. This
+is a package manager for installing an ever-increasing number of open source
+utilities and packages, notably packages available on Linux but missing from
+(default) macOS. Importantly, it will install these with minimal effort on your part,
+as opposed to the old school method of downloading source code, configuring,
+compiling, and repeating again with required dependencies, all while debugging
+errors. It's the modern equivalent of MacPorts or Finks, for those old enough to
+remember those projects.
+
+Follow the instructions on the [Homebrew](https://brew.sh) website to install. On
+older Intel Macs, this will install under `/usr/local`, which makes integrating with
+other software relatively painless. However, under modern Apple Silicon Macs, it
+installs under `/opt/homebrew`, which can sometimes make finding libraries and
+development headers, especially when building older software projects, a bit more
+challenging.
+
+
 ## Installing your own Perl
 
 Apple is generally a little slow in updating their Perl compared to the latest available
@@ -63,6 +82,12 @@ for the source of the  solution.
 
 	$ install_name_tool -change libBigWig.so /path/to/lib/libBigWig.so blib/arch/auto/Bio/DB/Big/Big.bundle
 
+There are limitations to the length of the new path to be linked. For example, if you
+attempting to link to a buried Alien::Lib path in a local library, it may not work
+`because larger updated load commands do not fit` error messages. Your best bet is to
+install it in `/usr/local`; that almost always works.
+
+
 ## DB_File errors
 
 There are [reports of issues](https://github.com/bioperl/bioperl-live/issues/267) 
@@ -93,16 +118,18 @@ Set::IntervalTree, so force install ExtUtils::CppGuess and try re-installing
 Set::IntervalTree again – it will probably work.
 
 
-## libBigWig
+## Curl support in libBigWig
 
-When manually installing libBigWig on recent versions of macOS (observed with Sonoma,
-14.x and libBigWig v0.4.7), the compilation may fail at first. To check for libCurl
-dependencies, it attempts to compile a small test program and runs the command
-`mktemp --suffix=.c`. While that `--suffix` option is available to versions on Linux
-platforms, it is not available to the version on macOS, thus breaking the detection
-of libCurl. To work around this, we just have to tell it that, yes, we have libCurl.
-Edit the `Makefile` and comment out the five lines after 
-`# Create a simple test-program...` and add a new line
+When manually installing recent versions of
+[libBigWig](https://github.com/dpryan79/libBigWig) ,v 0.4.6 and later, the
+compilation may fail at first. To check for libcurl dependencies, it attempts to
+compile a small test program and runs the command `mktemp --suffix=.c`. While that
+`--suffix` option is available to versions on Linux platforms, it is not available to
+the version on macOS, thus breaking the detection of libcurl. 
+
+To work around this, we just have to tell it that, yes, we have libcurl (it's part of
+XCode). Edit the `Makefile` and comment out the five lines after `# Create a simple
+test-program...` and add a new line
 
 	HAVE_CURL=YES
 
@@ -115,7 +142,46 @@ its testing, albeit through a fake remote test with Test::Fake::HTTPD (if it's
 installed). However, empirical testing with real remote data (via https) seems to work ok.
 
 
+## Compiling UCSC library
+
+Compiling the UCSC library and/or utilities on macOS requires additional libraries,
+which are best installed using [Homebrew](https://brew.sh).
+
+	brew install libpng libssl
+
+If you're intending to compile the command line utilities, you will also need
+MySQL. For purposes here, MariaDB seems sufficient. You do not need to set up
+or configure a database.
+
+	brew install mariadb
+
+Generate the `localEnvironment.mk` file
+
+	cd path/to/userApps
+	make installEnvironment
+
+This will generate `kent/src/inc/localEnvironment.mk` for your local machine. If
+you are just building the library, edit this file to add at the end:
+
+	CFLAGS = -fPIC
+	L+=/opt/homebrew/lib/libssl.a
+	L+=/opt/homebrew/lib/libcrypto.a
+	PNGLIB=/opt/homebrew/lib/libpng.a
+	PNGINCL=-I/opt/homebrew/include
 
+However, if you are intending to build the command line utilities, your best bet is
+to simply hack `kent/src/inc/common.mk` file directly. This file performs a bunch of
+shenanigans to locate various libraries and set different compile options separately,
+_especially_ for MySQL, so there is no single variable you can simply define. Your
+best bet is to search for the `/opt/local` path for the following variables and
+change it to `/opt/homebrew`. An example of what you need to change is below:
 
+	L+=/opt/homebrew/lib/libssl.a
+	L+=/opt/homebrew/lib/libcrypto.a
+	PNGLIB=/opt/homebrew/lib/libpng.a
+	PNGINCL=-I/opt/homebrew/include
+	MYSQLINC=/opt/homebrew/include/mysql
+	MYSQLLIBS=/opt/homebrew/lib/libmysqlclient.a
 
+The libraries and utilities should then compile ok.