Merge pull request #333 from sshanks-kx/refactor

refactor

Author: vcalescu

docs/basics/internal.md

@@ -17,8 +17,8 @@ The operator `!` with a negative integer as left argument calls an internal func
 [-10!x](#-10x-type-enum)       type enum                     -5!   [parse](../ref/parse.md)
 [-11!](#-11-streaming-execute)        streaming execute             -6!   [eval](../ref/eval.md)
 [-14!x](#-14x-quote-escape)       quote escape                  -7!   [hcount](../ref/hcount.md)
-[-16!x](#-16x-ref-count)       ref count                     -12!  [.Q.host](../ref/dotq.md#host-hostname)
-[-18!x](#-18x-compress-byte)       compress byte                 -13!  [.Q.addr](../ref/dotq.md#addr-ip-address)
+[-16!x](#-16x-ref-count)       ref count                     -12!  [.Q.host](../ref/dotq.md#host-ip-to-hostname)
+[-18!x](#-18x-compress-byte)       compress byte                 -13!  [.Q.addr](../ref/dotq.md#addr-iphost-as-int)
 [-21!x](#-21x-compressionencryption-stats)       compression/encryption stats  -15!  [md5](../ref/md5.md)
 [-22!x](#-22x-uncompressed-length)       uncompressed length           -19!  [set](../ref/get.md#set)
 [-23!x](#-23x-memory-map)       memory map                    -20!  [.Q.gc](../ref/dotq.md#gc-garbage-collect)
@@ -359,7 +359,7 @@ q)-27!(3i;0 1+123456789.4567)
 "123456790.457"
 ```
 
-This is a more precise, built-in version of [`.Q.f`](../ref/dotq.md#f-format) but uses IEEE754 rounding:
+This is a more precise, built-in version of [`.Q.f`](../ref/dotq.md#f-precision-format) but uses IEEE754 rounding:
 
 ```q
 q).045

docs/basics/ipc.md

@@ -398,8 +398,8 @@ The compression/decompression algorithms are proprietary and implemented as the
 <br>
 [`.z` namespace](../ref/dotz.md) for callback functions
 <br> 
-[`.Q.addr`](../ref/dotq.md#addr-ip-address) (IP address), 
-[`.Q.host`](../ref/dotq.md#host-hostname) (hostname), 
+[`.Q.addr`](../ref/dotq.md#addr-iphost-as-int) (IP/host as int), 
+[`.Q.host`](../ref/dotq.md#host-ip-to-hostname) (IP to hostname), 
 <br>
 :fontawesome-solid-book-open:
 [Connection handles](handles.md)

docs/basics/syscmds.md

@@ -530,8 +530,8 @@ q)1%3
     ```
 
 :fontawesome-solid-book:
-[`.Q.f`](../ref/dotq.md#f-format),
-[`.Q.fmt`](../ref/dotq.md#fmt-format)
+[`.Q.f`](../ref/dotq.md#f-precision-format) (precision format),
+[`.Q.fmt`](../ref/dotq.md#fmt-precision-format) (precision format with length)
 <br>
 :fontawesome-solid-book-open:
 [Precision](precision.md),

docs/cloud/aws-lambda/index.md

@@ -335,7 +335,7 @@ From the response payload we see the function was successful and calculated the
 
 ## Stream data from Amazon S3
 
-To demonstrate a q/kdb+ Lambda function processing multiple events, we detail how to stream data from AWS Simple Storage Service (S3). Using FIFO named pipes and [`.Q.fps`](../../ref/dotq.md#fps-streaming-algorithm) within q, data can be streamed in for processing. To illustrate this example, we create 100 files each containing 1 million Black-Scholes input parameters. The files are placed in a S3 bucket. This S3 bucket is the trigger for the Lambda function.
+To demonstrate a q/kdb+ Lambda function processing multiple events, we detail how to stream data from AWS Simple Storage Service (S3). Using FIFO named pipes and [`.Q.fps`](../../ref/dotq.md#fps-pipe-streaming) (pipe streaming) within q, data can be streamed in for processing. To illustrate this example, we create 100 files each containing 1 million Black-Scholes input parameters. The files are placed in a S3 bucket. This S3 bucket is the trigger for the Lambda function.
 
 :fontawesome-brands-aws:
 [Configuring Amazon S3 Event Notifications](https://docs.aws.amazon.com/AmazonS3/latest/dev/NotificationHowTo.html)
@@ -365,7 +365,7 @@ The steps in the `process_s3data.q` code are as follows.
 1.  Call S3 function from `Stream_Data` script, initiate FIFO pipe and stream in S3 data.
 1.  Load `blackScholes.q`.
 1.  Create inputs table to store input parameters.
-1.  Use [`.Q.fps`](../../ref/dotq.md#fps-streaming-algorithm) to stream in the S3 data from the FIFO `pipe_stream` to inputs table.
+1.  Use [`.Q.fps`](../../ref/dotq.md#fps-pipe-streaming) to stream in the S3 data from the FIFO `pipe_stream` to inputs table.
 1.  Use [`.Q.fu`](../../ref/dotq.md#fu-apply-unique) to run the inputs through the `blackScholes` formula.
 1.  `black_scholes_data` contains the input parameters and the calculated option prices.
 

docs/github.md

@@ -117,6 +117,13 @@ GitHub topic queries:  
 </td>
 </tr>
 <tr markdown>
+<td markdown class="nowrap"><a href="https://github.com/lukebrit27/surv-cloud">surv-cloud</a></td>
+<td markdown>Small market surveillance application for cloud/kubernetes. <span class="author"><a href="https://github.com/lukebrit27?tab=repositories">
+                        (Luke Britton)
+                    </a></span>
+</td>
+</tr>
+<tr markdown>
 <td markdown class="nowrap"><a href="https://github.com/simongarland/tickrecover">tickrecover</a></td>
 <td markdown>Recover from tickerplant crash. <span class="author"><a href="https://github.com/simongarland?tab=repositories">
 						(Simon Garland)

docs/interfaces/q-server-for-odbc3.md

@@ -70,6 +70,8 @@ Ensure you have `ps.k` loaded into the kdb+ process specified in your DSN:
 q)\l ps.k
 ```
 
+The kdb+ process should also be [listening on port](../basics/listening-port.md) which relates to the port choosen and defined in the odbc configuration.
+
 
 ## Notes
 

docs/kb/loading-from-large-files.md

@@ -13,7 +13,7 @@ The [Load CSV](../ref/file-text.md#load-csv) form of the File Text operator load
 
 If the data in the CSV file is too large to fit into memory, we need to break the large CSV file into manageable chunks and process them in sequence. 
 
-Function [`.Q.fs`](../ref/dotq.md#fs-streaming-algorithm) and its variants help automate this process. `.Q.fs` loops over a file in conveniently-sized chunks of complete records, and applies a function to each chunk. This lets you implement a _streaming algorithm_ to convert a large CSV file into an on-disk database without holding all the data in memory at once.
+Function [`.Q.fs`](../ref/dotq.md#fs-file-streaming) (file streaming) and its variants help automate this process. `.Q.fs` loops over a file in conveniently-sized chunks of complete records, and applies a function to each chunk. This lets you implement a _streaming algorithm_ to convert a large CSV file into an on-disk database without holding all the data in memory at once.
 
 
 ## Using `.Q.fs`
@@ -96,11 +96,11 @@ date       open  high  low   close volume   sym
 Variants of `.Q.fs` extend it to [named pipes](named-pipes.md) and control chunk size.
 
 :fontawesome-solid-book:
-[`.Q.fsn`](../ref/dotq.md#fsn-streaming-algorithm) for chunk size
+[`.Q.fsn`](../ref/dotq.md#fsn-file-streaming) for chunk size
 <br>
 :fontawesome-solid-book:
-[`.Q.fps`](../ref/dotq.md#fps-streaming-algorithm),
-[`.Q.fpn`](../ref/dotq.md#fpn-streaming-algorithm) for named pipes
+[`.Q.fps`](../ref/dotq.md#fps-pipe-streaming),
+[`.Q.fpn`](../ref/dotq.md#fpn-pipe-streaming) for named pipes
 
 <!-- 
 To write to a partitioned database, some utility functions generalizing [`.Q.dpft`](../ref/dotq.md#dpft-save-table) are useful.
@@ -163,7 +163,7 @@ Unfortunately it is not always possible or is too expensive to structure the inp
 
 We must
 
--   read data in chunks using [`.Q.fsn`](../ref/dotq.md#fsn-streaming-algorithm)
+-   read data in chunks using [`.Q.fsn`](../ref/dotq.md#fsn-file-streaming)
 -   append data to splayed tables using manual enumerations and [`upsert`](../basics/qsql.md#upsert)
 -   re-sort and set attributes on disk when all the data is loaded
 -   write a daily statistics table as a splayed table at the top level of the database
@@ -444,7 +444,7 @@ openssl enc -aes-256-cbc -d –k password -in trades.csv.dat > named_pipe &
 
 :fontawesome-solid-book:
 [`set`](../ref/get.md#set), 
-[`.Q.fps`](../ref/dotq.md#fps-streaming-algorithm)
+[`.Q.fps`](../ref/dotq.md#fps-pipe-streaming) (pipe streaming)
 <br>
 :fontawesome-solid-laptop:
 [Named pipes](../kb/named-pipes.md)

docs/kb/named-pipes.md

@@ -5,8 +5,7 @@ keywords: fifo, kdb+, named, pipe, q, unix
 ---
 # Named pipes
 
-
-
+## Overview
 
 Since V3.4 it has been possible to read FIFOs/named pipes on Unix.
 
@@ -19,24 +18,69 @@ q)/ At most, n bytes will be read, perhaps fewer
 q)hclose h             / Close the file to clean up
 ```
 
-[`.Q.fps`](../ref/dotq.md#fps-streaming-algorithm "streaming algorithm") is [`.Q.fs`](../ref/dotq.md#fs-streaming-algorithm "streaming algorithm") for pipes. 
-(`.Q.fpn` corresponds to [`.Q.fsn`](../ref/dotq.md#fsn-streaming-algorithm "streaming algorithm").) 
+A `` `:fifo://`` handle is also useful for reading certain non-seekable or zero-length (therefore, unsuitable for the regular `read1`) system files or devices, e.g.
+
+```q
+q)a:hopen`:fifo:///dev/urandom
+q)read1 (a;8)
+0x8f172b7ea00b85e6
+q)hclose a
+```
+
+## Streaming
+
+[`.Q.fps`](../ref/dotq.md#fps-pipe-streaming) and [`.Q.fpn`](../ref/dotq.md#fpn-pipe-streaming) provide the ability to streaming data from a fifo/named pipe.
 
-The following example loads a CSV via FIFO, avoiding decompressing to disk:
+This can be useful for various applications, such as streaming data in from a compressed file without having to decompress the contents to disk.
+
+For example, using a csv file (t.csv) with the contents
+```csv
+MSFT,12:01:10.000,A,O,300,55.60
+APPL,12:01:20.000,B,O,500,67.70
+IBM,12:01:20.100,A,O,100,61.11
+MSFT,12:01:10.100,A,O,300,55.60
+APPL,12:01:20.100,B,O,500,67.70
+IBM,12:01:20.200,A,O,100,61.11
+MSFT,12:01:10.200,A,O,300,55.60
+APPL,12:01:20.200,B,O,500,67.70
+IBM,12:01:20.200,A,O,100,61.11
+MSFT,12:01:10.300,A,O,300,55.60
+APPL,12:01:20.400,B,O,500,67.70
+IBM,12:01:20.500,A,O,100,61.11
+MSFT,12:01:10.500,A,O,300,55.60
+APPL,12:01:20.600,B,O,500,67.70
+IBM,12:01:20.600,A,O,100,61.11
+MSFT,12:01:10.700,A,O,300,55.60
+APPL,12:01:20.700,B,O,500,67.70
+IBM,12:01:20.800,A,O,100,61.11
+MSFT,12:01:10.900,A,O,300,55.60
+APPL,12:01:20.900,B,O,500,67.70
+IBM,12:01:20.990,A,O,100,61.11
+```
+
+If the file is compressed into a ZIP archive (t.zip), the system command `unzip` has the option to uncompress to stdout, which can be combined with a `fifo`.
+The following loads the CSV file through a FIFO without having the intermediary step of creating the unzipped file:
 
 ```q
 q)system"rm -f fifo && mkfifo fifo"
-q)system"unzip -p t.zip t.csv > fifo &"
 q)trade:flip `sym`time`ex`cond`size`price!"STCCFF"$\:()
+q)system"unzip -p t.zip > fifo &"
 q).Q.fps[{`trade insert ("STCCFF";",")0:x}]`:fifo
+q)trade
 ```
 
-A `` `:fifo://`` handle is also useful for reading certain non-seekable or zero-length (therefore, unsuitable for the regular `read1`) system files or devices, e.g.
+Alternatively, if the file was compressed using gzip (t.gz), the system command `gunzip` can be used:
 
 ```q
-q)a:hopen`:fifo:///dev/urandom
-q)read1 (a;8)
-0x8f172b7ea00b85e6
-q)hclose a
+q)system"rm -f fifo && mkfifo fifo"
+q)trade:flip `sym`time`ex`cond`size`price!"STCCFF"$\:()
+q)system"gunzip -cf t.gz > fifo &"
+q).Q.fps[{`trade insert ("STCCFF";",")0:x}]`:fifo
+q)trade
 ```
 
+:fontawesome-regular-hand-point-right:
+[`0:`](../ref/file-text.md#load-csv) (load csv)<br>
+:fontawesome-solid-book-open:
+[mkfifo](https://linux.die.net/man/1/mkfifo),
+[unzip](https://linux.die.net/man/1/unzip), [gunzip](https://linux.die.net/man/1/unzip)

docs/kb/using-dotz.md

@@ -430,3 +430,4 @@ q)
 -   [Changes in V2.4](../releases/ChangesIn2.4.md)
 -   _Q for Mortals_: [§11.6 Interprocess Communication](/q4m3/11_IO/#116-interprocess-communication)
 -   [Authentication and Authorization](../basics/ipc.md#authentication-authorization)
+-   [Permissions with kdb+](../wp/permissions/index.md)

docs/ref/doth.md

@@ -94,7 +94,7 @@ q).h.cd (`a`b`c;1 2 3;"xyz")
 Columns can be nested vectors, in which case [`.h.d`](#hd-delimiter) is used to separate subitems. (Since V4.0 2020.03.17.)
 
 :fontawesome-regular-hand-point-right:
-[0: load csv](file-text.md#load-csv)
+[0: load csv](file-text.md#load-csv), [save](save.md#save) (save and format data)
 
 
 ## `.h.code` (code after Tab)
@@ -161,6 +161,8 @@ q).h.ed ([]a:1 2 3;b:`x`y`z)
 "<Workbook xmlns=\"urn:schemas-microsoft-com:office:spreadsheet\" xmlns:o=\"u..
 ```
 
+:fontawesome-regular-hand-point-right:
+[save](save.md#save) (save and format data)
 
 ## `.h.edsn` (Excel from tables)
 
@@ -203,7 +205,8 @@ q)`:/Users/sjt/tmp/excel.xls 0: .h.edsn `test1`test2!(t1;t2)
 
 ![excel.xls](../img/h.edsn.png "Excel spreadsheet")
 
-
+:fontawesome-regular-hand-point-right:
+[save](save.md#save) (save and format data)
 
 ## `.h.fram` (frame)
 
@@ -819,6 +822,9 @@ q).h.xd ([]a:1 2 3;b:`x`y`z)
 "</R>"
 ```
 
+:fontawesome-regular-hand-point-right:
+[save](save.md#save) (save and format data)
+
 
 ## `.h.xmp` (XMP)
 
@@ -865,4 +871,4 @@ foo| "bar"
 ```
 
 :fontawesome-regular-hand-point-right:
-[`.j` namespace](dotj.md) (JSON de/serialization)
+[`.j` namespace](dotj.md) (JSON de/serialization), [save](save.md#save) (save and format data)