v0.3.3

Author: shenwei356

doc/docs/download.md

@@ -6,31 +6,30 @@ SeqKit is implemented in [Golang](https://golang.org/) programming language,
 
 ## Latest Version
 
-[SeqKit v0.3.2](https://github.com/shenwei356/seqkit/releases/tag/v0.3.2)
+[SeqKit v0.3.3](https://github.com/shenwei356/seqkit/releases/tag/v0.3.3)
 
 
 ***64-bit versions are highly recommended.***
 
 ### Links
 
-
 - **Linux**
-    - [seqkit_linux_386.tar.gz](https://github.com/shenwei356/seqkit/releases/download/v0.3.2/seqkit_linux_386.tar.gz)
-    - [seqkit_linux_amd64.tar.gz](https://github.com/shenwei356/seqkit/releases/download/v0.3.2/seqkit_linux_amd64.tar.gz)
-    - [seqkit_linux_arm.tar.gz](https://github.com/shenwei356/seqkit/releases/download/v0.3.2/seqkit_linux_arm.tar.gz)
+    - [seqkit_linux_386.tar.gz](https://github.com/shenwei356/seqkit/releases/download/v0.3.3/seqkit_linux_386.tar.gz)
+    - [seqkit_linux_amd64.tar.gz](https://github.com/shenwei356/seqkit/releases/download/v0.3.3/seqkit_linux_amd64.tar.gz)
+    - [seqkit_linux_arm.tar.gz](https://github.com/shenwei356/seqkit/releases/download/v0.3.3/seqkit_linux_arm.tar.gz)
 - **Mac OS X**
-    - [seqkit_darwin_386.tar.gz](https://github.com/shenwei356/seqkit/releases/download/v0.3.2/seqkit_darwin_386.tar.gz)
-    - [seqkit_darwin_amd64.tar.gz](https://github.com/shenwei356/seqkit/releases/download/v0.3.2/seqkit_darwin_amd64.tar.gz)
+    - [seqkit_darwin_386.tar.gz](https://github.com/shenwei356/seqkit/releases/download/v0.3.3/seqkit_darwin_386.tar.gz)
+    - [seqkit_darwin_amd64.tar.gz](https://github.com/shenwei356/seqkit/releases/download/v0.3.3/seqkit_darwin_amd64.tar.gz)
 - **Windows**
-    - [seqkit_windows_386.exe.tar.gz](https://github.com/shenwei356/seqkit/releases/download/v0.3.2/seqkit_windows_386.exe.tar.gz)
-    - [seqkit_windows_amd64.exe.tar.gz](https://github.com/shenwei356/seqkit/releases/download/v0.3.2/seqkit_windows_amd64.exe.tar.gz)
+    - [seqkit_windows_386.exe.tar.gz](https://github.com/shenwei356/seqkit/releases/download/v0.3.3/seqkit_windows_386.exe.tar.gz)
+    - [seqkit_windows_amd64.exe.tar.gz](https://github.com/shenwei356/seqkit/releases/download/v0.3.3/seqkit_windows_amd64.exe.tar.gz)
 - **FreeBSD**
-    - [seqkit_freebsd_386.tar.gz](https://github.com/shenwei356/seqkit/releases/download/v0.3.2/seqkit_freebsd_386.tar.gz)
-    - [seqkit_freebsd_amd64.tar.gz](https://github.com/shenwei356/seqkit/releases/download/v0.3.2/seqkit_freebsd_amd64.tar.gz)
-    - [seqkit_freebsd_arm.tar.gz](https://github.com/shenwei356/seqkit/releases/download/v0.3.2/seqkit_freebsd_arm.tar.gz)
+    - [seqkit_freebsd_386.tar.gz](https://github.com/shenwei356/seqkit/releases/download/v0.3.3/seqkit_freebsd_386.tar.gz)
+    - [seqkit_freebsd_amd64.tar.gz](https://github.com/shenwei356/seqkit/releases/download/v0.3.3/seqkit_freebsd_amd64.tar.gz)
+    - [seqkit_freebsd_arm.tar.gz](https://github.com/shenwei356/seqkit/releases/download/v0.3.3/seqkit_freebsd_arm.tar.gz)
 - **OpenBSD**
-    - [seqkit_openbsd_386.tar.gz](https://github.com/shenwei356/seqkit/releases/download/v0.3.2/seqkit_openbsd_386.tar.gz)
-    - [seqkit_openbsd_amd64.tar.gz](https://github.com/shenwei356/seqkit/releases/download/v0.3.2/seqkit_openbsd_amd64.tar.gz)
+    - [seqkit_openbsd_386.tar.gz](https://github.com/shenwei356/seqkit/releases/download/v0.3.3/seqkit_openbsd_386.tar.gz)
+    - [seqkit_openbsd_amd64.tar.gz](https://github.com/shenwei356/seqkit/releases/download/v0.3.3/seqkit_openbsd_amd64.tar.gz)
 
 ### Mirror site for Chinese user
 
@@ -78,6 +77,11 @@ For Go developer, just one command:
 
 ## Release History
 
+- [SeqKit v0.3.3](https://github.com/shenwei356/seqkit/releases/tag/v0.3.3)
+    - fix bug of `seqkit replace`, wrongly starting from 2 when using `{nr}`
+      in `-r` (`--replacement`)
+    - new feature: `seqkit replace` supports replacement symbols `{nr}` (record number)
+      and `{kv}` (corresponding value of the key ($1) by key-value file)
 - [SeqKit v0.3.2](https://github.com/shenwei356/seqkit/releases/tag/v0.3.2)
     - fix bug of `seqkit split`, error when target file is in a directory.
     - improve performance of `seqkit spliding` for big sequences, and output

doc/docs/usage.md

@@ -99,7 +99,7 @@ Usage
 ```
 SeqKit -- a cross-platform and ultrafast toolkit for FASTA/Q file manipulation
 
-Version: 0.3.1.1
+Version: 0.3.3
 
 Author: Wei Shen <[email protected]>
 
@@ -1022,7 +1022,7 @@ Examples
 Usage
 
 ```
-replace name/sequence/by regular expression.
+replace name/sequence by regular expression.
 
 Note that the replacement supports capture variables.
 e.g. $1 represents the text of the first submatch.
@@ -1038,14 +1038,20 @@ Or use the \ escape character.
 
 more on: http://shenwei356.github.io/seqkit/usage/#replace
 
+Special repalcement symbols:
+
+        {nr}    Record number, starting from 1
+        {kv}    Corresponding value of the key ($1) by key-value file
+
 Usage:
   seqkit replace [flags]
 
 Flags:
   -s, --by-seq               replace seq
   -i, --ignore-case          ignore case
+  -k, --kv-file string       tab-delimited key-value file for replacing key with value when using "{kv}" in -r (--replacement)
   -p, --pattern string       search regular expression
-  -r, --replacement string   replacement. supporting capture variables.  e.g. $1 represents the text of the first submatch. ATTENTION: use SINGLE quote NOT double quotes in *nix OS or use the \ escape character. record number is also supported by "{NR}"
+  -r, --replacement string   replacement. supporting capture variables.  e.g. $1 represents the text of the first submatch. ATTENTION: use SINGLE quote NOT double quotes in *nix OS or use the \ escape character. Record number is also supported by "{nr}"
 
 ```
 

doc/site

@@ -1 +1 @@
-Subproject commit 782503376f2ab54df795036d02c9dadfab6155f7
+Subproject commit 463f2ef8617e0d1ebfcc764124b8e70bf5f76ebf

seqkit/cmd/helper.go

@@ -43,7 +43,7 @@ import (
 )
 
 // VERSION of seqkit
-const VERSION = "0.3.2"
+const VERSION = "0.3.3"
 
 func checkError(err error) {
 	if err != nil {
@@ -378,3 +378,34 @@ var bufferedByteSliceWrapper *byteutil.BufferedByteSliceWrapper
 // func init() {
 // 	bufferedByteSliceWrapper = byteutil.NewBufferedByteSliceWrapper(1, defaultBytesBufferSize)
 // }
+
+func readKVs(file string) (map[string]string, error) {
+	type KV [2]string
+	fn := func(line string) (interface{}, bool, error) {
+		if len(line) == 0 {
+			return nil, false, nil
+		}
+		items := strings.Split(strings.TrimRight(line, "\r\n"), "\t")
+		if len(items) < 2 {
+			return nil, false, nil
+		}
+
+		return KV([2]string{items[0], items[1]}), true, nil
+	}
+	kvs := make(map[string]string)
+	reader, err := breader.NewBufferedReader(file, 2, 10, fn)
+	if err != nil {
+		return kvs, err
+	}
+	var items KV
+	for chunk := range reader.Ch {
+		if chunk.Err != nil {
+			return kvs, err
+		}
+		for _, data := range chunk.Data {
+			items = data.(KV)
+			kvs[items[0]] = items[1]
+		}
+	}
+	return kvs, nil
+}

seqkit/cmd/replace.go

@@ -26,6 +26,7 @@ import (
 	"regexp"
 	"runtime"
 	"strconv"
+	"strings"
 
 	"github.com/shenwei356/bio/seq"
 	"github.com/shenwei356/bio/seqio/fastx"
@@ -53,6 +54,11 @@ Or use the \ escape character.
 
 more on: http://shenwei356.github.io/seqkit/usage/#replace
 
+Special repalcement symbols:
+
+	{nr}	Record number, starting from 1
+	{kv}	Corresponding value of the key ($1) by key-value file
+
 `,
 	Run: func(cmd *cobra.Command, args []string) {
 		config := getConfigs(cmd)
@@ -66,10 +72,7 @@ more on: http://shenwei356.github.io/seqkit/usage/#replace
 
 		pattern := getFlagString(cmd, "pattern")
 		replacement := []byte(getFlagString(cmd, "replacement"))
-		var replaceeWithNR bool
-		if reNR.Match(replacement) {
-			replaceeWithNR = true
-		}
+		kvFile := getFlagString(cmd, "kv-file")
 
 		bySeq := getFlagBool(cmd, "by-seq")
 		// byName := getFlagBool(cmd, "by-name")
@@ -78,26 +81,63 @@ more on: http://shenwei356.github.io/seqkit/usage/#replace
 		if pattern == "" {
 			checkError(fmt.Errorf("flags -p (--pattern) needed"))
 		}
-
 		p := pattern
 		if ignoreCase {
 			p = "(?i)" + p
 		}
 		patternRegexp, err := regexp.Compile(p)
 		checkError(err)
 
+		var replaceWithNR bool
+		if reNR.Match(replacement) {
+			replaceWithNR = true
+		}
+
+		var replaceWithKV bool
+		var kvs map[string]string
+		if reKV.Match(replacement) {
+			replaceWithKV = true
+			if !regexp.MustCompile(`\(.+\)`).MatchString(pattern) {
+				checkError(fmt.Errorf(`value of -p (--pattern) must contains "(" and ")" to capture data which is used specify the KEY`))
+			}
+			if kvFile == "" {
+				checkError(fmt.Errorf(`since repalcement symbol "{kv}"/"{KV}" found in value of flag -r (--replacement), tab-delimited key-value file should be given by flag -k (--kv-file)`))
+			}
+			log.Infof("read key-value file: %s", kvFile)
+			kvs, err = readKVs(kvFile)
+			if err != nil {
+				checkError(fmt.Errorf("read key-value file: %s", err))
+			}
+			if len(kvs) == 0 {
+				checkError(fmt.Errorf("no valid data in key-value file: %s", kvFile))
+			}
+
+			if ignoreCase {
+				kvs2 := make(map[string]string, len(kvs))
+				for k, v := range kvs {
+					kvs2[strings.ToLower(k)] = v
+				}
+				kvs = kvs2
+			}
+
+			log.Infof("%d pairs of key-value loaded", len(kvs))
+		}
+
 		files := getFileList(args)
 
 		outfh, err := xopen.Wopen(outFile)
 		checkError(err)
 		defer outfh.Close()
 
 		var r []byte
+		var found [][]byte
+		var k string
+		var ok bool
 		for _, file := range files {
 
 			fastxReader, err := fastx.NewReader(alphabet, file, idRegexp)
 			checkError(err)
-			nr := 1
+			nr := 0
 			for {
 				record, err := fastxReader.Read()
 				if err != nil {
@@ -113,12 +153,30 @@ more on: http://shenwei356.github.io/seqkit/usage/#replace
 					record.Seq.Seq = patternRegexp.ReplaceAll(record.Seq.Seq, replacement)
 				} else {
 					r = replacement
-					if replaceeWithNR {
-						r = reNR.ReplaceAll(replacement, []byte(strconv.Itoa(nr)))
+
+					if replaceWithNR {
+						r = reNR.ReplaceAll(r, []byte(strconv.Itoa(nr)))
+					}
+
+					if replaceWithKV {
+						found = patternRegexp.FindSubmatch(record.Name)
+						if len(found) > 0 {
+							k = string(found[1])
+							if ignoreCase {
+								k = strings.ToLower(k)
+							}
+							if _, ok = kvs[k]; ok {
+								r = reKV.ReplaceAll(r, []byte(kvs[k]))
+							} else {
+								r = reKV.ReplaceAll(r, found[1])
+							}
+						}
 					}
-					record.Name = patternRegexp.ReplaceAll(record.Name, r)
+
 				}
 
+				record.Name = patternRegexp.ReplaceAll(record.Name, r)
+
 				record.FormatToWriter(outfh, lineWidth)
 			}
 
@@ -133,10 +191,13 @@ func init() {
 		"replacement. supporting capture variables. "+
 			" e.g. $1 represents the text of the first submatch. "+
 			"ATTENTION: use SINGLE quote NOT double quotes in *nix OS or "+
-			`use the \ escape character. Record number is also supported by "{NR}"`)
+			`use the \ escape character. Record number is also supported by "{nr}"`)
 	// replaceCmd.Flags().BoolP("by-name", "n", false, "replace full name instead of just id")
 	replaceCmd.Flags().BoolP("by-seq", "s", false, "replace seq")
 	replaceCmd.Flags().BoolP("ignore-case", "i", false, "ignore case")
+	replaceCmd.Flags().StringP("kv-file", "k", "",
+		`tab-delimited key-value file for replacing key with value when using "{kv}" in -r (--replacement)`)
 }
 
 var reNR = regexp.MustCompile(`\{(NR|nr)\}`)
+var reKV = regexp.MustCompile(`\{(KV|kv)\}`)