Skip to content

Commit b32f418

Browse files
authored
Adding support for storing snapshots with any time resolution in CSV files and File System Repository. (#268)
# Describe Request This is to allow using different date time formats to store snapshots at various resolutions and no longer just days. Added support for configuring the CSV file, and the File System Repository. - `asset.Snapshot.Date` no longer has a date format declared as a struct tag. - `helper.NewCsv` takes options to configure headers, logger, and the default date time format. - `asset.NewFileSystemRepository` also takes CSV options to configure the CSV files. Example: ```golang csv, err := helper.NewCsv[Row]( helper.WithCsvDefaultDateTimeFormat[Row]("2006-01-02 15:04:05"), ) ``` Related #266 # Change Type New feature. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit Based on the comprehensive summary, here are the release notes: - **New Features** - Enhanced CSV handling with more flexible configuration options - Introduced functional options for CSV processing - Added new configuration methods for CSV operations - **Improvements** - Simplified CSV reading functions - Removed hardcoded boolean parameters in CSV file reading - Updated default date format handling - **Refactoring** - Restructured CSV-related functions to use variadic options - Modified function signatures across multiple packages - Streamlined CSV data processing methods <!-- end of auto-generated comment: release notes by coderabbit.ai -->
1 parent a3e7cbd commit b32f418

File tree

107 files changed

+442
-262
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

107 files changed

+442
-262
lines changed

asset/README.md

Lines changed: 11 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -34,7 +34,7 @@ The information provided on this project is strictly for informational purposes
3434
- [func SnapshotsAsOpenings\(snapshots \<\-chan \*Snapshot\) \<\-chan float64](<#SnapshotsAsOpenings>)
3535
- [func SnapshotsAsVolumes\(snapshots \<\-chan \*Snapshot\) \<\-chan float64](<#SnapshotsAsVolumes>)
3636
- [type FileSystemRepository](<#FileSystemRepository>)
37-
- [func NewFileSystemRepository\(base string\) \*FileSystemRepository](<#NewFileSystemRepository>)
37+
- [func NewFileSystemRepository\(base string, csvOptions ...helper.CsvOption\[Snapshot\]\) \*FileSystemRepository](<#NewFileSystemRepository>)
3838
- [func \(r \*FileSystemRepository\) Append\(name string, snapshots \<\-chan \*Snapshot\) error](<#FileSystemRepository.Append>)
3939
- [func \(r \*FileSystemRepository\) Assets\(\) \(\[\]string, error\)](<#FileSystemRepository.Assets>)
4040
- [func \(r \*FileSystemRepository\) Get\(name string\) \(\<\-chan \*Snapshot, error\)](<#FileSystemRepository.Get>)
@@ -183,7 +183,7 @@ func SnapshotsAsVolumes(snapshots <-chan *Snapshot) <-chan float64
183183
SnapshotsAsVolumes extracts the volume field from each snapshot in the provided channel and returns a new channel containing only those volume values.The original snapshots channel can no longer be directly used afterward.
184184

185185
<a name="FileSystemRepository"></a>
186-
## type [FileSystemRepository](<https://github.com/cinar/indicator/blob/master/asset/file_system_repository.go#L20-L23>)
186+
## type [FileSystemRepository](<https://github.com/cinar/indicator/blob/master/asset/file_system_repository.go#L20-L26>)
187187

188188
FileSystemRepository stores and retrieves asset snapshots using the local file system.
189189

@@ -194,16 +194,16 @@ type FileSystemRepository struct {
194194
```
195195

196196
<a name="NewFileSystemRepository"></a>
197-
### func [NewFileSystemRepository](<https://github.com/cinar/indicator/blob/master/asset/file_system_repository.go#L27>)
197+
### func [NewFileSystemRepository](<https://github.com/cinar/indicator/blob/master/asset/file_system_repository.go#L30>)
198198

199199
```go
200-
func NewFileSystemRepository(base string) *FileSystemRepository
200+
func NewFileSystemRepository(base string, csvOptions ...helper.CsvOption[Snapshot]) *FileSystemRepository
201201
```
202202

203-
NewFileSystemRepository initializes a file system repository with the given base directory.
203+
NewFileSystemRepository initializes a file system repository with the given base directory and the CSV options.
204204

205205
<a name="FileSystemRepository.Append"></a>
206-
### func \(\*FileSystemRepository\) [Append](<https://github.com/cinar/indicator/blob/master/asset/file_system_repository.go#L92>)
206+
### func \(\*FileSystemRepository\) [Append](<https://github.com/cinar/indicator/blob/master/asset/file_system_repository.go#L96>)
207207

208208
```go
209209
func (r *FileSystemRepository) Append(name string, snapshots <-chan *Snapshot) error
@@ -212,7 +212,7 @@ func (r *FileSystemRepository) Append(name string, snapshots <-chan *Snapshot) e
212212
Append adds the given snapshows to the asset with the given name.
213213

214214
<a name="FileSystemRepository.Assets"></a>
215-
### func \(\*FileSystemRepository\) [Assets](<https://github.com/cinar/indicator/blob/master/asset/file_system_repository.go#L34>)
215+
### func \(\*FileSystemRepository\) [Assets](<https://github.com/cinar/indicator/blob/master/asset/file_system_repository.go#L38>)
216216

217217
```go
218218
func (r *FileSystemRepository) Assets() ([]string, error)
@@ -221,7 +221,7 @@ func (r *FileSystemRepository) Assets() ([]string, error)
221221
Assets returns the names of all assets in the repository.
222222

223223
<a name="FileSystemRepository.Get"></a>
224-
### func \(\*FileSystemRepository\) [Get](<https://github.com/cinar/indicator/blob/master/asset/file_system_repository.go#L56>)
224+
### func \(\*FileSystemRepository\) [Get](<https://github.com/cinar/indicator/blob/master/asset/file_system_repository.go#L60>)
225225

226226
```go
227227
func (r *FileSystemRepository) Get(name string) (<-chan *Snapshot, error)
@@ -230,7 +230,7 @@ func (r *FileSystemRepository) Get(name string) (<-chan *Snapshot, error)
230230
Get attempts to return a channel of snapshots for the asset with the given name.
231231

232232
<a name="FileSystemRepository.GetSince"></a>
233-
### func \(\*FileSystemRepository\) [GetSince](<https://github.com/cinar/indicator/blob/master/asset/file_system_repository.go#L61>)
233+
### func \(\*FileSystemRepository\) [GetSince](<https://github.com/cinar/indicator/blob/master/asset/file_system_repository.go#L65>)
234234

235235
```go
236236
func (r *FileSystemRepository) GetSince(name string, date time.Time) (<-chan *Snapshot, error)
@@ -239,7 +239,7 @@ func (r *FileSystemRepository) GetSince(name string, date time.Time) (<-chan *Sn
239239
GetSince attempts to return a channel of snapshots for the asset with the given name since the given date.
240240

241241
<a name="FileSystemRepository.LastDate"></a>
242-
### func \(\*FileSystemRepository\) [LastDate](<https://github.com/cinar/indicator/blob/master/asset/file_system_repository.go#L75>)
242+
### func \(\*FileSystemRepository\) [LastDate](<https://github.com/cinar/indicator/blob/master/asset/file_system_repository.go#L79>)
243243

244244
```go
245245
func (r *FileSystemRepository) LastDate(name string) (time.Time, error)
@@ -476,7 +476,7 @@ Snapshot captures a single observation of an asset's price at a specific moment.
476476
```go
477477
type Snapshot struct {
478478
// Date represents the specific timestamp.
479-
Date time.Time `format:"2006-01-02"`
479+
Date time.Time
480480

481481
// Open represents the opening price for the
482482
// snapshot period.

asset/file_system_repository.go

Lines changed: 9 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -20,13 +20,17 @@ import (
2020
type FileSystemRepository struct {
2121
// base is the root directory where asset snapshots are stored.
2222
base string
23+
24+
// csvOptions are the CSV options used for reading and writing snapshots.
25+
csvOptions []helper.CsvOption[Snapshot]
2326
}
2427

2528
// NewFileSystemRepository initializes a file system repository with
26-
// the given base directory.
27-
func NewFileSystemRepository(base string) *FileSystemRepository {
29+
// the given base directory and the CSV options.
30+
func NewFileSystemRepository(base string, csvOptions ...helper.CsvOption[Snapshot]) *FileSystemRepository {
2831
return &FileSystemRepository{
29-
base: base,
32+
base: base,
33+
csvOptions: csvOptions,
3034
}
3135
}
3236

@@ -54,7 +58,7 @@ func (r *FileSystemRepository) Assets() ([]string, error) {
5458

5559
// Get attempts to return a channel of snapshots for the asset with the given name.
5660
func (r *FileSystemRepository) Get(name string) (<-chan *Snapshot, error) {
57-
return helper.ReadFromCsvFile[Snapshot](r.getCsvFileName(name), true)
61+
return helper.ReadFromCsvFile[Snapshot](r.getCsvFileName(name), r.csvOptions...)
5862
}
5963

6064
// GetSince attempts to return a channel of snapshots for the asset with the given name since the given date.
@@ -90,7 +94,7 @@ func (r *FileSystemRepository) LastDate(name string) (time.Time, error) {
9094

9195
// Append adds the given snapshows to the asset with the given name.
9296
func (r *FileSystemRepository) Append(name string, snapshots <-chan *Snapshot) error {
93-
return helper.AppendOrWriteToCsvFile(r.getCsvFileName(name), true, snapshots)
97+
return helper.AppendOrWriteToCsvFile(r.getCsvFileName(name), snapshots, r.csvOptions...)
9498
}
9599

96100
// getCsvFileName gets the CSV file name for the given asset name.

asset/file_system_repository_test.go

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -69,7 +69,7 @@ func TestFileSystemRepositoryGetSince(t *testing.T) {
6969
t.Fatal(err)
7070
}
7171

72-
expected, err := helper.ReadFromCsvFile[asset.Snapshot]("testdata/since.csv", true)
72+
expected, err := helper.ReadFromCsvFile[asset.Snapshot]("testdata/since.csv")
7373
if err != nil {
7474
t.Fatal(err)
7575
}

asset/snapshot.go

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@ import (
1414
// at a specific moment.
1515
type Snapshot struct {
1616
// Date represents the specific timestamp.
17-
Date time.Time `format:"2006-01-02"`
17+
Date time.Time
1818

1919
// Open represents the opening price for the
2020
// snapshot period.

asset/snapshot_test.go

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@ import (
1212
)
1313

1414
func TestSnapshotsAs(t *testing.T) {
15-
snapshots, err := helper.ReadFromCsvFile[asset.Snapshot]("testdata/repository/brk-b.csv", true)
15+
snapshots, err := helper.ReadFromCsvFile[asset.Snapshot]("testdata/repository/brk-b.csv")
1616
if err != nil {
1717
t.Fatal(err)
1818
}

helper/README.md

Lines changed: 56 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,7 @@ The information provided on this project is strictly for informational purposes
2727
- [Constants](<#constants>)
2828
- [func Abs\[T Number\]\(c \<\-chan T\) \<\-chan T](<#Abs>)
2929
- [func Add\[T Number\]\(ac, bc \<\-chan T\) \<\-chan T](<#Add>)
30-
- [func AppendOrWriteToCsvFile\[T any\]\(fileName string, hasHeader bool, rows \<\-chan \*T\) error](<#AppendOrWriteToCsvFile>)
30+
- [func AppendOrWriteToCsvFile\[T any\]\(fileName string, rows \<\-chan \*T, options ...CsvOption\[T\]\) error](<#AppendOrWriteToCsvFile>)
3131
- [func Apply\[T Number\]\(c \<\-chan T, f func\(T\) T\) \<\-chan T](<#Apply>)
3232
- [func Buffered\[T any\]\(c \<\-chan T, size int\) \<\-chan T](<#Buffered>)
3333
- [func ChanToJSON\[T any\]\(c \<\-chan T, w io.Writer\) error](<#ChanToJSON>)
@@ -69,7 +69,7 @@ The information provided on this project is strictly for informational purposes
6969
- [func Operate3\[A any, B any, C any, R any\]\(ac \<\-chan A, bc \<\-chan B, cc \<\-chan C, o func\(A, B, C\) R\) \<\-chan R](<#Operate3>)
7070
- [func Pipe\[T any\]\(f \<\-chan T, t chan\<\- T\)](<#Pipe>)
7171
- [func Pow\[T Number\]\(c \<\-chan T, y T\) \<\-chan T](<#Pow>)
72-
- [func ReadFromCsvFile\[T any\]\(fileName string, hasHeader bool\) \(\<\-chan \*T, error\)](<#ReadFromCsvFile>)
72+
- [func ReadFromCsvFile\[T any\]\(fileName string, options ...CsvOption\[T\]\) \(\<\-chan \*T, error\)](<#ReadFromCsvFile>)
7373
- [func Remove\(t \*testing.T, name string\)](<#Remove>)
7474
- [func RemoveAll\(t \*testing.T, path string\)](<#RemoveAll>)
7575
- [func RoundDigit\[T Number\]\(n T, d int\) T](<#RoundDigit>)
@@ -93,11 +93,15 @@ The information provided on this project is strictly for informational purposes
9393
- [func \(b \*Bst\[T\]\) Remove\(value T\) bool](<#Bst[T].Remove>)
9494
- [type BstNode](<#BstNode>)
9595
- [type Csv](<#Csv>)
96-
- [func NewCsv\[T any\]\(hasHeader bool\) \(\*Csv\[T\], error\)](<#NewCsv>)
96+
- [func NewCsv\[T any\]\(options ...CsvOption\[T\]\) \(\*Csv\[T\], error\)](<#NewCsv>)
9797
- [func \(c \*Csv\[T\]\) AppendToFile\(fileName string, rows \<\-chan \*T\) error](<#Csv[T].AppendToFile>)
9898
- [func \(c \*Csv\[T\]\) ReadFromFile\(fileName string\) \(\<\-chan \*T, error\)](<#Csv[T].ReadFromFile>)
9999
- [func \(c \*Csv\[T\]\) ReadFromReader\(reader io.Reader\) \<\-chan \*T](<#Csv[T].ReadFromReader>)
100100
- [func \(c \*Csv\[T\]\) WriteToFile\(fileName string, rows \<\-chan \*T\) error](<#Csv[T].WriteToFile>)
101+
- [type CsvOption](<#CsvOption>)
102+
- [func WithCsvDefaultDateTimeFormat\[T any\]\(format string\) CsvOption\[T\]](<#WithCsvDefaultDateTimeFormat>)
103+
- [func WithCsvLogger\[T any\]\(logger \*slog.Logger\) CsvOption\[T\]](<#WithCsvLogger>)
104+
- [func WithoutCsvHeader\[T any\]\(\) CsvOption\[T\]](<#WithoutCsvHeader>)
101105
- [type Float](<#Float>)
102106
- [type Integer](<#Integer>)
103107
- [type Number](<#Number>)
@@ -132,7 +136,7 @@ const (
132136
CsvFormatTag = "format"
133137

134138
// DefaultDateTimeFormat denotes the default format of a date and time column.
135-
DefaultDateTimeFormat = "2006-01-02 15:04:05"
139+
DefaultDateTimeFormat = "2006-01-02"
136140
)
137141
```
138142

@@ -182,10 +186,10 @@ fmt.Println(actual) // [2, 4, 6, 8, 10, 12, 14, 16, 18, 20]
182186
```
183187

184188
<a name="AppendOrWriteToCsvFile"></a>
185-
## func [AppendOrWriteToCsvFile](<https://github.com/cinar/indicator/blob/master/helper/csv.go#L291>)
189+
## func [AppendOrWriteToCsvFile](<https://github.com/cinar/indicator/blob/master/helper/csv.go#L323>)
186190

187191
```go
188-
func AppendOrWriteToCsvFile[T any](fileName string, hasHeader bool, rows <-chan *T) error
192+
func AppendOrWriteToCsvFile[T any](fileName string, rows <-chan *T, options ...CsvOption[T]) error
189193
```
190194

191195
AppendOrWriteToCsvFile writes the provided rows of data to the specified file, appending to the existing file if it exists or creating a new one if it doesn't. In append mode, the function assumes that the existing file's column order matches the field order of the given row struct to ensure consistent data structure.
@@ -805,10 +809,10 @@ fmt.Println(helper.ChanToSlice(squared)) // [4, 9, 25, 100]
805809
```
806810

807811
<a name="ReadFromCsvFile"></a>
808-
## func [ReadFromCsvFile](<https://github.com/cinar/indicator/blob/master/helper/csv.go#L278>)
812+
## func [ReadFromCsvFile](<https://github.com/cinar/indicator/blob/master/helper/csv.go#L310>)
809813

810814
```go
811-
func ReadFromCsvFile[T any](fileName string, hasHeader bool) (<-chan *T, error)
815+
func ReadFromCsvFile[T any](fileName string, options ...CsvOption[T]) (<-chan *T, error)
812816
```
813817

814818
ReadFromCsvFile creates a CSV instance, parses CSV data from the provided filename, maps the data to corresponding struct fields, and delivers it through the channel.
@@ -1095,7 +1099,7 @@ type BstNode[T Number] struct {
10951099
```
10961100

10971101
<a name="Csv"></a>
1098-
## type [Csv](<https://github.com/cinar/indicator/blob/master/helper/csv.go#L40-L50>)
1102+
## type [Csv](<https://github.com/cinar/indicator/blob/master/helper/csv.go#L40-L53>)
10991103

11001104
Csv represents the configuration for CSV reader and writer.
11011105

@@ -1109,16 +1113,16 @@ type Csv[T any] struct {
11091113
```
11101114

11111115
<a name="NewCsv"></a>
1112-
### func [NewCsv](<https://github.com/cinar/indicator/blob/master/helper/csv.go#L54>)
1116+
### func [NewCsv](<https://github.com/cinar/indicator/blob/master/helper/csv.go#L80>)
11131117

11141118
```go
1115-
func NewCsv[T any](hasHeader bool) (*Csv[T], error)
1119+
func NewCsv[T any](options ...CsvOption[T]) (*Csv[T], error)
11161120
```
11171121

1118-
NewCsv function initializes a new CSV instance. The parameter hasHeader indicates whether the CSV contains a header row.
1122+
NewCsv creates a new CSV instance with the provided options.
11191123

11201124
<a name="Csv[T].AppendToFile"></a>
1121-
### func \(\*Csv\[T\]\) [AppendToFile](<https://github.com/cinar/indicator/blob/master/helper/csv.go#L174>)
1125+
### func \(\*Csv\[T\]\) [AppendToFile](<https://github.com/cinar/indicator/blob/master/helper/csv.go#L206>)
11221126

11231127
```go
11241128
func (c *Csv[T]) AppendToFile(fileName string, rows <-chan *T) error
@@ -1127,7 +1131,7 @@ func (c *Csv[T]) AppendToFile(fileName string, rows <-chan *T) error
11271131
AppendToFile appends the provided rows of data to the end of the specified file, creating the file if it doesn't exist. In append mode, the function assumes that the existing file's column order matches the field order of the given row struct to ensure consistent data structure.
11281132

11291133
<a name="Csv[T].ReadFromFile"></a>
1130-
### func \(\*Csv\[T\]\) [ReadFromFile](<https://github.com/cinar/indicator/blob/master/helper/csv.go#L150>)
1134+
### func \(\*Csv\[T\]\) [ReadFromFile](<https://github.com/cinar/indicator/blob/master/helper/csv.go#L182>)
11311135

11321136
```go
11331137
func (c *Csv[T]) ReadFromFile(fileName string) (<-chan *T, error)
@@ -1136,7 +1140,7 @@ func (c *Csv[T]) ReadFromFile(fileName string) (<-chan *T, error)
11361140
ReadFromFile parses the CSV data from the provided file name, maps the data to corresponding struct fields, and delivers the resulting rows through the channel.
11371141

11381142
<a name="Csv[T].ReadFromReader"></a>
1139-
### func \(\*Csv\[T\]\) [ReadFromReader](<https://github.com/cinar/indicator/blob/master/helper/csv.go#L95>)
1143+
### func \(\*Csv\[T\]\) [ReadFromReader](<https://github.com/cinar/indicator/blob/master/helper/csv.go#L127>)
11401144

11411145
```go
11421146
func (c *Csv[T]) ReadFromReader(reader io.Reader) <-chan *T
@@ -1145,14 +1149,50 @@ func (c *Csv[T]) ReadFromReader(reader io.Reader) <-chan *T
11451149
ReadFromReader parses the CSV data from the provided reader, maps the data to corresponding struct fields, and delivers the resulting it through the channel.
11461150

11471151
<a name="Csv[T].WriteToFile"></a>
1148-
### func \(\*Csv\[T\]\) [WriteToFile](<https://github.com/cinar/indicator/blob/master/helper/csv.go#L190>)
1152+
### func \(\*Csv\[T\]\) [WriteToFile](<https://github.com/cinar/indicator/blob/master/helper/csv.go#L222>)
11491153

11501154
```go
11511155
func (c *Csv[T]) WriteToFile(fileName string, rows <-chan *T) error
11521156
```
11531157

11541158
WriteToFile creates a new file with the given name and writes the provided rows of data to it, overwriting any existing content.
11551159

1160+
<a name="CsvOption"></a>
1161+
## type [CsvOption](<https://github.com/cinar/indicator/blob/master/helper/csv.go#L56>)
1162+
1163+
CsvOption represents a functional option for configuring the CSV instance.
1164+
1165+
```go
1166+
type CsvOption[T any] func(*Csv[T])
1167+
```
1168+
1169+
<a name="WithCsvDefaultDateTimeFormat"></a>
1170+
### func [WithCsvDefaultDateTimeFormat](<https://github.com/cinar/indicator/blob/master/helper/csv.go#L73>)
1171+
1172+
```go
1173+
func WithCsvDefaultDateTimeFormat[T any](format string) CsvOption[T]
1174+
```
1175+
1176+
WithCsvDefaultDateTimeFormat sets the default date and time format for the CSV instance.
1177+
1178+
<a name="WithCsvLogger"></a>
1179+
### func [WithCsvLogger](<https://github.com/cinar/indicator/blob/master/helper/csv.go#L66>)
1180+
1181+
```go
1182+
func WithCsvLogger[T any](logger *slog.Logger) CsvOption[T]
1183+
```
1184+
1185+
WithCsvLogger sets the logger for the CSV instance.
1186+
1187+
<a name="WithoutCsvHeader"></a>
1188+
### func [WithoutCsvHeader](<https://github.com/cinar/indicator/blob/master/helper/csv.go#L59>)
1189+
1190+
```go
1191+
func WithoutCsvHeader[T any]() CsvOption[T]
1192+
```
1193+
1194+
WithoutCsvHeader disables the header row in the CSV.
1195+
11561196
<a name="Float"></a>
11571197
## type [Float](<https://github.com/cinar/indicator/blob/master/helper/helper.go#L27-L29>)
11581198

0 commit comments

Comments
 (0)