Skip to content

Commit 6e6ea00

Browse files
authored
Merging changes after rename (#220)
* Removed executable bit from all Go sources * Updated tutorial according to API changes * Added tutorial presentation * Fixed typos in presentation * Fix eof detection in pcap read * Added scalar IPv4 longest prefix match support * Update Makefile (#198) * Update Makefile Add non-parallel testing option to overcome hugepages error on Travis CI. * Refactoring - SetAffinity returns error - rename ParseCPU to HanpleCPUList - parseCPU is internal and does only parsing - common test now does not not require many cores * Added ring reorder test, minor bug fixing * Switched private and public network ports for convenince * Remove redundant packet functions * Simplify init packet functions * Fixed hostnames in tests. * Fixed Docker checkout URL. They renamed back to docker * Rewritten tests to make them run with go test * Updated test system and vagrant script for automatic testing. - Added docker server setup and configuration to VMs config script. - Removed docker port from launching in Makefiles because now it should be specified in YANFF_HOSTS. - Moved test framework to its own subdirectory. - Some cosmetic changes in ignore files. - Changed test framework command line. Added hosts command line override. - Switched vagrant script VMs to Fedora leaving Ubuntu as an option. - Disabled FSGSBASE and F16C instructions to allow working in VM. - Added verbose message after vagrant provisioning. - In Makefiles added user name to docker images names to allow simultaneous work in multiuser mode. - In test framework added username to images name. Added reaction to console interrupt with Ctrl-C. - Added final statistics after tests finish. - In Makefile set generated file dependency from types.go * Refactored tests: * renamed to camel-case instead of underscores * fixed stability tests to fail with go test if they fail (was only printing fail before) * created citesting target to test with travis and testing target with full stability tests * Updated code and scripts to reflect renaming from yanff to nff-go * Updates to scripts and testing to run automated tests - Changed YANFF_HOSTS format to comma separated list host1:port,host2:port. - Removed host-name and docker-port from test config files. Now they have to be specified on command line. - Added environment variables with DPDK and Pktgen archive locations. Now it is possible to download them once and use file:// URLs instead of downloading them every time. - Made yanff-base image with username prefix too. This allows parallel work for multiple users on the same host. - Fixed renaming artefacts. - Fixed dependencies to build NFF-Go. - Added optional docker image prefix. - Added non-zero return status if some tests failed. * Fixed variable name in if * Add handling of configuration requests to KNI * Fixed Fedora nat server script * Implemented stability tests and improved infra scripts - Split test VM hosts configuration into permanent and transient. - Added customizable VM name to allow multiple VM groups configured by one script. - Added docker setup by default. - Added test files generation for web server. - Switched to Go 1.9.4. - Added VM reboot in vagrant script. * Fixes in tests: * Fixed testCksum logic (was: passed when 0 packets received) * Fixed stability config (was: wrong image names) * Added ports to test configuration * Fixed all tests to work correctly in current VM setup - In many cases it is necessary to limit generator speed because VMs cannot handle packets quickly. - In many cases it is necessary to limit number of target packets because tests timeout 60 seconds is too short for default value. - Fixed port numbers for VM when port 0 is connected to port 0 and port 1 connected to port 1. - Added default MAKEOPTS for VMs to limit number of jobs to 4.
1 parent 8d7df01 commit 6e6ea00

File tree

180 files changed

+3266
-2866
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

180 files changed

+3266
-2866
lines changed

.gitignore

-10
Original file line numberDiff line numberDiff line change
@@ -1,13 +1,3 @@
11
# emacs
22
*~
3-
4-
examples/demo
5-
examples/Firewall
6-
examples/Forwarding
7-
examples/dump
8-
examples/clonable_pcap_dumper
9-
examples/kni
10-
examples/copy
11-
examples/gopacket_parser_example
12-
test/stash/send_fixed_number
133
doc

.travis.yml

+1-1
Original file line numberDiff line numberDiff line change
@@ -35,4 +35,4 @@ script:
3535
- sudo sh -c 'echo 1024 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages'
3636
- sudo mkdir /mnt/huge
3737
- sudo mount -t hugetlbfs nodev /mnt/huge
38-
- sudo -E env "PATH=$PATH GOROOT=$GOROOT GOPATH=$GOPATH" make testing
38+
- sudo -E env "PATH=$PATH GOROOT=$GOROOT GOPATH=$GOPATH" make citesting

CONTRIBUTING.md

+9-9
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,10 @@
1-
# Contributing to YANFF
1+
# Contributing to NFF-GO
22

3-
YANFF is an open source project licensed under the BSD-style license which can be found in the LICENSE file.
3+
NFF-GO is an open source project licensed under the BSD-style license which can be found in the LICENSE file.
44

55
## Coding Style
66

7-
YANFF follows the standard formatting recommendations and language idioms set out
7+
NFF-GO follows the standard formatting recommendations and language idioms set out
88
in the [Effective Go](https://golang.org/doc/effective_go.html) guide. It's
99
definitely worth reading - but the relevant sections are
1010
[formatting](https://golang.org/doc/effective_go.html#formatting)
@@ -34,17 +34,17 @@ Signed-off-by: <[email protected]>
3434

3535
We accept github pull requests.
3636

37-
If you want to work on github.com/intel-go/yanff and your fork on the same workstation you will need to use multiple GOPATHs. Assuming this is the case
37+
If you want to work on github.com/intel-go/nff-go and your fork on the same workstation you will need to use multiple GOPATHs. Assuming this is the case
3838

3939
1. Open a terminal
4040
2. mkdir -p ~/go-fork/src/github.com/intel-go (replacing go-fork with your preferred location)
4141
3. export GOPATH=~/go-fork
42-
4. cd $GOPATH/src/github.com/intel-go/yanff
43-
5. git clone https://github.com/GITHUB-USERNAME/yanff.git (replace GITHUB-USERNAME with your username)
44-
6. cd yanff
42+
4. cd $GOPATH/src/github.com/intel-go/nff-go
43+
5. git clone https://github.com/GITHUB-USERNAME/nff-go.git (replace GITHUB-USERNAME with your username)
44+
6. cd nff-go
4545
7. go install ./...
4646

47-
Once you've finished making your changes push them to your fork and send the PR via the github UI. If you don't need to maintain the github.com/intel-go/yanff repo and your fork on the same workstation you can skip steps 2 and 3.
47+
Once you've finished making your changes push them to your fork and send the PR via the github UI. If you don't need to maintain the github.com/intel-go/nff-go repo and your fork on the same workstation you can skip steps 2 and 3.
4848

4949
## Quality Controls
5050

@@ -56,7 +56,7 @@ We request you give quality assurance some consideration by:
5656
If you have a problem, please let us know. Write e-mail to [maintainers](mailto:[email protected])
5757

5858
If a bug is not already documented, by all means please [open an
59-
issue in github](https://github.com/intel-go/YANFF/issues/new) so we all get visibility
59+
issue in github](https://github.com/intel-go/NFF-GO/issues/new) so we all get visibility
6060
the problem and work toward resolution.
6161

6262
For feature requests we're also using github issues, with the label

Makefile

+11-4
Original file line numberDiff line numberDiff line change
@@ -3,13 +3,14 @@
33
# license that can be found in the LICENSE file.
44

55
PATH_TO_MK = mk
6-
SUBDIRS = yanff-base dpdk test examples
6+
SUBDIRS = nff-go-base dpdk test examples
77
DOC_TARGETS = flow packet
8-
TESTING_TARGETS = packet
8+
CI_TESTING_TARGETS = packet low common
9+
TESTING_TARGETS = $(CI_TESTING_TARGETS) test/stability
910

1011
all: $(SUBDIRS)
1112

12-
dpdk: yanff-base
13+
dpdk: nff-go-base
1314

1415
test: dpdk
1516

@@ -19,13 +20,19 @@ perf_testing:
1920
$(MAKE) -C test perf_testing
2021

2122
.PHONY: testing $(TESTING_TARGETS)
23+
.NOTPARALLEL: testing $(TESTING_TARGETS)
2224
testing: $(TESTING_TARGETS)
25+
26+
.PHONY: citesting $(CI_TESTING_TARGETS)
27+
.NOTPARALLEL: citesting $(CI_TESTING_TARGETS)
28+
citesting: $(CI_TESTING_TARGETS)
29+
2330
$(TESTING_TARGETS):
2431
$(MAKE) -C $@ testing
2532

2633
.PHONY: doc
2734
doc: $(DOC_TARGETS)
2835
mkdir doc
29-
$(foreach package,$(DOC_TARGETS),godoc -analysis=type -analysis=pointer -html github.com/intel-go/yanff/$(package) > doc/$(package).html;)
36+
$(foreach package,$(DOC_TARGETS),godoc -analysis=type -analysis=pointer -html github.com/intel-go/nff-go/$(package) > doc/$(package).html;)
3037

3138
include $(PATH_TO_MK)/intermediate.mk

README.md

+19-24
Original file line numberDiff line numberDiff line change
@@ -34,7 +34,7 @@ func CheckFatal(err error) {
3434
}
3535

3636
func main() {
37-
// Initialize NFF-Go library to use 8 cores max.
37+
// Initialize NFF-GO library to use 8 cores max.
3838
config := flow.Config{
3939
CPUCoresNumber: 8,
4040
}
@@ -69,35 +69,32 @@ func L3Separator(currentPacket *packet.Packet, context flow.UserContext) bool {
6969
return currentPacket.L3ACLPermit(L3Rules)
7070
}
7171
```
72-
73-
NFF-Go is an Open Source BSD licensed project that runs mostly in Linux user
72+
NFF-GO is an Open Source BSD licensed project that runs mostly in Linux user
7473
land. The most recent patches and enhancements provided by the community are
7574
available in the master branch.
7675

77-
## Getting NFF-Go
78-
79-
Use the **go get** command to download NFF-Go. You must first set your GOPATH
76+
## Getting NFF-GO
8077

78+
Use the **go get** command to download NFF-GO. You must first set your GOPATH
8179
export GOPATH=/my/local/directory
8280
go get -v -d github.com/intel-go/nff-go
8381

84-
Go will download the sources into $GOPATH/src. It will try to build NFF-Go and
82+
Go will download the sources into $GOPATH/src. It will try to build NFF-GO and
8583
fail with a message:
8684

87-
can't load package: package github.com/intel-go/yanff: no buildable Go source files in /localdisk/work/rscohn1/ws/nff-test/src/github.com/intel-go/nff-go
88-
85+
can't load package: package github.com/intel-go/nff-go: no buildable Go source files in /localdisk/work/rscohn1/ws/nff-go-test/src/github.com/intel-go/nff-go
8986
Ignore the message for now. We need to install some dependencies before you can
9087
build.
9188

9289
### Working with a github fork
9390

94-
If you are working on a fork, then the **go get** command will not put yanff in
91+
If you are working on a fork, then the **go get** command will not put nff-go in
9592
$GOPATH/src/github.com/intel-go. However, imports will continue to reference
96-
githb.com/intel-go. This is a feature of Go and not a problem in the way yanff
93+
githb.com/intel-go. This is a feature of Go and not a problem in the way nff-go
9794
is written. See [stackoverflow
9895
article](https://stackoverflow.com/questions/14323872/using-forked-package-import-in-go)
9996
for a discussion. A simple way to resolve the problem is to use a symlink. If
100-
you are rscohn2 on github, and you forked yanff into your personal account,
97+
you are rscohn2 on github, and you forked nff-go into your personal account,
10198
then do this:
10299

103100
cd $GOPATH/src/github.com
@@ -109,7 +106,7 @@ then do this:
109106

110107
### DPDK
111108

112-
NFF-Go uses DPDK, so you must setup your system to build and run DPDK. See [System
109+
NFF-GO uses DPDK, so you must setup your system to build and run DPDK. See [System
113110
Requirements in the DPDK Getting Started Guide for
114111
Linux](http://dpdk.org/doc/guides/linux_gsg/sys_reqs.html) for more
115112
information.
@@ -142,13 +139,12 @@ Use Go version 1.9 or higher. To check the version of Go, do:
142139

143140
export PATH="$PATH:$GOPATH"/bin
144141

145-
## Building NFF
142+
## Building NFF-GO
146143

147144
cd $GOPATH/src/github.com/intel-go/nff-go
148145
make -j8
149146

150-
## Running NFF
151-
147+
# Running NFF-GO
152148

153149
## Documentation
154150

@@ -167,7 +163,7 @@ and browse the following URLs:
167163
## Tests
168164

169165
Invoking make in the top-level directory builds the testing framework and
170-
examples. NFF-Go distributed tests are packaged inside of Docker container
166+
examples. NFF-GO distributed tests are packaged inside of Docker container
171167
images. There are also single node unit tests in some packages that you can
172168
run using the command:
173169

@@ -182,8 +178,8 @@ variable), use the **make images** command.
182178
To deploy Docker images for use in distributed testing, use the **make deploy**
183179
command. This command requires two environment variables:
184180

185-
* YANFF_HOSTS="hostname1 hostname2 ... hostnameN"* - a list of all hostnames for deployed test Docker images
186-
* DOCKER_PORT=2375* - the port number to connect to Docker daemons running on hosts in the YANFF_HOSTS variable
181+
* NFF_GO_HOSTS="hostname1 hostname2 ... hostnameN"* - a list of all hostnames for deployed test Docker images
182+
* DOCKER_PORT=2375* - the port number to connect to Docker daemons running on hosts in the NFF_GO_HOSTS variable
187183

188184
To delete generated images in the default Docker target, use the **make
189185
clean-images** command.
@@ -194,17 +190,17 @@ After the Docker images are deployed on all test hosts, you can run distributed
194190
network tests. The test framework is located in the test/main directory and
195191
accepts a JSON file with a test specification. There are predefined configs for
196192
performance and stability tests in the same directory. To run these tests,
197-
change **hostname1** and **hostname2** to the hosts from the YANFF_HOSTS list
193+
change **hostname1** and **hostname2** to the hosts from the NFF_GO_HOSTS list
198194
in these JSON files.
199195

200196
## Cleaning-up
201197

202198
To clean all generated binaries, use the **make clean** command. To delete all
203-
deployed images listed in YANFF_HOSTS, use the **make cleanall** command.
199+
deployed images listed in NFF_GO_HOSTS, use the **make cleanall** command.
204200

205201
## Changing the DPDK sources
206202

207-
If you use the **make** command from NFF-Go directories, the DPDK driver is
203+
If you use the **make** command from NFF-GO directories, the DPDK driver is
208204
downloaded automatically.
209205

210206
## Contributing
@@ -215,7 +211,6 @@ recommend checking the 'janitorial' bugs in our list of open issues; these bugs
215211
can be solved without an extensive knowledge of NFF-Go. We would love to help
216212
you start contributing.
217213

218-
You can reach the NFF-Go development team via our [mailing
219-
list](mailto:[email protected]).
214+
You can reach the NFF-Go development team via our [mailing list](mailto:[email protected]).
220215

221216

common/Makefile

+2-1
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,7 @@ include $(PATH_TO_MK)/include.mk
99
testing:
1010
go test
1111

12+
.PHONY: coverage
1213
coverage:
1314
go test -cover -coverprofile=c.out
14-
go tool cover -html=c.out -o common_coverage.html
15+
go tool cover -html=c.out -o common_coverage.html

common/common.go

+33-19
Original file line numberDiff line numberDiff line change
@@ -10,13 +10,16 @@ import (
1010
"fmt"
1111
"io"
1212
"log"
13+
"math"
1314
"os"
14-
"runtime"
1515
"strconv"
1616

1717
"github.com/pkg/errors"
1818
)
1919

20+
// Max array length for type conversions
21+
const MaxLength = math.MaxInt32
22+
2023
// Length of addresses.
2124
const (
2225
EtherAddrLen = 6
@@ -128,9 +131,10 @@ const (
128131
PcapReadFail
129132
PcapWriteFail
130133
InvalidCPURangeErr
134+
SetAffinityErr
131135
)
132136

133-
// NFError is error type returned by yanff functions
137+
// NFError is error type returned by nff-go functions
134138
type NFError struct {
135139
Code ErrorCode
136140
Message string
@@ -301,7 +305,7 @@ func GetDPDKLogLevel() string {
301305
}
302306
}
303307

304-
// GetDefaultCPUs returns default core list {0, 1, ..., GOMAXPROCS-1}
308+
// GetDefaultCPUs returns default core list {0, 1, ..., NumCPU}
305309
func GetDefaultCPUs(cpuNumber uint) []uint {
306310
cpus := make([]uint, cpuNumber, cpuNumber)
307311
for i := uint(0); i < cpuNumber; i++ {
@@ -310,9 +314,17 @@ func GetDefaultCPUs(cpuNumber uint) []uint {
310314
return cpus
311315
}
312316

313-
// ParseCPUs parses cpu list string into array of cpu numbers
314-
// and truncate the list according to given coresNumber (GOMAXPROCS)
315-
func ParseCPUs(s string, coresNumber uint) ([]uint, error) {
317+
// HandleCPUs parses cpu list string into array of valid core numbers.
318+
// Removes duplicates
319+
func HandleCPUList(s string, maxcpu uint) ([]uint, error) {
320+
nums, err := parseCPUs(s)
321+
nums = removeDuplicates(nums)
322+
nums = dropInvalidCPUs(nums, maxcpu)
323+
return nums, err
324+
}
325+
326+
// parseCPUs parses cpu list string into array of cpu numbers
327+
func parseCPUs(s string) ([]uint, error) {
316328
var startRange, k int
317329
nums := make([]uint, 0, 256)
318330
if s == "" {
@@ -351,19 +363,6 @@ func ParseCPUs(s string, coresNumber uint) ([]uint, error) {
351363
j = i + 1
352364
}
353365
}
354-
355-
nums = removeDuplicates(nums)
356-
357-
numCPU := uint(runtime.NumCPU())
358-
for _, cpu := range nums {
359-
if cpu >= numCPU {
360-
return []uint{}, WrapWithNFError(nil, "requested cpu exceeds maximum cores number on machine", MaxCPUExceedErr)
361-
}
362-
}
363-
if len(nums) > int(coresNumber) {
364-
return nums[:coresNumber], nil
365-
}
366-
367366
return nums, nil
368367
}
369368

@@ -378,3 +377,18 @@ func removeDuplicates(array []uint) []uint {
378377
}
379378
return result
380379
}
380+
381+
// dropInvalidCPUs validates cpu list. Takes array of cpu ids and checks it is < maxcpu on machine,
382+
// invalid are excluded. Returns list of valid cpu ids.
383+
func dropInvalidCPUs(nums []uint, maxcpu uint) []uint {
384+
i := 0
385+
for _, x := range nums {
386+
if x < maxcpu {
387+
nums[i] = x
388+
i++
389+
} else {
390+
LogWarning(Initialization, "Requested cpu", x, "exceeds maximum cores number on machine, skip it")
391+
}
392+
}
393+
return nums[:i]
394+
}

0 commit comments

Comments
 (0)