Skip to content

Commit 98f3520

Browse files
committed
remove pagination, update readme
1 parent c6ab2ec commit 98f3520

File tree

3 files changed

+114
-30
lines changed

3 files changed

+114
-30
lines changed

README.md

Lines changed: 68 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,6 @@
1-
## w3kipedia
1+
## zwi-fave
22

3-
[w3kipedia](https://github.com/onepeerlabs/w3kipedia) was originally a try to participate for the [WAM](https://www.wearemillions.online/) hackathon for a better wikipedia on swarm.
4-
5-
Now [w3kipedia-fave](https://github.com/onepeerlabs/w3kipedia-fave) is a modified version of w3kipedia that uses FaVe to store, index and search the content.
3+
This project uses FaVe and fairdatasociety/huggingface-vectorizer to index, store and search zwi files content.
64

75
It has two components
86

@@ -14,8 +12,6 @@ It has two components
1412

1513
Server will start a http-server that can fetch content from FaVe and display them in the web browser.
1614

17-
18-
1915
### How to serve?
2016

2117
```
@@ -25,7 +21,7 @@ go build
2521

2622
Docker:
2723
```
28-
docker build -f Dockerfile.server --tag w3ki-server .
24+
docker build -f Dockerfile.server --tag zwi-fave-server .
2925
```
3026

3127
#### Help :
@@ -43,7 +39,7 @@ Usage of ./server:
4339

4440
Docker:
4541
```
46-
docker run w3ki-server -h
42+
docker run zwi-fave-server -h
4743
```
4844

4945

@@ -58,20 +54,75 @@ Docker:
5854
```
5955
docker run \
6056
-p 8080:8080 \
61-
w3ki-server -fave=<FAVE_API_ENDPOINT> -collection=<COLLECTION_NAME>
57+
zwi-fave-server -fave=<FAVE_API_ENDPOINT> -collection=<COLLECTION_NAME>
6258
```
6359

64-
This will start a local http-serve which will serve wikipedia content on port `:8080`.
60+
This will start a local http-serve which will serve wikipedia content on port `:8080`.
6561

66-
#### How Indexer works:
62+
this project [uses code](https://github.com/akhenakh/gozim/blob) that is [MIT licensed](https://github.com/akhenakh/gozim/blob/master/LICENSE)
6763

68-
Indexer uses [FaVe](https://github.com/fairDataSociety/FaVe) to store content and index them for doing semantic search. It is sanitizing the Article content and vectorizing it to do FaVe Nearest Neighbour Search.
64+
### Docker Compose
6965

70-
#### How Server works:
66+
Best way to run the server is to use a single docker-compose file that will start vectorizer, FaVe and the server.
7167

72-
Server lists all the items in the server with "text/html" mimetype.
68+
Copy the following docker-compose file and save it as `docker-compose.yml` in a directory.
69+
```
70+
version: '3'
71+
services:
72+
vectorizer:
73+
command:
74+
- --model-name
75+
- sentence-transformers/all-mpnet-base-v2
76+
image: fairdatasociety/huggingface-vectorizer:latest
77+
ports:
78+
- 9876:9876
79+
restart: on-failure:0
80+
81+
fave:
82+
command:
83+
- --host
84+
- 0.0.0.0
85+
- --port
86+
- '1234'
87+
- --write-timeout
88+
- 1500m
89+
- --read-timeout
90+
- 1500m
91+
image: fairdatasociety/fave:latest
92+
ports:
93+
- 1234:1234
94+
restart: on-failure:0
95+
environment:
96+
BEE_API: <BEE_URL>
97+
RPC_API: <RPC_ENDPOINT_FOR_ENS>
98+
STAMP_ID: 0
99+
USER: <USER_NAME>
100+
PASSWORD: <PASSWORD>
101+
POD: <POD_NAME>
102+
VECTORIZER_URL: http://vectorizer:9876
103+
VERBOSE: true
104+
zwi-fave:
105+
command:
106+
- -collection
107+
- <COLLECTION_NAME>
108+
- -port
109+
- '8526'
110+
- -fave
111+
- http://fave:1234/v1
112+
image: fairdatasociety/zwi-fave:latest
113+
ports:
114+
- 8526:8526
115+
restart: on-failure:0
116+
```
73117

74-
For showing the content it checks "GET" request on "/wiki/XXXX". it reads the "XXXX" relative url, then finds the entry for that item in the FaVe/FairOS-dfs document store,
75-
downloads the content and sends it back as response.
118+
Change the environment variables to match your setup.
119+
```
120+
BEE_API: <BEE_URL> # Bee api endpoint (Make sure to use local ip (e.g. 192.168.x.x), not "localhost" or "127.0.0.1")
121+
RPC_API: <RPC_ENDPOINT_FOR_ENS> # This is for fairOS-dfs to authenticate with ENS
122+
USER: <USER_NAME> # FairOS-dfs user name
123+
PASSWORD: <PASSWORD> # FairOS-dfs password
124+
POD: <POD_NAME> # FairOS-dfs pod name to store content
125+
COLLECTION_NAME: <COLLECTION_NAME> # Collection name to store content on FaVe
126+
```
76127

77-
this project [uses code](https://github.com/akhenakh/gozim/blob) that is [MIT licensed](https://github.com/akhenakh/gozim/blob/master/LICENSE)
128+
Then run `docker-compose up` to start the server. The server should be available on port `:8526`.

cmd/server/templates/searchResult.html

Lines changed: 0 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -70,19 +70,6 @@
7070
{{end}}
7171
</div> <!-- /container -->
7272

73-
<div id="footer">
74-
<div class="container">
75-
<ul class="pager">
76-
<li class="previous">
77-
<a href="/search/?search_data={{ .Query }}&page={{ .PreviousPage }}">← Previous</a>
78-
</li>
79-
<li class="next">
80-
<a href="/search/?search_data={{ .Query }}&page={{ .NextPage }}">Next →</a>
81-
</li>
82-
</ul>
83-
</div>
84-
</div>
85-
8673
<script src="/static/js/jquery.min.js"></script>
8774
<script src="/static/js/bootstrap.min.js"></script>
8875

docker-compose.yaml

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
version: '3'
2+
services:
3+
vectorizer:
4+
command:
5+
- --model-name
6+
- sentence-transformers/all-mpnet-base-v2
7+
image: fairdatasociety/huggingface-vectorizer:latest
8+
ports:
9+
- 9876:9876
10+
restart: on-failure:0
11+
12+
fave:
13+
command:
14+
- --host
15+
- 0.0.0.0
16+
- --port
17+
- '1234'
18+
- --write-timeout
19+
- 1500m
20+
- --read-timeout
21+
- 1500m
22+
image: fairdatasociety/fave:latest
23+
ports:
24+
- 1234:1234
25+
restart: on-failure:0
26+
environment:
27+
BEE_API: <BEE_URL>
28+
RPC_API: <RPC_ENDPOINT_FOR_ENS>
29+
STAMP_ID: <STAMP_ID>
30+
USER: <USER_NAME>
31+
PASSWORD: <PASSWORD>
32+
POD: <POD_NAME>
33+
VECTORIZER_URL: http://vectorizer:9876
34+
VERBOSE: true
35+
zwi-fave:
36+
command:
37+
- -collection
38+
- <COLLECTION_NAME>
39+
- -port
40+
- '8526'
41+
- -fave
42+
- http://fave:1234/v1
43+
image: fairdatasociety/zwi-fave:latest
44+
ports:
45+
- 8526:8526
46+
restart: on-failure:0

0 commit comments

Comments
 (0)