Skip to content

Commit d83d6b4

Browse files
SarkososSarkosos
authored
Merge pull request #383 from macrocosm-os/staging
Staging
2 parents aec8edb + 7dc938f commit d83d6b4

25 files changed

+1650
-163
lines changed

‎assets/folding_api.svg

+3
Loading

‎documentation/api/README.md

+370
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,370 @@
1+
# Protein Folding API Documentation
2+
3+
The Protein Folding API provides endpoints for submitting and managing protein folding jobs on the Bittensor network (Subnet 25). The API consists of two main components:
4+
5+
1. **Folding API** - For distributing protein folding jobs across multiple validators
6+
2. **Organic API** - For submitting jobs directly to individual validators (restricted access)
7+
8+
## Authentication
9+
10+
All endpoints use Epistula headers for request authentication. Each request must include:
11+
12+
- `Epistula-Version`: Must be "2"
13+
- `Epistula-Timestamp`: Current timestamp
14+
- `Epistula-Uuid`: Unique request identifier
15+
- `Epistula-Signed-By`: Sender's hotkey
16+
- `Epistula-Request-Signature`: Request signature
17+
18+
## Folding API
19+
20+
The Folding API allows researchers and developers to distribute protein folding jobs across multiple validators.
21+
22+
### Base URL
23+
```
24+
http://localhost:8029
25+
```
26+
27+
### Endpoints
28+
29+
#### POST /fold
30+
31+
Submit a new folding job to be distributed across multiple validators.
32+
33+
**Request Body Schema:**
34+
```json
35+
{
36+
"pdb_id": "string",
37+
"source": "rcsb" | "pdbe",
38+
"ff": "string",
39+
"water": "string",
40+
"box": "cube" | "box",
41+
"temperature": "float",
42+
"friction": "float",
43+
"epsilon": "float",
44+
"validator_uids": "array[int]",
45+
"num_validators_to_sample": "int (optional)",
46+
"timeout": "int (default: 5)"
47+
}
48+
```
49+
50+
**Response Schema:**
51+
```json
52+
{
53+
"uids": "array[int]",
54+
"hotkeys": "array[string]",
55+
"status_codes": "array[any]",
56+
"job_id": "array[string]"
57+
}
58+
```
59+
60+
**Example Usage:**
61+
```python
62+
import requests
63+
import json
64+
from atom.epistula.epistula import Epistula
65+
66+
epistula = Epistula()
67+
68+
body = {
69+
"pdb_id": "1ubq",
70+
"source": "rcsb",
71+
"ff": "charmm36.xml",
72+
"water": "charmm36/water.xml",
73+
"box": "cube",
74+
"temperature": 300.0,
75+
"friction": 1.0,
76+
"epsilon": 0.01,
77+
"validator_uids": [1, 2, 3],
78+
"num_validators_to_sample": 3
79+
}
80+
81+
body_bytes = json.dumps(body, sort_keys=True).encode('utf-8')
82+
headers = epistula.generate_header(your_hotkey, body_bytes)
83+
84+
response = requests.post(
85+
"http://localhost:8029/fold",
86+
headers=headers,
87+
data=body_bytes
88+
)
89+
90+
print(response.json())
91+
```
92+
93+
## Organic API
94+
95+
The Organic API enables direct submission of folding jobs to individual validators. Access is restricted to whitelisted hotkeys.
96+
97+
### Base URL
98+
```
99+
http://{validator_ip}:8031
100+
```
101+
102+
### Endpoints
103+
104+
#### POST /organic
105+
106+
Submit a folding job directly to a specific validator.
107+
108+
**Request Body Schema:**
109+
```json
110+
{
111+
"pdb_id": "string",
112+
"source": "rcsb" | "pdbe",
113+
"ff": "string",
114+
"water": "string",
115+
"box": "cube" | "box",
116+
"temperature": "float",
117+
"friction": "float",
118+
"epsilon": "float"
119+
}
120+
```
121+
122+
**Response Schema:**
123+
```json
124+
{
125+
"job_id": "string"
126+
}
127+
```
128+
129+
**Example Usage:**
130+
```python
131+
import requests
132+
import json
133+
from atom.epistula.epistula import Epistula
134+
135+
epistula = Epistula()
136+
137+
body = {
138+
"pdb_id": "1ubq",
139+
"source": "rcsb",
140+
"ff": "charmm36.xml",
141+
"water": "charmm36/water.xml",
142+
"box": "cube",
143+
"temperature": 300.0,
144+
"friction": 1.0,
145+
"epsilon": 0.01
146+
}
147+
148+
body_bytes = json.dumps(body, sort_keys=True).encode('utf-8')
149+
headers = epistula.generate_header(your_hotkey, body_bytes)
150+
151+
validator_address = "http://validator_ip:8031"
152+
response = requests.post(
153+
f"{validator_address}/organic",
154+
headers=headers,
155+
data=body_bytes
156+
)
157+
158+
print(response.json())
159+
```
160+
161+
## Organic API Startup Process
162+
163+
The Organic API is automatically started with each validator and follows this process:
164+
165+
1. **API Initialization**
166+
- Each validator starts its own Organic API instance on port 8031 (default)
167+
- The API is initialized only if the validator has not disabled the axon (`--neuron.axon_off`) or organic scoring (`--neuron.organic_disabled`)
168+
169+
2. **Address Registration**
170+
- The validator obtains its external IP address using a public service (checkip.amazonaws.com)
171+
- The full API address is constructed as: `http://{external_ip}:8031`
172+
- This address is registered on-chain as the validator's "commitment" using the Bittensor subtensor
173+
174+
3. **Address Discovery**
175+
- Other network participants can discover a validator's Organic API address by querying the validator's commitment on-chain
176+
- The commitment is automatically updated if the validator's IP address changes
177+
178+
Example of how a validator starts the Organic API:
179+
```python
180+
from folding.organic.api import start_organic_api
181+
import requests
182+
183+
# Get external IP
184+
external_ip = requests.get("https://checkip.amazonaws.com").text.strip()
185+
186+
# Construct API address
187+
commitment = f"http://{external_ip}:8031"
188+
189+
# Register address on-chain
190+
subtensor.commit(
191+
wallet=wallet,
192+
netuid=netuid,
193+
data=commitment
194+
)
195+
196+
# Start the API
197+
await start_organic_api(organic_validator, config)
198+
```
199+
200+
To query a validator's Organic API address:
201+
```python
202+
# Get validator's commitment (API address)
203+
validator_address = subtensor.get_commitment(netuid, validator_uid)
204+
```
205+
206+
### Configuration Options
207+
208+
The Organic API can be configured using the following command-line arguments:
209+
210+
- `--neuron.axon_off`: Disables the axon and Organic API
211+
- `--neuron.organic_enabled`: enables only organic scoring
212+
- `--neuron.organic_api.port`: Sets the API port (default: 8031)
213+
- `--organic_whitelist`: List of hotkeys allowed to use the Organic API
214+
215+
## Common Parameters
216+
217+
### Simulation Parameters
218+
- `pdb_id`: The PDB identifier for the protein structure
219+
- `source`: The source database for the PDB file (`rcsb` or `pdbe`)
220+
- `ff`: Force field specification file (e.g., "charmm36.xml")
221+
- `water`: Water model specification file (e.g., "charmm36/water.xml")
222+
- `box`: Simulation box type (`cube` or `box`)
223+
- `temperature`: Simulation temperature in Kelvin
224+
- `friction`: Friction coefficient for the simulation
225+
- `epsilon`: Base epsilon value for the challenge (represented as %/100)
226+
227+
### Folding API Specific Parameters
228+
- `validator_uids`: List of validator UIDs to target
229+
- `num_validators_to_sample`: Optional number of validators to sample from the provided UIDs
230+
- `timeout`: Request timeout in seconds (default: 5)
231+
232+
## Error Handling
233+
234+
The API returns standard HTTP status codes:
235+
236+
- 200: Successful request
237+
- 400: Bad request (invalid parameters)
238+
- 403: Forbidden (authentication failure or unauthorized access)
239+
- 429: Too many requests (rate limit exceeded)
240+
- 500: Internal server error
241+
242+
## Rate Limiting
243+
244+
The API implements rate limiting based on the client's IP address using the `slowapi` library.
245+
246+
## Monitoring
247+
248+
The Folding API includes Prometheus metrics for monitoring through the FastAPI Instrumentator. Metrics are exposed at the `/metrics` endpoint.
249+
250+
## Development
251+
252+
The APIs are built using FastAPI and include:
253+
254+
- Request/Response schemas using Pydantic models
255+
- Async request handling
256+
- Prometheus metrics integration (Folding API)
257+
- Rate limiting
258+
- Structured logging
259+
- Access control (Organic API)
260+
261+
For local development of the Folding API:
262+
263+
```bash
264+
uvicorn folding_api.main:app --host 0.0.0.0 --port 8029 --reload
265+
```
266+
267+
The Organic API starts automatically with each validator service.
268+
269+
## Validator Registry
270+
271+
The Folding API maintains a validator registry that manages the availability and selection of validators for protein folding jobs. The registry:
272+
273+
1. Tracks validator status and performance
274+
2. Manages validator cooldown periods
275+
3. Handles validator selection based on stake-weighted sampling
276+
277+
### Validator Selection
278+
279+
When submitting a job through the Folding API, validators are selected based on:
280+
281+
1. **Specified UIDs**: If `validator_uids` are provided and all UIDs are available in the registry, these validators will be used
282+
2. **Random Sampling**: If no UIDs are specified or some are unavailable, validators are randomly selected based on their stake weight
283+
284+
### Validator Availability
285+
286+
Validators are considered available based on:
287+
- Minimum stake requirement (≥ 10,000 TAO)
288+
- Active status on the network
289+
- Not being in a cooldown period
290+
- Having a valid commitment address
291+
292+
### Cooldown Mechanism
293+
294+
The registry implements an exponential backoff mechanism for failed requests:
295+
- Initial timeout: 1 second
296+
- On failure: Timeout doubles (capped at 24 hours)
297+
- On success: Timeout resets to 1 second
298+
299+
Example of validator selection and request handling:
300+
```python
301+
import requests
302+
import json
303+
from atom.epistula.epistula import Epistula
304+
305+
epistula = Epistula()
306+
307+
# Submit to specific validators
308+
body = {
309+
"pdb_id": "1ubq",
310+
"source": "rcsb",
311+
"ff": "charmm36.xml",
312+
"water": "charmm36/water.xml",
313+
"box": "cube",
314+
"temperature": 300.0,
315+
"friction": 1.0,
316+
"epsilon": 0.01,
317+
"validator_uids": [1, 2, 3] # Specific validators
318+
}
319+
320+
# Or let the registry select validators
321+
body_random = {
322+
"pdb_id": "1ubq",
323+
"source": "rcsb",
324+
"ff": "charmm36.xml",
325+
"water": "charmm36/water.xml",
326+
"box": "cube",
327+
"temperature": 300.0,
328+
"friction": 1.0,
329+
"epsilon": 0.01,
330+
"num_validators_to_sample": 3 # Number of validators to randomly select
331+
}
332+
```
333+
334+
## Starting the Folding API
335+
336+
The Folding API can be started in different network configurations. Here are the common ways to start it:
337+
338+
### Development/Test Network
339+
340+
For development or testing, start the API connected to the test network:
341+
342+
```bash
343+
python folding_api/main.py \
344+
--netuid 141 \
345+
--subtensor.network test \
346+
--wallet.name folding_testnet \
347+
--wallet.hotkey your_hotkey
348+
```
349+
350+
### Production Network
351+
352+
For production, connect to the main network (Subnet 25):
353+
354+
```bash
355+
python folding_api/main.py \
356+
--netuid 25 \
357+
--wallet.name your_wallet \
358+
--wallet.hotkey your_hotkey
359+
```
360+
361+
### Configuration Parameters
362+
363+
- `netuid`: The subnet ID (25 for mainnet, 141 for testnet)
364+
- `subtensor.network`: Network to connect to (`finney` for mainnet, `test` for testnet)
365+
- `wallet.name`: The name of your Bittensor wallet
366+
- `wallet.hotkey`: The hotkey to use for signing requests
367+
368+
## API Diagram
369+
370+
![API Diagram](../../assets/folding_api.svg)

‎folding/__init__.py

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
__version__ = "2.1.4"
1+
__version__ = "2.2.0"
22
version_split = __version__.split(".")
33
__spec_version__ = (
44
(10000 * int(version_split[0]))

0 commit comments

Comments
 (0)