Skip to content

Commit 9c888e1

Browse files
likplikp
authored
Instructions for replacing the default selection (#27)
* Improve formatting and readability of configuration documentation * Enhance documentation for saved queries storage options and database requirements * Refactor database table definitions for SavedQueryMeta2 and DefaultSelection for improved clarity and consistency * Add documentation for default selection and saved queries management * Reworked the texts with help of Copilot * Fixed misspellings
1 parent 5ec08d8 commit 9c888e1

File tree

1 file changed

+185
-22
lines changed

1 file changed

+185
-22
lines changed

‎docs/PxWebApi/documentation/configuration.md‎

Lines changed: 185 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -1,17 +1,31 @@
11
# Configuring PxWebApi 2
2-
The configuration is split into several different files, each with a specific purpose. The main file you will need to modify is `appsettings.json`.
2+
3+
The configuration is split into several different files, each with a specific
4+
purpose. The main file you will need to modify is `appsettings.json`.
35

46
## appsettings.json
5-
This file contains all functional settings. A recommended configuration after installation is the Admin protection settings, which control access to the admin API. The settings in `appsettings.json` can also be overridden by environment variables. For example, to set the DataSourceType, define an environment variable called `DataSource__DataSourceType`.
7+
8+
This file contains all functional settings. A recommended configuration after
9+
installation is the Admin protection settings, which control access to the admin
10+
API. The settings in `appsettings.json` can also be overridden by environment
11+
variables. For example, to set the DataSourceType, define an environment variable
12+
called `DataSource__DataSourceType`.
613

714
### General settings
815

916
#### Language
10-
Specify which languages should be available by editing the `PxApiConfiguration.Languages` section. Set the default language by assigning `PxApiConfiguration.DefaultLanguage` to the `id` of one of the languages listed under `PxApiConfiguration.Languages`.
17+
18+
Specify which languages should be available by editing the
19+
`PxApiConfiguration.Languages` section. Set the default language by assigning
20+
`PxApiConfiguration.DefaultLanguage` to the `id` of one of the languages listed
21+
under `PxApiConfiguration.Languages`.
1122

1223
#### Downloading data
13-
You can limit the size of data that can be downloaded using the `PxApiConfiguration.MaxDataCells` setting.
14-
You can also control which formats are available for downloading data by setting `PxApiConfiguration.OutputFormats`.
24+
25+
You can limit the size of data that can be downloaded using the
26+
`PxApiConfiguration.MaxDataCells` setting.
27+
You can also control which formats are available for downloading data by setting
28+
`PxApiConfiguration.OutputFormats`.
1529
The following formats can be enabled or disabled:
1630

1731
- Excel (`xlsx`)
@@ -29,58 +43,207 @@ The following formats are always available and cannot be disabled:
2943
JSON-stat v1 has been deprecated and is no longer supported as an output format.
3044

3145
#### Attribution
32-
You can specify how you would like others to reference your data by setting `PxApiConfiguration.SourceReference`. Make sure to provide a section for each language specified under `PxApiConfiguration.Languages`.
46+
47+
You can specify how you would like others to reference your data by setting
48+
`PxApiConfiguration.SourceReference`. Make sure to provide a section for each
49+
language specified under `PxApiConfiguration.Languages`.
3350

3451
You can also specify the license for the data under `PxApiConfiguration.License`.
3552

3653
#### Caching
37-
You can control how long a response should be cached by setting `PxApiConfiguration.CacheTime` (in seconds).
54+
55+
You can control how long a response should be cached by setting
56+
`PxApiConfiguration.CacheTime` (in seconds).
3857

3958
You also have the option to clear the cache at a specific time by setting `PxApiConfiguration.CacheClearTime`.
4059

4160
### Datasource
42-
PxWebApi 2.0 supports two different data sources: PX file-based or CNMM. You can only configure one at a time. Set `DataSource.DataSourceType` to either `PX` or `CNMM` to specify which you want to use.
61+
62+
PxWebApi 2.0 supports two different data sources: PX file-based or CNMM. You can
63+
only configure one at a time. Set `DataSource.DataSourceType` to either `PX` or
64+
`CNMM` to specify which you want to use.
4365

4466
#### PX datasource
45-
For the PX file datasource, you can set `DataSource.DataSourceType.PX.StrictAggregations` to either `true` or `false` to determine if aggregations can be overlapping or not.
67+
68+
For the PX file datasource, you can set `DataSource.DataSourceType.PX.StrictAggregations`
69+
to either `true` or `false` to determine if aggregations can be overlapping or not.
4670

4771
#### CNMM datasource
48-
You should only have one database specified, and you must set `DataSource.DataSourceType.CNMM.DatabaseId` to match the `id` of the database in `SqlDb.config`.
4972

50-
When you configure the CNMM datasource, a health check is automatically set up to verify the database connection. It connects to the default database in `SqlDb.config` and executes a statement. The default statement is `select 1`, but you can change this by setting `DataSource.DataSourceType.CNMM.HealthCheckQuery`. Avoid using queries that take a long time to process.
73+
You should only have one database specified, and you must set `DataSource.DataSourceType.CNMM.DatabaseId`
74+
to match the `id` of the database in `SqlDb.config`.
75+
76+
When you configure the CNMM datasource, a health check is automatically set up
77+
to verify the database connection. It connects to the default database in `SqlDb.config`
78+
and executes a statement. The default statement is `select 1`, but you can change
79+
this by setting `DataSource.DataSourceType.CNMM.HealthCheckQuery`. Avoid using
80+
queries that take a long time to process.
5181

5282
### Saved query storage
53-
Saved queries are an essential part of the API. Users can create, view, and run saved queries. Saved queries can also be used as the default selection when a user does not specify any selection when downloading data.
5483

55-
You can store saved queries either as files on the file system or in a relational database. Select the storage method by setting `SavedQuery.Backend`.
84+
Saved queries can be viewed as templates for repeatable data extractions. Users
85+
can create, list, and execute them. A saved query can also serve as the default selection
86+
for a table when a request omits an explicit selection (see
87+
[Default selection](#default-selection)).
88+
89+
Supported storage backends:
90+
91+
- File system
92+
- Relational database (SQL Server or Oracle)
93+
94+
Choose backend with `SavedQuery.Backend` (e.g. `"File"` or `"Database"`).
5695

5796
#### File
58-
Specify the path where saved queries should be stored by setting `SavedQuery.FileStorage.Path`.
97+
98+
Each saved query is represented by two sibling files sharing the same base name
99+
but having diffrent suffixes:
100+
101+
- `.sqa` – the serialized saved query (definition)
102+
- `.sqs` – metadata (creation date, last run, etc.)
103+
104+
Directory layout: Saved queries are grouped in subfolders under
105+
`SavedQuery.FileStorage.Path`. The subfolder name is the first character of the
106+
saved query id. This prevents extremely large single directories.
107+
108+
Change the root storage path with `SavedQuery.FileStorage.Path`.
59109

60110
#### Database
61-
Specify the connection string and other relevant settings for storing saved queries in a database.
111+
112+
Provide:
113+
114+
- Connection string: `SavedQuery.DatabaseStorage.ConnectionString`
115+
- Vendor: `SavedQuery.DatabaseStorage.DatabaseVendor` (`SqlServer` or `Oracle`)
116+
117+
The database requires the tables: `SavedQueryMeta2` and `DefaultSelection`.
118+
119+
Schema (conceptual):
120+
121+
```sql
122+
SavedQueryMeta2 (
123+
QueryId int NOT NULL,
124+
DataSourceType varchar(10) NOT NULL,
125+
DatabaseId varchar(500) NULL,
126+
DataSourceId varchar(500) NOT NULL,
127+
Status char(1) NOT NULL,
128+
StatusUse char(1) NOT NULL,
129+
StatusChange char(1) NOT NULL,
130+
OwnerId varchar(80) NOT NULL,
131+
MyDescription varchar(250) NOT NULL,
132+
Tags varchar(250) NULL,
133+
CreatedDate smalldatetime NOT NULL,
134+
ChangedDate smalldatetime NULL,
135+
ChangedBy varchar(80) NULL,
136+
UsedDate smalldatetime NULL,
137+
DataSourceUpdateDate smalldatetime NULL,
138+
SavedQueryFormat varchar(10) NOT NULL,
139+
SavedQueryStorage char(1) NOT NULL,
140+
QueryText varchar(max) NOT NULL,
141+
Runs int NOT NULL,
142+
Fails int NOT NULL,
143+
)
144+
145+
DefaultSelection (
146+
TableId varchar(20) NOT NULL,
147+
SavedQueryId int NOT NULL,
148+
)
149+
```
150+
151+
Notes:
152+
153+
- `QueryText` stores the actual saved query (e.g. JSON) used for execution.
154+
- `Runs` / `Fails` enable simple usage statistics.
155+
- `DefaultSelection` maps a table id to the saved query used as its default.
62156

63157
### Rate limiting
64-
API rate limiting settings are found under `IpRateLimiting`. You can read more about the options [here](https://github.com/stefanprodan/aspnetcoreratelimit/wiki/ipratelimitmiddleware). The most important settings are `IpRateLimiting.GeneralRules.Period` and `IpRateLimiting.GeneralRules.Limit`, which control the time window and the number of calls allowed during that period.
65-
If you do not run behind a proxy or load balancer, you should remove the `IpRateLimiting.RealIpHeader` setting.
66158

159+
API rate limiting settings are found under `IpRateLimiting`. You can read more
160+
about the options [here](https://github.com/stefanprodan/aspnetcoreratelimit/wiki/ipratelimitmiddleware).
161+
The most important settings are `IpRateLimiting.GeneralRules.Period` and
162+
`IpRateLimiting.GeneralRules.Limit`, which control the time window and the number
163+
of calls allowed during that period.
164+
165+
If you do not run behind a proxy or load balancer, you should remove the
166+
`IpRateLimiting.RealIpHeader` setting.
67167

68168
### Admin protection
69-
The admin API is protected by both an IP filter and a token that must be specified in the request. You can set these options using `AdminProtection.IpWhitelist` and `AdminProtection.AdminKey`. You can specify more than one IP in `AdminProtection.IpWhitelist`, including ranges of IPs in CIDR notation.
70169

170+
The admin API is protected by both an IP filter and a token that must be specified
171+
in the request. You can set these options using `AdminProtection.IpWhitelist` and
172+
`AdminProtection.AdminKey`. You can specify more than one IP in
173+
`AdminProtection.IpWhitelist`, including ranges of IPs in CIDR notation.
71174

72175
## app.config
176+
73177
This is for legacy configuration. It contains paths for:
178+
74179
- language files
75180
- aggregation and valueset files
76181
- the name of the CNMM configuration file
77-
In most cases, the default settings are sufficient and you do not need to modify this file.
182+
183+
In most cases, the default settings are sufficient and you do not need to modify
184+
this file.
78185

79186
## web.config
80-
The `web.config` file is used to configure PxWebApi 2.0 when running on Microsoft Internet Information Services (IIS). In most cases, the default settings are sufficient and you do not need to modify this file. Only advanced scenarios or specific server requirements may require changes.
187+
188+
The `web.config` file is used to configure PxWebApi 2.0 when running on Microsoft
189+
Internet Information Services (IIS). In most cases, the default settings are
190+
sufficient and you do not need to modify this file. Only advanced scenarios or
191+
specific server requirements may require changes.
81192

82193
## SqlDb.config
83-
Currently, this file is the same as in PxWeb 2023, with one exception: you should only have one database specified.
194+
195+
Currently, this file is the same as in PxWeb 2023, with one exception: you should
196+
only have one database specified.
84197

85198
## log4net.config
86-
The `log4net.config` file manages how PxWebApi 2.0 creates and stores logs. By default, logs are saved in the `/logs` directory. PxWebApi 2.0 relies on log4net for logging, which allows you to customize log levels, formats, and destinations. For more information on how to configure log4net, please refer to the official documentation [here](https://logging.apache.org/log4net/manual/configuration.html).
199+
200+
The `log4net.config` file manages how PxWebApi 2.0 creates and stores logs. By
201+
default, logs are saved in the `/logs` directory. PxWebApi 2.0 relies on log4net
202+
for logging, which allows you to customize log levels, formats, and destinations.
203+
For more information on how to configure log4net, please refer to the official
204+
documentation [here](https://logging.apache.org/log4net/manual/configuration.html).
205+
206+
## Default selection
207+
208+
When a data request is submitted without any explicit selection, PxWebApi builds
209+
one automatically. The built-in algorithm picks a small, representative set of
210+
variable values based on variable types and cardinalities.
211+
212+
If that automatic selection is not suitable, you can override it with a saved
213+
query. The override process depends on the saved query backend.
214+
215+
!!! note
216+
The saved query must target the same table (`tableId` match) or the override
217+
will fail or have unexpected behavior.
218+
219+
Also note that this is an manual process to replace the default selection.
220+
221+
### File backend override
222+
223+
Steps:
224+
225+
1. Identify (or create) the saved query you want to use (its `.sqa` + `.sqs`).
226+
2. Copy both files.
227+
3. Rename the copies so the base name equals the target table id (keep suffixes).
228+
4. Place them in the correct subfolder under `SavedQuery.FileStorage.Path`.
229+
(Subfolder name = first character of the table id.)
230+
5. Clear cache if needed; subsequent no-selection requests now use the saved query.
231+
232+
### Database backend override
233+
234+
Add or update the mapping in `DefaultSelection`:
235+
236+
```sql
237+
-- Pseudocode mapping description
238+
TableId = <table id>
239+
SavedQueryId = <QueryId from SavedQueryMeta2>
240+
```
241+
242+
Procedure:
243+
244+
1. Confirm the saved query exists (`SavedQueryMeta2.QueryId`).
245+
2. UPDATE if present, INSERT if absent.
246+
3. Clear caches if immediate effect is required.
247+
248+
!!! note
249+
`TableId` should acts as primary key; use UPDATE if a row already exists.

0 commit comments

Comments
 (0)