Skip to content

Commit ce6f144

Browse files
shukriadamsshukriadams
committed
Merge branch 'master' into develop
2 parents 45a64b9 + 40875f8 commit ce6f144

File tree

14 files changed

+244
-246
lines changed

14 files changed

+244
-246
lines changed

‎docs/development.md

+6-7
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,10 @@ If you want to develop Linux on Windows, Vagrant is an excellent and convenient
1616
- Dotnet 6.0 SDK
1717
- Visual Studio 2019 (Windows) or Visual Studio Code with C# extension. If you're using Visual Studio make sure you've update to the latest version. Visual Studio 2017 isn't supported.
1818

19+
Unit test coverage is done with https://github.com/SteveGilham/altcover, install ReportGenerator with
20+
21+
dotnet tool install --global dotnet-reportgenerator-globaltool
22+
1923
## Running from Visual Studio
2024

2125
- Start Visual Studio as administrator
@@ -73,15 +77,10 @@ To run tests in Visual Studio, open Test > Windows > Test Explorer, run desired
7377

7478
## Test coverage reports
7579

76-
Unit test coverage is done with https://github.com/SteveGilham/altcover. To visualize your tests
77-
78-
- Install ReportGenerator
79-
80-
dotnet tool install --global dotnet-reportgenerator-globaltool
81-
82-
- after testing run
80+
After testing run
8381

8482
cd /src
83+
8584
reportgenerator -reports:./Tetrifact.Tests/coverage.xml -targetdir:./Tetrifact.Tests/coverage -assemblyfilters:+Tetrifact.*;-Tetrifact.Tests -classfilters:-Tetrifact.Core.ThreadDefault;-Tetrifact.Web.DaemonProcessRunner;-Tetrifact.Web.Pager;-Tetrifact.Web.Program;-Tetrifact.Web.Startup;-Tetrifact.Web.ReadLevel;-Tetrifact.Web.WriteLevel;-*f__*
8685

8786
The HTML report is at

‎docs/use.md

+158-123
Original file line numberDiff line numberDiff line change
@@ -1,123 +1,158 @@
1-
# Using Tetrifact
2-
3-
Tetrifact stores builds as packages. A package is a group of files which are added and retrieved as a unit, and should correspond to a single build in your CI system, which in turn should of course correspond to a COMMIT or REVISION in your VCS (version control system). It is highly recommended you name your Tetrifact packages after the revision number (or hash) in your VCS the package's contents were compiled from, as this allows you to easily trace a package back to the source code from which it was built.
4-
5-
Don't worry if your revision hashes are difficult to pass around, share or remember, packages can be tagged with additional human-friendly names too.
6-
7-
## Changing log level
8-
9-
The default log level as of version 1.8.4 is `Warning`. You can override this with the environment variable
10-
11-
Logging__LogLevel__Microsoft=<YOUR LEVEL>
12-
13-
Allowed levels are the standard internal Dotnet values : Trace|Debug|Information|Warning|Error|Critical|None
14-
15-
## REST
16-
17-
All Tetrifact's functionality is exposed via a REST API, so it should be familiar.
18-
19-
## Adding a package
20-
21-
### As individual files
22-
23-
HTTP METHOD :
24-
25-
POST
26-
27-
ENDPOINT :
28-
29-
/v1/packages/myPackageName
30-
31-
HEADER :
32-
33-
Content-Type: multipart/form-data; boundary=-------------------------acebdf13572468
34-
35-
BODY :
36-
37-
---------------------------acebdf13572468
38-
Content-Disposition: form-data; name="Files"; filename="1.txt"
39-
40-
file 1 text or binary content here
41-
---------------------------acebdf13572468
42-
Content-Disposition: form-data; name="Files" ; filename="path/to/2.txt"
43-
44-
file 2 text or binary content here
45-
---------------------------acebdf13572468
46-
47-
Each file's "name" property must be "Files". The filename property should be the path and filename of the file you want to store, relative to the root of the package.
48-
49-
Were you posting actual files with CURL it should look like
50-
51-
curl -X POST \
52-
-H "Content-Type: multipart/form-data" \
53-
-H "Transfer-Encoding: chunked" \
54-
-F "Files=@~/mybuild/1.txt;filename=1.txt" \
55-
-F "Files=@~/mybuild/path/to/2.txt;filename=path/to/2.txt" \
56-
http://myTetrifact.server/v1/packages/myPackageName
57-
58-
### As an archive
59-
60-
To post your build as an archive, use the following.
61-
62-
HTTP METHOD :
63-
64-
POST
65-
66-
ENDPOINT :
67-
68-
/v1/packages/myPackageName?IsArchive=true
69-
70-
HEADER :
71-
72-
Content-Type: multipart/form-data; boundary=-------------------------acebdf13572468
73-
74-
BODY :
75-
76-
---------------------------acebdf13572468
77-
Content-Disposition: form-data; name="FILES"; filename="whatever"
78-
Content-Type: application/octet-stream
79-
80-
<@INCLUDE *~/path/to/archive.zip*@>
81-
---------------------------acebdf13572468
82-
83-
When posting a zip, note the following
84-
85-
- only a single file can be attached to your post. If you add more than one, you'll get an error.
86-
- filename doesn't matter, as it won't be used
87-
- the archive's root will be treated as the root of the project, meaning all file paths will be mapped relative to this.
88-
89-
With CURL it would look like
90-
91-
curl -X POST -H "Content-Type: multipart/form-data" -H "Transfer-Encoding: chunked" -F "Files=@path/to/archive" http://tetriserver.example.com/v1/packages/myPackage?isArchive=true
92-
93-
Chunking the upload is important if your archive is large, as this might exceeded the multipart body attachment size.
94-
95-
### Posting an archive from NodeJS
96-
97-
This uses the request package (https://www.npmjs.com/package/request).
98-
99-
let request = require('request'),
100-
fs = require('fs'),
101-
formdata = {
102-
Files : fs.createReadStream('path/to/archive')
103-
};
104-
105-
request.post({url: 'http://tetriserver.example.com', formData: formdata}, function(err, httpResponse, body) {
106-
if (err) {
107-
return console.error('upload failed : ', err);
108-
}
109-
110-
console.log('Upload succeeded : ', body);
111-
});
112-
113-
## Tagging
114-
115-
### Curl
116-
117-
To add the tag "MyTag" to the package "MyPackage, use
118-
119-
curl -X POST http://tetriserver.example.com/v1/tags/MyTag/MyPackage
120-
121-
To remove the tag
122-
123-
curl -X DELETE http://tetriserver.example.com/v1/tags/MyTag/MyPackage
1+
# Using Tetrifact
2+
3+
Tetrifact stores builds as packages. A package is a group of files which are added and retrieved as a unit, and should correspond to a single build in your CI system, which in turn should of course correspond to a COMMIT or REVISION in your VCS (version control system). It is highly recommended you name your Tetrifact packages after the revision number (or hash) in your VCS the package's contents were compiled from, as this allows you to easily trace a package back to the source code from which it was built.
4+
5+
Don't worry if your revision hashes are difficult to pass around, share or remember, packages can be tagged with additional human-friendly names too.
6+
7+
## Changing log level
8+
9+
The default log level as of version 1.8.4 is `Warning`. You can override this with the environment variable
10+
11+
Logging__LogLevel__Microsoft=<YOUR LEVEL>
12+
13+
Allowed levels are the standard internal Dotnet values : Trace|Debug|Information|Warning|Error|Critical|None
14+
15+
## REST
16+
17+
All Tetrifact's functionality is exposed via a REST API, so it should be familiar.
18+
19+
## Adding a package
20+
21+
### As individual files
22+
23+
HTTP METHOD :
24+
25+
POST
26+
27+
ENDPOINT :
28+
29+
/v1/packages/myPackageName
30+
31+
HEADER :
32+
33+
Content-Type: multipart/form-data; boundary=-------------------------acebdf13572468
34+
35+
BODY :
36+
37+
---------------------------acebdf13572468
38+
Content-Disposition: form-data; name="Files"; filename="1.txt"
39+
40+
file 1 text or binary content here
41+
---------------------------acebdf13572468
42+
Content-Disposition: form-data; name="Files" ; filename="path/to/2.txt"
43+
44+
file 2 text or binary content here
45+
---------------------------acebdf13572468
46+
47+
Each file's "name" property must be "Files". The filename property should be the path and filename of the file you want to store, relative to the root of the package.
48+
49+
Were you posting actual files with CURL it should look like
50+
51+
curl -X POST \
52+
-H "Content-Type: multipart/form-data" \
53+
-H "Transfer-Encoding: chunked" \
54+
-F "Files=@~/mybuild/1.txt;filename=1.txt" \
55+
-F "Files=@~/mybuild/path/to/2.txt;filename=path/to/2.txt" \
56+
http://myTetrifact.server/v1/packages/myPackageName
57+
58+
### As an archive
59+
60+
To post your build as an archive, use the following.
61+
62+
HTTP METHOD :
63+
64+
POST
65+
66+
ENDPOINT :
67+
68+
/v1/packages/myPackageName?IsArchive=true
69+
70+
HEADER :
71+
72+
Content-Type: multipart/form-data; boundary=-------------------------acebdf13572468
73+
74+
BODY :
75+
76+
---------------------------acebdf13572468
77+
Content-Disposition: form-data; name="FILES"; filename="whatever"
78+
Content-Type: application/octet-stream
79+
80+
<@INCLUDE *~/path/to/archive.zip*@>
81+
---------------------------acebdf13572468
82+
83+
When posting a zip, note the following
84+
85+
- only a single file can be attached to your post. If you add more than one, you'll get an error.
86+
- filename doesn't matter, as it won't be used
87+
- the archive's root will be treated as the root of the project, meaning all file paths will be mapped relative to this.
88+
89+
With CURL it would look like
90+
91+
curl -X POST -H "Content-Type: multipart/form-data" -H "Transfer-Encoding: chunked" -F "Files=@path/to/archive" http://tetriserver.example.com/v1/packages/myPackage?isArchive=true
92+
93+
Chunking the upload is important if your archive is large, as this might exceeded the multipart body attachment size.
94+
95+
### Posting an archive from NodeJS
96+
97+
This uses the request package (https://www.npmjs.com/package/request).
98+
99+
let request = require('request'),
100+
fs = require('fs'),
101+
formdata = {
102+
Files : fs.createReadStream('path/to/archive')
103+
};
104+
105+
request.post({url: 'http://tetriserver.example.com', formData: formdata}, function(err, httpResponse, body) {
106+
if (err) {
107+
return console.error('upload failed : ', err);
108+
}
109+
110+
console.log('Upload succeeded : ', body);
111+
});
112+
113+
## Tagging
114+
115+
### Curl
116+
117+
To add the tag "MyTag" to the package "MyPackage, use
118+
119+
curl -X POST http://tetriserver.example.com/v1/tags/MyTag/MyPackage
120+
121+
To remove the tag
122+
123+
curl -X DELETE http://tetriserver.example.com/v1/tags/MyTag/MyPackage
124+
125+
126+
## Pruning
127+
128+
Tetrifact has automated pruning to delete older packages to prevent disk overuse. All prune settings are kept in config.yml.
129+
130+
Pruning is disabled by default, to enable it use
131+
132+
...
133+
PruneEnabled : True
134+
...
135+
136+
Pruning behaviour works in brackets. A bracket is a period for the age of packages, and x number of packages of a given age can be kept. Using the example below
137+
138+
PruneBrackets :
139+
- Days: 1
140+
Amount: -1
141+
- Days : 3
142+
Amount : 5
143+
- Days : 5
144+
Amount : 4
145+
146+
The first bracket is for all packages up to 1 day old, an amount of -1 means prune none, ie, keep all packages that are up to a day old. The second bracket indicates that for all packages created more than 1 ago and less than 3 days ago, keep 5 packages.
147+
148+
Pruning can ignore packages marked with one or more tags, use the list
149+
150+
PruneIgnoreTags:
151+
- MyKeepTag
152+
- MyImportTag
153+
154+
Pruning runs at 2am server time by default, you can change this with
155+
156+
PruneCronMask: "0 3 * * *"
157+
158+

‎src/Tetrifact.Core/ArchiveService.cs

+10-30
Original file line numberDiff line numberDiff line change
@@ -238,36 +238,16 @@ private async Task ArchiveDotNetZip(string packageId, string archivePathTemp)
238238
ZipArchiveEntry zipEntry = archive.CreateEntry(file.Path, _settings.ArchiveCompression);
239239

240240
using (Stream zipEntryStream = zipEntry.Open())
241-
{
242-
if (manifest.IsCompressed)
243-
{
244-
GetFileResponse fileLookup = _indexReader.GetFile(file.Id);
245-
if (fileLookup == null)
246-
throw new Exception($"Failed to find expected package file {file.Id} - repository is likely corrupt");
247-
248-
using (var storageArchive = new ZipArchive(fileLookup.Content))
249-
{
250-
ZipArchiveEntry storageArchiveEntry = storageArchive.Entries[0];
251-
using (var storageArchiveStream = storageArchiveEntry.Open())
252-
{
253-
StreamProgressCopy copy = new StreamProgressCopy(storageArchiveStream, zipEntryStream, copyStepSize);
254-
copy.OnProgress += progressEvent;
255-
await copy.Work();
256-
}
257-
}
258-
}
259-
else
260-
{
261-
GetFileResponse fileLookup = _indexReader.GetFile(file.Id);
262-
if (fileLookup == null)
263-
throw new Exception($"Failed to find expected package file {file.Id}- repository is likely corrupt");
264-
265-
using (Stream fileStream = fileLookup.Content)
266-
{
267-
StreamProgressCopy copy = new StreamProgressCopy(fileStream, zipEntryStream, copyStepSize);
268-
copy.OnProgress += progressEvent;
269-
await copy.Work();
270-
}
241+
{
242+
GetFileResponse fileLookup = _indexReader.GetFile(file.Id);
243+
if (fileLookup == null)
244+
throw new Exception($"Failed to find expected package file {file.Id}- repository is likely corrupt");
245+
246+
using (Stream fileStream = fileLookup.Content)
247+
{
248+
StreamProgressCopy copy = new StreamProgressCopy(fileStream, zipEntryStream, copyStepSize);
249+
copy.OnProgress += progressEvent;
250+
await copy.Work();
271251
}
272252

273253
_log.LogDebug($"Added file \"{file.Path}\" to archive for package \"{packageId}\"");

‎src/Tetrifact.Core/ISettings.cs

-5
Original file line numberDiff line numberDiff line change
@@ -135,11 +135,6 @@ public interface ISettings
135135
/// </summary>
136136
IEnumerable<string> AccessTokens { get; set; }
137137

138-
/// <summary>
139-
/// If true, package data will be compressed on storage. This will consume less disk space, but required more CPU power to read packages.
140-
/// </summary>
141-
bool StorageCompressionEnabled { get; set; }
142-
143138
/// <summary>
144139
/// Zip compression for downloadable builds. Allowed values Optimal | Fastest | NoCompression | SmallestSize.
145140
/// </summary>

0 commit comments

Comments
 (0)