Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: added CreateIfNotExists option #47

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

feat: added CreateIfNotExists option #47

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 6 additions & 5 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,11 +32,11 @@ PackageReference
Use one the following ways to initialize `CosmosDbStorage`

```csharp
GlobalConfiguration.Configuration.UseAzureCosmosDbStorage("<url>", "<authSecret>", "<databaseName>", "<collectionName>");
GlobalConfiguration.Configuration.UseAzureCosmosDbStorage("<url>", "<authSecret>", "<databaseName>", "<containerName>");

------------------------------------------------

Hangfire.Azure.CosmosDbStorage storage = Hangfire.Azure.CosmosDbStorage.Create("<url>", "<authSecret>", "<databaseName>", "<collectionName>");
Hangfire.Azure.CosmosDbStorage storage = Hangfire.Azure.CosmosDbStorage.Create("<url>", "<authSecret>", "<databaseName>", "<containerName>");
GlobalConfiguration.Configuration.UseStorage(storage);
```

Expand All @@ -46,14 +46,15 @@ Hangfire.Azure.CosmosDbStorageOptions options = new Hangfire.Azure.CosmosDbStora
{
ExpirationCheckInterval = TimeSpan.FromMinutes(2),
CountersAggregateInterval = TimeSpan.FromMinutes(2),
QueuePollInterval = TimeSpan.FromSeconds(15)
QueuePollInterval = TimeSpan.FromSeconds(15),
CreateIfNotExists = true
};

GlobalConfiguration.Configuration.UseAzureCosmosDbStorage("<url>", "<authSecret>", "<databaseName>", "<collectionName>", cosmoClientOptions, options);
GlobalConfiguration.Configuration.UseAzureCosmosDbStorage("<url>", "<authSecret>", "<databaseName>", "<containerName>", cosmoClientOptions, options);

------------------------------------------------

Hangfire.Azure.CosmosDbStorage storage = Hangfire.Azure.CosmosDbStorage.Create("<url>", "<authSecret>", "<databaseName>", "<collectionName>", cosmoClientOptions, options);
Hangfire.Azure.CosmosDbStorage storage = Hangfire.Azure.CosmosDbStorage.Create("<url>", "<authSecret>", "<databaseName>", "<containerName>", cosmoClientOptions, options);
GlobalConfiguration.Configuration.UseStorage(storage);
```

Expand Down
17 changes: 17 additions & 0 deletions src/CosmosDbStorage.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
using System.Collections.ObjectModel;
using System.IO;
using System.Linq;
using System.Net;
using System.Reflection;
using System.Text;
using System.Threading;
Expand Down Expand Up @@ -150,7 +151,9 @@ public override void WriteOptionsToLog(ILog log)
info.AppendLine($" Region: [{Client.ClientOptions.ApplicationRegion}]");
info.AppendLine($" Max Retry Attempts On Rate Limited Requests: [{Client.ClientOptions.MaxRetryAttemptsOnRateLimitedRequests}]");
info.AppendLine($" Max Retry Wait Time On Rate Limited Requests: [{Client.ClientOptions.MaxRetryWaitTimeOnRateLimitedRequests!.Value}]");
info.AppendLine($" Create Storage If Not Exists: [{StorageOptions.CreateIfNotExists}]");
info.AppendLine($" Counter Aggregator Max Items: [{StorageOptions.CountersAggregateMaxItemCount}]");
info.AppendLine($" Transactional Lock Timeout: [{StorageOptions.TransactionalLockTimeout}]");
info.AppendLine($" Counter Aggregate Interval: [{StorageOptions.CountersAggregateInterval}]");
info.AppendLine($" Queue Poll Interval: [{StorageOptions.QueuePollInterval}]");
info.AppendLine($" Expiration Check Interval: [{StorageOptions.ExpirationCheckInterval}]");
Expand Down Expand Up @@ -239,6 +242,20 @@ public static async Task<CosmosDbStorage> CreateAsync(CosmosClient cosmosClient,

private async Task InitializeAsync(CancellationToken cancellationToken = default)
{
if (!StorageOptions.CreateIfNotExists)
{
// check if container exists within database
try
{
Container = await Client.GetContainer(databaseName, containerName).ReadContainerAsync(cancellationToken: cancellationToken);
return;
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cannot exit from here, as there are some dependencies that need to be created on the container.

Copy link
Author

@f1nzer f1nzer Aug 11, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The idea behind this change is that I have a hangfire server app (responsible for job processing) and a hangfire client app (job scheduling only).

As for the hangfire server, I want to use CreateIfNotExists=true to preserve my database up-to-date with possible changes on this library side.

On the other hand, my clients are just clients, and I want to make them:

  1. use ManagedIdentity (now it supports only data-plane operations)
  2. have a fast startup/first query time (no need to upload multiple stored procedures and so on)

There is a DistributedCache library where the same flag is implemented: https://github.com/Azure/Microsoft.Extensions.Caching.Cosmos/blob/master/src/CosmosCache.cs#L447
Of course, there are no stored procedures, but the database might also be broken due to this flag (incorrect pk, etc.)

Copy link
Author

@f1nzer f1nzer Aug 11, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Anyway, another option (suggestion) is to store some kind of hash, that is based on a procedure's content as a single entry in a database (something similar to EF migration tables). So, if the hash entry is missing or the hash value differs from the calculated value then there is no need to upload stored procedures. It will use a bit more CPU time and 1 GET request instead of calling Get + Create/Replace for every procedure.

Using this hash-based approach we could get rid of several (non-friendly for RBAC) calls and speedup initialization in general.

}
catch (CosmosException ex) when (ex.StatusCode == HttpStatusCode.NotFound)
{
throw new InvalidOperationException($"Cannot find an existing container named {containerName} within database {databaseName}");
}
}

// create database
logger.Info($"Creating database : [{databaseName}]");
DatabaseResponse databaseResponse = await Client.CreateDatabaseIfNotExistsAsync(databaseName, cancellationToken: cancellationToken);
Expand Down
6 changes: 6 additions & 0 deletions src/CosmosDbStorageOptions.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,12 @@ namespace Hangfire.Azure;
/// </summary>
public class CosmosDbStorageOptions
{
/// <summary>
/// Gets or sets a value indicating whether initialization will check for the Container existence and create it if it doesn't exist.
/// </summary>
/// <value>Default value is true</value>
public bool CreateIfNotExists { get; set; } = true;

/// <summary>
/// Get or set the interval timespan to process expired entries. Default value 30 minutes.
/// Expired items under "locks", "jobs", "lists", "sets", "hashs", "counters", "state" will be checked
Expand Down