From 1d9194465aaa9476a07ed2a37223a69f3cb3fa75 Mon Sep 17 00:00:00 2001 From: Thomas Gummerer Date: Mon, 17 Nov 2025 16:53:19 +0100 Subject: [PATCH 01/14] Add journaling blog post Add blog post for journaling (https://github.com/pulumi/pulumi/issues/13502) --- content/blog/journaling/index.md | 311 +++++++++++++++++++++++++++++++ content/blog/journaling/meta.png | Bin 0 -> 24665 bytes 2 files changed, 311 insertions(+) create mode 100644 content/blog/journaling/index.md create mode 100644 content/blog/journaling/meta.png diff --git a/content/blog/journaling/index.md b/content/blog/journaling/index.md new file mode 100644 index 000000000000..b08beb42f531 --- /dev/null +++ b/content/blog/journaling/index.md @@ -0,0 +1,311 @@ +--- +title: "Speeding up Pulumi deployments by up to 10x" + +date: 2025-12-08T17:57:55+02:00 + +draft: false + +# Use the meta_desc property to provide a brief summary (one or two sentences) +# of the content of the post, which is useful for targeting search results or +# social-media previews. This field is required or the build will fail the +# linter test. Max length is 160 characters. +meta_desc: Learn how journaling helps speed up your deployments in large stacks up to 10x, while maintaining full data integrity guarantees throughout the update + +# The meta_image appears in social-media previews and on the blog home page. A +# placeholder image representing the recommended format, dimensions and aspect +# ratio has been provided for you. +meta_image: meta.png + +# At least one author is required. The values in this list correspond with the +# `id` properties of the team member files at /data/team/team. Create a file for +# yourself if you don't already have one. +authors: + - thomas-gummerer + +# At least one tag is required. Lowercase, hyphen-delimited is recommended. +tags: + - journaling + - performance + - data-integrity + + +# The social copy used to promote this post on Twitter and Linkedin. These +# properties do not actually create the post and have no effect on the +# generated blog page. They are here strictly for reference. + +# Here are some examples of posts we have made in the past for inspiration: +# https://www.linkedin.com/feed/update/urn:li:activity:7171191945841561601 +# https://www.linkedin.com/feed/update/urn:li:activity:7169021002394296320 +# https://www.linkedin.com/feed/update/urn:li:activity:7155606616455737345 +# https://twitter.com/PulumiCorp/status/1763265391042654623 +# https://twitter.com/PulumiCorp/status/1762900472489185492 +# https://twitter.com/PulumiCorp/status/1755637618631405655 + +social: + twitter: Speeding up your Pulumi deployments by 10x + linkedin: Want faster Pulumi deployments on big stacks? Read on how we improved the performance of Pulumi deployments with lots of resources by up to 10x, and how you can get access to it today. + +# See the blogging docs at https://github.com/pulumi/docs/blob/master/BLOGGING.md +# for details, and please remove these comments before submitting for review. +--- + +Pulumi saves a snapshot of the current state of your cloud infrastructure at every deployment, and also at every step of the deployment. This means that Pulumi always has a current view of the state even if there are crashes during an operation. However this comes with a performance penalty especially for large stacks. Today we're introducing an improvement that can speed up deployments up to 10x. Read on for benchmarks and some technical details of the implementation. + + + +## Benchmarks + +Before getting into the more technical details, here's a number of benchmarks demonstrating what this new experience looks like. To run the benchmarks end we picked a couple of Pulumi projects, one that can be set up massively parallel, which is the worst case scenario for the old snapshot system, and another one that looks a little more like a real world example. Note that all of these benchmarks were conducted in Europe connecting to Pulumi Cloud, which runs in `us-west-2`, so exact numbers may vary on based on your location and internet connection. This should however give a good indication of the performance improvements. + +We're benchmarking two somewhat large stacks, both of which are or were used at Pulumi. The first program sets up a website using AWS bucket objects. We're using the [example-ts-static-website](https://github.com/pulumi/examples/tree/master/aws-ts-static-website) example here, but expand it a little bit to set up what is a version of our docs site. This means we're setting up more than 3000 bucket objects, with 3222 resources in total. + +The time for the benchmarks is used in the console using the `time` built-in command, and we're capturing the network traffic using `tcpdump`, and then use `tshark` to count the bytes. + +| | Time | Bytes sent | +|--------------------|--------|------------| +| Without journaling | 58m26s | 14MB | +| With journaling | 04m31s | 10.2MB | // TODO: rerun this test with the final implementation +| Skip checkpoints | 01m33s | 0.5MB | + +The second example is setting up an instance of the Pulumi app and API. Here we'll have an example that's a bit more dominated by the cost of setting up the actual infrastructure in the cloud, but we still have a very noticeable improvement in the time it takes to set up the stack. + +| | Time | Bytes sent | +|--------------------|------|------------| +| Without journaling | | | // TODO +| With journaling |9m45s | | // TODO +| Skip checkpoints |8m39s | 2MB | + +Note that this feature is still behind a feature flag, but we are ready for testers. To get enrolled in the feature flag, please reach out to us, either on the [Community Slack](https://slack.pulumi.com/), or through our [Support channels](https://support.pulumi.com/hc/en-us). Once that's done, all you need to do is to set the `PULUMI_ENABLE_JOURNALING` environment variable to `true`, and your deployments will start finishing faster. + +If you are interested in the more technical details read on! + +## Introduction into snapshotting + +Pulumi keeps track of all resources in a stack in a snapshot. This snapshot is stored in the chosen backend, either in Pulumi Cloud, or in a DIY backend. Further deployments of the stack then use this snapshot to figure out which resources need to be created, updated or deleted. + +To make sure there are never any resources that are not tracked, even if a deployment is aborted unexpectedly (for example due to network issues, power outages, or bugs), Pulumi creates a new snapshot at the beginning and at the end of each operation. + +At the beginning of the operation, Pulumi adds a new "pending operation" to the snapshot. Pending operations declare the intent to mutate a resource. If a pending operation is left in the snapshot (in other words the operation started, but Pulumi couldn't record the end of it), in the next operation Pulumi asks the user to check the actual state of the resource, and then either removes it from the snapshot, or imports it depending on the users input. This is because it is possible that the resource has been set up correctly, or it is possible that the resource creation failed. If Pulumi aborted midway through the operation it's impossible to know which it is. + +Once an operation finished, the pending operation is removed, as we now know the final state of the resource, and the final state of the resource is updated in the snapshot. + +There's also some additional metadata that is stored in the snapshot, that is only updated infrequently. + +Here's how the snapshot looks in code. This snapshot is serialized and sent to the backend. `Resources` holds the list of known resource state, and is updated after each operation finishes, and `PendingOperations` is the list of pending operations described above. + +```go +type Snapshot struct { + Manifest Manifest // a deployment manifest of versions, checksums, and so on. + SecretsManager secrets.Manager // the secrets manager to use when serializing this snapshot. + Resources []*resource.State // all resources and their associated states. + PendingOperations []resource.Operation // all currently pending resource operations. + Metadata SnapshotMetadata // metadata associated with the snapshot. +} +``` + +Before we dive in deeper, we also need to understand a little bit about how the Pulumi engine works internally. Whenever a Pulumi operation, e.g. `pulumi up`, `pulumi destroy`, `pulumi refresh` etc. is run, the engine internally generates a series of steps, to create, update, delete etc. resources. This series of steps is then executed. To maintain the correct relationship, the steps need to be executed in a partial order, where all steps whose dependent steps have been executed already can be executed in parallel. + +As each step is responsible for updating a single resource, we can generate a snapshot of the state before each step starts, and after it completes. Before each step starts, we create a pending operation, and add it to the `PendingOperations` list. After that step completes, we remove the pending operation from that list, and update the `Resources` list, either adding a resource, removing it, or updating it, depending on the kind of operation we just executed. + +After this introduction, we can dive into what's slow, how we fixed it, and some benchmarks. + +## Why is it slow? + +To make sure the state is always as up-to-date as possible, even if there are any network hiccups/power outages etc., a step won't start until the snapshot that includes the pending operation. Similarly an operation won't be considered finished until the snapshot with an updated resources list is confirmed to be stored in the backend. + +To send the current state to the backend, we simply serialize it as a JSON file, and send it to the backend. However, as mentioned above, steps can be executed in parallel. If we uploaded the snapshot at the beginning and end of every step with no serialization, there would be a risk that we overwrite the a new snapshot with an older one, leading to incorrect data. + +Our workaround for that is to serialize the snapshot uploads, uploading one snapshot at a time. This gives us the data integrity properties we want, however it can slow step execution down, especially on internet connections with lower bandwidth, and/or high latency. + +This impacts performance especially for large stacks, as we upload the whole snapshot every time, which can take some time if the snapshot is getting big. For the Pulumi Cloud backend we improved on this a little [at the end of 2022](https://github.com/pulumi/pulumi/pull/10788). We implemented a diff based protocol, which is especially helpful for large snapshots, as we only need to send the diff between the old and the new snapshot, and Pulumi Cloud can then reconstruct the full snapshot based on that. This reduces the amount of data that needs to be transferred, thus improving performance. + +However the snapshotting is still a major bottleneck for large Pulumi deployments. Having to serially upload the snapshot twice for each step does still have a big impact on performance, especially if many resources are modified in parallel. + +## Fast, but lacking data integrity? + +As long as Pulumi can complete its operation, there's no need for the intermediate checkpoints. It is possible to set the `PULUMI_SKIP_CHECKPOINTS` variable to a truthy value, and skip all the uploading of the intermittent checkpoints to the backend. This, of course, avoids the single serialization point we have sending the snapshots to the backend, and thus makes the operation much more performant. + +However it also has the big disadvantage that it's compromising some of the data integrity guarantees Pulumi gives you. If anything goes wrong during the update, Pulumi has no notion of what happened until then, potentially leaving orphaned resources in the provider, or leaving resources in the state that no longer exist. + +Neither of these solutions is very satisfying, as the tradeoff is either performance or data integrity. We would like to have our cake and eat it too here, and that's exactly what we're doing. + +## Enter journaling + +To achieve this, we went back to the drawing board, and asked ourselves, "What would a solution look like that's both performant *and* preserves data integrity throughout the update?". + +Making that happen is possible because of three facts: + +- We always start with the same snapshot on the backend and the CLI. +- Every step the engine executes affects only one resource. +- We have a service that can reconstruct a snapshot from what is given to it. + +(The third point here already hints at it, but this feature is only available and made possible by Pulumi Cloud, but not on the DIY backend). + +What if instead of sending the whole snapshot, or a diff of the snapshot, we could send the individual changes to the base snapshot to the service, which could then apply it, and reconstruct a full snapshot from it? This is exactly what we are doing here, in the form of what we call journal entries. Each journal entry has the following form: + +```go +const ( + JournalEntryKindBegin JournalEntryKind = 0 + JournalEntryKindSuccess JournalEntryKind = 1 + JournalEntryKindFailure JournalEntryKind = 2 + JournalEntryKindRefreshSuccess JournalEntryKind = 3 + JournalEntryKindOutputs JournalEntryKind = 4 + JournalEntryKindWrite JournalEntryKind = 5 + JournalEntryKindSecretsManager JournalEntryKind = 6 + JournalEntryKindRebuiltBaseState JournalEntryKind = 7 +) + +type JournalEntry struct { + // Version of the journal entry format. + Version int `json:"version"` + // Kind of journal entry. + Kind JournalEntryKind `json:"kind"` + // Sequence ID of the operation. + SequenceID int64 `json:"sequenceID"` + // ID of the operation this journal entry is associated with. + OperationID int64 `json:"operationID"` + // ID for the delete Operation that this journal entry is associated with. + RemoveOld *int64 `json:"removeOld"` + // ID for the delete Operation that this journal entry is associated with. + RemoveNew *int64 `json:"removeNew"` + // PendingReplacementOld is the index of the resource that's to be marked as pending replacement + PendingReplacementOld *int64 `json:"pendingReplacementOld,omitempty"` + // PendingReplacementNew is the operation ID of the new resource to be marked as pending replacement + PendingReplacementNew *int64 `json:"pendingReplacementNew,omitempty"` + // DeleteOld is the index of the resource that's to be marked as deleted. + DeleteOld *int64 `json:"deleteOld,omitempty"` + // DeleteNew is the operation ID of the new resource to be marked as deleted. + DeleteNew *int64 `json:"deleteNew,omitempty"` + // The resource state associated with this journal entry. + State *ResourceV3 `json:"state,omitempty"` + // The operation associated with this journal entry, if any. + Operation *OperationV2 `json:"operation,omitempty"` + // If true, this journal entry is part of a refresh operation. + RebuildDependencies bool `json:"isRefresh,omitempty"` + // The secrets manager associated with this journal entry, if any. + SecretsProvider *SecretsProvidersV1 `json:"secretsProvider,omitempty"` + + // NewSnapshot is the new snapshot that this journal entry is associated with. + NewSnapshot *DeploymentV3 `json:"newSnapshot,omitempty"` +} +``` + +These journal entries encode all the information needed to reconstruct the snapshot from them. Each journal entry can be sent in parallel from the engine, and the snapshot will still be fully valid. All journal entries have a Sequence ID attached to them, and they need to be replayed in that order on the service side to make sure we get a valid snapshot. It is however okay to replay with journal entries that have not yet been received by the service, and whose sequence ID is thus missing. This is because the engine only sends entries in parallel whose parents/dependencies have been fully created and confirmed by the service. + +This way we make sure that the resources list is always in the correct partial order that is required by the engine to function correctly, and for the snapshot to be considered valid. + +The algorithm looks as follows: + +``` +# Apply snapshot writes. This replaces the full snapshot we have on the service. +# We do this if default providers change, because we don't emit steps for that, as +# we do for the rest of the operations. +snapshot = find_write_journal_entry_or_use_base(base, journal) + +# Track changes +deletes, snapshot_deletes, mark_deleted, mark_pending = set(), set(), set(), set() +operation_id_to_resource_index = {} + +# Process journal entries. This is the main algorithm, that adds new resources +# to the snapshot, removes existing ones, deals with refreshes, and operations +# that update outputs. +incomplete_ops = {} +has_refresh = false + +index = 0 +for entry in journal: + match entry.type: + case BEGIN: + incomplete_ops[entry.op_id] = entry + + case SUCCESS: + del incomplete_ops[entry.op_id] + + if entry.state and entry.op_id: + resources.append(entry.state) + operation_id_to_resource_index.add(entry.op_id, index) + index++ + if entry.remove_old: + snapshot_deletes.add(entry.remove_old) + if entry.remove_new: + deletes[remove_new] = true + if entry.pending_replacement: + mark_pending(entry.pending_replacement) + if entry.delete: + mark_deleted(entry.delete) + has_refresh |= entry.is_refresh + + case REFRESH_SUCCESS: + del incomplete_ops[entry.op_id] + has_refresh = true + if entry.remove_old: + if entry.state: + snapshot_replacements[entry.remove_old] = entry.state + else: + snapshot_deletes.add(entry.remove_old) + if entry.remove_new: + if entry.state: + deletes[entry.remove_new] = true + else: + resources.replace(operation_id_to_resource_index(entry.remove_new), entry.state) + case REFRESH_SUCCESS: + resources.replace(operation_id_to_resource_index(entry.remove_new), entry.state) + case FAILURE: + del incomplete_ops[entry.op_id] + + case OUTPUTS: + if entry.state and entry.remove_old: + snapshot_replacements[entry.remove_old] = entry.state + if entry.state and entry.remove_new: + resources.replace(operation_id_to_resource_index(entry.remove_new), entry.state) + +deletes = deletes.map(|i| => operation_id_to_resource_index[i]) + +# Now that we have marked all the operations, and created a new list of resources, we can +# go through them, and merge the list of new resources and old resources from the snapshot +# that remain together. +for i, res in resources: + if i in deletes: + remove_from_resources(resources, i) + +# Merge snapshot resources. These resources have not been touched by the update, and will +# thus be appended to the end of the resource list. We also need to mark existing resources as +# `Delete` and `PendingReplacement` here. +for i, res in enumerate(snapshot.resources): + if i not in snapshot_deletes: + if i in snapshot_replacements: + resources.append(snapshot_replacements[i]) + else: + if i in mark_deleted: + res.delete = true + if i in mark_pending: + res.pending_replacement = true + resources.append(res) + +# Collect pending operations. These are stored separately from the resources list +# in the snapshot. +pending_ops = [op.operation for op in incomplete_ops.values() if op.operation] +pending_ops.extend([op for op in snapshot.pending_ops if op.type == CREATE]) + +# Rebuild dependencies if necessary. Refreshes can delete parents or dependencies +# of resources, without affecting the resource itself directly. We need to now remove +# these relationships to make sure the snapshot remains valid. +if has_refresh: + rebuild_dependencies(resources) +``` + +The full documentation of the algorithm can be found in our [developer docs](https://pulumi-developer-docs.readthedocs.io/latest/docs/architecture/deployment-execution/state.html#snapshot-journaling). + +### Rollout + +Pulumi state is a very central part of Pulumi, so we wanted to be extra careful with the rollout to make sure we don't break anything. We did this in a few stages: + +- We implemented the replay interface inside the Pulumi CLI, and ran it in parallel with the current snapshotting implementation in our tests. The snapshots were then compared automatically, and tests made to fail when the result didn't match. +- Since tests can't cover all possible edge cases, the next step was to run the journaler in parallel with the current snapshotting implementation internally. This was still without sending the results to the service. However we would compare the snapshot, and send an error event to the service if the snapshot didn't match. In our data warehouse we could then inspect any mismatches, and fix them. Since this does involve the service in a minor way, we would only do this if the user is using the Cloud backend. +- Next up was adding a feature flag for the service, so journaling could be turned on selectively for some orgs. At the same time we implemented an opt-in environment variable in the CLI (`PULUMI_ENABLE_JOURNALING`), so the feature could be selectively turned on by users, if both the feature flag is enabled and the user sets the environment variable. This way we could slowly start enabling this in our repos, e.g. first in the integration tests for `pulumi/pulumi`, then in the tests for `pulumi/examples` and `pulumi/templates`, etc. +- Allow users to start opting in. If you want to opt-in with your org, please reach out to us, either on the [Community Slack](https://slack.pulumi.com/), or through our [Support channels](https://support.pulumi.com/hc/en-us), and we'll opt your org into the feature flag. Then you can begin seeing the performance improvements by setting the `PULUMI_ENABLE_JOURNALING` env variable to true. +- Enable this for everyone. After some time with more orgs opted in, we'll enable this feature for everyone using Pulumi Cloud, and by default in the Pulumi CLI. + +## What's next + +While these performance improvements hopefully make your day to day use of Pulumi quicker and more enjoyable, we're not quite done here. We're looking at some other performance improvements, that will hopefully speed up your workflows even more. diff --git a/content/blog/journaling/meta.png b/content/blog/journaling/meta.png new file mode 100644 index 0000000000000000000000000000000000000000..44a346913c3498ce31b90f943b1e2830cd0ca816 GIT binary patch literal 24665 zcmeFZcT|&I_CFeIU`J7U6+xwh-m8G1R22jSLJvKJ5<&-+rqWb|&=C<3Y0?d$Nk^Ix zAata6kP?s*xCh^N-kG^;=KGubyZ5g9#~qfgBu~zB&OZC>`q}%*i~E`?=g-iefj}VV z@7+~;2!WjV41pZer#J?#kOH4;fS1!wcOSb#AZMwL{*yuC6KTPWWv=OoXz*(|DZ*`R?|Qkw zwY@ZTEWGS3BrLDV$+Aj&NP-CvaFiLV2g1S8RnkM|+V6QK!TY1jeAifi4?)?>T)Ta= zAgjK{eO5)J3!GI1D#~jiAS}WvCIJ-?6%v+^;9(Wy7ZB&;7v~cY;uR2;6y}!{6lDGL z?;2Rm#nMXhp_1~SYk^NP*KANICrLg&cXxNFyATxVV$CNYAtAxXFUTh-$O}gBx_UaI z%shA?_sk-t%+XFa0I@`x%? zxP+LMkPxqgrGyx-u$iSWcqbyt3%9g16O|A!hYMN>USs|Hx{62#q>Bd967XE;NFRR7 zrQ}^(S8#rwe`=UE-1*OU4z{enMN!hs;z(O$u2~$NG~DvqpOIK-|hoTtHOR(#qm*`~PS9|AjpMZ`1!LgDq^#9IfHN2=iV0w>z>xI-=n&|8Oc! zW-eyH_QG9UWv*GdAQ7x)PEHQC7G_6B$cJ{c{L`fUB^0bEBPmMWvfk6Ut#XHzRX~uRzYZhH&(8{CAl84h zY`>>DTJwKnntzb_e@WNxxmo|okp9DRzg6@1f&Z~XzQ51$PdxZNMFQKtyx& z_ODP1eE2IEgF6D{cLAXm#eTyi1fqNIp3-d{kHqDXMQ{3nr>lFtr&Ld!xcg4(4ZNk$ z0lSQzKxN?PSK_XFb6U>~n+oNz2)X}uX#N=FiGf|Ooh+u8 z_|o{67=5^130De1Sc8E9w?I63?CbaX4@E)6ngnjI^zM%Ym}8j=-bXg2wZVk($2y*< z%9g@SeKuNkM`JSF!`=C;N(Sj*tsq-33Cj9Jmt~1#T@ynFfmSQ0C2M3xM!Ke(uhr6+ zIKxhBsCTsAQOR5HvFX8QSqggIhjDXlUN&#iS(U9NRHA$Ri<&AQ_9P29AZ62p{Y^2cc9Uqj9*PngO%dwj=pvO7OMe&y>2 zLCAN$>4)o z-`T9I>)i}7lo8f{+ zf3ty!w30q;U~%`$^|pR zVZ@!<*YonMHv`3GB<`hO_T{@*`C5}B#IkWKK4?=)r+`{MUlTR5tNAH2v(Nol%^7g4 zMn5V_PK*hU&XwmadO@RJtMSr=Hjx&226>`Y)!W<3jg0EZ#OuDbzq2B(JS6*}pu-7y z7lq>TbfU!Z&pgWRtt8A?4=wiI7r6{%xT}Fy0V1yF z+94<`taOR|iET0EF?sJ70i-m`G4d1fy1BYURjh9IwCjK!QHNsO1*=FE<#+h2zc*VG zbD@S~)&(n~Q8YS^UnCAZWQJq_PQD9w;7k&ib}GL@}Q18vUGlkdRpsENU`c_B-i7hoKK2`Qt{%tcvR^pW?`m3E&E%|} zcybklW;W4x>s5Rwa1RTUNry?nPCzh^St%3a<3!4q_Z8mG z+%}~Jg7!lux*Kl8{8qK$JuFn^xV&MgT5HtnWOAwwKBc(5Gqjh%G9I1jw#;wmM9#b$ z1F906U@~_={BYa!;oA42Xc1@H{N!Y`$AAYbzZA>) z6VHxw(>^&1IQB!vv8-wST7SzjR)>YW=KC3yjM~>(K2SsP4vcEUjOCmQP9e3WrTNMO zN?L`Zr6UQ8G99yt&>+OVpITJyad|CH&N`|nGq(B!v2f)|wm>H=OfTPG5q4tTCSC=9H=yQKS`het(glmmLo&{~VvS~|lrZjJR3ciJ z;}aq^JKH2+AXw(_0qXSqTbix5P*LNALE0CW5z{PL5Tqm_0g?6K2;Co=orFx z6B-ku@KL4s9WUW!7b90mBZKMUM|w?p#u+==z?P>!jA7|ZvyE55y~Qy7aG_mp<%Wb> zP8UABMfmin@?qXqVbwFjVsB&Ku8-`Phv}4EdbqMok}79Z2$CLYa9}HVIW8g8IHFt) zt0O7>Jo;rr%xtCc>gJMQ`x-@1C1s7zW)&&YjUm8UwOE}0-9 z={-Pq@dPj+YOVuh(5Q=BmpwCEA*S!>!*`u!O}MC48x*uoe-_Foc=ygG#8QNCaxM>A zr@U=VA6d@rNb&V!layj%(tqthu<3DkFFo6|b0Hth7n7H(f9`UL9kC^CwH;yKdRsH| z&0fLC1-|dXi0MX+{7&RrI3wz`Ux6U&v`d_VtaU4V_T{1~E9Dst^;vh=*f(ObQY5x5rRNxrfyO)1+bxDT? zL)U}_=;C7;JEXp19jx4EABp8NuG6E*GjH5DatRrOxweDh31zuGnMvmZfmH&&L@d7o z?dvA)|B8kSqjofi6Q-_wt(Aa0_wKIDEsQD#N<2Boyo=c}vXMogQm%!^q8 zaLH&?g|}Gvi?J_K#l>2@cjOJc8GR;P*;l9b zVONjR?Qk!4z#83Y9!yJY|F^H7EfqvvEzPqZ7R)vCq5Jcs1*r$6E`RCzc;mvwh&QKk zXq(p+A2Z&BaA%7#Ej8bVV$>aaKZ+?Yo;vUJB3$_@xx83;+<3b)h_D8tSQz7ORvUro z=!wed59y)})Mjn(NYVDo^BK~fT_f5R%VrcPSmj4-?A$J;7ojJT`tz{QN0A}qYwEkr z6K(gpon_1Rc6(o;a5m`oi=D~)KYm=-pIIq6&32|~=K;3(7={5b_j>TB$hjbwICZQ( zqosAGa++~|Z#!ZLUAZ)_BO@bHW|nSLT6WzRzRO_S!%$-9rW120+vVoU(D#Yc{}ksr z!@3`jd+mD7jqQXN*7}V_qe{z6w+aPzw>}x?#|{0|NHas;gTyMbo=_gWD|{}QLrUsu zXj6bKSMlX&^2e2>&8YUAF8;Wf)WT#=FSow&+ZFmev2*#O?blu@zBkBqtn#=FGh+Vv zQ2{Szqp78}#CsaznkNo}Kw_o}<7hfrB8}*Ufh_6qrKBX^=Y=TWOM5Uj=p)KydmqG6 zmBt8{0d=ibKk>W=$fe3rP3A4>>C2GlOB7EpQICxcjnFj;@XEYqD(}wvfImt0JsKYs zMWdy%F8kI&anYsCaKhAIiSIMuXObSD?H2iWj$>^6Z>E-a#^aVc%1QlD_mZdMz>EqS zgj?~_6B|3SF*Ku3=~G8PBS4gD$d~;3G#M&Do+2!{t5Rlc0K^U3L~c9&n(y>PVVgN0 za=RYKl&WGM9MLZim>MEMh;0}Zo3Sw~Bge!6_Pon?^ga_6KI zh@<&cp^t-dcwvOBuaLT#hsHiz7C7F}dqx?;H(R(r;nROR!kXqUZ?6`XHV4++w-sK% z5GSa{#(KGtG~?=-Z!kRPRm<~MtNf->w;<>Ez>M@g_X6JJEqdp>Cqhp`*aG`U2C8ys zSR|U-fPKUo4Mr|Y!V>*j;DT*~Cx1wvP%u4VG%>Wn1bi{(i{GiRm-S!$#>H1rw{|}o z3V^#2Bo%+b0MWR=lv7Ag7xf3kB za}Jfr!Rgjo^UP)74}C(=Qh*FrLD>({aFSLoq;I z9%kvw$SiBG0cKkmBUvez@CRgxnOz^Zvqsvc(LoAPlTUxtPfdAZl8Ok4<^}h~J6L92 zNw^>V_R^FTL7X=@>#XHMr!FD#vK!3VqBNt`IEGrab9F7erR*0ON>}M>?y^%JQ_MA5 zAS`=ghOS1alPgkX6ve3EIg5YRTk2E1?u5Gzv^mRK+tZDCRgzpteIdiWr%v%WdzgBU zoj^G7Fh;bswYz^Ip_1iUX(>*mg%VyiC_tke<5*E1`q*+WVP3{K5VYBW`1Nbma@i0t zyOPl#hE6`Q0_W+Gl$^M<((ms*I*r!Q>Wt5Rm1J!$C@nx2BSJf5ZyW9vfE%oN0tPX1 zP>HVGXJAN3hzw@e?sMme8sD!qmxGE^Yjk$B^C{(;z+~dl%e!{D2JIiT=TV=Gix7R& zb#aquo}ctF-C2);_A^l7p)X=M^M^v7UxC}F z3RyFoQ<7w+P{PDi8mENsZQ2YD*P2_}H8zIYrHzV9Kj(w;4i$<)Fc6?P4+qo-M^ghP zpVq7s8D`JJ3tnG3YvL$dR1kyq_yWBBocNh@^1wDNI=&SZEiJC-t~RR~Z`NU@L{((* zm!l9~R(V=dV((XP`*g-+)S|fPUZ{Ke`Yqy)Ih_WH2MxEFErgD~~GkpS(EW7wh9d#RSjE}vRjr_m$&)g#ryHWVru zjw4ms;o$>?Wxo!RdH2YiPE%4mHG8X>=};l#?>#vR;o?TL^YV=9S0nV!XXx;3*^bgf zPT}IoGp%R)`owm7Zj*Ad;szKN%SMu)qJ7`c^gq=ISn4WH03x)h3Y;d78io8dAV*D& zSVje^sIt{zCQD$SpUVH7*$qaxmssG!gAq~O5b@TSnXfxRHmJ3DB5CtFCB^6$i7)CO z!^1}y9Xijl1*+it97jqys5mc9f;8{PWmYifjn|)NBR4mfhJm|1j*}UC8tz__-D@Wq zU0lTiK@4~raHaX4;f|<`Cf+>JQ z=~Ns$-AyN9%Q{@QIZn+*sY}A0FHSyvQbfa&PjCkUWHMGejh2W$HAY=CZ45Ik8{K%e zXAe06&RVoo_#zNK&5`6)`O)4|;{f27bj+Xlfs_@5R=Qj7!ozt@s?GE~)d7yNTdvwZ z?9(%l_e+5yG9!2Ri>>pGthY&O8pmh>mTvmIzZ=+!y**#|VPQaMtA_&p>8E~o4IP<{ z42+$nZ{6EC`v4aVEdytl7@v_HL|7~mF9YIdYDWh-x?La1+|$^otD8G2w8Cw;Vkfo` zKIt6bCHsf+n9(Ts#b?daTfpLLv*g@Jw22Uz^BA6L#KU@!tHF-Yx zf+u8QbIA_c6N<LUAn~H+w%epX>0mj1`N}zs zem%+rjm|+KvbOKR#%9>p_w(N(c+Nd}4I+WyXANBeiXs@r?F{SL9QEUOwQkfvskPDv z20U@9=2GAs3IfWn(De!5d~AEYhj?Z~^f=d%b^NMo&C6E*xqbTkkx&(!rIty87$z>1 z&m)?SKV@T->E|(5$?tNq9Ng91ZJOPPTL~QdD3;T9E)~F7w3i@PW}F2zMZJr&{zwhf9ZIua&i!{-luy? z>69Ez7MOZ^BS1O0Y09#Gvs69yg1%+B=Y5KC;tqo6-l4v&3`4Z;eYU`%Wq~p_-&HaY z2a$z5ZV>_}|EkmVdwR?BAo;sz*-SMmjfEqOt+#s}EfQ4q6H)b%0XM#DGxS!V@`}%4gk=yp|c+Ja83m=@Ok4L#NV_NsS#gET$ zQXZRQ0-K7Bj*w&*=)8@|IoIX6F%urnLH5MTal08L_54qdD5+OCJN~iF_51n!0ce`S z^Y|c<5>P4AQD!~C*vBvBH7l475+YG6zrj-(iU2F-4>`Iid_Z1YVS^D>_wDV)zb>{P zTD$BI_4iNH$={4iULW7wtRH;?zP&E#A@QRumY)d%F|7e+>|?jGb?G(~3or)= zB;a3E>FVA$E?oNaJANK@tuG5M1;(J-_vEO;rd zzOlgUqnmzTp?BL@6!0n*U{C{t-d#Ax_9XQvVL&dX0z3p#bLIC19>#>xt%d&HvnH+$ z4byp42=!>)KCe0-!%bF!c__q`|05fj)98Fl%Zt2(p}uLf8(zT%&e^>%r_+A#5)A;h zoXaU-5)vXNnlmFOLE0xzQRAt*8|qbA1vQ!9ip}95@P;87`SAm8#n%F(HEW3m?Tv3X zBcmUbI7T)#0{%e700=k>@4kf}^71`^ns|a^)RV4!C|WlDVm z-9`j`bjQV!CbxI5CA7x^B7y^sK^Tvyyw2Sopzz$(=KUVcp3jVsR@&%I>a!VQEKhW} z%)EvGBG~xmTyXS>gm%XBp7~wCVQiK@mA~kI4*~&3*-3rEbWAlZMpU#W&GZGPyQ=hC{DbN<65Zdr8dZSiVO&lVB8;Dm2aD@fkA2|YPSyr(y zZpw3UegKjbFSd_2T=EwR`_$Xm&V4t@g5fNiyLP{ zD;>MFY4AmtU$7yr7x!<7es8_sHh$@hM*L{04$lQ^mLDKpnylCvFjpzq=7ZEc1}k5< z%eZ8i*q@fUbLYHlN~dbkLueFT?W3g}%{D5EFGlQGhotIl&Iv2>0vIs$X_X@g?t~C_ zJUaRWI`Qhnp0}=O*~om$i-wu6UExP&ntRQd z;1-%Bgut|aXys6Ab?G*_0{PAj=6DK(+LO4BZAh}Quo!QCi$$<&3+axQ1DYSbynB5@ zH=exVrV7R4+QCmJe~-o>gU&4wc+5}JkwLzJ;E6U-IqhI!+BK9+nL9Hm@VGOXADG&OB5$~8@*wKTo&cUoXAgZSQyxv zx9V}o{nJk=Ygn6wKQ5LNVeRcIyrOoG6YX^<9^~ql8B4qJQ-PYpk|Xl*@Qdi=vO0@z z1}vs*N&sX*zivTs+4|}rZVF4hk2+2UNd&g(yE{N-_>yHyInEEhF#{ol|M0>$?+c|o zEzYqWTNO;q_##(ohSNcmA>_`Pf@aW!g6R-T^Q+X5qj*ccI$A9qWZH&Wd~aZMjT8Ms zw=gXg9=kR*%Yo?h+o3YZkdW5t($Tv&8NUSr)W&@_nn(&{*JhWYzVZhvV?zaS$_$i*@qOC(Ic!kZu(M5#eR^LZ4kZfj6jRRmlyy=q>7ZY>+5!f$v zMWL;&txK8$@)ruky(XMiFu;VTTpP?&wdM!6lFJmQ zcfSpW&PFcVtw)FH&;7=QS2C>-!V5Qc??EvwRuQ|XXGVPv7a_FQ!4YfPt#TIUh%FGk zDs^-FKzS)MGu`}aUi5Xz1e51m-fn?@A1JZ_)NV9O^eQRHmQ7jxL~w7IcEOFTxZ6KZ zNlb1|Gsx0;*)Un9Y@agbk7ao4fLs^R0SO8x``$_f`7yo9DgngOr_LLJUnI9gmkA8S zg?Zbwu8%t3Zg{0XytPdnsu&(%hVb(PD~4!hb|mF>8QXZF9U)|mX||zi1QR5ZxXO!y z5!^`PDT`x@jD#cUFOy^I_@{tkPgo$i2V>_da@Jr(Qf3dOnj-! zWX6!|Y45K&uTPOdN`VdZurEsqm@A*v0hQ7(Mk^(TWqglYyGv z&(Y|cH?R@pAdIA7Qxc70Zp%qc3Idr~`t6o!mocNUFT!#BjB)h+;%#iG7QylLZ{aOJ z)0cbCL7dJ2&rhiU5Zff^v)JS$9NM?{tME-vPzSM8@RUx12Gv0|IIo{S2H6dir;rB@ zd9ppAE8V^hA2Z!mY{%aq~vw`nx6`60OjjCIJG*|(?$_0U0brmhAawskIGT9 z>Pe7@?<=5m;${CpApnRq@v~;Wuz1$N&&hjuW<7dnvY?Wm=FRCtVh6Bc^!qq>3xMud z`B}UQJ#Wl}v-2`cL0ZPARmxUlzjhDK@$B2|CKh(54z9AubrH#`Z$TK2&J~nb$~DKb z*W^vnvh)-u$bTAd#T)8}skfgq(@ve7m%I|+_90T;VQyPW2ob4*cN{H`YX1PRo77#= zq%&L08hGJ$C4F!{>^;p?6el7#B&Zm68;_ULWN00`9fj$0KAX~oVj-H^)NUpm`+OM?U^$}a_e-UwfdEBJTU6VBwrOqkv<46VOXrpcEW>XS6HJJtrMdKGSJ z4Ma|iAQo&r`fh#zdvTT2oMkoYRNr>h)C>JqvP4t4ZT~)yflK9E?LEp zsJS^DQ;?m8GJV>LuykA^o6C-i!y1W>WGdHoA+9=b|AxyF5hnDX6v4(xlarNOHH^WJ z1f=6R8T>9@+$xGI)@}f}zzhv_Qg{>apaAX40|&~y6L83A0&KscQjk; zo!LkhT?4kQN)^*8BO-#gkK7%=UXuD;!vQiC*1#OP&O6IVns@*uvg1LshD|R$&a%E% zT~egE%SQ3)$@8IvylStVial`w8L3&~&Yr=C6`(DH#NM5thDjf2wCXvYDco@0{QN*1 zcE^lXebEI=gUPEN9bNE?!tQ)5V(SY8c+a@_7)g&uYom&~1^ras$Tm(kY%Zb?v+J7tolqX20NTrLOgFs{b~N7 zCIZG}h{FoFxKlk*JIX*EJ|0W`{Oa`IFx4G8eaaSe9@2Ax(i)OfBvL3#Yw7pKHB?peUK`!+7pl!>v4j5AbIwN>8%g6urM z`yuOTn&SI}Il(El59rC-Q0#JROy8&8j*V@gy!7&{bQ>QjRJjk#1+{lk$3Hv-*VqDo+E^%#^7KWCA?bl9 z22OdoY`&`)m?)Sdf#X+2yB;LJ)7nD2`zJWOX_tF`k$Pa3x7($o6}E3%r^)=Ixd08_ zAd9)cFe6U^vfkEkxBvgOEHch)eMy2IJur6Eu*x`t_HILnqkHW^MGPeL^vBqKe17rb zo+|?=Q*Xr_btZf|IyF#*&~N}108ioety*wV{??J08Rzpc#x1yI1AX_qatd5l1!Frx zp)i0gmHU%^H-><#`bWqS=y(CG5~#S-vo!BP$<~kNH28+=2*mxGW(!Q^{$$e33ui10 zcvO((LLmLq0hLOl^MU}i1bs30GQXui+=zmf0TY-U7nhcK6AWm<|_^_qE%zsZU*8t(S&dN-6v7s@^8j`6&(YPQje_NtnT^bKZNfrHxRks1h!KH^ zN$5nomv$aRIrD?yxfGqfvnNPe4$Vj-kVKk(S=DK4Crv&UO%S+n_pIleZ=Ei_ZY8ef zs|QSmC|CXcgSDNxiJ-FLz!H4wo=kCNFe+iS)k#u%#|h0_6-mhBzV1gughrPYrZ!Y8-4Ph%})R%sk%`+b$Ax}bDr%cIn! zvTe37((f0P(bfbOy4p0wkULg|$CEo`OGvn*->OodX&Nbm=(|%!GSyoESFI69cCzV!yJm^zZvg3mwZM72(Nqe3s zTYkbT7wlVA&T{1Su;oh8HRxUeY|>ep+R*s%vY#Q0z~Z;9fjKRm0`x%Tx@66fvVT(yS4R(b)~WQ+R_V3bGCBJ z%1~3a$*FqFUxOu!?WDPvK?cXvjX;bAoDZ&i|k&LNvRL8 zO1bxMTH@y`N0>Y}CT>RWvs9Ci2&6m;>1DNLEqC~Shi=D$-+*=n~iK1jpCc7=m!`&O}!0StCW; zRcNedXTnpS<}EhVI$v{Um2&|Z?dnz(?4ay$`ITV3UIEY@$3Dx@pzYO<)XdFx<4=P&`DY6H*WyQVu zBp4%nws!cYSbB1EX{^01ubw?Qsa5iN@p{=1yJAvXV18Md;ri6LnGr<+B}}l?1t+u7 z)d1)uUn3Pg{Yl&^%~2f2?mfV;EK8T?F_fH$%qUs@Ku>Y%_*YYu^L$6hwq41tC$3P? zXm+V|-5+>WpAC#$i`UO0>&WoIPS0K=3#ZV|#WJ{XQYFsL$`_?AR5nC)L8W%Qd2l!P zm)}y?#j%zXaBnVtzjMllUb2{xk%1Kz8AFPKJugEF0T~%@)Z3g#*)Qq>pvT9HG9$yU zf2M?$yKbk&G+)RbkW%v9-SUPHJ6vAQ(^R0yCxC7SYpYpZ-9AhE>iw<5dCXyV zweG~mHz!w8-{LZ2K$IIvXB2xnH|0X5v7&>ph!Fe3_eM~lv)lgr?-HmVcxEb6I9nh{ zAt7Vv9p$zDp}f7bT@65oU9dglnL{}e=ab!&Xx5(xv3E73N3O`-<)BKce$6H!@}%|U zV`Z`p6znk4|Bz4cyrz2a;Tq;JYwy$Nyv0V=#Dryf$nmc~1_ycB0?)be9+npsaj@j% zj@TI<$8C(uL}r|3D=jtd8AFb|Ie8(7Gd?9TDFL&ZzUZ~FLG(h{F9uPXBS@vhPos6K zRRVUWQ+y9XJ4z5`LqeawrB4yPw6fXLTTHR5_^DnSivUm&c%#B{<<(1|ZI@=rOP&wAUs z(RF91zu6&)sc}1qa@~F*S-J~Ya8QU}r<+{WBnKj!Z*uq`Srzml2N688GEcie`Xl5i zVrSRBkPe*L>es(50Hw@MZp`~|Ib{9oI?;93;yDxsnkLci1_$%3ReUT)CJ2A>!R4)H zRRq{I=b(=2k(M0Vqh5Ctw`ab4?@m3)HIU&e*}7iEBGeeQU)L$i&|z0FwEXyaRc_Ul zh{z<~wvSFYnImti>?U0`_K2fIuW|l+`OI4dl(Pu4mWRJn<=_dk0AW=!+TF zR-FuMQVOCea=$;|Y=68NtMc)At7=sgCriEJ^VIa`sES-nWSMA+d(_mPC4}&5ky~sk zK+iCNrb;|oy&1|($b;qg4x)UK+4?t|sX(E-agq3q-;;7Izid%@tZnYWM3*qRO3bl2 zfY9)3^#wBf?7w(nU@GZ*ouxSgTN)-;H|>&z_0#6LGDRW)zjEPL&&M8M&Ksj<9}xyi zZRg6NLudN>9AZD2?e(hd)Wj4#C*ix%FCA&m4=PYdO!T0A5l~TsrA%upxQbF7vU4=l z)~u&Ah>~Tra;*CD1?7VK{40aQ{xloVD-@@~=lZ43w>-H+gPE!@Qg{wrdm>lx+@VN4 zAX^ zY5X<9Ur>>XYW;S>&uyvpAooZB`Y^Rpdub80^IXW%1`u;P2rNyf@C)N@)zg5mbr=_ z6qAdtrL@OQeY!hF#21P275IV*m#^Kd)}@0q3cpB`>oX&5I0j6kZ)ItiAAwmzDu4UT z6E+aVCz=&=p$^!WH+H83X7W@TzcnY#J`!4RZHFyE6JL zn4Ihsab|L|!so{tKLU0!>=dGF({1P@Ha~t@j&0!bTvzb`w#*_97^JD=X;0I0Q47lo zL#`WVx2Rh#HI|QWY;+}y3O7Chp@QGog}ItvyNSGwkUX{`S0qD$l!}SPtJK&1xYz0x z?QXyiObHq1>um+J&lWOWaKMvZ-)SNo{thUAK+^2&!LLw65(u2tbZV@{VaKJ_E;#!j zLslQYf6bL%1iP0a?R7I=cdyq?rix`~(5C=bgfREf#N@_DWE1RFRFJr~_QbIS8ia;M zXEQnQyUS8yyPt@XVvRvzjw$UW{qp>viS&YYYtN#{WQQhyzb#$Bg9o=T(*dSUkLU&k z^Q?lNSp*0J=q+-vfVFKMtjB^*z>gXfLBfOw;fe0$fj ze6Q^I5Am2J2;=>^-M*4b6!`hKT}b?#qZeDs7$0{IuspP-H!$zHu0*6wtR}(2eQx4D zm*yxT)prm3>b9b~+)dvOx>9_$a7uHjumSyf)a(l=OlsBo@hwbQ zc>=7cu54pF`7CvOsez;D`Pip9mVdcU6Kji;d~La@Z0pU9Kf|4Q>i2^J)8Ltpx9+MTYor+^njvrAz}VPx zxRK>fBrT(o@_m)_?Lur{u^B`0bjq)ogxwWVTgj|Ycv`9y` z2%57UoBe6f`%8y~H!pb3rpZ{^`x^HEc!e#7D%Mi6wpg@!0lbm89&>oTUl zX-tG7f-*BYvWYRR99ZfU-ttvLN06SND{)sxuB5SK*8Rw5E4+jQ9x?q*d0q<1jFxI& zSZov4Yy%MvzC}OvEU-|caKpBLd^>H?tJSD4cm}>`s4pP`F#^qkH)jFBZUpzR-w4Xt zURrTztu85CPKR^sHp%($1;!@Fkp?=)^74swbx6>`o+9JT2s5B>czVr*v{J@&;Ko)C z9u(*#Ihj>9#Z2|xpjJ^YDR%B%C;n@7B22(1`s(1*-8aKB^}bEAv|ApVRf z`fBV)Lx$tYNz-dK$2~A?*kIc5e$Z{(8vQkv)(T9EFS}J{$E;=l<+Dcj_i-iw{cfxn z`D{g2>n!jb4<-j4A`yf|m|=Bg$ra67ne#?Q-Lw&KPr)eWlC z-U;^Q+g&8`r=uYfZlW?mIKRD)^qkk5lTJx+=iZVIgb@h+k$FR(_wSvx;3zB2y$Mvk zkmLj&*LlLyxc<#ft*n=tiIXEa_ME>*ezqH5vb2}SxIsjNLMlP?bE7&}$3ljR@1bT0 znTcYU@>1nUQjENf`Z+uvrLsnf9sabO!%tIsk6wM>PqN6x4sz5Ld%vqO3CCP!4I+fD zt-^{zE-CSnBOPh`E`8(a&@#X%WK=nX;@C5fnXz6Gge@8vKSnsVf1HDR4&fd`rHr$` zd>*xkh^~V_gXIe2g^FaWi3tU9Z6q zTUO(jO@dUu`$pL?gZG!Vm{`9jMe*%9y{O6zK#htL4<;{yM|=UzLSp7NKg@kSXL zR$(_XGpsE0Yj^J?%#Mxl$n9mK5Tw4rU0w<7M*)|~CDlzA^WdVQXi!K18S={Ph1dBs zbzNbfzZ4I<$a77vKj3-{#TX-zQbRXfKP<{LV(c9k-px3+1el#G{PZZB=0Z(qlQ}Q@ z>4fZqQn44}=H`Mt3>S&2cQ$`X6$>h6g@=V2n)+GA3gaUT-2I;JRf!&- zy4JomDw0eKT7sSPKd`9rT`;yWSCE}^YY(2eDt@dZw+0|8VWFc>4EK`CgWKAwV6U9q zt%vl#&k;MgcTAlNLgp>Qa?3#+sNJH}8r8UZBT8byPY-lqoMroQZeaQ$MVIIG;I64| zqdL(TmYlhy_1FR*C(L4Loc@cdy`l2)<{?4Cr(U+i6n4f45FUQ>_z5tHQ^T5W5+B~2 zPJ=Q>oFi-x#S~a=E1+zr8x2DA4yHRzdOtnFUUyd$dY4$FN8#*2>+aMI!zo?Euk+Z% z1^rTJ(}tjD$tZLDEF6pspV5ZC4Uwc$!Ui zXLIAiV#IZ-ko9GdG4ZxC&{wAf{5pV?>mUDomN)Q;P#AF_lxek|f$F;O7)or#hanqM z)>}-yuOxQI?(vH%25a1#jRqmG>TSd2V%xK{qEx2;~)pl9#XE zwvdb?8Zs1S?N=Oz`MFzTrjir9r7W5raNUwJjTE*pzb8>(x6MFr(K(YAT8jCk22!Kf zjav;|xbYs5dYe;k!}}X^PevC{! z6oB>Y>v;`4uf@ezMH}H zC@)Xlj1F6Mr%##yDG#o6tMvWGuaMQnO07?+g#biQKAZ)6>!8kHXhaE)6u&nz!LqS1 z)^8j7(E@rJ5wCAVGDqxoHZE_`E+>v1F6YDi5_JU4!eoRZdVk35uQ1Tr?jJa-aN=AR z=r4mPu$rPlfvt2KZMihy#xBW!jnrLomb1j};B~Qj!xY+^dUWJc|Dc!I=0h_6DUgy1 zg?UY_bZmL1cGyIS;}qgA-#Ezs=83KBwr_y#&U0Puy;Jr-RVW_U538a{7XZhHrmiGY<-puqO1wpP=_ zayGC__#36X=U1<=cY+7X9Az~N?*H0C`oE5xHW2aU8J1We4vrn{mRwTx-;7GRP<8H& zHi2@|dVgw7(Q|q=D|IwhvvHLLIe&66Mu2L4bVajOZe-_Ya@XBo0N`M(U6g)O1h!-m z7JDPp#rD!zWXo4tW({1`prmNaTwD>Z$hXv5uhMvLhR^>^WcZGPlnoHIIDxHvEV&Lq zkV-zfJf5bGkBt8N{M^t;Z&k5d-|fJ@5ZFK_nq(EJ=p zDHVH6mRgoCW z(3uc+^zC)}-SK(#f z*6MGiuK`#I@{-Apf-Xl&*g~#}E-S6i&$Sz4ej6qGW7b+M2t=j+b7=50sz$~&uk`nN zTun_oX=zjS^;4d%LQ75PqIt~?$>7Wk^`5gJ&kx)2gW{J2buZ|xwON)9pvvE4@LAiO zp&L_Q54Hf$85ltmmrLJ=ssRX3QRJsb2hSjY zVBGYkrFzF}jZAx@!S2oIPPp6=pd>R$6En8BasHjq6i7?~ZjHPPN@j<@@(&;TFX7v(wL@0G~5>4h>SEYSg+b z3|k0~R*) z2xhDSVCni+l{x)Q^$zlI)d%&U&nAeH!M#oW8V$_o%qc~G(r#oS*W+wm@A%y@r*T~d z_m#NpZeNe>15EXf`89xDc^^#Zwt>xT?Kdo_;J_84ia_Pkt$cXjJMkv(_fD@JTEc=(VZCA&3P_7+H@v)2?*S$oXY zUFjU2aPTmWtW>H^tD7;Z;Io-^ovR%La~Y;gb^|+-tR=Vi9x=7C*Tmwpe{Yr(GdIzF zBgJnN{i$<0y4Qbqy4pxzMiO_K^myE9>T6SqhcM`I1NQNjSeIZ~ak4TSMi7d(oT%a8hzr%OlSAoLfa{SrMDBG=r9znz#V+_Ib=$QoS z_iBKx{6Es!$!L#$J0}Tx+$pO$PlLHEGjs6X@8X9`s%K;q6-AI&xg6}M22*U*AWM7Z z~ibKwipVq;!frt7`$%>tx zaejXKtH#pV<>CAJgfpu%3UfcrC+CuU^Nmf0@pk>j}wV7 z7OOB$!7B2`M{EVjr8yRLTdY z(KQ9$l=pQu>Qp8v(fCyOMKny&041Bmw?BC;lrVl0kuQ`qR(#w7Npr9@)pe7ZrB&|X z8^Y|yu1`|6%2C#}0{S#SJx%@rithKX3H|rCf)tL*jQ{q>VH`8hLLjscfy0jz-`w>9 zDCY&fyP4YH`4E+a`1lAYFHo$a{2YS)n*P#Heo!~qA-!(i^xW%lBN02YCw(+2F2%?kDj79$NfXmnYg9}dFLkWE?l3cCVh?T`o0zM1S?Zk~#*i3Yd>Q>8dT zFZ(YkyRHhpF(+4v?LWHI6YlE7fr=P!<7&z1mekGpk9BcSY|eETK(O3vt3Pciu-B@` zl^}ZK?8U4>mCgx0tsfLy`mDzDPND|=2n>WIfi9uHzrO}h;}d^5WeieOq`hSBbtucC zoP?pn7P+JuV%>rC>G@&kCkgv38kTuy7r5WQ0q^jGOz1(RYq(ImVf)t*xnu`D6~zTm zCxnef{()=Q&(4|0l^&mt)QOqqPFu>^JZTe1>mqBQ9Tu`Q<)%xW<3 zJ@LNGx#%1XzhT9ZrBG{u9d^68h(8z@)3IUm!^2IjfdkMc*+jkkgMVIYt6Sgq%~?c3~0z}1C5!efav{lAnvr?@;hUh$+gH?^IU`03990z@k zjk4iketWwNmje@kPf=fp8wYjw(Je*=p%xgyay)rF-QC^}dIeBnx2+G(G&Fk#?gK8) zu(lJbeM^ED;eeM&=pMIO95n&zL=M&gUC-C;{d_;Z2PP+Q$Yd+B2Hpruf|!_rZAF2l z3z26F;7E%p8feD?mYM3>sovDBs!wk6>4PnUPUul*EG`FkYXyiOe-CQS*6Z(}R_d|5 z@sFTX<31ycKYmOs@;-Fn7Gl|^ROdfqE5oMfbf)8-TH zp_K9oI$-0k)`43ER3!f>XXN}vupyk+9dbYBMOya9aQss%kB?8wqAYT@v-o^?jdevmz@`V1Y0Qf#UsevcRmMU`r9}euQ55V`ttI|DpIKOWoGSc zIve$*-vO5cO9di${+-Lw--s;dr@B15qWj;4RbDuNDe6(T_2;LQWdtOmTQH>DaWc9U z>eiqkhW9t=#Fw+{P|m`dSNYz?tn^6> z-MpH1l)A=2%C_KE)JfF{HJ54`$ks7onCGr+g2w zZ3YIBNO)cz%E9dto=$k#oWea!LJ&9s06E5snV_EVW2AYuTOcz6=3ZT?Gj$8Z+iM?F z^l8{U}fh<-kPF#F?)1&E`~nzDZ5^J6n?tocRITz}UuOHz7;QCN>c&PmJ?25%SU`HC zw1V=z-^p>x^`4WvrGq8Oqelh~Z&%=54LVV9k+gI`lc#ieY@EqmI_tk+pZ%~l!Hm7F zf$v9oTM)OY51Ln8&ck`8^`8jR%&_@t3bt47I(5PiSnj!4DxH8GMrAZr2x49jH^;6k zSP&+hSYb=L>f*A!UZx2dD`4|AZCLFO4s0JPF%xYhg242MxK+^o{yinTcYXHOz%44t z56Pk>3hsx7&%v$*wp(OFc~Jm{-JGUrL9JXqK&CUPlMSeqpV0$7D{PPn%j@=(V@R+- zl0`XZMm5b_VN!cqDG+2u@Eur&r+rsRWuY=Q3X;Yt`b0*%m)n^wPx|GAE{k5m-RTNC z(A>QGGq&BV1IBc(oI7b+#iR=^<=si&TnpMv?|bW&q}L_Pq=qBRkR zMRi{6d3f}GKZU_qDvc7m;i_vCNz6hBibg81+oDfokf)&oP8Q8JiJ6Jj?}QUAgR1Ed zzjV0Phc5&~Y0yMBw7_ILcU1O*3S|ni7Szx;Ft*+&%V~#F7a^ryUc3d!Y*$YlhKEfGhIvH37=(OT^i;wPirSC;Fb~>l2p+MiM;spZb?|a- zv4se$R;pg=017LuxXHe*z*D78I=S zQ61N4q85Hn+U^cpYkoU}Yr_@DZUvn#0F_{PX*5Y;rZ)BtFw=F+l(^Ws11}j`^;^u^ zh!?n2?U-Tjfcp+8opV)%el1XxApZ+%h0zo`&=HWUVCKc;fET@iEGvwP?CYpjaIQ>t zp2)0)63NA4&oSe(SNX=@=l7_>AM=%iZP_<(!eLkh=IK)zCts*^AuFZBT}zy#e?Y*$ zZ)> Date: Tue, 2 Dec 2025 14:58:44 +0100 Subject: [PATCH 02/14] update benchmarks --- content/blog/journaling/index.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/content/blog/journaling/index.md b/content/blog/journaling/index.md index b08beb42f531..9741dd0227b0 100644 --- a/content/blog/journaling/index.md +++ b/content/blog/journaling/index.md @@ -63,17 +63,17 @@ The time for the benchmarks is used in the console using the `time` built-in com | | Time | Bytes sent | |--------------------|--------|------------| -| Without journaling | 58m26s | 14MB | -| With journaling | 04m31s | 10.2MB | // TODO: rerun this test with the final implementation +| Without journaling | 58m26s | 16.5MB | +| With journaling | 03m05s | 2.3MB | | Skip checkpoints | 01m33s | 0.5MB | The second example is setting up an instance of the Pulumi app and API. Here we'll have an example that's a bit more dominated by the cost of setting up the actual infrastructure in the cloud, but we still have a very noticeable improvement in the time it takes to set up the stack. -| | Time | Bytes sent | -|--------------------|------|------------| -| Without journaling | | | // TODO -| With journaling |9m45s | | // TODO -| Skip checkpoints |8m39s | 2MB | +| | Time | Bytes sent | +|--------------------|--------|------------| +| Without journaling | 17m52s | 18.5MB | +| With journaling | 9m45s | 5.9MB | +| Skip checkpoints | 8m39s | 2MB | Note that this feature is still behind a feature flag, but we are ready for testers. To get enrolled in the feature flag, please reach out to us, either on the [Community Slack](https://slack.pulumi.com/), or through our [Support channels](https://support.pulumi.com/hc/en-us). Once that's done, all you need to do is to set the `PULUMI_ENABLE_JOURNALING` environment variable to `true`, and your deployments will start finishing faster. From 1bbce7981fb447bcb476dde8b61287c1eb086561 Mon Sep 17 00:00:00 2001 From: Thomas Gummerer Date: Tue, 2 Dec 2025 15:39:01 +0100 Subject: [PATCH 03/14] more fixes --- content/blog/journaling/index.md | 64 ++++++++++++++++---------------- 1 file changed, 33 insertions(+), 31 deletions(-) diff --git a/content/blog/journaling/index.md b/content/blog/journaling/index.md index 9741dd0227b0..31eb216f6c9d 100644 --- a/content/blog/journaling/index.md +++ b/content/blog/journaling/index.md @@ -49,17 +49,19 @@ social: # for details, and please remove these comments before submitting for review. --- -Pulumi saves a snapshot of the current state of your cloud infrastructure at every deployment, and also at every step of the deployment. This means that Pulumi always has a current view of the state even if there are crashes during an operation. However this comes with a performance penalty especially for large stacks. Today we're introducing an improvement that can speed up deployments up to 10x. Read on for benchmarks and some technical details of the implementation. +Pulumi saves a snapshot of the current state of your cloud infrastructure at every deployment, and also at every step of the deployment. This means that Pulumi always has a current view of the state even if there are crashes during an operation. However, this comes with a performance penalty especially for large stacks. Today we're introducing an improvement that can speed up deployments up to 10x. Read on for benchmarks and some technical details of the implementation. ## Benchmarks -Before getting into the more technical details, here's a number of benchmarks demonstrating what this new experience looks like. To run the benchmarks end we picked a couple of Pulumi projects, one that can be set up massively parallel, which is the worst case scenario for the old snapshot system, and another one that looks a little more like a real world example. Note that all of these benchmarks were conducted in Europe connecting to Pulumi Cloud, which runs in `us-west-2`, so exact numbers may vary on based on your location and internet connection. This should however give a good indication of the performance improvements. +Before getting into the more technical details, here's a number of benchmarks demonstrating what this new experience looks like. To run the benchmarks we picked a couple of Pulumi projects, one that can be set up massively parallel, which is the worst case scenario for the old snapshot system, and another one that looks a little more like a real world example. Note that all of these benchmarks were conducted in Europe connecting to Pulumi Cloud, which runs in `us-west-2`, so exact numbers may vary based on your location and internet connection. This should however give a good indication of the performance improvements. -We're benchmarking two somewhat large stacks, both of which are or were used at Pulumi. The first program sets up a website using AWS bucket objects. We're using the [example-ts-static-website](https://github.com/pulumi/examples/tree/master/aws-ts-static-website) example here, but expand it a little bit to set up what is a version of our docs site. This means we're setting up more than 3000 bucket objects, with 3222 resources in total. +We're benchmarking two somewhat large stacks, both of which are or were used at Pulumi. The first program sets up a website using AWS bucket objects. We're using the [example-ts-static-website](https://github.com/pulumi/examples/tree/master/aws-ts-static-website) example here, but expand it a little bit to set up a version of our docs site. This means we're setting up more than 3000 bucket objects, with 3222 resources in total. -The time for the benchmarks is used in the console using the `time` built-in command, and we're capturing the network traffic using `tcpdump`, and then use `tshark` to count the bytes. +The benchmarks were measured using `time` built-in command and using the best time in a best-of-three benchmarks. The network traffic using `tcpdump`, limiting the measured traffic to only the IP addresses for Pulumi Cloud. Finally `tshark` was used to process the packet captures and count the bytes sent. + +All the benchmarks are run with journaling off (the default experience), with journaling on (the new experience), and finally with `PULUMI_SKIP_CHECKPOINTS=true` set. The last one means we skip uploading intermediate checkpoints to the backend, which in turn means potentially losing track of changes that are in flight if Pulumi exits unexpectedly due to any reason. | | Time | Bytes sent | |--------------------|--------|------------| @@ -85,7 +87,9 @@ Pulumi keeps track of all resources in a stack in a snapshot. This snapshot is s To make sure there are never any resources that are not tracked, even if a deployment is aborted unexpectedly (for example due to network issues, power outages, or bugs), Pulumi creates a new snapshot at the beginning and at the end of each operation. -At the beginning of the operation, Pulumi adds a new "pending operation" to the snapshot. Pending operations declare the intent to mutate a resource. If a pending operation is left in the snapshot (in other words the operation started, but Pulumi couldn't record the end of it), in the next operation Pulumi asks the user to check the actual state of the resource, and then either removes it from the snapshot, or imports it depending on the users input. This is because it is possible that the resource has been set up correctly, or it is possible that the resource creation failed. If Pulumi aborted midway through the operation it's impossible to know which it is. +At the beginning of the operation, Pulumi adds a new "pending operation" to the snapshot. Pending operations declare the intent to mutate a resource. If a pending operation is left in the snapshot (in other words the operation started, but Pulumi couldn't record the end of it), in the next operation Pulumi asks the user to check the actual state of the resource, and then either removes it from the snapshot, or imports it depending on the users input. + +This is because it is possible that the resource has been set up correctly, or it is possible that the resource creation failed. If Pulumi aborted midway through the operation it's impossible to know which it is. Once an operation finished, the pending operation is removed, as we now know the final state of the resource, and the final state of the resource is updated in the snapshot. @@ -111,21 +115,21 @@ After this introduction, we can dive into what's slow, how we fixed it, and some ## Why is it slow? -To make sure the state is always as up-to-date as possible, even if there are any network hiccups/power outages etc., a step won't start until the snapshot that includes the pending operation. Similarly an operation won't be considered finished until the snapshot with an updated resources list is confirmed to be stored in the backend. +To make sure the state is always as up-to-date as possible, even if there are any network hiccups/power outages etc., a step won't start until the snapshot that includes the pending operation is confirmed to be stored in the backend. Similarly an operation won't be considered finished until the snapshot with an updated resources list is confirmed to be stored in the backend. -To send the current state to the backend, we simply serialize it as a JSON file, and send it to the backend. However, as mentioned above, steps can be executed in parallel. If we uploaded the snapshot at the beginning and end of every step with no serialization, there would be a risk that we overwrite the a new snapshot with an older one, leading to incorrect data. +To send the current state to the backend, we simply serialize it as a JSON file, and send it to the backend. However, as mentioned above, steps can be executed in parallel. If we uploaded the snapshot at the beginning and end of every step with no serialization, there would be a risk that we overwrite a new snapshot with an older one, leading to incorrect data. Our workaround for that is to serialize the snapshot uploads, uploading one snapshot at a time. This gives us the data integrity properties we want, however it can slow step execution down, especially on internet connections with lower bandwidth, and/or high latency. This impacts performance especially for large stacks, as we upload the whole snapshot every time, which can take some time if the snapshot is getting big. For the Pulumi Cloud backend we improved on this a little [at the end of 2022](https://github.com/pulumi/pulumi/pull/10788). We implemented a diff based protocol, which is especially helpful for large snapshots, as we only need to send the diff between the old and the new snapshot, and Pulumi Cloud can then reconstruct the full snapshot based on that. This reduces the amount of data that needs to be transferred, thus improving performance. -However the snapshotting is still a major bottleneck for large Pulumi deployments. Having to serially upload the snapshot twice for each step does still have a big impact on performance, especially if many resources are modified in parallel. +However, the snapshotting is still a major bottleneck for large Pulumi deployments. Having to serially upload the snapshot twice for each step does still have a big impact on performance, especially if many resources are modified in parallel. ## Fast, but lacking data integrity? As long as Pulumi can complete its operation, there's no need for the intermediate checkpoints. It is possible to set the `PULUMI_SKIP_CHECKPOINTS` variable to a truthy value, and skip all the uploading of the intermittent checkpoints to the backend. This, of course, avoids the single serialization point we have sending the snapshots to the backend, and thus makes the operation much more performant. -However it also has the big disadvantage that it's compromising some of the data integrity guarantees Pulumi gives you. If anything goes wrong during the update, Pulumi has no notion of what happened until then, potentially leaving orphaned resources in the provider, or leaving resources in the state that no longer exist. +However, it also has the big disadvantage that it's compromising some of the data integrity guarantees Pulumi gives you. If anything goes wrong during the update, Pulumi has no notion of what happened until then, potentially leaving orphaned resources in the provider, or leaving resources in the state that no longer exist. Neither of these solutions is very satisfying, as the tradeoff is either performance or data integrity. We would like to have our cake and eat it too here, and that's exactly what we're doing. @@ -139,7 +143,7 @@ Making that happen is possible because of three facts: - Every step the engine executes affects only one resource. - We have a service that can reconstruct a snapshot from what is given to it. -(The third point here already hints at it, but this feature is only available and made possible by Pulumi Cloud, but not on the DIY backend). +(The third point here already hints at it, but this feature is only available and made possible by Pulumi Cloud, not on the DIY backend). What if instead of sending the whole snapshot, or a diff of the snapshot, we could send the individual changes to the base snapshot to the service, which could then apply it, and reconstruct a full snapshot from it? This is exactly what we are doing here, in the form of what we call journal entries. Each journal entry has the following form: @@ -190,7 +194,7 @@ type JournalEntry struct { } ``` -These journal entries encode all the information needed to reconstruct the snapshot from them. Each journal entry can be sent in parallel from the engine, and the snapshot will still be fully valid. All journal entries have a Sequence ID attached to them, and they need to be replayed in that order on the service side to make sure we get a valid snapshot. It is however okay to replay with journal entries that have not yet been received by the service, and whose sequence ID is thus missing. This is because the engine only sends entries in parallel whose parents/dependencies have been fully created and confirmed by the service. +These journal entries encode all the information needed to reconstruct the snapshot from them. Each journal entry can be sent in parallel from the engine, and the snapshot will still be fully valid. All journal entries have a Sequence ID attached to them, and they need to be replayed in that order on the service side to make sure we get a valid snapshot. It is however okay to replay with journal entries that have not yet been received by the service, and whose sequence ID is thus missing. This is safe because the engine only sends entries in parallel whose parents/dependencies have been fully created and confirmed by the service. This way we make sure that the resources list is always in the correct partial order that is required by the engine to function correctly, and for the snapshot to be considered valid. @@ -223,16 +227,16 @@ for entry in journal: if entry.state and entry.op_id: resources.append(entry.state) - operation_id_to_resource_index.add(entry.op_id, index) - index++ + operation_id_to_resource_index.add(entry.op_id, index) + index++ if entry.remove_old: snapshot_deletes.add(entry.remove_old) - if entry.remove_new: - deletes[remove_new] = true - if entry.pending_replacement: - mark_pending(entry.pending_replacement) - if entry.delete: - mark_deleted(entry.delete) + if entry.remove_new: + deletes[remove_new] = true + if entry.pending_replacement: + mark_pending(entry.pending_replacement) + if entry.delete: + mark_deleted(entry.delete) has_refresh |= entry.is_refresh case REFRESH_SUCCESS: @@ -243,21 +247,19 @@ for entry in journal: snapshot_replacements[entry.remove_old] = entry.state else: snapshot_deletes.add(entry.remove_old) - if entry.remove_new: - if entry.state: - deletes[entry.remove_new] = true - else: - resources.replace(operation_id_to_resource_index(entry.remove_new), entry.state) - case REFRESH_SUCCESS: - resources.replace(operation_id_to_resource_index(entry.remove_new), entry.state) + if entry.remove_new: + if entry.state: + deletes[entry.remove_new] = true + else: + resources.replace(operation_id_to_resource_index(entry.remove_new), entry.state) case FAILURE: del incomplete_ops[entry.op_id] case OUTPUTS: if entry.state and entry.remove_old: snapshot_replacements[entry.remove_old] = entry.state - if entry.state and entry.remove_new: - resources.replace(operation_id_to_resource_index(entry.remove_new), entry.state) + if entry.state and entry.remove_new: + resources.replace(operation_id_to_resource_index(entry.remove_new), entry.state) deletes = deletes.map(|i| => operation_id_to_resource_index[i]) @@ -266,7 +268,7 @@ deletes = deletes.map(|i| => operation_id_to_resource_index[i]) # that remain together. for i, res in resources: if i in deletes: - remove_from_resources(resources, i) + remove_from_resources(resources, i) # Merge snapshot resources. These resources have not been touched by the update, and will # thus be appended to the end of the resource list. We also need to mark existing resources as @@ -278,8 +280,8 @@ for i, res in enumerate(snapshot.resources): else: if i in mark_deleted: res.delete = true - if i in mark_pending: - res.pending_replacement = true + if i in mark_pending: + res.pending_replacement = true resources.append(res) # Collect pending operations. These are stored separately from the resources list From 8b41732c90747cd77441760e5cb961c04048f612 Mon Sep 17 00:00:00 2001 From: Thomas Gummerer Date: Wed, 3 Dec 2025 10:40:32 +0100 Subject: [PATCH 04/14] Apply suggestions from code review Co-authored-by: Julien --- content/blog/journaling/index.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/content/blog/journaling/index.md b/content/blog/journaling/index.md index 31eb216f6c9d..89f267f01c13 100644 --- a/content/blog/journaling/index.md +++ b/content/blog/journaling/index.md @@ -49,17 +49,17 @@ social: # for details, and please remove these comments before submitting for review. --- -Pulumi saves a snapshot of the current state of your cloud infrastructure at every deployment, and also at every step of the deployment. This means that Pulumi always has a current view of the state even if there are crashes during an operation. However, this comes with a performance penalty especially for large stacks. Today we're introducing an improvement that can speed up deployments up to 10x. Read on for benchmarks and some technical details of the implementation. +Pulumi saves a snapshot of the current state of your cloud infrastructure at every deployment, and also at every step of the deployment. This means that Pulumi always has a current view of the state even if there is an issue during an operation. However, this comes with a performance penalty especially for large stacks. Today we're introducing an improvement that can speed up deployments up to 10x. Read on for benchmarks and some technical details of the implementation. ## Benchmarks -Before getting into the more technical details, here's a number of benchmarks demonstrating what this new experience looks like. To run the benchmarks we picked a couple of Pulumi projects, one that can be set up massively parallel, which is the worst case scenario for the old snapshot system, and another one that looks a little more like a real world example. Note that all of these benchmarks were conducted in Europe connecting to Pulumi Cloud, which runs in `us-west-2`, so exact numbers may vary based on your location and internet connection. This should however give a good indication of the performance improvements. +Before getting into the more technical details, here are a number of benchmarks demonstrating what this new experience looks like. To run the benchmarks we picked a couple of Pulumi projects, one that can be set up massively parallel, which is the worst case scenario for the old snapshot system, and another one that looks a little more like a real world example. Note that all of these benchmarks were conducted in Europe connecting to Pulumi Cloud, which runs in `us-west-2`, so exact numbers may vary based on your location and internet connection. This should however give a good indication of the performance improvements. We're benchmarking two somewhat large stacks, both of which are or were used at Pulumi. The first program sets up a website using AWS bucket objects. We're using the [example-ts-static-website](https://github.com/pulumi/examples/tree/master/aws-ts-static-website) example here, but expand it a little bit to set up a version of our docs site. This means we're setting up more than 3000 bucket objects, with 3222 resources in total. -The benchmarks were measured using `time` built-in command and using the best time in a best-of-three benchmarks. The network traffic using `tcpdump`, limiting the measured traffic to only the IP addresses for Pulumi Cloud. Finally `tshark` was used to process the packet captures and count the bytes sent. +The benchmarks were measured using `time` built-in command and using the best time in a best-of-three benchmark. The network traffic was measured using `tcpdump`, limiting the measured traffic to only the IP addresses for Pulumi Cloud. Finally `tshark` was used to process the packet captures and count the bytes sent. All the benchmarks are run with journaling off (the default experience), with journaling on (the new experience), and finally with `PULUMI_SKIP_CHECKPOINTS=true` set. The last one means we skip uploading intermediate checkpoints to the backend, which in turn means potentially losing track of changes that are in flight if Pulumi exits unexpectedly due to any reason. @@ -91,11 +91,11 @@ At the beginning of the operation, Pulumi adds a new "pending operation" to the This is because it is possible that the resource has been set up correctly, or it is possible that the resource creation failed. If Pulumi aborted midway through the operation it's impossible to know which it is. -Once an operation finished, the pending operation is removed, as we now know the final state of the resource, and the final state of the resource is updated in the snapshot. +Once an operation finishes, the pending operation is removed, as we now know the final state of the resource, and the final state of the resource is updated in the snapshot. There's also some additional metadata that is stored in the snapshot, that is only updated infrequently. -Here's how the snapshot looks in code. This snapshot is serialized and sent to the backend. `Resources` holds the list of known resource state, and is updated after each operation finishes, and `PendingOperations` is the list of pending operations described above. +Here's how the snapshot looks in code. This snapshot is serialized and sent to the backend. `Resources` holds the list of known resource states, and is updated after each operation finishes, and `PendingOperations` is the list of pending operations described above. ```go type Snapshot struct { From cfc1d31a51261251017e02e2fff5ad5bc7e720fb Mon Sep 17 00:00:00 2001 From: Thomas Gummerer Date: Tue, 9 Dec 2025 11:14:05 +0100 Subject: [PATCH 05/14] Update content/blog/journaling/index.md Co-authored-by: Mark --- content/blog/journaling/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/blog/journaling/index.md b/content/blog/journaling/index.md index 89f267f01c13..af5ddb9de062 100644 --- a/content/blog/journaling/index.md +++ b/content/blog/journaling/index.md @@ -49,7 +49,7 @@ social: # for details, and please remove these comments before submitting for review. --- -Pulumi saves a snapshot of the current state of your cloud infrastructure at every deployment, and also at every step of the deployment. This means that Pulumi always has a current view of the state even if there is an issue during an operation. However, this comes with a performance penalty especially for large stacks. Today we're introducing an improvement that can speed up deployments up to 10x. Read on for benchmarks and some technical details of the implementation. +Today we're introducing an improvement that can speed up deployments up to 10x. At every deployment, and at every step within a deployment, Pulumi saves a snapshot of your cloud infrastructure. This gives Pulumi a current view of state even if something fails mid-operation, but it comes with a performance penalty for large stacks. Here's how we fixed it. From b64d315551fcde0da880f91df122bcf37d7675fa Mon Sep 17 00:00:00 2001 From: Thomas Gummerer Date: Tue, 9 Dec 2025 11:17:51 +0100 Subject: [PATCH 06/14] Update content/blog/journaling/index.md Co-authored-by: Mark --- content/blog/journaling/index.md | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/content/blog/journaling/index.md b/content/blog/journaling/index.md index af5ddb9de062..8211bbe89012 100644 --- a/content/blog/journaling/index.md +++ b/content/blog/journaling/index.md @@ -87,9 +87,7 @@ Pulumi keeps track of all resources in a stack in a snapshot. This snapshot is s To make sure there are never any resources that are not tracked, even if a deployment is aborted unexpectedly (for example due to network issues, power outages, or bugs), Pulumi creates a new snapshot at the beginning and at the end of each operation. -At the beginning of the operation, Pulumi adds a new "pending operation" to the snapshot. Pending operations declare the intent to mutate a resource. If a pending operation is left in the snapshot (in other words the operation started, but Pulumi couldn't record the end of it), in the next operation Pulumi asks the user to check the actual state of the resource, and then either removes it from the snapshot, or imports it depending on the users input. - -This is because it is possible that the resource has been set up correctly, or it is possible that the resource creation failed. If Pulumi aborted midway through the operation it's impossible to know which it is. +At the beginning of the operation, Pulumi adds a new "pending operation" to the snapshot. Pending operations declare the intent to mutate a resource. If a pending operation is left in the snapshot (in other words the operation started, but Pulumi couldn't record the end of it), in the next operation Pulumi asks the user to check the actual state of the resource, and then either removes it from the snapshot, or imports it depending on the users input. This is because it is possible that the resource has been set up correctly, or it is possible that the resource creation failed. If Pulumi aborted midway through the operation it's impossible to know which it is. Once an operation finishes, the pending operation is removed, as we now know the final state of the resource, and the final state of the resource is updated in the snapshot. From 2b67a88a86d4b845c48869537268d6461c27c4a2 Mon Sep 17 00:00:00 2001 From: Thomas Gummerer Date: Tue, 9 Dec 2025 11:18:18 +0100 Subject: [PATCH 07/14] Update content/blog/journaling/index.md Co-authored-by: Mark --- content/blog/journaling/index.md | 13 ------------- 1 file changed, 13 deletions(-) diff --git a/content/blog/journaling/index.md b/content/blog/journaling/index.md index 8211bbe89012..77b48d87cbaf 100644 --- a/content/blog/journaling/index.md +++ b/content/blog/journaling/index.md @@ -28,19 +28,6 @@ tags: - performance - data-integrity - -# The social copy used to promote this post on Twitter and Linkedin. These -# properties do not actually create the post and have no effect on the -# generated blog page. They are here strictly for reference. - -# Here are some examples of posts we have made in the past for inspiration: -# https://www.linkedin.com/feed/update/urn:li:activity:7171191945841561601 -# https://www.linkedin.com/feed/update/urn:li:activity:7169021002394296320 -# https://www.linkedin.com/feed/update/urn:li:activity:7155606616455737345 -# https://twitter.com/PulumiCorp/status/1763265391042654623 -# https://twitter.com/PulumiCorp/status/1762900472489185492 -# https://twitter.com/PulumiCorp/status/1755637618631405655 - social: twitter: Speeding up your Pulumi deployments by 10x linkedin: Want faster Pulumi deployments on big stacks? Read on how we improved the performance of Pulumi deployments with lots of resources by up to 10x, and how you can get access to it today. From b33baf8f3620b744a38fef53649f720ecae7698f Mon Sep 17 00:00:00 2001 From: Thomas Gummerer Date: Tue, 9 Dec 2025 11:18:31 +0100 Subject: [PATCH 08/14] Update content/blog/journaling/index.md Co-authored-by: Mark --- content/blog/journaling/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/blog/journaling/index.md b/content/blog/journaling/index.md index 77b48d87cbaf..71774fc9a5e5 100644 --- a/content/blog/journaling/index.md +++ b/content/blog/journaling/index.md @@ -9,7 +9,7 @@ draft: false # of the content of the post, which is useful for targeting search results or # social-media previews. This field is required or the build will fail the # linter test. Max length is 160 characters. -meta_desc: Learn how journaling helps speed up your deployments in large stacks up to 10x, while maintaining full data integrity guarantees throughout the update +meta_desc: Pulumi deployments get up to 10x faster with journaling, a new snapshotting approach that speeds up large stacks while keeping full data integrity. # The meta_image appears in social-media previews and on the blog home page. A # placeholder image representing the recommended format, dimensions and aspect From d313ff3d45d40400f6a2305d964eecfecb0af1b4 Mon Sep 17 00:00:00 2001 From: Thomas Gummerer Date: Tue, 9 Dec 2025 11:20:44 +0100 Subject: [PATCH 09/14] suggestions --- content/blog/journaling/index.md | 43 ++++++++++++++++++++++++-------- 1 file changed, 32 insertions(+), 11 deletions(-) diff --git a/content/blog/journaling/index.md b/content/blog/journaling/index.md index 71774fc9a5e5..b5869ab2defd 100644 --- a/content/blog/journaling/index.md +++ b/content/blog/journaling/index.md @@ -11,29 +11,50 @@ draft: false # linter test. Max length is 160 characters. meta_desc: Pulumi deployments get up to 10x faster with journaling, a new snapshotting approach that speeds up large stacks while keeping full data integrity. -# The meta_image appears in social-media previews and on the blog home page. A -# placeholder image representing the recommended format, dimensions and aspect -# ratio has been provided for you. meta_image: meta.png -# At least one author is required. The values in this list correspond with the -# `id` properties of the team member files at /data/team/team. Create a file for -# yourself if you don't already have one. authors: - thomas-gummerer -# At least one tag is required. Lowercase, hyphen-delimited is recommended. tags: - journaling - performance - data-integrity social: - twitter: Speeding up your Pulumi deployments by 10x - linkedin: Want faster Pulumi deployments on big stacks? Read on how we improved the performance of Pulumi deployments with lots of resources by up to 10x, and how you can get access to it today. + twitter: | +Pulumi deployments just got up to 10x faster: + +- Journaling: Send only changes, not full snapshots +- Data integrity: No compromise on reliability +- Network traffic cut by 85%+ on large stacks + +Try it: [link] + linkedin: | +# Pulumi Deployments Get Up to 10x Faster + +Large Pulumi stacks just got a major performance boost. Here's what changed. + +# The Problem +Pulumi saves snapshots at every deployment step to maintain data integrity. For large stacks, this creates a bottleneck: uploading the full snapshot serially slows everything down. + +# The Solution: Journaling +Instead of sending the whole snapshot, journaling sends only individual changes. These journal entries can go in parallel, and Pulumi Cloud reconstructs the full snapshot on the backend. + +# The Results +In benchmarks on a 3,000+ resource stack: + + Time dropped from 58 minutes to 3 minutes + Network traffic cut from 16.5MB to 2.3MB + +# Why This Matters +You get the speed without sacrificing data integrity. Unlike SKIP_CHECKPOINTS, journaling still tracks all in-flight operations. If something fails mid-deployment, Pulumi still knows exactly what happened. + +# Get Started +This feature is in opt-in testing. Reach out on Pulumi Community Slack or through Support to get your org enrolled. Then set PULUMI_ENABLE_JOURNALING=true. + +Read more: [link] -# See the blogging docs at https://github.com/pulumi/docs/blob/master/BLOGGING.md -# for details, and please remove these comments before submitting for review. --- Today we're introducing an improvement that can speed up deployments up to 10x. At every deployment, and at every step within a deployment, Pulumi saves a snapshot of your cloud infrastructure. This gives Pulumi a current view of state even if something fails mid-operation, but it comes with a performance penalty for large stacks. Here's how we fixed it. From ebe660869b60e0725bd35adf54dd86d6ee808957 Mon Sep 17 00:00:00 2001 From: Thomas Gummerer Date: Tue, 9 Dec 2025 14:59:48 +0100 Subject: [PATCH 10/14] add time and size charts --- content/blog/journaling/index.md | 6 +++++- content/blog/journaling/size.png | Bin 0 -> 12921 bytes content/blog/journaling/time.png | Bin 0 -> 17757 bytes 3 files changed, 5 insertions(+), 1 deletion(-) create mode 100644 content/blog/journaling/size.png create mode 100644 content/blog/journaling/time.png diff --git a/content/blog/journaling/index.md b/content/blog/journaling/index.md index b5869ab2defd..9679721e0f68 100644 --- a/content/blog/journaling/index.md +++ b/content/blog/journaling/index.md @@ -85,7 +85,11 @@ The second example is setting up an instance of the Pulumi app and API. Here we' | With journaling | 9m45s | 5.9MB | | Skip checkpoints | 8m39s | 2MB | -Note that this feature is still behind a feature flag, but we are ready for testers. To get enrolled in the feature flag, please reach out to us, either on the [Community Slack](https://slack.pulumi.com/), or through our [Support channels](https://support.pulumi.com/hc/en-us). Once that's done, all you need to do is to set the `PULUMI_ENABLE_JOURNALING` environment variable to `true`, and your deployments will start finishing faster. +![Time taken](time.png) + +![Data sent](data.png) + +*Note that this feature is still behind a feature flag, but we are ready for testers. To get enrolled in the feature flag, please reach out to us, either on the [Community Slack](https://slack.pulumi.com/), or through our [Support channels](https://support.pulumi.com/hc/en-us). Once that's done, all you need to do is to set the `PULUMI_ENABLE_JOURNALING` environment variable to `true`, and your deployments will start finishing faster.* If you are interested in the more technical details read on! diff --git a/content/blog/journaling/size.png b/content/blog/journaling/size.png new file mode 100644 index 0000000000000000000000000000000000000000..5c452323b66458216361d24403296f78716c0667 GIT binary patch literal 12921 zcmeHuXH-;Mw`Cy)Lg87%+p^pZ-W<3V`3`AkiPX1pk(>OOnH1WJ$OP#4TL4D+t7OzYI(S z;;lq41_Dt}jYc56C-I#Sh;J|a1rdnH{Qu6!)zzIbS~X}hLL*keg&ZtJT$>^_K#~H4EX_~N%qN0b79z7cJ6@Thq>1=0rk{@z2 zp{6D%PxskYZJN@&jHG1mz(Br!d9Ww{lOHGw5m#Btw8yq8LgD-z<$nEv6rNhAZll42WOmX zSoj?0-u6OQ^1;^pc=fB|q10}!l+sb;Fg5y*lxEaA<*+|GrmlT`W8+}8YHeZh@bEAv zCueHP`0$CfwY9u_bAPUWmGg>x#}GW&R72qK{?x0@nWlF)8A+uKrKR7J8OqTS2(Y!! z3~ODJV|jG;^iEb{;?Tj09V;w+pu77_nU&aqKUOQ3X0J>y<+M+%Jgm3&xtx*FSa+%d z9;N8=^0L9BM|I9Ce=c1k{@K9uA#kON1bcR7W;-pG7TdM-jr+Q8T=T~IdSgpVOGgJI zYNYCA{-ZiiR^sf841fIa*jTG4Pr6Di2O1k23knJ-y^c4UKiv%^z9;Cm%StWDO571A zRPJ?lO6k7-z{rS#dZy_Eq0_8hbzto2C@11`H_*@va+&fW(?FOzk|u|YUO;1kxO^mU9Rvra+Z;iNmL!o z*3S32f~OqCX}Uh~{p_H_Ya!xy2<6`9=H~i377ormCMNUB-0f{gFRwGFrG8%&%Ei^y zsKVYC|Dm>axp^1O%a?nwt2Td17PmUYZ1d-91h9pPA3C~q7bQzz4HZA@AbI3Ixwckk zH~lLkBZIx}*uE=C>hh;0HEwsD_914!e4!#CiJX|2h>az4oy)H~MK3KaEi5pNE(MaXA08aMa&zdtkKcC7*u%>Sz@Sn*qONe~!1e_r%2G!_=;AEK!&B$FNnX z&@>tO-Rn(lZ8*5N6r3jhetv%LwD&vp`~^QH=jN_VP1#TX!a+9fEDel~j2IXiF8cA= zj(t33rbDqBd@XR>TNQF%z9r(yO+7R4$P4G%wWF2taVu+UJaR$0FjNl}2 zE-o(W>FJT-GKz_bF)++Q0LAdx=4EEyz`EEH$u8!yK9Q?mZe?XvWRZGM@RA{nH2r(hy$XC0kN4qOxXUE0SdVawYOlTeMl?q=OKm4S$H0O<} z1~^utyIL>!V^JuSZQThk8(SVkx?{)Ww_2;@;u%@+*w*Ir{WGsB4*Voc>Mj%$B!Q;>;1fT`n0!?pLN6=R=eEi<#p~!Q{v&_ zX=rGGv^w1QO_^_0gI-)b*co_q6WJaw;pR6Kura-?MWxOga%{gF|* zjUlWLu#(qy?0A=Ua9g%1o;oZnOjJ~q&t^ncQSta>Z~Rf+>HO;}=CqP=ESnn}6#%T% zGjM7Y6cj#}agxS!$G_T1>~>hv=H=x%J3FI7F)%L^5)#hL&B2zwCl`vq?=F6(A|zBb zGc(h#Wxs>g)YjHM>=!Lh=Py}iAPii!(T zScKR-ztobYZy=%e0uWHRo&A&U*Wu>2OCpXwhQMu z_D;xV%xOPInFw+kz-^$nSJ>;c8j1=7149h2byi-U=l=R+D7C~`NyPT0db47B0=ehU zGvnjqx!nL*o12^8zkjc%39mheKwNmPq|~MyC-~{pr<%;d5_)RX@98FD+AvuFeW?&R zEe#DF9UbmrpRzJuk?+QZKaKBGqdMl778kp??y<6teGFTX`D^R7$m7a|3sIqv9hg^e zxw*Oh8;k>z#xx!#6G3e-O&H|M?aRA<{W@d_TrJJbr$^&w%F*2Klmqwo_o1-CF7X*k zX=_KuPi(kccr7kiS5i{qFyFpi5*;1Ajjme^oNA8~>b@&=?_O?E5$BLo>Pk$1r+H5s z#RK)8IvZ|F*OU}Uly7Fmb;p}6MMXuv*GM4#t7cV58?R_PWo1CNmX?+l78XJuF#m!- zEiH|ecxid4I5;@CdtgS4lbf3m>tgqS&Z}1yVPQ97+9FPQaUQ60csMmV^^=B_CRzRdZ8sEHuy3l$pP$ zvJBOy7CrzV`MaS3fGmIZlH=+~MNv_ByjJtgBf$T0ue0fP9ZXCgK)Sx(-p)_kE5m$j zY&I)@N`pk_=uqni?ygBmNyuGt@8zwb&Q5W!jFFF!tdWreabhB#$AfTncB&p;&V!yq zxigK2`;TL>Oo8uFURqk2c~=r(YrLQn3n%BpLt*iw2fdjzxPH@%i?@l0=xc1&*VhND zUfTBD;Nsya4G0J@6I$%gjq&wufI5c6dbPcv=ImS+6B82^rK_fP~N$kf(BL_|cK(967x41U+` z3@hY+Rh}cx~JKmK04FB&5Dh*#&rb7uZWE1StG4SweF1(F%uWuCZnME<%|{^kZnsgq0N)ICwXv}=SR5ecwgett2sDnbUcKV6;I-(@fau{na*l5>hErbp_8Qa3 z$OtOlL7riiqmz@8qGFm-3^6`FhZ9-@FwZr#06HZpDXAha{*>0%R$l8r+)sa9S5fJJ zp7&5IER0shzGCDCt#oUG*!aLeIX!#t{inY^e){yv)wMmE=dQ4@u$b7R!@8a^=whCu z(P(Hnp#H^hJo%xYXZuHGI7CYQo|vHR{KSO#IHi#5_QI3;w-pr?Je41Im0;f?_)m_H ze}zzr!ODZZj2Ttpg&pQvp)uS>_q8-M%)&O`zkk1|ocx|n!I#QPS2*AzOUNAPK78N4 zm6nw3Oh9$%u=Yb1CnqPz#l>|%b^^+n@mZLdltA3;7MtNv2)i6?&OUhX02SJmECVg5 zJ79ioZ7rq8c35DbLNO13MUFv*edm`4LqkIf5vaM8kYn^Cgb)gE>?AG($l}Wp8)4zpoFvm1kB~tRh<@6%No@2vc;7kJrJbJdlEEx>4mcBZm9= zLs?m@gpP6O?ffV`nFz5jUHD$|yvKQUq-lODE;g2!-6-c95~kZ3+WC-uf5{{Oey_vD zbPEd$6uyW^4fIoRc^vJ=iFw82KU7nDVrKShev60a>_A2yZu=7wtL+3IhN7P4unJwW zpr9ah%pEYxl$WR9S%4fuqvvL3m~vp$M^u#TXg3xY6^R}$B*PCC4_PIuHqP3o27WcL z(bUvT-o13`lCQ6?^#~IJQQZLv+}hTrKUS-x27L}O8Cf-q9e`*zktL<2rtH%!EG#a< zM#-AzAJU{mAjI!9dFeL;iofykf$YJ=#9R~rXmj73xhi4^6=}AU|cq;5rn}PlKIBp8`&fAxvvc9Ei2avLHa_)c(0N zIsDW^V0zP`&Q!EM5>Moy;r5Vb2~NoeysWqm8Fl2>SB&bf9qO0u>q-i#qhD+Hbr( zaAxqwk;i1|vowyUP{i)UOnp(#X10(`*7J3HjW=ZM5)IvcVAeLvimiR*c(xRy1Q z<@%&1E!a$(L;>k9$-2nXU3-S|By$R`$8zJz(;EupG>yEzM&hmo>(e|9r=$$SU-zPf z=T_iUQ1NZJ8Sq&k`g)YmLT9?nRcgDtH$`<8=WEP4#I*6Nj&m|7D0?6H8LJW2?jaCZ z)F$=c&X}rDb-K=LT&9tT>#yNJ{(VyGpG4`Sy^&R;R1(9G>NuyZ)S_$D5$$t6^I!EV zoHU)O5eRZ@ZKoAwDf?jj@JPK-_x0MMYy@J#9GI(g8UY#gOk)H>@bbUNw{+xs%WDK; z1*1GQ<2rF6sre#zt;%BHroP(C)M9H7SC>}DzT)2D)vBs4()L7;lX^cnN*nue>Kh7z z8F~osmsgUMHQyVOeBz909DkSzH{2Zg%F2v#+tA3qs!+pD5Vl$d69=)jS}%q=$cwO3&h~N zhLxP>CYFOsKJggiUQ~_Q{AXSy@|^Ofci=pfehE~1aTXL6Y<8zvHgw1f78t>*F54)m z8462KJPG?Dlka?G)yw%l?SfwzYN!0NZn;rfoTsC)oYiD<(nbuX0Gf=&_F*-QLg4E*j=*}k%W`wuAL~mwZ zv-YPc*ZVR)5?F~$0r<#wrnez6GRLEUO?RV zCx5@qF_bA>Rb6S2k(oTy*AR|Rb3?&3sANqnky0jZn&XF2P`1`?lHw3OCL$w=S85ah z*~!r{m{LqxEGIkrX`#-Dm{Kgi+6|)GfvGyBl7(kw``rd^kA}G(#WA8FI$hXF71@LI zqbAm0pG+w@jELF2GxXHbVArxJL?Bq?PlY&u9C3$<9C`$WdFbgSBsV~1aR!*%T%(N{I|+>aj# zA(=X9__ZgM{%Zv@3RBaD0%T;gy09RN6(&nZ@J>=)y;$vzgTt0q-)}@@BaUh0I?OU` z&1@JNgzcJ-S)_0ui@NW>m}v}y9?rPXr7fBV-hTV`E%x$xsl(lri3<$87THn*vwCl) zdWFGi;nUy8Cnu4tx>K{WrdC$&d#j_gGQms*s_^s&jbh$I%N(w-2)OQ`p#-XOxZl6^ ze}e7)C&2K(&!zuA`|W=So&4}9`@6aXJdeLRwpTdJ0~72b9A8)GrKYC#uXs90w%M9% zgPw6^w5r@OSe)uJj3m(9AGp_k_|OAvG?5zMqIgUrr4uyp>B@2E;y}Bcq9P18J9~RT z1cmtf`v(R>6st>y!9i|*5JSVD0o2KI?EClE&u1Pz_>{IbA0uPBQXiaaNfeE(!VPR+ z12*l$(3V*(O-<$kFSt`6gSmhm81bQ&uK@wr6d?l{N-#n^d6Ia64?Fi$czDt>0YdNM zJWKVvsHmulidQYecRA6nyUS6ZKg+6+5EDbc2}2)`#Zw!bv8=;e69g5B%LJ#t8XBaA z=)D7eRA4KnHe$5GY$fb@%+Ji6SzGG~N&Nl$cT!T)|0rf19UYC0jfvR|GV=2i;^JzY zmb5MrgFZw>d2odzhFp4YXU78)^#8}~|4fYHF(pAD)_JF5-y9Z4N``QKG{lZhZ_oGHDa8t}fdvQm&OGCMO< zPK=DptQAI6qCqi)xB}tTY`jZ~WL9HA_(|=Pjcj@jM**@luLP$90|h!5f1eyX{vy#s zqvcs{3I@|$;egagOnkDE5bA*VLMYXziF+u0Ij-co;B%+`&X0zU_va6$FnmgS`+Tg( zmaA?VHNVQsp-H4=-ptAa>IV|xHTeZZFEjcw)*3F1mfO<8=Se3Tj3ZKVpo*d@^h~q z?Zhs3CmL4S)_E<#76ctfqVRcm=W#|6$xNJR2;vKvnB~^P5s{Jelan_{NwdDSV<7b2 zNHw}9gwf`#9Nvn~*&z-47{^@2{i9x6QOyJQ_}OpUazmwiN~N!wvu+B8!?S}FP;<_X zy7pyh-r==s?C2;5s>)hjRaI3-C*xZX(lRwAg{kVIH*sQz=8;z3C&TY`M=Z9+9knR@ z__uFSsznNLDl!)sZy+rslg5t9E2H(AwTYM_|4m9modU9voSghBHB=#*Fxj zY&AWuzj!yI>U`B}dTI*5KwK(wuHx`8eNZbmH8rnrd0=Z)a0!Qlcij0bEyVOCQucD4 zqM{txb`Tb|aX@q4gxAaIDVO0fdxO47qHVT^Gl;o*KRlUWvZ%PYd&GeQIlAz*#^&Z3 z-%c?Qr(uN=yCI$rw!%FN$EqfMz3c22n2?PfD&4qg>D@RC2?CsRy=TiiIHxNouOHs^ z4Nh*%76jReULnHe)vK$%=St6Cwao$HO^jV)hgD)18du!?^t0pP;AduXqZ>?7Z|d&R zvz}jyg&r`Lm0YUDy!oF~b^mWyB8z4=#7Bu(byct3@CCcK&`rlujB-okYA!ve9Y=>( z%Qm&_NNz`4b4QHc4KFx#mJiI+4Lvk=jkZ1^8lF2Ui@1hZqmmqTlRs~`X*|PvUK~WY ztI5=yDs37QJ)6*sNQ>GpN638_Eo&f)YB26r&71plBy(C=N~pn!sD8@feGvaJEH=5x z@VaKiH{R=ypAX#(#%(!P&tjdl4O|L*S4rK)L63ebIU4!`Cd@TvN%HCzbl|%*W|5|D z4r-@-6OxBJ`B<9l%DedEj-kS55vsF5m;K#vZjL8=?HevjU>p5R zhb51^ zb`I$3@pP|RLuB%@dJMs=Co;8iv#&a(zSuIO;>b zglF3FFXH0O>`JcEFMi3I5c%@=kfU)k30E&gQm4jo0^cD%_yJ??{W4tkyZc27HLN*x z_IrydY5g6Iy~-U_>YWeNeskWE`t(cv$Fr2gSeAYEmd6)ig`&w>Ad2~CAU91gGTZ2J zX2XwbSWljis}mtINUuLe2D;C#(#F+BCEPS#W2)NzAc}dOq?O@YNwPY#W)?&DH496~ z2YJsJl{aWLFo#WOhK+ftQLaw@{+7DVOFGg|bu{rLn^sZyWp-G@#zwz2E+Mimt6j-; zSlICCCY}5l@4<}J;oM@cbr(I2lt6_lNyyxBM1LJV93plg9ku$gx{+U(RwYwVCF7`b zA90T^A~Zjlu4|rthlYUPpXh*IMnQdu=RWCWjzl6G;+f+YN6m@eS1qh~m(~K0bF5gz zj&}ci6j)k4J=C{*ZI8xuk&$dr2wzoWcI%BlQCZQY7;+_}q$>xvMv2DdIxFWNnf0$f zusS*3xhMEJUQdgb4#j^v3ZRdHWbSUVxc5ux`w6TGtz{f^Z!1lzN$zH=4qduWetrlk zX|DDoVdV8bvF}eSTkiWj+D)lO;oEc4di2yR@z8XA*bJ}oeWm;>&35%FLiXqH<)&J% z_wNnP;x^I0ftNI=g*p6Q(cK+ohB_-_tM`uWEdniGp5a<|iF0?i&q8b`paxkd@uDqW zL{`13h;h<1&P1$*CJ@lkU*PP+QyZwocj}}Wcis0~njAKtlMfNI887c$ba!w4h|7*W zG3tEs5`C0F7g?5(X?I$2zvR+o4h+P{30=0Grn-jTXAO&HIm-{nFo9C^4%T9&ny&k_ zd>TG&6`0r(>_V6vZ{Fmip@V()?Ah<>4H!5A0|E$#1vujW zd+w+?6Cu9b3V1L0TAaiBPYFmGn+NXDo;C!M+%0f$q0EqN@6yP+g)qMcLkAiT*4Z#y9qLt9RkVyyon z4?kTv0>4fuW?&MN)Xq*lo*1a9lG4)Mcsn~gYa|0MK1o3LMVIF1Qh@j`*cXAvn2n?@;Fr z6!HnMkidMHLHSKhO>NG*ph23nT0=|l4kX^lmRWnKx`$wkc)2#F`S=V(A_|Vj-XO^p z7TSr6i)(2SQ=5PhKp~Qi9%W)?2EG8p*D;uy3~ogrRtg9SF)}dJW^R17o5rOOX8Nu5 zP+k2B4(WEf4v7%mxP35er^t^QN<`uUwi@>e2IvpuwVZH{;|2EvI(64rI_b2i1qaJ^sr3Oda}V#!~bDifUx=*3AyP+bK-)ti)bhZG5|bU6h{tPo$(SNvXMudi_9MH3U5bbWmes8Y#7_SFd^; zZWor8!g%%4-M!js9yDVho+CwbGBcZ+o0FK%hxBvnfy+h`3|Q~p$+@`jOR0g@xH?pv z?&DMc!}r<}8ZBr%pZJtVWGFT^*7IOC5@fNR<)IJpv~d*R%s4yU6AMV<=Ha<^?b>4M zwj-#_phUa4?1Ll;&H*P7Mqu;?Uz8h&TZB^djEuvBgVi2Y-5;upiZ+u&M2m`xmqx4h zK(b;AfbL5lJPD&YB_$dP3JO&Cvf7~3%t9ch;01zsqX05oQ&(3PJUKkl-o{2#Q&VEJ zTc7M6=ywhv*Mqo)?HZgYh3~W$m*#b;yQfEhOxe)T5F~yOld`k2=m?sC4*-!(S*7m?fDo*&tIHgxB10~v zb`3VN%K4s@8pI#4`sKiwypsg8*l?M(zTdKlV@4XIK3E3ep5Z?kM__^wAZrIt7*jwr zr>Q)=ajqXAsY8C>1=EmRubj$ehuE1rXlexSQc6mC|NIf;;n~?;83y^%u^7|^;QgDN zo#xxgt*uu;X@vve;pUc|SLm1l8JjKg==i`PAPJ;Iu9vF@SrytiH7`L~D+Eq{e?1Wy zZ)7fT{qcqSe0(H<4PXt~*w|1{SMKZYH!IWuJJtP@LU{v&5zuL_QrpmzRe>VICHP#5>pV zF9RZgdj*z9g}P2IxD7~}0=qz+l9WV4M`r@06bSV8!kC@J{(@A%Nr4dW>go!2T)upH zTMne$cmDZQby)?N=k}!&F$N_Pr%Co*$H0&Ru#xu z5^P^3K0O12>L>M=0V^ymm%zbO1L_?49{$H>z{r+au+dl#70trtZtM2yfgrto(XYO% zyE|Mtuwe?G+$|goA)Sn4;3_$A7p11-=jZ3;Ed#&@EC3gqb%_lqXUR!P_V)IxV>S9( zTIZ7OP%6PY6G&W8U*Q_}*o%{-p>C_Oz@i41PHRgG7aw2b)4jwXd?#^Sa>1FkwGQLL z_d!96F0`nR5fRa!KjR}!o{4%KmgMEN4-UG3WeMcuJA5{=+|LP4{-i--t%1pByCf{E zYj50(oe!6C*56hTMGWuV*O0yu(VVL2XmaW>HwFEcMsSy_2^bCU-3f}afC!Jcn^ z$R<-+0+3Q$YedZ>t6~p&{@1Vkhd=MKvZmDaGK0B;mvKS9b&E_O(;z}f-brE)F`ElO6Q2t% z3wREIBb^#o9%?!!CME_3X(mE>y72S#xp)yDDIG#lPSXDD)B}i1D5Rjg&(6-? zK!UHv!~t0MhzMT0DM?pW2;L`9Lm)=B2XU1EJ*e>Uk-y&Gc37WyaM&>d4jNoC{!Y6p zoMy8pPt;UZ#S^ZAI%ZqB4MY83X0Rfi9A2C3E;&)N*3XTNvH?ke;Sp?x(c$4ptlPpa zFVxh!RK!kt&9}qx?Ux4&fgJ08&WMjiLrW_nBJ!>^%0NrY%-T9PJ^lCm79JVD_?f+K zvNlfKNLg7~u~{3~Obm+6l8+99qp|>D?W3r2T#CzFqbi_ zrx6kpTbP@J5X?YFmyw;VaOXXRh#(yu9UtFbXQBj1>ty^d9!p60)YP2RdK?KJ3m{Sf zW)i@p0ohwx>Lf1yEma}XBr*>a=&|Zoqm@o~L|jXt%v|$him}+GI~%AwEd^^N7%{8ii&`+PFu!hc~AXQ;@X&rzcoN=)6 z>)YGQV>LojYTZ(R#$Yvqa*V=PQBlFc!C{MRncg_uTk~2S5kNMAvGmQGH-v@y2u1G< zOjU2FZ|$G&KZP6xl!Ii&#l=-mQ=~?re0`xpHgzTHu%6C*PystoZ(rZveoHnH`x$@u z==LNWCD?B6vampL>zP96t}KRCs6 zbP9ib`}PNF6AgDq@FDAqnC7{9q zyh0yJfCOfDp-U|jo&yTShD$Y&+29yjbO-(hK$yk-8OE3ZJaPjEC+Fd$A7yuUH|Q~d zN3dgTgxIEc&fSHu(rd)5R476cl7asI-Gwe6ioVwNb~a|_Fl&Q{4~3Z|sH)}hhSl_35AszR|w`e$Y`hx~3iRhN^KgT~g%uV_!d`&ZDNm*94Vcu@Sz zkvK%E!>R?lArnOUDKzv;qX(wT9`LAyRQK*#PyKv{MHvP7GS8Dk>Ey^FPA)F6 zgM-qbC}!pQ9BbVZL}#XN?{9O79xWFGo`DCm0rn7DMQDw;&u&to4C*{ZU?D�*Jr` zjUC;&3k^c{cYnWgV<CIO-HD z^xUqlI8KPY9{-ewoDl<&8WbNkNT*W)~nIz##PL&Id*nu>5;&HZca|U7LtQ{Vhlv ioXZ;j{C{5@c?lG6tBlfV-+&1WAt$9QnTLG*=Dz?rtAu|5 literal 0 HcmV?d00001 diff --git a/content/blog/journaling/time.png b/content/blog/journaling/time.png new file mode 100644 index 0000000000000000000000000000000000000000..bd9106adc6bd81e7bcf64c029cbb85b874c4c2b4 GIT binary patch literal 17757 zcmd741yt1Sn?F3ZfP%DufDS1sjilm$ND9&^DGkz%sFZ++NSAaDCEX$+E!`>I(#?DE zJim9(?*4Yq*>~UF^Z(B|&!ddY%=dfW*L}q&#_zez)0>#2mZszHwDX}0bn(P;PS?&=5*#|CM!e`X^ z^(Z|*8qzYS?YTp&uKKtLBi(15JSK@>#qxE&f4Ir`fs;%kw4m2uS->M;jpoe<)5q-b zd;Y?Poeyp1Vq!fzJ62Ze@|YwyI8{|unbiu7R^&KI9LwV{;&NWSO#f_xqE;>ywU#Tia^w|iVGnS zlf=~jjbHaq&%ahnj9E6l)UiyOGWYsswg$TL$eDD!MkJxjsI9H-#@bW{CKeVtJNxom zSe{jZ`{>=PduQ24(}XJ(%EipZRzW!{b-ztWJm2TzypmEA znpe`A<=BnkB$ZZGjSLM{)Wf-adbN4)!yPT#6*1StkP#K$DE=acbaf%iae+{Yoz;>F zYn-P6x4SYG8Bc!b{P6bj@(K+Pe`RRMqFOLh?}xv(w)XMk$C}f_jD`UaH)h@w*6c*-klunv1(Tf z@bHBD^4YKK?+j>{79Et4mUJ36qyZesw!vmdtWVgKRKj}cl{g@p<;DbxrcwrGMWvg^m zI@#pRgloG;^4TrfoU){(rZzS<_DfU;1l+h(-o0phHKv@T7=5)nG*r0gW|q=``YHQv z1FQ@g-*!!pNs&mxMx+v(qx#Ln78hqS&7I#SYZEoXHgj$5(L6`{`{@dq@HsM3ocCFB z;v!tjzBwkG-%{yd67lv3=GEFGSLsKO9@WAffBW{$kN*Dsj3++WH-5~_&*OI;Y|Y2U z#zym4p7zKJw?}a;H&J>BIBzd}x&LB!ZA=oShPJPJk6~GTU|8dR0-q|2%E-vDtU0lN z{`~p+cohzXfH;bnRcmFuiofKZMOk8^?#f{Pd}lo6Lu>tZ9Y;q;E7r!})6>p7OMPhd z64-nLLqohS9O2^!uR8AJM7j^Pwwlhigqn{Oqcbx%cXv-u_9iCW_Qq;!Yu9Ve;!H{K z@q6HfZ{EC_`r$J_Gt)b!Bq1RoF8;G*!kyo2Fz@TvuSrjX@Pg$tV%QBN!dW!ye*A#{ z^SK>vhwoGl%wJi<63h|gGF!jZ6~$$0*cO3{hgYEA{Ml}~-)(p3)r(Xq5>BJ%XQwA+ zaS#rpH6Ca1$#H@%{fFPOCX}jNW_&2!H@3ExCu+{%dd5>E!^4jh`d`m?keiyCQc_YP z+1c4YeE6VNEn4Ibf-wd!HniMhxo*&lexohzXHFgj8@~aTk4&jo-U6LY1ThG==3-~K0ZC( z>MSoB8XEfcs`GR(cIMZPgPLGI(KQ>c?_FI-29v$LI6{R+F_gXD92WrKQ;9 zd{%-_MMd5Ar+obU{CZNQ{XcxjED0%hp{AnJgV=Q0T^TAe9q_z#EpEqb^Wb2AUqM;f zW3!p=>Bl=6>FE!+x%=Y<@#Ww|m%jcz*B1GLJC|v{Mx|r^+U?61AB%|8L9AE2>WqWCU{Wt$X?Dq5i_3I+ zf4toz`~Ca(G?~~BK|!)IGAyhfhXxpq6ynvX_jq~tXPQG$Um_xI-?}xqzTR0CCTzE) z&|Xtqw105m-GW3OS~p~v%ESzRs5R9^8~@h`tgTJ3P#+jyc{aIg2Z;K_ev~N17TM)lfEI z6F!o1EgkMG9}Y)yyxLvtc{P})Q`A4Lr2?;leO=p}Z9UwC(1QoO48IGqT%}<D0yGV47kDo%;4rkR4lSnb`&o;*6S?4r%jL$V*7_T~N3?w7O zY=D$KQe^td(y|8LqNSz9bRfrWb$I8s>7s|n<%^y&X^c!v5c<7OgHuur0i$U?kGiHU_J z=b&E50v+BmJj};%{>V&EH##}op?dmprcSRjPLLs#fse0zje&@WX!NZVZAk0v;bvW( zh>eYng(@}PV(!h}lN4S^fu{o9y}gC$>FIOjhr4!7&rN%F)*fH;IUDg8=ih*T&OeP_Z5&Zrj}8hU7OqU~7@`M1d9 z>=ZK33goE90Fs+A7jYvaBNsc_mj#`ihqQiHr!PnDcHqYi!YLr(Fl16Iq-SASgPLu; zk7#ReZ*6HQofUJ^&DALT9CN%8r*3Mx4DZB9fN4R!I#Lqk@2_5JrGqXJ_e)Psey>i? z%#1#G`*m(BLn*6_CwaM`sA#nxJu>vvVzgA})vH%}dPyFF;nC5pS4_s(dWUVRT=G0^ zi2YDdj~=@2Z+s-@cRyUDJJ~|8VtmduzJ=-a`0-;%Y(4RcmoHz=)~qOAo8<}?S5Q+^ zv!sL8)5_ldbUC-icL9!o!`kTLXxX=U{*EiFM?z@;xnLqh}kpM!^om6>_0+|IPU(zSqR!qo>e^u>>D&|5I1Xy7XC!cl~q;3hYLyYihe#OJv}|Yg|)GY z`}gn1^4WEV^YiobnD#T#(t7)HTa2dHP?dziztYjrj8(fu`$B#afL;^1G*+>6kP=3FB`SRusVoe8HCb*pnyJe*;Rk(NIgP9O#?G=t2-=LYfNycrv+;0x2A2MHY zUS3{FNl-w*9bT(Rm-VXo#l>b#IoGXro~^Ad(^K*;8(UkAa@$2H9dF;h#lFK^J7$MR zO3Dx{ez?1O)Jzw@z0{X=ucqXQ_l^4u449ahq{8m53JMDL_Qj_!Uds}ZklZKoudrQY zhbo0jB_=SnGC%8d>L#cBW3MBm2$EIBmVND;3y^VFM$2r_F1D0SO---%H>TibDEMrR z;NrDU@X2s-|4jK%LYvq;GBPqSu=g{V3W6ZR=~@qzECxnKkCUB&DAR|$yxVJIF7Rqt zyx-%ZzaQNAX401#NXB!xzrQk8;Q-lg$D+5t-yUi*BkN2*}f0{*Vi=eB#iD3OjiC;WX@%%_cnwiQ2+jL63Gc8RUpMrvd^(bWJX7X9UT?dzOxYc;$ zoy^c06OMI(vcTpqY)BmCcTe}F;N+x*pFG>W+AW-)CL0fDi;SY5jj;==(Wb9a+}C}*|?mXqm>1<-mpAH(2nn5BTx^ul zCFUcBwfK_A*0d5Gtqo1P!2MzJw42hzYpbrJZ`16GcJkasF6nXncCIO3vHSX*9y+P3 zH*RkTI9OHE!`{h=i!VSk^PK(;w5Uc#3*UBM3MA*&HlJeLyp8{?N?UW6JzFJT7xDrb zkA*8&w97?EN={pIffRyjS$)Np6LBt0%aIQZ$*-V23`1qhi|^%;?BCJ5?8s(Sgbx$w z>`qnXRhUQF|9=mjKSI|3ZWkecWvHh-r-$mv%)|tZMSqTFE8%#h69+xLI728D>fP6H5g633FUcNhDuPNh28Y#B|jMGs! zj#)+Fgkxi4Gb&~OY;8S)Gf-OR>+1_8p*@zrL^_(g=6KV;rbZa|d?1*cn~Qhd^ox#G zfA-8zoEnLQzS*EP?8>!k{JgyN_4SU<&i&~2&dxLFa(CA!WKk(8DG*_hGaDOa&(3mS z+LwJf`7pksqC$!`OOY44fQ=0+WJ_Bcw1G32f`WoQJv|)k>>8SyWiNY(@w&WklEIO9 zVQDTV=3R}Cb^h?r&Sv@{hJ`?U-9w(yxj=*GD~?vm>ND{})e%Zie}+nznaSYhSN+Mi z)FWNfa`m@T#(g?EIv$?AD>48iL?Pjar%#)%a=QmNH{tp>=;oJZXSenCe({x%lr&(E zRA69cWb7%K!rZ}GMk~@2aLI0(pzB^+pFf+in#cj+uIcxLRnZ?SoI14vtVRpKluFlbE(5R5{ZP4SOMUHn3y|c zWc<4I*U8ADVX%VvKC5pDS|G+yiM7*km`I6x*Aety07dA1rtMZCnbkWjSPa;1 zE}V>u?(M_(u#Q~39IaZ1l&yI@N~QIwZyY3szhdRFE1Dbi#(Eab+Gn=@t1)N2##yzb zfT1&2T*ZBQ&TmAc%*86}4bBNuv}(@E&Q?|=F`_ohbH>zJB+;2@jOk-+ach-J*5j@d zC7gY)zLWLk>&gA8{IAx_JnsAW+LAKkRuN9=^|UOdU8qsvjjXA#-#=*`a=L#-L(lX1 z1KZ}WJoUgV(+?Mx-G#lgd+50=#x#yjYQ~R}aK;4Ro!z*CIO8lG9;$zEv9X~> zv^z|aFOcO#JXNM?xO^mHC|Y`yzvHKZ6iS}VxC^c8rIGN7@e3gXvcNjO(rziIKt%DA zRz$qo%h}DVxNaL0zH`;>`u5$|c>Gx`W&1M=2lhs9eRxkBMvLk-m#?ip<9FQJHxHB; zDc!n)aGd(&Djg8qE3QwS&6GJ!p+uzg%v_2QixG!aC^b_aKe)HCaGh9?ZiRw>Vfi^r zqeX1_GEr}LZ%h67NJrh!^y%y!B;To??qjvbsuoUdy*0ixNNY1w?bEM`m0vY+~3XgAc_x|IkPDcacg*8=JIQJ$?ZwJ~Ru}6fG|9lL7lN-o25A zRSt(}74bt?D2F6ZEi(z1n*UIq-&?YDc#Czb-$F+u(^|kMDsEgK_aQo+Aq?GThE!DE z*17X10t*A%KSqGnH-q>p- zUKp)1rB{oo6KjLYzYr@hVeP+Gtg$2aDK;HS5k(7;T;OngY&4fY64_*d8!V1QCi;75 z$SG!YoVhpG4T%`4A9-B7gy7T^ucj4KOi@zI(C3z4OK4Ndtl4~q3hN*;KXef^%XrRq z5s57M8kX^gM6IZ}AbqK0h-o0F=-IHPQ~#0*8)Y^brKT0-!7I-31%V$15F zgh1rp`kcF1X16Jm_LJud@^SR3R{aB231 zl9dy$ny@Q>U*@&J>ps1wmk_^tC(pLHZu;}P&%a>5~yHu{~jEZRc5$r z7)(|-;B}I!lrHF8dPR1bp@_S^I>zPhp_S8Pul%<3iP3R%d2ug2*TA&JougBNEWM|e z&U-(*e?M$?;=rd{F629zM}Fqk<~x0o*ek}TujAojS#jL^_L4R^OSO99#NkS^toF{# z1%wtowRX>QjssF+KzpP3G*&PF597T5yD0L1<=wB+#)~BQ1_Tu5fdM@zysg?8HyO8CsoPQUX@UkzjpH9(yT2YRySX8|Fenwl~xQez2!DkherR+O2XoD5(V07hK@^5#%SZ00*coAu$fNn(K?0t0Ch zp&3|MFv@u>C)WjY4n84aigpT2;B|G0O2W`L(8t< z&`^UxBdG7&H=YRD zw6o+hQf1>$0Way*5WRcr_;5LQbQ6#gzz|&lhm=badrc#sQJ9@AH4(A=JmGn-Kyy|R zA(Ubn3yYi!FA(p}p|E@(1mbygNJt0_rI*OaKD(l3W@kgAqFTlhBR_whUR_1!mRrxZ z0M7gN-8(<=3yyJqoTM-lo-hEc3bp+OJpnB(t%eWBtB%5g z0)W5fNoVHf7(yk8iNniZ*jZb%n_)_~;_~$kK5_gxFi>)S0UH`OT;o}hNZqbPQNRV7 zo0}mPsKf&9CPJ8OZ`%qB*K~Jx!}JdifQ+h7c1NBDQ7C5hxgG5aT1_Hw$$72rK70t> z#Tu#*u!QMp-G6RsJUq^--Hyg#(t|bn3n}4aVru*ObKy$I9a!b&+L>rUm*T3bs*sQu z(9uKFTvb`=Yt;OiM!m!WT~d{ja*ikp9j$Nx*aX{};Y{L+uR2UvO|RAYc&Z0N|k&k6^nU#+~K?qysEQ ztJLbAvLocKuV3$GC;+@a0#osTTFUq^#1eq7KlKYfefo5K?E1qSc+BS^p3NmMzh5lV zhy$4J)29cPckADSPT1drj-OF~wkjlL zRDDXgOd3BB2Y^(%Y%jp#O)V`k;Q5)L3;s@dz3S1y!3h)ZfgAMl5>-;_w`n7Qp4?Yl zzm^YDI$88s%*@R4 zq8%dA2Wo$ffX{^TRhWqU0mGA$mi9|8EL5Nk=`+#O(MgccfHM!2QF~h(46A^WZ5I_O z1xuvZnG#}(wue+Ymwbk_6&-C_OHligm_@^AB&+vrVAi3`{O@OgZ^ zytm{eBtDdslvIB(7y{tvDGCMq+>chrA>&^8mGA`0)4yb{Q3l9dy}8qii>f94EiDW} zLUE>_qNB4E7y!{;lIerwly{$#`RRg!9?sM~7(xF&w`?13)sV(T{fQKnlxXSbfE@XF zo9(SHOkLeQJtW?$z|lbkOX@R$DH2#WKHG(tdV2V`ZU$UtC|8MIG&} zvjWsx0w6&(kHsjjKyz6xM?l+2M30oapsnm>oYS1q@;jg$piX_ zR0IADZp&}soPqv1hLk z`cw->h1%8_qJN4n3^gVF<~PS5h^=s0qrk1&YrXE%Z;z8j(w$j2apf%8(sI>=%R)W* zbgKUu|5{wdfc;m80SrwTKMyi+cH4zjKGS#*;=l8tztnj^fGMi4pA5%7B%}vOvSsd4 z+r`fn6-PipZ_}Kfp2EnF=Lh5-FE4MeiO0Xosy1E_ds{HmH|s-6IygA+zDbDCD*lun zC(7fmysKqny%Pxo@a+pPzU}@l&5x@Z9XHa+U>y>$7%8ZwEQ_a-{Fp}Gl0KwmVjdL| zv-c#vXtcA+e~8kYB|{`QJ>nE;gKd-rr80Rl^r zr5YU(!3)qn(9}_0)@8@j4gP(9Js>$0@!k0+$O>N1&Tie8sR+0cIlp}jpIxTTlPhn+ zKj1!KIc5G_tkG?3yZDsdj<3Ietg&s}YmuQKVcuHl(=i zXaMcu@w26c9a{E{&Fk+kB6W+H_CED3qQsD~O5wp)TN%stq&ssdXyvlmx7*i7D;#jx zF<;;$pK-aYADn4jKVh#g4-KuZ3XCpVuMkRe7uuRmyWpz(lbq#0s#t;+gt0QjP41Z1jpYAc^20xbp%PFbw*$+t|I^x6 zyDa(-tf0F)h8_8NNMc7>D86k^FrINF#Ew)Gy;*#zYF)EZJ4{^$wZy>n;{l2~%(qVI zhET;+9r}fXNl_CU1`4X@-{teFd*=7nRz@za&UM$klL*3640b=@W- zTQ-GOVtuS4?KwT@6-u#Rb;KAO7!XoYc5?mvfAPnna%lPC@y}g2xZ+i=cd>Mg&a8=dQPB)>aIkLyvLmW)v$nCG2 z4wUE8;l>^>-uUs0?nvm)>DUOJyFhHb+T`@WD7wP5mbWI-FQklepmsR--38K*I;#C| z`V<)Y-ecfHH^36oOvIwG|NR=}Ejbhl1;h6oQ1UQiL*r&zq!wl%iUd)WTOo{VfZle3 zP6TcUy06wSX5d5VaDhGol>G9QD+UGzRUqnk_UxI3g#~E4826BCz&L@Dq^-I6TrCB% zFlZgLS7xF={o4VhlE_L*q{rap=66Q#8upFy?`#;-#MLF|jK%+!X6|3tZ=}fz3`doS z;p{to<5W>(Bmnm6WD$ua2%@ws;O|D)W412T3O`P2qQEt_G|0f0;aR)=^&ZIfHe`P- z&DT1!oYiA$yE^pi=-Z1p8!Yn+&W-|Z&*{V}WvQI`=+C^AJL+$Vy_V@f@qmOw z#BDBdPRx5-JLK8j^h2F@hi#&-U#8LRr84F|SRyeZ=w8*Ks;3a>YI|Qbu2jtSWPHSh z(frtRHRHG8bqewj2DE)|9GcM?`_4@Np+Ei~8(E5DF5AI4b>orl9CGZlJb5%}sasM| zY*LNhmYSQp%Nj4PDrvM84i*DrYBk}5o)X5r=L+)?X4`{QNF{fP@FET0nXF45k0%fE zvT0XSc?aX)i^w%`Q@^x*c#vILINh3(EuYffq7x8^A|Ltm=wb#ih&Q#p2j2$8DLSKiQdXMxW|>PMSMd z-lHm2itwnZY&!}k3^nOXd3z^gaN+#ojWaieAwvp+$c8_g*;M{zuQ2OBS>B-9{%XG`atJ zo5AQ4&oU`xrF+(eAvefsZ3>dMa!>wE3B*CLrqFoP?nPe%D7KY}sF&Q{y@dHjoqs)y(q zMJ0_gcVpG=`@}Y_USo&$c@1nsUnXO4;(}zJD*WE0qPiRCefgr?HA^HCEu-D=0E#e| z+M3QmAuD5z)_tpj-{Z~ry!_W!vX|Eb#b;Joc0F44b(4RkMCz^@cAIE-t$fYU@#Ewp z$Hk!394EkxFbNc+zIDUiR2oI9l$MKk{WenGnCpAHCeK)@b#CRhMWn0xFc*&`BXa5G z@}z~aG>UC;AV4(n0=m+~B0b3e=$iRN0QqKIZq*CaXnbb4vs3|VzNNd!=JzdJKVOPj z%97_>H*xaCwHvIl>U@d}C)&f^xT2YJ70A(lnkSmJX|b1>)`p+Gv@{hc#mVYg*rcc| zJb2X8c=G`7M&vdZtrl!xwrzEOG4a9ZZ%zXcP;`&(A50aY<)n9?%}AUc;PEVkj<*VdK^f2;j`% zu^V#IK&R`Gt**L&iYM&_xzND_Y!Yu=f7M;TT+h(GqVPl0kE*7fg|8$(_GjwP(iOdO z<9v!PBtH7IdHO?M(Q|E0e%07g*YvFYWh<+3=aDI1ThS}Ng3S;dtSxR+nKILlYDTO( zbOth);u#&el69#fgXOele+!wF-@^P9 zf8^xsLTg@NxgDVwn;}QWdT1LQ{T^Usjni_U>is$9VPsoQNQ_r@neahCbI-@-j0w|` zLjBER|G6MS*StqrSzSsg3Akrnr!Iw)-(}>k@fdb=sv-H}TUZ8@`3n^y{!}b1N!$0y z#8y`+J^fMB>(1$W|4@?C?Kb{FQJBH~+=xc^EX}RR$qvdy^@l5;h3)K`5L~dPP8cK3 znRqfGN?9`H67`s|P8nSdq9ge3;@V9a?(!ixbV;xfmWcOGn`r}xul2U%CVpJB- z`*t9H!$WcznR04)JRiB;Q?=U^FfNLc98f6|GqOB-|HtS>8hbZ);+oT2rLCd?;k(6+ z+81IaTo4t0nl8YVh@9p69r?@@3<-2Os zxUEVlraeN0#b+;i1>=J`fAe_`NEk^5g@(&y*_ES>o0%k6%~rVh#!g2TQN7>$A7}NL zcZ$oI6lJl*uKwQK;OACNe4E)~*+Jf!n|*t_Kgu_Um~fPZnPi}&#Ia@cTFBYNyXG^? zCG`s&t`(;bc}1vi)hBDKJM2_SWCcpsXrbj?RE2jVdh=};E*=?2JYq`Bvj})D-dr;1 zg7b4nQ`+k-OLD$EK5aG`6~V7B4iT5;Rgn@kvgD%?{mi#bZAkCbCXZ7gqy8jWOwno< zeW1n04ZtAu73E}7`J^dMh{+e+X8*_V?J!3e`v+&)TQ+yXpT=K=Y9`5E7|&Ak6}d## zHx)ulDfa16ZCI%CS((3LhOdrP<9o(Ea`gVRdT~06-4%gLvUB4W@369aRRQ~nGflfU#`hU1du(OZkmW}SQ?yu@|GwbkIHbR&iB2U zD7)fBPmC>%`4x3{&-e0;ECILg^|)d>jF3_75ZG1S-_TI|My7dqV=UGyIg5xhn%;`^ zNr1{ro`5ey$>_UEblZm?+4g8ag-0E$oOTc_}!mb z*%9{ovY~3sFC%WA_%I8&6wXF9IJC_8zdN$Ya&@?uzqOua(c-T=;U0A4TbDb)0(;>y zO549fMjGN?)et4x${KUH{stFsN-wCnKq6Bq_+8%*pIyrbP7O0gqOEjT2DA!y>3n!7 z6jj+$LX)4%Kj$xxZPjXdxg|-zc7y*b))SPTNaWP_00oh6RF_3ADBd^T(JueG8xVPa z(Kgv_`(o#COwKrJQ}D{#>ihYIi0DWeimehnNvxr>_USM$w3>KL#AXTMdy!^ePfp#J z?d&WB%d3IDm$8dedzyHRIEUH06+^mykgeXV!X zUCS_Y7(C*VjEam*bK`xdj%{V~TF6=+t(barRc7yIHW(G;=$G8c2@Ic1V zcvoaBcae=!6UY~RK#4fh`A(UfDbw6V%4aq=cpf-@`npA@hFqF&svHZwm-x{lpt=)> zc-fu*D^=pRtoJB_-nhuTFM{@7JFj2O9S2Cp%KF#{?Z28DPuJUa;Jp4!SvtTs-)&xQ zHHjf7P)5~~CqVUSWA8il{uI~rhb9RX4@>5p5@^3m*^CB<#uF4q?Bbi z7%6sCVoHRURHKL+tb)P_9~8zLj8Tzm5DG54-#($j-`3wrM33#u-A;&o;4#cV^Kj%> zCQ02Nf4ge@Lh`Z5UG(j@bWrky{U3>->V?W1V)|}Tow}F@G9=dY3FWkMFN;(`65a@Fd*3E?zhq&G@aL$A(qAbXZK6P!&yN3D!f4_l zcYSz%(Oo^Zz!e%_{*#)l@J&p$8s(=bRl)ex{t#MgvcG)9s`NFBc5p`vZuM# zBkPD(sgQV4>>u?8yE$0(C^lr-zhZq8@K!W6+R_Q5S(zl<9h7vk~oKDp{LFpro zO-|*9VhRIgi-C3uzZn-yYRcoD2i%tivuEowhbr6a3j(q)<&zd$;o!2dQ|fY=opN{I zA99&s>vy9Zz)*i3s5pxB^kcEXR?mx7;Zo0g2gyf0I8`Hh6IcCptZkp!IBnw!TABBu z(PttLWaaDdBDbgC_Ho9F&#sG#v@j<=Yb^@xqkd_$B`|en-uv*q@!Cv2V=AVfk%q_j zjhBsl51lSTXYXqq7RmM#`;;kXd6+ed5gUEA6(0O z3%c`Ins+|A%6=D^O_JMiRvBG9YNV)p@-0Nj97v=-Ap#ywBzfMpt*L*b_SE(&#n$m>SPS=@lnxVd|n#$&%4xjo5F)&U9z#Y!^9( zGHn~{r6~HE*1VC;(^b0H{KjDK9=?1=q{VbTHyzV*R5H#$7 zZ!e^sg|0o&^W17;wWtR#)|3g|f6;`sFF<))Bsw%{*ac-nm4w@L*Q~FOzR>4qD;l>6 zfsn@SIQR=eMI~2-rpWM)mFgh?Y;C^gZL~f3yOQ7aL3d0?P2CpFbM8fuMU($~Sn_Pw^6hee%k1XQpFb6|ltRCJ zv0WK_z0{j=RTl5`52@fTZ*KDg#ANdgsp`SOL6CH|=(B@F3^?EhfN0O*0$W?#s-i1b zu7Cpm@Q_6Q^X<_f083L~(4K6& zfOr!R@7&fV1+D>$AyCN)3dZ^xwnq_CP-y)nkITx+5~t2AYWYtjbC+IWz`pw4b0`JX zevuk;uz0Bto?mie;#*%}wzf-$&qZ_3+7(5fJn>5(A8tZ}x@pT>#9D#x# zpO*gFKzwX&s$(0v4KKJuPEIyBSS!or`0t3Ky^wzkZPkJFd1+)o-SpqXTXtznGc)>w z=X_ZFf6if31H4pT#LdkO)(L-NR?rKAHG zhY9wVgFa<#!5lDyRe-X9hl|S`gq!m6c*;hWmLAjfeocK8Lau{9emr3a#lHREYhK>3 zymKXIZ+#xq$lw%+7&n$CUHD;h!PvM_-ebfGK!2bhrV;E$rpfptYyqE{cdzgX2te~d z9W0(KZw&4&Zp-oGi~Kbp`~X7h&Ye4E=H_66!anz!L1fLX`E(m`0;22e8U1eHd%_bR z@J#?eAWjX0EI1W_f^^l;1O6OHF0#B(h~&P?_{x5JAXfqEwA^@_JY#(@n6DQNelR_~ z-&0e6#vCRVdHzTxaXH#=WW!~)xgX4igB~7s53HVTAZG?$Fl*-g-;gmp3o>r-AGrb$ z-q6qh0`=)r0-)p0_G-@Dk-p8oSqhw$04jkgYvctG4afWgFSst({a{H~Tz#$f&2jK=DQt5Ff)h&+Xl z=o6L_7Y7X^JeXQ7#>$Jp6#=pc*qxf!zpu6y@WomH-Ft4b8m~O~Oe3ihR^@hNUt%#f zIyR=MsY%SF^8DMYPzjn_*8wu!rdeEEBn_N9KG*`B4nR8)XNEWHgZkJ|l^YmBAgnZH{RG7jqYu~{2`FML5oAfOKZ>Xng4@*kvzCVRu+*)7;wkLoY;V)olu(HYv4-e1H zT?Y-XO4w>|#tSnurU5gET+MRZj56Kl&wqgycn1gUAnK~B2kkr)fD)IO^f7RA!#E`! zFEj?sOGAUf_Cl9QQIx?DQ0WIpM?lv&Ecb&4L;Y|lpXsQ9So;Pkr+8MM&{+@GPTfOF zfpBnMiKldf79=}6J4pD{alRuK@N&=ykqSC<;zHD)0LKr01c*SbYS#letYJ*5-Jo0z z2ndjbpaq=^5X%#xpb!#p-1v0w2?%-WH61oLHo(52fsTv>zgsI9QJf)aKw<(p7M40K zH5E>l%jQgTAiIMjNue6^j|Yb*<5uAK1mGCL2uRm%fL_7WQ3h14HfZ$0=V9>@2S{~L z`B0r}mmvQF0iB~&ZB?Z!vb#UU%$9uSgd*JcAt8UQfOh4v@kor|o(!WtVI0`ENR21FQ0ZUFc@ zJ3G6&{s0*cm&U=t;qT{{pO=?D)CUR)u^@^naAYYfD^pU&M?^$8m)wRzvA#Z$qsfnq zdZ1GW38+YIl`-Oer5Pb1;nsZTyp{|}-nmJLnVH#3LJ~xGen++BmwAA1*hI3uIzV<6x$n6kLCLSIhm@OeG>_Wop#Vm<>G%b5A>{-6 z1j1=>h~_|E0^g$$@atc{)^>M)X5{4Ip-Y6Ua^0&A{C>m)1Z@yWU`XPsfN}}a=16vf z443mW#nk z^W(5<9<=u})jTTn1l-4pa$q2-s&WJC5UAxKmP+eD-~dPX^0jN{W*Rm&Ikb|ClM@%{ zP5R4W{~&Jj{#I%1|M?vl5CB1wlzT4Y2ZsjuKW=+N$?pJG791+utVgBmd~;!RGBHMm zhK7(g200Kn-`fZ|vcUT&Wc7hW1USO;OU2BBOrMzv-ak{K@fwes(oz*_JTQG}Ryps0 zSq_d)mE)hp>MU5qb0d{HHyaxXmx%%#^V!z$Ws6==4}z7_fdY=+;}tIBLw*fN-w+IC z8n~EVD@p4^^w5J0onk0<>Oq4vif#benuo2RR^6Se@X{j6KMZm|Hi$38Im#?zr7jy_qk5y+%)M3w+=y%8w^6f*RS(s(x`wOYtW>f zZJdPGg^+M3k@m`t$fl=fugMPZAD|OEfjs2fFbzI&Fq~WS=z#mY8uka)2BhYLAn=Fp z2!QitWo4zNHe9h@-8#1053cZY_Y+V23q?gP>zSXx1TWa~g&71<3cH^kJFRWqxRbBa zHYNe?2XLN81P4DxfwTGyY@gZ(D^n>p_V(58Cw#c~2|mCE^=7s-HwU-QqP9yVr#q?X z=;nV133qgMg794zw1=akqaeKyL7i{%devU(SrE5wy)U$9t78@6SR#T@2!V%ej~A}V z%>3YMF;QIwZd;HqAt5S9?}A%)xWtl`02AzQaG}BAKmdV17;BHg8Hp+cLlk&CZK}#D zDm>0k_So3iU`53%x(o(KMJ6JUJOWL=zrWvGnF1xt9RAzc*jND179>K| zFzDINz%L6Yac*U$(x?j=X3!bSKfk_e(EH#dg_B`6>OZkF0A z=V*{ym6aWU$J=A6_dYmJL0$tUr1io4AKl&7kj+5xxC$~pIJjdj{AZxI1mzdBqJrxo zSV5laf60iy_sRVCG%^2u9n7zK;HN=0$+b(5hj=s8^Adj=XQSmD*p*M7afdHcbe zzns5;VjM4P5lCwro7VPrnna)~O~IahZ<Ar}w{BA$uLpmIcBzWsjy+%oNQ literal 0 HcmV?d00001 From 472008ce98ae288b6026a3af220b89ab286bec4a Mon Sep 17 00:00:00 2001 From: Thomas Gummerer Date: Tue, 9 Dec 2025 15:12:47 +0100 Subject: [PATCH 11/14] maybe? --- content/blog/journaling/index.md | 40 ++++++++++++++++---------------- 1 file changed, 20 insertions(+), 20 deletions(-) diff --git a/content/blog/journaling/index.md b/content/blog/journaling/index.md index 9679721e0f68..58689b15a3e9 100644 --- a/content/blog/journaling/index.md +++ b/content/blog/journaling/index.md @@ -23,37 +23,37 @@ tags: social: twitter: | -Pulumi deployments just got up to 10x faster: + Pulumi deployments just got up to 10x faster: -- Journaling: Send only changes, not full snapshots -- Data integrity: No compromise on reliability -- Network traffic cut by 85%+ on large stacks + - Journaling: Send only changes, not full snapshots + - Data integrity: No compromise on reliability + - Network traffic cut by 85%+ on large stacks -Try it: [link] + Try it: [link] linkedin: | -# Pulumi Deployments Get Up to 10x Faster + # Pulumi Deployments Get Up to 10x Faster -Large Pulumi stacks just got a major performance boost. Here's what changed. + Large Pulumi stacks just got a major performance boost. Here's what changed. -# The Problem -Pulumi saves snapshots at every deployment step to maintain data integrity. For large stacks, this creates a bottleneck: uploading the full snapshot serially slows everything down. + # The Problem + Pulumi saves snapshots at every deployment step to maintain data integrity. For large stacks, this creates a bottleneck: uploading the full snapshot serially slows everything down. -# The Solution: Journaling -Instead of sending the whole snapshot, journaling sends only individual changes. These journal entries can go in parallel, and Pulumi Cloud reconstructs the full snapshot on the backend. + # The Solution: Journaling + Instead of sending the whole snapshot, journaling sends only individual changes. These journal entries can go in parallel, and Pulumi Cloud reconstructs the full snapshot on the backend. -# The Results -In benchmarks on a 3,000+ resource stack: + # The Results + In benchmarks on a 3,000+ resource stack: - Time dropped from 58 minutes to 3 minutes - Network traffic cut from 16.5MB to 2.3MB + Time dropped from 58 minutes to 3 minutes + Network traffic cut from 16.5MB to 2.3MB -# Why This Matters -You get the speed without sacrificing data integrity. Unlike SKIP_CHECKPOINTS, journaling still tracks all in-flight operations. If something fails mid-deployment, Pulumi still knows exactly what happened. + # Why This Matters + You get the speed without sacrificing data integrity. Unlike SKIP_CHECKPOINTS, journaling still tracks all in-flight operations. If something fails mid-deployment, Pulumi still knows exactly what happened. -# Get Started -This feature is in opt-in testing. Reach out on Pulumi Community Slack or through Support to get your org enrolled. Then set PULUMI_ENABLE_JOURNALING=true. + # Get Started + This feature is in opt-in testing. Reach out on Pulumi Community Slack or through Support to get your org enrolled. Then set PULUMI_ENABLE_JOURNALING=true. -Read more: [link] + Read more: [link] --- From 26f8f2b0ea78a564e463e015a9e51a191fff12e1 Mon Sep 17 00:00:00 2001 From: Thomas Gummerer Date: Tue, 9 Dec 2025 15:15:38 +0100 Subject: [PATCH 12/14] update --- content/blog/journaling/index.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/content/blog/journaling/index.md b/content/blog/journaling/index.md index 58689b15a3e9..ee9e23fde4c1 100644 --- a/content/blog/journaling/index.md +++ b/content/blog/journaling/index.md @@ -1,5 +1,5 @@ --- -title: "Speeding up Pulumi deployments by up to 10x" +title: "Speeding up Pulumi deployments by up to 20x" date: 2025-12-08T17:57:55+02:00 @@ -9,7 +9,7 @@ draft: false # of the content of the post, which is useful for targeting search results or # social-media previews. This field is required or the build will fail the # linter test. Max length is 160 characters. -meta_desc: Pulumi deployments get up to 10x faster with journaling, a new snapshotting approach that speeds up large stacks while keeping full data integrity. +meta_desc: Pulumi deployments get up to 20x faster with journaling, a new snapshotting approach that speeds up large stacks while keeping full data integrity. meta_image: meta.png @@ -23,7 +23,7 @@ tags: social: twitter: | - Pulumi deployments just got up to 10x faster: + Pulumi deployments just got up to 20x faster: - Journaling: Send only changes, not full snapshots - Data integrity: No compromise on reliability @@ -31,7 +31,7 @@ social: Try it: [link] linkedin: | - # Pulumi Deployments Get Up to 10x Faster + # Pulumi Deployments Get Up to 20x Faster Large Pulumi stacks just got a major performance boost. Here's what changed. @@ -57,7 +57,7 @@ social: --- -Today we're introducing an improvement that can speed up deployments up to 10x. At every deployment, and at every step within a deployment, Pulumi saves a snapshot of your cloud infrastructure. This gives Pulumi a current view of state even if something fails mid-operation, but it comes with a performance penalty for large stacks. Here's how we fixed it. +Today we're introducing an improvement that can speed up deployments up to 20x. At every deployment, and at every step within a deployment, Pulumi saves a snapshot of your cloud infrastructure. This gives Pulumi a current view of state even if something fails mid-operation, but it comes with a performance penalty for large stacks. Here's how we fixed it. @@ -87,7 +87,7 @@ The second example is setting up an instance of the Pulumi app and API. Here we' ![Time taken](time.png) -![Data sent](data.png) +![Data sent](size.png) *Note that this feature is still behind a feature flag, but we are ready for testers. To get enrolled in the feature flag, please reach out to us, either on the [Community Slack](https://slack.pulumi.com/), or through our [Support channels](https://support.pulumi.com/hc/en-us). Once that's done, all you need to do is to set the `PULUMI_ENABLE_JOURNALING` environment variable to `true`, and your deployments will start finishing faster.* @@ -316,7 +316,7 @@ Pulumi state is a very central part of Pulumi, so we wanted to be extra careful - Since tests can't cover all possible edge cases, the next step was to run the journaler in parallel with the current snapshotting implementation internally. This was still without sending the results to the service. However we would compare the snapshot, and send an error event to the service if the snapshot didn't match. In our data warehouse we could then inspect any mismatches, and fix them. Since this does involve the service in a minor way, we would only do this if the user is using the Cloud backend. - Next up was adding a feature flag for the service, so journaling could be turned on selectively for some orgs. At the same time we implemented an opt-in environment variable in the CLI (`PULUMI_ENABLE_JOURNALING`), so the feature could be selectively turned on by users, if both the feature flag is enabled and the user sets the environment variable. This way we could slowly start enabling this in our repos, e.g. first in the integration tests for `pulumi/pulumi`, then in the tests for `pulumi/examples` and `pulumi/templates`, etc. - Allow users to start opting in. If you want to opt-in with your org, please reach out to us, either on the [Community Slack](https://slack.pulumi.com/), or through our [Support channels](https://support.pulumi.com/hc/en-us), and we'll opt your org into the feature flag. Then you can begin seeing the performance improvements by setting the `PULUMI_ENABLE_JOURNALING` env variable to true. -- Enable this for everyone. After some time with more orgs opted in, we'll enable this feature for everyone using Pulumi Cloud, and by default in the Pulumi CLI. +- Enable this for everyone. After some time with more orgs opted in, we'll enable this feature for everyone using Pulumi Cloud, and by default in the Pulumi CLI. This is currently planned for the end of January. ## What's next From 3cee18bf3071ba37f34ca4ad702e1e6ff44f50d3 Mon Sep 17 00:00:00 2001 From: Thomas Gummerer Date: Wed, 10 Dec 2025 18:38:36 +0100 Subject: [PATCH 13/14] Apply suggestions from code review Co-authored-by: Pat Gavlin --- content/blog/journaling/index.md | 30 +++++++++++++++--------------- 1 file changed, 15 insertions(+), 15 deletions(-) diff --git a/content/blog/journaling/index.md b/content/blog/journaling/index.md index ee9e23fde4c1..77b80b7ec1dd 100644 --- a/content/blog/journaling/index.md +++ b/content/blog/journaling/index.md @@ -9,7 +9,7 @@ draft: false # of the content of the post, which is useful for targeting search results or # social-media previews. This field is required or the build will fail the # linter test. Max length is 160 characters. -meta_desc: Pulumi deployments get up to 20x faster with journaling, a new snapshotting approach that speeds up large stacks while keeping full data integrity. +meta_desc: Pulumi operations get up to 20x faster with journaling, a new snapshotting approach that speeds up large stacks while keeping full data integrity. meta_image: meta.png @@ -39,10 +39,10 @@ social: Pulumi saves snapshots at every deployment step to maintain data integrity. For large stacks, this creates a bottleneck: uploading the full snapshot serially slows everything down. # The Solution: Journaling - Instead of sending the whole snapshot, journaling sends only individual changes. These journal entries can go in parallel, and Pulumi Cloud reconstructs the full snapshot on the backend. + Instead of sending the whole snapshot, journaling sends only individual changes. These journal entries can be written in parallel, and Pulumi Cloud can reconstruct the full snapshot on the backend. # The Results - In benchmarks on a 3,000+ resource stack: + In benchmarks when creating a 3,000+ resource stack: Time dropped from 58 minutes to 3 minutes Network traffic cut from 16.5MB to 2.3MB @@ -63,13 +63,13 @@ Today we're introducing an improvement that can speed up deployments up to 20x. ## Benchmarks -Before getting into the more technical details, here are a number of benchmarks demonstrating what this new experience looks like. To run the benchmarks we picked a couple of Pulumi projects, one that can be set up massively parallel, which is the worst case scenario for the old snapshot system, and another one that looks a little more like a real world example. Note that all of these benchmarks were conducted in Europe connecting to Pulumi Cloud, which runs in `us-west-2`, so exact numbers may vary based on your location and internet connection. This should however give a good indication of the performance improvements. +Before getting into the more technical details, here are a number of benchmarks demonstrating what this new experience looks like. To run the benchmarks we picked a couple of Pulumi projects: one that can be set up massively parallel, which is the worst case scenario for the old snapshot system, and another that looks a little more like a real world example. Note that all of these benchmarks were conducted in Europe connecting to Pulumi Cloud, which runs in AWS's `us-west-2` region, so exact numbers may vary based on your location and internet connection. This should however give a good indication of the performance improvements. -We're benchmarking two somewhat large stacks, both of which are or were used at Pulumi. The first program sets up a website using AWS bucket objects. We're using the [example-ts-static-website](https://github.com/pulumi/examples/tree/master/aws-ts-static-website) example here, but expand it a little bit to set up a version of our docs site. This means we're setting up more than 3000 bucket objects, with 3222 resources in total. +We're benchmarking two somewhat large stacks, both of which are or were used at Pulumi. The first program sets up a website using AWS bucket objects. We're using the [aws-ts-static-website](https://github.com/pulumi/examples/tree/master/aws-ts-static-website) example here, but expand it a little bit to set up a version of our docs site. This means we're setting up more than 3000 bucket objects, with 3222 resources in total. The benchmarks were measured using `time` built-in command and using the best time in a best-of-three benchmark. The network traffic was measured using `tcpdump`, limiting the measured traffic to only the IP addresses for Pulumi Cloud. Finally `tshark` was used to process the packet captures and count the bytes sent. -All the benchmarks are run with journaling off (the default experience), with journaling on (the new experience), and finally with `PULUMI_SKIP_CHECKPOINTS=true` set. The last one means we skip uploading intermediate checkpoints to the backend, which in turn means potentially losing track of changes that are in flight if Pulumi exits unexpectedly due to any reason. +All the benchmarks are run with journaling off (the default experience), with journaling on (the new experience), and finally with `PULUMI_SKIP_CHECKPOINTS=true` set. The last configuration skips uploading intermediate checkpoints to the backend, which increases performance at the cost of potentially losing track of changes that are in flight if the `pulumi` CLI loses connectivity or exits unexpectedly. | | Time | Bytes sent | |--------------------|--------|------------| @@ -93,17 +93,17 @@ The second example is setting up an instance of the Pulumi app and API. Here we' If you are interested in the more technical details read on! -## Introduction into snapshotting +## Introduction to snapshotting -Pulumi keeps track of all resources in a stack in a snapshot. This snapshot is stored in the chosen backend, either in Pulumi Cloud, or in a DIY backend. Further deployments of the stack then use this snapshot to figure out which resources need to be created, updated or deleted. +Pulumi keeps track of all resources in a stack in a snapshot. This snapshot is stored in the stack's configured backend, which is either the Pulumi Cloud or a DIY backend. Further deployments of the stack then use this snapshot to figure out which resources need to be created, updated or deleted. To make sure there are never any resources that are not tracked, even if a deployment is aborted unexpectedly (for example due to network issues, power outages, or bugs), Pulumi creates a new snapshot at the beginning and at the end of each operation. -At the beginning of the operation, Pulumi adds a new "pending operation" to the snapshot. Pending operations declare the intent to mutate a resource. If a pending operation is left in the snapshot (in other words the operation started, but Pulumi couldn't record the end of it), in the next operation Pulumi asks the user to check the actual state of the resource, and then either removes it from the snapshot, or imports it depending on the users input. This is because it is possible that the resource has been set up correctly, or it is possible that the resource creation failed. If Pulumi aborted midway through the operation it's impossible to know which it is. +At the beginning of the operation, `pulumi` adds a new "pending operation" to the snapshot. Pending operations declare the intent to mutate a resource. If a pending operation is left in the snapshot (in other words the operation started, but `pulumi` couldn't record the end of it), the next operation will ask the user to check the actual state of the resource. Depending on the user's response, `pulumi` will either remove the operation from the snapshot or import the resource . This is because it is possible that the resource has been set up correctly or that the resource creation failed. If `pulumi` aborted midway through the operation, it's impossible to know which state the resource is in. -Once an operation finishes, the pending operation is removed, as we now know the final state of the resource, and the final state of the resource is updated in the snapshot. +Once an operation finishes, the pending operation is removed and the resource's final state is recorded in the snapshot. -There's also some additional metadata that is stored in the snapshot, that is only updated infrequently. +There's also some additional metadata that is stored in the snapshot that is only updated infrequently. Here's how the snapshot looks in code. This snapshot is serialized and sent to the backend. `Resources` holds the list of known resource states, and is updated after each operation finishes, and `PendingOperations` is the list of pending operations described above. @@ -117,7 +117,7 @@ type Snapshot struct { } ``` -Before we dive in deeper, we also need to understand a little bit about how the Pulumi engine works internally. Whenever a Pulumi operation, e.g. `pulumi up`, `pulumi destroy`, `pulumi refresh` etc. is run, the engine internally generates a series of steps, to create, update, delete etc. resources. This series of steps is then executed. To maintain the correct relationship, the steps need to be executed in a partial order, where all steps whose dependent steps have been executed already can be executed in parallel. +Before we dive in deeper, we also need to understand a little bit about how the Pulumi engine works internally. Whenever a Pulumi operation, e.g. `pulumi up`, `pulumi destroy`, `pulumi refresh` etc. is run, the engine internally generates and executes a series of steps, to create, update, delete etc. resources. To maintain correct relationships between resources, the steps need to be executed in a partial order such that no step is executed until all of its step dependencies have successfully executed. Steps may otherwise execute concurrently. As each step is responsible for updating a single resource, we can generate a snapshot of the state before each step starts, and after it completes. Before each step starts, we create a pending operation, and add it to the `PendingOperations` list. After that step completes, we remove the pending operation from that list, and update the `Resources` list, either adding a resource, removing it, or updating it, depending on the kind of operation we just executed. @@ -133,15 +133,15 @@ Our workaround for that is to serialize the snapshot uploads, uploading one snap This impacts performance especially for large stacks, as we upload the whole snapshot every time, which can take some time if the snapshot is getting big. For the Pulumi Cloud backend we improved on this a little [at the end of 2022](https://github.com/pulumi/pulumi/pull/10788). We implemented a diff based protocol, which is especially helpful for large snapshots, as we only need to send the diff between the old and the new snapshot, and Pulumi Cloud can then reconstruct the full snapshot based on that. This reduces the amount of data that needs to be transferred, thus improving performance. -However, the snapshotting is still a major bottleneck for large Pulumi deployments. Having to serially upload the snapshot twice for each step does still have a big impact on performance, especially if many resources are modified in parallel. +However, the snapshotting is still a major bottleneck for large Pulumi deployments. Having to serially upload the snapshot twice for each step does still have a big impact on performance, especially if many resources are modified in parallel. Furthermore, the time spent performing textual diffs between snapshots scales in proportion to the size of the data being processed, which adds additional execution time to each operation. ## Fast, but lacking data integrity? As long as Pulumi can complete its operation, there's no need for the intermediate checkpoints. It is possible to set the `PULUMI_SKIP_CHECKPOINTS` variable to a truthy value, and skip all the uploading of the intermittent checkpoints to the backend. This, of course, avoids the single serialization point we have sending the snapshots to the backend, and thus makes the operation much more performant. -However, it also has the big disadvantage that it's compromising some of the data integrity guarantees Pulumi gives you. If anything goes wrong during the update, Pulumi has no notion of what happened until then, potentially leaving orphaned resources in the provider, or leaving resources in the state that no longer exist. +However, it also has the serious disadvantage of compromising some of the data integrity guarantees Pulumi gives you. If anything goes wrong during the update, Pulumi has no notion of what happened until then, potentially leaving orphaned resources in the provider, or leaving resources in the state that no longer exist. -Neither of these solutions is very satisfying, as the tradeoff is either performance or data integrity. We would like to have our cake and eat it too here, and that's exactly what we're doing. +Neither of these solutions is very satisfying, as the tradeoff is either performance or data integrity. We would like to have our cake and eat it too, and that's exactly what we're doing with journaling. ## Enter journaling From e6ae7fe9ba3003947f99c0b764966bc9954b06d6 Mon Sep 17 00:00:00 2001 From: Thomas Gummerer Date: Wed, 10 Dec 2025 18:41:39 +0100 Subject: [PATCH 14/14] deployments -> operations; Pulumi -> `pulumi` --- content/blog/journaling/index.md | 32 ++++++++++++++++---------------- 1 file changed, 16 insertions(+), 16 deletions(-) diff --git a/content/blog/journaling/index.md b/content/blog/journaling/index.md index 77b80b7ec1dd..5e3a584838ef 100644 --- a/content/blog/journaling/index.md +++ b/content/blog/journaling/index.md @@ -1,5 +1,5 @@ --- -title: "Speeding up Pulumi deployments by up to 20x" +title: "Speeding up `pulumi` operations by up to 20x" date: 2025-12-08T17:57:55+02:00 @@ -23,7 +23,7 @@ tags: social: twitter: | - Pulumi deployments just got up to 20x faster: + Pulumi operations just got up to 20x faster: - Journaling: Send only changes, not full snapshots - Data integrity: No compromise on reliability @@ -31,7 +31,7 @@ social: Try it: [link] linkedin: | - # Pulumi Deployments Get Up to 20x Faster + # Pulumi Operations Get Up to 20x Faster Large Pulumi stacks just got a major performance boost. Here's what changed. @@ -48,7 +48,7 @@ social: Network traffic cut from 16.5MB to 2.3MB # Why This Matters - You get the speed without sacrificing data integrity. Unlike SKIP_CHECKPOINTS, journaling still tracks all in-flight operations. If something fails mid-deployment, Pulumi still knows exactly what happened. + You get the speed without sacrificing data integrity. Unlike SKIP_CHECKPOINTS, journaling still tracks all in-flight operations. If something fails mid-deployment, `pulumi` still knows exactly what happened. # Get Started This feature is in opt-in testing. Reach out on Pulumi Community Slack or through Support to get your org enrolled. Then set PULUMI_ENABLE_JOURNALING=true. @@ -57,7 +57,7 @@ social: --- -Today we're introducing an improvement that can speed up deployments up to 20x. At every deployment, and at every step within a deployment, Pulumi saves a snapshot of your cloud infrastructure. This gives Pulumi a current view of state even if something fails mid-operation, but it comes with a performance penalty for large stacks. Here's how we fixed it. +Today we're introducing an improvement that can speed up operations up to 20x. At every deployment, and at every step within a deployment, `pulumi` saves a snapshot of your cloud infrastructure. This gives `pulumi` a current view of state even if something fails mid-operation, but it comes with a performance penalty for large stacks. Here's how we fixed it. @@ -89,15 +89,15 @@ The second example is setting up an instance of the Pulumi app and API. Here we' ![Data sent](size.png) -*Note that this feature is still behind a feature flag, but we are ready for testers. To get enrolled in the feature flag, please reach out to us, either on the [Community Slack](https://slack.pulumi.com/), or through our [Support channels](https://support.pulumi.com/hc/en-us). Once that's done, all you need to do is to set the `PULUMI_ENABLE_JOURNALING` environment variable to `true`, and your deployments will start finishing faster.* +*Note that this feature is still behind a feature flag, but we are ready for testers. To get enrolled in the feature flag, please reach out to us, either on the [Community Slack](https://slack.pulumi.com/), or through our [Support channels](https://support.pulumi.com/hc/en-us). Once that's done, all you need to do is to set the `PULUMI_ENABLE_JOURNALING` environment variable to `true`, and your operations will start finishing faster.* If you are interested in the more technical details read on! ## Introduction to snapshotting -Pulumi keeps track of all resources in a stack in a snapshot. This snapshot is stored in the stack's configured backend, which is either the Pulumi Cloud or a DIY backend. Further deployments of the stack then use this snapshot to figure out which resources need to be created, updated or deleted. +`pulumi` keeps track of all resources in a stack in a snapshot. This snapshot is stored in the stack's configured backend, which is either the Pulumi Cloud or a DIY backend. Further operations of the stack then use this snapshot to figure out which resources need to be created, updated or deleted. -To make sure there are never any resources that are not tracked, even if a deployment is aborted unexpectedly (for example due to network issues, power outages, or bugs), Pulumi creates a new snapshot at the beginning and at the end of each operation. +To make sure there are never any resources that are not tracked, even if a deployment is aborted unexpectedly (for example due to network issues, power outages, or bugs), `pulumi` creates a new snapshot at the beginning and at the end of each operation. At the beginning of the operation, `pulumi` adds a new "pending operation" to the snapshot. Pending operations declare the intent to mutate a resource. If a pending operation is left in the snapshot (in other words the operation started, but `pulumi` couldn't record the end of it), the next operation will ask the user to check the actual state of the resource. Depending on the user's response, `pulumi` will either remove the operation from the snapshot or import the resource . This is because it is possible that the resource has been set up correctly or that the resource creation failed. If `pulumi` aborted midway through the operation, it's impossible to know which state the resource is in. @@ -117,7 +117,7 @@ type Snapshot struct { } ``` -Before we dive in deeper, we also need to understand a little bit about how the Pulumi engine works internally. Whenever a Pulumi operation, e.g. `pulumi up`, `pulumi destroy`, `pulumi refresh` etc. is run, the engine internally generates and executes a series of steps, to create, update, delete etc. resources. To maintain correct relationships between resources, the steps need to be executed in a partial order such that no step is executed until all of its step dependencies have successfully executed. Steps may otherwise execute concurrently. +Before we dive in deeper, we also need to understand a little bit about how the `pulumi` engine works internally. Whenever a `pulumi` operation, e.g. `pulumi up`, `pulumi destroy`, `pulumi refresh` etc. is run, the engine internally generates and executes a series of steps, to create, update, delete etc. resources. To maintain correct relationships between resources, the steps need to be executed in a partial order such that no step is executed until all of its step dependencies have successfully executed. Steps may otherwise execute concurrently. As each step is responsible for updating a single resource, we can generate a snapshot of the state before each step starts, and after it completes. Before each step starts, we create a pending operation, and add it to the `PendingOperations` list. After that step completes, we remove the pending operation from that list, and update the `Resources` list, either adding a resource, removing it, or updating it, depending on the kind of operation we just executed. @@ -133,13 +133,13 @@ Our workaround for that is to serialize the snapshot uploads, uploading one snap This impacts performance especially for large stacks, as we upload the whole snapshot every time, which can take some time if the snapshot is getting big. For the Pulumi Cloud backend we improved on this a little [at the end of 2022](https://github.com/pulumi/pulumi/pull/10788). We implemented a diff based protocol, which is especially helpful for large snapshots, as we only need to send the diff between the old and the new snapshot, and Pulumi Cloud can then reconstruct the full snapshot based on that. This reduces the amount of data that needs to be transferred, thus improving performance. -However, the snapshotting is still a major bottleneck for large Pulumi deployments. Having to serially upload the snapshot twice for each step does still have a big impact on performance, especially if many resources are modified in parallel. Furthermore, the time spent performing textual diffs between snapshots scales in proportion to the size of the data being processed, which adds additional execution time to each operation. +However, the snapshotting is still a major bottleneck for large `pulumi` operations. Having to serially upload the snapshot twice for each step does still have a big impact on performance, especially if many resources are modified in parallel. Furthermore, the time spent performing textual diffs between snapshots scales in proportion to the size of the data being processed, which adds additional execution time to each operation. ## Fast, but lacking data integrity? -As long as Pulumi can complete its operation, there's no need for the intermediate checkpoints. It is possible to set the `PULUMI_SKIP_CHECKPOINTS` variable to a truthy value, and skip all the uploading of the intermittent checkpoints to the backend. This, of course, avoids the single serialization point we have sending the snapshots to the backend, and thus makes the operation much more performant. +As long as `pulumi` can complete its operation, there's no need for the intermediate checkpoints. It is possible to set the `PULUMI_SKIP_CHECKPOINTS` variable to a truthy value, and skip all the uploading of the intermittent checkpoints to the backend. This, of course, avoids the single serialization point we have sending the snapshots to the backend, and thus makes the operation much more performant. -However, it also has the serious disadvantage of compromising some of the data integrity guarantees Pulumi gives you. If anything goes wrong during the update, Pulumi has no notion of what happened until then, potentially leaving orphaned resources in the provider, or leaving resources in the state that no longer exist. +However, it also has the serious disadvantage of compromising some of the data integrity guarantees `pulumi` gives you. If anything goes wrong during the update, `pulumi` has no notion of what happened until then, potentially leaving orphaned resources in the provider, or leaving resources in the state that no longer exist. Neither of these solutions is very satisfying, as the tradeoff is either performance or data integrity. We would like to have our cake and eat it too, and that's exactly what we're doing with journaling. @@ -310,14 +310,14 @@ The full documentation of the algorithm can be found in our [developer docs](htt ### Rollout -Pulumi state is a very central part of Pulumi, so we wanted to be extra careful with the rollout to make sure we don't break anything. We did this in a few stages: +`pulumi` state is a very central part of `pulumi`, so we wanted to be extra careful with the rollout to make sure we don't break anything. We did this in a few stages: -- We implemented the replay interface inside the Pulumi CLI, and ran it in parallel with the current snapshotting implementation in our tests. The snapshots were then compared automatically, and tests made to fail when the result didn't match. +- We implemented the replay interface inside the `pulumi` CLI, and ran it in parallel with the current snapshotting implementation in our tests. The snapshots were then compared automatically, and tests made to fail when the result didn't match. - Since tests can't cover all possible edge cases, the next step was to run the journaler in parallel with the current snapshotting implementation internally. This was still without sending the results to the service. However we would compare the snapshot, and send an error event to the service if the snapshot didn't match. In our data warehouse we could then inspect any mismatches, and fix them. Since this does involve the service in a minor way, we would only do this if the user is using the Cloud backend. - Next up was adding a feature flag for the service, so journaling could be turned on selectively for some orgs. At the same time we implemented an opt-in environment variable in the CLI (`PULUMI_ENABLE_JOURNALING`), so the feature could be selectively turned on by users, if both the feature flag is enabled and the user sets the environment variable. This way we could slowly start enabling this in our repos, e.g. first in the integration tests for `pulumi/pulumi`, then in the tests for `pulumi/examples` and `pulumi/templates`, etc. - Allow users to start opting in. If you want to opt-in with your org, please reach out to us, either on the [Community Slack](https://slack.pulumi.com/), or through our [Support channels](https://support.pulumi.com/hc/en-us), and we'll opt your org into the feature flag. Then you can begin seeing the performance improvements by setting the `PULUMI_ENABLE_JOURNALING` env variable to true. -- Enable this for everyone. After some time with more orgs opted in, we'll enable this feature for everyone using Pulumi Cloud, and by default in the Pulumi CLI. This is currently planned for the end of January. +- Enable this for everyone. After some time with more orgs opted in, we'll enable this feature for everyone using Pulumi Cloud, and by default in the `pulumi` CLI. This is currently planned for the end of January. ## What's next -While these performance improvements hopefully make your day to day use of Pulumi quicker and more enjoyable, we're not quite done here. We're looking at some other performance improvements, that will hopefully speed up your workflows even more. +While these performance improvements hopefully make your day to day use of `pulumi` quicker and more enjoyable, we're not quite done here. We're looking at some other performance improvements, that will hopefully speed up your workflows even more.