Messaging: Weekly Digest

[joethreepwood]

Messaging

Blockers

• Need to confirm with @raquelmsmith that all the events are correctly set up
• The design (especially on the Hogstradamus section) could do with some @corywatilo love. May be a nice opportunity to move to the proposed new email designs? Feel strongly that this would be a big and impactful change for a lot of users.

Pitch

Continuing from the hackathon: @raquelmsmith's idea is to have a weekly digest of certain events sent to all users. We've set up the intrumentation for this, and the messaging is broadly done. I've also added this to the new product notifications subscription tag, so users can unsubscribe from this if they don't want it.

Here's the trigger:

The goal, broadly, is to encourage more feature usage and log ins. So, I'd propose adding it to a hold-out test where 50% of users would get the email and 50% won't. We'll then monitor as a success metric if users log in more and activate with more products.

What I was trying to do with the Hogstradamus section was mockup something like this, but it... is shit.

Here's a peek of what the code looks like for each section:

Last week your team made {% if event.new_dashboards_in_last_7_days %}{% if event.new_dashboards_in_last_7_days > 1 %}<a href="https://us.posthog.com/dashboard" target="_blank" style="color: #f54e00;">{{ event.new_dashboards_in_last_7_days }}</a> dashboards{% elsif event.new_dashboards_in_last_7_days == 1 %}<a href="https://us.posthog.com/dashboard" target="_blank" style="color: #f54e00;">1</a> dashboard{% elsif event.new_dashboards_in_last_7_days < 1 %}no new dashboards{% endif %}{% else %}<a href="https://us.posthog.com/dashboard" target="_blank" style="color: #f54e00;">some new dashboards</a>{% endif %} and {% if event.new_event_definitions_in_last_7_days %}{% if event.new_event_definitions_in_last_7_days > 1 %}<a href="https://us.posthog.com/data-management/events" target="_blank" style="color: #f54e00;">{{ event.new_event_definitions_in_last_7_days }} new events</a> {% elsif event.new_event_definitions_in_last_7_days == 1 %}<a href="https://us.posthog.com/data-management/events" target="_blank" style="color: #f54e00;">1 new event</a> {% elsif event.new_event_definitions_in_last_7_days < 1 %}no new events{% endif %}{% else %}<a href="https://us.posthog.com/data-management/events" target="_blank" style="color: #f54e00;">some new events</a>{% endif %}. Check out our templates for more ideas.

And here's how it looks with the fallback error copy:

[raquelmsmith]

@joethreepwood I think we should include more info about the items. Basically list out the experiments that were started, the flags that were made, etc. Something like:

• New experiments
  • Green vs Blue Button Color started on Nov 14
  • Move onboarding buttons started on Nov 16
  • (one more)
  • and 3 other experiments (link to experiments list)

The hogs are fairly overwhelming IMO. I would remove most of them.

@corywatilo can you help us with some design here?

[joethreepwood]

Yep, def agree on the hogs. TBH I had a much grander idea for the design originally (wanted it to look like an actual newspaper) but...hackathon 🤷

Let me look at modifying the liquid for the other stuff tomorrow.

[jamesefhawkins]

a bunch of customers will have poor usage - this will probably cause them to churn. do you think we should consider this scenario?

[joethreepwood]

Raquel and I discussed this briefly at the offsite, as I agree it's a risk. One idea I've since had is basically setting up a filter on the campaign that means it only triggers if users have usage above a certain amount. This would mean we'd probably want to change this to a monthly digest, however.

We could then set it so that it only sends if a user has created more than 3 dashboards and either 1 playlist, experiment, or warehouse source (for example).

I also think a monthly cadence would be less annoying for users here.

[joethreepwood]

@corywatilo can you help us with some design here?

TBH, if @corywatilo has time to spend on emails I'd rather have that focused on the new email templates that work in dark mode. I'll want to move this to that template as soon as they're available anyway.

I've tinkered together some improvements in the mean time and am working on adding in the new liquid. See below.

a bunch of customers will have poor usage - this will probably cause them to churn. do you think we should consider this scenario?

I agree with this btw, @raquelmsmith -- I've changed the copy to be monthly, which I think will work better. Easy to change to something else if you want though.

[joethreepwood]

ok, new code is all in place

[raquelmsmith]

@joethreepwood I'd like this to be weekly because I'm planning on adding a lot more to it.

We should just not show a section if they aren't using it or there is no content for it.

If there are fewer than 2 sections with content we can just not send it.

I think it should actually list out the data like I mentioned here.

Just saying "some new experiments were launched" isn't actionable at all. But saying "an experiment called change homepage background color was launched 3 days ago -> check results" is much more actionable.

[joethreepwood]

@joethreepwood I'd like this to be weekly because I'm planning on adding a lot more to it.

I do feel somewhat strongly this is too often, but for now I'll change the copy back to weekly and we can watch the unsubscribe rate.

Just saying "some new experiments were launched" isn't actionable at all. But saying "an experiment called change homepage background color was launched 3 days ago -> check results" is much more actionable.

Yeah, what you're seeing now is just the fall-back error code that I put in for testing. This only displays if the event is completely not present. This can't actually occur IRL because the event itself is the campaign trigger - so this is just for testing until we have your PR merged and real data sent.

The issue with formatting the data the way you cited is length. If someone launches a lot of experiments, has a lot of dashboards, etc then that bullet list you suggested will get very long and unwieldy - especially if they have complex names. Both of these feel like likely situations.

I'm deliberately omitting info about time at the moment -- it just gets too much to say A launched X days ago, B launched Y days ago, and C launched Z days ago. Also, D completed W days ago, E completed V days ago. etc. The important info is the state (launched/completed) and the name (which is descriptive), not the period.

The actual outputs from the current Liquid code (see bottom) compromise on this. If a user has multiple launched and completed experiments:

You launched 3 experiments last month, including Experiment A, Experiment B, Experiment C. You also completed 2 experiments, including Experiment X, Experiment Y. [Check the results here!](https://us.posthog.com/experiments)

If no experiments launch/complete, we omit that sentence:

You completed 1 experiment last month, called Experiment X. [Check the results here!](https://us.posthog.com/experiments) or You launched 1 experiment last month, called Experiment A. [Check the progress here!](https://us.posthog.com/experiments)

And if the data is present but has no values (e.g. an error), we say:

It looks like we couldn’t retrieve your experiments data. If you think this is an error, please [open a support ticket](http://app.posthog.com/home#supportModal) for assistance!

We should just not show a section if they aren't using it or there is no content for it.

Customer.io doesn't have a reliable way to remove the entire row if nothing launched or completed. We could not display text, but that'd leave the rest of the space, the image, and available heading. We can remove these elements to accomodate a text only solution, but I think that also becomes less actionable and is something we should iterate on once we have data.

Right now, if there's no data on the event then we display the below. I decided to link to the docs because they're more actively updated and therefore less repetitive than any guides we have -- but once this is out I can definitely iterate variations into this.

You didn’t launch or complete any new experiments last month. Maybe our [experiments guide](https://posthog.com/docs/experiments) can help!

If there are fewer than 2 sections with content we can just not send it.

I presume you mean not send the email, rather than the event? I'll have to give some thought to how I'd build that trigger.

---

Here's the code I wrote to power the above, if you want to check it. There's similar code for each section.

{% if event is not none %}
    {% assign launched_count = event.new_experiments_launched_in_last_7_days %}
    {% assign completed_count = event.new_experiments_completed_in_last_7_days %}
    {% if launched_count or completed_count %}
        {% if launched_count %}
            You launched {{ launched_count | default: 0 }} experiment{% if launched_count > 1 %}s{% endif %} last month
            {% if launched_count > 1 %}, including {% for experiment in launched_count %}
                {{ experiment.name }}{% if forloop.last == false %}, {% else %}.{% endif %}
            {% endfor %}
            {% else %}, called {{ launched_count.0.name }}.{% endif %}
        {% endif %}
        {% if completed_count %}
            {% if launched_count %} You also completed {% else %} You completed {% endif %}
            {{ completed_count | default: 0 }} experiment{% if completed_count > 1 %}s{% endif %}
            {% if completed_count > 1 %}, including {% for experiment in completed_count %}
                {{ experiment.name }}{% if forloop.last == false %}, {% else %}.{% endif %}
            {% endfor %}
            {% else %}, called {{ completed_count.0.name }}.{% endif %}
            <a href="https://us.posthog.com/experiments" target="_blank" style="color: #f54e00;">Check the results here!</a>
        {% else %}
            <a href="https://us.posthog.com/experiments" target="_blank" style="color: #f54e00;">Check the progress here!</a>
        {% endif %}
    {% else %}
        You didn’t launch or complete any new experiments last month. Maybe our <a href="https://posthog.com/docs/experiments" target="_blank" style="color: #f54e00;">experiments guide</a> can help!
    {% endif %}
{% else %}
    It looks like we couldn’t retrieve your experiments data. If you think this is an error, please <a href="http://app.posthog.com/home#supportModal" target="_blank" style="color: #f54e00;">open a support ticket</a> for assistance!
{% endif %}

[joethreepwood]

Changed back to weekly ✅

If there are fewer than 2 sections with content we can just not send it.

This is actually proving incredibly complex to do because of how I believe the event is currently setup.

Currently, this campaign is triggered by the transactional_email event. Customer.io allows us to filter down on that event to only trigger based on certain properties but only using an ALL condition. We therefore can't use this to filter for At Least N

I've looked into solutions using TRUE/FALSE branches in the workflow, but the most workable solution in Customer.io is a segment filter that checks every possible variation of 2 event attributes existing - e.g. new_dashboards_in_last_7_days + new_sources_in_last_7_days, new_dashboards_in_last_7_days + new_surveys_in_last_7_days.

That's doable, but it gives us 21 possible variations right now - and would be a hassle to scale if you plan to add more here.

So, before we fall back to that: is there a solution we could work into your PR where we ensure weekly_digest only sends for users with 2 or more sections, or that there is a single additional eligibility value I can check in the trigger conditions?

[raquelmsmith]

Yeah I can put a single var like number_of_product_with_updates that just sums them up? Can you filter on that, so that it just has to be greater than 2?

[joethreepwood]

Yep, that's doable!

[raquelmsmith]

The issue with formatting the data the way you cited is length. If someone launches a lot of experiments, has a lot of dashboards, etc then that bullet list you suggested will get very long and unwieldy - especially if they have complex names. Both of these feel like likely situations.

We should list the first 3 and then say "and 8 more" that just links to the relevant page (eg the experiments list)

I don't think this should have a lot of "english" in it. It should be a concise list that's easy to scan and click what you want. Paragraphs are not the best solution here imo.

If they are just list items, we can show the time. It's also easier to deal with formatting and sentence structure.

Customer.io doesn't have a reliable way to remove the entire row if nothing launched or completed. We could not display text, but that'd leave the rest of the space, the image, and available heading. We can remove these elements to accomodate a text only solution, but I think that also becomes less actionable and is something we should iterate on once we have data.

I really think we should figure out how to not show entire sections. Having sections that are unused makes this whole email difficult to read and not very actionable, and presents the churn problem that james mentioned. "huh my team isn't doing X, maybe we should stop paying for it"

[raquelmsmith]

Here's a very rough mockup of what I was thinking:

Cory said he can pick this up next week!