Packaging... This mess...

[nathan-curtis]

Whats the 'best way to package this up... Is it packages? Blueprints?

[teskanoo]

I have a similar system to yours, it's all packages and included blocks with!include.

Almost all the jinja is in 5 custom templates making it harder to redistribute.

I have experimented with rewrite of all jinja to scripts but it degrades performance too much.

Such a shame we cannot specify alternative/additional folders from which t load custom templates.

[nathan-curtis]

Yeah and if you guys are tracking im actually planning on breaking the hardlogic in the scripts down to custom templates

Reason if I do it thag way the same module is Available both in 'the library - live context' and 'the prompt - at prompt time' in essence I'm writing a front door template that boils it all down into the core logic the llm will need building prompts, logging in, pulling the prompt source from cabinets, assembling itself, validate the hash and finally load authoritatively (some kind of public key exchange to the 'system' authorizing load.)

I can't write that crap twice. I'm having hard enough time organizing all the domains we have now.

Anyway the senos_ai command interpreter will be the front door for anything we built and I'll be moving all the script core logic out to it.

You want this. I get some really cool tricks if we do.

Ive got 1/3 of the identity script finished. (people, resolve guid to person dump person stuff.) now I have to have it lookup / dump households and families...

(im going to build a contextual family tree from the loaded cabinets and spell it out for the ai if youvd been following you see we put the framework into the format)

All that tk say yeah one big honkin or maybe two or three custom templates or six - teen... I don't know yet.

[teskanoo]

There is a fair bit of repetition in the scripts.
That is not a criticism at all - stuff grows organically and to get things done we have a tendency to borrow and evolve what we have borrowed, then things drift :)

I have been inspired by your project, borrowed from it and moved a lot into macros. Partly so I can get a better comprehension and partly so the code is all in one place. My macros return mostly JSON, I have not found this costly at all, but I'm not on a RPi either.

Here are some examples of utility macros I have implemented to simplify the YAML code around volume/cabinet and drawer retrieval - this has allowed me to more easily cope with schema/structural changes I make as I evolve my application.

Interestingly my library is also Zen related, but I chose it from Blake's 7 - Zen was the computer that controlled the Liberator spaceship.

What Follows is relevant to your filecabinet system / manifest / summarizer etc.

{#- zen_library.jinja -#}
{#- usage: import 'zen/zen_library.jinja' as LIBRARY -#}
{#-
########################################################################################
macro all_cabinets()
Returns a list of all cabinets in the application
########################################################################################
Mine have (historically) a device class and
unit_of_measurement set in YAML
-#}
{%- macro all_cabinets() -%}
    {%- set _candidates = states.sensor
        | selectattr('attributes.device_class', 'defined')
        | selectattr('attributes.variables', 'defined')
        | selectattr('attributes.device_class', 'in', ['data_size'])
        | selectattr('attributes.unit_of_measurement', 'in', ['B'])
        | map(attribute='entity_id')
        | sort
        | list -%}
    {%- set ns = namespace(keep=[]) -%}

    {%- for cab_entity_id in _candidates -%}
        {%- set volume_config = get_volume_config(cab_entity_id) | from_json() -%}
        {%- if volume_config -%}
            {%- set ns.keep = ns.keep + [cab_entity_id] -%}
        {%- endif -%}
    {%- endfor -%}
    {{- ns.keep | to_json(pretty_print=true) -}}
{%- endmacro -%}

{#-
########################################################################################
macro get_volume_config(entity_id: string)
Returns the configuration for a cabinet/book
########################################################################################
-#}
{%- macro get_volume_config(entity_id) -%}
    {%- set configuration_drawer_key = get_configuration_drawer_key() -%}
    {%- set rv = get_volume_drawer_value(entity_id, configuration_drawer_key, none) -%}
    {{- rv | to_json(pretty_print=true) -}}
{%- endmacro -%}

{#-
########################################################################################
macro get_volume(entity_id: string)
Returns the 'variables' attribute for a cabinet/book
########################################################################################
-#}
{%- macro get_volume(entity_id) -%}
    {%- set rv = state_attr(entity_id, 'variables') | default( dict({}), true) -%}
    {%- if rv is not mapping -%}
        {%- set rv = dict({}) -%}
    {%- endif -%}
    {{- rv | to_json(pretty_print=true) -}}
{%- endmacro -%}

{#-
########################################################################################
macro get_volume_drawer(entity_id: string, drawer_key: string)
Returns a 'specific' drawer from a cabinet/book
########################################################################################
-#}
{%- macro get_volume_drawer(entity_id, drawer_key) -%}
    {%- set rv = none -%}
    {%- set volume_variables = get_volume(entity_id) | from_json() -%}
    {%- if drawer_key in volume_variables -%}
        {%- set drawer = volume_variables.get(drawer_key) -%}
        {%- if drawer is mapping -%}
            {%- set rv = drawer -%}
        {%- endif -%}
    {%- endif -%}
    {{- rv | to_json(pretty_print=true) -}}
{%- endmacro -%}

{#-
########################################################################################
macro get_volume_drawer_exists(entity_id: string, drawer_key: string)
Returns true/false indicating if a cabinet/book contains a 'specific' drawer
########################################################################################
-#}
{%- macro get_volume_drawer_exists(entity_id, drawer_key) -%}
    {%- set candidate = get_volume_drawer(entity_id, drawer_key) | from_json() -%}
    {%- set rv = candidate is not none -%}
    {{- rv | to_json(pretty_print=true) -}}
{%- endmacro -%}

{#-
########################################################################################
macro get_volume_drawer_value(entity_id: string, drawer_key: string, _default: any)
Returns the contents ('value') of a 'specific' drawer from a cabinet/book
########################################################################################
-#}
{%- macro get_volume_drawer_value(entity_id, drawer_key, _default=none) -%}
    {%- set rv = _default -%}
    {%- set d = get_volume_drawer(entity_id, drawer_key) | from_json() -%}
    {%- if d is not none -%}
        {%- set rv = d.get('value', none) -%}
    {%- endif -%}
    {{- rv | to_json(pretty_print=true) -}}
{%- endmacro -%}


{#-
########################################################################################
macro get_volume_label_index(entity_id: string)
Returns the 'label_index' from a cabinet/book
########################################################################################
-#}
{%- macro get_volume_label_index(entity_id) -%}
    {%- set volume_variables = get_volume(entity_id) | from_json() -%}
    {%- set rv = get_volume_label_index_from_volume_variables(volume_variables) | from_json() -%}
    {{- rv | to_json(pretty_print=true) -}}
{%- endmacro -%}

{#-
########################################################################################
macro get_volume_label_index_from_volume_variables(entity_id: mapping)
Returns the 'label_index' from the volume_variables of a cabinet/book
########################################################################################
-#}
{%- macro get_volume_label_index_from_volume_variables(volume_variables) -%}
    {%- set rv = volume_variables.get('_label_index', {}).get('value', {}) -%}
    {{- rv | to_json(pretty_print=true) -}}
{%- endmacro -%}

[teskanoo]

I am using something like this for ACL
__USERS is actually just generated by iterating person entities...

{#- users.jinja -#}
{%- set __USERS = [
    dict(
        person             = 'person.teskanoo',
        id                 = state_attr( 'person.teskanoo', 'id' ) | default(none),
        friendly_name      = state_attr( 'person.teskanoo', 'friendly_name' ) | default(none),
        user_id            = state_attr( 'person.teskanoo', 'user_id' ) | default(none),
        device_trackers    = state_attr( 'person.teskanoo', 'device_trackers' ) | default([]),
        entity_picture     = state_attr( 'person.teskanoo', 'entity_picture' ) | default(none),
        entity_count       = label_entities( 'person_teskanoo' ) | count(),
    ),
    dict(
        person             = 'person.xxxx',
        id                 = state_attr( 'person.xxxx', 'id' ) | default(none),
        friendly_name      = state_attr( 'person.xxxx', 'friendly_name' ) | default(none),
        user_id            = state_attr( 'person.xxxx', 'user_id' ) | default(none),
        device_trackers    = state_attr( 'person.xxxx', 'device_trackers' ) | default([]),
        entity_picture     = state_attr( 'person.xxxx', 'entity_picture' ) | default(none),
        entity_count       = label_entities( 'person_xxxx' ) | count(),
    ),
    dict(
        person             = 'person.zen_ai',
        id                 = state_attr( 'person.zen_ai', 'id' ) | default(none),
        friendly_name      = state_attr( 'person.zen_ai', 'friendly_name' ) | default(none),
        user_id            = state_attr( 'person.zen_ai', 'user_id' ) | default(none),
        device_trackers    = state_attr( 'person.zen_ai', 'device_trackers' ) | default([]),
        entity_picture     = state_attr( 'person.zen_ai', 'entity_picture' ) | default(none),
        entity_count       = label_entities( 'person_zen_ai' ) | count(),
    )
] -%}

{%- macro users() -%}
    {{- __USERS | to_json( pretty_print=true ) -}}
{%- endmacro -%}

{%- macro find( user_id ) -%}
    {%- set match = __USERS
        | selectattr('user_id', 'eq', user_id)
        | list
        | first
        | default({})
    -%}
    {{- match | to_json(pretty_print=true) -}}
{%- endmacro -%}

And in zen/acl.jinja ...

{#- acl.jinja -#}
{%- import "zen/users.jinja" as USERS -%}
{%- macro user_can( user_id, action, object ) -%}
    {%- set rv = false -%}

    {%- set user_id = user_id | default('') -%}
    {%- set action = action | default('') -%}
    {%- set object = object | default('') -%}

    {%- set user = USERS.find(user_id) | from_json() | default({}) -%}
    {%- set acl = {} -%}
    {%- if object -%}
        {%- set acl = state_attr(object, 'acl') | default({}) -%}
    {%- endif -%}

    {%- set permissions = acl.get(action) | default({}) -%}
    {%- set person = user.person | default(none) -%}

    {%- if acl -%}
        {%- if user -%}
            {%- if permissions -%}
                {%- if person -%}
                    {%- if person in permissions -%}
                        {%- set rv = true -%}
                    {%- endif -%}
                {%- endif -%}
            {%- endif -%}
        {%- endif -%}
    {%- endif -%}
    {%- set data =
        dict(
            user = user,
            action = action,
            object = object,
            permissions = permissions,
            person = person,
            granted = rv,
            acl = acl,
        )
        | to_json(pretty_print=true) -%}

    {{- data -}}


{%- endmacro -%}

finally in my todo_conroller.yaml

    # -----------------------------------------------------------------------
    # Process Variables - ACL
    # -----------------------------------------------------------------------
    - alias: Variables - ACL - User
      variables:

        zen_ai_user_id: >
          {%- import "zeni/users.jinja" as USERS -%}
          {{- USERS.users() | from_json() | selectattr('person', 'eq', 'person.zen_ai') | map(attribute='user_id') | first  -}}

        acl_user_id: >
          {{- this.get('context',{}).get('user_id', none) or zen_ai_user_id -}}

        acl_user: >
          {%- import "zeni/users.jinja" as USERS -%}
          {{- USERS.find(acl_user_id) | from_json() -}}

    - alias: Variables - ACL Data
      variables:

        # -------------------------------------------------------------------
        # When the user is attempting to set an item 'completed', they need
        # specific permission to do that so we 'modify' the acl_action to
        # ensure that we check against that special form of 'update'
        # -------------------------------------------------------------------
        acl_action: >
          {%- set rv = v_request_data.get('action_type','') -%}
          
          {%- if rv in ['update'] -%}
            {%- for todo_entry in target_items -%}
              {%- set item_status = todo_entry.get('status','') -%}
              {%- if item_status in ['completed'] -%}
                {%- set rv = 'completed' -%}
              {%- endif -%}
            {%- endfor -%}
          {%- endif -%}
          {{- rv -}}

        acl_objects: >
          {%- set rv = [] -%}
          {%- if acl_action in ['read'] -%}
            {%- if read_sub_mode in ['Mode_Read_Todos'] -%}
              {%- set rv = [] -%}
            {%- else -%}
              {%- set rv = lists_to_read -%}
            {%- endif -%}
          {%- else -%}
            {%- set rv = lists_to_write -%}
          {%- endif -%}
          {{- rv -}}

    # -----------------------------------------------------------------------
    # Process ACL validation
    # -----------------------------------------------------------------------
    - alias: 'Variables: acl_errors'
      variables:
        acl_errors: >
          {%- import "zen/acl.jinja" as ACL -%}
          {%- set ns = namespace(errors=[]) -%}
          {%- set req = v_request_data -%}

          {#- ========================================================================= -#}
          {#- ========================================================================= -#}
          {#- ========================================================================= -#}
          {%- if (acl_objects | length) > 0 -%}
          
            {%- for target_object in acl_objects -%}
          
              {%- if not acl_user_id -%}
                {%- set e = "Error: Permission denied user_id not defined in context" -%}
                {%- set ns.errors = ns.errors + [e] -%}
              {%- elif not acl_user -%}
                {%- set e = "Error: Permission denied user {} not found".format(acl_user_id) -%}
                {%- set ns.errors = ns.errors + [e] -%}
              {%- else -%}
                {%- set test = ACL.user_can(acl_user_id, acl_action, target_object) -%}

                {%- if ( not test ) -%}
                  {%- set e = "Error: Permission denied user {} cannot do action {} on {}. Please request access if you think you should perhaps be able to perform this action".format(acl_user.get('person'), acl_action, target_object) -%}
                  {%- set ns.errors = ns.errors + [e] -%}
                {%- endif -%}
              {%- endif -%}
          
            {%- endfor -%}
          {%- endif -%}
          
          {%- if ( (ns.errors|length) > 0 ) -%}
            {%- set e = "Please request access if you think you should perhaps be able to perform this action" -%}
            {%- set ns.errors = ns.errors + [e] -%}
          {%- endif -%}
          
          {{- ns.errors -}}

    - if:
        condition: template
        value_template: "{{- acl_errors | length > 0 -}}"
      then:
      - variables:
          v_script_response: >
            {%- import 'zen/zen_response.jinja' as ZR -%}
            {%- set v_script_response = ZR.set_response_status(v_script_response, 'error', acl_errors) | from_json() -%}
            {{- v_script_response -}}

      - stop: Return ACL Errors to LLM
        response_variable: v_script_response

[teskanoo]

OK, so I have finally decided to give a whirl at adding this to my installation and have the following observation regarding the packaging.

If this is going to be (easily) addable to an existing installation, it really needs to be more simple than it currently is. The structure of the existing repo presumes that users are including their scripts and automations and templates using

automation: !include_dir_named xxxx/xxxx/automations/
script: !include_dir_named xxxx/xxxx/scripts/
template: !include_dir_named xxxx/xxxx/templates/
packages: !include_dir_named xxxx/xxxx/packages/

This creates a requirement to place everything in the right place in your installation which is (IMO) not really necessary and overcomplicates the installation.

With the files in a hierarchy as follows, the package becomes 'more packaged' and self-contained which will make versioning and updates more simple. All you will need is as follows (and to copy the contents of custom_templates to custom_templates/zenos_ai

I have highlighted the relocated files

homeassistant:
  packages: !include_dir_named xxxx/xxxx/packages

[nathan-curtis]

You're reading my mind and I'm actually thinking something similar in my head. So you're saying keep the YAML scripts each independent or as necessary (for maint) and then the zenos_foo_*.jinja all in custom_templates... Because HA.

I'm also considering two core pieces. A template sensor pack for system health. They're essential in the onboarding and a default cabinet pack and a custom addon that let's people BYO-cabinet without touching core files. Since packages don't care structure maybe /cabinets /health or?

[teskanoo]

Yes, I am saying keep everything under /packages/zeonos/
There are solutions to the custom_template 'situation' but they are a bit invasive for most people's systems (file write direct to the '/custom_templates/zenos/' folder)

I have relocated the files as a baseline in this pull request

#29

I am very enthusiastic about your project (it may show :) and I have cycles right now as I am not back at my day job until January (programmer)

[nathan-curtis]

What the current plan looks like and current todos to be 'RC shippable'

I have to

• finish the formatting script (ID and format all existing vols.) this goes in the Redirector (the other reason I wrote it - it's really a filesystem driver and will handle volume creation and health) that's today or tomorrow. I have testers waiting.
• package it (I'm building it as a package this time) then we'll release it for testing, and I'll start moving all the scripts and automations to the packages dir. Note when I do - I'll be moving Todo into core - mostly because I have something for it to do in the prompt engine. Possibly calendar as well. ...so, they'll ship too.
• setup the default prompts and prompt components, I'm considering shipping them as template sensors - (Basically like ROM) Then we can ship a new sensor to update them (or the user can do their own by tagging a specifically crafted alternate. This will let me start the boot prompt without a default cabinet and solves a fun little Catch-22 I ran into this week with init.
• finally, I have two intent_scripts one for select/input_select and one for number/input_number we need to get in there too

I'm doing final debugging today and will have the template sensors likely posted tonight. In my system I have intents as an include - you'll see them show up in the tree under /intents

[nathan-curtis]

ANd I want to get you those scripts asap - good job on that invalid YML in the index, completely missed that.