Observer not working for `smapi`

[ydirson]

In v24.39.1, we noticed that the observer.py wrapper does not get called when an observer is enabled on the smapi component.

The traces added in #6280 reveal the following exception:

Jan 28 10:48:01 xcpng83-bzkcpvhy xapi: [error||0 |check if component smapi is enabled  D:181f882cc62e|xapi_observer_components] is_component_enabled(smapi) inner got exception: Db_exn.DBCache_NotFound("missing table", "Observer", "")

Readablized stack trace:

Raised at Xapi_database__Db_cache_types.TableSet.find in file "ocaml/database/db_cache_types.ml", line 262, characters 22-73
Called from Xapi_database__Db_cache_impl.read_refs in file "ocaml/database/db_cache_impl.ml", line 255, characters 12-70
Called from Db_actions.DB_Action.Observer.get_all in file "ocaml/xapi/db_actions.ml", line 21421, characters 40-69
Called from Xapi_observer_components.is_component_enabled.(fun) in file "ocaml/xapi/xapi_observer_components.ml", line 83, characters 26-56

From what I understand:

• the code calls Db.Observer.get_all
• that one is generated code making use of an in-memory cache
• said cache misses the Observer table even when the XAPI state.db has it

What could be causing this?

[contificate]

This is very peculiar. Usually, this exception is seen at startup (e.g. if you moved an old build to a host).

---

It would be worth instrumenting in some way to see what database the Context.t used by Xapi_observer_components believes it has: the exec_with_new_task handling should just yield a context that takes reference to the master database.

Do other DB actions - regarding Observer - also fail? Something like Observer.get_by_name_label (fully auto-generated, Server's local action will immediately call into Db_actions).

[ydirson]

Do other DB actions - regarding Observer - also fail? Something like Observer.get_by_name_label

From python it looks pretty OK:

s = get_xapi_session()
xapi = s.xenapi
res = xapi.Observer.get_by_name_label("ydi-obs2")
print(f"got: {res}")
print(f"{xapi.Observer.get_name_label(res[0])}")

got: ['OpaqueRef:b1423e14-2859-7f0d-4a5a-dc901b401e07']
ydi-obs2

[ydirson]

Do other DB actions - regarding Observer - also fail? Something like Observer.get_by_name_label

From python it looks pretty OK:

Just ignore that - I should have checked that first... from python the get_all() call works well too, digging...

[contificate]

Is there a possibility that this is doing a DB action prior to the database being initialised on startup?

A lot of code in Xapi is written in a way where evaluation of required ambient state is delayed until first use: e.g. any code using Xapi_globs can't reliably be done at module initialisation, as the values likely won't have been initialised yet. These cases technically amount to trying to mitigate race conditions, but usually it's expected that enough of Xapi must be "up" for various code paths to be reached anyway.

[ydirson]

Is there a possibility that this is doing a DB action prior to the database being initialised on startup?

I don't think so, as this happens systematically, every time I do a common operation (mainly testing with VM.start and VM.shutdown), and the observer does send telemetry data for xapi and xenopsd components.

[ydirson]

Checked adding a call to get_by_name_label() in is_component_enabled, it does fail with a similar exception.

[contificate]

Checked adding a call to get_by_name_label() in is_component_enabled, it does fail with a similar exception.

It may be worth trying some other DB actions, affecting other tables, in the same place.

---

As an aside, it would be worth adding stats : Db_ref.t -> (string * int) list to the DB_ACCESS signature and creating relevant implementations for the remote DB protocol implementations. Then, you'd be able to call it like:

let open Xapi_database in
let r = Context.database_of __context in
let module DB = (val Db_cache.get r : Db_interface.DB_ACCESS) in
let stats = DB.stats r in
List.iter (fun (table, count) -> D.debug "%s => %d" table count) stats; ...

This really should've been part of the DB_ACCESS signature, as it's simpler when you just want rough statistics.