New user questions

[NickEastNL]

Hi everyone. So, I predominantly use Obsidian for writing, worldbuilding, and roleplaying. Ever since I started experimenting with AI assistants a few months ago, I've found that they significantly boost my creativity and nearly eliminate any of the struggles I've been facing for a few years now.

I've been looking for a way to use Obsidian and use my own notes to allow the LLMs to produce better and more relevant responses. For a while I've used GitHub Copilot in VSCode to work directly with the Markdown files, and it has worked for a bit. But it's obviously not tailored for it.

To be honest, I've been hesitant to try Obsidian Copilot, because I want to avoid subscriptions as much as possible (especially if I also have to pay for model APIs). I also have certain more personal requirements. I've therefore been working on my own assistant for Obsidian, but it's proving to take much more time and energy than I can realistically spare. So at this point I'd rather pay a bit more money to get something that "just works". But I still have my own personal requirements, so I'm wondering what the possibilities are.

1. If I understand correctly, Copilot doesn't split notes into chunks, it includes the entire note in the context. If so, why not? It's my understanding that splitting allows for more accurate and efficient use of context. And, well, I can have plenty of really long notes that logically don't need to be included in their entirety, leaving room for other relevant context.
2. I've been using the Fountain screenplay syntax to write, well, screenplays within Obsidian. I've also been planning to create a dedicated screenplay editor plugin that uses a custom view, so I'm not stuck with the limited Markdown post processor. But I notice that Copilot can only index Markdown files. Might it (some day) be possible to index non-Markdown (text) files or include them as context?
3. In addition, I've also been using CSV files as light-weight databases (using custom Dataview JS views) so I don't have to create dozens or hundreds of empty notes with only a handful of properties, while still allowing for querying and filtering. Is it possible to use these as context within chats? They're not for pure data storage, but just a convenient way to manage tables that are a bit more complex than regular Markdown tables.
4. I noticed that the Believer tier is being phased out. Is there a timeline for that? I'm willing to invest in this if it means no subscription, but Copilot doesn't perfectly fit my workflow yet. And I honestly cannot fully justify the price for what this might get in the future.

Perhaps a bit of an unusual request or remark, but I might be able and willing to assist with adding the features I would like to have (if they are considered appropriate of course), if it means getting them faster. As I've only just begun using Copilot, I'm not sure exactly what is part of the open source plugin, and what is part of the closed-source backend. All I know is that this plugin is much more mature than anything I could've made, and if I can somehow make it exactly what I'm looking, the better.

I apologize for this really long post, it's a bit difficult to describe my use case without being verbose. And I'm looking for as much information as possible, given the limited time left to get the Believer tier.

[WetHat]

@NickEastNL I totally get where you’re coming from. The idea of something that “just works” is appealing, but I haven’t seen any AI tool yet that truly does that out of the box. Every single one I’ve tried (ChatGPT, Gemini AI, Open-Webui, GitHub Copilot, Copilot Studio) needed effort, skills, and tinkering to get it aligned with my workflow. For me, the sweet spot has been using Obsidian Copilot projects to build domain‑specific assistants. Once you set them up, they feel much closer to “just works” because they’re tailored to my own notes and context.

On subscriptions: I hear the hesitation. But with the believer plan you actually get unlimited access to chat and embedding models (copilot‑plus‑flash, which currently looks like Gemini Pro) without extra API costs. The believer lifetime option is priced roughly at three years of subscription, so if you’re planning to stick with Obsidian Copilot long‑term, it’s a fair offer compared to juggling separate API bills.

About Fountain syntax: Fountain is just plain text. That means your screenplays can live inside Markdown files, and you can pull them directly into chat or project context.

For CSVs: I’ve shifted to dense Markdown pipe tables instead. Basically, I replace commas with |. It works great for priming context in chats and projects, though it’s a little more effort when querying with Dataview.

And on the bigger picture, Copilot really is the only choice right now. If you want AI deeply integrated with your vault, this is where the development energy is. Contributing to Obsidian Copilot’s growth seems like the best way forward, both to shape the features you’d like to see and to ensure the tool keeps evolving in directions that benefit workflows like yours.

[NickEastNL]

Hey, thanks for the reply.

With "just works" I was actually referring to using a working solution versus my own plugin. I've been working on my own tailored AI assistant for Obsidian for a few weeks now, but progress was frustratingly slow. I had hoped to be able to use the core features by December for my "real" projects, but it looks like it would still be a few months out before I can actually use it in "production". That would just be too long. Copilot here is a solution that already works, even if it doesn't quite have the features I was looking for (support for specific file types like Fountain and CSV as I mentioned, but also others).

As for LLMs, I like the freedom OpenRouter gives, and different models are suitable for different things. Since I shift a lot between writing, worldbuilding, roleplaying, and other creative and technical work, being able to switch to a wide variety of different models (and costs) is appealing. I'm fine with paying for that, but it means I'm less inclined to invest in additional costs for software or plugins that bring AI into my preferred workflow. While the Believer plan is still a hefty investment for me at the moment, the fact that it's a one-time payment is a good compromise. It's just that it'll disappear "soon" as I read, with no clear timeline, and I'm not 100% sold on it yet given the cost.

The reasons I'm not quite sold yet is largely because of the freedom I get with Obsidian as a whole, while being somewhat limited by the closed-source and "premium" nature of Copilot. But as I said before, I'm not sure what the closed-source backend actually does versus the open-source plugin. This is largely because of the plugins I use and the, arguably unusual, workflows I follow with Obsidian.

For Fountain, I currently use inline syntax in Markdown files, using a modified version of the "Fountain Editor" plugin (so it also renders properly in read view). But the Obsidian API is a little too limited when it comes to Markdown processing/rendering. So I was planning to write a dedicated screenplay editor plugin that creates a custom view for Fountain files and includes an editor and parser specific to Fountain. In addition, for the future I was also planning to use XML (based on another screenplay spec) for more control over the formatting. To avoid interfering with regular Markdown, I use a different extension so the views are registered only for those files. But Copilot (free) only sees Markdown files. I know from the docs that Copilot Plus allows for more file formats as context, but it's unclear if that also means it treats other file formats registered via plugins as vault files (for indexing and context), or as attachments, like PDFs and images.

As for CSVs, I'm unsure how I could use Markdown tables with Dataview in the same way. Admittedly, the way I use CSVs is perhaps a little counterintuitive, but that's what I love about Obsidian; it's not forcing me into a specific workflow. And it's still a little experimental, so it's not like I'm married to it. I simply want to avoid having a lot of empty notes to store a handful of properties just so I can use Bases or Dataview; while still being able to query, sort, and filter collections of data to create different tables in different notes. I had hoped Bases would basically do that; a single file with structured data; but I guess not.

Then there's the matter of context size. While doing research for my own plugin, I learned about splitting long documents into more manageable chunks, not just to optimize how much context is sent, but also to make retrieval more accurate; long documents may contain multiple (sub)topics that are not all relevant for a single query. This leaves more context space for other chunks that are more relevant. I noticed that Copilot always includes entire notes, and I've had initial prompts that were around 20,000 tokens, compared to other services I experimented with (particularly Novelcrafter, which encourages short snippets). Most notes aren't very long, but when it comes to Fountain scripts, I tend to write hour-long scripts (~60 pages) in a single file, and it's rarely useful to include the whole file as context unless the query is about the whole script (e.g. summarizing or analyzing it). It's also unclear what happens if I include a folder as context, because it doesn't look like it includes all files all the time.

Granted, I've only just scratched the surface of even the free version. I'll be spending more time with it by working on my actual projects now.

[WetHat]

@NickEastNL

how I could use Markdown tables with Dataview

I'm not sure what your exact use-case for CSV & Dataview is. This is how I handle Markdown tables:

const file = await app.vault.getAbstractFileByPath("data/mydata.md");
const content = await app.vault.read(file);

// Parse dense table (simple split by lines/pipe)
const rows = content.split("\n").map(r => r.split("|"));

dv.table(rows[0], rows.slice(2));
//            ^- header    ^- data

... Then there's the matter of context size.

Yup, I totally agree. I'm having issues with that as well. It appears that this is a transient issue, and once Obsidian Copilot nears release candidate status we should be having full access to the model's context window (that would be ~1M tokens for Gemini; ~200k for Claude; 400k for OpenAI ). That should be good even for larger Fountain scripts, but you'd still have the issue of context dilution (garbage in the context). Particularly, OpenAI models have an issue with that. I completely support the idea of context splitting e.g. by allowing [[note#section]] or [[note#^id]] for manual splitting. I'm not so much a friend of rule based or token limit splitting because they are not designed to provide semantically coherent chunks, and still would load garbage into the context.

...being able to switch to a wide variety of different models (and costs) is appealing

Indeed! I do that too.

I think Obsidian Copilot's development trajectory is quite appealing, but your guess where it will end up is as good as mine. At this point the believer plan investment is a leap-of-faith.
You could mitigate the risk by actively engaging in Copilot development, though

[NickEastNL]

Thanks for the tip, I'll give it a shot. That's an elegant solution for the table, I hadn't thought of that. In fact, I might have come up with one or two other solutions that would work for me. The reason I began trying CSVs is for a police procedural story/campaign I'm writing, to collect a list of clues and evidence for each "case" with various properties and organized by type. That way, it's easier to just add more when I want to, and also to prompt AI to come up with additional suggestions. I then reference the list in several notes, filtering it depending on whether I want to inspect the evidence left behind by the perpetrators, circumstantial evidence that may not be relevant, red herrings, and which clues have been found by the detectives. A lightweight "database" seemed appropriate, but I didn't want to put every little clue for every case in its own note as it's just a short description and a bunch of properties; that would defeat the goal.

But that's getting off-topic 😄

The idea of splitting is less about the maximum size and more about my concern for wasting tokens (and thus money). Being able to explicitly include sections of notes as context just as with entire notes is a nice idea, but it kind of defeats the point of RAG (at least based on how I understand it from my research). I can't always know what might be relevant, and in many cases the point of me using AI is so I can get answers quicker without having to manually search the vault. For my own assistant I got far enough to build a document splitter from scratch, using unified and remark to parse Markdown as an AST and split by specific tokens (mainly headings, but also code blocks and tables with a bit of leeway for special paragraphs). It produced fairly decent results, especially when I also added rules for my Fountain scripts.

But while I consider myself a fairly decent programmer, I'm not well-versed in advanced IT topics, especially AI. So I can't say for certain what is best. All I see is the amount of model calls and tokens used via the OpenRouter activity dashboard, and the resulting cost 😄 .

Which brings me to another thing that's quite important to me: transparency; knowing exactly what gets sent to the LLM. In pretty much any assistant I used that uses RAG, I don't get to see exactly what chunks it has found before they get sent to the LLM, especially when including a folder or relying on searching the entire "collection" (codebase, vault, etc.) without attaching any specific files. For my own assistant I implemented a "dry run" where the retrieval step happened before sending the whole prompt to the LLM so the user can inspect it. I found that Copilot includes a debugger which kind of shows the steps, and I also noticed that it does multiple steps (and multiple LLM calls), which I didn't really need myself; so whether a dry run is feasible in this case, I don't know.

In any case, thanks for the pointers. I'll wait and see if there will be more replies to this. In the mean time, I'll be testing the free version for a bit and see how far I can get with it. If I can at least wait until the end of December to get the Believer plan before it gets removed, I'll be satisfied.