AI-empowered Personal Knowledge Graphs in Obsidian.
The topic of personal knowledge graphs is quite wide, and even definitions could be pretty diverse :
Knowledge about user
The knowledge that users care about
Snapshot of EKG or OKG with the context of the user
We could 100% agree that that is user-centric, so the center or graph is the user node.
You use personal knowledge graphs every day, even without knowing. Google creates pkg of your search habits and behavior. Your medical insurance probably has a pkg about you.
Why should we care?
How often have you had situations when you collect papers, bookmark pages, and make notes and, in a few days, have no idea where they are?
We have a few gigantic challenges in the modern world :
information overload
Information fragmentation
Information retrieval
Overload
Individuals process gigabytes of data every day. Too much information often turns it from value to noise. Personal knowledge graphs are a form of information compression and filtering, together with a form of classification and structuring of knowledge. We could use a personal knowledge graph as a filter.
Fragmentation
Well, we are locked in apps. Something is locked to our browser, and something stays in our Kindle. A lot remains in our device file system in various vendor-locked formats. Personal knowledge and knowledge graphs could be used as a cross-storage layer over heterogeneous data, gluing together different applications and data sources.
Obsidian and zettelkasten could be effective anti-fragmentation layers.
Retrieval
Finding information in the context of a new question or query is challenging.
Personal knowledge graphs and graph-oriented query languages paired with semantic search could change the complexity of search and retrieval
it is all about relations.
One of the most powerful features of personal knowledge graphs is reasoning and link prediction—we can create and reveal new knowledge from existing graphs and discover new links and contexts for old data.
do it yourself !!
The first, quite popular personal knowledge system that used a superpower of graphs was a zettelkasten.
It was an utterly offline system based on a shoebox and a6 card of ideas. Every card had a unique ID and space for the number of IDs of connected cards.
You can read more in my article.
Personal Knowledge Graphs in Obsidian
Personal Knowledge Graphs
in Obsidian Personal Knowledge Graphsvolodymyrpavlyshyn.medium.com
Modern obsidian is enough to do it perfectly out of the box without even any extra plugins.
Why do we need more?
As I built my zettelkasten over time, I discovered that I needed more powerful graph capabilities to see links.
How Zettelkasten Solves a Problem?
Zettelkasten is a simple framework for knowledge management is based on a nonlinear and network approach developed in the 70th in the time of full offline and paper tools.
The process consists of a few activities
summarizing ideas in one's words.
linking ideas together
keeps links to sources of idea
You keep only knowledge in your words in a context that matters to you. All is a permanent note that contains links to each other, so you build a unified and consistent layer on top of different data sources
The Flow of Zettelkasten
I zellelkasten you operate with short notes written on A6 paper. Not all notes are created equal. The system has a few utility tools, like fleeting and literature notes, that allow users to capture ideas for future processing.
Fleeting Notes The Zettelkasten method starts with what is called “fleeting notes.” These are quick, temporary notes that you jot down when an idea strikes you They can be as simple as a sentence or a keyword. The purpose is to capture the thought before it escapes your mind.
Literature Notes: When you read a book, article, or any other source of information, you create “literature notes.” These are summaries or paraphrases of the material, written in your own words. The aim is to distill the essence of what you’ve read, making it easier to review later.
Permanent Notes The next step is to convert these literature notes into “permanent notes.” These are well-thought-out notes that you integrate into your Zettelkasten system. They are written in a way that makes them understandable, even when taken out of context. Each permanent note should focus on a single idea or concept.
Linking and Indexing: The real magic of Zettelkasten comes into play when you connect these permanent notes. By creating links between related notes, you’re building a web of interconnected ideas. This enables you to see the relationships between different pieces of information, facilitating more profound understanding and creative thinking.
More Power with Plugins
Pure zettelkasten insists that you should capture only your ideas in notes and reformulate them with your words. It is forced memorization and requires a lot of work.
I found it time-consuming but valuable. I have missed opportunities overlooked before the digital era and information fragmentation.
We often work with data on different applications and devices, and I want a convenient way of making notes and captures while I'm on the go.
Another big challenge that is overlooked is that the internet is not permanent storage. I recommend storing your Obsidian app PDF and related documents as a snapshot of web pages.
I had some cases where the page gave me 404 instead of content
So my workflow is described in the following diagram
Obsidian Clipper—The Internet is not immutable. Pages and articles similar to those in PDF could be deleted. Clipper helps you convert an article to a Markdown Obsidian note. It can improve readability and remove the visual nose, so you get immediate benefits without any extra processing.
Kindle Export and Kindel highlights—All who use and read on Kindle must have this. You get a note per book with all your notes and highlights. So you are halfway done turning it into permanent notes and idea destinations.
Obsidian Annotator — Sync a separate note to all PDF and epub highlights and notes.
Highlighter-Plugin helps with progressive summarization. read more about progressive summaries in an article
Highlight them All
OBSIDIAN and Progressive summarization
From Zettelkasten to Personal Personal Knowledge Graph
Now, our Graph looks at a connection of permanent notes with reference to literature notes that refer to article clippings or highlighter notes or annotation notes from a PDF etc. I also use an Excel drawing, but it is a topic for a separate article about visual thinking. To make a Knowledge Graph out of it, we need the ability to add context and metadata to links. I have already covered this topic in my previous article.
Why It Matters to You
The transformation from a General Knowledge Graph to a Personal Knowledge Graph is simple but powerful.
You need to add yourself as a “Me” node in the center of the graph and filter and categorize information in a user-centric way. How is every data point related to you, your interests, projects, etc.?
If you see a set of notes without a path to the Me node, why do these notes matter to you?
Too big and to Reach
A6 limits you in writing and forces you to distill ideas in your own words. It is cool, but it is too big even in this context. Your note is full of entities and supportive context that is hard to link manually. So, we need semiautomatic entity extraction for notes.
To make notes more smart and digestible, we need two simple procedures
- entity extraction
- link discovery
We extract categorized entities and find links between them as an extra step.
As a result, we have a local knowledge graph that is closer to a concept map.
It improves information retrieval and understanding.
Entity links as bridges
time to go global!
We could merge local graphs to one global semantic graph. We could keep this resulting graph as a second layer that keeps links to notes the same as to semantic entities.
We link notes together, so we manually link to rich blocks of entities and reveal only a few possible connections.
If we get the possibility to create a connection of entities in the same personal knowledge graph, we will see more complex and rich structures that will give more capabilities for reasoning.
more structure with ontology.
So, together with reference or literature notes, we have flitting notes and our cornerstone of everything — permanent notes. Side by side with a zettelkasten ontology that postpones categorization and loves changing a context, we could have a small domain-specific ontology for entities that reach nodes with metainformation.
So, things are not strings. Every entity could represent structured knowledge about a person or location or any other robust data point that gives a better understanding and context.
Another ontology superpower is templates and constraints for relations and nodes. We could identify missed data or derive new data point.
How do we get there?
Well, we’re still good to go with Obsidian, but now we need an AI assistant with a few tools or small language models together with a vector search database.
So we need :
entity extraction and summarization SLM
Slm or Ilm that know how to build a graph and extract relationships for discovered entities
categorization and ontology expansion LLMs
Vector search db that is able to search for entities
management layer that creates a processing pipeline
In the new article, we could use Local LLM and Obsidian plugins to make it happen.
If you have no time to wait, try mykin.ai, which builds a semantic personal
Kin. A personal AI for your private life.
Get inspired, talk things through, navigate situations or get personalized guidance with Kin. Built for privacy…
knowledge graph from your conversations.
Stay tuned.