Build a knowledge management system based on Obsidian

In 2020, I wrote an article How to Build Your Own Knowledge System?, which mainly reflects on the daily accumulation of technology, leaning towards methodology.

This article will explore the possibilities of personal knowledge management from the perspective of tools and practical use, based on Obsidian, and attempt to integrate it with the Hexo blog. Of course, this preference for tools and practices has a strong subjectivity; this article is entirely based on my personal practice and experience; there is no best tool, only the one that suits you best.

Preface

I have used many note-taking software, from Evernote to Leanote, from Gist to Typora, as well as various messy Markdown editors. From centralized platforms to self-built backends, and finally to pure text with Dropbox for syncing content, I increasingly feel that there is not much difference between them simply from the perspective of text editing. As for some third-party content platforms, such as Blog Garden, JianShu, and Zhihu, I have written about them; their biggest problems are content review and gradual entertainment, making me feel that any centralized third-party content platform is not user-friendly and unreliable.

Before using Obsidian, I was using Hexo to deploy my blog, Typora to edit MD files, and syncing everything through Dropbox across multiple devices. However, as content increased, I desperately wished for a tool that could categorize, organize, and quickly retrieve knowledge base content.

Sometimes I want to quickly locate certain previous content, but Hexo blog’s online search performs poorly when the number of words on a single page is very large, and unpublished content is also not searchable. Using Typora to edit a single file locally is quite convenient, but it is inconvenient for batch management, so I primarily replaced it with SublimeText’s directory search function.

However, the drawbacks are apparent; every time I search, I have to open SublimeText and specify the directory, and I cannot preview the MD effect in real-time. Additionally, based on pure-text search, it is inconvenient to categorize and filter content, much like the Tags or Categories of a blog.

Thus, I entered another cycle of tool exploration, like searching for a new hammer with a nail.

I must say that the current note-taking tool market is evolving rapidly; Evernote has long declined, and now it’s the era of linked notes.

Currently, more popular tools include Notion, Obsidian, and Logseq. After previous experiences, I have confirmed the following requirements for choosing a knowledge management tool:

1. Must natively support Markdown
2. Should not rely on any cloud or account system; data should be entirely controlled by oneself
3. Must have powerful search and categorization capabilities
4. Must be compatible with Hexo blog’s source files and directory structure without interfering with one another. The blog remains an important channel for my knowledge management.
5. Functionality should be customizable and extendable
6. Should be easy to migrate to other tools

After comprehensive evaluation, I found that Obsidian meets my needs relatively well; although the official version promotes its own account syncing service, it is not mandatory. My original goal was merely to use it as a purely local tool; I can also sync based on S3 services, which allows for more controlled data, and I will introduce the method later.

Moreover, it has a plugin market, with a good number of plugins available and the ability to develop plugins based on JS, making it quite convenient to extend functionality. There’s also the Dataview plugin, which allows me to write code to control the knowledge base, offering great flexibility for programmers.

Getting Acquainted with Obsidian

Let GPT-4 briefly introduce the features and advantages of Obsidian:

[!ai]+ AI
Obsidian is a knowledge management tool that organizes information through linking. Features include: Markdown-based text editing, with optional local or cloud storage and strong file compatibility; cross-platform support, allowing editing and viewing anytime, anywhere; and support for various file formats beyond text, including audio and video. Its advantages are: integrated concept management and note organization, second-level search functionality, the freedom to create a network of knowledge links, continuously building and optimizing your own knowledge system, and a graph view that visually displays thought processes and knowledge links.

What is referred to here as a network of knowledge links is the bi-directional linking. Content from one note can be directly referenced in another note without needing to copy it, so for identical content, only one version needs to be maintained, and if it requires revision, it can affect all references to it. Furthermore, this method also establishes a connection among knowledge, allowing related knowledge to be linked through one point.

The idea is great and very convenient.

However, the more features there are, the more reliance there will be on the current tools. My thought is that tools can assist in writing, but the content should ideally not be tool-dependent. Although I use Obsidian, my final output goal is in pure Markdown format.

Additionally, since my Hexo deployed blog does not support this linking method, I only use the universal Markdown syntax for publicly published content, while other content can use bi-directional links to a limited extent.

For instance, for a comprehensive collation of a technology or some knowledge, I could link to notes in individual pages or a canvas, bringing them together.

Moreover, the syntax is very simple:

1. [[file path#title]] is a link reference that redirects upon clicking.
2. Adding ! in front enables it to reference the content directly, effectively pasting the content and allowing for edits.

Using the Canvas whiteboard feature to organize content is very intuitive:

Data Migration

When I decided to use Obsidian, the first step was to migrate existing content. The vast majority of my previous articles and notes were hosted in the Hexo blog, committed to the GitHub repository; copying the Markdown source files of the blog into the Vault created by Obsidian can easily be recognized.

This is the benefit of managing note files based on Markdown; as long as the target tool supports Markdown format, data migration can be achieved smoothly. Moreover, the organizational format of files in the previous blog is relatively standardized, so I can simply treat the source file paths of the blog as the root path for the Obsidian Vault.

My goal is to use Obsidian to manage all source files, including blogs and notes, without affecting the normal publishing of blog content. Additionally, I want to have precise control over what content goes to the blog instead of publishing everything in the repository.

Files generated by Obsidian, such as .obsidian/.trash, etc., are isolated from the blog, and the content managed by Obsidian is stored in the _obsidian directory.

The .obsidian directory holds software settings, plugins, themes, and other resources. The .trash folder is where files deleted from Obsidian go; they are not deleted directly, providing an extra layer of assurance.

Integration with Hexo Blog

In a previous article How to Write a Good Technical Article#Publishing, I mentioned that I synchronize source files through Dropbox, push commands on my VPS, submit MD files to a separate Git repository, and trigger builds via GitHub Actions.

Upon connecting Obsidian, I did not want to change this workflow, so I made the blog and Obsidian’s Vault the same repository.

1. Dropbox + GitHub double backup
2. Unified management of Obsidian and blog

To achieve this, I need to configure the Hexo site to ignore all files under .obsidian/.trash and _obsidian during generation.

Modify the site’s _config.yml:

exclude:
  - "_obsidian/*"
  - ".obsidian/*"
  - ".trash/*"
  - "essay/tweets/*"
  - "notes/ue/*"

This way, all notes under _obsidian will not be generated by Hexo, achieving a private effect.

Then, I can simply commit all files containing Obsidian data to the blog-md repository:

No modifications whatsoever to my original blog management process are required; when I want to update an article, I just need to push this repository (even editing online via GitHub) to trigger CI automatic generation.

Plugin Recommendations

Having used Obsidian for over a year and installed dozens of plugins, most are effective in improving efficiency or expanding originally unsupported functionalities.

Functional Extensions

• remotely-save: By default, syncing is not available without an Obsidian account, unless using some third-party tools like Dropbox, etc. It works well on PC but is inconvenient on mobile; remotely-save supports S3 storage buckets for syncing Vault content in an object storage fashion. (The only drawback is that synchronizing the .obsidian/plugins folder across multiple devices only allows additions and modifications; uninstalling plugins cannot be automatically removed.)
• dataview: I consider this an essential plugin in Obsidian; it allows embedding JS code in notes, enabling dynamic effects like calculating dates, weather, or filtering notes in Obsidian by specified rules, all of which can be achieved through your own code. I have also written some scripts, which I will discuss later in Usage Optimization.
• pangu: Supports board features in Obsidian, which I use for task categorization and management.
• Natural Language Dates: Supports inserting dates in natural language; for example, to insert the current time in a note, just input @now, and it fills automatically.
• Full Calendar: Expands calendar functionality, allowing event addition.
• drawio: Opens a draw.io canvas in Obsidian.
• gist: Supports inserting gist code via code blocks.
• excalidraw: Can be used to draw hand-drawn style flowcharts.
• Paste image rename: Automatically renames images after inserting (if the image exists locally).
• Image auto upload Plugin: Calls PicGo for uploading when inserting an image and pastes the uploaded image link.
• Markdown Prettifier: Automatically formats MD files and can set modified values for properties, like automatically adjusting updated time after edits, but it may lag with large files (of 100k words).
• fuzzy chinese pinyin: Allows fuzzy searching by pinyin.
• text generator: Integrates GPT in Obsidian for text generation.

Appearance

• Hider: Can hide elements in Obsidian; anything visually displeasing can be hidden. I prefer a minimalist form, so I’ve hidden many elements.
• File Tree Alternative Plugin: Enhanced version of the directory structure, allows side-by-side display.
• Novel word count: Displays the total word count under the current directory in Obsidian.
• front matter title: Allows displaying the title item from notes instead of file names in Obsidian; the articles I migrated from my blog are all timestamp-named, so this tool provides a clear view of article names.
• hover editor: Previews other notes in a hovering window.
• homepage: Supports automatically opening a specific page when the software is opened.

I use the Things theme, which is quite minimalist and suits my taste.

Usage Optimization

Date

I hope that when opening Obsidian, the homepage shows some important information, such as the current date, the lunar calendar, and the week’s, month’s, and year’s progress. After searching, I found there weren’t any existing solutions, so I wrote one using JS. The effect is as shown in the content at the top (mobile users need to install the Natural Language Dates plugin).

Hello, today is Thursday, September 7, 2023, the 23rd day of the 7th lunar month. There are 3 days left in this week, 23 days left in September, and 115 days remaining in 2023.

Using dataviewjs to embed JS code:

await dv.view("_obsidian/_data/_scripts/countdownView")

The actual script code is:

CountdownView();

// Main function to create the view
async function CountdownView(){
    var now_time = new Date();
    var today_time = now_time.toLocaleString("en-US", {timeZone: "Asia/Shanghai"});
    var day_number = moment(today_time).format('DD');
    var month_number = moment(today_time).format('MMMM');
    var year_number = moment(today_time).format('YYYY');
    
    var formattedDate = moment(today_time).format('YYYY年MM月DD日dddd');
    dv.el('span','Hello，今天是<span class="ob-special-font-style">'+formattedDate+'</span>，')

    var today_local_month = now_time.toLocaleString('zh-u-ca-chinese', {month: "numeric"})
    var today_local_day = now_time.toLocaleString('zh-u-ca-chinese', {day: "numeric"})
    var today_local_day_num = Number(today_local_day.split('日')[0])

    dv.el('span','农历<span class="ob-special-font-style">' + today_local_month + (today_local_day_num > 10 ? "" : "初") + today_local_day + '</span>。')
    
    const nldates = app.plugins.getPlugin("nldates-obsidian")
    var weekDiff,monthDiff,yearDiff;
    if(nldates)
    {
        const today = nldates.parseDate("today").moment
        const week =  nldates.parseDate("next week").moment
        const month = nldates.parseDate("next month").moment
        const year = nldates.parseDate("next year").moment
        weekDiff = week.diff(today,'days')
        monthDiff = month.diff(today,'days')
        const daysInMonth = today.daysInMonth()
        yearDiff = year.diff(today,'days')
    }
    // momentjs
    const NativeWeekDiff = Math.floor(moment(moment().startOf('week').add(1, "week")).diff(moment(), 'days'))
    const NativeMonthDiff = moment(moment(moment().format("YYYY,MM")).add(1, "month").format("YYYY,MM[,1]")).diff(moment(), "days")
    const NativeYearDiff = moment(moment(moment().format("YYYY")).add(1, "year").format("YYYY[,1,1]")).diff(moment(), "days")
    if(NativeWeekDiff && NativeWeekDiff != NaN){ weekDiff = NativeWeekDiff; }
    if(NativeMonthDiff && NativeMonthDiff != NaN){ monthDiff = NativeMonthDiff; }
    if(NativeYearDiff && NativeYearDiff != NaN){ yearDiff = NativeYearDiff; }

    dv.el('span','本周还剩<span class="ob-special-font-style">'+weekDiff+'</span>天，')
    if(monthDiff==0){
        dv.el('span','也是<span class="ob-special-font-style">'+month_number+'</span>的最后一天，')
    }else{
        dv.el('span',month_number+'还剩<span class="ob-special-font-style">'+monthDiff+'</span>天，')
    }
    dv.el('spwn',year_number+'年还剩<span class="ob-special-font-style">'+yearDiff+'</span>天。')
}

The CSS used (changing the font color of the date to blue; if not used, the ordinary text color will remain the same):

.ob-special-font-style {
  color: #4C8CE6;
  font-size: 1em;
  margin: 0 0.2em 0 0.2em;
}

Weather

I also hope that it can display the recent two days’ weather upon opening. I found a community-based implementation using the HeWeather API; it meets my needs. I made some modifications, and the effect is as follows:

The dataviewjs code is:

let setting = {};

// API key created in HeWeather
setting.key = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx";
setting.city = "Shenzhen"; // City name
setting.days = 2; // Weather forecast days
setting.headerLevel = 0; // Level of title added
setting.addDesc = false; // Whether to add description
setting.onlyToday = false; // Whether to only display for today
setting.anotherCity = ""; // Add another city

// Path where the script file weatherView.js is located
await dv.view("_obsidian/_data/_scripts/weatherView",setting)

dv.view calls the weatherView JS code: weatherView.js

Vault Overview

I also hope to be able to output real-time statistics of all content in the current Obsidian vault, such as:

A total of 383 documents created, 248 tags, 5 boards, 155 to-dos, 30 diaries. 180 blog articles, 16 drafts.

This can also be achieved using dataviewjs; the code is:

let nofold = '!"_obsidian/_data/_Template"'
let allFile = dv.pages(nofold).file
let totalMd = "共创建 "+ allFile.length+" 篇文档"
let totalTag = allFile.etags.distinct().length+" 个标签"
let totalTask = allFile.tasks.length+" 个待办"

let kanbanFile = dv.pages('#看板 and !"_obsidian/_data"').file
let kanbanMddiarys = kanbanFile.length + " 个[[Homepage#看板|看板]]"

let diary_rule = '"_obsidian/_diary"'
let diaryFile = dv.pages(diary_rule).file
let totalMddiarys = diaryFile.length + " 篇[[ObHome#日记|日记]]"

let posts_rule = '"_posts"'
let postsFile = dv.pages(posts_rule).file
let draft_rule = '"_draft"'
let draftFile = dv.pages(draft_rule).file
let totalMdArticles = postsFile.length - 1 + " 篇[[BlogHome#已发布|博客文章]]，" + draftFile.length+" 篇[[BlogHome#草稿|草稿]]。"

dv.paragraph(
	totalMd+"、"+totalTag+"、"+ kanbanMddiarys + "，" +totalTask +"、" + totalMddiarys + "。" + totalMdArticles
)

Content Filtering

Lastly, listing all non-article pages in my blog and allowing navigation in the browser:

Implementation code (dataview):

table WITHOUT ID link(file.link,file.link.title) AS "笔记",elink("https://imzlp.com/"+file.folder,substring(file.folder,6)) AS 链接,file.link.description as "描述"
from "notes" and !"notes/index"
sort file.size desc
limit 100

Using this method allows traversing all content in Obsidian with code and accessing its metadata data for filtering.

Plugin Development and Optimization

Obsidian provides a very powerful plugin mechanism. The previously listed plugins that I often use can be customized or new ones developed if I’m not completely satisfied with certain plugins.

• Development Documentation: Build a plugin
• Plugin Development Template: obsidianmd/obsidian-sample-plugin

Obsidian’s plugin development uses TypeScript and requires compilation to output usable plugins. If you are unsatisfied with a plugin, you can find its GitHub repository, modify the TypeScript code, and generate a new plugin.

Taking the file-tree-alternative plugin as an example: the default version can only display file names, but the file names of my Hexo articles are timestamp-based, making them less intuitive. Therefore, I modified the default version based on the original repository so that it can read the title item from articles for display. For more details see: imzlp/ob-file-tree-alternative. Download the file from the release and replace the main.js of the plugin.

Comparison images of the official version and my modified version:

Isn’t it much more intuitive now? This is one of the advantages of an open plugin ecosystem.

Changes Brought by Tools

Switching to Obsidian has made my knowledge management significantly less cumbersome. It’s not merely because of the numerous unique features of Obsidian; rather, it brings together all the tools I frequently use and customizes them according to my needs.

1. It can sync across all platforms.
2. It allows content retrieval more conveniently (file management, TAG, search, etc.).
3. I can directly use more complex tools (boards, draw.io, excalidraw, Canvas, etc.).
4. Task management has become more precise; in the Calendar, I can click dates to create diaries and add scheduled events, allowing me to clearly see what needs to be done upon opening Obsidian.
5. Blog management has become easier; in Obsidian, I can quickly navigate to a specific blog page and create an empty article format file based on a template. Moving it to the _posts directory enables publication.
6. It simplifies operational steps, channeling focus back to content itself, making me more eager to write.

My Obsidian main page:

Conclusion

What is the most important thing in knowledge management? Content! Content! Still content! Tools that do not produce content are meaningless to tinker with.

To do a good job, one must first sharpen their tools. The goal of sharpening tools is to do things better and faster, not just to make the tools themselves good, which would be missing the point.

Obsidian indeed allows me to write content conveniently and works well. However, I believe that the more important aspect is the attitude towards content: to write seriously, share actively, and unleash its maximum value — that is the essence of “knowledge management.”