Hypothes.is + Obsidian = Hypothesidian for easier note taking and formatting

Anyone who knows me knows that I love Hypothes.is for all my online highlighting, annotating, and general note taking. They also know that if one isn’t actively using their notes to some better end, then it’s likely not worth having taken them at all, so I store mine in markdown in Obsidian for future-proofing and portability.

Hypothes.is + Obsidian

A while back I came across RoamHacker’s work to dovetail Hypothes.is for use in Obsidian and finally managed to get it up and running with my Obsidian vault. I’ve previously outlined a method for pulling in my notes from Hypothes.is using RSS, however this doesn’t give one any formatting capabilities and it also doesn’t provide any of the Hypothes.is tags as RSS has no layer for taxonomies.

RoamHacker’s work, which leverages the Templater Plugin for Obsidian, fixes both of these problems. I suspect that I’ll keep my prior method in place to create the individual notes, but use this additional work to clean up my fleeting notes from Hypothes.is in my actual commonplace book. Since there’s no server involved, it’s harder to automate the entire process so that every time you create notes they’re automatically ported across either in real-time or in batches every few hours.

Formatting your notes

I did spend some time last night to modify some of RoamHacker’s code to re-format the annotations to better suit my current notes format and layout. I’m excerpting the most relevant part below, but the entire Gist can also be downloaded or further modified for easier copy/pasting into one’s own vault for the needed set up.

I’ve only modified the section of the original Gist at the bottom that follows the line:
/* TEMPLATE STARTS HERE */

The changes still keep all the relevant data fields, but reorder them and add a bit of formatting to fit the layout and the way I use my Obsidian notebook. I changed the formatting so that tags in Hypothes.is are turned into [[wikilinks]] rather than #⁠hashtags as in the original. (The original also doesn’t do so well with multi-word tags, which I use quite a lot.)

Hopefully the small changes I’ve made and comparison with the original Gist will allow those who aren’t as code-savvy to better understand the template and potentially let them make changes to suit their own needs.

/* TEMPLATE STARTS HERE */ 
if (tp.file.content.length==0) {
  //likely a new document, insert front matter
  tR += `---\n`;
  tR += `fileType: HypothesisAnnotations\n`;
  tR += `creationDate: ${tp.date.now('YYYY-MM-DD')} \n`;
  tR += `annotationDate: ${articleAnnotations[0].created.substring(0,10)}\n`;
  tR += `uri: ${articleAnnotations[0].uri}\n`;
  tR += `---\n`;
}

tR += `# ${articleAnnotations[0].title}\n`
tR += `URL: ${articleAnnotations[0].uri}\n\n`

for( a of articleAnnotations) {
  let tags = '';
  let user = '';
  if(a.tags.length>0) tags = ' ' + (a.tags.map(t=> '[['+ t + ']]')).join(' ');
  if(insertUser) user = ' _(' + a.user.replace('acct:','').replace('@hypothes.is','') + ')_';
  if(a.text) tR += `${a.text}\n—[[${user}]]\n\n`;
  tR += `## Source \n`;
  tR += `> ${a.highlight}[^1]\n\n`;
  tR += `[^1]: [${articleAnnotations[0].title}](${articleAnnotations[0].uri}) | [syndication link](tk) \n`;
  tR += `\n---\ntags: \nlinks: ${tags} \n- broader terms (BT):  \n- narrower terms (NT):  \n- related terms (RT):  \n- used for (UF) or aliases:  \nconnected ideas:  \nMOC:  \n\n---\n`;
}
%>

Creating Internal Backlinks for MediaWiki for Digital Gardeners

I’d spit-balled the general idea of showing backlinks or bidirectional links on a MediaWiki instance last year when thinking about and adding Webmention to one. Tonight I tinkered around and actually set it up on an instance. Within a MediaWiki, one can transclude all the backlinks from other pages to a particular page by adding a line for transcluding content like the following when editing a page:

{{Special:WhatLinksHere/PageName|limit=1000}}

You can see a live example of the practice on my user page on the IndieWeb wiki at https://indieweb.org/User:Boffosocko.com#Backlinks along with the code I used by clicking on the edit tab. The effect is rather nice, particularly when put into columns when there are lots of entries. I’ll have to look into automatically coding something like this into every page now, but being able to do it manually is most of the battle, right?

Doing this along with adding display for external webmentions quickly vaults MediaWiki to a solidly first class web-enabled digital commonplace book/digital garden/Memex/zettelkasten tool that can communicate with other similarly enabled tools. (Now if only Webmention were supported natively on MediaWiki… but there are definitely ways around this in the meanwhile.)

To go the extra mile, I know there’s the ability to interlink wikis with some custom syntax or even to show hovercards within a wiki. Both MediaWiki and Wikipedia already allow this after enabling page previews using hovercards in 2018. (I’ll have to check out if one could do hovercards across wikis as well!?!)

I’ve slowed down some of my experiments with my personal MediaWiki in preference to using Obsidian lately. Perhaps, for working in public, I’m going to have to resume some of my experiments and/or figure out a way to mirror the content?

Sister Heather Kristine in knowledge-management discord for Obsidian () for re-sparking idea.

Replied to Bird sound encoding by Douglas HoffDouglas Hoff (Art of Memory Forum)
Rey’s star book (already ordered!) is a wonderful way to rekindle my adolescent interest in the stars while learning more about memory methods like pareidolia to finally complete my identification of the skies. I never stored more than three or four constellations permanently. @chrisaldrich, I’ll be intested to see how you help bring together your knowledge to create a more mnemonic way to visualize and remember bird calls and traits. I’ve also added your blog to my news reader with all the goo...
I’ll apologize in advance for the noisy-ness of my website. I use it as a commonplace book and post almost everything I do on the web there first (including social media). If it gets to be too much, you can subscribe to individual topics of interest (like https://boffosocko.com/category/Memory/feed/, which is sure to include any bird related work) so that you’re just getting what you want instead of the overzealous firehose which can be upwards of 10 to 20,000 posts a year, depending on how much of my stream I make public.
Read Import from TiddlyWiki 5 to Obsidian (Obsidian Forum)
How to Export from TiddlyWiki to Obsidian Note: I am on Mac, so all the steps are specially for that OS. I followed the steps by @kepano and just tweaked/clarified the things that didn’t work for me initially. So this is very much his work. @pikacho, I hope this is helpful for you and others. Step 1: Set up TiddlyWiki in Node.js Download and install Node.js Open “Terminal”, run npm install -g tiddlywiki If an error occurs, run sudo chown -R $USER /usr/local/lib/node_modules (that gives you a...

Non-technical IndieWeb: Fun, Creativity, Community, and “Content”

I resemble that remark.

rakhim
–Credit: Rakhim

Um…

Er… I mean…

I resent that remark. 😉 

The point of having a website is putting something interesting on it right?

The IndieWeb wiki does tend toward the technical, but many of us are working toward remedying that. For those who haven’t found them yet, there are some pages around a variety of topics like poetry, crafts, hobbies, music, writing, journalism, education, and a variety of other businesses and use cases. How we don’t have one on art (yet) is beyond me… Hopefully these might help us begin to use our sites instead of incessantly building them, though this can be a happy hobby if you enjoy it.

If you’ve got an IndieWeb friendly site, why not use it to interact with others? Help aggregate people around other things in which you’re interested. One might interact with the micro.blog community around any of their tagmoji. (I’m personally hoping there will be one for the stationery, pen, and typewriter crowd.) One might also find some community on any of the various stubs (or by creating new stubs) on IndieWeb.xyz

For more practical advice and to borrow a proverbial page from the movie Finding Forrester, perhaps reading others’ words and borrowing or replying to them may also help you along. I find that starting and ending everything from my own website means that I’m never at a loss for content to consume or create. Just start a conversation, even if it’s just with yourself. This started out as a short reply, but grew into a longer post aggregating various ideas I’ve had banging around my head this month.

Rachel Syme recently made me think about “old school blogs”, and as interesting as her question was, I would recommend against getting stuck in that framing which can be a trap that limits your creativity. It’s your site, do what you want with it. Don’t make it a single topic. That will make it feel like work to use it.

The ever-wise Charlie Owen reminds of this and suggests a solution for others reading our content. 

Of course if building websites is your passion and you want to make a new one on a new platform every week, that’s cool too. Perhaps you could document the continuing refreshing of the process each time and that could be your content?

Of course if this isn’t enough, I’ll also recommend Matthias Ott‘s advice to Make it Personal. And for those with a more technical bent, Simon Collison has a recent and interesting take on how we might be a bit more creative with our technical skills in This Used to be Our Playground.

In any case, good luck and remember to have some fun!

A note taking problem and a proposed solution

tl;dr

It’s too painful to quickly get frequent notes into note taking and related platforms. Hypothes.is has an open API and a great UI that can be leveraged to simplify note taking processes.

Note taking tools

I’ve been keeping notes in systems like OneNote and Evernote for ages, but for my memory-related research and work in combination with my commonplace book for the last year, I’ve been alternately using TiddlyWiki (with TiddlyBlink) and WordPress (it’s way more than a blog.)

I’ve also dabbled significantly enough with related systems like Roam Research, Obsidian, Org mode/Org Roam, MediaWiki, DocuWiki, and many others to know what I’m looking for.

Many of these, particularly those that can be used alternately as commonplace books and zettelkasten appeal to me greatly when they include the idea of backlinks. (I’ve been using Webmention to leverage that functionality in WordPress settings, and MediaWiki gives it grudgingly with the “what links to this page” basic functionality that can be leveraged into better transclusion if necessary.)

The major problem with most note taking tools

The final remaining problem I’ve found with almost all of these platforms is being able to quickly and easily get data into them so that I can work with or manipulate it. For me the worst part of note taking is the actual taking of notes. Once I’ve got them, I can do some generally useful things with them—it’s literally the physical method of getting data from a web page, book, or other platform into the actual digital notebook that is the most painful, mindless, and useless thing for me.

Evernote and OneNote

Older note taking services like Evernote and OneNote come with browser bookmarklets or mobile share functionality that make taking notes and extracting data from web sources simple and straightforward. Then once the data is in your notebook you can actually do some work with it. Sadly neither of these services has the backlinking functionality that I find has become de rigueur for my note taking or knowledge wrangling needs.

WordPress

My WordPress solutions are pretty well set since that workflow is entirely web-based and because WordPress has both bookmarklet and Micropub support. There I’m primarily using a variety of feeds and services to format data into a usable form that I can use to ping my Micropub endpoint. The Micropub plugin handles the post and most of the meta data I care about.

It would be great if other web services had support for Micropub this way too, as I could see some massive benefits to MediaWiki, Roam Research, and TiddlyWiki if they had this sort of support. The idea of Micropub has such great potential for great user interfaces. I could also see many of these services modifying projects like Omnibear to extend themselves to create highlighting (quoting) and annotating functionality with a browser extension.

With this said, I’m finding that the user interface piece that I’m missing for almost all of these note taking tools is raw data collection.

I’m not the sort of person whose learning style (or memory) is benefited by writing or typing out notes into my notebooks. I’d far rather just have it magically happen. Even copying and pasting data from a web browser into my digital notebook is a painful and annoying process, especially when you’re reading and collecting/curating as many notes as I tend to. I’d rather be able to highlight, type some thoughts and have it appear in my notebook. This would prevent the flow of my reading, thinking, and short annotations from being subverted by the note collection process.

Different modalities for content consumption and note taking 

Based on my general experience there are only a handful of different spaces where I’m typically making notes.

Reading online

A large portion of my reading these days is done in online settings. From newspapers, magazines, journal articles and more, I’m usually reading them online and taking notes from them there.

.pdf texts

Some texts I want to read (often books and journal articles) only live in .pdf form. While reading them in an app-specific setting has previously been my preference, I’ve taken to reading them from within browsers. I’ll explain why in just a moment, but it has to do with a tool that treats this method the same as the general online modality. I’ll note that most of the .pdf  specific apps have dreadful data export—if any.

Reading e-books (Kindle, e-readers, etc.)

If it’s not online or in .pdf format, I’m usually reading books within a Kindle or other e-reading device. These are usually fairly easy to add highlights, annotations, and notes to. While there are some paid apps that can extract these notes, I don’t find it too difficult to find the raw file and cut and paste the data into my notebook of choice. Once there, going through my notes, reformatting them (if necessary), tagging them and expanding on them is not only relatively straightforward, but it also serves as a simple method for doing a first pass of spaced repetition and review for better long term recall.

Lectures

Naturally taking notes from live lectures, audiobooks, and other spoken events occurs, but more often in these cases, I’m typically able to type them directly into my notebook of preference or I’m using something like my digital Livescribe pen for notes which get converted by OCR and are easy enough to convert in bulk into a digital notebook. I won’t belabor this part further, though if others have quick methods, I’d love to hear them.

Physical books

While I love a physical book 10x more than the next 100 people, I’ve been trying to stay away from them because I find that though they’re easy to highlight, underline, and annotate the margins, it takes too much time and effort (generally useless for memory purposes for me) to transfer these notes into a digital notebook setting. And after all, it’s the time saving piece I’m after here, so my preference is to read in some digital format if at all possible.

A potential solution for most of these modalities

For several years now, I’ve been enamored of the online Hypothes.is annotation tool. It’s open source, allows me reasonable access to my data from the (free) hosted version, and has a simple, beautiful, and fast process for bookmarking, highlighting, and annotating online texts on desktop and mobile. It works exceptionally well for both web pages and when reading .pdf texts within a browser window.

I’ve used it daily to make several thousand annotations on 800+ online web pages and documents. I’m not sure how I managed without it before. It’s the note taking tool I wished I’d always had. It’s a fun and welcome part of my daily life. It does exactly what I want it to and generally stays out of the way otherwise. I love it and recommend it unreservedly. It’s helped me to think more deeply and interact more directly with countless texts.

When reading on desktop or mobile platforms, it’s very simple to tap a browser extension and have all their functionality immediately available. I can quickly highlight a section of a text and their UI pops open to allow me to annotate, tag it, and publish. I feel like it’s even faster than posting something to Twitter. It is fantastically elegant.

The one problem I have with it is that while it’s great for collecting and aggregating my note data into my Hypothes.is account, there’s not much I can do with it once it’s there. It’s missing the notebook functionality some of these other services provide. I wish I could plug all my annotation and highlight content into spaced repetition systems or move it around and modify it within a notebook where it might be more interactive and cross linked for the long term. Sadly I don’t think that any of this sort of functionality is on Hypothes.is’ roadmap any time soon.

There is some great news however! Hypothes.is is open source and has a reasonable API. This portends some exciting things! This means that any of these wiki, zettelkasten, note taking, or spaced repetition services could leverage the UI for collecting data and pipe it into their interfaces for direct use.

As an example, what if I could quickly tell Obsidian to import all my pre-existing and future Hypothes.is data directly into my Obsidian vault for manipulating as notes? (And wouldn’t you know, the small atomic notes I get by highlighting and annotating are just the sort that one would like in a zettelkasten!) What if I could pick and choose specific course-related data from my reading and note taking in Hypothes.is (perhaps by tag or group) for import into Anki to quickly create some flash cards for spaced repetition review? For me, this combination would be my dream application!

These small pieces, loosely joined can provide some awesome opportunities for knowledge workers, students, researchers, and others. The education focused direction that Hypothes.is, many of these note taking platforms, and spaced repetition systems are all facing positions them to make a super-product that we all want and need.

An experiment

So today, as a somewhat limited experiment, I played around with my Hypothes.is atom feed (https://hypothes.is/stream.atom?user=chrisaldrich, because you know you want to subscribe to this) and piped it into IFTTT. Each post creates a new document in a OneDrive file which I can convert to a markdown .md file that can be picked up by my Obsidian client. While I can’t easily get the tags the way I’d like (because they’re not included in the feed) and the formatting is incredibly close, but not quite there, the result is actually quite nice.

Since I can “drop” all my new notes into a particular folder, I can easily process them all at a later date/time if necessary. In fact, I find that the fact that I might want to revisit all my notes to do quick tweaks or adding links or additional thoughts provides the added benefit of a first round of spaced repetition for the notes I took.

Some notes may end up being deleted or reshuffled, but one thing is clear: I’ve never been able to so simply highlight, annotate, and take notes on documents online and get them into my notebook so quickly. And when I want to do something with them, there they are, already sitting in my notebook for manipulation, cross-linking, spaced repetition, and review.

So if the developers of any of these platforms are paying attention, I (and I’m sure others) really can’t wait for plugin integrations using the full power of the Hypothes.is API that allow us to all leverage Hypothes.is’ user interface to make our workflows seamlessly simple.

Listened to Episode 5: Caring What You're Sharing by Dr Laurie SantosDr Laurie Santos from The Happiness Lab

Sharing a good experience with another human deepens our enjoyment of the moment... but only if we abide by certain rules. Dr Laurie Santos shows us how we often get 'sharing' wrong and explains how we can all derive more happiness from ice cream, sunsets and a night in front of the TV.

Maryellis Bunn’s website

Erica Boothby website

Museum of Ice Cream website

Alix Barash website

There is some interesting discussion about exploring and interacting with the world here both with and without a camera and/or digital phone or other device in one’s hand. 

The research and examples in this episode could be useful  for UX/UI  designers in the social media and IndieWeb spaces. The ideas presented here could help us in designing interactions on the web for people in a much happier and healthier fashion. I particularly likes the concept that a museum specifically redesigned some of it’s exhibits so as to be able to minimize the use of phones and increase the human-to-human interaction.

The questions of whether we’re posting content for ourselves or to share with others is an intriguing one. I tend to post for myself (and my memory via my commonplace book) first in almost all cases. When I’m taking photos or checking in, I almost always do it in a way so as to minimize as much as possible the distraction of doing so to others. It’s exceptionally rare that I spend the time and effort to get the “perfect” photo when I’m with others in public.

The discussion about the museum experience being designed for or against photography and the research relating to memories of the experiences reminds a lot of Matt Maldre’s recent experience with a museum security guard who urged patrons to get their phones out and take close up photos of artworks. [#] She obviously intuitively knew something that the rest of us could have only guessed at. Or perhaps she’s just been reading all the most cutting-edge research and putting it into practice in her own work?

This also reminds me I ought to call Dan Cohen and have a conversation about these sort of design concepts (and particularly those relating to Frances Yates and memory techniques) for his forthcoming library.

Replied to Do you keep a diary? Or a life log? by Matt Maldre (Spudart)
Do you keep a daily diary? I have a couple questions for you: Length: Do you keep it really short? Or do you write longer form? Medium: Where do you write it? In a physical notebook? Or maybe online in a blog? Maybe some sort of software, or Google Drive. The short diary in a …
I keep the sort of diary/commonplace book you’re talking about. Generally it lives in two places. The biggest portion lives on my website where I can generally quickly bookmark almost everything I read, listen to, watch, annotate, reply to, or deal with online in some fashion. Not all of my posts there are public, but they’re archived there privately for search. 

My secondary backup is on OneNote (I’d used Evernote in the past and I find them roughly similar), where I’ll tend to keep some personal daily to do lists (not too dissimilar from a digital bullet journal) and other private things that are easier to keep there than on my own website.

I like that both OneNote and my website are available on almost all the platforms I regularly use, so they’re always accessible to me.

Annotation posts >> Highlight posts

Because they’re so similar, I’ve decided to discontinue the custom highlight posts my site had in lieu of the more prevalent annotation post kind. The layout and format of both as highlighted text quoted from another site was almost exactly the same with the primary difference being my additional commentary added to the highlighted text to call it an annotation. Conceptually I considered “highlight + commentary/reply = annotation”. The difference is marginal at best–pun intended.

Since I only had 13 highlight posts versus 121 annotation posts (plus various additional annotations and highlights which I’ve rolled up into the body of some of my read posts) over the last year and a half, I felt it seemed redundant and bothersome to maintain two separate, but nearly identical post kinds. Semantically one may think of a highlight on some text as an annotation anyway, thus the idea of annotation subsumes that of a simple highlight.

As of this evening, I’ve changed all the custom highlight posts to be of the annotation kind. Other than the one word visual difference of the post kind text changing from “highlight” to “annotation” this change won’t affect much except for those who may have been subscribed to the highlight feed. Going forward you may consider subscribing to my annotation feed instead.

I had created highlight posts first, but in the end annotation posts have won the day. And for those that don’t have them, fear not, because honestly annotation posts are really just glorified bookmarks with custom text in the context. (The glorification only entails a highligher icon instead of a bookmark icon and a bit of CSS to color the text yellow.) I do find having them delineated for my personal research purposes useful though.

Listened to The Curator's Code by Brooke Gladstone from On the Media | WNYC Studios

One of the greatest assets of the internet is that it leads to great content discoveries that readers might not otherwise be able to find. One of the biggest liabilities is that content is frequently repackaged without crediting its creators or where it was found. Brooke talks to Maria Popova, editor of the website Brain Pickings and one of the creators of the Curator's Code, which seeks to honor the way people discover content online.

hat tip: Martijn van der Ven and Jeremy Cherfas

Originally bookmarked to listen to on November 23, 2019 at 10:38AM

Replied to a tweet by Dr. Ryan StraightDr. Ryan Straight (Twitter)
What a great prompt! Here are a few interesting off-label use cases I’ve used, imagined, or seen in the wild:

Greg McVerry, Ian O’Byrne, and I have integrated Hypothes.is into our digital/online commonplace books in different ways. Greg’s are embedded at https://jgregorymcverry.com/annotations, Ian discusses his process on his site, while mine show up as annotation or highlight posts.

I’ve not published the full idea yet, but I’ve spent some time contemplating using Hypothes.is as a blogging platform/CMS. It might require a bit of flexibility, but it generally has reasonable support for:

  • Writing posts with a reasonably full-featured text editor and the ability to edit and delete posts later;
  • HTML and markdown support;
  • Public and private posting as well as sharing content with other private groups;
  • The ability to reply to other websites;
  • The ability for others to comment on your posts natively;
  • A robust tagging functionality;
  • The ability to socially bookmark web pages (blank page notes);
  • An RSS feed;
  • The ability to share posts to other social platforms including meta data for Twitter cards;
  • Naturally, it’s very easy to use for writing short notes, creating highlights and annotations, and keeping track of what you’ve read;
  • It has a pseudo-social media functionality in that your public posts appear on a global timeline where people can read and interact with them.
  • It’s also opensource, so you can self-host, modify it, or add new features.

I have been personally using Hypothes.is to follow the public feed, several tag feeds, and several friends’ specific feeds as a discovery tool for finding interesting content to read.

And a final off-label use case that could be compelling, but which could have some better UI and integration would be to use Hypothes.is as an embeddable commenting system for one’s own website. It has in-line commenting in much the same way that Medium does, but the entire thing could likely be embedded into a comment section under a traditional blog post and be used in much the same way people use Disqus on blogs. I’ll note that in practice, I find Hypothes.is far faster than Disqus ever was. I’ve yet to see anyone offloading the commenting functionality of their blog this way, but I’d be willing to bet dollars to donuts that someone could hack it together as a simple iframe or via the API pretty quickly and with solid results.

And naturally I’m missing many, potentially including some I’ve thought about before. Maybe worth checking the old Hypothes.is tag in my digital notebook?

If people have others, I’m enamored to hear them.

Checked into Beyond WordPress – easy WP automation and integration with no coding
Sessions after lunch starting a few minutes late.

Sabrina Liao is looking primarily looking at Zapier, Integromat, IFTTT, automate.io.

I use a huge number of automated pieces like these, particularly IFTTT, for driving my own personal online commonplace book.

I’ve been seemingly “away” from my online presence for several months, but felt it’s time to come back to posting publicly a bit more frequently. Some of the hiatus has been set aside for crafting some new workflows for posting more content privately for a slightly different version of my commonplace book. 

While I’ve learned a lot in my experiment, I’ll have to ruminate on it a bit longer before writing any specific takeaways. One of my favorite portions has been some work to pull more data out of the books I’m reading in terms of highlights, notes, and marginalia for research purposes. 

In general, not much has changed here directly, but I will have to get used to the the posting interface and some of my old workflows again. I’ll also need to get around to some general updates and fixes that I’ve let slide for far too long. I’ll also have to catch up on some change logs for things that have improved since my hiatus began.

I will say that while I’ve been very productive during my hiatus, which has included stepping away significantly from other forms of social media consumption as well, I also very much miss interacting with a lot of my online friends and colleagues on a more regular basis. It’ll be good to ease back into my feed reader and see what everyone has been up to for the past several months.