A note taking problem and a proposed solution

tl;dr

It’s too painful to quickly get frequent notes into note taking and related platforms. Hypothes.is has an open API and a great UI that can be leveraged to simplify note taking processes.

Note taking tools

I’ve been keeping notes in systems like OneNote and Evernote for ages, but for my memory-related research and work in combination with my commonplace book for the last year, I’ve been alternately using TiddlyWiki (with TiddlyBlink) and WordPress (it’s way more than a blog.)

I’ve also dabbled significantly enough with related systems like Roam Research, Obsidian, Org mode/Org Roam, MediaWiki, DocuWiki, and many others to know what I’m looking for.

Many of these, particularly those that can be used alternately as commonplace books and zettelkasten appeal to me greatly when they include the idea of backlinks. (I’ve been using Webmention to leverage that functionality in WordPress settings, and MediaWiki gives it grudgingly with the “what links to this page” basic functionality that can be leveraged into better transclusion if necessary.)

The major problem with most note taking tools

The final remaining problem I’ve found with almost all of these platforms is being able to quickly and easily get data into them so that I can work with or manipulate it. For me the worst part of note taking is the actual taking of notes. Once I’ve got them, I can do some generally useful things with them—it’s literally the physical method of getting data from a web page, book, or other platform into the actual digital notebook that is the most painful, mindless, and useless thing for me.

Evernote and OneNote

Older note taking services like Evernote and OneNote come with browser bookmarklets or mobile share functionality that make taking notes and extracting data from web sources simple and straightforward. Then once the data is in your notebook you can actually do some work with it. Sadly neither of these services has the backlinking functionality that I find has become de rigueur for my note taking or knowledge wrangling needs.

WordPress

My WordPress solutions are pretty well set since that workflow is entirely web-based and because WordPress has both bookmarklet and Micropub support. There I’m primarily using a variety of feeds and services to format data into a usable form that I can use to ping my Micropub endpoint. The Micropub plugin handles the post and most of the meta data I care about.

It would be great if other web services had support for Micropub this way too, as I could see some massive benefits to MediaWiki, Roam Research, and TiddlyWiki if they had this sort of support. The idea of Micropub has such great potential for great user interfaces. I could also see many of these services modifying projects like Omnibear to extend themselves to create highlighting (quoting) and annotating functionality with a browser extension.

With this said, I’m finding that the user interface piece that I’m missing for almost all of these note taking tools is raw data collection.

I’m not the sort of person whose learning style (or memory) is benefited by writing or typing out notes into my notebooks. I’d far rather just have it magically happen. Even copying and pasting data from a web browser into my digital notebook is a painful and annoying process, especially when you’re reading and collecting/curating as many notes as I tend to. I’d rather be able to highlight, type some thoughts and have it appear in my notebook. This would prevent the flow of my reading, thinking, and short annotations from being subverted by the note collection process.

Different modalities for content consumption and note taking 

Based on my general experience there are only a handful of different spaces where I’m typically making notes.

Reading online

A large portion of my reading these days is done in online settings. From newspapers, magazines, journal articles and more, I’m usually reading them online and taking notes from them there.

.pdf texts

Some texts I want to read (often books and journal articles) only live in .pdf form. While reading them in an app-specific setting has previously been my preference, I’ve taken to reading them from within browsers. I’ll explain why in just a moment, but it has to do with a tool that treats this method the same as the general online modality. I’ll note that most of the .pdf  specific apps have dreadful data export—if any.

Reading e-books (Kindle, e-readers, etc.)

If it’s not online or in .pdf format, I’m usually reading books within a Kindle or other e-reading device. These are usually fairly easy to add highlights, annotations, and notes to. While there are some paid apps that can extract these notes, I don’t find it too difficult to find the raw file and cut and paste the data into my notebook of choice. Once there, going through my notes, reformatting them (if necessary), tagging them and expanding on them is not only relatively straightforward, but it also serves as a simple method for doing a first pass of spaced repetition and review for better long term recall.

Lectures

Naturally taking notes from live lectures, audiobooks, and other spoken events occurs, but more often in these cases, I’m typically able to type them directly into my notebook of preference or I’m using something like my digital Livescribe pen for notes which get converted by OCR and are easy enough to convert in bulk into a digital notebook. I won’t belabor this part further, though if others have quick methods, I’d love to hear them.

Physical books

While I love a physical book 10x more than the next 100 people, I’ve been trying to stay away from them because I find that though they’re easy to highlight, underline, and annotate the margins, it takes too much time and effort (generally useless for memory purposes for me) to transfer these notes into a digital notebook setting. And after all, it’s the time saving piece I’m after here, so my preference is to read in some digital format if at all possible.

A potential solution for most of these modalities

For several years now, I’ve been enamored of the online Hypothes.is annotation tool. It’s open source, allows me reasonable access to my data from the (free) hosted version, and has a simple, beautiful, and fast process for bookmarking, highlighting, and annotating online texts on desktop and mobile. It works exceptionally well for both web pages and when reading .pdf texts within a browser window.

I’ve used it daily to make several thousand annotations on 800+ online web pages and documents. I’m not sure how I managed without it before. It’s the note taking tool I wished I’d always had. It’s a fun and welcome part of my daily life. It does exactly what I want it to and generally stays out of the way otherwise. I love it and recommend it unreservedly. It’s helped me to think more deeply and interact more directly with countless texts.

When reading on desktop or mobile platforms, it’s very simple to tap a browser extension and have all their functionality immediately available. I can quickly highlight a section of a text and their UI pops open to allow me to annotate, tag it, and publish. I feel like it’s even faster than posting something to Twitter. It is fantastically elegant.

The one problem I have with it is that while it’s great for collecting and aggregating my note data into my Hypothes.is account, there’s not much I can do with it once it’s there. It’s missing the notebook functionality some of these other services provide. I wish I could plug all my annotation and highlight content into spaced repetition systems or move it around and modify it within a notebook where it might be more interactive and cross linked for the long term. Sadly I don’t think that any of this sort of functionality is on Hypothes.is’ roadmap any time soon.

There is some great news however! Hypothes.is is open source and has a reasonable API. This portends some exciting things! This means that any of these wiki, zettelkasten, note taking, or spaced repetition services could leverage the UI for collecting data and pipe it into their interfaces for direct use.

As an example, what if I could quickly tell Obsidian to import all my pre-existing and future Hypothes.is data directly into my Obsidian vault for manipulating as notes? (And wouldn’t you know, the small atomic notes I get by highlighting and annotating are just the sort that one would like in a zettelkasten!) What if I could pick and choose specific course-related data from my reading and note taking in Hypothes.is (perhaps by tag or group) for import into Anki to quickly create some flash cards for spaced repetition review? For me, this combination would be my dream application!

These small pieces, loosely joined can provide some awesome opportunities for knowledge workers, students, researchers, and others. The education focused direction that Hypothes.is, many of these note taking platforms, and spaced repetition systems are all facing positions them to make a super-product that we all want and need.

An experiment

So today, as a somewhat limited experiment, I played around with my Hypothes.is atom feed (https://hypothes.is/stream.atom?user=chrisaldrich, because you know you want to subscribe to this) and piped it into IFTTT. Each post creates a new document in a OneDrive file which I can convert to a markdown .md file that can be picked up by my Obsidian client. While I can’t easily get the tags the way I’d like (because they’re not included in the feed) and the formatting is incredibly close, but not quite there, the result is actually quite nice.

Since I can “drop” all my new notes into a particular folder, I can easily process them all at a later date/time if necessary. In fact, I find that the fact that I might want to revisit all my notes to do quick tweaks or adding links or additional thoughts provides the added benefit of a first round of spaced repetition for the notes I took.

Some notes may end up being deleted or reshuffled, but one thing is clear: I’ve never been able to so simply highlight, annotate, and take notes on documents online and get them into my notebook so quickly. And when I want to do something with them, there they are, already sitting in my notebook for manipulation, cross-linking, spaced repetition, and review.

So if the developers of any of these platforms are paying attention, I (and I’m sure others) really can’t wait for plugin integrations using the full power of the Hypothes.is API that allow us to all leverage Hypothes.is’ user interface to make our workflows seamlessly simple.

Published by

Chris Aldrich

I'm a biomedical and electrical engineer with interests in information theory, complexity, evolution, genetics, signal processing, IndieWeb, theoretical mathematics, and big history. I'm also a talent manager-producer-publisher in the entertainment industry with expertise in representation, distribution, finance, production, content delivery, and new media.

24 thoughts on “A note taking problem and a proposed solution”

    1. Mostly I really have one main account (my own website) and I syndicate out things from there. Here’s a longer post with more detail: https://boffosocko.com/2016/10/17/how-many-social-media-related-accounts-can-one-person-have-on-the-web/

      I suspect some of the ones you originally linked to on microblog were actually “shadow” accounts that the service made for different people to subscribe to on their platform, but it’s still an apt question. I’m pretty sure that the @boffosocko.com is one of those, and I don’t have any control over what appears there or why to my knowledge. The mastodon one is just a mirror of my sites content which appears on Mastodon and that’s a common pattern as MB allows users to subscribe to/follow sites outside of it on their platform.

      Here’s a fun one: “In 2021, the average 18 to 34-year-old runs 8.4 profiles on social media.”

      Sorry I missed your comment before, but better late than never… Hopefully this manages to find you where you are now.

      Reply

  8. Vorig weekend deelde ik deze bookmark naar een artikel van Chris Aldrich. Ik herken zijn reflecties op web-based notities. Veel apps en services zijn prima ingesteld om in de app zelf te kunnen organiseren, zoeken en linken. Maar het zit vaak in de manier van annoteren en verzamelen waar de frictie zit.
     

    the user interface piece that I’m missing for almost all of these note taking tools is raw data collection.
    @chrisaldrich https://boffosocko.com/?p=55775767

    Herkenbaar. Op basis van Chris’ eerdere ervaringen met Hypothes.is en zijn eigen reflecties in het artikel heb ik Hypothesis de afgelopen dagen zo nu en dan gebruikt. Deze open source dienst werkt behoorlijk aardig op de desktop en redelijk OK op mobiel. Ik kan snel annotaties maken op webpagina’s en deze zijn goed terug te vinden op de site zelf.

    Ondanks dat het wéér een extra tool is, het biedt me wel een interessante optie. Ik kan specifieke annotaties doorplaatsen naar mijn site. Zoals destijds del.icio.us had, Pinboard dit heeft en vast nog andere diensten, heeft Hypothesis een goede RSS-dienst. Ik kan met elke combinatie van gebruikersnaam en specifieke tag een RSS feed maken. Die kan ik weer in Integromat laden en zo een blogpost maken van annotaties die ik door wil plaatsen naar mijn site.

    Een zijpad: Ja, bovenstaande link naar de tag-pagina met blogposts heeft net zo goed weer een RSS-feed. Zo houdt het systeem zichzelf heerlijk in stand en kun je een oneindige loop creëeren van feeds en posts. Wat is het internet toch fantastisch.
    Deze annotaties zijn eigenlijk een extra manier om bookmarks op te slaan op deze site. Iets wat ik al langer doe via het Micropub protocol met bijbehorende plugin. Zit er een verschil in annotaties en bookmarks? Misschien wel voor dit blog. Tegelijk kan ik de annotaties van Hypothesis ook weer lokaal opslaan in Obsidian om zo allerlei losse flodders die ik online vind aan elkaar te knopen. De ruwe data collectie krijgt zo een extra component. Een manier om mijn eigen Frankopedia, mijn Wunderkammer, weer verder te bouwen.
    Je kunt trouwens mijn stroom aan annotaties in je eigen feedreader laden. Ik zou er zelfs een nieuwsbrief van kunnen maken. Wat is het open web toch mooi, al die losse blokken die je zelf in elkaar kunt klikken.
    <!–

    –>

    Reply

  10. I had MarkDownload installed in Chrome before and had used it a bit. Spent some time tonight and did some custom configuration in the settings to make it easier to save web pages directly to my Obsidian commonplace book. This can be an interesting method for quickly saving and linking lots of data, though it also means saving a lot more than may be strictly necessary, particularly at scale. I’m still not giving up on my Hypothes.is methods though.
    Once configured for Obsidian, MarkDownload can be a very powerful tool.

    Syndicated copies:

    Reply

  14. One thing expected from the note-taking tools, makes me particularly skeptical: their collaborative/ public use. I think the lifecycle of notes cannot be continuous from capturing to communication, unless I forgo the possibility of cryptic, sloppy, abbreviated shorthand meant just for the “me later” that Magdalena Böttger depicted so aptly in 2005. ❧

    Some of the value of notes being done and readable in public means that one typically puts a bit more effort into them at the start. This can make them much more useful and valuable later on. It also means that they usually have more substance and context for use by others in collaboration or other reuses.
    Short notes are often called fleeting notes which may or may not be processed into something more substantive. The ones that do become more substantive can more easily be reused in other future settings.
    Sonke Ahrens’ book How to Take Smart Notes is one of the better arguments for the why and how of note taking.
    Annotated on July 06, 2021 at 10:24AM

    Note that such careful treatment applies only to a certain kind of my notes. While many project-related notes go straight to simple folders of the operating system, the notes that don’t fit in one of the folders, deserve special attention. I don’t know yet where I might deploy them — possibly in multiple places. ❧

    I like that you’ve got such a fascinating system. It’s very similar in form and substance to one that I use, but which relies on a wholly different technology stack: https://boffosocko.com/2020/08/29/a-note-taking-problem-and-a-proposed-solution/
    Annotated on July 06, 2021 at 10:25AM

    Which makes them similar to “commonplace”: reusable in many places. But this connotation has led to a pejorative flavor of the German translation “Gemeinplatz” which means platitude. That’s why I prefer to call them ‘evergreen’ notes, although I am not sure if I am using this differentiation correctly. ❧

    I’ve only run across the German “Gemeinplatz” a few times with this translation attached. Sad to think that this negative connotation has apparently taken hold. Even in English the word commonplace can have a somewhat negative connotation as well meaning “everyday, ordinary, unexceptional” when the point of commonplacing notes is specifically because they are surprising or extraordinary by definition.
    Your phrasing of “evergreen notes” seems close enough. I’ve seen some who might call the shorter notes you’re making either “seedlings” or “budding” notes. Some may wait for bigger expansions of their ideas into 500-2000 word essays before they consider them “evergreen” notes. (Compare: https://maggieappleton.com/garden-history and https://notes.andymatuschak.org/Evergreen_notes). Of course this does vary quite a bit from person to person in my experience, so your phrasing certainly fits.
    I’ve not seen it crop up in the digital gardens or zettelkasten circles specifically but the word “evergreen” is used in the journalism space to describe a fully formed article that can be re-used wholesale on a recurring basis. Usually they’re related to recurring festivals, holidays, or cyclical stories like “How to cook the perfect Turkey” which might get recycled a week before Thanksgiving every year.
    Annotated on July 06, 2021 at 10:37AM

    Syndicated copies:

    Reply

  15. Anyone who knows me knows that I love Hypothes.is for all my online highlighting, annotating, and general note taking. They also know that if one isn’t actively using their notes to some better end, then it’s likely not worth having taken them at all, so I store mine in markdown in Obsidian for future-proofing and portability.
    Hypothes.is + Obsidian
    A while back I came across RoamHacker’s work to dovetail Hypothes.is for use in Obsidian and finally managed to get it up and running with my Obsidian vault. I’ve previously outlined a method for pulling in my notes from Hypothes.is using RSS, however this doesn’t give one any formatting capabilities and it also doesn’t provide any of the Hypothes.is tags as RSS has no layer for taxonomies.
    RoamHacker’s work, which leverages the Templater Plugin for Obsidian, fixes both of these problems. I suspect that I’ll keep my prior method in place to create the individual notes, but use this additional work to clean up my fleeting notes from Hypothes.is in my actual commonplace book. Since there’s no server involved, it’s harder to automate the entire process so that every time you create notes they’re automatically ported across either in real-time or in batches every few hours.
    Formatting your notes
    I did spend some time last night to modify some of RoamHacker’s code to re-format the annotations to better suit my current notes format and layout. I’m excerpting the most relevant part below, but the entire Gist can also be downloaded or further modified for easier copy/pasting into one’s own vault for the needed set up.
    I’ve only modified the section of the original Gist at the bottom that follows the line:
    /* TEMPLATE STARTS HERE */
    The changes still keep all the relevant data fields, but reorder them and add a bit of formatting to fit the layout and the way I use my Obsidian notebook. I changed the formatting so that tags in Hypothes.is are turned into [[wikilinks]] rather than #⁠hashtags as in the original. (The original also doesn’t do so well with multi-word tags, which I use quite a lot.)
    Hopefully the small changes I’ve made and comparison with the original Gist will allow those who aren’t as code-savvy to better understand the template and potentially let them make changes to suit their own needs.
    /* TEMPLATE STARTS HERE */
    if (tp.file.content.length==0) {
    //likely a new document, insert front matter
    tR += ---n;
    tR += fileType: HypothesisAnnotationsn;
    tR += creationDate: ${tp.date.now('YYYY-MM-DD')} n;
    tR += annotationDate: ${articleAnnotations[0].created.substring(0,10)}n;
    tR += uri: ${articleAnnotations[0].uri}n;
    tR += ---n;
    }

    tR += # ${articleAnnotations[0].title}n
    tR += URL: ${articleAnnotations[0].uri}nn

    for( a of articleAnnotations) {
    let tags = ”;
    let user = ”;
    if(a.tags.length>0) tags = ‘ ‘ + (a.tags.map(t=> ‘[[‘+ t + ‘]]’)).join(‘ ‘);
    if(insertUser) user = ‘ (‘ + a.user.replace(‘acct:’,”).replace(‘@hypothes.is’,”) + ‘)‘;
    if(a.text) tR += ${a.text}n&amp;mdash;[[${user}]]nn;
    tR += ## Source n;
    tR += &gt; ${a.highlight}[^1]nn;
    tR += [^1]: [${articleAnnotations[0].title}](${articleAnnotations[0].uri}) | [syndication link](tk) n;
    tR += n---ntags: nlinks: ${tags} n- broader terms (BT): n- narrower terms (NT): n- related terms (RT): n- used for (UF) or aliases: nconnected ideas: nMOC: nn---n;
    }
    %>

    Reply

  16. @dragonman225 This looks like a lot of the affordances made possible by the open source @Hypothes_is project whose data can also be ported to Obsidian, Roam, or your favorite note-taking app.
    Example: https://via.hypothes.is/https://julian.digital/2020/09/04/a-meta-layer-for-notes/
    See also: https://boffosocko.com/2020/08/29/a-note-taking-problem-and-a-proposed-solution/
    cc: @julianlehr

    ~ New essay ~
    –
    • What Hey gets right about note taking
    • Why notes need a spatial interface
    • The digital equivalent of sticky notes
    ↓https://t.co/T3t9dhISjv
    — Julian Lehr (@julianlehr) September 4, 2020

    Syndicated copies:

    Reply

  17. FWIW, I’ve been using Readwise for some time (for my primary source being highlights from Instapaper, which I read on an e-ink tablet) and the ability to auto-export to Obsidian is so useful it’s caused me to move my private notes management and drafting there.

    Reply

    1. Yes,that’s another good solution too. I passed on it as I didn’t want yet another paid subscription service. It is nice that they support several subsidiary services. I’d heard they were adding Hypithes.is, but haven’t checked into it yet.

      Reply

  19. Interesting read. I’m determined to work on reducing the “latency” when using digital note-taking systems.

    I recently spoke to someone using hypothesis and Readwise together, to import atoms into @protolyst for backlinking. This also retains the link to the source content


    Reply

    1. I can’t wait for improved user interfaces (especially for the masses) that make this sort of functionality easier to use and more commonplace.

      Syndicated copies:

      Reply

Likes

Mentions

Reposts

Leave a Reply

Your email address will not be published. Required fields are marked *

To respond to a post on this site using your own website, create your post making sure to include the (target) URL/permalink for my post in your response. Then enter the URL/permalink of your response in the (source) box and click the 'Ping me' button. Your response will appear (possibly after moderation) on my page. Want to update or remove your response? Update or delete your post and re-enter your post's URL again. (Learn More)