Creating a commonplace book or zettelkasten index from Hypothes.is tags

I thought it might be useful to have a relatively complete list of cross linked topical headings in my digital notebook (currently Obsidian) which is a mélange of wiki, zettelkasten, journal, project management tool, notebook, and productivity tool. First, let’s be honest that mélange is too poetic based on what I see of how others use Obsidian and similar tools. My version is structured to have very clear delineations between these forms even though I’m using the same tool for various functions.

I find that indexed subject headings can be useful for creating links between my wiki-like pages as well as links between atomic ideas in my digital zettelkasten. Gradually as one’s zettelkasten becomes larger and one works with it more, it becomes easier to recall individual ideas and cross link them. Until this happens or for smaller zettelkasten it can be useful to cross reference subject headings from one zettel to see what those link to and use those as a way to potential create links to other zettels. This method can also be used as a search/discovery aide for connecting edge ideas in new areas to pre-existing portions of one’s zettelkasten as well. Of course at massive scale with decades of work, I suspect this index will have increased value as well.

I don’t hear people talking about these types of indices for their zettelkasten in any of the influencer spaces or on social media. Are people simply skipping this valuable tool? For those enamored of Niklas Luhmann, we should mention that having and maintaining a subject index was a powerful portion of his system, even if the digitized version of his zettelkasten hasn’t yet been fully digitized. I haven’t seen the whole collection myself, but based on the condition of some of the cards in his index, Luhmann heavily used his subject index. (Note to self: I wonder what his whole system would look like in Obsidian?) Having a general key word/subject heading/topic heading index of all the material in one’s system can be very useful for general search and discovery as well. This is one of the reasons that John Locke wrote about a system for indexing one’s commonplace book in 1685. His work here is likely the distal reason Luhmann had one in his system.

Systems that have graphical knowledge graphs may make this process easier as one can look from one zettel out one or two levels to see where those link to.

Since such a large swath of my note taking practice starts by using Hypothes.is as my tool of choice, I’m able to leverage several years of using it to my benefit. Within it I’ve got 9,314 annotations, highlights, and bookmarks tagged with over 3,326 subject headings as of this writing.

To get all my subject heading tags, I used Jon Udell’s excellent facet tool to go to the tag editing interface. There I entered a “max” number larger than my total number of annotations and left the “tag” field empty to have it return the entire list of my tags. I was then able to edit a few of them to concatenate duplicates, fix misspellings, and remove some spurious tags.

An alternate way of doing this is to use a method described in this GitHub issue which shows how to get the tags out of local storage in your web browser. Your mileage may vary though if you use Hypothes.is in multiple browsers, which I do.

I moved this list from the tag editor into a spreadsheet software to massage the list a bit, clean up any character encodings, and then spit out a list of [[wikilinked]] index keywords. I then cut and pasted it into my notebook and threw in some alphabetical headings so that I could more easily jump around the list.

Now I’ve got an excellent tool and interface for more easily searching and browsing the various areas of my multi-purpose digital notebook.

I’m sure there are other methods within various tools of doing this, including searching all files and cutting and pasting those into a page, though in my case this doesn’t capture non-existing files. One might also try a search for a regex phrase like: /(?:(?:(?:<([^ ]+)(?:.*)>)\[\[(?:<\/\1>))|(?:\[\[))(?:(?:(?:<([^ ]+)(?:.*)>)(.+?)(?:<\/\2>))|(.+?))(?:(?:(?:<([^ ]+)(?:.*)>)\]\](?:<\/\5>))|(?:\]\]))/ (found here) or something as simple as /\[\[.*\]\]/ though in my case they don’t quite return what I really want or need.

I’ll likely keep using more local search and discovery, but perhaps having a centralized store of subject headings will offer some more interesting affordances for search and browsing?

Have you created an index for your system? How did you do it?

Unable to search or find public replies to annotations in public stream

Filed an Issue GitHub - hypothesis/client: The Hypothesis web-based annotation client. (GitHub)
The Hypothesis web-based annotation client. Contribute to hypothesis/client development by creating an account on GitHub.

Replies (with or without tags) to primary/original annotations are unable to subsequently be found in the main public stream or via search at https://hypothes.is/search.

Steps to reproduce

  1. Make a reply to any public annotation (with or without tags)
  2. Use https://hypothes.is/search to search the username of the reply or one of the original tags
  3. The reply can’t be found

The original (more complicated) example that uncovered the issue

From https://doi.org/10.6092/issn.1971-8853/8350 which redirects to https://sociologica.unibo.it/article/view/8350, I can click on the pdf icon to get to https://sociologica.unibo.it/article/view/8350/8272 which I can download locally and then reopen in Chrome to annotate with the Hypothes.is client.

I was able to make an original public annotation: https://hypothes.is/a/Nysv1HyTEeyaC2cnv3ZCPQ

Having subscribed to my public individual user feed, this annotation (via the annotation permalink and not via the original document) was found in Ton Zijlstra‘s RSS reader, and he was able to reply to it: https://hypothes.is/a/p3uUBJc8EeyuRmfRyGEGfQ.

Oddly the URL https://sociologica.unibo.it/article/view/8350/8272 when activated for Hypothes.is doesn’t show any of the annotations though I would suspect that the .pdf fingerprint should match that of the downloaded and annotated version. Alternately visiting https://uni-bielefeld.de/soz/luhmann-archiv/pdf/jschmidt_niklas-luhmanns-card-index_-sociologica_2018_12-1.pdf shows 51 annotations in the Chrome extension, though none of them are visible and the .pdf file doesn’t load on the page which returns a 404. Ton Zijlstra, having none of these URLs would otherwise not have been able to find or reply to annotations I’ve made other than having the original pointer via his RSS feed.

This last part non-withstanding, after making his reply to my annotation (directly at https://hypothes.is/a/Nysv1HyTEeyaC2cnv3ZCPQ), Ton Zijlstra is now no longer able to find his original annotation in the https://hypothes.is/search online interface. It’s as if it’s completely disappeared as the main web search interface is unable to find it via username and/or tags and (likely by design) the main public thread only shows top level annotations and not replies.

I’ve tried some similar experiments on my own replies to annotations. I’m unable to search my own annotations (via https://hypothes.is/users/chrisaldrich) or use either a user-based or tag-based search to find those same annotations after they were made, thus they’re essentially lost to me and others unless I can find the original document and trace my way back to them. These replies are obviously available via feeds (RSS/ATOM) and the API (using the urn:x-pdf:471902ab75f5683c53518d14f95f0dfe key), but they are essentially lost to the vast number of users who won’t have recourse to these methods.

Similarly searching Ton Zijlstra’s user name: https://hypothes.is/users/tonz, one will see no public annotations despite his public reply to a public annotation. The reply can be found at https://hypothes.is/stream.atom?user=tonz and via API calls.

Expected behaviour

After having made a reply to an annotation (with or without tags), one should expect to be able to search their own annotations or specific tags and find those public replies to annotations again.

Whether or not the main web stream (https://hypothes.is/search) filters out replies, they should still be able to be found via subsequent direct search.

Actual behaviour

Searching for one’s previous replies, via user, tag, or otherwise doesn’t find them, though they certainly exist and are findable in feeds and API.

Additional details

Related, possibly helpful for the above

Browser/system information

I’ve tried on other platforms and browsers and platforms with similar results, but I’m using Windows 10 and see the same behavior in both Chrome (Version 98.0.4758.102 (Official Build) (64-bit)) and Firefox (97.0.1 (64-bit)).

I’m excited to join Dan Allosso‘s book club on How to Take Smart Notes as a means of turning my active reading, annotating, and note taking into papers, articles and books using Obsidian.md and Hypothes.is

Details: 

cc: Ian O’Byrne, Remi Kalir

Another really clever reimagining of the @Hypothes_is UI from @peterhagen_. Lots of color! A graph of annotations over time! And you can annotate on the page.

Try it at https://annotations.lindylearn.io/page/?url=https://www.theatlantic.com/magazine/archive/1945/07/as-we-may-think/303881/

Alternate Hypothes.is UI featuring the Atlantic article As We May Think with rainbow colored highlights on one side and annotations to the right. At the top is a graph of the article's popularity showing the number of annotations over time.

Replied to a tweet by dragonman225 (Twitter)
@dragonman225 This looks like a lot of the affordances made possible by the open source @Hypothes_is project whose data can also be ported to Obsidian, Roam, or your favorite note-taking app.
Example: https://via.hypothes.is/https://julian.digital/2020/09/04/a-meta-layer-for-notes/
See also: https://boffosocko.com/2020/08/29/a-note-taking-problem-and-a-proposed-solution/
cc: @julianlehr

Replied to a thread by Roy Scholten and Sonja Drimmer (Twitter)
@Hypothes_is, you guys are working on this, right? 😜

Annotation Colors Using Hypothes.is: Problems and Solutions

I frequently see the feature request from Hypothes.is users to have the tool add the ability to choose different colors for highlights.

The first occurrence of the issue is probably documented here: https://github.com/hypothesis/vision/issues/123, by the head of the company.

I suspect it may take a while before such a color feature might be built in, if ever. (Here I’ll note that I don’t work for or speak for the company or any of the other open source developers on the project, but I am one what one would consider a “heavy user”.) If they have the time (I know they’re very busy), perhaps they may chime in with a potential roadmap or other ideas.

Color highlights are a difficult user interface problem

While I’m thinking about it, in an academic context for students, colors may be slightly better indicators of different users’ annotations of a particular text as a means of differentiating one annotator from another more subtly, particularly on texts that are extensively marked up.

Just this difference points out what a mixed bag of functionality colored highlights brings from a usability, design, and user interface perspective. While colored highlights is a seemingly “simple” sounding feature in the analog world where a single document is only annotated by one user, mapping it into a digital shared context is a difficult engineering problem to navigate and solve for. What if your color meanings aren’t the same as those of another?, for example.

While colors can be useful for individuals, do they have the same place in a social annotation product?

I already find it difficult to annotate heavily annotated pages that all use the same color, much less a rainbow of others’ colors. (If this is also you, I’ll note that there’s a handy “eye” icon in the annotation drawer that will allow you to turn them on/off.)

Potential Color Highlight Hacks

While the value of colors may be useful in some contexts, you could potentially use a few other features, functionality, and methods to creatively achieve a similar feature in Hypothes.is for yourself today. Below are a few potential creative “hacks” that some might try.

Use the tagging functionality

You could use the tagging system to create specific tags to stand in for your desired colors: As an example, in some systems I might use the following color designators:

  • Yellow—general highlights and highlights which don’t fit under another category below
  • Orange—Vocabulary word; interesting and/or rare word
  • Green—Reference to read
  • Blue—Interesting Quote
  • Gray—Typography Problem
  • Red—Example to work through

Instead of colors in Hypothesis, for example, one could use the tags “words” or “vocabulary”, “reference” or “citation“, “typo“, “quotes“, or “examples” to stand in for these particular “colors” respectively. I sometimes practice some of these which you can find by clicking on the links, though you may note that in practice I use other tags for them.

In some sense, this is what the software would be doing, particularly with regard to search for these after-the-fact. If you wanted a list of all your “citations” for example, you’d have to search for the color for that and be able to find them all, presuming this search functionality existed with such a color feature. This isn’t really much different than simply tagging all those particular highlights with words like “citation” or “reference” in the first place.

Use the Group Functionality

You could created different “groups” (private or public) to stand in for the colors you wish you had, thus a “yellow group” could be used for one “color” of highlight and a “green group” for another. ( See Annotating with Groups for more details.)

Switching between groups for annotating isn’t going to be drastically different than a user interface for switching colors of highlighter. The one drawback (or perhaps it’s a feature?) here is that you will only be able to see one “color” at a time.

Roll your own solution with open source

As ever, with some work, you could self-host the open source software and modify your copy to add this functionality in for yourself.

Some clever hacks in your browser with CSS might also give you your preferred output. I know some users have done custom work to the Hypothes.is UI in the past: eg. https://tomcritchlow.com/2019/02/12/annotations/, see his gist at the bottom of the post.

Another custom solution which may give you ideas can also be found at https://web.hypothes.is/blog/do-it-yourself-anchoring-and-the-evolution-of-the-hypothesis-toolkit/.

Perhaps adding custom classes on the tags or usernames might allow people the ability to target highlights on a page so that one could define custom CSS rules for each highlight using either usernames of tags as well? Of course, just like the “eye icon” described above, I’m sure there are times that people will appreciate the ability to turn these colors on and off. I personally don’t want the clean interface dressed up in Josephs Amazing Technicolor Annotation Dreamcoat.

Other solutions and problems?

Are there ideas or potential solutions for color highlights I’ve missed? How about design problems that might be encountered in implementing color-coded highlights in the older single document/single user model being transferred to a multi-user space with infinite scale? Is color the best and most accessible solution? Are there better things that could be done with color in the product?

Feel free to comment below with your ideas or links to examples.

Interested in other Hypothes.is hacks, tips, and ideas? Try browsing my Hypothes.is archive.

Annotated a tweet by @AmberRegis (Twitter)
I’d probably have done it digitally with @Hypothes_is to share with others, but kudos to those who can still fathom the analog.
https://via.hypothes.is/https://www.gutenberg.org/files/145/145-h/145-h.htm
Replied to a thread by Annie Murphy Paul and Howard Rheingold (Twitter)
Anecdotally, I’ve seen this sort of behavior happening in the margins of ethical educational tools like @Hypothes_is. Some of them are in smaller groups and private groups, which stay hidden. I try to do much of my thinking & commenting there in public: https://hypothes.is/users/chrisaldrich
Replied to Twitter thread by CatoMinor & Flancian (Twitter)
Me too:

  • https://forum.obsidian.md/t/using-hypothes-is-to-quickly-place-notes-into-my-digital-notebook/5010
  • https://boffosocko.com/2021/07/08/hypothes-is-obsidian-hypothesidian-for-easier-note-taking-and-formatting/; or
  • https://electricarchaeology.ca/2021/02/14/from-hypothesis-annotation-to-obsidian-note/
Read 25 Years of Ed Tech – Blogs by JR DingwallJR Dingwall (jrdingwall.ca)
This week I was able to catch up a bit on some podcasts I subscribe to. One of the casts I’ve been enjoying lately is 25 Years of Ed Tech, a serialized version of Martin Weller’s book by the same title. Now audio books are plenty good by themselves, but this particular podcast has an addition episode per chapter called “between the chapters” where a host interviews members of the ed tech community (those around Martin in some way) about the topic of the previous chapter. This week was all about blogs.
JR writes about some of his journey into blogging. I appreciate some of the last part about the 9x9x25 blogs. For JR it seems like some smaller prompts got him into more regular writing.

He mentions Stephen Downes‘ regular workflow as well. I think mine is fairly similar to Stephen’s. To some extent, I write much more on my own website now than I ever had before. This is because I post a lot more frequently to my own site, in part because it’s just so easy to do. I’ll bookmark things or post about what I’ve recently read or watched. My short commentary on some of these is just that—short commentary. But occasionally I discover, depending on the subject, that those short notes and bookmark posts will spring into something bigger or larger. Sometimes it’s a handful of small posts over a few days or weeks that ultimately inspires the longer thing. The key seems to be to write something.

Perhaps a snowball analogy will work? I take a tiny snowball of words and give it a proverbial roll. Sometimes it sits there and other times it rolls down the hill and turns into a much larger snowball. Other times I get a group of them and build a full snowman.

Of course lately a lot of my writing starts, like this did, as an annotation (using Hypothes.is) to something I was reading. It then posts to my website with some context and we’re off to the races.

It’s just this sort of workflow that I was considering when I recently suggested that those using annotation as a classroom social annotation tool, might also consider using it to help students create commonplace books to help students spur their writing. The key is to create small/low initial stakes that have the potential to build up into something bigger. Something akin to the user interface of Twitter (and their tweetstorm functionality). Write a short sentence or two on which you can hit publish, but if the mood strikes, then write another, and another until you’ve eventually gotten to something that could be a blog post (or article). Of course if you do this, you should own it.

This is also the sort of perspective which Sönke Ahrens takes in his excellent book How to Take Smart Notes: One Simple Technique to Boost Writing, Learning and Thinking – for Students, Academics and Nonfiction Book Writers, though there he’s prescribing something for general note taking when I might suggest it’s a prescription for a pedagogy behind living and writing.

Read Creating a Commonplace Book (CPB) by Colleen E. KennedyColleen E. Kennedy
During the sixteenth and seventeenth centuries, one of the most important tools of a reader or writer was a commonplace book (CPB). Peter Beal, leading expert on English manuscript studies, defines a commonplace book as “a manuscript book in which quotations or passages from reading matter, precepts, proverbs and aphorisms, useful rhetorical figures or exemplary phrasing, words and ideas, or other notes and memoranda are entered for ready reference under general subject headings.” Your sources can include, first and foremost, the assigned readings and supplementary materials, as well as any other useful texts you come across. I encourage you to supplement CPB entries with extra-curricular material: quotations from readings for other classes, lyrics from songs, lines from movies, tweets with relevant hashtags, an occasional quotation from a classmate during discussion, etc. These extra-curricular commonplace passages, however, are in addition to and not in place of the required passages as described below.
I love this outline/syllabus for creating a commonplace book (as a potential replacement for a term paper).

I’d be curious to see those who are using Hypothes.is as a social annotation tool in coursework utilize this outline (or similar ones) in combination with their annotation practices.

Curating one’s annotations and placing them into a commonplace book or zettelkasten would be a fantastic rhetorical exercise to extend the value of one’s notes and ideas.

Annotated Helping Hands on the Medieval Page by Erik Kwakkel (medievalbooks)
We are taught not to point. Pointing with your finger is rude, even though it is often extremely convenient and efficient. Medieval readers do not seem to have been hindered by this convention: in …
👈

As I’m thinking about this, I can’t help but think that Hypothes.is, if only for fun, ought to add a manicule functionality to their annotation product.

I totally want to be able to highlight portions of my reading with an octopus manicule!

I can see their new tagline now:

Helping hands on the digital page.

I’m off to draw some octopi…

I take a lot of notes, the majority of them in public using Hypothes.is. I follow a handful of others’ public notes myself, but until today I didn’t think anyone was following back or reading.

I wish this were a more common practice.

https://diggingthedigital.com/abonneren-op-aantekeningen/